CSL-SHARE: A Multimodal Wearable Sensor-Based Human Activity Dataset

Liu, Hui; Hartmann, Yale; Schultz, Tanja

doi:10.3389/fcomp.2021.759136

DATA REPORT article

Front. Comput. Sci., 15 October 2021
Sec. Mobile and Ubiquitous Computing
Volume 3 - 2021 | https://doi.org/10.3389/fcomp.2021.759136

CSL-SHARE: A Multimodal Wearable Sensor-Based Human Activity Dataset

Hui Liu*

Yale Hartmann

Tanja Schultz

Cognitive Systems Lab, Department for Mathematics and Computer Science, University of Bremen, Bremen, Germany

1 Background

In this digital age, human activity recognition (HAR) plays an increasingly important role in almost all aspects of life to improve people’s quality of life, such as auxiliary medical care, rehabilitation technology, and interactive entertainment. Besides external sensing, sensor-based internal sensing for HAR is also intensively studied. A large body of research involves recognizing various kinds of everyday human activities, including walking, standing, jumping, and performing gestures. HAR research relies on large amounts of data, which includes the collection of laboratory data that meet in-house research goals, as well as the usage of external and public databases to verify models and methods. Therefore, data collection is an essential part of our entire HAR research work, for which we will detail this extensive progress.

Many public HAR datasets are available online, providing various sorts of collected data, some of which have some similarities with our in-house data acquisition in terms of purpose, sensor selection, or protocol design. For instance, the Opportunity benchmark database (Chavarriaga et al., 2013) contains naturalistic daily living activities recorded with a large set of on-body sensors. The UniMiB SHAR dataset (Micucci et al., 2017) includes 11,771 samples of both human activities and falls divided into 17 fine-grained classes. The GaitAnalysisDataBase (Loose et al., 2020) contains 3D walking kinematics and muscle activity data from healthy adults walking at normal, slow or fast pace on the flat ground or at incremental speeds on a treadmill. The RealWorld dataset (Sztyler and Stuckenschmidt, 2016) covers acceleration, GPS, gyroscope, light, magnetic field, and sound level data of the activities climbing stairs down and up, jumping, lying, standing, sitting, running/jogging, and walking of 15 subjects. The FORTH-TRACE dataset (Karagiannaki et al., 2016) is collected from 15 participants wearing five Shimmer wearable sensor nodes on the left/right wrist, the torso, the right thigh, and the left ankle. The ENABL3S dataset (Hu et al., 2018) contains bilateral electromyography (EMG) and joint and limb kinematics recorded from wearable sensors for ten able-bodied individuals as they freely transitioned between sitting, standing, and five walking-related activities.

In this article, we disclose our in-house collected sensor-based dataset, CSL-SHARE (Cognitive Systems Lab Sensor-based Human Activity REcordings). Based on the improvement of the recording plan and organization through the experience gathered from the pilot datasets’ collection of CSL17 (one subject, seven activities of daily living, 15 minutes) and CSL18 (four subjects, 18 activities of daily living and sports, 90 minutes), the CSL-SHARE dataset covers 22 types of activities of daily living and sports from 20 subjects in a total time of 691 minutes, of which 363 minutes are segmented and annotated. In this dataset, we used two triaxial accelerometers, two triaxial gyroscopes, four surface electromyography (sEMG) sensors, one biaxial electrogoniometer, and one airborne microphone integrated into a knee bandage, bringing the total number of channels to 19, as these sensors can provide usable and reliable biosignals for HAR research, gait analysis, and health assessment according to existing studies, such as Whittle (1996), Rowe et al. (2000), Mathie et al. (2003), Kwapisz et al. (2010), Rebelo et al. (2013), and Teague et al. (2016). We also tried to use a piezoelectric microphone and a force sensor for sensing the acoustic and physical pressure signals from the knee during the acquisition. Nevertheless, in subsequent analysis and research, we did not have evidence to support their contribution to HAR research. Therefore, we removed these two channels of signal from the public dataset. In addition, although our two pilot datasets mentioned above, CSL17 and CSL18, are not publicly available due to the relatively smaller data volume, they can also be obtained from us for scientific research purposes.

2 Dataset Description

2.1 Devices, Sensors, and Sensor Placement

We chose the biosignalsplux Researcher Kit¹ with the selected various types of sensors supplied together. One hub from the kit records biosignals from eight channels, each up to 16 bits, simultaneously. Since we needed to record over 20 channels, we connected 3 hubs via synchronization cables that connect the hubs and synchronize all channels automatically between the hubs at the beginning of each recording session, which ensured the synchronicity during the entire recording sessions.

The sensor positioning on the right-leg-worn knee bandage was decided in collaboration with kinesiologists of the Institute of Sport and Sports Science at Karlsruhe Institute of Technology based on their substantial research experience in knee kinematics (Stetter et al., 2018; Stetter et al., 2019) to capture ambulation activities. The CSL-SHARE sensor positions and their measured muscles/body parts are listed as follows:

• Triaxial accelerometer 1 (upper): thigh, proximal ventral

• Triaxial accelerometer 2 (lower): shank, distal ventral

• Triaxial gyroscope 1 (upper): thigh, proximal ventral

• Triaxial gyroscope 2 (lower): shank, distal ventral

• EMG 1 (upper-front): musculus vastus medialis

• EMG 2 (lower-front): musculus tibialis anterior

• EMG 3 (upper-back): musculus biceps femoris

• EMG 4 (lower-back): musculus gastrocnemius

• Biaxial electrogoniometer (lateral): knee of the right leg

• Airborne microphone (lateral): knee of the right leg.

EMG and microphone signals were recorded with a sampling rate of 1000 Hz and all other signals with 100 Hz. The low-sampled channels at 100 Hz were up-sampled to 1000 Hz to be synchronized and aligned with high-sampled channels. All channels have a quantization resolution of 16 Bits.

2.2 Software for Data Acquisition

We developed a software called Activity Signal Kit (ASK) with a Graphical User Interface (GUI) and multi-functionalities using the driver library provided by biosignalsplux, as introduced in (Liu and Schultz, 2018). ASK automatically connects and synchronizes several recording hubs, then collects up to 24-channel sensor data from all hubs simultaneously and continuously. All recorded data are archived automatically in HDF5 files with the filename of dates and timestamps for further research.

A protocol-for-pushbutton mechanism of segmentation and annotation has been implemented in the ASK software, which will be introduced in Section 2.3. Moreover, the baseline ASK software also provides the functionalities of digital signal processing, feature extraction, modeling, training, and recognition by applying our in-house developed HMM-based decoder BioKIT (Telaar et al., 2014).

2.3 Annotation and Segmentation

The task of segmentation in HAR research is to split a relatively long sequence of activities into several segments of single activity, while annotation is the process of labeling each segment, such as “walk,” “run,” “stand-to-sit,” among others. Segmentation, which can be performed manually (Rebelo et al., 2013), semi-supervised Barbič et al. (2004), or automatically (Guenterberg et al., 2009; Micucci et al., 2017), is undoubtedly a prerequisite for annotation, and its output will be input for digital signal processing and feature extraction. Annotation, which can be performed directly after each segmentation subtask, helps two follow-up operations: training and evaluation.

In our research, we applied the pushbutton of the biosignalsplux Research Kit in our proposed semi-automated segmentation and annotation solution. In subsequent research, the applicability of the semi-automated segmented data has been verified for our research purpose during numerous experiments (see Section 4), so we have been applying this mechanism to our successively acquired datasets.

The so-called protocol-for-pushbutton mechanism of segmentation and annotation has been implemented in the ASK software (Liu and Schultz, 2018). When the “segmentation and annotation” mode is switched on during the data acquisition, a predefined activity sequence protocol will be loaded into the software, which prompts the user to perform the activities one after the other. Each activity is displayed on the screen one-by-one while the user controls the activity recording by pushing, holding, and releasing the pushbutton. The user follows the instructions of the software step-by-step. For example, the prompted activity states “walk,” the user sees the instruction “Please hold the pushbutton and do: walk.” The user prepares for it, then pushes the button and starts to “walk.” She/he keeps holding the pushbutton while walking for a duration at will, then releases the pushbutton to finish this activity. With the release, the system displays the next activity instruction, e.g., “stand-to-sit,” the process continues until the predefined acquisition protocol is fully processed.

The ASK software records all timestamps/sample numbers of each button push and button release during the data recording. These data are archived in CSV files as segmentation and annotation results for each activity. Since we synchronized all data at 1,000 Hz, each sample represents data from 1 ms. For example, a line “sit, 3647, 6163” in a CSV file means that the activity segment labeled “sit” lasts 2,516 samples from the timestamp 3,647 to 6,162, which corresponds to 2.516 seconds. The corresponding 2,516 samples form one segment for training the activity model “sit,” or for the recognition results evaluation. The protocol-for-pushbutton mechanism was implemented to reduce the time and labor costs of manual annotation. The resulting segmentations are excellent and required little to no manual correction, and lay a good foundation for subsequent research. Nevertheless, this mechanism has some limitations:

• The mechanism can only be applied during acquisition and is incapable of segmenting archived data

• Clear activity start-/endpoints need to be defined, which is impossible in cases like field studies

• Activities requiring both hands are not possible due to participants holding the pushbutton

• The pushbutton operation may consciously or subconsciously affect the activity execution

• The participant forgetting to push or release the button results in subsequent segmentation errors.

None of these limitations, except forgetting to release the pushbutton, hold in a laboratory setting with clear instructions and protocols. Hence, misapplication of the pushbutton was addressed by real-time human monitoring of the incoming sensor signals, including the pushbutton channel, during acquisition. Additionally, a mobile phone video camera for post verification and adjustments was used (see Section 2.4).

2.4 Post Verification

Although the “segmentation and annotation” mode of the ASK software was switched on to segment and annotate the recorded data efficiently, a mobile phone video camera was used in addition to record the whole biosignal acquisition sessions to manually correct the misoperation of pushing/holding/releasing after the data recording.

After each recording event with one subject, the collected data and the automatically generated segments with annotation labels were examined thoroughly based on the video. Segments with minor human-factor errors were corrected by shifting the start-/endpoints forward/backward a short distance manually, while segments with problems that cannot be easily corrected were discarded, which is one of the reasons leading to the slight divergence among the activity occurrences in Table 1 (see Section 2.8 for another reason). A script to automatically detect the activity length outlier was also implemented to assist the post verification. After finishing the correction and verification, we deleted all recorded videos to preserve privacy.

TABLE 1

TABLE 1. Statistics of segmented corpus in the CSL-SHARE dataset. The minimum (Min), maximum (Max), mean, and standard deviation (std) values are in seconds. #Seg: number of segments.

2.5 Activities and Protocols

The CSL-SHARE dataset was recorded in a controlled laboratory environment at the Cognitive Systems Lab, University of Bremen, comprised of 22 daily living and sports-based activities. The acquisition protocols of CSL-SHARE recording events were strictly and normatively designed. The body steering angles and the number of steps related to the activity parameters are restricted. Most of the acquisition protocols contain only one activity. However, there are two protocols with two activities and one protocol with four activities because these activities can be practically and logically recorded one after another in a sequence, which also keeps the balance of the activity occurrences. To follow the logical sequence of the activities and the protocol-for-pushbutton segmentation and annotation mechanism (see Section 2.3), the order of the activities in these three multi-activity protocols must be observed during recording.

Figure 1 illustrates the diagrammatic sketch of all recording protocols, helping more intuitively understand the recording procedure and activity details. The 22 activities and the 17 acquisition protocols are described as follows:

•Protocol 1: walk (Figure 1A)

<push + hold> walk forward with three gait cycles, left foot starts, i.e., left-right-left-right-left-right <release> → (turn around in place) → …

20 repetitions (20 activities with 60 gait cycles) per subject.

Note: the “turn around in place” between two “walk”/“run” activities is due to the limited space in our laboratory.

•Protocol 2: walk-curve-left (90°) (Figure 1B)

<push + hold> turn left 90° with three gait cycles, left foot starts <release> → (turn left 90° in place) → …

20 repetitions (20 activities with 60 gait cycles) per subject.

•Protocol 3: spin-left-left-first (Figure 1C)

<push + hold> turn left 90° in one step, left foot starts <release> → …