Abstract
Wearable-based human activity recognition (HAR) has emerged as a valuable method for capturing activities across diverse domains, including rehabilitation, occupational ergonomics, sports, and human-computer interaction (HCI). While recognition performance can be significantly enhanced by leveraging multiple complementary sensors, this approach requires accurately synchronized time bases across all devices. Although previous studies on synchronization in HAR suggested that sub-second accuracy is advisable while sub-100 ms accuracy is unnecessary, the specific effect of time discrepancies on machine learning models has, so far, remained unexplored. We address this gap by introducing an experimental paradigm for systematically evaluating the impact of time discrepancies in multi-wearable HAR, which we evaluated in two experiments. In our first experiment, we use the example of multi-stage temporal convolutional networks (MS-TCN) for sequence-to-sequence action segmentation, simulating the time discrepancies of time offset and clock skew via rational resampling. Our evaluation spanned 30,025 training and validation runs across different model configurations, totaling over one million core-hours of computation. Our results reveal that time offsets larger than 167 ms should be avoided in training datasets, and offsets beyond 333 ms can already significantly degrade HAR performance for typical activities of daily living (ADLs). Subsequently, we performed a second experiment focusing on the impact of time offsets on inference in models trained on synchronized datasets. Our evaluation spanned temporal convolutional networks, LSTMs, and Transformer architectures across five architectural configurations, each with two different temporal input lengths. The results indicate that LSTMs for action segmentation are more robust to desynchronization, while other architectures exhibited a marked performance degradation beyond desynchronization offsets spanning 167 ms. Our findings have implications for the design and deployment of multi-wearable HAR systems and may extend to other multi-sensor contexts.
1 Introduction
Human activity recognition (HAR) involves identifying and categorizing human movements from sensor data, ranging from general activities of daily living (ADL) to domain-specific activities. While simple tilt switches () were popular in the early days of wearable sensing, inertial measurement units (IMUs) have since emerged as a popular choice for HAR due to their small form factor, low power consumption, and ability to capture rich kinematic information. The field has gained substantial momentum over the past two decades, driven by applications such as sports (), environment interactions (), or manufacturing (). The evolution of HAR methodologies reflects broader trends in sensor technology and computational intelligence. As the demand for movement assessment grows, wearable sensing has evolved from simple rule-based systems that provide users with basic metrics, such as the number of steps, to sophisticated machine learning pipelines capable of recognizing complex activity patterns (). The performance of these models depends on the completeness of the available data, with sensors mounted at a single body location providing only a local and therefore limited perspective on whole-body movement. Consequently, using multiple distributed devices to capture complementary signals from different body locations has been shown to improve the recognition of complex and ambiguous movements (; ). This multi-sensor paradigm aligns with the increasing sophistication of HAR models, which benefit from richer input data to distinguish fine-grained activities more effectively.
A fundamental challenge in multi-wearable systems arises from the independent operation of devices. Since multiple devices operate with clocks that run autonomously from each other, the inherent inaccuracy of local clock sources can degrade recording quality. As the timing error grows approximately linearly over time, this becomes an issue in longer recordings such as activity recognition datasets. Even typical CMOS MEMS oscillator with inaccuracies of ± 500 ppm () can lead to a temporal misalignment of about 150 ms after just 5 min of recording. Therefore, multi-wearable HAR requires a synchronization mechanism, otherwise time discrepancies would successively result in mismatched time recordings (). As online synchronization through wireless radio communication is usually not available due to costs and resource constraints (), recordings are often aligned offline in a post-processing step (; ), which results in lower synchronization accuracy (). In practice, the application of machine learning methods on data from multi-device systems suffers from these temporal inaccuracies and events' unmatched time, which inevitably results in blurry models and weak performance (; ). As a practical guideline, early research by suggested that a sub-second synchronization accuracy would be sufficient for the recognition of ADLs and that a sub-100 ms performance is unnecessary. However, the actual impact of time discrepancies on machine learning models for HAR has not been investigated systematically and empirically yet.
We aim to address this gap by a systematic evaluation of time discrepancies on HAR performance using temporal action segmentation and sequence-to-sequence models. In a two-staged experiment, we first investigated the impact of (1) time offset and (2) clock skew applied using the example convolutional neural networks (CNNs) by with (3) varying depth and (4) input lengths. In total, the evaluation spanned 30,025 training runs with different variations of these parameters, amounting to 1.2 million core-h computations. In the second stage of the experiment, we further investigated the impact of time offset on a larger set of models with different architectures during inference. This included architecture variants of MS-TCN () in two different configurations, DeepConvLSTM () with unidirectional and bidirectional LSTMs, and IF-ConvTransformer (). Each model was trained using two different temporal input lengths. The evaluation was performed using the public Opportunity Activity Recognition dataset (,), which contains hardware-synchronized sensor readings from five body-worn IMU sensors, comprising accelerometer and gyroscope recordings, which were modified according to the experiments through time offsets and rational resampling to simulate clock skew
Results from the first experiment reveal that a time offset of 167 ms between training dataset and data in deployment can already lead to a considerable performance drop. Moreover, we show that an offset of 333 ms can cause a drop in accuracy of 2.7 %, 6.6 % for F1 at 0.5 overlap, and 3.9 % for the edit distance, thus remarkably degrading HAR performance. At the same time, skew did not impact the model, even with input sequence lengths spanning up to 2,000 samples (66.67 s). Furthermore, results from our second experiment demonstrate that network architectures exhibit clear differences in their robustness to desynchronization. In particular, models based on LSTMs exhibited robustness without a noticeable performance drop up to a maximum desynchronization spanning around half a second between the five sensors. On the other hand, our experiment indicates that transformers might suffer from performance loss even with slight discrepancies. In evaluating the distribution of the discrepancies across sensors, we found the maximum offset discrepancy being the main driver for reduced performance. The findings offer practical implications for the design and deployment of multi-wearable HAR systems, and local standards for accuracy of synchronization in datasets, and may extend to other multi-sensor contexts. We conclude by discussing the practical impacts of improving synchronization in real-world applications.
The contribution of our work is threefold: We (1) propose an experimental paradigm for systematically evaluating the impact of insufficient device synchronization in multi-wearable HAR and (2) provide empirical insights from two different experiments spanning different desynchronization setups. Lastly, we (3) discuss the deployment implications of our research, highlight the need for adequate synchronization, and the importance of providing accuracy information.
2 Related work
2.1 Wearable human activity recognition
Early research relied on coarse data from rudimentary sensors like tilt switches (). The emergence of affordable micro-electro-mechanical systems (MEMS) accelerometers and IMUs then stimulated the development of computer-assisted methods for activity analysis. In general, wearable sensing encompasses a broad spectrum of approaches. Commercial devices frequently employ heuristic algorithms, such as thresholding and peak detection for tasks including step counting and simple motion event detection. While these approaches are computationally efficient and effective for small sets of well-defined tasks (), they are limited in their ability to distinguish between activities with complex patterns across multiple body parts. This has prompted research into machine learning techniques for the discriminative analysis of motion data.
Nowadays, the recognition of ADLs commonly involves machine learning to identify patterns in time series (; ; ). This development has been driven by the need for more objective and quantitative assessment methods than traditional observational procedures, particularly in healthcare and rehabilitation (). Early algorithms for wearable HAR employed signal processing techniques and handcrafted feature extraction, including time-domain features such as mean and standard deviation, frequency-based or duration-based features, followed by classification using support vector machines or decision trees ().
While these handcrafted feature-based approaches enabled first classification of more complex motion, their performance depends on the quality of manually engineered features (). More recently, deep learning approaches have become dominant in HAR research, as they enable training from raw data using architectures that involve models spanning from recurrent neural networks (; ; ), CNNs (; ; ), and lately Transformers (; ). Modern architectures for HAR often combine multiple architectural features, and in these networks, CNNs are typically integrated in the first layers to capture temporal patterns (). CNNs remain a popular choice due to their high capacity to extract distinctive features from raw data (), and the integration with other architectures such as LSTMs () or Transformers (; ) achieves cutting-edge performance.
Recently, the focus has begun to shift from coarse activity classification toward fine-grained action segmentation and temporal localization in continuous IMU streams. A classical activity recognition chain uses a sliding window of a fixed length and a certain overlap between segments (). However, determining the optimal size of the sliding window is challenging and often leads to the multi-class window problem, i.e., when the sliding window is large and covers a set of more than one action (). Conversely, action segmentation or temporal localization involves methods that do not impose fixed window lengths, and instead localize activities directly in the temporal data stream (; ). Such models can identify the moment a person begins an action, when they stop, or when they take a break during the activity.
As noted by , such approaches yield more coherent predictions with reduced fragmentation, while achieving performance comparable to traditional window-based classification. Beyond classification accuracy, segmentation enables a richer representation of user behavior by explicitly modeling temporal structure and action boundaries (). Recent work has been supported by the introduction of datasets with fine-grained annotations, such as WEAR (). Prior studies (; ) demonstrate that architectures originally developed for temporal action segmentation in other domains can be effectively transferred to IMU data with minimal adaptation. Similarly, in our previous work (), we show that models from the video domain, such as MS-TCN (), can be applied to sequence-to-sequence segmentation tasks directly using raw IMU data.
2.2 Synchronization in wearable human activity recognition
In wearable HAR, the information gathered with a solitary wearable device is often not rich enough to distinguish similar activities with complex and ambiguous signal patterns (). Therefore, employing multiple sensors with complementary perspectives has proven to be significantly more accurate than data only from single body parts (; ). The resource constraints of wearables necessitate either the development of efficient and streamlined algorithms, or the transfer of processing to a hub, e.g., a smartphone, or even the cloud (). Synchronization methods ensure that the data collected by multiple distributed devices with autonomous clocks are accurately aligned in time to allow for a meaningful analysis. The applied methods are typically categorized as online and offline approaches. Online synchronization methods rely on network communication to dynamically update, negotiate, and adjust devices' local time during runtime. However, off-the-shelf wearables used in HAR typically do not support synchronization during operation () as they would require specific built-in hardware or communication protocols whose overhead would exceed the energy budget (; ). Originated in HAR, offline methods identify and leverage external events in recordings, such as “clapping, jumping, and hitting a surface” (), but defer the synchronization to a data-driven post-processing step. Depending on sensor modality, sampling rate, and the leveraged signal's characteristics, offline approaches achieve different levels of accuracy. Initiated by , most approaches are motion-based with accuracies of 300 ms (), 49 ms (), 20 ms (), or even 8.6 ms (). Others leverages sporadic cough events with 46 ms (), artificial ECG landmark signals with 6.25 ms (; ), the natural human pulse with 2.9 ms (), or electromagnetic fields with 8-10 ms (; ) or even 2.6 ms ().
The important fundamental HAR research by mentions that “in realistic scenarios achieving high synchronization performance is not trivial” and the required synchronization accuracy largely depends on the target application. They formulated a well-considered rule of thumb stating that a sub-second accuracy is advisable for the recognition of “motion-related activities in daily life” while a sub-100 ms accuracy is “typically not needed”.
Beyond inaccuracies across the input signals, also the accurate alignment with ground truth plays an important role. Labels that are not placed well can result in the analysis window overlapping two or more actions, similar to the multi-class window problem (). While this inherently lowers accuracy, prior research has shown that it can partially be tackled with weakly supervised learning () or smooth activity transitions that are computed around the action boundaries in ground truth (). In contrast, the effect of the synchronization of multiple devices and its impact on HAR accuracy lacks empirical results that could guide practical applications.
2.3 Time discrepancy between wearables
The timestamps of measurements collected using multiple distributed wearable devices are inherently affected by time discrepancies that are observable as an unavoidable characteristic of any clocked system in comparison to a reference time. Among other factors, such as variable transmission times and fluctuating processing times (), the main cause for these discrepancies is the inherent inaccuracy of local, autonomous clock sources.
Oscillators show an accuracy, specified in parts per million (ppm), that typically ranges from ± 500 ppm of CMOS MEMS oscillators, over ± 20 ppm of conventional crystal oscillators (XO), to ± 5 ppm of temperature-compensated crystal oscillators (TCXO) (). Beyond the inherent inaccuracies from manufacturing and aging, their stability is also affected by various environmental factors, including temperature fluctuations, barometric pressure variations, shock and acceleration forces, magnetic field interference, and electromagnetic noise (; ). While oscillators with superior accuracy exist, they would entail higher costs and increased energy consumption (). Consequently, developers commonly choose a cost-effective compromise. Nevertheless, even with highly accurate clock sources, a deviation will sooner or later be measurable (). Therefore, the regular synchronization of all devices' local time is necessary.
In general, time discrepancies result in timestamps with scale and frequency inconsistency, drift, or even absence (). However, the terminology used to describe the accuracy of clock sources and the resulting time bases can vary across domains. Furthermore, the terms are sometimes used interchangeably and cause confusion. Therefore, in this work, we follow the terminology proposed by .
2.3.1 Offset
The time offset of two clocks Ci and Cj is given by their constant difference Ci(t)−Cj(t) over time t, measured in time units or phase degrees. The absolute timing error refers to the absolute reference clock Cr(t) = t and is denoted as Δti = Ci(t)−t, with Δti = 0 for an ideal, Δti>0 for an early, and Δti <0 a late clock.
2.3.2 Skew
The clock skew of two clocks Ci and Cj is given by the constant difference of their frequencies through the first derivative of the clock values with respect to t. Skew refers to phenomena that cause the clocks to not run at the same rate and, hence, gradually desynchronize, with Δfi <0 for a slow and Δfi>0 for a fast clock. Oscillators specify their characteristic deviation from the targeted center frequency fc through the ± peak variations ppm around fc in units of ppm, as defined in Equation 1:
The accumulated time offset Δt along the time period T caused by the clock skew of ppm is defined in Equation 2:
For example, a clock source with ± 500 ppm might gradually reach a time offset Δt= 150 ms after T≥ 5 minutes while a clock source with ± 20 ppm would take at least T≥ 125 minutes, and hence require fewer costly time synchronizations.
2.3.3 Drift
The clock drift among two clocks Ci and Cj is given by the differences of their second derivative with respect to t and refers to variations of skew due to environmental influences. In addition to manufacturing imprecision and the relatively small effect of aging (), temperature fluctuations are primarily responsible for non-linear drift in the range of ± 10–20 ppm ().
As these rely on regularly sampled time series with accurately matched time bases, the inadequate synchronization of time results in a misalignment of signal patterns, although they have been captured simultaneously. This disassociation of coincidences (; ) can result in wrong conclusions from the analyzed situation, and causes a loss of information and correlation between measurements for causation (). Therefore, time discrepancies inevitably result in a poor classification performance of machine learning models ().
3 Methodology
3.1 Modeling time discrepancies
Our research requires a careful consideration of how to realistically introduce time discrepancies into the original time series without downgrading data integrity and utility. In our initial paradigm for Experiment 1, published in , we proposed two processing strategies to implement and investigate time offset and clock skew with their different characteristics. These are then applied to the five available sensors in our dataset, by permuting the discrepancies to randomly allocate the discrepancy to the individual sensors. This resembles a uniform distribution and ensures maximum effect across the sensor channels. However, the equidistant, uniform allocation is not realistic for scenarios involving devices that exhibit clock skew of various degrees, which causes the local time bases to drift apart over time. Therefore, for Experiment 2, we decided to describe the discrepancy through the two parameters span and skew, to account for the shift of emphasis.
3.1.1 Time offset
The time offset is a constant toff that shifts the local time Ci(t) = Cr(t)+toff relative to the reference time Cr. Only integer multiples N∈ℕ+ of the sampling period Ts = 1/fs are used with N = [toff/Ts] to avoid the necessity of rational sampling. The process involves a shift only in the indices corresponding to the timestamps, thus preserving the integrity of the sampled data.
3.1.2 Clock skew
The introduction of the clock skew Δfs requires the modification of the original sampling rate fs by a small, non-integer fraction in the scale of ppm. As described by and , rational resampling is, therefore, necessary to insert or remove fractional, off-grid samples without introducing signal artifacts or aliasing. In the cascaded processing chain, first, an expander increases the sample coverage by Nup and interpolates the values between the original samples. Subsequently, a low-pass filter limits the signal bandwidth to the Nyquist frequency to prevent aliasing and hence smooth the signal. Eventually, a decimator removes the excessive samples by Ndown to obtain the targeted rate . Ideally, the rational factor Nup/Ndown is a common integer multiple of the clock skew Δfs, or reproduces it with sufficient accuracy, at least.
3.1.3 Uniform discrepancy allocation
To evaluate the impact of time discrepancies, care must be taken to consider realistic scenarios in which the discrepancies are modeled over the five sensor channels available in the dataset. For instance, if only one sensor is affected by the discrepancy, the value could directly be assigned to the sensor, e.g., A = [0, 0, 0, 0, 100]. In this way, sensor 5 would show a discrepancy of 100 ms relative to the others, while there are no relative differences across channels 1 to 4. However, this scenario would apply only in a very limited setting where all nodes share an identical time base, with one sensor presenting an outlier. To better reflect reality with a wide distribution of discrepancies, these are split across the sensor channels using an equidistant, uniform distribution Punif(x), as illustrated in Figure 1A and defined in Equation 5. Accordingly, a discrepancy with a standard deviation σ = ±50 can be modeled with values ranging from −50 ms to +50 ms over all sensors, e.g., A = [−50, −25, 0, 25, 50]. As the maximum relative discrepancy across the channels is still 100 ms, we define this range from the minimum Smin = min(A) to the maximum Smax = max(A), as given in Equation 4.
Figure 1
3.1.4 Skew normal discrepancy allocation
For Experiment 2, we searched for a more realistic model to distribute discrepancies across sensor channels. Therefore, we decided to map the initial uniform distribution onto a Gaussian normal distribution Pnorm(x), according to Equation 6. In this model, the minimum Smin and maximum Smax values are mapped to the standard deviation −σ and +σ, respectively, while the center is in line with the expected value μ. As illustrated in Figure 1B, the two remaining intermediate values are then positioned equidistantly at −σ/2 and +σ/2 to complete the distribution. Since the simulation is only sensitive to the relative distances of the discrepancies, the specific value of μ is irrelevant and just has to be at the center of the distribution, e.g., A = [-50, -25, 0, 25, 50] with S = 100, σ = ±50, and μ = 0 ⇔ A = [0, 25, 50, 75, 100] with S = 100, σ = ±50, and μ = 50.
For a small number of sensor channels, the discrepancy allocation through a normal distribution remains indistinguishable from the one of an equidistant, uniform distribution. However, transitioning to a normal distribution served as a critical intermediary step toward implementing a skew normal distribution. By eventually introducing the parameter skewness, we aim to simulate more realistic scenarios involving devices with larger clock skews, which cause the local time bases to drift apart over time. As illustrated in Figure 1C, this skew normal distribution results in a cluster of channels' discrepancies aligned more closely, while a few outliers exhibit considerably larger discrepancies. The skew normal distribution Pskew(x) is defined through its location ξ, scale ω, and the skewness α, as given in Equation 7.
It is a composition of the standard normal probability distribution function (PDF) ϕ(x), with μ = 0 and σ = 1, defined in Equation 8, and the cumulative distribution function (CDF) Φ(x), where erf is the common error function in Equation 9.
To determine and map the corresponding discrepancies, the probabilities are derived from the mode-normalized, relative standard normal distribution ϕ*(x) following Equation 10.
By applying the set of allocated values A* to this relative distribution, the mapped weights can be obtained, as presented in Equation 11.
In this allocation, the central value is scaled to unity, while the remaining sensor channels are assigned decreasing weights that follow the Gaussian curve as they move further from the center (μ = 0). This approach provides a symmetric, mathematically grounded framework for distributing discrepancy levels across the five available channels. Once the relative weights are determined, the values from ϕ*(A*) can be mapped onto the mode-normalized, relative skew normal distribution . This process ensures that the structural properties of the skewness are correctly captured while maintaining the intended relative proportions between the channels. Unfortunately, does not automatically maintain the span |S|. Therefore, the resulting skew normal distribution is scaled and shifted by modifying ξ and ω until the PDF precisely matches the targeted span |S|, as illustrated in Figure 1D. The characteristic distribution of the channels is not affected by this adjustment.
It is important to note that, in the context of this simulation, the mode-normalized versions of normal and skew normal distributions are easier to work with as the discrepancy at the mode (maximum peak) becomes a full multiplier of 1, while all other offsets are represented as a percentage of that peak.
As illustrated in Figures 2A–C, the impact of quantization becomes increasingly bigger as the span decreases. This quantization error originates in the conversion of the calculated allocations within the PDF, originally represented as floating-point values, into the rounded discrete integer values. To address this inherent inaccuracy, we applied dithering by artificially introducing uniform quantization noise, denoted as ϵ∈[−0.5, 0.5]. By incorporating ϵ before rounding the targeted real to the integer position, the allocation effectively jumps between neighboring values with a probability determined by the uniform noise distribution around the real boundary, thereby smoothing the transition from continuous values to the integer grid. While this smoothing causes the span to fluctuate slightly in size between |S|real−2ϵ ≤ |S|noise ≤ |S|real+2ϵ, the statistical mean of these quantization noise-induced, rounded integer spans |S|noise converges to the targeted span value |S|real, as determined by Equation 12. In contrast to this statistical smoothing, the constant rounding of the targeted positions would cause a consistent yet persistent over- or undervaluation |S|round.
Figure 2
3.2 Dataset and models
3.2.1 Dataset
Our experiment involved the public Opportunity Activity Recognition dataset by and , which contains activities performed by four subjects in six runs each, resulting in 25 h of data on 27,000 atomic ADLs. These were recorded with 72 sensors worn on the body, integrated in the environment, or in objects, and grouped in 15 wired and wireless networks. The dataset provides different levels of labels, which enable fine-grained, sample-level classification. We decided for the subset of labels for locomotion, which distinguishes between ADLs such as standing, walking, sitting, lying, and a “background” class, meaning neither of the actions.
While the first five runs were performed freely during the dataset recording, the sixth ‘drill' run followed a predefined protocol (), and hence was neglected to avoid any bias. The performed activities are imbalanced and unequally represented, with standing 34.68 ± 6.08 %, walking 20.80 ± 5.15 %, sitting 18.59 ± 4.69 %, lying 4.07 ± 1.51 %, and nothing 21.86 ± 6.08 %. The first lines of Figures 3A, B illustrate exemplary ground truth labels along the timeline.
Figure 3
Out of the 72 sensors, five commercial XSense IMUs were attached to a custom “motion jacket” made by . The sensor nodes were placed on the lower arms, the upper arms, and the back of the torso. As described by and , they are networked through RS-485 and assumed to be recorded simultaneously with the same local time and, therefore, to be adequately synchronized in hardware. The IMU sensors provide six inherently synchronized sensor channels of a 3-axis accelerometer and a 3-axis gyroscope. The terminating NaN values at the end of the recordings are discarded, but apart from that, the time series of the selected channels do not contain any missing values or labels.
3.2.2 Recognition approach and evaluation
To investigate the impact of time discrepancies on machine learning models, we opted for the action segmentation approach using sequence-to-sequence prediction. Unlike in a traditional window-based approach, where the performance can be affected through an incorrect choice of sliding window length, this approach is invariant to the length of the input, thus avoiding the multi-class window problem (). Lately, segmentation and temporal localization have been gaining interest in the community, as they provide more action context due to being able to model action boundaries (). Due to the sensitivity of our models to the choice of window size, and our hypothesis that desynchronization might particularly affect transitions between actions that window-based approaches cannot capture, we adopt a segmentation-based approach rather than classical window-based methods.
Sequence-to-sequence segmentation can be implemented using established HAR architectures, including LSTM-based models () and Transformer-based approaches (), originally designed for window-based classification. These architectures can be adapted by omitting temporal aggregation prior to prediction, enabling dense, sample-wise outputs and thus per-timestep labeling instead of a single prediction for the whole time window.
In line with approaches for action segmentation (), we used mean accuracy, edit score, and F1 score at different segment overlaps as evaluation metrics. Although accuracy, calculated as the correct assignment of each timestamp to a set of classes, is the most commonly used metric for action recognition, it does not capture the alignment of segments and is often overly impacted by long activities. Therefore, we also computed the F1 score at 0.1 and 0.5 overlap thresholds using the intersection over union (IoU) ratio, as described by , and the edit distance to account for the performance.
To ensure that all sensor data contributed to the model training, before applying the time discrepancies, we performed a verification step to assess each sensor's impact on model performance. This was performed by systematically omitting individual sensor channels during exploratory model training and then assessing them in the evaluation step. Reducing the input down to four sensors always reduced the model performance. On average, omitting a sensor reduced the validation accuracy by 1.28 ± 1.69 %, and the F1 score at 0.1 overlap by 8.85 ± 7.75 %, thus confirming that all sensors provide complimentary input to the model. Sensor 2 on the right arm had the largest influence on model accuracy with up to 3.5 % performance drop, while the sensor on the back has shown to reduce the F1 score at 0.1 overlap by up to 20 %. This demonstrates that, while sensors on arms help to distinguish activities apart, the sensor on the back has the potentially strongest contribution toward accurately identifying the temporal boundaries of the activities.
3.2.3 Sensor permutations
In settings with multiple sensors, time discrepancies are relative measures across the channels, and the allocation has to be modeled to adequately simulate realistic settings. However, with five example sensor channels, all possible sensor permutations equal to 5! = 120 variations. To manage this complexity and evaluate whether a reduced number of permutations is sufficient to yield convergence, we performed an evaluation for both time offset and clock skew. In this evaluation, we applied all 120 possible sensor permutations and evaluated the resulting accuracies of selected models, averaged over increasing numbers. A Gaussian smoothing filter was then applied to analyze the trend in mean and standard deviation of the achieved accuracies. Results of our analysis showed that the variance declines quickly at first while significant improvements plateau around 10 permutations. Therefore, 10 random permutations per training run were used Experiment 1 while we opted for 20 permutations for Experiment 2, striking a balance between computational efficiency and statistical validity.
3.3 Experiment 1: impact of time offset and clock skew
To date, CNNs play a central role in wearable-based HAR due to their ability to capture local temporal patterns from raw sensor data with high computational efficiency. They are widely integrated into state-of-the-art end-to-end HAR models (; ; ) as well as more recent approaches incorporating cross-modal transfer from large pretrained models such as LLMs ().
In our first experiments, we aim to evaluate how inaccurate synchronization affects model performance, particularly in terms of local feature distortions and temporal recognition patterns. To this end, we adopt a fully convolutional architecture based on multi-stage temporal convolutional networks (MS-TCNs) by as a representative example. Given the strong role of convolutional structures in modeling temporal dynamics in HAR, this choice allows us to systematically study how such models are affected by temporal misalignment.
To generally evaluate how time discrepancies affect the performance of CNNs in HAR, we investigated the impact of (1) time offset and (2) clock skew on CNNs with varying (3) network depths and (4) input lengths. We also performed the evaluation varying the parameters during training and validation steps, to simulate discrepancies when the model is trained on a dataset affected by incorrect synchronization and applied in settings with proper synchronization, or vice versa. In total, our first evaluation spanned 30,025 training runs with different variations of these parameters, amounting to 1.2 million core-h of computations. The summary of the parameters is provided in Table 1.
Table 1
| Parameters | Iterated values |
|---|---|
| Time offset in s at 30 Hz | 0.00, 0.06, 0.15, 0.30, 0.61, 1.52, 3.03, 6.06, 15.15, 30.30, 60.60 |
| Clock skew in ppm | 0, 2, 5, 10, 20, 50, 100, 200, 500, 1,000, 2,000 |
| Network depth | 2 (shallow), 3 (medium), and 4 (deep) stages |
| input length in s at 30 Hz | 6.6, 11.0, 17.6, 22.0, 27.5, 33.0, 44.0, 66.0, and 111.0 |
| target | Training, validation, and both |
| sensor assignment | 10 random assignments of five sensors with discrepancies modeled as [−σ, −σ/2, 0, σ/2, σ] |
Summary of the model parameters that have been assigned and iterated throughout Experiment 1.
To train the models, we set the number of feature maps to 128, applied a learning rate of 0.005, and used the loss function from the original implementation of . The training and test data was split 75:25 in each run among the four participants, and the fourth participant was consistently left out for validation. Moreover, we sampled the data from participants' recording sessions equally to ensure an equal representation of activity classes and motion sequences. During training and validation, we randomly selected segments from the subsets with an input length according to the experiment protocol to avoid overfitting. The training of each model was performed for 50 epochs, and the best model was saved every ten epochs. We implemented early stopping, with the training ceasing if the validation loss did not improve for three consecutive epochs. As the parameter combinations between time offset and clock skew were generated using a partial Latin square design (Figure 4) rather than a full factorial design, we limited our analysis to a descriptive characterization of the observed response.
Figure 4
3.4 Experiment 2: fine-grained impact analysis of time offset
In our second experiment, we evaluate how different architectures vary in their robustness to desynchronization-induced misalignment. We consider a larger set of architectures under a finer hyperparameter grid, including variants of MS-TCN () in two configurations, DeepConvLSTM () in two configurations, and IF-ConvTransformer (). Each model was trained using two different temporal input lengths, resulting in 10 model configurations.
For MS-TCN, we follow the configuration from the previous experiment, varying only the number of refinement stages. Specifically, we evaluate a shallow variant with one stage and a medium-depth variant with two stages. For DeepConvLSTM and IF-ConvTransformer, which were originally designed for window-based classification, we remove temporal aggregation layers prior to prediction. This minimal modification enables dense, sample-wise predictions in a manner comparable to MS-TCN.
For DeepConvLSTM, we consider both unidirectional and bidirectional LSTM variants. The motivation is that bidirectional models access context before and after each considered timestamp, providing a comparable temporal receptive field to MS-TCN, allowing for a more controlled comparison of temporal modeling capabilities. For IF-ConvTransformer, we use the original architecture without aggregation, resulting in sample-wise predictions. We omit the gravity component due to its absence in the Opportunity dataset, as it was originally derived via a low-pass filtering approximation in the authors' implementation. The sensor fusion attention mechanism is retained as proposed in the original model. Although the MS-TCN and DeepConvLSTM architectures are time-invariant, as transformers rely on positional encoding, we used unified input lengths of 198 and 330 samples for all networks to limit the receptive field. This resulted in a combination of 10 network architectures that were trained on the opportunity dataset.
To train the models, we applied a learning rate of 0.005, and used the loss proposed by , consisting of sample-based cross entropy loss and temporal MSE smoothing to align segmentation. The training of each model was performed for 100 epochs, and we implemented early stopping, with the training ceasing if the validation loss did not improve for three consecutive epochs. We performed the training and evaluation using LOSO, meaning leaving one of the four subjects out for training and testing purposes. Over four splits, this resulted in 40 models trained on the dataset without any discrepancies. For evaluation, we iterated through all the values provided in Table 2. Following the evaluation, we conducted a statistical analysis by computing Spearman's rank correlation coefficient ρ to quantify the strength and direction of the relationship between model performance and span.
Table 2
| Parameters | Iterated values |
|---|---|
| time offset spans S in s at 30 Hz | 0.03, 0.06, 0.10, 0.13, 0.17, 0.20, 0.27, 0.33, 0.40, 0.47, 0.53, 0.60, 0.67, 0.83, 1.00, 1.17, 1.33, 1.50, 1.67, 2, 2.33, 2.67, 3 |
| Architecture | Five model architectures, each trained using input length of 6.6 and 11 s: (1) MS-TCN in shallow configuration (2 layers, one classification and one refinement layer) (2) MS-TCN in medium (3 layers) configuration (3) DeepConvLSTM architecture with four CNN layers and uni-directinal LSTMs for segmentation (4) DeepConvLSTM architecture with bi-directional LSTMs for segmentation (5) IF-ConvTransformer |
| Target | Validation |
| Sensor assignment | 20 random assignments of 5 sensors with discrepancies modeled as mode-normalized skew normal distribution [SN(−ω, α), SN(−ω/2, α), SN(0, α), SN(ω/2, α), SN(ω, α)] |
| Skew parameter α | 0, 5, 10 |
Summary of the model parameters that have been assigned and iterated throughout Experiment 2.
4 Results
4.1 Experiment 1
To evaluate the impact of time discrepancies, we ran the different parameter and model combinations, and computed the mean performance. Figure 4 illustrates averaged curves for accuracy, F1 score at 0.5 overlap threshold, and the edit distance over different time offsets with clock skew set to zero, and vice versa.
We begin our evaluation by demonstrating the effect of time offset, which can occur through inaccurate synchronization or accumulated clock skew. A drop in the performance is observable starting from σ of 5 to 10 (167 to 333 ms, Figure 5A). For a σ of 20 (667 ms), our results display an accuracy drop of 3.9 %, 6.6 % for F1 at 0.5 overlap, and 5.1 % for segmentational edit distance. In contrast, the clock skew values of typical quartz oscillators do not appear to considerably affect the performance of the HAR model in the short term (Figure 5B). In our experiment, the averaged metrics for accuracy, F1 scores, and edit distance remained largely constant across variations of clock skew. Likewise, when both discrepancies in time offset and clock skew are present, as illustrated in Figure 4, the degradation in performance is primarily driven by time offset.
Figure 5
To address how these findings could translate into practical applications, we evaluated the impact of time offset grouped under training and validation conditions (Figure 5C). The validation curve illustrates how time discrepancies would affect a model that was trained on well-synchronized data, but deployed in settings affected by different levels of time offset. We observe that such models may fail to generalize to data that is affected by temporal misalignment. Interestingly, even when both training and validation data are affected, performance still drops, which we attribute to the distributed time offsets impacting how single sensors contribute toward the model's output. Moreover, the pattern toward lower performance is observed across models with different network depths (Figure 5D), suggesting that local inaccuracies propagate through features captured from raw data in the first layers into deeper layers.
To further explore how temporal patterns in activity segmentation are affected by time discrepancies, we provide qualitative results from one representative sample from the dataset. Figure 3 illustrates the ground truth labels and recognized segments for a single model configuration across various time offset and clock skew values. While clock skew (Figure 3B) does not lead to significant changes in recognition patterns, time offset (Figure 3A) introduces substantial variation in model segmentation performance. From the qualitative data, although we find that both static and dynamic activities are affected, the most noticeable impact occurs during frequent transitions between activities, such as going from ‘walking' to ‘standing'.
4.2 Experiment 2
To assess the robustness of different machine learning architectures to offset desynchronization, we ran the experiment and computed mean performance metrics over model × temporal size × span. The means were computed over all permutations and the three skews α of 0, 5, and 10. Due to the use of skew normal distribution, and overall lower σ values in our fine-grained impact analysis, we report the results using the span S between the sensor misalignment instead of σ. In addition, we report Spearman's ρ coefficient to quantify the strength and direction of the monotonic relationship between the tested span variable and the performance metrics, i.e., accuracy, edit score, and F1-score, for each model. The results are summarized in Table 3.
Table 3
| Network/metric | Accuracy | Edit | F1@0.1 | F1@0.5 |
|---|---|---|---|---|
| MS-TCN with one Refinement Layer | -0.592 | -0.426 | -0.654 | -0.764 |
| MS-TCN with two Refinement Layers | -0.379 | -0.297 | -0.617 | -0.635 |
| IF-ConvTransformer | -0.501 | -0.353 | -0.776 | -0.798 |
| DeepConvLSTM unidirectional | -0.407 | -0.225 | -0.458 | -0.549 |
| DeepConvLSTM bidirectional | -0.226 | -0.187 | -0.596 | -0.731 |
Summary of the model parameters that have been assigned and iterated throughout Experiment 2.
Overall, models with larger temporal input sizes achieve consistently better performance on synchronized data. Across all five architecture variants, models with an input length of 330 samples outperform their counterparts with 198 samples. The best results are obtained by DeepConvLSTM models with bi-directional LSTMs, achieving an accuracy of 74.80%, edit distance of 59.62%, and F1@0.5 score of 59.62% 45.72% in the 330 sample variant. The second best model on synchronized data was the MS-TCN model with two refinement layers, which achieved 72% accuracy, an edit distance of 57.23%, and an F1 score at 0.5 overlap of 40.95%. While the IF-ConvTransformer models demonstrated competitive performance in terms of per-sample accuracy, their edit distance and F1 score were lower than those of other evaluated models. This suggests that, although the model excels at identifying activities, its segmentation performance is comparatively weaker. Overview of the models' performance on synchronized and desynchronized data is displayed in Figure 6.
Figure 6
In terms of quantitative relationships, all models exhibited negative Spearman's ρ values, indicating moderate to strong negative monotonic associations. Across all models, the strongest negative correlations were observed for the F1-score, ranging from -0.549 for DeepConvLSTM to -0.798 for IF-ConvTransformer. In contrast, the edit score showed the weakest correlations (Figure 6), with Spearman's ρ ranging from -0.187 for bidirectional DeepConvLSTM to -0.426 for MS-TCN with one refinement layer. Overall, these results indicate that increasing values of the tested span variable are consistently associated with lower performance across all models and metrics.
On the level of single networks, MS-TCN models show moderate sensitivity to misalignment due to the increasing time offsets across the sensor channels. At an offset span S of 5 (167 ms), accuracy decreases by approximately 1 % with a drop of about 2.5 % in F1@0.5 for the single-refinement variant. For the model with an additional refinement layer, accuracy decreases by approximately 0.5 %, with a similar decrease in F1@0.5. At S of 10 (330 ms), accuracy drops by around 2 %, while F1@0.5 decreases by approximately four to five points in both models over both temporal window lengths.
The IF-ConvTransformer model exhibits the strongest sensitivity to temporal misalignment. In the 198-sample configuration, accuracy decreases by approximately 2 % at S of 5 (167 ms), and by about 1.5 % in the 330-sample setting. Overall, accuracy drops sharply by up to 6 % at S of 16 (530 ms), after which the decline becomes more gradual. Notably, F1@0.5 decreases by approximately one third already at S of 10 (330 ms).
In contrast, the DeepConvLSTM architecture appears comparatively robust to temporal discrepancies. Both accuracy and edit score remain largely stable across all tested offsets, with noticeable degradation in F1@0.5 only occurring beyond S of 16 (530 ms).
5 Discussion
The use of multiple wearables with complementary sensor positions is beneficial for HAR but hinges on the synchronization of the devices' local clocks. Most available datasets do not adequately report the applied synchronization method and accuracy achieved. Understanding the effect of time discrepancies on sensor fusion and machine learning models is, therefore, crucial to select sufficiently accurate synchronization methods for data acquisition. In practical settings, these techniques must strike a balance between energy consumption, cost, and accuracy. This trade-off is particularly important when collecting new benchmark datasets for machine learning models. In HAR, typical synchronization actions like clapping and jumping often serve as landmarks in IMU sensor channels for manual or automatic alignment, but, as described by , proved unfortunately to be less accurate in practice.
The often applied rule of thumb by , suggesting that a sub-second accuracy might be sufficient for the recognition of “motion-related activities in daily life” while a sub-100 ms accuracy is “typically not needed”, is thus challenged by our findings. Through our empirical analysis with CNNs in Experiment 1, we can show that even time offsets as small as 60 – 150 ms may negatively impact model performance, especially in cases where the training and validation data, or data during inference, are not equally synchronized. Our results also demonstrate that offsets exceeding 300 ms already lead to a significant drop in the HAR model performance, which highlights the need for adequate synchronization of training datasets.
Results from our Experiment 2 reveal differences in robustness to desynchronization across architectural families, with patterns that can be traced to how temporal information is represented and processed. In our experiment, the IF-ConvTranformers () model was the most strongly affected by temporal misalignments. We believe that can be attributed to their reliance on positional encodings, which impose an explicit temporal structure on the input. Hence, when signals are desynchronized, this structure becomes inconsistent, leading to degraded temporal representations and reduced performance.
At the same time, MS-TCNs () exhibit a more nuanced behavior. The variant with a single refinement stage was more sensitive to desynchronization, likely due to its limited temporal receptive field, which makes it vulnerable to shifts in local temporal patterns. In contrast, the deeper MS-TCN variant was more robust. This suggests that the additional refinement stage provided an implicit correction mechanism that mitigated temporal noise and partially realigned predictions. This behavior indicates that the hierarchical refinement could improve robustness to input misalignment.
Interestingly, DeepConvLSTM () showed stable performance under desynchronization. Despite using convolutional layers for feature extraction prior to the recurrent component, the architecture appears largely unaffected by temporal shifts. This indicates that robustness may be primarily driven by the LSTM-based temporal modeling stage rather than the convolutional feature extractor.
The recurrent hidden state enables temporal integration in which inconsistent inputs might be gradually forgotten, while salient information is preserved. This observation suggests that robustness in action segmentation might be driven more by the design of the temporal modeling head than by the feature extraction backbone. Both MS-TCN and DeepConvLSTM indicated that temporal modeling can compensate for imperfect alignment in the input signal, though through different mechanisms. MS TCN uses iterative refinement through multiple segmentation layers, while the behavior of LSTMs stems from recurrent state dynamics.
Finally, bidirectional LSTMs achieve the strongest overall performance. This result indicates that access to context before and after each timestamp improves robustness to temporal jitter, as predictions can be stabilized using information from both directions. Although bidirectional architectures are less commonly used in real-time wearable settings, these findings suggest that temporal context within a larger time-window can act as an implicit alignment mechanism, reducing sensitivity to local temporal shifts. In practical settings, this means that by passing ongoing detection backwards using bi-directional LSTMs could refine past predictions affected by discrepancies. Overall, our findings suggest that robustness to desynchronization is primarily determined by the temporal modeling strategy rather than the feature extraction stage. Architectures that incorporate either iterative refinement or recurrent state-based integration appear more resilient to temporal misalignment.
In our first experiment, while clock skew on its own appears to have minimal short-term impact, it should not be dismissed. Although CMOS MEMS clock sources with an accuracy of ± 500 ppm, according to , should not directly have an effect, they would potentially show an accrued time offset of 150 ms after just 5 minutes, which again considerably impacts model performance. Therefore, we recommend to consider accurate oscillators for deployment, such as standard XOs with ± 20 ppm. However, achieving optimal timestamp alignment is not solely a hardware issue. Also efficient and accurate synchronization methods can contribute toward re-aligning mismatched recordings. Creative approaches have demonstrated great accuracy relying on motion for synchronization, e.g., with 20 ms and with 8.6 ms, as well as approaches beyond motion, e.g., leveraging sporadic cough events with 46 ms, and () using artificial landmark signals detectable in the ECG channel of off-the-shelf wearables with 6.25 ms, using the continuously available, natural human pulse with 2.9 ms, and using electromagnetic fields with 8-10 ms, or to achieve even 2.6 ms.
Overall, our findings underscore the importance of precise and consistent synchronization strategies, particularly for training datasets, where timestamp inaccuracies may have a cascading effect on downstream generalization. Based on our experiments, we believe that ensuring that time offset remains below 100 ms should be treated a core requirement for HAR datasets and models. Beyond synchronization, we also recommend that future wearable datasets document the synchronization methodology and, if available, the achieved accuracy. To the best of our knowledge, among available datasets, only the Opportunity dataset (,) reports the synchronization procedure. Reporting this information would improve the transparency, reproducibility, and comparability of benchmarks and facilitate a more informed choice for transferring results into applications.
Furthermore, the relevance of these findings extends beyond deep learning-based HAR. Multi-wearable sensing systems are inherently susceptible to synchronization inaccuracies. Such inaccuracies have also been shown to affect applications such as clinical gait analysis that do not rely on machine learning (). Consequently, synchronization should be regarded as a fundamental consideration for wearable sensing systems in general, including HAR pipelines and other applications that depend on temporally aligned sensor streams.
To contextualize these findings, we must consider the inherent limitations of our experiment design, in particular, the low sampling rate of the opportunity dataset. Although common in IMU datasets, the sampling frequency of 30 Hz is lower than what is typical in more recent HAR datasets (; ; ). However, many of these datasets lack proper synchronization, which is why we chose the Opportunity dataset. The lower sample coverage reduces the precision of interpolated values between samples, increasing the risk of signal artifacts and distortion. Our current implementation introduces time offsets by shifting time indices associated with signal samples in discrete steps. A higher sampling rate would have enabled more granular time shifts, even at a sub-sample level, and facilitated better reconstruction of higher-frequency components. For clock skew correction, it would have allowed interpolation to insert or remove samples at ppm scale, which is infeasible at low resolutions. In both scenarios, higher rates would have improved signal integrity and model robustness. Moreover, the dataset contains only ADLs, so future work should also explore the impact on faster activities like sports or physiological signals, where precise timing is likely more critical. Furthermore, our experimental setup enabled robust statistical analysis only for Experiment 2. In Experiment 1, we were unable to quantify the strength of the relationships observed, as the parameter sweep did not follow a full factorial design, meaning that the parameters time offset and clock skew were not sampled independently across the explored parameter space. As a result, the sampled parameter values exhibited non-negligible correlations, introducing confounding between their effects. This limited our analysis in Experiment 1 to descriptive characterization.
Future work could move beyond evaluating desynchronization in controlled settings, investigating how HAR architectures cope with the combined variability encountered in real-world deployments, including aspects such as sensor noise or interpolated data due to intermittent data loss. Additionally, extending the analysis to a broader range of models and approaches, encompassing both deep learning models and more traditional approaches, would provide a more comprehensive understanding of synchronization robustness. Finally, field studies involving instrumented populations are needed to determine the extent to which these findings generalize to practical settings and to establish evidence-based guidelines for the minimum synchronization accuracy required across different application domains.
6 Conclusion
Although wearable-based human activity recognition (HAR) relies on multiple sensors, the effects of inaccurate synchronization across them on machine learning models remain underexplored. We systematically investigated these effects in two experiments using the Opportunity Activity Recognition dataset, which contains five hardware-synchronized IMU sensor channels. In the first experiment, we introduced desynchronization into both training and validation data. Results demonstrate that time offsets exceeding 167 ms in training degrade model performance, while offsets beyond 333 ms significantly impair recognition of typical activities of daily living (ADLs). Although clock skew has negligible immediate effects, its accumulation introduces offsets that must be corrected regularly. In the second experiment, we evaluated the robustness of different architectures to desynchronization during inference. Models trained on synchronized data showed marked performance degradation beyond offsets of 167 ms, except for the DeepConvLSTM architecture, which exhibited greater resilience. These findings underscore the critical role of synchronization in multi-wearable HAR and highlight architectural trade-offs in handling temporal misalignment. To ensure robust performance, we recommend maintaining offsets below 100 ms and employing synchronization strategies capable of sub-50 ms precision when feasible. These insights extend beyond HAR to other multi-sensor applications where temporal alignment is critical.
Statements
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.
Author contributions
DK: Conceptualization, Validation, Methodology, Data curation, Investigation, Writing – review & editing, Writing – original draft, Software, Formal analysis. FW: Writing – original draft, Formal analysis, Supervision, Validation, Methodology, Visualization, Software, Conceptualization, Investigation, Writing – review & editing. SS: Conceptualization, Supervision, Writing – review & editing, Methodology. FM: Writing – review & editing, Conceptualization, Methodology, Supervision.
Funding
The author(s) declared that financial support was received for this work and/or its publication. The authors acknowledge TU Wien Bibliothek for financial support in through its Open Access Funding Program.
Acknowledgments
We would like to thank Patrick Trollmann for implementing the fundamental concept of this research in his Master's thesis, previously published in . The computational results presented have been achieved using the Austrian Scientific Computing (ASC) infrastructure.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI was used for grammar correction. The paper remains an accurate representation of the author(s)' underlying work and intellectual contributions.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AguiletaA. A.BrenaR. F.MayoraO.Molino-Minero-ReE.TrejoL. A. (2019). Multi-sensor fusion for activity recognition-a survey. Sensors (Basel, Switzerland)19:3808. doi: 10.3390/s19173808
2
AhmedM. E.YuH.VassalloM.KoufakiP. (2025). Advancing real-world applications: A scoping review on emerging wearable technologies for recognizing activities of daily living. Smart Health36:100555. doi: 10.1016/j.smhl.2025.100555
3
AhmedT.AhmedM. Y.RahmanM. M.NematiE.IslamB.VatanparvarK.et al. (2020). “Automated time synchronization of cough events from multimodal sensors in mobile devices,” in International Conference on Multimodal Interaction (New York, NY: ACM), 614–619.
4
BannachD.AmftO.LukowiczP. (2009). “Automatic event-based synchronization of multimodal data streams from wearable and ambient sensors,” in Smart Sensing and Context, EuroSSC'09 (Cham: Springer), 135–148.
5
BarrettL. F.BarrettD. J. (2001). An introduction to computerized experience sampling in psychology. Soc. Sci. Comp. Rev. 19, 175–185. doi: 10.1177/089443930101900204
6
BennettT. R.GansN.JafariR. (2015a). “A data-driven synchronization technique for cyber-physical systems,” in Proceedings of the Second International Workshop on the Swarm at the Edge of the Cloud (New York, NY: ACM), 49–54.
7
BennettT. R.GansN.JafariR. (2015b). “Multi-sensor data-driven: synchronization using wearable sensors,” in Proceedings of the 2015 ACM International Symposium on Wearable Computers, ISWC '15 (New York, NY: ACM), 113–116.
8
BennettT. R.GansN.JafariR. (2017). Data-driven synchronization for internet-of-things systems. ACM Trans. Embedded Comp. Syst. 16, 1–24. doi: 10.1145/2983627
9
BerlinE.LiuJ.Van LaerhovenK.SchieleB. (2010). “Coming to grips with the objects we grasp,” in Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction - TEI '10 (New York, NY: ACM), 57.
10
BhattacharyaS.LaneN. D. (2016). “From smart to deep: Robust activity recognition on smartwatches using deep learning,” in 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops) (Sydney, NSW: IEEE), 1–6.
11
BockM.HölzemannA.MoellerM.Van LaerhovenK. (2021). “Improving deep learning for har with shallow LSTMS,” in Proceedings of the 2021 ACM International Symposium on Wearable Computers (New York, NY: ACM), 7–12.
12
BockM.KuehneH.Van LaerhovenK.MoellerM. (2024a). Wear: an outdoor sports dataset for wearable and egocentric activity recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8:3699776. doi: 10.1145/3699776
13
BockM.MoellerM.Van LaerhovenK. (2024b). Temporal action localization for inertial-based human activity recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8:3699770. doi: 10.1145/3699770
14
BorawskaA.DudaJ.BiercewiczK. (2021). Best practices of neurophysiological data collection for media message evaluation in social campaigns. Procedia Comp. Sci. 192, 4017–4026. doi: 10.1016/j.procs.2021.09.176
15
BromanD.DerlerP.EidsonJ. (2013). Temporal issues in cyber-physical systems. J. Indian Instit. Sci. 93, 389–402.
16
BullingA.BlankeU.SchieleB. (2014). A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 46:2499621. doi: 10.1145/2499621
17
CrochiereR.RabinerL. (1975). Optimum fir digital filter implementations for decimation, interpolation, and narrow-band filtering. IEEE Trans. Acoust. Speech Signal Proc. 23, 444–456. doi: 10.1109/TASSP.1975.1162719
18
DumphartB.SlijepcevicD.UnglaubeF.KranzlA.BacaA.HorsakB. (2025). The effect of inaccurate initial contact events on kinematics in healthy and pathological gait. Gait & Posture2025:110012. doi: 10.2139/ssrn.5095753
19
FarhaY. A.GallJ. (2019). “MS-TCN: Multi-stage temporal convolutional network for action segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Long Beach, CA: IEEE).
20
FillerR.VigJ. (1993). Long-term aging of oscillators. IEEE Trans. Ultrason. Ferroelect. Freq. Cont. 40, 387–394. doi: 10.1109/58.251287
21
GilbertT. J.LinZ.DayS.HamiltonA. F. d. C WardJ. A. (2024). A magnetometer-based method for in-situ syncing of wearable inertial measurement units. Front. Comp. Sci. 6:1385392. doi: 10.3389/fcomp.2024.1385392
22
HanY.GilbertT. J.TanX.WardJ. A. (2024). “The wand chooses the imu - open source hardware for synchronising wearables using magnetometers,” in Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp '24 (New York, NY: ACM), 939–943.
23
KhanA.HammerlaN.MellorS.PlötzT. (2016). Optimising sampling rates for accelerometer-based human activity recognition. Pattern Recognit. Lett. 73, 33–40. doi: 10.1016/j.patrec.2016.01.001
24
KwonH.AbowdG. D.PlötzT. (2019). “Handling annotation uncertainty in human activity recognition,” in Proceedings of the 2019 ACM International Symposium on Wearable Computers, ISWC '19 (New York, NY: ACM), 109–117.
25
LaputG.HarrisonC. (2019). “Sensing fine-grained hand activity with smartwatches,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19 (New York, NY: Association for Computing Machinery), 1–13.
26
LatifzadehK.LeivaL. A. (2022). “Gustav: Cross-device cross-computer synchronization of sensory signals,” in Adj. Pub. of the 35th ACM Symposium on User Interface Software and Technology, ed. M. Agrawala (New York, NY: ACM), 1–3.
27
LawalI. A.BanoS. (2020). Deep human activity recognition with localisation of wearable sensors. IEEE Access8, 155060–155070. doi: 10.1109/ACCESS.2020.3017681
28
LengZ.BhattacharjeeA.RajasekharH.ZhangL.BrudaE.KwonH.et al. (2024). IMUGPT 2.0: language-based cross modality transfer for sensor-based human activity recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8:1–32. doi: 10.1145/3678545
29
LiP.YinJ.OuyangL.PanS.WangG.DingH.et al. (2026). WS-IMUBench: can weakly supervised methods from audio, image, and video be adapted for imu-based temporal action localization? arXiv [preprint] arXiv:2602.01850. doi: 10.48550/arXiv.2602.01850
30
LyonsR. G. (2011). Understanding Digital Signal Processing. Upper Saddle River, NJ: Prentice Hall.
31
NwekeH. F.TehY. W.Al-garadiM. A.AloU. R. (2018). Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: state of the art and research challenges. Expert Syst. Appl. 105, 233–261. doi: 10.1016/j.eswa.2018.03.056
32
OhmuraR.NayaF.NomaH.KogureK. (2006). “B-Pack: a bluetooth-based wearable sensing device for nursing activity recognition,” in Wireless Pervasive Computing (ISWPC) (Phuket: IEEE), 1–6.
33
OmetovA.ShubinaV.KlusL.SkibińskaJ.SaafiS.PascacioP.et al. (2021). A survey on wearable technology: History, state-of-the-art and current challenges. Comp. Netw. 193:108074. doi: 10.1016/j.comnet.2021.108074
34
Ordó nezF. J.RoggenD. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors16:115. doi: 10.3390/s16010115
35
RoggenD.CalatroniA.Nguyen-DinhL.-V.ChavarriagaR.SaghaH. (2010a). OPPORTUNITY Activity Recognition. Irvine, CA: UCI Machine Learning Repository.
36
RoggenD.CalatroniA.RossiM.HolleczekT.FörsterK.TrösterG.et al. (2010b). “Collecting complex activity datasets in highly rich networked sensor environments,” in 2010 Seventh International Conference on Networked Sensing Systems (INSS) (Kassel: IEEE), 233–240. doi: 10.1109/INSS.2010.5573462
37
SahaB.SamantaR.GhoshS. K.RoyR. B. (2023). “From wrist to world: harnessing wearable IMU sensors and TinyML to enable smart environment interactions,” in Proceedings of the Third International Conference on AI-ML Systems (Bangalore: ACM), 1–3.
38
SchmidT.CharbiwalaZ.FriedmanJ.ChoY. H.SrivastavaM. B. (2008). Exploiting manufacturing variations for compensating environment-induced clock drift in time synchronization. SIGMETRICS Perform. Eval. Rev. 36, 97–108. doi: 10.1145/1384529.1375469
39
SchmidtP.ReissA.DuerichenR.MarbergerC.Van LaerhovenK. (2018). “Introducing wesad, a multimodal dataset for wearable stress and affect detection,” in Proceedings of the 20th ACM International Conference on Multimodal Interaction, ICMI '18 (New York, NY: ACM), 400–408.
40
ShiD.ZhongY.CaoQ.MaL.LiJ.TaoD. (2023). “TriDet: Temporal action detection with relative boundary modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (Vancouver, BC: IEEE), 18857–18866.
41
SopidisG.HaslgrüblerM.AzadiB.Anzengruber-TánaseB.AhmadA.FerschaA.et al. (2022). “Micro-activity recognition in industrial assembly process with imu data and deep learning,” in Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (Corfu: ACM), 103–112.
42
SpilzA.MunzM. (2023). Synchronisation of wearable inertial measurement units based on magnetometer data. Biomed. Eng. 68, 263–273. doi: 10.1515/bmt-2021-0329
43
StiefmeierT.RoggenD.OgrisG.LukowiczP.TrösterG. (2008). Wearable activity tracking in car manufacturing. IEEE Pervas. Comput. 7, 42–50. doi: 10.1109/MPRV.2008.40
44
TaoW.LaiZ.-H.LeuM. C.YinZ. (2018). Worker activity recognition in smart manufacturing using imu and semg signals with convolutional neural networks. Procedia Manufact. 26, 1159–1166. doi: 10.1016/j.promfg.2018.07.152
45
Tirado-AndrésF.AraujoA. (2019). Performance of clock sources and their influence on time synchronization in wireless sensor networks. Int. J. Distrib. Sensor Netw. 15, 16. doi: 10.1177/1550147719879372
46
TodaT.InoueS.TanakaS.UedaN. (2014). “Training human activity recognition for labels with inaccurate time stamps,” in Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, UbiComp '14 Adjunct (New York, NY: ACM), 863–872.
47
Trujillo-GuerreroM. F.Román-NiemesS.Jaén-VargasM.CadizA.FonsecaR.Serrano-OlmedoJ. J. (2023). Accuracy comparison of cnn, lstm, and transformer for activity recognition using imu and visual markers. IEEE Access11, 106650–106669. doi: 10.1109/ACCESS.2023.3318563
48
Van LaerhovenK.GellersenH.-W. (2004). “Spine versus porcupine: a study in distributed wearable activity recognition,” in Eighth International Symposium on Wearable Computers (Arlington, VA : IEEE), 142–149.
49
VigJ. R. (1992). Introduction to quartz frequency standards. NASA STI/Recon Tech. Report N92:29498.
50
WangC.SarsenbayevaZ.LuoC.GoncalvesJ.KostakosV. (2019). “Improving wearable sensor data quality using context markers,” in UbiComp/ISWC '19 (New York, NY: ACM).
51
WollingF.HuynhC. D.Van LaerhovenK. (2021a). “IBSync: Intra-body synchronization of wearable devices using artificial ecg landmarks,” in ISWC, ISWC '21 (New York, NY: ACM), 102–107.
52
WollingF.KostolaniD.TrollmannP.MichahellesF. (2025). Impact of Time Discrepancies on Machine Learning Performance for Multi-Wearable Human Activity Recognition. New York, NY: Association for Computing Machinery.
53
WollingF.Van LaerhovenK. (2022). IBSync: Intra-body synchronization and implicit contextualization of wearable devices using artificial ECG landmarks. Front. Comp. Sci. 4:915448. doi: 10.3389/fcomp.2022.915448
54
WollingF.Van LaerhovenK.SiirtolaP.RöningJ. (2021b). “PulSync: The heart rate variability as a unique fingerprint for the alignment of sensor data across multiple wearable devices,” in Proceedings of the 2021 IEEE International Conference on Pervasive Computing and Communications Workshops, PerHealth Workshop (Kassel: IEEE), 188–193.
55
XiaS.ChuL.PeiL.YangJ.YuW.QiuR. C. (2024). Timestamp-supervised wearable-based activity segmentation and recognition with contrastive learning and order-preserving optimal transport. IEEE Trans. Mobile Comp. 23, 10734–10751. doi: 10.1109/TMC.2024.3381171
56
XiaS.ChuL.PeiL.YuW.QiuR. C. (2022). A boundary consistency-aware multitask learning framework for joint activity segmentation and recognition with wearable sensors. IEEE Trans.Indust. Inform. 19, 2984–2996. doi: 10.1109/TII.2022.3173957
57
XuT.GuoA.MaJ.WangK. I.-K. (2017). “Feature-based temporal statistical modeling of data streams from multiple wearable devices,” in DASC'17 (Piscataway, NJ: IEEE), 119–126.
58
ZengM.GaoH.YuT.MengshoelO. J.LangsethH.LaneI.et al. (2018). “Understanding and improving recurrent networks for human activity recognition by continuous attention,” in Proceedings of the 2018 ACM International Symposium on Wearable Computers (New York, NY: ACM), 56–63.
59
ZhangY.WangL.ChenH.TianA.ZhouS.GuoY. (2022). If-convtransformer: A framework for human activity recognition using imu fusion and convtransformer. Proc. ACM on Interact. Mobile Wearable Ubiquit Technol. 6, 1–26. doi: 10.1145/3534584
60
ZhaoY.YangR.ChevalierG.XuX.ZhangZ. (2018). Deep residual Bidir-LSTM for human activity recognition using wearable sensors. Mathem. Prob. Eng. 2018:7316954. doi: 10.1155/2018/7316954
61
ZhouY.ZhaoH.HuangY.RiedelT.HefenbrockM.BeiglM. (2022). “TinyHAR: a lightweight deep learning model designed for human activity recognition,” in Proceedings of the 2022 ACM International Symposium on Wearable Computers (New York, NY: ACM), 89–93.
Summary
Keywords
action segmentation, desynchronization, human activity recognition, machine learning, synchronization, time discrepancy, wearable computing
Citation
Kostolani D, Wolling F, Schlund S and Michahelles F (2026) Broken time, stable models? Evaluating desynchronization robustness in wearable human activity recognition. Front. Comput. Sci. 8:1873283. doi: 10.3389/fcomp.2026.1873283
Received
05 May 2026
Revised
07 July 2026
Accepted
21 July 2026
Published
31 August 2026
Volume
8 - 2026
Edited by
Sizhen Bian, German Research Center for Artificial Intelligence (DFKI), Germany
Reviewed by
Tomas Duminis, Queen Mary University of London, United Kingdom
Hugo Vega-Huerta, National University of San Marcos, Peru
Updates
Copyright
© 2026 Kostolani, Wolling, Schlund and Michahelles.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: David Kostolani, david.kostolani@tuwien.ac.at; Florian Wolling, florian.wolling@tuwien.ac.at
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.