Impact Factor 6.050 | CiteScore 7.8
More on impact ›


Front. Cardiovasc. Med., 26 April 2021 |

Serial EXTEM, FIBTEM, and tPA Rotational Thromboelastometry Observations in the Maastricht Intensive Care COVID Cohort—Persistence of Hypercoagulability and Hypofibrinolysis Despite Anticoagulation

Anne-Marije Hulshof1,2*†‡, Renée A. G. Brüggemann3†‡, Mark M. G. Mulder4, Tom W. van de Berg2, Jan-Willem E. M. Sels4, Renske H. Olie2,3,5, Bart Spaetgens3, Alexander S. Streng1, Paul Verhezen1, Iwan C. C. van der Horst2,4, Hugo Ten Cate2,3,5, Henri M. H. Spronk2, Bas C. T van Bussel4,6 and Yvonne M. C. Henskens1 on behalf of the Dutch Covid-19 Thrombosis Consortium (DCTC)
  • 1Central Diagnostic Laboratory, Maastricht University Medical Centre+, Maastricht, Netherlands
  • 2Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, Netherlands
  • 3Department of Internal Medicine, Maastricht University Medical Centre+, Maastricht, Netherlands
  • 4Department of Intensive Care Medicine, Maastricht University Medical Centre+, Maastricht, Netherlands
  • 5Thrombosis Expert Centre Maastricht, Maastricht University Medical Centre+, Maastricht, Netherlands
  • 6Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands

Background: Coronavirus Disease 2019 (COVID-19) patients often present with thromboembolic events. In COVID-19 patients, routine hemostatic assays cannot correctly identify patients at risk for thromboembolic events. Viscoelastic testing with rotational thromboelastometry (ROTEM) might improve the characterization of COVID-19-associated coagulopathy.

Objective: To unravel underlying coagulopathy and fibrinolysis over time as measured by serial assessment heparin-independent (FIBTEM and EXTEM) and fibrinolysis illustrating (tissue plasminogen activator; tPA) ROTEM assays.

Patients/Methods: Between April 23 and June 12, consecutive adult patients enrolled within the Maastricht Intensive Care COVID (MaastrICCht) cohort were included, and a comprehensive set of clinical, physiological, pharmaceutical, and laboratory variables were collected daily. Twice per week, EXTEM, FIBTEM, and tPA ROTEM were performed. Clotting time (CT), clot formation time (CFT), maximum clot firmness (MCF), lysis onset time (LOT), and lysis time (LT) were determined to assess clot development and breakdown and were compared to routine hemostatic assays.

Results: In 36 patients, 96 EXTEM/FIBTEM and 87 tPA ROTEM tests were performed during a 6-week follow-up. CT prolongation was present in 54% of EXTEM measurements, which were not matched by prothrombin time (PT) in 37%. Respectively, 81 and 99% of all EXTEM and FIBTEM MCF values were above the reference range, and median MCF remained elevated during follow-up. The ROTEM fibrinolysis parameters remained prolonged with median LOT consequently >49 min and unmeasurable LT in 56% of measurements, suggesting a severe hypofibrinolytic phenotype.

Conclusion: ROTEM tests in COVID-19 ICU patients show hypercoagulability and severe hypofibrinolysis persisting over at least 6 weeks.


Early after the onset of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, anecdotal reports of COVID-19 patients suffering from extensive thromboembolic complications, mainly pulmonary embolisms, emerged (1, 2). Currently, more extensive studies depicted the incidence of thromboembolic complications in COVID-19 intensive care unit (ICU) patients between 20 and 77%, depending on the screening strategy and reported outcomes (312). Additionally, subtraction CT angiography (sCTA) showed abnormally decreased iodine distribution in areas of morphologically normal lung parenchyma of COVID-19 patients (13). These findings indicate a severe ventilation/perfusion mismatch, suggesting a role for the procoagulant state and microthrombi development in disease progression. Evidence indicates that COVID-19-associated coagulopathy plays a pivotal role in morbidity and mortality in critically ill patients (14). However, it is currently unknown whether the observed coagulopathy is directly virus-induced, a response to extensive systemic inflammation, or both. Moreover, a recent study suggests that COVID-19 causes hypercoagulability and leads to a decrease of fibrinolysis, subsequently associated with venous thromboembolism (VTE) (15). Identifying COVID-19 patients at the most significant risk for thromboembolic events is essential to guide optimal anticoagulant prophylaxis and treatment strategies limiting thromboembolic deaths.

Evaluation of COVID-19-associated coagulopathy has proven to be challenging as routine hemostatic assays cannot correctly identify patients at risk for COVID-19-associated thromboembolic complications (16). In part, this may be caused by the fact that multiple systems, including cellular immunity (neutrophils), endothelial cells, complement activation, contact activation, and hypofibrinolysis, are engaged, which mostly remain undetected in routine assays (17, 18). Viscoelastic testing might better represent coagulation in vivo than routine hemostatic assays due to its whole blood dynamic assessment. Assessment of the complete fibrinolytic system remains notoriously difficult, but tissue plasminogen activator rotational thromboelastometry (tPA ROTEM) is thought to be more sensitive to hypofibrinolysis than other clinically used assays (19, 20).

We aimed to unravel the underlying coagulopathy as measured with serial heparin-independent EXTEM and FIBTEM, and tPA ROTEM tests in a mechanically ventilated COVID-19 ICU population over several weeks. The prospective and longitudinal aspect of this study, combined with serial tPA ROTEM measurements, is unprecedented in the current field of hemostatic testing in COVID-19.

Materials and Methods

The manuscript was written following the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines (21).

MaastrICCht Study

The Maastricht Intensive Care COVID (MaastrICCht) cohort study design has been described more extensively elsewhere (22). This prospective cohort study was conducted in patients admitted to the ICU of the Maastricht University Medical Centre+ (MUMC+). The study was designed to foster other datasets and registries according to the FAIR data principle in collaboration (23). The local institutional review board [Medisch Ethische Toetsingscomissie (METC) 2020-1565/ 300523] of the MUMC+ approved the study, based on the reglations of the declaration of Helsinki. The study is registered in the Netherlands Trial Register (registration number NL8613). Patients provided consent for use of the collected data for research purposes.

This study included all participants with respiratory insufficiency requiring mechanical ventilation and at least one real-time polymerase chain reaction (RT-PCR) positive for SARS-CoV-2 RNA and/or a chest CT scan strongly suggestive for SARS-CoV-2 infection, based on a CORADS-score of 4–5 scored by a radiologist (24). Participants were followed until the primary outcome was reached (i.e., either died in the ICU or discharge from the ICU). Every day, a comprehensive and uniform set of clinical, physiological, and laboratory variables was collected, reducing the chance of missing data. In addition, when patients were not available for blood withdrawal or laboratory testing failed, the measurement would be rescheduled for the next blood withdrawal. Computed tomography pulmonary angiography (CTPA) was performed when pulmonary embolism was clinically suspected. Due to logistical reasons, no routine compression ultrasonography for the detection of deep venous thrombosis was performed.


During the COVID-19 pandemic, incident thrombosis frequently appeared, resulting in new thromboprophylaxis recommendations. Consequently, from April 23, 2020, the dosage in the ICU was increased (Nadroparin 5,700, 7,600, and 11,400 IU for, respectively, <70, 70–90, and >90 kg) after the release of a national guidance document (25). Patients who required therapeutic anticoagulants before hospital admission were started on therapeutic low molecular weight heparin (LMWH) upon ICU admission; vitamin K antagonists (VKAs) and direct oral anticoagulants (DOACs) were discontinued since oral medication might not be resorbed optimally in critically ill patients. Patients on extracorporeal membrane oxygenation (ECMO) or continuous renal replacement therapy (CRRT) received unfractionated heparin (UFH), monitored using the aPTT [heparin therapeutic range (HTR) 50–80 s, based on an aPTT ratio of 1.5–2.5] and the anti-Xa (HTR 0.3–0.7 IU/ml) according to local guidelines.

Hemostasis Subcohort

N = 94 patients were enrolled from March 25 until June 12 in the MaastrICCht cohort. Enrollment for the hemostasis subcohort started later, on April 23. To align patients within their severe COVID-19 disease course, measurements were classified per week after intubation. ROTEM measurements were started in newly admitted patients from April 23 onwards. Additionally, patients that were on the ICU on April 23 who were already included in the MaastrICCht cohort were also included in the hemostasis subcohort and ROTEMs were measured. When patients were transferred from another ICU to the MUMC+, the original date of intubation was used. This means that some patients could be included in their first, second, third, or even fourth week after intubation. Thus, timing of inclusion in the hemostasis subcohort varied by design. ROTEM measurements were performed twice weekly if logistically possible during a pandemic situation. ROTEM measurements were taken until discharge, transfer to other care facility, or death.

Blood Withdrawal and Preparation

Daily blood samples from all patients were collected from an arterial line using 7.2 mg K2 EDTA (4.0 ml), serum, and 3.2% (w/v) citrate Vacutainer vacuum blood tubes (Becton Dickinson, Plymouth, UK). Additional 3.2% (w/v) citrate blood tubes were collected twice a week for global hemostasis assays, including ROTEM. Whole blood tests were performed within 4 h after blood withdrawal. Platelet-free plasma (PFP) was obtained using two subsequent centrifugation steps: initial centrifugation of 2,490 g for 5 min, followed by 10,000 g for 10 min.

Laboratory Measurements

Complete blood count (leukocyte count, platelet count, hemoglobin, and hematocrit) was performed on a Sysmex XN-9000 analyzer (Sysmex Corporation, Kobe, Hyogo, Japan) using EDTA blood. PT (Dade Innovin; Siemens, Marburg, Germany), aPTT (Dade Actin FSL; Siemens), fibrinogen concentration (Clauss method, Dade Thrombin Reagent; Siemens), D-dimer concentration (Innovance; Siemens), and Anti-Xa activity (Biophen Heparin LRT; Hyphen Biomed, Neuville-Sur-Oise, France) were measured on a Sysmex CS2100i (Sysmex Corporation) in 3.2% citrated blood. For the anti-Xa measurement, COVID-19 patient samples (18 μl) were three times diluted with reference pooled plasma (36 μl). Anti-Xa activity was determined using an LMWH calibration line (aXa-LMWH; Hyphen Biomed). UFH activity was subsequently calculated with a previously determined formula: UFH anti-Xa = 1.55 * LMWH anti-Xa (26). Bilirubin (Bilirubin Total, third-generation; Roche Diagnostics, Basel, Switzerland) and creatinine (Enzymatic Reagent; Roche Diagnostics) were determined in serum on a COBAS® 8000 (Roche Diagnostics).


ROTEM (Werfen; Barcelona, Spain) is a point-of-care viscoelastic assay performed in 3.2% (w/v) citrated whole blood. The ROTEM sigma cartridge-based system was used to perform EXTEM and FIBTEM to illustrate clot development's dynamic properties. EXTEM and FIBTEM are activated with tissue factor, with additional cytochalasin D added to the FIBTEM reagent to eliminate platelet contribution to clot formation. FIBTEM highlights the fibrinogen contribution to clot formation, as opposed to EXTEM, which illustrates all components' contribution to clot buildup. Polybrene is present in both assays to inhibit a potential anticoagulant effect of heparin. Standard ROTEM parameters were analyzed: clotting time (CT, in seconds), clot formation time (CFT, in seconds), and maximum clot firmness (MCF, in millimeters).

TPA ROTEM was performed to assess the fibrinolytic properties in COVID-19 patients. Recombinant tissue plasminogen activator (r-tPA) is used to create an EXTEM-like assay, previously validated by Kuiper et al. (20). In brief, 125 ng/ml r-tPA (Actilyse; Boehringer Ingelheim BV) is added in combination with tissue factor (Dade Innovin, Siemens) and calcium to activate clot formation and fibrinolysis simultaneously. ROTEM parameters used to evaluate fibrinolysis are Ly60 (percentage of MCF still present at 60 min), lysis onset time (LOT; time until 15% breakdown of MCF, in minutes), and lysis time (LT; time until 90% breakdown of MCF, in minutes). The tPA assay was performed on a ROTEM delta, enabling manual addition of reagents and lasted for a maximum of 2 h. If LOT and/or LT were not reached within 2 h, these results were capped at 120 min for analysis. For clinical reference, it is important to note that prolonged LOT and LT indicate delayed or absent fibrinolysis, defined in the current manuscript as hypofibrinolysis. Reversely, shorter LOT and LT indicate hyperfibrinolysis.

Statistical Analyses

Descriptive results were presented per week with the time of intubation as reference. The time between the day of intubation and the first ROTEM test was variable by design, as several patients were transferred from other ICUs and were intubated before MUMC+ ICU admission and ROTEM assays started from April 23 onwards. This allowed us to investigate weekly ROTEM parameters on the cohort level over several weeks, aiming to study patients at a similar disease progression stage. We used only the first measurement per week for each individual patient and studied the first 6 weeks in pooled analyses (i.e., week 1: days 1–7 after intubation; week 2: days 8–14 after intubation; until week 6). However, when we repeated the analyses using the average ROTEM measurements per week results did not materially change (data not shown). Measurements over 6 weeks after intubation (n = 11) were excluded from pooled analyses since the number of weekly measurements fell below 10, and no reliable analysis could be performed. A detailed overview of this cohort was provided for more insight as Supplementary Figure 1. Descriptive data are presented as mean (standard deviation; SD) or median [interquartile range; IQR] as appropriate.

Linear regression was performed on all measurements (weeks 1–10). Linear regression analyses were used to evaluate ROTEM parameters' determinants, and presented are the standardized β coefficient with 95% confidence intervals (95%CI) and R2. A p-value of <0.05 was considered statistically significant. Statistical analyses were performed in IBM SPSS Statistics 25.0, and figures were created using GraphPad Prism.


Demographic Characteristics

Thirty-six COVID-19 ICU patients were included in the present study. The median age was 61 years (IQR 55–70 years, range 17–75 years), and 81% was male. Baseline characteristics, ICU admission, and events are presented in Table 1. All patients were mechanically ventilated during ICU stay, and few comorbidities were present. Twenty patients (55.6%) had a confirmed pulmonary embolism, and mortality during ICU admission was 19% (n = 7).


Table 1. Baseline characteristics, ICU admission, and events.

The median Sequential Organ Failure Assessment (SOFA) score was 7 (IQR 6.5–7) in the first week after intubation (Table 2 and Figure 1A) and remained elevated throughout the 6-week follow-up. The involvement of multi-organ failure during the clinical course of mechanically ventilated patients was shown previously in the MaastrICCht cohort (27). Platelet count was mostly within the normal range (Figure 1B). D-dimer was increased in all patients and did not normalize over time (Figure 1C). Fibrinogen was highly elevated during the first 2 weeks after intubation and slowly declined (Figure 1D), showing a normalization trend during the following weeks. Our laboratory's fibrinogen measurements are capped at 9.0 g/L for routine assays, and no further dilution in fibrinogen samples was performed. Therefore, fibrinogen levels of 9.0 g/L are considered to be even more elevated.


Table 2. SOFA score, routine laboratory, standard ROTEM, and tPA ROTEM in COVID-19 patients per week from intubation.


Figure 1. Sequential Organ Failure Assessment (SOFA) score (A), platelet count (B), D-dimer (C), and fibrinogen (D) measurements over 6 weeks from intubation in a critically ill COVID-19 population. Results are presented as mean with standard deviation (yes) or median with interquartile range (yes); gray symbols illustrate individual measurements. Dashed lines represent reference values for thrombocytopenia (<150 × 109/L) and hyperfibrinogenemia (>4 g/L). D-dimer reference range is age dependent and is defined as <500, <510, <560, <610, <660, and <710 ng/ml for ages <50, 51–55, 56–60, 61–65, 66–70, and 71–75, respectively. Red dots illustrate D-dimer values >10,000 ng/ml.

All patients received some form of anticoagulation during their ICU treatment. Table 3 shows the different types and dosages of anticoagulation and anti-Xa levels.


Table 3. Anticoagulation day prior to ROTEM blood withdrawal in COVID-19 patients per week from intubation.


The median [IQR] time between the day of intubation and first ROTEM measurement was 18 [6-25] days. We included a total of 36 patients in the hemostasis subcohort, who had a total number of 107 ROTEM measurements varying between 1 and 6 per patient. This led to a weekly total number of ROTEM measurements of 12 for week 1, 15 for week 2, 22 for week 3, 19 for week 4, 17 for week 5, 11 for week 6, and 11 for >week 6. In 97 samples (91%), tPA ROTEM was also performed. ROTEM results per week from intubation are presented in Table 2 and Figure 2.


Figure 2. ROTEM EXTEM (A–C), FIBTEM (D,E), and tissue plasminogen activator (TPA) (F–H) measurements over 6 weeks from intubation in a critically ill COVID-19 population. Results are presented as mean with standard deviation (yes) or median with interquartile range (yes); gray symbols illustrate individual measurements. Red dashed lines represent MUMC+ ROTEM reference ranges and blue dashed lines represent tPA ROTEM reference ranges as determined by Kuiper et al. (20). CT, clotting time; CFT, clot formation time; MCF, maximum clot firmness; LY60, lysis index at 60 min; LOT, lysis onset time; LT, lysis time.

EXTEM CT was largely prolonged during the first 2 weeks after intubation in COVID-19 ICU patients, as illustrated in Figure 2A. However, over time median and IQR decreased toward the reference range. Forty measurements (37.4%) with normal PT presented with prolonged EXTEM CT. Though the number of prolonged CT values remains consistently higher than PT, Table 4 illustrates that both tests classify fewer patients as prolonged as weeks from intubation increase.


Table 4. Number (percentage) of prolonged EXTEM clotting time (CT) and prothrombin time (PT) measurements.

The results show an increase in MCF for both EXTEM and FIBTEM assays, which persisted over 6 weeks, as illustrated with medians above the reference range's upper limit in Figures 2C,E. This hypercoagulability was considered more evident in FIBTEM compared to EXTEM as median values (and IQR) exceeded the reference range more prominently. Respectively, 81 and 99% of all EXTEM and FIBTEM MCF values were above the reference range. Linear regression analyses (Figure 3) showed that fibrinogen was more positively associated with FIBTEM MCF [β (95% CI) = 0.86 (0.77–0.98)] as compared to EXTEM MCF [β (95% CI) = 0.48 (0.31–0.66)]. Additionally, EXTEM MCF was positively associated with platelet count [R2 = 0.55, β (95% CI) = 0.74 (0.62–0.87), p-value < 0.001].


Figure 3. Linear regression between fibrinogen and maximum clot firmness (MCF) of EXTEM and FIBTEM. Presented are R2 and standardized β coefficient (95% CI). **p < 0.001.

TPA ROTEM fibrinolysis parameters, presented in Figures 2F,G,H, illustrate initiation (LOT) and propagation (LT and Ly60) of clot breakdown. LOT and LT were unmeasurable within 2 h in, respectively, 11 and 56 samples, suggesting no or little clot breakdown occurred. Clot breakdown of 90% (LT) was not achieved in more than half of the samples.

A hypofibrinolytic phenotype was illustrated in all tPA ROTEM parameters, though most evident in LT. The median for LT in most weeks is consistent with the tPA measurements' cutoff value (120 min). Figure 2F further illustrates that the median of LOT is continuously above the reference range.

Data stratified by mortality are provided in Supplementary Tables 1, 2. Deceased patients had persisting higher D-dimer and prolonged aPTT values, reflecting the over-representation of ECMO and RRT in deceased patients. Additional sensitivity analyses excluding the six patients on ECMO during their stay in the ICU did not alter any of the conclusions.


Our serial and comprehensive study to characterize COVID-19-associated coagulopathy by (tPA) thromboelastometry has four main findings: (1) a persisting increase in MCF that was more prominent in FIBTEM compared to EXTEM, illustrating hypercoagulability that is at least partly fibrinogen dependent; (2) a prolonged EXTEM CT in 54% of all measurements; (3) a discordance between EXTEM CT and PT observed in 37% of the samples, possibly suggesting missing coagulopathy recognition in routine hemostatic assays; and (4) a persistently increased LOT and LT measured with tPA ROTEM, illustrating hypofibrinolysis. Overall, COVID-19-positive ICU patients presented with persisting hypercoagulability and hypofibrinolysis despite anticoagulant treatment.

Of note, patients received different, individualized anticoagulant treatment regimens. COVID-19 patients mainly received prophylactic LMWH, therapeutic LMWH, or (therapeutic) UFH, as illustrated in Table 3. The inclusion of all mechanically ventilated COVID-19 ICU patients, irrespective of ECMO and/or CRRT, not only makes for a heterogeneous group but also reflects the clinical reality of the COVID-19 ICU population. Sensitivity analyses excluding patients receiving ECMO during ICU admission did not alter any of the conclusions. Additionally, heparins execute limited effects on the ROTEM parameters; EXTEM and FIBTEM contain a heparin neutralizing agent (polybrene) and no clear fibrinolytic effect of heparins can be observed in tPA ROTEM (28). Taken together, the individualized treatment regimens, though illustrating the heterogeneous ICU population, do not in itself affect ROTEM results.

Although previous studies (15, 2937) partly addressed these issues, measurements were either not serial, prospective, or did not assess both coagulation and fibrinolysis within a well-defined cohort (22) (Supplementary Figure 1). Hypercoagulability in COVID-19-positive ICU patients, illustrated by increased clot stiffness/amplitude in viscoelastic assays, has been extensively reported in the literature (15, 2937). Routine hemostatic assays, such as PT, were often within reference ranges, highlighting the relevance of the addition of global hemostatic assays to identify coagulopathy in these patients. Only two studies performed viscoelastic testing in COVID-19 patients on multiple occasions, with a follow-up period of 10 and 14 days showing a significant decrease in clot stiffness over time and a significant decrease in FIBTEM MCF, respectively (29, 30). This decrease in clot stiffness (MCF) over time was not observed in our cohort. Notably, the decrease in fibrinogen over the first 2 weeks was more pronounced in both previous studies (29, 30), compared to our cohort, possibly explaining the observed lack of decrease in clot stiffness. The diagnostic accuracy of ROTEM to classify sepsis-associated coagulopathy has been studied and demonstrated an evident ability of EXTEM to discriminate patients with and without disseminated intravascular coagulation (DIC) (38). Though COVID-19-associated coagulopathy is distinct from DIC (14), these results suggest viscoelastic testing's applicability in identifying coagulopathy in critically ill patients.

Several studies report decreased fibrinolysis in viscoelastic tests (15, 3436). COVID-19 patients present with paradoxical fibrinolytic findings; severely increased D-dimer on the one hand, but a hypofibrinolytic viscoelastic profile on the other. Additionally, both pro-fibrinolytic tPA and anti-fibrinolytic plasminogen activator inhibitor-1 (PAI1) are increased (37, 39). Assessment of fibrinolysis in routine viscoelastic assays remains limited since little fibrinolysis is likewise seen in patients and healthy controls. TPA-based ROTEM assays may provide a complete view of the fibrinolytic properties in these patients. Kuiper et al. (20) demonstrated a striking difference in ROTEM fibrinolysis parameters between healthy controls and septic patients, the latter presenting with a hypofibrinolytic profile. Nougier et al. (39) showed a similar hypofibrinolytic profile with tPA-activated ROTEM in a small cohort of COVID-19 ICU patients (n = 23), using a higher tPA concentration (0.625 μg/ml) as compared to our assay. A decreased fibrinolytic potential was associated with thromboembolic events in COVID-19 patients in other studies (15, 39). Though confirming a general hypofibrinolytic state persisting over several weeks, the extent of hypofibrinolysis remains somewhat elusive in our cohort since most patients presented with LT at cutoff value (120 min). A higher in vitro tPA concentration might be beneficial to differentiate the level of hypofibrinolysis in COVID-19 ICU patients. However, other ROTEM fibrinolysis markers (LOT and LY60) might be able to provide this differentiation in our current validated and standardized assay. We hypothesize that tPA ROTEM fibrinolysis parameters predict the occurrence of thromboembolic events in COVID-19 ICU patients, although further research analyzing clinical outcomes is required. In addition to tPA ROTEM, other fibrinolysis markers (e.g., α2-antiplasmin, tPA, and PAI-1) are of interest to unravel the pathophysiology underlying the observed hypofibrinolytic phenotype.

The major determinants of overall clot strength in viscoelastic testing are platelets and fibrinogen (40). The FIBTEM assay, in which platelets are inhibited by cytochalasin D, is mostly dependent on fibrinogen for clot formation. Increased FIBTEM MCF, as measured in COVID-19 patients, is likely caused by high fibrinogen levels. One might consider the hypercoagulability, as measured with EXTEM MCF above the reference range, to be dependent on fibrinogen as well. However, linear regression shows a large difference in the association of fibrinogen with FIBTEM compared to EXTEM.

Additionally, the decrease in fibrinogen and, consequently, FIBTEM MCF was not observed in EXTEM MCF. Together these data may suggest that platelet (activation) contributes to increased clot stiffness. Indeed, Manne et al. showed increased platelet activation and aggregation in COVID-19 ICU patients (41), and EXTEM MCF was positively associated with platelet count in our patients. Thus, although hyperfibrinogenemia is undoubtedly associated with increased clot stiffness, it is likely that other factors, including platelets, contribute to COVID-19 hypercoagulability.

An unexpected EXTEM CT prolongation (>77 s) was observed in 29 (81%) patients and 54% of all measurements. Considering the high doses of heparins administered (specifically UFH), the heparin inhibiting potential of polybrene in the EXTEM reagent might be exceeded. However, according to the manufacturer's instructions, the anti-Xa activity of 2–3 IU/ml should not result in a prolonged EXTEM CT. Since the highest measured anti-Xa level was 1.85 IU/ml, it is unlikely that heparins cause this prolongation. In addition, the EXTEM CT prolongation was not matched by PT. Although these assays have different reagents and measuring techniques, both represent the initiation of clot formation and use tissue factor as the activator. Previous studies in pediatric surgery patients and patients with various hemostatic disorders (including patients undergoing cardiac surgery) illustrated PT prolongations with normal EXTEM CT results (42, 43). Our results show a contrary course with a predominantly normal PT and a prolonged EXTEM CT, as shown in Table 4. Even though these assays both reflect the initiation of clot formation via the extrinsic pathway, the detection mechanism, reagent concentrations, and patient material (whole blood vs. plasma) used are completely different. These factors likely contribute to the observed discordance between the assays. Since the prolonged EXTEM CT is more in line with the observed COVID-19-associated coagulopathy compared to the predominantly normal PT results, we believe that ROTEM results may describe the hemostatic status in these patients more accurately.

Alternatively, a prolonged clotting time in COVID-19 patients could also be due to increased concentrations of D-dimer, fibrinogen, and, thus, fibrinogen degradation products (FDPs). As such, several animal studies have shown an inhibitory effect of FDPs on clotting assays (44, 45). However, the concentration of FDPs in these studies (0.05–2.5 g/L) largely exceeded our patients' expected concentration, deeming a causative role unlikely. Additionally, only a minimal association between D-dimer and EXTEM CT and PT was achieved (data not shown). All in all, we observe a prolonged EXTEM CT, which was unexplained.

Our study has some strengths and limitations. Strengths include the prospective and longitudinal nature of the study. To our knowledge, no serial assessment of tPA-based ROTEMs or follow-up after more than 2 weeks have been reported in the current literature, making this the first study to perform such comprehensive follow-up in a COVID-19 population. Furthermore, no previous study assessed both coagulation and fibrinolysis as thoroughly in these patients.

Limitations to our study mainly involve its generalizability and applicability in clinical practice. Due to the buildup of our cohort, not all patients were followed from ICU admission. As a result of this, we collected extensive follow-up over a long period, but conclusions concerning individual disease courses cannot be drawn from our data. The inclusion of all mechanically ventilated COVID-19 patients, irrespective of ECMO and CRRT, makes for a heterogeneous population, suggesting careful interpretation of the results and limited generalizability to other ICU populations. Additionally, because of the relatively small number of patients, no conclusions can be drawn regarding the risk for adverse outcome (VTE or mortality), which would have added to the study value. We focused on studying whether altered ROTEM parameters persisted or changed over time in severe COVID-19 infection. Therefore, we used descriptives of serial measurements. Our observations indicate that any altered ROTEM parameter studied during the course of severe COVID-19 infection can be used to investigate its associations with other coagulation markers and outcome, independent of the moment of measurement. This is important when pooling ROTEM data of several COVID-19 observational studies that used different sampling moments in time during the course of severe COVID-19 infection. Furthermore, beyond 6 weeks, insufficient measurements remained for analysis. However, the consistency of the results during the first 6 weeks of ICU admission suggests that little change is expected by extending this timeframe.

ROTEM analysis was not performed in all patients of the MaastrICCht cohort, as explained previously. Therefore, we compared the baseline characteristics of both the MaastrICCht and hemostasis subcohort to exclude selection bias (data not shown). Disease severity in the two cohorts is nearly identical, as indicated by similar SOFA and APACHE scores, and almost all other baseline characteristics are comparable except for comorbidities. Notably, within our study population, only a few comorbidities were present compared to the entire MaastrICCht cohort and other study populations. It is common practice in the Netherlands to initiate conservative (and/or palliative) care in COVID-19 patients with preexisting frailty or severe comorbidities. Therefore, only relatively healthy patients were transferred to the ICU for mechanical ventilation. A recent study in our center showed that, of all admitted COVID-19 patients, 35.2% had some form of treatment restrictions (46). This shows that COVID-19 related coagulopathy is not limited to patients with multiple comorbidities since our study sample with relatively healthy individuals presents with significant hemostatic abnormalities. This implies that the severity of the COVID-19-associated coagulopathy might be even more pronounced in other, more severely affected, ICU populations.

From a practical point of view, it must be acknowledged that tPA ROTEM measurement is labor-intensive and requires the “old” ROTEM delta device with manual pipetting, limiting its applicability to smaller hospitals. It might also not be feasible for huge patient volumes. Nonetheless, we did manage to set up and execute tPA ROTEM measurements during the first COVID-19 wave and expect that, with proper logistics, larger hospitals could still manage routine tPA ROTEM measurements for their ICU patients. Application of tPA ROTEM might aid in the characterization of COVID-19 and similar septic pathologies.

In conclusion, we found a persisting hypercoagulability and hypofibrinolysis, indicating severe and persisting COVID-19-associated coagulopathy as measured with (tPA) ROTEM. We hypothesize that hypercoagulability and hypofibrinolysis are associated with thromboembolic complications and aim to further study the association between ROTEM parameters, clinical variables, and outcomes in the MaastrICCht cohort. Future research should focus on applying ROTEM and other global hemostatic assays, such as thrombin generation, on identifying critically ill COVID-19 patients at risk for thrombosis. Comprehensive laboratory testing might help guide the optimal anticoagulant treatment strategy to ultimately improve morbidity and mortality in COVID-19 patients.

Data Availability Statement

The datasets presented in this article are not readily available because no concrete agreements on data sharing have been made yet. Before any data is shared outside the MUMC+, a datasharing plan will be drawn up, in consultation with the data officer, that conforms to relevant laws and regulations concerning personal data. Requests to access the datasets should be directed to Anne-Marije Hulshof,

Ethics Statement

The studies involving human participants were reviewed and approved by Medisch Ethische Toetsingscomissie (METC) azM/UM, Maastricht UMC+, P. Debyelaan 25, 5800, 6202 AZ Maastricht azM/UM (2020-1565/ 300523). The patients/participants provided their informed consent to participate in this study.

Author Contributions

A-MH and RB contributed to the design of the manuscript, performed statistical analysis, and were the main authors. MM collected clinical data. PV performed laboratory analysis. BB and IH developed the design of the MaastrICCht cohort. MM, TB, JS, RO, BS, AS, HS, PV, and IH critically reviewed the manuscript. BB, HT, and YH contributed to the concept and design of the manuscript, and revised the intellectual content. All authors contributed to the article and approved the submitted version.

Conflict of Interest

RO has received research support an honoraria from Bayer, Pfizer/BMS, Leo Pharma, Portola, and Sanofi. HT has received grants from Bayer and Pfizen, is a consultant for Alveron, and a share holder of Coagulation Profile. YH reports that ROTEM reagents were previously provided for research and development.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


We would like to thank all patients for participating in this research. We would also like to extend our gratitude to all the nurses, technicians, doctors, and medical students for their contribution to the MaastrICCht cohort.

Supplementary Material

The Supplementary Material for this article can be found online at:


1. Magro C, Mulvey JJ, Berlin D, Nuovo G, Salvatore S, Harp J, et al. Complement associated microvascular injury and thrombosis in the pathogenesis of severe COVID-19 infection: a report of five cases. Transl Res. (2020) 220:1–13. doi: 10.1016/j.trsl.2020.04.007

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Wichmann D, Sperhake JP, Lutgehetmann M, Steurer S, Edler C, Heinemann A, et al. Autopsy findings and venous thromboembolism in patients with COVID-19: a prospective cohort study. Ann Intern Med. (2020) 173:268–77. doi: 10.7326/M20-2003

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Grillet F, Behr J, Calame P, Aubry S, Delabrousse E. Acute pulmonary embolism associated with COVID-19 pneumonia detected with pulmonary CT angiography. Radiology. (2020) 296:E186–8. doi: 10.1148/radiol.2020201544

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Klok FA, Kruip M, van der Meer NJM, Arbous MS, Gommers D, Kant KM, et al. Incidence of thrombotic complications in critically ill ICU patients with COVID-19. Thromb Res. (2020) 191:145–7. doi: 10.1016/j.thromres.2020.04.013

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Klok FA, Kruip M, van der Meer NJM, Arbous MS, Gommers D, Kant KM, et al. Confirmation of the high cumulative incidence of thrombotic complications in critically ill ICU patients with COVID-19: An updated analysis. Thromb Res. (2020) 191:148–50. doi: 10.1016/j.thromres.2020.04.041

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Llitjos JF, Leclerc M, Chochois C, Monsallier JM, Ramakers M, Auvray M, et al. High incidence of venous thromboembolic events in anticoagulated severe COVID-19 patients. J Thromb Haemost. (2020) 18:1743–6. doi: 10.1111/jth.14869

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Middeldorp S, Coppens M, van Haaps TF, Foppen M, Vlaar AP, Muller MCA, et al. Incidence of venous thromboembolism in hospitalized patients with COVID-19. J Thromb Haemost. (2020) 18:1995–2002. doi: 10.1111/jth.14888

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Poissy J, Goutay J, Caplan M, Parmentier E, Duburcq T, Lassalle F, et al. Pulmonary embolism in patients with COVID-19: awareness of an increased prevalence. Circulation. (2020) 142:184–6. doi: 10.1161/CIRCULATIONAHA.120.047430

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Thomas W, Varley J, Johnston A, Symington E, Robinson M, Sheares K, et al. Thrombotic complications of patients admitted to intensive care with COVID-19 at a teaching hospital in the United Kingdom. Thromb Res. (2020) 191:76–7. doi: 10.1016/j.thromres.2020.04.028

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Cui S, Chen S, Li X, Liu S, Wang F. Prevalence of venous thromboembolism in patients with severe novel coronavirus pneumonia. J Thromb Haemost. (2020) 18:1421–4. doi: 10.1111/jth.14830

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Nopp S, Moik F, Jilma B, Pabinger I, Ay C. Risk of venous thromboembolism in patients with COVID-19: a systematic review and meta-analysis. Res Pract Thromb Haemost. (2020) 4:1178–91. doi: 10.1002/rth2.12439

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Bruggemann RAG, Spaetgens B, Gietema HA, Brouns SHA, Stassen PM, Magdelijns FJ, et al. The prevalence of pulmonary embolism in patients with COVID-19 and respiratory decline: a three-setting comparison. Thromb Res. (2020) 196:486–90. doi: 10.1016/j.thromres.2020.10.012

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Santamarina MG, Boisier Riscal D, Beddings I, Contreras R, Baque M, Volpacchio M, et al. COVID-19: what iodine maps from perfusion CT can reveal-a prospective cohort study. Crit Care. (2020) 24:619. doi: 10.1186/s13054-020-03333-3

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Levi M, Thachil J, Iba T, Levy JH. Coagulation abnormalities and thrombosis in patients with COVID-19. Lancet Haematol. (2020) 7:e438–40. doi: 10.1016/S2352-3026(20)30145-9

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Wright FL, Vogler TO, Moore EE, Moore HB, Wohlauer MV, Urban S, et al. Fibrinolysis shutdown correlation with thromboembolic events in severe COVID-19 infection. J Am Coll Surg. (2020) 231:193–203.e1. doi: 10.1016/j.jamcollsurg.2020.05.007

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Al-Samkari H, Karp Leaf RS, Dzik WH, Carlson JCT, Fogerty AE, Waheed A, et al. COVID-19 and coagulation: bleeding and thrombotic manifestations of SARS-CoV-2 infection. Blood. (2020) 136:489–500. doi: 10.1182/blood.2020006520

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Busch MH, Timmermans S, Nagy M, Visser M, Huckriede J, Aendekerk JP, et al. Neutrophils and contact activation of coagulation as potential drivers of Covid-19. Circulation. (2020) 142:1787–90. doi: 10.1161/CIRCULATIONAHA.120.050656

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Medcalf RL, Keragala CB, Myles PS. Fibrinolysis and COVID-19: A plasmin paradox. J Thromb Haemost. (2020) 18:2118–22. doi: 10.1111/jth.14960

CrossRef Full Text | Google Scholar

19. Vries MJA, Macrae F, Nelemans PJ, Kuiper G, Wetzels RJH, Bowman P, et al. Assessment and determinants of whole blood and plasma fibrinolysis in patients with mild bleeding symptoms. Thromb Res. (2019) 174:88–94. doi: 10.1016/j.thromres.2018.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Kuiper GJ, Kleinegris MC, van Oerle R, Spronk HM, Lance MD, Ten Cate H, et al. Validation of a modified thromboelastometry approach to detect changes in fibrinolytic activity. Thromb J. (2016) 14:1. doi: 10.1186/s12959-016-0076-2

PubMed Abstract | CrossRef Full Text | Google Scholar

21. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. (2007) 4:e296. doi: 10.1371/journal.pmed.0040296

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Tas J, van Gassel RJJ, heines SJH, Mulder MMG, heijnen NFL, Acampo - de Jong MJ, et al. Serial measurements in COVID-19-induced acute respiratory disease to unravel heterogeneity of the disease course: design of the Maastricht Intensive Care COVID cohort (MaastrICCht). BMJ Open. (2020) 10:e040175.

PubMed Abstract | Google Scholar

23. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. (2016) 3:160018. doi: 10.1038/sdata.2016.18

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Prokop M, van Everdingen W, van Rees Vellinga T, Quarles van Ufford H, Stoger L, Beenen L, et al. CO-RADS: a categorical CT assessment scheme for patients suspected of having COVID-19-definition and evaluation. Radiology. (2020) 296:E97–104. doi: 10.1148/radiol.2020201473

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Federatie Medisch Specialisten. Leidraad COVID-19 Coagulopathie. Available from: (accessed April 16, 2020).

Google Scholar

26. Streng AS, Delnoij TS, Mulder MM, Sels JW, Wetzels RJ, Verhezen PW, et al. Monitoring of unfractionated heparin in severe COVID-19: an observational study of patients on CRRT ECMO. TH Open. (2020) 04:e365–75. doi: 10.1055/s-0040-1719083

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Bels JLM, van Kuijk SMJ, Ghossein-Doha C, Tijssen FH, van Gassel RJJ, Tas J, et al. Decreased serial scores of severe organ failure assessments are associated with survival in mechanically ventilated patients; the prospective Maastricht Intensive Care COVID cohort. J Crit Care. (2020) 62:38–45. doi: 10.1016/j.jcrc.2020.11.006

PubMed Abstract | CrossRef Full Text | Google Scholar

28. van de Berg TW, Hulshof AM, Nagy M, van Oerle R, Sels JEM, van Bussel BCT, et al. Suggestions for global coagulation assays for the assessment of COVID-19 associated hypercoagulability. Thromb Res. (2020) 201:84–9. doi: 10.1016/j.thromres.2021.02.026

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Ranucci M, Ballotta A, Di Dedda U, Bayshnikova E, Dei Poli M, Resta M, et al. The procoagulant pattern of patients with COVID-19 acute respiratory distress syndrome. J Thromb Haemost. (2020) 18:1747–51. doi: 10.1111/jth.14854

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Pavoni V, Gianesello L, Pazzi M, Stera C, Meconi T, Frigieri FC. Evaluation of coagulation function by rotation thromboelastometry in critically ill patients with severe COVID-19 pneumonia. J Thromb Thrombolysis. (2020) 50:281–6. doi: 10.1007/s11239-020-02130-7

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Masi P, Hékimian G, Lejeune M, Chommeloux J, Desnos C, Chambrun MPD, et al. Systemic inflammatory response syndrome is a major contributor to COVID-19-associated coagulopathy: insights from a prospective singlecenter cohort study. Circulation. (2020) 142:611–4. doi: 10.1161/CIRCULATIONAHA.120.048925

CrossRef Full Text | Google Scholar

32. Spiezia L, Boscolo A, Poletto F, Cerruti L, Tiberio I, Campello E, et al. COVID-19-related severe hypercoagulability in patients admitted to intensive care unit for acute respiratory failure. Thromb Haemost. (2020) 120:998–1000. doi: 10.1055/s-0040-1714350

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Maatman TK, Jalali F, Feizpour C, Douglas A 2nd, McGuire SP, Kinnaman G, et al. Routine venous thromboembolism prophylaxis may be inadequate in the hypercoagulable state of severe coronavirus disease 2019. Crit Care Med. (2020) 48:e783–90. doi: 10.1097/CCM.0000000000004466

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Creel-Bulos C, Auld SC, Caridi-Scheible M, Barker N, Friend S, Gaddh M, et al. Fibrinolysis shutdown and thrombosis in a COVID-19 ICU. Shock. (2020) 3:316–20. doi: 10.1097/SHK.0000000000001635

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Panigada M, Bottino N, Tagliabue P, Grasselli G, Novembrino C, Chantarangkul V, et al. Hypercoagulability of COVID-19 patients in intensive care unit: a report of thromboelastography findings and other parameters of hemostasis. J Thromb Haemost. (2020) 18:1738–42. doi: 10.1111/jth.14850

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Mortus JR, Manek SE, Brubaker LS, Loor M, Cruz MA, Trautner BW, et al. Thromboelastographic results and hypercoagulability syndrome in patients with Coronavirus Disease 2019 who are critically ill. JAMA Netw Open. (2020) 3:e2011192. doi: 10.1001/jamanetworkopen.2020.11192

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Blasi A, von Meijenfeldt FA, Adelmeijer J, Calvo A, Ibanez C, Perdomo J, et al. In vitro hypercoagulability and ongoing in vivo activation of coagulation and fibrinolysis in COVID-19 patients on anticoagulation. J Thromb Haemost. (2020) 18:2646–53. doi: 10.1111/jth.15043

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Muller MCA, Meijers JC, van Meenen DM, Thachil J, Juffermans NP. Thromboelastometry in critically ill patients with disseminated intravascular coagulation. Blood Coagul Fibrinolysis. (2019) 30:181–7. doi: 10.1097/MBC.0000000000000808

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Nougier C, Benoit R, Simon M, Desmurs-Clavel H, Marcotte G, Argaud L, et al. Hypofibrinolytic state and high thrombin generation may play a major role in sars-cov2 associated thrombosis. J Thromb Haemost. (2020) 18:2215–9. doi: 10.1111/jth.15016

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Bolliger D, Seeberger MD, Tanaka KA. Principles and practice of thromboelastography in clinical coagulation management and transfusion practice. Transfus Med Rev. (2012) 26:1–13. doi: 10.1016/j.tmrv.2011.07.005

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Manne BK, Denorme F, Middleton EA, Portier I, Rowley JW, Stubben C, et al. Platelet gene expression and function in patients with COVID-19. Blood. (2020) 136:1317–29. doi: 10.1182/blood.2020007214

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Haas T, Spielmann N, Mauch J, Madjdpour C, Speer O, Schmugge M, et al. Comparison of thromboelastometry (ROTEM(R)) with standard plasmatic coagulation testing in paediatric surgery. Br J Anaesth. (2012) 108:36–41. doi: 10.1093/bja/aer342

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Scala E, Coutaz C, Gomez F, Alberio L, Marcucci C. Comparison of ROTEM sigma to standard laboratory tests and development of an algorithm for the management of coagulopathic bleeding in a tertiary center. J Cardiothorac Vasc Anesth. (2020) 34:640–9. doi: 10.1053/j.jvca.2019.10.016

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Mischke R, Wolling H. Influence of fibrinogen degradation products on thrombin time, activated partial thromboplastin time and prothrombin time of canine plasma. Haemostasis. (2000) 30:123–30. doi: 10.1159/000022534

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Mischke R, Wolling H, Nolte I. Detection of anticoagulant activities of isolated canine fibrinogen degradation products X, Y, D and E using resonance thrombography. Blood Coagul Fibrinolysis. (2004) 15:81–8. doi: 10.1097/00001721-200401000-00013

PubMed Abstract | CrossRef Full Text | Google Scholar

46. van Dam PMEL, Zelis N, van Kuijk SMJ, Linkens AEMJH, Bruggemann RRAG, Spaetgens B, et al. Performance of prediction models for short term outcome in COVID-19 patients in the emergency department: a retrospective study. Ann Med. (2020) 53:402–9. doi: 10.1101/2020.11.25.20238527

PubMed Abstract | CrossRef Full Text | Google Scholar

Keywords: COVID-19, fibrinolysis, pulmonary embolism, thromboelastometry (ROTEM®), thrombosis

Citation: Hulshof AM, Brüggemann RAG, Mulder MMG, van de Berg TW, Sels JEM, Olie RH, Spaetgens B, Streng AS, Verhezen P, van der Horst ICC, Ten Cate H, Spronk HMH, van Bussel BCT and Henskens YMC (2021) Serial EXTEM, FIBTEM, and tPA Rotational Thromboelastometry Observations in the Maastricht Intensive Care COVID Cohort—Persistence of Hypercoagulability and Hypofibrinolysis Despite Anticoagulation. Front. Cardiovasc. Med. 8:654174. doi: 10.3389/fcvm.2021.654174

Received: 15 January 2021; Accepted: 05 March 2021;
Published: 26 April 2021.

Edited by:

Marlien Pieters, North-West University, South Africa

Reviewed by:

Job Harenberg, Heidelberg University, Germany
Robert Campbell, The University of Utah, United States

Copyright © 2021 Hulshof, Brüggemann, Mulder, van de Berg, Sels, Olie, Spaetgens, Streng, Verhezen, van der Horst, Ten Cate, Spronk, van Bussel and Henskens. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Anne-Marije Hulshof,

These authors have contributed equally to this work

ORCID: Anne-Marije Hulshof
Renée A. G. Brüggemann
Renske H. Olie
Bart Spaetgens
Iwan C. C. van der Horst
Hugo ten Cate
Henri M. H. Spronk
Bas C. T van Bussel