In this study, the protocol was applied for testing three novel PDVs and one clinical approved ventilator as reference: the GirVent (Girtec AG, Nänikon, Switzerland), the High Energy Ventilator (HEV, CERN, Geneva, Switzerland) (8), and the breathe (ETH, Zurich, Switzerland) (9). The three PDVs selected represent the three different most commonly used functional principles. Each PDV tested in this study is developed by the corresponding team and is still under development. The test status presented here dates from December 2020. The GirVent is a pressure-controlled ventilator based on a turbine blower with two oxygen (O₂) ports: one port is realized by a direct injection of O₂ into the patient tubing (LP1), while the other includes a 2-L balloon as reservoir (LP2). The HEV ventilator is based on a controlled valve system with a buffer reservoir in which the O₂ concentration of the supplied air can be adjusted. The volume-controlled breathe ventilator (13) is built on the concept of the MIT E-Vent (14) and consists of a resuscitator bag (AmbuBag, Ambu, Ballerup, Denmark), which is periodically compressed by two paddles. The Hamilton T1 (Hamilton, Bonaduz, Switzerland) was included as a reference ventilator and was tested using the same test protocol as the PDVs.

Table 1 depicts the relevant features of the ventilators tested in this study. Each ventilator system was tested using the intended auxiliary materials supplied by the developer, that is, tubing, sensors, high-efficiency particulate air filters, and PEEP valves. The ventilation modes tested were pressure-controlled continuous mandatory ventilation (PC-CMV) and volume-controlled continuous mandatory ventilation (VC-CMV). For the Hamilton T1, the volume-targeted (S)CMV+ and pressure-controlled PCV+ mode was tested. In each ventilator system, the flow and pressure sensors are placed at the distal end of the patient tubing. The breathe and GirVent ventilators do not have an O₂ sensor and do not include a closed-loop FiO₂ control system. Both ventilators operate with a low-pressure O₂ supply in feed-forward settings where the O₂ flow is adjusted directly on the O₂ source. The HEV ventilator operates with a high-pressure O₂ supply and comprises a sensor that measures the O₂ concentration in the buffer reservoir. Hence, in the HEV, FiO₂ can be controlled in a closed loop similarly to the Hamilton T1. The Hamilton T1 can be operated with a low- (feed-forward) or a high-pressure (closed-loop) supply.

Each test described in this study was performed on a mechanical lung simulator (TestChest V3, Organis GmbH, Landquart, Switzerland). In brief, the TestChest can model different lung mechanics by varying airway resistance (R_aw) and respiratory system compliance (C_rs). Spontaneous breathing cycles can be set by means of a negative occlusion pressure (P_0.1) and the respiratory rate (RR). The TestChest includes a pressure sensor measuring the airway pressure (P_aw) and an O₂ sensor with a response time <6 s for 90% of the final value placed in the airway. A flow sensor SFM3019 (Sensirion AG, Stäfa, Switzerland) was placed at the inlet of the TestChest to measure the resultant air flow. The ventilators were connected to the TestChest via an endotracheal tube (ETT) (SafetyClear I.D. 7.5, Teleflex Medical GmbH, Fellbach, Germany), as shown in Figure 1. For the efficiency of the O₂ use and the response to a change of O₂ supply, a medical-grade O₂ tank was used, which includes an adjustable flow rate valve for low-pressure supply and a high-pressure port (four bar).

The test protocol is divided into three categories: testing of (1) the accuracy of control systems and instruments, (2) the use of O₂ and the response to a change in O₂ supply, and (3) the ventilator triggering. The TestChest was calibrated with its internal program prior to the initiation of the study.

Signals were recorded via the TestChest V3 and the SFM3019 mass flow sensor at 100 Hz and processed in MATLAB R2020a (The Mathworks Inc., Natick, USA). The data of each experiment were shifted and windowed to a length of 50 s to eliminate phase shift differences between the single measurement sets, as shown in Table 2. The signal of P_aw and the flow measured in the TestChest (Q_TestChest) were processed using a Gaussian filter with, a window length of 200 ms (20 samples). The RR was calculated using the time difference between two onsets of the inspiratory flow. The onset of the inspiratory flow starts when Q_TestChest reached a specified threshold. This threshold was defined per ventilator as the most sensitive value at which the fluctuation of Q_TestChest does not cause an onset detection. The inspiratory time (TI) and expiratory time (TE) are defined as the time during which Q_TestChest is above the positive threshold (inspiratory flow) and below the negative threshold (expiratory flow), respectively.

For each test performed, P_aw was measured inside the TestChest (Figure 1); hence, the PIP and PEEP displayed in the results report the pressure recorded downstream to the ETT. The PIP was detected as the peak value of P_aw averaged over 40 ms (four samples) symmetrically around the detected point signal. The PEEP value was determined as the minimum value of P_aw in the time interval of two consecutive breaths that was closer to the inspiration of the latter breath. The V_t was calculated by integrating the flow sensor signal. I:E is defined as the ratio of inspiratory time (T_I) to expiratory time (T_E) (T_I:T_E).

A Gaussian filter with a window length of 30 s (3,000 samples) was applied to the FiO₂ signal recorded by the TestChest. The filtered signal averages the FiO₂ values, while the unfiltered signal shows the FiO₂ fluctuations during each breath cycle. The starting time of the t₉₀ test was set when the unfiltered FiO₂ signal exceeded an FiO₂ of 22%. Maximum O₂ step response and the 90% O₂ step response were determined using the filtered signal.

Parameters for the trigger signals (PD, TPM, TDT, PTP) were evaluated from the P_aw signal. The start of inspiration is defined when exceeding 10% of the maximum gradient recorded during the pressure drop in the inspiratory phase.

The overall relative errors of individual parameters recorded over all experiments (Table 2) are depicted in Figure 3. The relative error was calculated as the ratio of the measured error to the set value with the measured error being the difference between the value measured and the set value on the ventilator. Table 3 lists all the measured mean values, including standard deviation (SD). The PDVs showed generally larger relative mean errors and SD than the Hamilton T1 (PCV+ mode). In addition, more pronounced outliers were observed at individual measurement settings. Only the V_t measured in the breathe and the RR in all PDVs were in a range similar to those of the Hamilton T1, with no parameter changes having a substantial effect on the accuracy. The two passive PEEP valves used in the HEV and the breathe showed a larger variation than the actively applied PEEP in the GirVent.

Figures 4, 5 show the effects of the varying parameters on the PIP and the V_t. An increase in the RR, a decrease in the inspiratory time, and an increase in R_aw showed the largest influence on the accuracy of the PIP. Moreover, the largest PIP error in each pressure-controlled ventilator was observed in the same setting of a PIP of 30 cmH₂O and an RR of 20 bpm. By contrast, in the volume-controlled mode, only increasing the RR (breathe) or increasing R_aw (Hamilton T1 (S)CMV+ mode) had a prominent influence on the V_t error. Other parameter changes had only a marginal impact on the accuracy of the parameters tested. Detailed plots and data of each measurement and parameter change, as well as the flow profiles, recorded for the base setting for each ventilator are provided in the Supplementary material.

The ISO 80601-2-12 norm and the MHRA specify an acceptable performance tolerance of ± (2 + (4 % of the actual reading)) cmH₂O for the pressure value and ± (4,0 + (15% of the actual volume expired through the patient connection port)) mL. The acceptable tolerance is given in Figures 4, 5. For the respiratory rate and I:E, no performance tolerance was stated in the norm. Therefore, for the tests performed in this study, we defined a tolerance of 5 and 10%, respectively.

The achieved fractions of inspired O₂ (FiO₂) at three O₂ flow rates (2 L min⁻¹, 4 L min⁻¹, and 6 L min⁻¹) for the low- and high-pressure supply (four bar) of each ventilator are presented in Figure 6. Fluctuations of FiO₂ were observed in each ventilator and are indicated by error bars (minimum and maximum values). Generally, since the results are all relative values obtained by using the O₂ sensor in steady state and the tests were all performed with the same O₂ sensor, the delay of the sensor was neglectable.

The achieved mean FiO₂ at steady state with the breathe was in a similar range or higher than that achieved with the Hamilton T1 with the low-pressure supply. By contrast, FiO₂ measured with the GirVent was considerably low in the current design stage. In the tests with the high-pressure O₂ supply, the Hamilton T1 achieved the intended O₂ concentration with minor deviations from the set value. FiO₂ provided by the HEV was constantly higher than the set concentration in the graphic UI.

The t₉₀ values for the low-pressure supply (12 L min⁻¹) and the high-pressure supply (4 bar, O₂ at 100%), as well as the FiO₂ achieved in the airway (FiO_{2, t90}), are shown in Figure 6. GirVent showed for both O₂ ports (LP-1 and LP-2) t₉₀ values similar to those of the Hamilton T1; however, the achieved FiO_{2, t90} values were also lower than those of the other PDVs. For the breathe (LP O₂ port) and the HEV (HP O₂ port), the t₉₀ values were higher than the values of the corresponding ports in the Hamilton T1, but both ventilators achieved FiO_{2, t90} of 1.00.

The trigger signals and pressurization characteristics of each ventilator for the two P_0.1 level set (2 cmH₂O/100 ms and 4 cmH₂O/100 ms) are shown in Figure 7. Exact values for all measurements are depicted in the Supplementary material.

For both values of P_0.1, PD and TDT values observed in the PDVs were generally larger than those of the reference ventilator. The HEV presented the smallest difference to the reference ventilator and achieved faster TDT at a P_0.1 of 2 cmH₂O/100 ms.

The pressurization capacity of the PDVs was notably lower than that of the reference ventilator, which achieved values between 51.5 and 80.1%. The HEV presented values between 24.7% (PC300) and 72.5% (PC500), while those of the GirVent ranged between 40.4 and 64.0%, respectively. The values observed in the breathe ranged between 5.4 and 15.6%.

Generally, the design stage tested of the PDVs showed satisfactory accuracy of the control systems and instruments (12). Nonetheless, individual mechanisms of the PDVs resulted in some strong outliers and in increased inaccuracies at individual test settings compared to those of the reference ventilator.

All ventilators showed a decelerating flow profile, except the breathe which showed an ascending flow profile. Hence, a direct comparison between the ventilators is not desirable. However, the results presented evaluate their accuracy regarding their main control variables, independent of their different functional principles.

The applied tests resulted in a critical differentiation of functional performance of the evaluated PDVs, such as the passive PEEP valves (HEV and breathe) showed larger relative errors and variations compared to the actively controlled PEEP by blowers (Hamilton T1 and GirVent). The PEEP generated by a passive PEEP valve is more prone to errors and potentially less stable under clinical conditions of respiratory diseases.

Each pressure-controlled ventilator displayed maximum PIP error at a PIP of 30 cm H₂O, with an RR of 20 bpm. As shown in Figure 4, the PIP error increases when inspiratory time is decreased. This effect is mainly caused by the resistance of the ETT used in the experiments. Figure 8 shows an example of the pressure profile (HEV ventilator) measured with and without an ETT in the setting PIP = 30 cm H₂O and RR = 20 bpm.

The main target regions for PDVs are environments with low or limited access to a supply of O₂. The O₂ performance with a limited flow feed-forward control is highly relevant for PDVs, and the economic use of O₂ is of great importance. However, concerning clinical implications, it is crucial that a PDV is also capable of delivering a high O₂ concentration.

According to the ISO 80601-2-12 norm and the MHRA specification, the accepted accuracy for the oxygen concentrations is ± 5% of the set value. As such, this can only be evaluated for the feedback control of the high-pressure O₂ supply tests. However, the norm and the MHRA specification do not state any accepted performance for the low-pressure O₂ efficiency.

In the low-pressure O₂ supply mode (2, 4, 6 L min⁻¹), the breathe achieved higher FiO₂ at steady state than the Hamilton T1. The resuscitator bag serves as a large reservoir, which premixes air and O₂, thereby allowing a homogeneous gas mix. However, this system has no other form of O₂ control than the supply flow set. In sharp contrast, the GirVent achieved considerably lower FiO₂ with both O₂ port modes implemented. A solenoid valve is used here to limit the O₂ injection exclusively to the inspiratory phase. This design results in a sparing use of O₂; however, high FiO₂ cannot be achieved. The HEV ventilator with the high-pressure O₂ port showed a larger error than 5%, with continuous overshoot to the set FiO₂. Thus, it requires additional tuning of the software and calibration of the O₂ sensor. By contrast, the Hamilton T1 showed high accuracy in the high-pressure O₂ tests, ranging within the 5% tolerance.

The t₉₀ value represents the response to an FiO₂ change. A limited increase in maximal FiO₂ may result in a low t₉₀ value, but it also indicates that the ventilator system does not manage to provide the patient with a high FiO₂. For instance, a very low t₉₀value was observed in the GirVent, but the maximal FiO₂ achieved is likewise very low, whereas with the breathe and the HEV, a long t₉₀ value was observed, but the achieved FiO₂ concentrations approach 1.00. In general, the O₂ operation modes of the HEV and the breathe show that premixing O₂ in a reservoir is beneficial for a homogeneous O₂ supply. In addition, the implementation of an O₂ sensor is recommendable even if FiO₂ can be regulated with the setting of the O₂ source.

The correct detection of and fast reaction to a patient's breathing efforts is crucial to prevent patient–ventilator asynchrony, which may cause discomfort or even lung injury (21, 22). A higher initial flow after triggering results in a better pressurization capacity of the ventilator and in decreased patient effort (23, 24). Overall, the PDVs tested here detected the applied P_0.1 levels and suitably triggered breathing cycles generated by the TestChest. Concerning the acceptable performance for the trigger detection and reaction, neither the ISO 80601-2-12 standard nor the MHRA specification specifies any tolerances.

Relative to results reported in commercial ventilators, our measurements showed generally higher TDT, including the Hamilton T1 (17–20). This may be explained by the additional resistance imposed by the ETT, which causes a restricted and delayed flow signal, which in turn requires more sensitive triggering. Compared to studies that used a PEEP level of 0 cmH₂O or 5 cmH₂O (17–20), the use of an increased PEEP level (10 cmH₂O) requires a less sensitive trigger to avoid auto-triggering (17–20). This directly results in a slower detection and a longer TDT.

Although the HEV and the GirVent display slightly higher PD and TDT levels and lower pressurization capacities than the reference ventilator, the results are encouraging, also considering the fact that the devices are still in an early development state. Furthermore, Delgado et al. (18) reported pressurization capacities of different commercial ventilators in a similar magnitude to those we observed in the HEV and the GirVent.

In this study, we showed that the presented test protocol enables rapid and conclusive evaluations of PDVs during their early development phase. The test protocol and the evaluations of the individual parameters were compared with the performance of the Hamilton T1 as a reference system. The simplicity of the protocol in combination with the automated detection and evaluation algorithm, fast quantification of the performance, as well as iterative and test-oriented development of PDVs, is supported. Safety features required according to the norm ISO 80601-2-12:2020 (12) and the MHRA recommendation (5) were considered but are not comprehensively incorporated in the present study. The objective of this simplified protocol was to enable data-driven development based on realistic in vitro tests. However, our test protocol does not replace the tests required for certification.

The norm ISO 80601-2-12:2020 (12) states to use a linear resistance that accounts for the resistance induced by the ETT and the pulmonary system. However, the restricting cross-sectional area of an ETT introduces a relevant parabolic resistance and flow disturbance that might not be accounted for solely with a linear resistance implemented in the mechanical lung, especially at high flow rates. The impact of the ETT is also evident in the trigger signal evaluation. Overall, the use of ETTs is of critical importance as this represents clinical reality of the gold standard therapeutic strategy, and it is beneficial to investigate representative clinical scenarios.

Other limitations of the test protocol presented include the lack of testing of the efficiency of O₂ use of each ventilator. To determine their actual efficiency, the amount of O₂ used from the source to achieve the desired FiO₂ needs to be measured. Also, the robustness of software and hardware of each ventilator ought to be assessed under high load. Last, the patient cycling that describes the transition from inspiration to expiration during assisted ventilation (25) was not measured in this study.

In comparison to in vivo testing, our test approach provides simplicity and complete experimental control. Nonetheless, it is physiologically limited, whereas in vivo tests provide more specific and reliable observation of biological effects but with strict regulations and compliance standards.

In conclusion, we presented a systematic test approach, which helps develop pandemic ventilators in a short time, independent of their ventilation principle. The test protocol covers the essential functional aspects for the initial development phase of PDVs. In addition, the automated and modular evaluation scripts allows for performing iterative investigations and drawing conclusions regarding specific ventilator characteristics.