Edited by: Raymond Scott Turner, Georgetown University, United States
Reviewed by: Luke Esposito, Allen Institute for Brain Science, United States; Alberto Lleo, Hospital Sant Pau, Spain
Specialty section: This article was submitted to Neurodegeneration, a section of the journal Frontiers in Neurology
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Before implementation in clinical practice, biomarker assays need to be thoroughly analytically validated. There is currently a strong interest in implementation of the ratio of amyloid-β peptide 1-42 and 1-40 (Aβ42/Aβ40) in clinical routine. Therefore, in this study, we compared the analytical performance of six assays detecting Aβ40 in cerebrospinal fluid (CSF) in six laboratories according to a recently standard operating procedure (SOP) developed for implementation of ELISA assays for clinical routine.
Aβ40 assays of six vendors were validated in up to three centers per assay according to recently proposed international consensus validation protocols. The performance parameters included sensitivity, precision, dilutional linearity, recovery, and parallelism. Inter-laboratory variation was determined using a set of 20 CSF samples. In addition, test results were used to critically evaluate the SOPs that were used to validate the assays.
Most performance parameters of the different Aβ40 assays were similar between labs and within the predefined acceptance criteria. The only exceptions were the out-of-range results of recovery for the majority of experiments and of parallelism by three laboratories. Additionally, experiments to define the dilutional linearity and hook-effect were not executed correctly in part of the centers. The inter-laboratory variation showed acceptable low levels for all assays. Absolute concentrations measured by the assays varied by a factor up to 4.7 for the extremes.
All validated Aβ40 assays appeared to be of good technical quality and performed generally well according to predefined criteria. A novel version of the validation SOP is developed based on these findings, to further facilitate implementation of novel immunoassays in clinical practice.
Cerebrospinal fluid (CSF) biomarkers have proven to be helpful in early diagnosis of Alzheimer’s disease (AD). They reflect preclinical early events in AD by as many as 10–15 years before clinical symptoms occur (
The CSF biomarkers most prominently used in Alzheimer’s diagnostics are amyloid-β peptide 1-42 (Aβ42), total tau protein (t-tau), and tau phosphorylated at threonine 181 (p-tau), because they reflect the pathological hallmarks of AD (
The inter-laboratory variation was tested by using 20 CSF samples centrally distributed in aliquots by one center. The samples were shipped on dry ice and upon receipt stored at −80°C in all laboratories. These samples were patient samples from an outpatient clinic, mostly patients who came for dementia diagnostic screening. Participants or their legal representatives gave informed consent. The study conforms with The Code of Ethics of the World Medical Association (Declaration of Helsinki) (
Overview of Aβ40 assays tested in different laboratories.
Vendor | Kit | Cat (#) | Read-out | Standards (#) | Controls (#) | Sample dilution (factor) | Standard range (pg/mL) | Lab #1 | Lab #2 | Lab #3 | Lab #4 | Lab #5 | Lab #6 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
MSD | Aβ peptide panel I | K15199E | Electrochemiluminescence | 8 | 2× | 4–15,000 | x | x | x | ||||
(4G8) | |||||||||||||
IBL | Amyloid-β (1-40) | RE59651 | Optical density | 6 | 2 | 20× | 0–1,900 | x | x | x | |||
CSF ELISA | |||||||||||||
Invitrogen | ELISA kit | LNB0001 | Optical density | 8 | 4× | 0–500 | x | x | x | ||||
Human Aβ40 | |||||||||||||
Novex | Human Aβ40 | LHB3481 | Luminescence | 8 | 2× | 0–5,000 | x | x | |||||
Singleplex bead kit | |||||||||||||
Fujirebio | INNOTEST | 81585 | Optical density | 6 | 2 | 100× | 5–1,000 | x | x | x | |||
β-amyloid (1-40) | |||||||||||||
Euroimmun | β-amyloid | EQ 6511-9601-L | Optical density | 6 | 2 | 21× | 0–800 | x | x | x | |||
(1-40) ELISA | |||||||||||||
VUmc | In-house Aβ40 | ( |
Optical density | 8 | 2 | 25× | 0–25,000 | x |
Six laboratories participated in this study and validated seven Aβ40 assays (Table
For the determination of the LLOQs, 16 blank samples were measured in one plate for each assay. The calibration curves were calculated using a four parameter logistic curve fit for all assays, which gave the optimal fit.
Intra-assay variation (repeatability) was determined by analysis of samples (
Inter-assay variation (intermediate precision) was measured to determine the variation of analyses between different days. To quantify inter-assay variation, samples with low, medium, and high concentrations were selected from the samples used for the intra-assay variation [quality control (QC) low, QC medium and QC high]. These samples were measured in duplicate in four different plates at identical positions in the assay plates on four different days. In Lab #2, three different plates were tested on three different days for the Invitrogen enzyme-linked immunosorbent assay (ELISA). The mean %CV was calculated for all samples. A %CV <20% was defined as acceptable.
Intra-plate variation was determined to explore the influence of different positions within a plate on the measured concentrations (stability of the plate). QC low, QC medium, and QC high samples were measured in four replicates at different positions of the plate (columns 3/4 vs. columns 11/12). The mean %CV was calculated for all samples. A %CV <20% was defined as acceptable.
Three different CSF samples were used to perform the dilution linearity experiments. In Table S1 in Supplementary Material, the spiked Aβ40 calibrator concentration and the dilution factors used to serially dilute the samples per laboratory and assay are summarized. These samples were analyzed in duplicate. The dilutional linearity was calculated and expressed as follows:
A linearity between 80 and 120% was defined as acceptable.
To compare dilutional linearity for the kits from the various vendors and analyses by the different labs, we determined the length of dilution range in which dilutional linearity was within the acceptable range of 80–120%. This dilution range length was calculated using the following formula:
Despite that protocols were distributed among the participating laboratories, some of them deviated from this protocol in the execution of the experiments to assess the dilutional linearity, since these laboratories did not spike the calibrator into the CSF samples. In addition, different interpretations of the distributed protocols may have resulted in the large differences in spiked Aβ40 calibrator concentrations by the various laboratories to assess dilutional linearity (Table S1 in Supplementary Material).
Five different CSF samples, measured in duplicate, were diluted according to manufacturers’ protocols and spiked with recombinant Aβ40 calibrator at three different levels. An overview of the spike low, medium, and high concentrations for each laboratory in every assay can be found in Table S2 in Supplementary Material. For neat samples, buffer without calibrator was added to the diluted CSF. Spike recoveries were calculated according to the formula:
A recovery between 80 and 120% was defined as acceptable.
Five different CSF samples, with high endogenous protein concentrations, were serially diluted. Both reciprocal relative dilution factor and OD450 absorbance signals of the samples and calibrator were log-transformed to be able to use linear regression to calculate the slopes of the sample and calibrator curves. The slope of the linear parts of the log–log transformed calibrator and sample dilution series were compared to determine the degree of parallelism by calculating the “in range%” using the following formula:
A calculated in range% between 80 and 120% was defined as acceptable.
Bland–Altman plots were drawn to define if differences in results between assays were dependent on the concentration and to define the % deviation of each assay from the overall mean results for all clinical CSF samples. Correlation coefficients for comparison of results between vendors and between labs were performed by Spearman’s ρ. A
An overview of the mean LLOQs per assay kit and for each laboratory per assay are presented in Table
Overview of analytical performance parameters.
Sensitivity | Precision | Dilutional linearity range (pg/ml) | Parallelism (%P) | Intra-assay variation in clinical samples |
Correlation coefficient (ρ) between laboratories | |||||
---|---|---|---|---|---|---|---|---|---|---|
LLOQ (pg/mL) | Intra-assay (%CV) | Inter-assay (%CV) | Intra-plate (%CV) | |||||||
Vendor | Kit | Mean (SD) | %CV | Mean (SD) | Mean (SD) | Mean (SD) | Range (length) | Mean (SD) | Mean (SD) | Mean (SD) |
MSD | Aβ peptide panel I (4G8) | 65 (64) | 99 | 7.1 (5.1) | 13 (2.7) | 6.1 (0.8) | 7–2,825 (348) | 92 (18) | 5.8 (3.6) | 0.86 (0.09) |
IBL | Amyloid-β (1-40) CSF ELISA | 41 (32) | 79 | 3.0 (0.5) | 10 (3.8) | 3.2 (1.9) | 15–480 (32) | 100 (5.5) | 2.4 (1.3) | 0.93 (0.03) |
Invitrogen | ELISA kit human Aβ40 | 28 (12) | 40 | 7.3 (5.3) | 18 (14) | 7.0 (1.7) | 2,500–97,656 (39) | 189 (134) | 3.7 (1.1) | 0.91 (0.04) |
Novex | Human Aβ40 singleplex bead kit | 64 (36) | 56 | 14 (1.9) | 17 (8.0) | 12 (5.4) | 2,500–97,656 (39) | 95 (12) | 8.4 (1.1) | 0.82 (–) |
Fujirebio | INNOTEST β-amyloid (1-40) | 1.8 (1.7) | 99 | 5.1 (3.5) | 13 (2.8) | 4.0 (1.9) | 73 (42) | 3.6 (1.3) | 0.80 (0.11) | |
Euroimmun | β-amyloid (1-40) ELISA | 23 (1.9) | 8 | 4.1 (2.5) | 14 (6.0) | 5.2 (3.6) | 2,500–15,625 (6) | 98 (6.7) | 2.9 (1.8) | 0.89 (0.03) |
VUmc | In-house Aβ40 | 1,089 (−) | – | 1.9 (−) | 12 (−) | 1.7 (−) | 25– 125 (5) | 93 (−) | 1.1 (−) | – |
The mean intra-assay CVs in all laboratories were below the predefined value of 20% for all assays (Table
The mean inter-assay CVs for all assays, but not for all individual laboratories, were below the predefined value of 20% (Table
The mean intra-plate CVs for all assays in all laboratories were below the predefined value of 20% (Table
No correct dilutional linearity data were obtained for the Fujirebio assay, Lab #2 and Lab #5. The remaining results are shown in Table
A hook effect, i.e., suppression of signal at concentrations above the upper limit of quantification, was observed for one sample tested in the Invitrogen ELISA by Lab #2 (between dilution factor 50 and 250; data not shown). Other samples tested by this and other laboratories neither showed a hook effect for the Invitrogen ELISA nor for any other assay (data not shown).
The results of the recovery (%R) experiments for all assays of different vendors are detailed in Table S5 in Supplementary Material. A large variation in recovery results between laboratories was observed for several assays. As a result, acceptable recovery within the predefined range of 80–120% for all three laboratories was only obtained for the IBL assay. In addition, acceptable recovery was obtained by individual laboratories for the MSD (1/3 laboratories), Invitrogen (1/3 laboratories), Novex (1/2 laboratories), Euroimmun (1/3 laboratories), and Fujirebio (1/3 laboratories) assays. Due to this large inter-laboratory variation, an overview of recovery results per vendor is not included in Table
Parallelism results are displayed in Table
The intra-assay variation in Aβ40 concentrations was below 10% for all CSF samples (
Correlation of concentrations in clinical cerebrospinal fluid (CSF) samples between laboratories. The mean concentrations of
The mean Aβ40 concentrations of all CSF samples varied up from ~2,000 pg/mL for the Novex assay at the one extreme to ~9,500 pg/mL for the IBL assay at the other end. The %difference in Aβ40 concentrations per assay compared to the mean concentration of all assays per sample showed that the highest deviation was obtained by the IBL and Novex assays (mean deviation of 65 and −64%, respectively) (Bland–Altman plots in Figure
Variation in Aβ40 concentrations in clinical cerebrospinal fluid (CSF) samples between assays. Bland–Altman plot: the %difference (
The CSF Aβ42 levels reflect the extent of amyloid-β accumulation in the form of plaques in the brain. However, it has been shown in previous studies that the concentration of Aβ42 not only depends on the presence or absence of plaques but also may on the total turnover of Aβ peptides in the brain, resulting in different total Aβ concentrations in the CSF (
The assignment of the different assays to the laboratories was performed first based on the availability of the MSD and Luminex systems in the laboratories and second as random as possible, to overcome that two laboratories measured the same set of assays as a source of center bias. We have chosen to have the validation performed by up to three centers per assays, which we expect to provide sufficient insight into real-life performance of the assays and deemed to be cost-effective. We reasoned that testing all assays in a larger number of laboratories would not lead to dramatically different results for the assays than presented in this study.
The performance parameters of all assays were within the predefined ranges, except for the recovery results, for which no definite conclusions could be drawn due to large inter-laboratory variation, and the parallelism results of Invitrogen and Fujirebio assays, for each of which one out of three laboratories obtained results out of the predefined range. The variation between laboratories in the Aβ40 concentrations in the clinical samples was small in the duplicate measurements for each assay in every laboratory (%CV <10%). Furthermore, high mean correlation coefficients (ρ > 0.8) per vendor between sample concentrations were found for all assays. In terms of variation in absolute values between vendors, the least deviation was found for the MSD assay compared to the mean Aβ40 concentration of all assays per sample.
The LLOQs were slightly higher compared to the kit specifications provided by most vendors, but well below the ranges found in clinical samples. The IBL and Euroimmun assays are the exceptions, because the LLOQ was lower in our results compared to the kit specifications (data not shown). It should be noted, however, that the LLOQs are calculated differently by the vendors (mean + 2 to 3*SD), while we did use a more strict method of calculation (mean + 10*SD).
Although the mean precision data per assay were well below the predefined CV of 20% for all assays, if we would consider a more strict maximum of a mean CV of all laboratories below 15%, the results of Invitrogen and Novex assays are out of range.
The dilutional linearity description in the SOP gave room for many different interpretation options and should be revised. Some laboratories pre-diluted the CSF samples and, therefore, did not test the lower dilutions in which a hook effect still could be present. In the Invitrogen assay, a high variation was found, probably because Lab #6 tested lower steps of dilution (a factor 2.5) than the other two labs did (factor 4 and 5). Also, the amount of spiked antigen differed per laboratory; some laboratories did not spike antigen and, therefore, actually performed a parallelism experiment instead of the dilutional linearity experiment and had to be excluded from analysis for this parameter.
The recovery was out-of-range for the majority of the experiments. This could be due to the large variability between the laboratories in terms of preparation of the samples for the recovery experiments, e.g., the variation in concentrations chosen for spiking. In some laboratories, too high concentrations of Aβ40 were spiked, which resulted in an overflow or too low concentrations were spiked, resulting in very low precision of the result and thus low recovery. We have adapted the wording in the SOP to increase the compliance in execution (Supplemental Data for the improved version), e.g., by expanding on the choice of spikes and adding notes to standardize the spike concentrations in multicenter studies. Moreover, we added a proposal to include analysis of the actual spiked concentrations in reagent diluent for calculation instead of using a theoretical spiked value, in order to track possible dilution errors.
The parallelism results for the majority of the assays were within the predefined ranges, except the results of the Invitrogen and Fujirebio assays, because one laboratory was out of range. No explanation could be given for these two deviating results. However, with regard to this occurrence, to get more insight, it may be helpful to include more laboratories to test each validation parameter in future validation studies.
The IBL, Invitrogen, and Euroimmun Aβ40 assays seem to perform slightly better in terms of inter-laboratory comparisons, as shown by consistent high correlation coefficients for the clinical samples between laboratories. The mean Aβ40 concentrations of all CSF samples varied between the vendors. This is a hurdle in implementation of the ratio of Aβ42/Aβ40 in clinical practice to improve diagnostic accuracy (
In conclusion, all Aβ40 assays perform generally well and are promising for implementing in clinical practice. We showed some deviating results in specific parameters, but these are more likely the result of inter-laboratory variation and the lack of reference materials. Furthermore, of note is that the absolute Aβ40 concentrations in clinical CSF varied between assays. Therefore, the assays could be improved by the availability of certified reference materials to calibrate kits as well as usage of automated platforms. The upcoming availability of Aβ40 assays in a European Conformity (CE)-certified format is an important first step for their routine implementation in Europe.
Participants or their legal representatives gave informed consent. The study conforms with The Code of Ethics of the World Medical Association (Declaration of Helsinki) (World_Medical_Association 1964). The biobank containing the coded patient samples was approved by the Local Ethical committee of the VU University Medical Center Amsterdam.
LWD: performed data analysis and drafted the manuscript. LK: supervision of data collection and study design, critical review of the manuscript. MK-S: study design and coordination, data collection, and analysis. HK, SE, AP-L, SL, MV, and EV: supervision of data collection and study design, critical review of the manuscript. AV: data collection, critical review of the manuscript. HS, HT, and AF: data collection and analysis, critical review of the manuscript. EV: supervision of data collection and study design, critical review of the manuscript. CT: study design and coordination, data analysis, critical review of the manuscript.
SE was/is consultant for and/or received research funding from Janssen Pharmaceutica, ADx NeuroSciences, Innogenetics/Fujirebio Europe, Lundbeck, Pfizer, Novartis, UCB, Roche diagnostics, Nutricia/Danone, and Eli Lilly. MV serves on an advisory board for Roche Diagnostics. CT served on the advisory board of Fujirebio and Roche, received research consumables from Euroimmun, IBL, Fujirebio, Invitrogen, and Meso Scale Discovery, and performed contract research for IBL, Shire, Boehringer, Roche, and Probiodrug. All other authors declare no conflict of interest.
We would like to thank Kees van Uffelen from VUmc for his technical assistance. Additionally, all used kits were provided for free by the vendors involved in this study. This work was supported by the BIOMARKAPD research project within the EU JPND. The project is supported through national funding organizations under the aegis of JPND (
The Supplementary Material for this article can be found online at
Aβ, amyloid-β peptide; CSF, cerebrospinal fluid; SOP, standard operating procedure; AD, Alzheimer’s disease; t-tau, total tau protein; p-tau, tau phosphorylated at threonine 181; MCI, mild cognitive impairment; PET, positron emission tomography; LLOQs, lower limits of quantification; 4PL, four parameter logistic; %CV, coefficient of variation; QC, quality control; ELISA, enzyme-linked immunosorbent assay; CE, European Conformity.
1
2
3
4
5