Precision and diagnostic accuracy verification of VITROS HIV combo assay

Introduction Human immunodeficiency virus (HIV) has emerged as the number one dreadful communicable disease in the last three decades. According to 2016 World Health Organization (WHO) report on global prevalence of HIV infection, 36.7 million people are living with HIV/AIDS worldwide and 1 million deaths have been reported due to HIV-related illnesses.1 Past decades have witnessed significant advances in the field of HIV prevention, diagnosis and management. Impact of early diagnosis and initiation of anti-retroviral therapy (ART) on longterm survival and infectivity of affected individuals had been studied extensively worldwide.2 This has urged an inevitable evolution of HIV testing aimed at early diagnosis. This is expressed as time taken for a particular HIV testing method to detect HIV from the time of infection, otherwise known as ‘window period’. Various platforms are available worldwide for HIV diagnosis and one of the most commonly used methods is immunoassay. HIV immunoassay has seen the recent addition of fifth generation assay.3 The advancing generations are targeted at substantial reduction in diagnostic window period through an improved efficiency.


Introduction
Human immunodeficiency virus (HIV) has emerged as the number one dreadful communicable disease in the last three decades. According to 2016 World Health Organization (WHO) report on global prevalence of HIV infection, 36.7 million people are living with HIV/AIDS worldwide and 1 million deaths have been reported due to HIV-related illnesses. 1 Past decades have witnessed significant advances in the field of HIV prevention, diagnosis and management. Impact of early diagnosis and initiation of anti-retroviral therapy (ART) on longterm survival and infectivity of affected individuals had been studied extensively worldwide. 2 This has urged an inevitable evolution of HIV testing aimed at early diagnosis. This is expressed as time taken for a particular HIV testing method to detect HIV from the time of infection, otherwise known as 'window period'. Various platforms are available worldwide for HIV diagnosis and one of the most commonly used methods is immunoassay. HIV immunoassay has seen the recent addition of fifth generation assay. 3 The advancing generations are targeted at substantial reduction in diagnostic window period through an improved efficiency.
In spite of the technological advancements, HIV is deprived of a 'gold standard' test for disease confirmation. Hence various national and international organizations have developed algorithms for diagnosis of HIV. In India, National AIDS Control Organisation (NACO) has published revised guideline for HIV testing in the year 2015. 4 Three 'fit for purpose' algorithms are available in NACO for HIV testing (algorithm I for blood transfusion donor screen, algorithm II for surveillance, and algorithm III for HIV diagnosis in asymptomatic individuals). 4 Algorithm III of NACO is being widely used by the clinical laboratories in India for HIV diagnosis. Algorithm III aids in laboratory diagnosis of HIV through a 'diagnostic review criteria' established based on findings of a cocktail of three different methods of HIV testing. The first-line testing method should be sensitive to arrive at an early diagnosis of HIV, and the second and third method should be specific to eliminate false positive results. It is the primary responsibility of the clinical laboratories intended to use algorithm III to ensure proper selection, verification and utilization of appropriate HIV testing methods fitting into the algorithm's groove. The present study has verified the potential of the current fourth generation VITROS® HIV combo assay to be used as a first-line method for HIV diagnosis (as per algorithm III) as opposed to the existing first-line HIV testing method (Abbott Architect HIV combo assay) in routine practice. The study has also attempted the verification of VITROS® HIV combo assay based on Clinical Laboratory Standards Institute (CLSI) EP15A3 guidelines (precision verification) and EP12A2 guidelines (diagnostic accuracy verification) .5,6

Materials and methods
The verification study was conducted at the Division of Clinical Biochemistry, MIOT hospitals, Chennai for a period of three months (August to October 2017). The testing method taken up for the study was fourth generation VITROS® HIV combo assay (Ortho Clinical Diagnostics). VITROS® HIV combo assay is a chemiluminescent immunoassay aimed at detection of anti-HIV 1 and 2 antibodies and p24 antigen. Antibody detection is achieved using recombinant transmembrane envelope proteins for HIV-1 group M and O, and HIV-2. Antigen detection is accomplished using monoclonal antibodies against HIV p24. 7 The present study was aimed at verifying the following: • Precision verification based on CLSI EP15A3: user estimation of precision and estimation of bias • Diagnostic accuracy verification based on CLSI EP12A2: user protocol for evaluation of qualitative test performance a) Precision verification study CLSI EP15A3 has been established for precision verification and trueness estimation of quantitative methods. But, VITROS® HIV combo assay is a qualitative immunoassay wherein the patients' results are interpreted as positive or negative. Though VITROS® HIV combo assay is a qualitative method, the present study adopted CLSI EP15A3 guideline due to the following reasons: 1. Results from VITROS® HIV combo assay are reported as positive/negative, but it provides numeric results for patients (sample/cut off) with a diagnostic cut off 1 (patients' results if ≥ 1 s/co are interpreted as positive). According to CLSI EP12A2, a qualitative test that produces numeric values can be verified, evaluated and monitored as a quantitative method. 6 Hence, the present study attempted to evaluate the precision around the diagnostic cut off (1 s/co) of our testing method by using EP15A3. 2. As per CLSI EP15A3, any testing method validated by the manufacturer adopting EP5A2 guideline, evaluation of precision performance of quantitative measurement methods shall be verified using EP15A3. 5 Since VITROS® HIV combo assay had been validated by manufacturer for precision using EP5A2. 7 The study used EP15A3 for precision verification of the same.
Testing materials used for precision verification study included two patient sample pools obtained after ethical committee approval. The pools included one positive pool (with value near diagnostic cut off: 1 s/co) and one negative pool. A five-run precision experiment was carried out as per EP15A3, with each run comprising of 5 replicates (5x5). After eliminating the outliers, two measures of imprecision were calculated based on statistical procedures adopted from EP15A3. These included 'within run imprecision', otherwise known as repeatability (% CVR), and 'within lab imprecision' (% CVWL). These imprecision estimates were compared against the manufacturer's claims for the same.

b) Diagnostic accuracy verification
The study attempted to measure the diagnostic accuracy rather than the measurement accuracy (comparing a testing method against the diagnostic accuracy criteria). This criterion was established as per NACO algorithm III, based on the findings of three methods of HIV testing for making a final positive/negative classification of infection. The three methodological platforms adopted in our laboratory for HIV diagnosis, as per NACO algorithm III, are as follows (Fig. 1). 1. Abbott architect HIV combo assay (A1): highly ensitive assay 2. VIDAS HIV DUO ultra assay (A2): highly specific assay differentiating antigen and antibody 9 3. Bio-Rad Geenius HIV ½ confirmatory assay (A3): highly specific assay differentiating HIV 1 and 2 infection 10 VITROS® HIV combo assay is a fourth-generation immunoassay to detect p24 antigen and anti-HIV antibodies (HIV 1 and 2). This has improved sensitivity (100%) with a shortened window period of detection. 3 According to NACO algorithm III, wherein three different methods are used for diagnosis of HIV infection, the first-line method of testing is a highly sensitive screening method. The current study has attempted to make a three-way comparison (based on CLSI EP 12A2) between the candidate method (VITROS® HIV combo assay) and existing first-line 'highly sensitive' immunoassay in the laboratory (Abbott Architect HIV combo assay) against the diagnostic accuracy criteria established based on NACO algorithm III. 8 The study considered 4994 samples obtained from hospital-based adult (20 to 60 years) patient population (3400 males and 1594 females). Among these samples, 4913 were confirmed negative and 52 were confirmed newly diagnosed positives (Table 1). Diagnostic accuracy was expressed in terms of sensitivity and specificity. These were estimated for both candidate and existing methods. The paired sensitivities and specificities of these two methods were compared and the differences, if statistically significant, were determined by calculating scored confidence limits based on statistical procedures adopted from EP 12A2. 5

A. Verification of precision
The study followed a stepwise approach to verify precision. The experiment was carried with two patient sample pools, which included a positive pool (with value near diagnostic cut off: 1 s/co) and negative pools. The findings of positive patient samples' pool have been presented in table 2.
Step 1: Compilation of data Results of twenty-five replicates are presented in table 2. Visual inspection of the data showed that there were no gross outliers. Significant statistical outliers were ruled out by Grubbs' test.
Step 2: Grubbs' test for outliers According to Grubbs' test, a result qualifies as an outlier, if that value lies more than the G SDs from the sample mean (Grubbs' limits) where G is the Grubbs' factor, and SD is the standard deviation of the raw data including the suspected outliers. Grubbs' factor G was calculated using Grubbs' table (Table 3). G was calculated as 3.135. This was followed by calculating Grubbs' limits.
Step 3: Imprecision estimate by one-way analysis of variance (ANOVA) 3.1 One-way ANOVA was used to find out the imprecision estimates of HIV within run and between run variability.5 A one-way ANOVA format was prepared using automated ANOVA calculation software (Table 4). From the ANOVA table, DF (degrees of freedom), DF total (total degrees of freedom), MS (mean squares), SS (sum of squares), SS total (total sum of squares) were calculated.

Calculation of variance:
The two components of variance were calculated from the ANOVA table (Table 3).

Calculation of imprecision in terms of standard deviation (SD)
The square root of variance yielded the standard deviation. Step: 4 Comparison of imprecision estimates with manufacturer's claims The imprecision estimates obtained were compared against the manufacturer's claims. The repeatability %CV (2.35%) was lesser than the manufacturer's claim, σ R (3.0%). Within lab %CV (3.77%) was lesser than the manufacturer's claim, σ WL (4.8%) ( Table 5).

B. Verification of diagnostic accuracy
The diagnostic accuracy was verified by the three-way comparison of VITROS® HIV combo assay (new method), Abbott architect HIV combo assay (old method) and the diagnostic accuracy criteria established according to NACO algorithm III. As per the guidelines, 22 positive cases reported by old (existing) method, turned out to be negative both in candidate method and confirmatory assay.
The results were released as indeterminate and later confirmed as true negative by repeat testing after 2 weeks by confirmatory assay (with an ability to differentiate HIV antigen and antibody). The following steps were covered in arriving at the diagnostic accuracy: Step 1: Computation of a three-way comparison ( Table  6)    Step 3: Finding out the statistically significant difference between the paired sensitivities and specificities of new and old method As per CLSI EP12A2, scored 95% CI was recommended and statistical procedures provided in the respective guidelines were adopted.
Interpretation: The difference between paired sensitivities value is zero (95% CI: -4.93 to +4.93). Hence the difference is not statistically significant.
Step Step 5: The 95% confidence interval (CI) for the difference between paired specificities Interpretation: The difference between paired specificities value is 0.3 (95% CI:0.43 to +0.57). Hence the difference is not statistically significant.

Discussion
Method verification has become the 'quality mantra of modern era' of clinical laboratories around the world. There is growing number of laboratories around the globe moving towards the path of accreditation to International Standards Organization (ISO) 15189:2012: 'International standard for Medical laboratories-quality and competence'. According to the clause 5.5 in ISO 15189:2012, ' it is the responsibility of end-user laboratories to perform independent verification of the methods that are already validated by the manufacturer before intending to use these methods for clinical purpose. But ISO 15189 is only a 'what to do' document, which focuses on minimum requirements of clinical laboratories for establishing a quality management system in compliance with the international standard. Hence, it is the purview of the individual laboratories to design their quality system procedures to 'how to do' to meet the requirements of ISO 15189:2012.
With respect to method verification of the analytical procedures in a clinical laboratory, global guidelines are available in CLSI. CLSI is a non-profitable organization aimed at developing standards and guidelines for various processes in a clinical laboratory including verification of validated methods in a clinical laboratory. CLSI has published specific guidelines for verification of quantitative (CLSI EP15A3 guidelines on precision verification and estimation of bias) and qualitative (EP12A2 on evaluation of qualitative test performance) methods accordingly, wherein quantitative methods yield numeric results and qualitative methods yield results reported as: positive/ negative, reactive/non-reactive etc. 6 The current study has attempted to verify and evaluate the quality of performance of VITROS® HIV combo assay, which is a qualitative method yielding results as positive/ negative, with a diagnostic cut off of 1.0 S/co. Patient values ≥1 are reported as positive and vice versa. The study has verified the following two essential facets of quality of performance: 1.
Precision verification (around the diagnostic cut off) 2.
Diagnostic accuracy verification

1) Precision verification study
The findings of precision verification study of positive pooled patients' samples yielded imprecision estimates in the form of %CV R (2.35%) and %CV WL (3.77%). Both of them were compared against the manufacturer's claims (σ R -3% and σ WL -4.8%). The comparison showed that the actual %CV of repeatability and within lab %CV were within acceptable limits.

2) Diagnostic accuracy verification
As per CLSI EP12A2, diagnostic accuracy verification is classified into 'high level' and 'low level' comparisons.
High level comparison comprises of comparing trueness of a testing method against a higher order reference method, if available. In circumstances where a higher order methodology does not exist, the comparison is made against diagnostic accuracy criterion, which refers to a final positive/negative classification of a disease based on combination of the results from different methods suiting the purpose. The estimates of diagnostic accuracy of high level comparison are expressed as sensitivity and specificity, wherein sensitivity (true positive rate) measures the proportion of positives and specificity (true negative rate) measures the proportion of negatives that are correctly identified as such.
Low level comparison is characterized by the evaluation of a testing method against a comparative method already in use. The estimates of low-level comparison are expressed in terms of positive and negative agreement and not as sensitivity and specificity, since information on correctness of the testing method is not available against a confirmed diagnosis.
In the present study, HIV combo assay does not have a higher order reference method, but a well-defined and established diagnostic accuracy criterion for HIV is available with NACO in the form of Algorithm III. Moreover, the diagnostic accuracy criteria have been established in the laboratory based on a combo of results from Abbott Architect HIV combo assay, VIDAS DUO HIV Ultra assay, and Bio-Rad Geenius HIV 1/2 confirmatory assay (Fig. 1).
A stepwise approach was adopted for comparison of accuracy of VITROS HIV combo assay against the diagnostic accuracy criteria. The first essential step in accuracy verification consisted of estimating the sensitivity and specificity of our testing method against the final diagnosis. The corresponding sensitivity and specificity yielded were (Sens new ) 100% and (Spec new ) 99.8%. This was followed by calculation of these estimates for the comparative method (Abbott Architect HIV assay). The estimates were found to have sensitivity (Sen sold ) of 99.8% and specificity (Spec old ) of 99.5%.
The next step of investigation comprised of comparing the sensitivities and specificities of old and new methods, otherwise known as paired sensitivites and specificities. The difference (D) between sensitivities of old and new methods (Sens new -Sens old ) was 0% and the specificities of old and new methods (Spec new -Spec old ) was 0.3%. The percentage of difference was accountable to the discrepancy in reporting false positive /negative results between the two methods including 29 patient samples.
The final goal of the study was to find out whether the differences in paired sensitivities and specificities were statistically significant. For this purpose, 95% CI of the paired data was calculated according to the statistical procedures suggested by CLSI EP12A2. The 95% CI for difference between paired sensitivities was -4.93 to +4.93. Since the actual difference of paired sensitivities was zero, the sensitivity difference was deemed to be statistically insignificant.
With respect to specificity comparison, the 95% CI for difference between paired specificities was -0.43 to +0.57. Since the actual difference of paired sensitivities was 0.3%, the specificity difference was deemed to be statistically insignificant.
The present study has demonstrated that VITROS® HIV combo assay shows an acceptable imprecision consistent with the manufacturer's claims and an acceptable diagnostic inaccuracy against the criteria established based on NACO algorithm III.