Aug
25
Written by:
Emily Sherman
8/25/2010 8:39 AM
If you have an autoimmune, infectious, or other disease, researchers need your help. Donate plasma and receive $400/donation for your time. Visit www.idonateplasma.com for details.
|
Preliminary Steps Toward Validating a Clinical Bioassay
A Case Study of the ELIspot Assay
Article Source: http://seracare.com/Portals/0/Papers/Preliminary%20Steps%20Towards%20Validating%20a%20Clinical%20Bioassay.pdf
The enzyme-linked immunospot (ELIspot) assay is sensitive and reproducible for measuring immune reactive T cells (1–4). For serum assays, the ELIspot assay is more sensitive than serial dilutions using an enzyme-linked immunosorbent assay (ELISA) (5). The ELIspot assay can quantitate the production — from a single cell — of cytokine in response to antigenic stimulation. Because it is so useful, it is rapidly being adapted for measuring responses to vaccination in cancer trials (6–9).
Because ELIspot is a bioassay and is used in heterologous populations, its performance can differ in each trial setting. The assay’s performance, therefore, needs to be characterized for the specific conditions of each trial. Once the assay has been developed and optimized for a specific clinical study, the precision of the assay needs to be determined. At a minimum, the intraassay, interassay, and biological variability should be assessed. Intraassay variability helps set the positive cutoff. The interassay variability helps decide if batch assaying is necessary. The biological variability is essential for determining a positive response to immunization. This case study describes the steps involved in characterizing an ELIspot assay and how that information can be used to evaluate assay results for a clinical trial.
Methods
The methods in this case study entail isolating peripheral blood mononuclear cells (PBMCs), using the ELIspot assay to enumerate human papilloma virus (HPV) specific effector cells within the PBMC population, and developing data analysis rotocols for intraassay, interassay, and biologic variability (see the “Experimental Methods and Data Analysis” sidebar for a complete description of experimental methods, material suppliers, laboratory equipment, and data analysis methods used in this case study).
Results and Interpretation
The results show that there can be a lot of variation when using a bioassay in clinical trials.
Intraassay variability. The interassay coefficient of variation (CV) describes the variation among multiple assay wells on the same plate from the same sample. Because most data are reported as the means of replicate values, the CV is an important indicator of the validity of the reported values. For the ELIspot, the CV was determined for each stimulating antigen and by each individual. As shown in Figure 1, there are differences in the CVs among the different individuals and among the antigens. The overall CVs (mean__) for media, Candida, and PHA were 32_16, 26_11, and 7_4, respectively. Because the spot-forming cell (SFC) values were lower for media (21_13 SFC) and Candida (68_42 SFC) than for PHA (435_61 SFC), it is possible that the range of CVs reflected the number of positive cells in the wells and not the stimulating antigen.
Mean CVs were, therefore, calculated by the SFC range within each antigen group. Table 1 shows that the CVs in each quantitative group are closely related, regardless of the stimulating antigen. The assay shows a high level of variation (CV_30) for SFC values below 20, which suggests that values below 20 are relatively imprecise. Precision improves as the SFC levels increase, with a plateau being reached at 200 SFC (CV_7–8). The CVs are inversely related to SFC levels — as the SFC level increases, the CVs decrease.
Interassay variability. The interassay variability is a measure of the variation among multiple assays performed on different days. This variable becomes important when serial samples from clinical trial participants are run on different days.
Figure 2 demonstrates that CVs varied by individual and by antigen, in a pattern similar to that seen for intraassay variation.
The overall interassay CV (media, 54_29; Candida, 55_30; and PHA, 17_11) was almost twofold greater than that found in the intraassay CV. This large interassay variation suggests that, when possible, serial samples from the same individual should be batch tested, which eliminates interassay variability.
Figure 1: Intraassay Variations measure differences on one microplate, that is, the differences among the different wells on one plate that all come from the same sample. The coefficient of variation (CV) describes the validity of the results, so the CV was determined for each stimulating antigen (a_media, b_Candida, and c_PHA) and for each individual (P3–P#14C). As shown, the CVs for media and Candida (boxes a and b) vary more among individuals. The mean of the variation by antigen (the last bar in each box) is again greater for media and Candida — which also had fewer spot-forming cells (SFCs) — than for PHA, so the variation might be related to the number of positive cells.
Biological variability. Biological variability
describes the day-to-day and week-to-week variation within each individual being evaluated. This is especially important in clinical trials when the purpose of the blood draw is to determine changes in a biomarker after immunization or treatment. The SFCs for these samples were similar to those found in interassay and intraassay results (media, 27_14 SFC; Candida, 109_49 SFC; and PHA, 495_83 SFC).
Figure 3 shows the CVs by individual and by antigen. As observed with the other variability indicators, there is variation among individuals and among antigens. The overall CVs for the biological samples (media, 58_26; Candida, 56_28; and PHA, 19_13) were almost identical to the CVs for interassay variability — high. This observation reinforces the suggestion that serial samples require batch testing. If results are determined using interassay variability and biological variability, that is, if they are run in real-time, subtle changes in the biomarkers would be completely overlooked.
Table 1 (left). The data from an intraassay variation test can be used to determine the positive cutoff (the level below which results are no longer considered positive) for clinical assays. By separating the mean coefficients of variation (CVs) from the intraassay tests by the number of spotforming cells (SFC), the data show that variation is significantly reduced for samples above 20 SPF — the precision of the assay increases at that point and the CVs drop below 30%.
Making Predictions from Assays
After these preliminary steps have been taken to determine the precision of an assay, the interassay, intraassay, and biological variations are evaluated. Once those determinations are made, they should be used to interpret assay results from the clinical trial using that same assay. For instance, the data from the variations in this case study can be used to interpret and predict assay results as follows.
Using intraassay variation results. The cutoff for a positive result (below which results are false positives) can be estimated from the intraassay variation, which shows that variability is significantly reduced when SFCs are greater than 20 (Table 1). That indicates that the assay’s precision increases when SFCs reach 20, and the CVs dropped below 30. Therefore, the minimum acceptable value for a positive result is 20 SFCs. However, 20 SFCs alone are insufficient for determining a positive result because there is also variation in the negative controls among individuals.
Because the mean response for media (the negative control) was 21_13 SFC, a media response could be incorrectly called positive — a false positive. So, not only does a positive from an antigen-stimulated well have to reach a minimum level of SFCs, the result must be sufficiently greater than the autologous negative control to account for intraassay variation.
In this case study, if the largest variation from the media controls is used (CV_38_25), and if a positive must be greater than 2_, a twofold increase above the media value is necessary for a positive result. This is calculated as 38% (mean CV) _ 2_25 (2_), which equals 88% variation or 1.8-fold. Because 2_ is approximately 95%, a false positive is unlikely (less than 5%) with a twofold greater level of SFCs. So for these calculations, a positive result must be greater than 20 SFCs and twofold higher than its autologous negative control.
Figure 2: Interassay Variations measure the variation among assays from different days for the same
clinical trial participant — and there are big differences when multiple assays are used, particularly
with Candida (box b). The variation between antigens (a_media, b_Candida, and c_PHA)
and for each individual (P3–P#14C) is high. In a pattern similar to the results of
intraassay testing (Figure 1), the CVs for media and Candida (boxes a and b) vary more
among individuals and by antigen (as shown by the mean, the last bar in each box) than
for PHA (box c). This large interassay variation suggests that, when possible, serial samples from
the same individual should be batch tested, which eliminates interassay variability.
Using interassay variability results. The interassay variability can help determine whether realtime testing of serial samples is practical and whether batch testing is necessary. In our study, interassay CVs were about twofold higher than intraassay CVs. Media and Candida CVs were greater than 50%. Using the 2_ calculation, SFC values of 2.2-fold or lower cannot be differentiated. Without considering biological variation, samples from two time points would have to differ by more than 2.2-fold to be considered different.
Because the range of SFCs expected from the trial antigen is similar to that observed for the Candida antigen, interassay variation for realtime testing can obscure the differences that might have occurred from immunization: Batch testing is recommended.
In an immunotherapy trial in which participants could already have a positive response to the test antigen, the biological variation is a critical factor in determining whether a positive response is from the immunization. There is, however, a caveat to these measurements: Samples from different days can mean different blood drawers and different cell processors. So calculations using biological variations include day-to-day processing variations. As day-to-day processing changes during a clinical trial (and from site to site), those changes are an important component in the overall predictive value of the measurement. The biological variation was similar to the interassay variation and relatively high.
For media and Candida, the CVs were greater than 50%. Using the 2_ calculation, two samples have to vary more than 2.2-fold to be considered different. So a test sample has to be more than 2.2-fold higher than its autologous baseline sample to be considered a positive response from immunization (assuming that an individual’s samples from different times are run in the same assay).
Again, the need for batch testing is confirmed. If samples from different times are run in different assays on different days, both interassay and biological variation would contribute to false positives. A sample after immunization has to be 4.4-fold above baseline in a real-time assay compared to 2.2-fold in a batch assay.
Seeking the Positive
The following conditions would be used to determine samples as positives from a clinical trial evaluated using the ELIspot assay.
• An individual sample must have 20 SFCs and be more than twofold greater than its autologous media (negative) control to be considered positive.
• After immunization, a sample must be more than 2.2-fold higher than its matching baseline to be considered a positive response to immunization.
• All an individual’s samples from different time points need to be run in the same assay. BPI
Experimental Methods and Data Analysis
The steps involved in characterizing an ELIspot assay are detailed
here.
Isolation of peripheral blood mononuclear cells. PBMCs were isolated from heparinized peripheral blood by centrifugation on a Ficoll–Paque gradient (Amersham Pharmacia Biotech AB, Uppsala, Sweden) and stored in a freezing medium containing 90% fetal calf serum (FCS, JRH Biosciences, Lenexa, KS), 10% dimethyl sulfoxide (DMSO, Sigma Aldrich, St. Louis, MO). All PBMC samples were aliquoted at a concentration of 5_106/mL and stored frozen overnight at 80°C before transfer to liquid nitrogen in which the samples were maintained in the vapor phase.
On the day of the assay, sufficient numbers of vials of PBMC were thawed and washed two times with PBMC wash medium (RPMI from JRH Lifescience) containing 1% penicillin–streptomycin (Life Technologies, Grand Island, NY) and 1% HEPES buffer (Life Technologies). PBMC samples were counted in trypan blue (Life Technologies). The samples were adjusted to the appropriate cell concentration in PBMC medium and used in the ELIspot assay.
ELIspot assay. A gamma-interferon (IFN-_) ELIspot assay was used to enumerate the frequency of HPV-specific effector cells within the PBMC populations. Human IFN-_ ELIspot kits were purchased from R&D Systems Inc. (Minneapolis). All steps involving plate development were performed according to the manufacturer’s instructions. The assays were performed with 4_105 PBMCs per well. All samples were tested with
• phytohemagglutinin-P (PHA) mitogen control (5 g/mL) (Sigma Aldrich, St. Louis, MO),
• Candida albicans, antigen-specific control (10 g/mL) (Greer Laboratories, Lenoir, NC), • and media only (no stimulant, the negative control) added directly to the PBMCs within each well.
ELIspot plates were blocked with complete PBMC medium (RPMI) with 10% human AB serum (C-Six Diagnostics, Germantown, WI), 10% HEPES buffer, 10% L-glutamine (Life Technologies), 10% penicillin–streptomycin, and 0.1% 2-mercaptoethanol (Life Technologies) for 10–15 minutes at room temperature (RT). Blocking medium was removed before the addition of cells to the plates. Triplicate test and control wells were set up for all assays. Cells were incubated in plates for 24 hours at 37°C, removed, and washed four times with wash buffer (prepared using manufacturer’s instructions).
Detection antibodies were added, followed by a 24-hour incubation at 4°C. The plates were washed four times, then streptavidin horseradish peroxidase (HRP) was added. Plates were incubated for two hours at RT, followed by four wash cycles, then an alkaline phosphatase chromogen (BCIP/NBT) substrate was added to all wells for spot development. Plates were incubated for one hour at RT in the dark, then rinsed with distilled water. Plates were dried at RT and then shipped to Zellnet Consulting Inc. (New York) for counting using an ELIspot reader system (Carl Zeiss Vision, Oberkochen, Germany) with KS ELIspot 4.0 software. Results are read as spot-forming cells (SFC) per 400,000 PBMCs. All assays were performed by the same two scientists using the same lots of ELIspot reagents.
Data analysis for intraassay variability. PBMC samples from the same blood draw of 19 individuals were assayed in triplicate on 1–6 different plates on 1–6 different days (median_2). The mean and standard deviation (_) were calculated for individual sets of triplicates (total n_46). Each coefficient of variation (CV) was determined by dividing each _ by each mean and converting to a percent (%CV). Mean CVs were calculated by individual (n_1–6, for each of the 19 individuals) and for the entire data set (n_46).
The relationship between the amount of intraassay variability and the amount of SFC was determined by grouping individual CVs by level of SFC (n_194). The groups were _20, 21–50, 51–100, 100–200, 200–400, and _400 SFC. Each group was compared with the adjacent group with the lesser SFC level using an unpaired T test and unequal variance (Excel 2000).
Data analysis for interassay variability was calculated using the mean of the triplicates from each separate assay performed on different days. The interassay CV by individual was determined from the mean and _ calculated from each set of individual samples (n_2–6, median_2). The overall interassay CV was calculated as the mean and _ of the CVs from each individual’s sample set (n_19).
Biological variability. PBMC samples from 15 individuals, derived from four blood draws 3–4 weeks apart were assayed in triplicate. All four triplicates from each individual were assayed on the same day, using the same plate and the same operator. The mean from each set of triplicates was used to determine the CV for the biological variability. The biological CV by individual was determined from the mean and _ calculated from each set of individual samples (n_4). The overall biological CV was calculated as the mean and _ of the CVs from each individual’s sample set (n_15).
Figure 3: Biological Variability describes the day-to-day and week-to-week variation within each individual
being evaluated, which can be used to set a level over baseline to show a positive response in
individuals for a given antigen. This is especially important in clinical trials when the purpose of the
blood draw is to determine changes in a biomarker after immunization or treatment. The variation
between antigens (a_media, b_Candida, and c_PHA) and for each individual (P3–P#14C) is
high. Because of the high levels of the CVs for media (box a) and for Candida (box b), test samples
that have been batch tested have to have a difference greater than 2.2-fold to be considered a positive
response to immunization and 4.4-fold above the baseline for assays run on different days.
Acknowledgments
The ELIspot assays were performed and quantitated by Janaki Sathiyaseelan, associate scientist, and Mark Matijevic, scientist, at Zycos, Inc., in Lexington, MA.
References
(1) Schmittel, A., Keilholz, U., and Scheibenbogen, C., “Evaluation of the Interferon-_ ELIspot Assay for Quantification of Peptide-Specific T Lymphocytes from Peripheral Blood,” J. Immunol. Methods 210, 167–174 (1997).
(2) Herr, W. et al., “The Use of Computer- Assisted Video Image Analysis for the Quantification of CD8_ T Lymphocytes Producing Tumor Necrosis Factor _ Spots in
Response to Peptide Antigens,” J. Immunol. Methods 203, 141–152 (1997).
(3) Scheibenbogen, C. et al., “Quantitation of Antigen-Reactive T Cells in Peripheral Blood by IFN_-ELIspot Assay and Chromium- Release Assay: A Four-Centre Comparative Trial,” J. Immunol. Methods 244, 82–89 (2000).
(4) Smith, J.G. et al., “Development and Validation of a Gamma Interferon ELIspot Assay for Quantitation of Cellular Immune Responses to Varicella-Zoster Virus,” Clin. Diag. Lab. Immunol. 8, 871–879 (2001).
(5) “ELIspot Assay Using Multi-Screen 96-Well Filtration Plates,” Tech Note TN044 (Millipore, Billerica, MA, accessed www.millipore.com site 11 February 2003).
(6) Asai, T., Storkus, W.J., and Whiteside, T.L., “Evaluation of the Modified ELIspot Assay for Gamma Interferon Production in Cancer Patients Receiving Antitumor Vaccines,” Clin. Diag. Lab. Immunol. 7, 145–154 (2000).
(7) Arlen, P. et al., “The Use of a Rapid ELIspot Assay to Analyze Peptide-Specific Immune Responses in Carcinoma Patients to Peptide Vs. Recombinant Poxvirus Vaccines,” Cancer Immunol. Immunother. 49, 517–529 (2000).
(8) Lewis, J.J. et al., “Evaluation of CD8+ T-Cell
Frequencies by the ELIspot Assay in Healthy Individuals and in Patients with Metastatic Melanoma Immunized with Tyrosinase Peptide,” Int. J. Cancer 87, 391–398 (2000).
(9) Klencke, B. et al., “Encapsulated Plasmid DNA Treatment for HPV-16 Associated Anal Dysplasia: A Phase I Study of ZYC101,” Clin. Cancer Res. 8, 1028–1037 (2002).