Abstract

---

The Vaccine Adverse Event Reporting System (VAERS) was created in 1 990 by the Center for Disease Control and Prevention (CDC) to track adverse events following inoculations (AEFIs). Less than 1% of AEFIs are reported and accurate recording of AEFIs is compromised on many levels. One contributing error to the accurate monitoring of vaccine safety may be the CDC’s apparent use of a truncated definition of what constitutes a serious adverse event (SAE) to rate cases rather than the statutory Code of Federal Regulation (CFR) definition. The authors set out to test if this criteria error affected rates of reported SAEs for the quadrivalent human papillomavirus (qHPV) vaccine, Gardasil, in the
FDA/CDC’s 2009 VAERS-based safety study, which concluded that 6.2% of AEFI reports were “serious” but did not signal a safety concern. A panel of volunteer, licensed physicians were asked to independently rate VAERS reports from the same data pool by applying both the 2009 study definition and the more inclusive CFR definition, respectively. The independent physicians rated 1 2% of the AEFIs as “serious” using the 2009 study definition and 24.2% of the AEFIs as “serious” using the CFR one, supporting the conclusion that errant interpretation of Federal Code applied to VAERS data reduced the ability of the 2009 study authors to detect significant SAEs, thereby compromising safety surveillance. Other serious problems with VAERS are also discussed herein.

Copyright © The Author — Published Under the Creative Commons License
Share/Alike (see https://creativecommons.org/licenses/)

Introduction

---

Following the introduction of a new vaccine, the Food and Drug Administration (FDA) and Center for Disease Control and Prevention (CDC) monitor safety systems designed to track adverse events in the general population. One such system is the Vaccine Adverse Event Reporting System (VAERS), which was created in 1986 as a result of the National Childhood Vaccine Injury Act (NCVIA).[1] VAERS is a single system in the United States for the collection and analysis of reports of Adverse Events Following Immunization (AEFI).[2] The FDA and CDC jointly implement the system, and a contractor hired by the CDC operates the system to distribute and collect AEFI reports. NCVIA stipulated that health care providers who administer vaccines and vaccine manufacturers licensed in the US must report AEFIs following specific vaccinations to the FDA and CDC. Nonetheless, VAERS is still regarded as a passive system, and reporting by the general public is voluntary. Indeed, under-reporting of AEFIs is one of the well-acknowledged limitations of the VAERS database.[3] The result of a 2010 study by the Agency for Healthcare Research and Quality (AHRQ) found that “fewer than 1% of vaccine adverse events are reported.”[4] It thereby becomes even more important that adverse events that are reported are both properly evaluated and properly recorded.

The CDC considers VAERS to be an essential, front-line system for monitoring newly licensed vaccines for safety regarding frequency and severity of AEFIs in the general public. The first step in the process is differentiating serious from non-serious reports. Secondly, case reports are broken down according to specific outcomes, such as “seizures”. Each outcome is assigned a standardized code from the Medical Dictionary for Regulatory Activities (MedDRA) and the data are then entered into a computer data- base.[5] However, if a VAERS report is rated inaccurately as a non-serious event initially, further review of that case may cease. For example, in a report regarding the increased number of post-vaccination syncope reports to VAERS between January 2005 and July 2007, primarily among females aged 11–18 years for newly licensed vaccines like Gardasil, the CDC acknowledged that the findings were subject to several limitations, including the fact that the “clinical details of non-serious reports were not reviewed.”[6] This suggests that errors in rating VAERS reports initially as non-serious can affect coding and review, which in turn could skew the interpretation of the safety profile.

In addition to VAERS, the CDC has two other systems in place to monitor the safety of all licensed vaccines: the Vaccine Safety Datalink (VSD) and the Clinical Immunization Safety Assessment (CISA) Network. The VSD, a collaborative project between CDC and eight managed care organizations, examines possible associations by comparing the number of AEFIs reported by the VAERS for selected outcomes with background rates for these events from the large VSD database.[7,8,9] A methodology used by the VSD since 2007 to provide active assessment of potential vaccine-safety signals is rapid cycle analysis.[10] However, the surveillance performed using the VSD is only for selected outcomes. In particular, rapid cycle analysis studies routinely extract and aggregate counts of electronic data on vaccinations from managed care organizations patient records only for pre- specified outcomes that occur during a pre- specified post-vaccination observation window. [11,12] The preselected outcomes are identified on the basis of data from prelicensure trials, early reports from VAERS, literature on similar vaccines, known biological properties of the vaccine, or some combination of these factors. While the role of the VSD is to investigate the epidemiologic and statistical significance of potential AEFIs, such AEFIs may not be identified if VAERS has failed to rate, code or count them correctly. It is therefore critical that the entry-level data are accurate since that is the foundation for safety investigation by other systems.

The CDC established the third safety system, the CISA Network, in 2001.[13] The CISA Network conducts research on specific AEFI at the individual or clinical level to determine possible genetic and other risk factors that may predispose certain people to a higher risk for vaccine adverse reactions. The CISA Network is a collaboration including six academic centers in the US whose purpose is to conduct clinical research around a series of specific immunization safety topics which are referred to them.[14,15] However, the CISA Network, like the VSD, does not provide oversight of VAERS and it is not designed to monitor whether or not VAERS is failing to detect serious safety signals.

Since a primary function of VAERS is to identify early signals of potential safety concern, how reliable is the system at recording accurate data at the entry level? The reference for our exploration was the CDC and the FDA post- licensure safety surveillance report for the recombinant human papillomavirus vaccine (qHPV) Gardasil which was published on August 19, 2009. Gardasil was licensed by the FDA on June 8, 2006 for routine vaccination of females aged 9 to 26 to prevent infection with genital HPV types 6, 11, 16 and 18, and it is widely promoted as a vaccine to prevent cervical cancer.[16]

The 2009 study by Slade et al. [17] provided information on the number and type of AEFIs that were reported to VAERS between June 1, 2006 and December 31, 2008. During that two- and-a-half-year period following the introduction of the qHPV vaccine into the US vaccination schedule, VAERS received 12,424 AEFIs attributed to qHPV vaccination. Of these, 772 (6.2%) were determined to be serious, including 32 deaths. The authors concluded that most AEFIs did not meet the definition of serious, a result which did not signal a safety concern. They acknowledged several limitations of their study in addition to under-reporting, including inconsistency in the quality and completeness of reported data, stimulated reporting due to extensive news coverage, reporting biases, and the fact that not all reports were systematically validated. Indeed, 68% of the VAERS reports for qHPV came from the vaccine manufacturers, and an astounding 89% or 7,519 cases did not include sufficient identifying information to allow for independent medical review of individual cases.[18] The frequency of SAEs reported is thereby reduced due to the manufacturer’s incomplete reports, while the volume of these reports inflates the denominator — thereby reducing the reporting rate of the serious events that are recorded. What might the data look like if reports from the manufacturer had been informative? Even with these omissions, the authors found a disproportionate reporting of the specific outcomes of syncope and venous thromboembolic events but stated that most of the AEFI rates were not greater than the background rates compared to other vaccines. [19]

The only qHPV-related outcomes analyzed by Slade et al.[20] and later by the VSD [21] were the prespecified outcomes, limited to: seizures, syncope, stroke, anaphylaxis, appendicitis, Guillain-Barré Syndrome (GBS), and venous thromboembolic events (VTE). This list was based in part on early reports from VAERS in a narrow window of time. Namely, the post-vaccination window for GBS and VTE was 1–42 days; for appendicitis, stroke, or seizures, 0–42 days; for anaphylaxis, 0–2 days; and for syncope, 0 days. Once the prespecified upper limit on length of surveillance is reached without a signal, surveillance stops.[22,23] It is clear, then, that VAERS reports of SAEs following qHPV vaccination that were filed within two and a half years post-licensure were key to determining what selected outcomes VSD would investigate. While the VAERS system is designed to rule out if a signal is real or not, it is not designed to monitor for failures in the detection of serious safety signals. Because VAERS is foundational to other safety monitoring systems, we were motivated to explore in more detail the process of classification of the early SAE reports to VAERS related to the qHPV vaccine.

The Slade et al. 2009 report for qHPV defined a “serious adverse event” as:

one that is life threatening; results in death, permanent disability, congenital anomaly, hospitalization or prolonged hospitalization; or necessitates medical or surgical intervention to preclude one of these outcomes.[24]

According to Slade et al, this definition is based on the US 21 CFR 314.80.

The authors also referenced the definition of a “serious adverse event” from Chen et al., which stated:

For an adverse event to be categorized as serious by the FDA, it must have resulted in one of the following: (1) death; (2) permanent disability; (3) hospitalization; (4) prolongation of hospitalization; or (5) have been determined to be life-threatening. [25]

A careful read of 21 CFR 314.80, however, reveals that it defines a serious adverse event as any of the following:

1. death;
2. a life-threatening adverse drug experience;
3. hospitalization;
4. prolongation of existing hospitalization;
5. a persistent or significant disability/incapacity; or
6. a congenital anomaly/birth defect.

Important medical events that may not result in death, be life-threatening, or require hospitalization may be considered a serious adverse drug experience when, based upon appropriate medical judgment, they may jeopardize the patient or subject and may require medical or surgical intervention to prevent one of the out- comes listed in the definition.[28,29]

Both the Slade et al. and Chen, et al. reports [30,31] are missing or have truncated Criterion 5: “a persistent or significant disability/incapacity”. While there certainly is an overlap between a “permanent disability” and “a persistent or significant disability/incapacity,” these two criteria are, however, not the same. Had the Federal Regulation 21 CFR 314.80 been changed? The history of 21 CFR 314.80 revealed that the criteria defining a serious adverse event was initially established in 1986 and is consistent with the definition cited by Chen et al., including the criterion “permanent disability.”[32,33] The original definition was amended in 1998, however, when “permanent disability” was changed to “a persistent or significant disability or incapacity.”[34] The broader definition is the one delineated in the current Code.

Thus, while the truncated definition was correctly quoted in the 1994 Chen et al.[35] article, it had been changed to a broader definition in 1998. The broader definition is, then, the one which should have been applied to rate AEFI reports for Gardasil cases and should have been quoted in the 2009 Slade et al. report.[36] While the FDA website posts the correct definition for a serious adverse event [37], both the current CDC website and the VAERS reporting form have inaccurately used the truncated definition.[38,39] Regulatory authorities should accurately define criteria for SAEs so those recording AEFIs know when to rate an event as serious. Did the exclusion or truncation of the criteria for an SAE compromise the identification of potentially serious events, particularly since many HPV vaccinated females reported persistent and significant incapacity or disability, the permanency of which was un- known? Moreover, did the possible exclusion of such events affect the coding rates of specific outcomes and early calculation rates of SAEs following qHPV vaccinations, thereby under- mining possible safety warning signals?

We proposed to investigate the accuracy of VAERS reporting by re-examining VAERS data from the same data pool Slade et al.[40] utilized to generate their 2009 qHPV vaccine safety surveillance report. The lead author (LT) conducted a preliminary pilot assessment to rate 2,000 randomly selected VAERS reports according to the legal criteria. The outcome from this preliminary assessment suggested that the rate of SAEs was much higher when one applied the legal criteria for an SAE. However, since LT is not a licensed physician, the authors deter- mined that the cases should be re-evaluated by a team of medically trained and licensed doctors.

The statistical and scientific objectives of the study were as follows:

1. To estimate the percent of AEFI reports that independent physicians would rate as serious if they were to assess all qHPV vaccine cases in VAERS’ records according to the truncated Federal Regulations criteria outlined in Definition 1 (Table 1);
2. To estimate the percent of AEFI reports physicians would rate as serious using the complete Federal Regulations criterion as per Definition 2 (Table 1);
3. To assess the statistical evidence that the percent of AEFI rated as serious by VAERS and by independent physicians are different (for either or both of Definitions 1 and 2).

Materials and Methods

---

Results

---

Overall, the author designated 15.29% of qHPV- related reports to VAERS as serious under the Definition 2 criteria. Under the VAERS rating scheme, only 7.5% of these AEFIs were designated as serious. According to stratified sampling estimates, independent physicians would rate 12% of cases as serious by Definition 1, had they rated all cases, and would rate 24.2% as serious by Definition 2.

As shown in Table 3, the 95% credible set for the physicians’ rate of designating an AEFI as serious, using Definition 1, is wide and contains the interval from VAERS. When the physicians used Definition 2, however, they gave a serious designation to qHPV-vaccine-related AEFIs at a much higher rate than when using Definition 1, and the 95% credible set for this rate did not overlap with that of VAERS. Point estimates of physician rates of “serious” designation for Definitions 1 and 2 are 13.2% and 25.0%, respectively.

Table 3 indicates point estimates and 95% credible sets for the ratio of the physicians’ “serious” assessment to that of VAERS. For Definition 1, the credible set includes 1.0, indicating that a hypothesis of equality cannot be discarded. The point estimate for the ratio (1.6) is substantially higher than 1.0, so the data do not strongly support equality; more data is needed to estimate this proportion more precisely. For Definition 2, however, the credible set for the ratio (3.2) is well removed from 1.0, so the evidence against the hypothesis of equality between proportions is strong.

Table 4 shows the cross-tabulation between each physician’s assessment according to the two definitions. There is a clear direction to the off- diagonal counts; an exact binomial test of the hypothesis that the two directions have equal probability (an exact version of McNemar’s Test) yielded p = 0. Since Definition 2 is uniformly more inclusive than Definition 1, this demonstrates that physicians are logically consistent.

Discussion

---

The purpose of this study was to investigate the VAERS system, particularly at the entry level, regarding the accuracy of recording SAEs, given the discrepancy in the criteria used to evaluate a serious event. In particular, qHPV vaccine SAE reporting was investigated as a case example. The method involved estimating the rate at which physicians would designate AEFIs reported following qHPV as serious, according to Definitions 1 and 2, and to determine whether their rating was different from that of VAERS. Table 3 indicates that the discrepancy between VAERS and the physicians’ rating of AEFIs as serious was much greater for Definition 2. It is important to note that Table 3 does not represent the actual number of cases rated as serious by the physicians; rather, it shows the overall rate at which we estimated physicians would designate AEFIs as serious, were they to rate all 1,988 reports. The results presented here thus suggest that there is a significant VAERS bias in underrating the AEFIs following qHPV vaccination as non-serious when they fit the criteria for serious. In particular, our analysis shows that compared to the VAERS rating, physicians’ rating of serious cases was more than 1.5 times higher for Definition 1 and more than 3 times higher for Definition 2 (Table 3).

Examples where VAERS rated an AEFI as non-serious while physicians rated the event as serious include the following reports after qHPV vaccination, by VAERS ID: cervical cancer 350859; 352921; muscular sclerosis 353172; severe cervical dysplasia 352921; lupus 318888, 338386; blindness 370051; Bell’s palsy 314140, 329722, 289753, 338586; embolisms 339415, 276871; non-Hodgkins lymphoma 364709; throat tightness with breathing difficulty 292869, 356463; hospitalization 329701; chronic severe joint pain, numbness 353070, 318759, respectively; autoimmune muscle disease 325193; stomach paralysis 298447.[43] The reader can access these case reports for qHPV vaccination adverse events at wonder.cdc.gov. Descriptions of two cases are given below in Table 6 to show how the reports vary in length, content, and source.

In our random sample, we also found 23 cases of spontaneous miscarriage among qHPV vaccinated women, VAERS ID: 284390, 306354, 300961, 342700, 313380, 311564, 310279, 311457, 374776, 301933, 371293, 356980, 284195, 313382, 291686, 270302, 363343, 341208, 291285, 290872, 301933, 295177, 304880. Sourced from: wonder.cdc.gov,[44] three reports of infant deaths following delivery from qHPV vaccinated mothers, VAERS ID: 346965, 325593, 346965, [45] and one report of infant congenital anomaly, 348547 [46]. All of these events were designated as non-serious by VAERS. The latter case was initially recorded by VAERS as non-serious but was later changed to “serious”. Examples of two of these pregnancy- related events are shown in Table 7. What is notable is that there are reports of spontaneous abortions and congenital anomalies similar to those shown in Table 7 which VAERS rated as serious. These are shown in Table 8. The criteria appear to be inconsistently applied.

The Slade et al.[47] post-licensure safety surveillance report stated that there were 236 VAERS reports of qHPV given shortly before or during pregnancy including 10 AEFI reports for hospitalization due to miscarriage and an addi- tional 143 reports from the Merck Pregnancy Registry for qHPV “were coded as miscarriage (spontaneous abortion)”. Yet only twelve of these AEFIs were coded as “serious” and none of these fetal deaths were counted among the 32 deaths noted in the Slade 2009 report. This outcome begs the questions (a) were multiple miscarriages and stillbirths discounted when determining rates of SAEs, and (b) was the reported number of such cases affected by the fact that 97% of the pregnancy-related AEFIs were provided by the manufacturer?[48]

Given that VAERS reports are not standardized and that the submission of reports is voluntary, the quantity and quality of information provided varies widely. When reviewing the VAERS reports for this study, physicians were allowed to make comments. These comments sometimes indicated that the synopsis provided in the VAERS records was insufficient to make a firm determination, and in such cases, physicians were inclined to give a non-serious assessment. As one might expect, there was variability among the raters, as discussed earlier (Figure 2). Our results find that inter-rater variation occurs more with some strata than with others, but in our study the strata were operationally rather than formally defined, and identifying sources would require deeper textual analysis. Of course, we have no way to estimate how multiple VAERS raters might differentially interpret cases. Nevertheless, even with some variation among physicians, the independent physicians in our study gave “serious” ratings more often than VAERS when applying the criteria for either of the two definitions (Table 3).

The disparity between VAERS and the physicians in their ratings was especially significant when applying Definition 2, which includes the criterion of “a persistent or significant disability/incapacity.” This suggests that many such cases may have been inaccurately rated by VAERS. Indeed, the VAERS form [49] itself narrowed this condition to “permanent disability” on a checklist of outcomes to be reported, thereby potentially shrinking the number of serious reports.

The CDC recently revised their VAERS form, now version 2.0 as of 10/18/19. The revision changed the category labeled “Resulted in permanent disability” to now read “Disability or permanent damages.”[50] The restatement still obfuscates that these are two different conditions and misleads the reporter into thinking that a persistent or significant disability or incapacity is not reportable unless it is permanent, thereby potentially reducing the number of entries in VAERS for serious conditions.

Since AEFIs are also coded and classified for specific outcomes based on terms from the Medical Dictionary for Regulatory Activities (MedDRA), one might argue that the possible inaccuracy of serious/non-serious classification should not impact adverse event signal detection.[51] MedDRA is an internationally utilized database of terminology used for converting an adverse event report into a hierarchical, biomedical framework with standardized codes. Once adverse events have been properly coded, incidences and frequencies of adverse events can be analyzed for safety signals. However, proper coding is a challenge and there can be great uncertainty on how AEFIs should be coded, which can result in misinterpretation and misclassification. Indeed, the CDC has acknowledged “the fact that MedDRA coding terms might not accurately reflect the diagnosis”.[52]

The process of condensing and recording data from thousands of reports is vulnerable to error at multiple levels. How reliable are the data, particularly at the entry level? Scholl et al.[53] conducted a systematic review of studies on intra- and inter-coder variation and other potential problems related to interpretation and translation of adverse events into coding terms. They concluded: “There is a lack of evidence that coding of adverse events is a reliable, unbiased and reproducible process.” A study by Toneatti et al.[54] found that two blinded coders using MedDRA coded the same adverse events differently 12% of the time at preferred-term level and that 13% of the adverse events were assessed by experts to be “non-accurate.”[55] The accuracy level of the initial coding will obviously affect the accuracy of the overall analysis. In an effort to increase more objective coding, MedDRA is constantly developing additional terms that might be more exact matches to the verbatim adverse event report, and it is updated biannually. Consequently, there are now more than 72,000 Lowest Level Terms (LLT) and more than 20,000 Preferred Terms (PT) including, for example, 50 LLTs for headache.[56] As the amount of terms increases, however, events are split into subcategories. This can result in signal dilution, thereby making it harder to statistically detect adverse events, thereby potentially compromising safety.[57]

It is not necessarily true that MedDRA would detect safety signals regardless of the inaccuracy of serious/non-serious classifications, given the established problems regarding differences in the medical aptitude of coders, consistency concerns, the accuracy of terms, discrepancies among different versions of MedDRA, bias, signal dilution [58] and, as noted earlier, failure to accurately identify a report as serious may mean the symptoms in the report are never recorded in MedDRA terms.

The International Federation of Pharmaceutical Manufacturers and Associations is a Trustee of the International Conference on Harmonisation (ICH) Steering Committee and holds the intellectual property rights to MedDRA, with technical and financial oversight.[59] One could pose the question as to whether the manufacturer who creates and profits from the sale of the vaccine and the government body who licenses, promotes, and profits from the vaccine should be the same entities that oversee post- licensure safety. Schroll et.al.[60] expressed surprise that “the system that forms the basis for all regulatory safety reporting has been subject to so little publicly available research on the topic.”

Regarding specific outcomes, Souayah et al.[61] identified 69 VAERS reports of GBS associated with the qHPV vaccine Gardasil in the US between 2006 and 2009. The estimated weekly reporting rate of post-Gardasil GBS within the first six weeks was higher than that of the general population and higher than post-Menactra and post-influenza vaccinations. In particular, there was nearly a 2.5- to 10-times- greater risk of acquiring GBS within six weeks after Gardasil vaccination when compared with the general population. Additionally, Gardasil vaccination was associated with approximately 8.5 times more emergency department visits, 12.5 times more hospitalizations, 10 times more life-threatening events and 26.5 times more disability than the Menactra vaccine.

One criticism of the finding by Souayah et al.[62] which was addressed by Slade et al.[63] is that the authors took the doses distributed and divided by three (for three doses) to estimate the number of persons at risk for an adverse event following qHPV. However, it has been reported that only 60% of the girls who were vaccinated received all three doses.[64] The assumption that all vaccinees received three doses thus reduces the denominator and falsely inflates the reporting rate. On the other hand, Slade et al. also made inaccurate calculations to determine reporting rates. They used the number of qHPV vaccine doses distributed rather than the number of vaccine doses administered, and they likewise did not adjust the distributed doses to account for the fact that 60% of the vaccinees received three doses and at least some of the remaining 40% received two doses.[65] The assumption that all doses distributed were administered and that all vaccinees received only one dose inflates the denominator and falsely deflates the rate of reporting of adverse events.

It is notable that Slade et al. reported that Gardasil had three times as many AEFI reports than there were for all other vaccines combined. The authors of the 2009 report dismissed these results as being due to the so-called “Weber effect.”[66] According to that effect, increased publicity generally follows after the introduction of any new drug into the market and presumably causes the adverse event reporting to peak by the end of the second year following introduction, declining thereafter.[67] As shown in Figure 1, the yearly percentages of serious HPV-vaccine- related AEFIs for the period since HPV vaccine introduction (2006–2014) do not follow the expected Weber effect pattern. Namely, although showing a peak in 2009 (approximately three years following HPV vaccine licensure) and then a decline in 2010, the percentages of SAEs linked to HPV vaccination have since increased steadily, and in 2013 and 2014 reached a new peak, higher than that observed in 2009. Weber himself noted that the decline in reporting observed after the general second year peak is due to a reduction in the reporting of clinically mild or trivial reactions, while the more serious events continue to be reported from year to year in a quite constant manner.[68] We can further conclude that although the number of adverse events reported are expected to increase following introduction of a new drug, it is incorrect to assume that this increase is a simple artifact of increased reporting due to increased public awareness. Rather, the observed increase may point to an actual safety warning signal, especially in instances where after an initial peak, SAEs continue to be reported at the same or increased rate which exceeds the baseline reported for products of a similar class.

Further analysis of VAERS data showed that AEFIs reported in relation to HPV vaccine administration not only continued to increase over time but they also exceed greatly those reported for other vaccines. Namely, 59.04– 76.97% of all serious and 41.63–64.32% of total AEFIs reported to VAERS yearly since 2007 to 2014 in females younger than 30 years were HPV-vaccine-related reports (Figure 1). The corresponding percentages for 2006 year reports were much lower since the HPV vaccine had only been licensed by the FDA in June 2006.[69] Furthermore, during that same period (2007 to 2014), of all AEFIs reported for HPV vaccines, 9.25–24.96% were classified by VAERS as serious. By comparison, the yearly percentages of SAEs for all other vaccines combined ranged 7.4–11.07 (Figure 2), which is approximately 1.3–3.0 lower than the corresponding percentages of HPV-vaccine-related SAEs. It is also notable that the percentages of SAEs reported yearly for Menactra, a suitable comparator vaccine since it is routinely given to the same age group as HPV, are in line with those reported for all other vaccines, ranging from 7.2–7.34% for the 2007–2014 period (Figure 2). Note that the Menactra vaccine was also introduced into the US vaccination schedule in 2005,[70] just prior to HPV vaccines (the quadrivalent vaccine Gardasil in 2006 [71] and the bivalent vaccine Cervarix in 2009 [72]). The increase in HPV- vaccine-related AEFIs is thus unlikely merely the result of the “Weber effect,” an effect that authors of a recent elaborate drug analysis found no evidence to support.[73]

While the CDC/FDA dismissed the very high incidence of AEFI reports for qHPV vaccination in the US, the Japanese Ministry of Health, Labor and Welfare (MHLW) did not dismiss the finding that AEFIs related to HPV vaccines exceeded greatly that reported in relation to other routine vaccinations in Japan. HPV vaccinations in Japan commenced in December 2009. By April 2013, the MHLW panel reported 1,968 AEFIs. Of these, 106 were rated as serious and related to cases of pain or body convulsions, joint pain or difficulty in walking. This figure translates into a rate of 12.8 SAEs per 1 million vaccinations, which is 6.1 times higher than that reported for the inactivated polio vaccine and 14.2 times higher than that reported for the influenza vaccine.[74,75] Ultimately, on June 14, 2013, the Japanese MHLW suspended its active recommendation for HPV vaccination due to increasing public concerns regarding SAEs.

The 12.8% rate of SAEs reported in Japan is remarkably similar to the 12.01% rate of SAEs assessed by the physicians in this study when they applied Definition 1 criteria to evaluate VAERS cases. Would the rate of serious cases have been 12% or higher if the CDC raters had been using the correct CFR definition to evaluate cases? Would the rate of serious SAEs have been 12% or higher if the 7,519 VAERS reports filed by the manufacturer had had enough information for clinical follow up? Would the CDC have found a 12% or higher post-licensure reporting rate for SAEs following Gardasil vaccination to be a signal for further investigation, as they did in Japan? Just what rate is high enough for the CDC to investigate the safety of the HPV vaccine, given that the HPV vaccine has consistently had AEFI and SAE reports that are many times greater than any other vaccine of a similar class (Figure 1 and Figure 2)?

Throughout 2014, long-term chronic illnesses with intense symptoms related to HPV vaccination continued to be reported in Japan, and Japanese researchers like Kinoshita et al.[76] investigated causes for the neurological mani- festations in young girls aged 11 to 17 years. The disorders, which included complex regional pain syndrome (CRPS), orthostatic hypotension, and postural orthostatic tachycardia syndrome, were sufficiently severe to cause significant disability. Would such disabling AEFIs have been recorded properly as serious with the US VAERS system, and would they have been investigated by the CDC? Researchers concluded that the symptoms observed for some in their study could be explained by abnormal peripheral sympathetic responses following HPV immunizations and that it was unlikely that the Japanese environment played a role in the pathogenesis of this unique autonomic disorder. Their conclusions are supported by reports of similar symptoms related to HPV vaccine administration worldwide, namely Denmark, [77,78] US [79, 80] Australia,[81] and elsewhere.[82,83] Japan’s AEFI reporting system was effective in guiding their health regulators to make informed choices regarding HPV vaccine safety and public health policy in a manner that a sensible and reliable vaccine adverse event system would allow.

Conclusion

---

For decades there has been broad awareness and passive acceptance of the fact that AEFIs for all vaccines are grossly under-reported to VAERS at less than 1%. Reasons for this on the submission side include lack of public and physician aware- ness of possible vaccine-related adverse events. For example, possible AEs are not mentioned in direct-to-consumer television advertisements for vaccines, while they are for drugs. Vaccine Information Sheets (VIS) do not include adequate information regarding possible AEs and are often not given to consumers until after the vaccine has been administered, if at all (both contra laws governing informed consent). Physicians may not be aware of the importance of reporting of adverse events or may be uncertain of their liabilities. (They have none under the NCVIA of 1986.) Consumers are often unaware that they too can report their own and loved ones’ adverse events to VAERS, and they may not be aware that they do not require their doctor’s permission or agreement. They, like physicians, may be uncertain about when, what and where to report. Other reasons include the absence of any penalty to vaccinating professionals (including pharmacists) for failure to report, incomplete forms, lack of standardization, and the inconvenience of reporting.

Factors leading to under-reporting of AEFIs on the recording side include the lack of a uniform probing questionnaire. For example, a standardized checklist of possible symptoms could have been sent to those reporting serious AEFIs for Gardasil to indicate what conditions existed before and after each subsequent inoculation. Such a checklist would be particularly relevant and helpful in assessing the safety of the qHPV vaccine, because this vaccine is given, depending on age, in two or three doses over a relatively short period of time. The subject could thus become his or her own baseline for examining pre- and post-injection AEFIs. Something like a standardized form would also eliminate the need for a coder to translate a report into MedDRA codes, which has led to problems with the inaccuracy of coding terms, coder bias or coder inexperience and signal dilution. To offer an alternative to the existing spontaneous reporting system, the AHRQ project proposed an electronic automated adverse event reporting system that would survey medical records electronically with algorithms designed to seek both expected and unexpected AEFIs. The system would interface with but be independent of the existing VAERS and VSD systems. The goal was to increase “the quality of physician detection” for AEFIs and “to substantially increase the number, completeness, validity, and timeliness of physician-approved case reports to VAERS.” However, the CDC failed to respond to “multiple requests to proceed with testing and evaluation.”[84]

Other reasons for under-reporting rates of AEFIs are the questionable practice of using the number of vaccines distributed rather than the number of vaccines used as a denominator to determine reporting rates, the failure to adjust the denominator when an individual receives a series of shots of the same vaccine, the failure to interview the families of those who die after vaccination to obtain more information, and one that is a focus of this paper and not reported elsewhere, using the wrong criteria to rate a serious event.

In particular, it appears that the outdated (by 22 years) rating criteria used by the CDC for vaccine safety surveillance to assess if a VAERS report is serious or non-serious do not necessarily conform to those stipulated by US, 21 CFR 314.80. An important medical event, namely “A persistent or significant disability/ incapacity,” is left out of the list of possible AEs to be considered serious. The results of our study show a significant 3–4X higher rate of reported serious AEFIs when cases are assessed according to the complete legal criteria for a serious AEFI as opposed to a truncated definition. Because of the limited number of physician raters who agreed to volunteer as raters for this study, it was not possible to extend our analysis to vaccines other than qHPV. However, given that the truncated criteria for evaluating SAEs appear to be the standard used by the VAERS, it is likely that serious events possibly related to vaccines other than qHPV vaccines may also be under-rated.

Our study demonstrates inter-rater variation attributed to completeness of criteria, however our study did not estimate inter-rater variability. While this could be a weakness, we did find a robust and significant difference when the full criteria were used. High inter-rater variability exists among VAERS raters themselves which may undermine the utility of VAERS as a system for tracking AEFIs and SAEs.

For decades, the CDC has failed to adequately address multiple problems with the VAERS system, including neglecting to consist- ently use the appropriate CFR criteria to rate and record serious cases. Under-reporting continues to the detriment of public health and the erosion of public confidence in vaccine safety. Given the importance of reliable reporting and recording of vaccine AEFIs, focused research efforts to improve the system are long overdue. New surveillance methods are needed. Perhaps it is time to create an independent agency with no conflict of interests — including no financial input from vaccine manufacturers, and no revenues received tied to vaccine uptake — to oversee vaccine safety surveillance.

References

---

Appendix

---

If it were possible to rate all cases by the author and by physicians in addition to VAERS, we could construct a 2 × 2 × 2 table of “Serious” or “Non-serious” ratings by VAERS, author, and physicians. Let pijk be the probability for each of the 8 cells in this table, where i, j, and k respectively indicate case designations by VAERS, the lead author, and physicians. We can represent these joint probabilities using the product of marginal and conditional probabilities:
P_ijk = P(VAERS = c_i, Author = c_j, Physicians = c_k)
= P(Physicians = c_k | VAERS = c_i, Author = c_j)
P(VAERS = c_i, Author = c_j)

We placed a Dirichlet(0.5, 0.5, 0.5, 0.5) prior over the four strata formed by the VAERS and initial assessments, and independent Beta(0.5, 0.5) priors over the probability of physicians’ “Serious” ratings for each of the strata; this is the Jeffries prior for the binomial probability.

Given counts n₁ through n₄ for the four strata, the posterior distribution over strata probabilities is Dirichlet(n₁ + 0.5, n₂ + 0.5, n₃ + 0.5, n₄ + 0.5). Similarly, given x_s “Serious” physician ratings out of n_s cases in stratum i, the posterior distribution for physician “Serious” ratings is Beta(x_s + 0.5, n_s – x_s + 0.5).

We generated 20,000 Monte Carlo realizations from the Dirichlet posterior for P(VAERS = c_i, Author = c_j) and similarly for each of the four stratum-specific Beta posteriors for P(Physicians = c_k | VAERS = c_i, Author = c_j). Multiplying, we obtained a joint posterior distribution for each cell of the 2 × 2 × 2 table. Posterior distributions for parameters of interest are obtained by summing over unneeded dimensions or by applying the appropriate function for the 20,000 realizations. For example, the rate of physicians’ “Serious” assessment is obtained by summing over the VAERS and lead author’s levels for each iteration. The ratio of physicians’ rate to VAERS rate is obtained by taking the ratio of the marginal quantities.

Acknowledgments

---

The authors thank Dr. William Reichel for his assistance in recruiting the
physician raters, and those raters for donating their valuable time to this study.