Ovarian cancer screening — a developing story that can help us, too
Today, one of the nation’s largest commercial clinical laboratories took a new ovarian cancer screening blood test off the market after the U.S. Food and Drug Administration had issued the company a warning letter on September 29th. The company revealed its decision Friday in a filing with the Securities and Exchange Commission. Some news stories are suggesting that this has raised questions about whether the FDA should be regulating medical diagnostic tests and that it’s just an attempt to squelch innovation and universities from profiting from their research — after all, everyone knows screening tests save lives.
That’s not even close to the real story.
This is a story of the FDA actually working to safeguard the public; of the peer review process actually working swiftly; of some medical professionals commendably speaking out for sound science; and how the enormous potential commercial value in the development of the first effective screening test for ovarian cancer can blemish the integrity of science.
This entire saga has many valuable take-home lessons, so let’s poke around…
Background
With no screening test yet available that can accurately detect ovarian cancer early, the race as been on for years to develop one. Several universities have been rivals in trying to be the first to develop, patent and license an ovarian cancer screening test. Companies are standing in line wanting to negotiate deals to commercialize the tests in development, said Anna Lokshin, assistant professor of medicine and pathology and a researcher at the University of Pittsburgh Medical Center, at the 2006 annual meeting of the American Association for Cancer Research. UPMC is developing its own test, which it was reporting at this meeting had achieved a specificity of 98% and a sensitivity of 93%.
Yale University had been developing one based on four protein biomarkers, and had already exclusively licensed its ovarian cancer screening test to Laboratory Corporation of America in the United States in February of 2005. On June 12, 2006, Yale University announced it had granted an exclusive license to SurExam Life Science & Technology Company for the commercialization of its ovarian cancer screening test in China. Teva Pharmaceuticals in Israel has been granted the license for the test there.
Yale’s preliminary report on its screening assay had been published in the Proceedings of the National Academy of Sciences on May 24, 2005, which described a study on 58 cancer patients and 28 healthy controls. As the Yale researchers reported, their screening assay had only achieved 95% specificity. In that paper, they noted that not only was the specificity too poor to be adopted as a screening test, another problem was that the bio markers were not specific to ovarian cancer, but also found in other cancers such as breast and uterine, which needs to be “investigated rigorously.” It means that a positive result may call for a larger range of follow-up tests/imaging, biopsies and surgeries to confirm or rule out each potential cancer diagnosis.
These degrees of accuracy might sound impressive, but not to scientists and medical professionals. Even according to the Yale authors in their introduction, because ovarian cancer occurs at such low rates among the general population (0.04% of postmenopausal women), in order for the potential benefit of early detection in a screening test to outweigh the risks from complications of additional follow-up tests and surgeries among those with false-positive results, an ovarian screening test must achieve a minimum of 99.6% specificity. Unlike a positive mammogram which can be confirmed with a biopsy or further imaging, for instance, a positive screening test for ovarian cancer can only be confirmed with surgery. Even low levels of false positives will result in large numbers of women being incorrectly told they have ovarian cancer.
Dr. Noah Kauff, director of Ovarian Cancer Screening and Prevention at Memorial Sloan-Kettering Cancer Center, explained this in more detail several years ago to the Genome Web publication, Proteo Monitor. The incidence of ovarian cancer in post-menopausal women is only about one per 2,500 women per year. So, even if an ovarian cancer screening test had 100% sensitivity, if it had 98% specificity that would mean that if 2,500 women had the test, one woman would correctly have her ovarian cancer detected, but 49 women would have false positives and undergo unnecessary surgery, with risk of associated complications. The goal up to this point, he said, is to develop a screening test for ovarian cancer that leads to one cancer found in every ten surgeries done as the result of an abnormal screen, which would require a test to have a 99.6% specificity to achieve this.
*** Please see the sidebar:“Understanding screening tests: sensitivity versus specificity, false positives and false negatives, and probabilities of having a disease.” Then come back….
Fast forward to 2008
Most Americans probably first learned of the ovarian cancer screening test, OvaSure™, this past February when Yale University issued a press release announcing that it had developed a new blood test for early stage ovarian cancer, using six biomarkers, that was 99% accurate. At that time, the news media across North America reported the story pretty much verbatim from the press release:
“The ability to recognize almost 100 percent of new tumours will have a major impact on the high death rates of this cancer,” lead author Gil Mor, associate professor in the Department of Obstetrics, Gynecology and Reproductive Sciences at Yale, was quoted as saying. “We hope this test will become the standard of care for women having routine examinations.”
In a Phase 2 clinical trial, researchers tested the blood serum of 350 patients without ovarian cancer and 150 ovarian cancer patients. The number of proteins in each participant's blood were counted. Once the results were tabulated, the Early Detection Research Network of the National Cancer Institute independently evaluated the findings… A Phase 3 trial is now underway, with 2,000 patients enrolled.
It appeared no medical writers had gone to the study they were reporting on or fully understood it. The news had, however, provided one clear piece of information that would have alerted informed consumers that the headlines and news stories were going beyond the science. Did you catch it?
It was a phase II trial. As reviewed here, all studies are not created equal. When new medical treatments, drugs, preventatives, diagnostics and screenings are being developed, they are clinically tested in phases, with each phase study investigating different things.
A Phase I clinical trial is the very first introduction of an investigational new drug or treatment in humans. These studies are done on a small group of people (20-80) to determine a drug’s metabolic and pharmacologic actions in people, the active dose, initial risks and side effects, and gather enough evidence to justify the design of a scientifically valid phase II trial.
A Phase II trial is the initial study to gain preliminary data on the possible effectiveness of a drug or treatment for a particular indication or to see if it acts on a specific disease or condition. It also looks for common short-term side effects to help evaluate a drug’s initial degree of safety. It is conducted on a slightly larger group of people (50-300).
A Phase III trial is done after a phase II suggests some effectiveness for a particular condition. A phase III trial is conducted on a larger group of people (1,000-3,000) more representative of end users, to evaluate if a drug is effective and works better than current standard treatment over a period of time, and also that it has an overall favorable benefit-risk ratio. These findings are submitted to the FDA to get approval for marketing and labeling requirements.
A Phase IV trial is a study occasionally done after a drug has been on the market to learn more about its longer-term risks, benefits, and optimal use.
So, when you hear glowing reports about a phase II trial, remember that it’s still years away from proving to be safe and effective for its intended application, and most importantly, to have shown that potential benefits appear to outweigh the risks for most people. [The National Institutes of Health also goes into detail about clinical trials at clinicaltrials.gov and the FDA/Center for Drug Evaluation and Research describes the drug development, review and approval processes in their CDER Handbook. Investigational/experimental interventions for life-and-death situations are a separate topic.]
The study* of the new ovarian screening test, published in the February 15, 2008 issue of the journal Clinical Cancer Research, had tested a larger group of biomarkers (leptin, prolactin, osteopontin, insulin-like growth factor II, macrophage inhibitory factor, and CA-125) that are associated with ovarian cancer. The authors collected the blood samples that had been drawn on 156 women newly clinically diagnosed with various stages of ovarian cancer (stage I to IV/X) at their gynecology oncology clinic between August 2002 and November 2006.
To explain it simply, they found that these six biomarkers were associated with 91.6% of the 36 cases of stage I and II cancers in their cohort, whereas 119 of the 120 stage II and IV samples had the biomarkers, a 100% level of accuracy. In other words, the test misclassified 11% of the early stage I and II samples. Of the 17 women who had come into their oncology clinic who had surgically been found to have had benign masses, their test misclassified 11% of those as having cancer and correctly identified the remaining 88% as normal.
They then compared these results among cancer patients to blood samples drawn on 118 matched healthy women who had come into their clinic for regular check-ups, and who were cancer free for 6 months after the blood samples were collected. They found the biomarker panels were significantly higher among the cancer patients compared to the healthy controls.
The authors conducted various statistical modelings using logistic regression on each of the markers singly and together, and calculated the sensitivity, specificity and total positive predictive value of their test. They concluded:
"Therefore, the statistical results for the final model are sensitivity 95.3% and specificity 99.4%. The PPV = TP/(TP+FP) for the test sample is 99.3% and the negative predictive value = TN/[FN+TN] is 99.2%. The final model provides more optimistic results compared with the test group, because it includes the training set; however, there are not major differences in terms of specificity and sensitivity for either model. Nevertheless, this final model will be validated in a multicenter validation study.
This study was clearly a Phase II trial and the screening test had not gone to Phase III trial stage yet to actually begin to be evaluated as it would be used among the public. Disconcertingly, too, neither this Phase II clinical trial or any upcoming Phase III trial has been registered at ClinicalTrials.gov.
Never the less, on June 23rd, LabCorp had announced the availability of OvaSure on the market to screen for early state ovarian cancer in high-risk women, at $220 to $240 each. Its press release and notice to shareholders cited the February study as having shown the test able to discriminate between disease-free women and ovarian cancer patients with high specificity and sensitivity.
Did you catch some of the problems in this study design as to why it cannot credibly be used to claim to be able to screen for ovarian cancer and ready for marketing as a screening test?
A test that finds certain markers to be more associated with cancer patients (people already known to have cancer) than in another group of known healthy women, does not translate to those markers being able to accurately predict the cancer among a group of seemingly healthy women being screened. Do you see the jump in logic? The study was carried out on two different and very selective populations, and was unable to show that it could accurately identify early cancer among a screening population. Nor has it shown that the screening test actually improves outcomes. These are just some of the flaws in the misuse of a preliminary Phase II trial that caught the attention of the FDA and other medical professionals.
No one doesn’t want a sound diagnostic test that can truly help women, and everyone supports researchers rigorously going forward with research to discover one. The problem comes in when commercial interests step ahead of the science. The FDA’s actions were not over some mundane issue with regulatory paperwork and the fact that the company hadn’t gotten FDA approval before commercially marketing the lab test. As Dr. Steven I. Gutman, M.D., MBA, FDA director of the Office of In Vitro Diagnostic Device Evaluation and Safety, and the Center for Devices and Radiological Health, wrote in a letter to the president of LabCorp on August 7th:
It appears that you are marketing the OvaSure™ Test with performance characteristics (specifically, 95.3% sensitivity and 99.4% specificity) that are identical to those reported in [the February study in Clinical Cancer Research]. We note that this research was carried out, and performance derived, on two populations that are strongly clinically biased for being healthy and normal, and for having already experienced ovarian cancer [selection bias]. Based on the available information, we do not believe the scientific community would consider the reported study sufficient to establish performance characteristics of a test in “high risk women who might have ovarian cancer”, i.e., in a clinical setting, as claimed in your intended use and promotional materials
Based on our review of your promotional materials and the research publication cited above, we believe you are offering a high risk test that has not received adequate clinical validation [completed phase III trials],and may harm the public health. We would like to discuss with you your offer of this test, and any validation strategies you have undertaken beyond those reported in the publication cited above.
We look forward to discussing this with you, and are committed to working with you as we strive to protect the public health without unnecessarily imposing regulatory burdens on the marketing of products of potential clinical importance.
During the post-publication peer review process, medical professionals described other problems with this study’s interpretations of its data, and with the premature commercial marketing of this screening test. Gynecologic oncologists with the Society of Gynecological Oncologists stated that they believed the test needed more rigorous testing. The editors of the journal actually issued a correction, which it posted with the article online, after finding the authors had not correctly calculated the study’s positive predictive value which were used to support its use for screening. It stated:
In the article on diagnostic markers for early detection of ovarian cancer in the February 15, 2008, issue of Clinical Cancer Research, the authors indicated that the novel blood test described has a PPV for the general population above the suggested 0.10 necessary to be used as a screening test. However, data were not provided to support this claim. The PPV of this test for the general population is 6.5%, or 0.065. Therefore, it was incorrect to suggest that this test may be used for screening the general population, and the authors do not support the use of this test for screening the general population.
After the FDA’s alert, the scientific community and media began coming forward with concerns. “You really need evidence that screening actually saves lives, or at least prolongs survival,” Dr. Robert Bast Jr., an ovarian biomarker expert at M.D. Anderson Cancer Center, cautioned on CNN on September 15th. Dr. Nicole Urban with the Fred Hutchinson Cancer Center in Seattle, WA, said that we don’t know how early various biomarkers give a signal of early ovarian cancer developing. The research is just beginning. Tracking seemingly healthy women well before their cancer is diagnosed is the only way to prove a test finds cancer early, said Dr. Michael Birrer of the National Cancer Institute. Yale University has not issued a statement.
The St. Louis Beacon published a good article for the public last month, describing the concerns of medical doctors and the cautions surrounding the potential harms of unsound ovarian cancer screening. It was written by a former editor of the St. Louis Post-Dispatch who had been diagnosed with ovarian cancer two years ago.
The continued marketing of the screening test to the public and through technical bulletins to doctors finally led the FDA to issue its Warning Letter on September 29th stating that marketing of the test is in violation of the law. According to the FDA warning, the company does not have premarket approval or an approved application for an investigational device exemption, nor did it notify the FDA of its intent to introduce the test into commercial distribution as required before marketing it. For all of the grief the FDA seems to get lately, it’s valuable to hear another side of the story once in awhile and, more importantly, to understand the scientific process in action when it works.
And hopefully, readers have garnered a few valuable take-home points for those other times when the scientific process doesn’t work so well.
© 2008 Sandy Szwarc
Grant support: Nicholas Brady, EDRN/NIH, and Laboratory Corporation of America. The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with18 U.S.C. Section1734 solely to indicate this fact.
* Study disclosure statement:
<< Home