- Steven Goodman — Associate Dean of Clinical and Translational Medicine, Stanford University
- Cari Tuna — Co-Founder, Good Ventures
- Holden Karnofsky — Co-Executive Director, GiveWell
- Stephanie Wykstra — Research Analyst, GiveWell
Note: This set of notes gives an overview of the major points made by Steven Goodman.
I know Kay Dickersin at the U.S. Cochrane Center very well. I was Associate Director of the U.S. Cochrane Center in its early stages in the 1990s. I was also her colleague for many years at Johns Hopkins before I left recently.
It's difficult for any scientist in the field of medicine to get funding for a project that crosses disease categories — i.e., doesn't pertain to a particular organ, disease, etc.
Cochrane is, for the most part, underfunded, and most of its reviewers do their work for free. The core infrastructure has secure funding in a few countries, but not in the U.S. In the 1990s, there were four centers in the U.S. and they had great people, but now most have gone away due to funding issues.
Cochrane's reliance on unpaid professionals produces some challenges for it. Which reviews they do, for the most part, depends on what those unpaid reviewers are interested in. This also makes it challenging to do the oversight necessary to assure the quality of the reviews. When I was involved, a big issue was how we could get reviews done in high-priority areas. Without much money, it was not possible to commission reviews on specific topics. We just had to hope that someone wanted to do reviews in those areas. I don’t know what the status of that issue is within the collaboration today.
On the role of country centers: if someone hears about Cochrane, and thinks they’d like to be involved, the centers are their point of contact. The centers mount a lot of educational activities and courses. Also, the centers are the main contact points for the review groups in a particular country. Centers can convene meetings for leaders of groups, help pool reviewers, etc. But they’re not well funded to do that, so their ability to bring people together is limited. They could arguably perform a much better role and could foster further development of the review groups, but that requires resources.
Cochrane’s general infrastructure is not well-supported. By infrastructure, I mean things as simple as having a paid staff person to answer the phone.
Other groups doing systematic reviews
Cochrane is an international collaboration of researchers devoted to producing reliable medical evidence; AHRQ is a U.S. government research funding agency that supports, among many other things, the conduct of reviews. AHRQ is most comfortable commissioning work in areas they or various US governmental or professional partners identify as priorities. They tend to support U.S. groups rather than working with international groups. AHRQ produces its own handbook and reviews rather than relying on Cochrane’s.
There have been some blockbuster reviews from Cochrane. However, its value is not just in blockbuster reviews, but in the breadth of coverage that you don’t get from any other source, and the credibility associated with the group. It’s not necessarily true that their reviews are always better than reviews commissioned by AHRQ or done by others, but Cochrane covers more topics, and is a generally reliable, comprehensive and independent source of information. In this sense it is somewhat similar to Wikipedia. This is what makes Cochrane the “go to” source for many people, though it’s not as widely known as it should be. I believe it has had more of an influence outside of the U.S. than it does here, although even people who have heard of it don’t really know how it works.
It is also important to view Cochrane through a very different lens; as a multifaceted effort to reform medical research practices and reporting. Its activities cover a far broader range than just systematic reviews; I would say that almost every researcher in the world who is interested in the truth of the medical literature has at some time either worked with or been influenced by Cochrane.
Systematic reviews are done by many groups for many purposes. Besides academics, payors such as CMS or private insurers use them extensively to help decide what treatments are effective and should be covered. Medical societies often commission them as the basis for practice guidelines. Sometimes payors turn to external groups who do reviews on a contract basis, or some have in-house groups. The National Blue Cross-Blue Shield Association has a very highly skilled internal technology assessment group that does reviews both for the participating plans and for AHRQ. For the plans, they do systematic reviews/technology assessments on medical therapies that are really at the cutting edge. I think that these reviews tend to be better than Cochrane’s, but it is because they have a highly skilled professional staff paid full-time to do these reviews, they incorporate non-randomized evidence that Cochrane reviews often ignore (important with emerging technologies), has close oversight by a prestigious medical advisory board, and they don't do nearly as many reviews as Cochrane does. (I serve as scientific advisor to that group, so I may be biased.)
The teams that do Cochrane reviews don’t always have all the expertise they need, a consequence of not being able to financially support the reviews. Also, their ability to incorporate non-randomized evidence is limited and inconsistent. Cochrane has tremendous strengths and some weaknesses, but there’s no other group that’s trying to cover the landscape in the way that it is. Cochrane produces as many reviews as it does because of the enthusiasm and commitment that it has inspired around the world. It’s a fundamentally very idealistic organization.
Other than systematic reviews done under the auspices of some organized group, there are many academics who do them - it's a major research methodology. I don't know the statistics, but I would bet that the majority of published systematic reviews are done by independent academics. Cochrane keeps track of these in an organized database of all reviews published in the medical literature.
There is at least one Cochrane center in China. It has a potentially huge impact in terms of leading the way on encouraging high-quality medical research in China.
Other Potentially Underfunded Areas
One area that has gotten a lot of attention is medical delivery reform, sometimes called quality improvement or safety science. That’s been a largely invisible field as a science. But there’s no question that avoidable medical errors kill a huge number of people in this country and around the world. If you have ever spent time in a hospital as patient, or with a friend or relative, you probably have witnessed lapses or frank errors. Reducing them has attracted some funding, but not enough. I think it has enormous potential to save lives relatively quickly.
Almost all of my work relates in some way to assessing or improving the quality of evidence in the medical literature. Many studies have shown that what is represented in the literature is a poor representation of what was really done, and too much is just done wrong. On a meta-level, medical knowledge is based on much shakier ground than most people suspect. Trial registration has been a key tool to establish some sort of base to make sure we know what’s not published in the trial arena, but the problems go well beyond publication bias. Even though everyone says they are interested in the quality of research (as assessed through meta-research), it is an area that’s really hard to get funded both because it cuts across disease areas, and because funders almost always want to fund research directly, not critical examination of or reform of research practices themselves.
Another underfunded area is drug safety. A huge number of people are harmed by drugs daily, sometimes because they’re taking them the wrong way, or because of the combinations of drugs that people take. Our ability to monitor drug related adverse events is still pretty crude compared to the magnitude of the problem. Congress isn’t very interested in this area except when crises occur, because they see too much concern with drug safety as an enemy of innovation.