A conversation with the Stanford Center on Philanthropy and Civil Society on January 8, 2013


Note: This set of notes gives an overview of the major points made by Walter Powell, Rob Reich, Paul Brest and Ira Katznelson.

General discussion of meta-research

Areas of meta-research

Meta-research can be broken down into at least three areas:

  1. Improving the quality and integrity of research
  2. Focusing research on the most important problems
  3. Stimulating demand for high-quality research on the most important problems

Meta-research efforts coming from outside vs. inside

Scholars often react negatively when outside institutions such as foundations attempt to change their research practices. They worry about being diverted from the science that they think they should be doing. A challenge for a funder in meta-research is that of offering resources to scientists to incentivize them, while cultivating a relationship of trust, and respecting their autonomy as much as possible.

Researchers are more receptive to initiatives that come from within their communities. For example, they would be more receptive to peers surveying them about how research practices in their field could be improved than they would be to surveys from an outside funder.

A way that a funder could have a positive impact on the output of a research community by funding meta-research while being sensitive to the community’s culture would be to fund institutions or scientists within the community to implement meta-research efforts, using their discretion based on their familiarity with the community.

Some problems in academic research

Disconnect between theoretical research and applications

About 80% of papers in the American Political Science Review (APSR) do institutional analysis without addressing governance issues. This research isn’t being applied, and more could be used to inform governance issues to address problems such as poverty.  More broadly, there is a need for closer attention to questions about impediments to successful governance, in the sense of addressing and solving large and vexing challenges.

It would be good if it were standard practice for papers to include a section about possible applications of the research discussed in the paper, and the applications’ implementation. Journals in the organizations research community routinely do this, particularly Administrative Science Quarterly.

Inferring causation from large correlations

John Levi Martin’s recent book, The Explanation of Social Action, argues that much social science research is flawed because it erroneously assumes that what he calls third-party explanations, such as the effects of patriarchy or capitalism, causes people’s actions. Social science in general is deeply suspicious of the explanations that individuals give for their behavior.

Limited data reporting from unsuccessful medical clinical trials

When a clinical trial in medicine isn’t successful, the data often isn’t reported. This is a problem in the social sciences as well. If the data were reported then it would be possible for other researchers to learn from this data. That could save a lot of money, and, more importantly, better direct subsequent investments in research.

Meta-research opportunities

Large scale & small scale

It would be a good idea for Good Ventures and/or GiveWell to fund some high level mapping of research fields while simultaneously funding some small scale pilot programs to improve research, so as to place the results of the pilot programs into a larger context and understand how the big picture features of research communities manifest themselves in specific examples.

Research mapping

The Templeton Foundation is funding a project by James Evans and colleagues at the University of Chicago to map the research landscape in chemistry, with a view toward determining its weak points, and is planning to expand the project to other fields.

Previously, there was another project at Stanford, by Dan McFarland, Chris Manning, Dan Jurafsky, and Walter Powell, on the sources of novelty in scientific research. They scraped the web for professors’ papers, patents and presentations at conferences, as well as their citations, as part of a mapping of the impact of various professors’ work. Professors were asked whether the assessments of the lists of their work were accurate, and 60% of the researchers responded, in some cases making corrections, and in some cases taking issue with the study. In particular, a number of humanities professors said that the methodology didn’t capture the emotional impact of their work on readers.

Surveying academics concerning the gaps in their fields

A survey of academics concerning how their fields of research could be improved should give variable weight to the respondents’ answers. Journal editors should get more weight than the average professor, on the basis that they probably know more about the breadth of research in their field than the average professor does.

The survey should include known and public skeptics of the research area in question, because they’ll be able to offer different perspectives than those of the others who are surveyed.

Meta-research for development economics vs. medical research and domestic social science research      

It’s often hard to know whether the results of a given study generalize. It seems harder to determine whether studies in development economics generalize (across countries) than it is to determine whether studies in medicine, or in social science pertaining to domestic affairs, generalize within their respective fields. So meta-research for medicine or domestic social science may be more leveraged than meta-research in development economics.

Specific social science research

The Social Science Research Council

The Social Science Research Council (SSRC) is a 90-year-old institution designed to:

  1. Encourage social scientists to do interdisciplinary work on important social problems
  2. Improve social science methodology
  3. Build the capacity of scholars and organizations to conduct rigorous and independent social science
  4. Represent social science knowledge to wider publics, including relevant policy communities

SSRC is currently working on three new projects that complement its existing portfolio: one having to do with cities, another having to do with anxieties of democracy, and a third having to do with the impact of new technology on the dissemination of knowledge.

Measure of America

SSRC is the home of Measure of America, a venture that aims to provide easy to use and methodologically sound tools for understanding wellbeing and opportunity in America. Its website has a map with data on the lifespan, educational level distribution and median income for each county in America, as well as a Human Development Index ranking of many locations across America. The project contains interactive tools that invite understanding of configurations of relationships and causal patterns.

Poor Communities

One feature of the SSRC’s urban program is an examination of the social geography of concentrated poverty.  Historically, low-income communities have been surrounded by other low-income communities. Recently, low-income and high-income communities have been more interspersed, due to gentrification in cities and migration of low-income people to suburbs. A potentially important area of research is that of whether people in low-income communities neighbored by high-income communities have better or worse chances of overcoming institutional isolation than do people in low-income communities neighbored by low-income communities. One could try to recruit the best researchers in concentrated poverty, intergenerational mobility and urban sociology to work on this problem together.

How Religion Divides and Unites Us

Robert D. Putnam has studied the effect of religion on social connectedness and has found that increased religiosity creates deeper connections both within and across religious communities, but creates barriers to connection between religious people and agnostic/atheist people. His book is called How Religion Divides and Unites Us. It would be desirable if this research were used to inform the public discussion of religion.