Archive for March 2016
By: David Pagliaccio, Ph.D.
Findings from the collaborative Reproducibility Project coordinated by the Center for Open Science were recently published in the journal, Science. This report raised a stir among both the scientific and lay communities. The study summarized an effort to replicate 100 studies previously published in three major psychology journals. The project found that only around one third of replication studies yielded significant results where the size of the effects observed in the replications was about half that of the original reports on average. The study was quickly met with statistical and methodological critique, and in turn by a criticism of this critique. With the concerns raised by the Reproducibility Project and the intense debate in the field, various organizations and media outlets have begun to spotlight this issue, noting that psychology may be experiencing a “reproducibility crisis.”
The authors of this recent study importantly indicted systemic forces that incentivize “scientists to prioritize novelty over replication” and journals for disregarding replication as unoriginal. Typically, a scientist’s career is most easily advanced through high profile and highly cited publications in well-respected journals. This rarely includes replication attempts as they are generally hailed as not novel or not progressing the field. Further, statistically significant results are prized while null results are often chalked up to insufficient power and are not given a forum to help shape the literature. These factors lead to a large publication bias towards often underpowered but significant findings, a rampant ‘file drawer problem’ of not being able to publish non-significant results, and what is known as “p-hacking”, where authors can analyze and reanalyze a given dataset in different ways to push towards significance of a desired result.
Several initiatives have been put forth to try to alleviate this “reproducibility crisis.” For example, the American Statistical Association released a statement regarding the use of p-values in scientific research. This served both as a general clarification on the use and interpretation of p-values in null hypothesis significance testing and an impetus to potentially include other measures in our understanding of effect size. This type of reframing is helpful in assuring good statistical practice and resisting the tendency to inaccurately interpret our arbitrary statistical threshold of p<0.05 as a marker of truth, which often biases scientific findings and reporting. Additionally, the National Institutes of Health (NIH) have adopted new guidelines for grant submissions to try to enhance rigor and reproducibility in science, for example by increasing transparency and setting basic expectations for biological covariates. Yet, the main investigator-initiated research grant from the NIH, the R01, still includes novelty as a main scoring criteria and does not have any specific provisions for including replication studies.
Publication-related initiatives have begun to be designed to help incentivize replication. For example, the Association for Psychological Science has created a registered replication report where, before data collection even begins, scientists can pre-register a replication study to be published in Perspectives on Psychological Science. This saves scientists from struggling to publish a direct replication and reframes the focus of replication away from whether a prior study was ‘true’ or ‘false’ but rather focuses on cumulative effect size across studies. While this is a step forward, few have yet to make use of this opportunity. Importantly, while the rare journal, like PLOS ONE, explicitly states that it accepts replication submissions, top-tier journals have generally not joined in on allowing for registered replications or for creating specific article submission formats to allow for replications that otherwise would not be considered ‘novel.’ Other interesting avenues for addressing this issue have begun to spring up, for example, the website, www.PsychFileDrawer.org, was created as an archive of attempts to replicate psychology studies. While this does provide a way to publicize failures to replicate that may otherwise not be publishable, these reports currently do not seem to be indexed by standard databases, like PubMed or PsychNet. Thus, while more failures to replicate can be made available and could help the field, the unofficial nature of this website does not easily help or incentivize investigators in terms of publication counts, citations, or other metrics often considered for hiring, tenure, etc.
Importantly, issues of reproducibility and publication bias can have vast consequences on society and policy as well as potentially eroding public trust in science and the scientific process. While an extreme case involving falsification, the long lasting consequences of Andrew Wakefield’s erroneous, retracted, and unreplicated paper linking the MMR vaccine to autism spectrum disorders truly underscore the potential impacts of unchecked scientific findings. A much more benign example was profiled in The Atlantic, concerning findings that bilingual individuals show better cognitive performance than monolingual individuals. While many published studies confirmed this effect in various ways, several large studies found no significant difference and many negative studies went unpublished. Similarly, as detailed in Slate, doubt has recently been cast upon a long research track examining ego depletion. Failed replications of this ego depletion effects are now coming to a head. This is after, for example, this research was formed into a book exploring willpower and how individuals can use this science to flex their self-control and willpower.
While these findings have not shaped major policy, it is not a far leap to see how difficult it may be to undo the effects of publications biases towards novel, but unreplicated research findings on a variety of policies. For example, education research also suffers from replication issues. One study pointed out that replication studies represented less than 1% of published studies in the top 100 education research journals. While many of these studies did replicate the original work, most were conceptual rather than direct replications, and replication success was somewhat lower when performed by a third-party rather than including authors from the original work. While the higher successful replication rate is encouraging, this study does call strongly for an increase in the number of replication studies performed.
Despite debates over the extent of the reproducibility problem, it is clear that psychology and science more broadly would benefit from greater attempts to replicate published findings. This will involve large-scale shifts in policies ranging from journal practices to tenure decisions and governmental funding to help alleviate these issues and to support and motivate high quality science and replication of published studies. These efforts will in turn have long-term benefits on the development of policies based on research in education, social psychology, mental health, and many other domains.
By: Courtney Pinard, Ph.D.
In one of the most impressive feats of synthetic biology so far, researchers have harnessed the ability of bacteria to fight and destroy viruses, and have been able to precisely and cheaply edit genetic code using a genetic technology called clustered, regularly-interspaced short palindromic repeats (CRISPR) and CRISPR-associated endonuclease protein 9 (Cas9). CRISPR has been used to find and detect mutations related to some of the world’s most deadly diseases, such as HIV and malaria. Although CRISPR holds great promise for treating disease, it raises numerous bioethical concerns, which were sparked by the first report of deliberate editing of the DNA of human embryos by Chinese researchers. Previous blog posts have described scientific discussion surrounding the promise of CRISPR. At least three scientific research papers per day are published using this technique, and biotech companies have already begun to invest in CRISPR to modify disease-related genes. However, the use of CRISPR, or any genetic editing technology, to permanently alter the genome of human embryos is an issue of concern to a much broader range of stakeholders, including clinicians, policymakers, international governments, advocacy groups, and the public at large. As CRISPR moves us forward into the realm of the newly possible, the larger global, social and policy implications deserve thorough consideration and discussion. Policies on human genetic editing should encourage extensive international cooperation, and require clear communication between scientists and the rest of society.
There is no question that CRISPR has the potential to help cure disease, both indirectly and directly. CRISPR won the Science Breakthrough of the Year for 2015, in part, for the creation of a “gene drive” designed to reprogram mosquito genomes to eliminate malaria. Using CRISPR-Cas9 technology, investigators at the Universities of California (UC) have engineered transgenic Anopheles stephensi mosquitoes to carry an anti-malaria parasite effector gene. This genetic tool could help wipe out the malaria pathogen within a targeted mosquito population, by spreading the dominant malaria-resistant gene in 99.5% of progeny. The gene snipping precision of CRISPR can also treat certain genetic diseases directly, such as certain cancers, and sickle cell disease. CRISPR can even be used to cut HIV out of the human genome, and prevent subsequent HIV infection.
There are limitations of CRISPR, which include the possibility of off-target genetic alterations, and unintended consequences of on-target alterations. For example, the embryos used in the Chinese study described above, were non-viable, less than 50% were edited, and some embryos started to divide before the edits were complete. Within a single embryo, some cells were edited, while other cells were not. In addition, researchers found lack of specificity; the target gene was inserted into DNA at the wrong locus. Little is known about the physiology of cells and tissues that have undergone genome editing, and there is evidence that complete loss of a gene could lead to compensatory adaptation in cells over time.
Another issue of concern is that CRISPR could lead scientists down the road to eugenics. On May 14th 2015, Stanford’s Center for Law and the Biosciences and Stanford’s Phi Beta Kappa Chapter co-hosted a panel discussion on editing the human germline genome, entitled Human Germline Modification: Medicine, Science, Ethics, and Law. Panelist Marcy Darnovsky, from the Center for Genetics and Society, called human germline modification a society-altering technology because of “the potential for a genetics arms race within and between countries, and a future world in which affluent parents purchase the latest upgrades for their offspring.” Because of its potential for dual use, genetic editing was recently declared a weapon of mass destruction.
In response to ethical concerns, the co-inventor of CRISPR, Dr. Jennifer Doudna, called for a self-imposed temporary moratorium on the use of CRISPR on germline cells. Eighteen scientists, including two Nobel Prize winners, agreed on the moratorium. Policy recommendations were published in the journal Science. In addition to a moratorium, recommendations include continuing research on the strengths and weaknesses of CRISPR, educating young researchers about these, and holding international meetings with all interested stakeholders to discuss progress and reach agreements on dual use. Not all scientists support such recommendations. Physician and science policy expert Henry Miller disagrees on a moratorium, and argues that it is unfair to restrict the development of CRISPR in germline gene therapy because we would be denying families cures to monstrous genetic diseases.
So far, the ethical debate has been mostly among scientists and academics. In her article published last December in The Hill Congress Blog, Darnovsky asks: “Where are the thought leaders who focus, for example, on environmental protection, disability rights, reproductive rights and justice, racial justice, labor, or children’s welfare?” More of these voices will be heard as social and policy implications catch up with the science.
In early February, the National Academy of Sciences and National Academy of Medicine held an information-gathering meeting to determine how American public attitudes and decision making intersect with the potential for developing therapeutics using human genetic editing technologies. The Committee’s report on recommendations and public opinion is expected later this year. One future recommendation may be to require Food and Drug Administration (FDA) regulation of genetic editing technology as a part of medical device regulation. Up until recently, the FDA has been slow to approve gene therapy products. Given the fast pace of CRISPR technology development, guidelines on dual use, as determined by recommendations from the National Academies, should be published before the end of the year. So far, U.S. guidelines call for strong discouragement of any attempts at genome modification of reproductive cells for clinical application in humans, until the social, environmental, and ethical implications are broadly discussed among scientific and governmental organizations.
International guidelines on the alteration of human embryos are absolutely necessary to help regulate genetic editing worldwide. According to a News Feature in Nature, many countries, including Japan, India, and China, have no enforceable rules on germline modification. Four laboratories in China, for example, continue to use CRISPR in non-viable human embryonic modification. Societal concerns about designer babies are not new. In the early 2000s, a Council of Europe Treaty on Human Rights and Biomedicine declared human genetic modification off-limits. However, the U.K. now allows the testing of CRISPR on human embryos.
In a global sense, employing tacit science diplomacy to developments in synthetic biology may mitigate unethical use of CRISPR. Tacit science diplomacy is diplomacy that uses honesty, fairness, objectivity, reliability, skepticism, accountability, and openness as common norms of behavior to accomplish scientific goals that benefit all of humanity. The National Science Advisory Board for Biosecurity (NSABB) is a federal advisory committee that addresses issues related to biosecurity and dual use research at the request of the United States Government. Although NSABB only acts in the U.S., the committee has the capacity to use tacit science diplomacy by providing guidance on CRISPR dual use concerns to both American citizen and foreign national scientists working in the U.S.
Under tacit science diplomacy, scientific studies misusing CRISPR would be condemned in the literature, in government agencies, and in diplomatic venues. Tacit science diplomacy was used when the Indonesian government refused to give the World Health Organization (WHO) samples of the bird flu virus, which temporarily prevented vaccine development. After five years of international negotiations on this issue, a preparedness framework was established that encouraged member states to share vaccines and technologies. A similar preparedness framework could be developed for genetic editing technology.
Institutional oversight and bioethical training for the responsible use of genetic editing technology are necessary, but not sufficient on their own. Tacit science diplomacy can help scientists working in the U.S. and abroad develop shared norms. Promoting international health advocacy and science policy discussions on this topic among scientists, government agencies, industry, advocacy groups, and the public will be instrumental in preventing unintended consequences and dual use of genetic editing technology.