Hannah Westlake, Fabrice David, Blandine Ribotta, Bruno Lemaitre

UPLEM, Global Health Institute, School of Life Science, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland. Email: bruno.lemaitre@epfl.ch

The data described in this website cannot be re-used in a publication without agreement of the authors. All the data will be available when published in an open access journal (scheduled for the end of 2025). Two articles, one discussing some issues in the field of Drosophila immunity and one metascience article focusing on the pattern of reproducibility in the field of Drosophila immunity have been posted in biorxiv.

This project was supported by the SNSF project grant 310030_189085 awarded to Bruno Lemaitre

Objectives of the fly immunity reproducibility projects

Confidence in published results is critical to sustain trust within a scientific community and to allow new research to build on previous findings. While recent reports have sparked intense discussion on the existence of a reproducibility crisis in life sciences, the accuracy of most scientific claims is rarely verified. Furthermore, claims known to be incorrect are either discreetly contradicted by subsequent articles, discussed informally, or simply ignored. Experts are generally aware of this, but those new to the field may waste time in basing interpretations or research plans on inaccurate claims. We therefore launched an ambitious project to comprehensively assess the reproducibility of articles published over a period of five decades (1959-2011) in the field Drosophila immunity.  Our ReproSci project, expected to be completed by the end of 2025, comprises several steps. First, we selected 400 articles that focus on the antimicrobial response of Drosophila and listed the major scientific claims of each article. We then assessed the claims of each article to determine if they were supported or refuted by subsequent publications. The claims were categorized into four broad groups: verified by subsequent studies, challenged by subsequent studies, uncontested, or mixed. Second, experimental work was performed (and is ongoing) in the laboratory to test the validity of a subset of uncontested claims using alternate methods. This community-accessible website catalogs our data and findings. We invite the scientific community with expertise on Drosophila immunity to join us in our reproducibility project by commenting on assessments of claims or contributing experiments to reproducing claims. The final goal is to provide a community-annotated dataset of articles, with assessments of their claims with a hindsight of at least ten years. Bibliometric analysis of this dataset will allow us to estimate the percentage and characteristics of articles that contain ‘irreproducible’ claims. The goal is in no way to target scientists or irreproducible claims, but to analyze patterns of irreproducibility in one field of experimental life sciences. It will also provide a valuable resource for researchers in the community, allowing faster and more focused progress by featuring quality results.

Reproducibility project methods

The purpose of the ReproSci reproducibility project is to determine whether a wide range of claims published in the field of Drosophila immunity are verifiable using independent sources. The aim is not to reproduce the experiments using the same tools as many reproducibility projects, but rather to test if the claim is robust when using another approach. A list of publications was generated using a curated search string on the publicly available PubMed database. This search string (Box1) included keywords, MeSH terms and prominent authors in the Drosophila immunity field, and generated an initial list of approximately 2000 publications. This pool was then manually curated to remove 1) reviews and opinion pieces, 2) topics unrelated to Drosophila immunity captured by the search string, and 3) immune topics outside the scope of Lemaitre lab expertise, including development, cell biology, immunity to viruses, immunity to parasitoid wasps, and immunity in other species. A few articles not captured by the search string but known and considered to be key articles in the field were also manually added to this selection. The list was then restricted to the date range of interest (1940-2010). This limitation was imposed to allow a time frame in which follow-up studies may be published that may verify, expand on or contradict the content of the publications selected for the study. These restrictions resulted in a list of approximately 400 primary articles.

The list of articles delivered by the first search string, those that were removed and added, and the final selected list can be found in Table S1.

Box 1. Search string used on PubMed to generate the initial pool of publications that was subsequently manually curated.

            Selected primary articles were annotated by a single researcher (HW) and reviewed by a second researcher (BL) to ensure consistency and accuracy across the project. An online database, ReproSci (https://reprosci.epfl.ch/) was constructed to organize the annotations and provide an interactive and searchable interface. Claims were extracted through careful reading of the selected primary articles and organised according to their importance within the paper: a single Main claim representing the title claim of the paper or overarching finding; 3-4 Major claims emphasized in the abstract or those contributing to the overarching finding; and a variable number of Minor claims representing other findings of note not emphasized in the abstract. Wording of the claims was paraphrased and structured to represent claims made directly by the authors, rather than interpretations made by the reader. Other supporting data such as the methods used in the paper and evidence used to support claims was also summarized.

            Claims were then cross-checked with evidence from previous, contemporary and subsequent publications and assigned a verification category (see Table 1). A short statement explaining the evidence and citing sources was written to justify the categorization and provide additional context for the claim; this assessment was linked to each claim and can be viewed on the ReproSci website. Supplementary documents summarizing complex or ambiguous topics were also written where necessary and can be viewed on the ReproSci website.

Table 1. Verification categories used to assess claims.

A subset of claims that lacked follow-up in the published literature were selected for direct experimental reproduction. This project is concerned with the accuracy of the conclusions themselves rather than the reliability of methods, so we employed indirect or inferential reproducibility, where experimental procedures that may be different from those originally used to reach a conclusion are used to independently verify it. In the event that appropriate new methods such as CRISPR mutant fly lines or next-generation RNAi were not available for a given gene of interest, direct reproducibility was also used. When complete, a document was generated detailing the experiments, including methods and conclusions, conducted to verify an individual claim. These short research projects were also uploaded to the ReproSci website and the assessment of the claim was adjusted from ‘Reproduction in progress’ to ‘Verified/Challenged by reproducibility project’ as appropriate. The ReproSci website will be made available to the Drosophila scientific community, who will be encouraged to comment on the annotations and verifications and contribute evidence, which may lead to reassessment of claims.

How to contribute?

Scientists with expertise on Drosophila immunity (hereafter ‘the contributors’) are welcome to join the ReproSci project (Hannah Westlake, Bruno Lemaitre and the Lemaitre lab) to contribute to this reproducibility project. After agreeing to the disclaimer, contributors need to register using an ORCID. They can then comment on any claim in the database. A comment can be a simple point of view or can be more substantial by referring to published literature or adding unpublished work testing the claim. These comments will be associated with the name of the contributors as defined upon signup with the ReproSci database. Contributors that wish to stay anonymous to other users can contact us to get their comments published under the name of ‘The Reproducibility Project’ or change your user name for 'anonymous' or a 'nickname' (follow the link with your email in the top menu).

There are different types of assessments associated with the claims:

• Unchallenged claims are claims that to our knowledge have never been validated by subsequent studies from another lab. Therefore, they are not yet part of the ‘confident’ data on Drosophila immunity although many are consistent with current knowledge. Many of these claims are also difficult to test using traditional genetic approaches, and consequently, there is no intention to test them experimentally in the Lemaitre lab as part of ReproSci. Contributors are however welcome to test them!
• Reproduction in progress claims are claims that we would like to test. We have generated research plans to address these claims. About 12 of them have already been tested in the Lemaitre lab and this unpublished data is included in our comments in the database. Contributors with relevant expertise and tools are also encouraged to test these claims, and can contact us to obtain and discuss our research plan if desired.
• Verified claims are claims that appear to be validated by other studies and are therefore part of the ‘confident’ knowledge on Drosophila immunity. However, it cannot be excluded (and it is in fact likely) that some claims labelled as verified are in fact not correct. Contributors are welcome to test them and provide conflicting evidence if found!
• Challenged claims are claims that appear to be fully or partially contradicted by other studies. However, it cannot be excluded that some claims labelled contradicted are in fact correct. Contributors are welcome to test them and provide additional evidence to support or challenge them!

In signing up to participate in the ReproSci database, I agree to be respectful to other users and researchers when commenting, and to discuss available data without making personal judgements on individuals either on the ReproSci database or other platforms such as social media. I acknowledge that the purpose of this project and my participation is to provide context for and make better use of existing data.

!! We welcome your help in this collaborative project     !!