Transparency Leaderboard(Pilot Project)
Audited transparency of authors' recently published articles (2019-2021).
Audit Process • Transparency Requirements • Motivations • FAQ • As of April 30, 2021
Audit Process • Transparency Requirements • Motivations • FAQ • As of April 30, 2021
- Announcement of our transparency (T) audit initiative (March 16) caused a bit of contention.
- We've carefully listened to the diverse feedback & have now improved our initiative in several ways (see below).
- Several additional changes are under consideration, which we're actively seeking feedback on (see below).
- 2 response videos: (1) addressing issues/misconceptions & (2) on our deeper motivations for conducting T audits.
- Additional ways to give feedback &/or join the conversation:
- Forum discussion about our T audits
- Upcoming open debate & live Q&A on T metrics (June 15, 2021; LeBel opening statement; full debate video)
- Tweet at us or send us an email (also, additional live Q&A being planned, stay tuned!)
New Updates (as of April 21, 2021)
| Researcher | # of Audited Articles |
%
OpenAccess Gold, Diamond, or Green OA available |
%
OpenData Minimal dataset (for all studies) publicly available when ethically possible (Link in article, [co-]author website publication list, or on Curate Scholar author page.)
|
%
COIDisclosures All financial/non-financial COIs disclosed (or grandfathered) All funding sources disclosed (Information disclosed in article, [co-]author website publication list, or on Curate Scholar author page.)
|
%
 standard Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
All COIs/funding sources disclosed
|
|
|---|---|---|---|---|---|---|
| 1 |
Eindhoven University of Technology | 100% | 100% | 100% | 100%
100% of author's last 12 articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Apr 30, 2021
All COIs/funding sources disclosed
| |
| 2 |
Leiden University | 100% | 91% | 100% | 91%
91% of author's last 11 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Feb 16, 2021
All COIs/funding sources disclosed
| |
| 3 |
Cardiff University | 90% | 100% | 100% | 90%
90% of author's last 21 articles (2019-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Apr 7, 2021
All COIs/funding sources disclosed
| |
| 3 |
Max Planck Institute | 90% | 100% | 100% | 90%
90% of author's last 10 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Mar 1, 2021
All COIs/funding sources disclosed
| |
| 3 |
University of Leipzig | 90% | 100% | 100% | 90%
90% of author's last 10 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Feb 25, 2021
All COIs/funding sources disclosed
| |
| 6 |
Princeton University | 100% | 82% | 100% | 82%
82% of author's last 11 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Mar 3, 2021
All COIs/funding sources disclosed
| |
| 6 |
Curate Science | 100% | 82% | 100% | 82%
82% of author's last 11 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Feb 14, 2021
All COIs/funding sources disclosed
| |
| 8 |
University of Cambridge | 100% | 80% | 100% | 80%
80% of author's last 10 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Mar 9, 2021
All COIs/funding sources disclosed
| |
| 9 |
Karolinska Institutet & Stockholm U | 100% | 71% | 100% | 78%
78% of author's last 18 articles (2019-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Apr 26, 2021
All COIs/funding sources disclosed
| |
| 10 |
Center for Open Science | 100% | 100% | 77% | 77%
77% of author's last 13 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Mar 9, 2021
All COIs/funding sources disclosed
| |
| 11 |
Yale University | 100% | 75% | 100% | 75%
75% of author's last 12 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Mar 25, 2021
All COIs/funding sources disclosed
| |
| 12 |
Western University | 82% | 76% | 71% | 53%
53% of author's last 17 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Feb 26, 2021
All COIs/funding sources disclosed
| |
| 13 |
Harvard University | 100% | 45% | 36% | 18%
18% of author's last 11 empirical articles (2016-2021) MEET
Curate Science transparency standard
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possibleEmpirical
As of Jan 27, 2021
All COIs/funding sources disclosed
|
Notes. ✝ = Author audited according to updated audit protocol: Empirical and non-empirical articles audited from last 2 years (or last 5 years for self-nominated authors; see protocol). * = Audited transparency verified by author. ** = Author is also initiative lead.
Audit Process
|
1. Identify Eligible Articles
|
|
2. Check Transparency of Articles
|
|
3. Author Verification Opportunity
|
|
4. Audit Results Posted
|
- Identify articles: First, eligible articles are identified by searching Google Scholar and the author's personal website, for any, and all, peer-reviewed articles (empirical and non-empirical articles) published in the eligibility time period. For our current audit #3 (NIH), this corresponds to articles published in 2015-2019. For our second audit (#2), this corresponds to articles published in the last 2 years (i.e., between Jan 2019 and March 2021; or last 5 years for self-nominated authors). For our 1st audit, this corresponded to empirical articles (only) published in the last 5 years (i.e., between Jan 2015 and Jan 2021 or prior to 2015 for early adopters; minimum N=10 articles).
- Check transparency: The transparency status (OA, open data, & COI disclosures) of each article is checked by searching for the relevant transparency information within the article full-text (a Sci-Hub PDF or preprint is used if full-text NOT available). This is done by 1 or 2 auditors (for our 1st audit, Etienne P. LeBel extracted transparency information for all articles; an anonymous PhD student, independently coded a subset of each author's articles (3-5 articles/author; N=34 total); all discrepancies (13) were easily resolved, see spreedsheet).
- Author verification: Audited researchers are then given an opportunity via email to verify/revise/add to their transparency information BEFORE audit results are publicly posted (2-week response window, 2 reminders maximum; see current email sent to authors (as of April 2, 2021).
- Post audit results: If audited researchers provide additional transparency information or corrections, the information is updated on the website, and authors are given a final opportunity to check the correctness of the updates before public release of audit results.
Transparency Standard Requirements
Curate Science transparency standard Empirical v0.5.0 2021
Gold, Diamond, or Green OA available
Minimal datasets (for all studies) publicly available (when ethically possible)Empirical
All COIs/funding sources disclosed
-
Open access (OA): Working link to article HTML/PDF full-text (version-of-record) at journal or (co-)author's website. Details:
- Gold, Diamond, or Green OA: Green OA includes any form of self-archiving of article (version-of-record) posted on a personal website, institutional or government repository (e.g., PubMed Central; Europe PMC), or pre-print server.
- Non-copy-edited version allowed
- Full-text access without having to login (hence, no Academia.edu or ResearchGate links)
-
Open data: Minimal dataset (for all studies) publicly posted online (for empirical articles ONLY). Working link to dataset included within article full-text, on [co-]author website/publication list, or on Curate Scholar author page (completeness of dataset(s) NOT verified). Allowed:
- Personal websites allowed
- Proprietary files allowed (e.g., SPSS .sav files, STATA .do files, etc.)
- Databook/codebook NOT required
Valid exemptions:- Accessible via registration for large public datasets (URL to dataset registration must be provided)
- Ethical/confidentiality reasons (exemption evidence needed, e.g., consent form)
-
COI disclosures: Following 2 disclosures made in article, on [co-]author website/publication list, or on Curate Scholar author page (as already done in reporting guidelines):
- Any, and all, (potentially) competing interests (financial or non-financial) that occurred within 5 years of conducting the research/preparing an article. Disclosures must be *specific* and *comprehensive* (see list of example financial/non-financial COIs that must be declared).✝
- Names of any, and all, funding sources/research grants received
✝"Grandfather" provisions: Any pre-existing vague and/or non-comprehensive COI disclosures are "grandfathered" to MEET the standard. However, once an author is audited, specific and comprehensive COI disclosures must be made in future publications.
A researcher's transparency track record is said to MEET the Curate Science transparency standard when 50% or more of their articles, within the last 5 years, MEET the article-level requirements (authors in green rows, e.g.,
Anna van 't Veer
91%
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possible
All COIs/funding sources disclosed
).
Anna van 't Veer
91%
91% of A van 't Veer's last 11 articles (2014-2021) MEET the
Curate Science transparency standard Empirical
Curate Science transparency standard Empirical
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possible
As of Feb 16, 2021
All COIs/funding sources disclosed
Rationale/Motivations
FAQ
About the transparency audits
1. How were authors chosen for the first round of transparency audits?
First, this initiative is a "proof of concept" pilot project to demonstrate the feasibility of conducting researcher transparency audits in science. In our fist audit, authors were chosen either because they were (1) transparency advocates/leaders/early adopters and/or (2) were high-impact/high-visibility researchers previously involved in open-science/replication-related discussions. Subsequent audits (in process) will focus on self-nominated authors (audit #2) or publicly-funded researchers from a broader population frame (e.g., NIH grantees, audit #3; incidentally, all authors in audit #1 have received public funding).
2. Aren't researchers just going to "game" their transparency?
It's of course possible that some researchers may try to game their transparency, but we think this is very unlikely given that:
- Gaming COI disclosures is unethical (with some arguing it perhaps should be considered researcher misconduct, e.g., Resnick, 2019)
- Open access by its very nature cannot be gamed,
- Gaming open data -- eg publicly posting a minimal dataset that fails to include all variables needed to reproduce all findings -- will eventually be transparently clear to the research community, which, if repeated over time, could negatively influence one’s reputation.
3. Aren't these audits *unfair* to junior researchers/early-career researchers (ECRs)?
No because the minimum transparency standard was chosen specifically to be as low & generous as possible. Plus, the N=10 article minimum actually provides extra generosity/leeway to junior researchers because it means any researcher with fewer than N=10 eligible articles are NOT eligible to be audited.
4. Aren't audits *unfair* to labs with very few resources?
Meeting the low & generous transparency level may involve some nominal costs, mostly in terms of time. But in science, transparency is not optional, so these costs should be budgeted in an a priori/front-loaded way. That said, the flexible & accommodating requirements, exemptions, and grandfather provisions offered mitigate against this concern.
5. What are the benefits of transparency audits?
Transparency audits offer a multitude of benefits for ALL research stakeholders (i.e., researchers, journals, universities, funding agencies, public policy analysts, and citizens/patients). Researchers can signal their transparency track record on their website/CV (e.g., via customizable widgets
91%
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possible
All COIs/funding sources disclosed
; see example of transparency widget used at the researcher- and article-levels).
Universities can use job candidate's transparency track record in hiring/promotion decisions. Funding agencies can base grant application/renewal decisions on researchers' transparency track record, and can also require all grant outputs/publications to meet a minimum transparency standard (in addition to several other applications).
91%
91% of A van 't Veer's last 11 articles (2014-2021) MEET the
Curate Science transparency standard Empirical
Curate Science transparency standard Empirical
Gold, Diamond, or Green OA available
Public minimal dataset when ethically possible
As of Feb 16, 2021
All COIs/funding sources disclosed
About the auditors/organization
6. What authority does Curate Science have to do transparency audits?
Curate Science -- which is currently operated as a non-profit -- does NOT technically have any official authority. However, our transparency audits, and transparency standards, are based on foundational first principles, and current ethical principles that all researchers are expected to abide by in their research (e.g., NWO code of conduct; NIH code of conduct). We also practice what we preach by conducting our transparency audits transparently (in fact, the curated article-level transparency information displayed in the leaderboard table makes it easy to verify the correctness of the audit results). Additionally, audit results go through various quality control processes PRIOR to public dissemination (e.g., author-verified checks), and we're committed to swiftly correcting any flagged errors/inaccuracies. Finally, we are working on partnering with (or becoming) an "external indepedent organization" (as recommended by Shamoo, 2016).
7. Who audits the auditors?
Given that we strive to conduct our transparency audits transparently, this means anyone can independently verify the correctness of our audits. In fact, we've intentionally "surfaced" article-level transparency information within the leaderboard table itself to make it easy for independent parties to verify the correctness of audited article's transparency information. Moving forward, we will further increase the transparency of our audit process by improving our technical tools, e.g., open database of all auditor-auditee interactions and other audit metadata (upon securing additional funding).
8. Who is funding this initiative?
The initiative is part of Curate Science, which is currently being operated using funds from a European Commission Marie-Curie grant. We are in the process of securing additional funds to improve our auditing tools so we can scale up transparency audits to cover all major scientific fields, including the medical, life, and social sciences.
9. Who is behind the audits?
The audits are spearheaded by Etienne P. LeBel, lead of Curate Science, as part of its broader mission to ensure the transparency of science (NOTE: Previous Curate Science advisory board members were - and are - NOT involved in developing/conducting the audits). Practising what we preach, below are COI disclosures for all individuals involved in conducting the audits:
- Etienne P. LeBel: Salary and project expenses covered by a European Commission Marie-Curie grant (2018-2020). Executive director of Curate Science (uncompensated). No other financial/non-financial COIs to declare (for activities occurring within the last 10 years).
- Anonymous (volunteer) PhD Student: No financial/non-financial COIs to declare (for activities occurring within the last 10 years).
About the minimum transparency standards
10. How were the minimum requirements for the transparency standards chosen?
Minimum requirements were chosen to be as generous/lenient as possible (i.e., we chose the fewest possible requirements) while still conforming to current ethical codes of conduct scientists are expected to abide by. For example, public posting of data when ethically possible (NWO/Netherlands, Section "2. Principles", Subsection "3. Transparency", sentence 2: "If parts of the research or data are *NOT* to be made public, the researcher must provide a good account of why this is not possible."; NIH/United States, Section "Publications", paragraph 3, sentence 2: The NIH position on reproducibility states that *ALL* essential data should be included in the published paper or be deposited in appropriate public databases or made available online.) and declaring any and all competing interests that could impair objectivity/impartiality (NWO/Netherlands, Section "3.4 Reporting Results", point 44: Be open and complete about the role of external stakeholders, commissioning parties, funding bodies, possible conflicts of interest and relevant ancillary activities.; ALLEA/European Union, Section "2.7 Publication & Dissemination", paragraph 5: All authors disclose any conflicts of interest and financial or other types of support for the research or for the publication of its results.). NOTE: Our standards can be considered "informal standards", though in the near future we will be working on achieving formal standards (e.g., using an ISO standard -- or ISO technical specification -- for the transparency standards; using ISO 19011:2018 Auditing Management System for our auditing system).
11. But isn't the minimum transparency standard too LOW?
Technically yes, but it’s still better for the research community to be meeting a transparency standard that might be TOO low, than not meeting any standard at all. That said, the chosen minimum transparency standard still reflects a higher transparency level than the vast majority of currently published research, which have very low open data rates, e.g., only 2% in psychology, 7% in the social sciences, and 13% in biomedicine.
12. But isn't there still some arbitrariness in these minimum standards?
To some extent yes, but it is still better for the research community to have its researchers meet a minimum transparency standard with small amounts of arbitrariness, than NOT meeting any transparency standard at all (the current situation).
13. What's this business with "grandfathering" pre-existing COI disclosures?
To smoothen the transition toward greater transparency in disclosing financial/non-financial COIs, we offer "grandfathered provisions" to audited authors' pre-existing COI disclosures. This means that a pre-existing VAGUE and/or NON-COMPREHENSIVE COI disclosure is said to "meet" our standard. Once an author has been audited, however, their COI disclosures in FUTURE publications will need to meet the NEW (more transparent) standard, which requires SPECIFIC and COMPREHENSIVE financial COI disclosures.
- *SPECIFIC* means disclosing specific details of each COI separately (e.g., speaking fees, consultant fees, stock ownership, etc; see list of example financial/non-financial COIs) rather than disclosing a general category of COIs.
- *COMPREHENSIVE* means disclosing any, and all, financial and non-financial COIs, even if a researcher believes a financial COI is NOT relevant to the reported research.
Grandfather provisions are offered because (1) many researchers have NOT received formal/proper ethics training, and (2) because humans are vulnerable to various cognitive and social psychological biases which leads them to justify VAGUE and NON-COMPREHENSIVE disclosures as sufficient.
Example #1:
VAGUE COI disclosure (meets standard under GRANDFATHER clause):
VAGUE COI disclosure (meets standard under GRANDFATHER clause):
John Smith is a consultant for Beneficial Nonprofit, a nonprofit organization with a mission to develop and deliver methods for making the world a better place.
SPECIFIC COI disclosure (meets NEW standard):
John Smith is a paid consultant who regularly receives consulting fees for giving speeches and workshops for Beneficial Nonprofit, a nonprofit organization with a mission to develop and deliver methods for making the world a better place.
Example #2:
VAGUE and NON-COMPREHENSIVE COI disclosure (meets standard under GRANDFATHER clause):
VAGUE and NON-COMPREHENSIVE COI disclosure (meets standard under GRANDFATHER clause):
John Smith is an employee for Ethical Nonprofit, a nonprofit organization with a mission to improve ethics in the workplace.
SPECIFIC and COMPREHENSIVE COI disclosure (meets NEW standard):
John Smith is a paid employee (executive director) for Ethical Nonprofit, a nonprofit organization with a mission to improve ethics in the workplace. John Smith is also a paid consultant who regularly receives consulting fees for giving speeches and workshops for Beneficial Nonprofit, a nonprofit organization with a mission to develop and deliver methods for making the world a better place.
Miscellaneous
14. Is it unethical to NOT meet the minimum transparency standard?
No. As an author, it is NOT officially unethical to NOT meet the requirements of our article- or researcher-level transparency standards. Ethicality issues like these are beyond the scope of our initiative, falling in the purview of ethicists and ethics committees. In contrast, our job/role is simply to audit/check whether an author's recent articles meet a specific transparency standard (which we've intentionally chosen to be as low/generous as possible).
15. Is research NOT meeting the minimum transparency standard *less* trustworthy?
Perhaps yes, but not necessarily so. Research that meets the minimum transparency standard, all else being equal, *is* more likely to be trustworthy compared to research NOT meeting such standard because transparent research can more thoroughly and easily be scrutinized. That said, transparent research is NOT automatically MORE trustworthy because transparent research can contain errors that have NOT YET been identified. Rather, transparency guarantees that your research will get the credibility it deserves. By corrollary, researchers with a transparent track record are guaranteed to earn the credibility they deserve as a scholar in their respective fields of research. This can act as a prophylactic against publishing tenuous findings given the realization that it will be easier for other researchers to identify errors in one's research.
16. How can I add missing transparency information to my (already) published articles?
You can do so by adding the information to your publication list via a Curate Scholar author page, or on your own website publication list (example 1; example 2; example 3).
17. But what happens if one of my co-authors does *NOT* want to publicy post data to meet the transparency standard?
As a co-author you "benefit" from a publication, but this means you also must share responsibility in terms of transparency and accountability. Of course, you cannot "force" a co-author/lead author to publicly post data, but you can try to convince them. In fact, this is an additional benefit of transparency audits: An audited author will have a heightened motivation to convince their co-authors to be more transparent by presenting various arguments to them (e.g., citing selfish reasons like the open data citation advantage, basic scientific hygiene, ethical duty, etc.). This also means researchers might become more judicious in co-authoring a paper with other researchers who may not share the same ethical values regarding data sharing.
Next steps
18. What are our next steps?
Our next steps involve (1) engaging in additional community consultations via various outreach activities (e.g., participating in an open debate about transparency audits; details coming soon) and (2) conducting additional audits to further demonstrate the feasibility of our approach. Then we will consider scaling up transparency audits.
19. Is it really feasible to do transparency audits at a larger/broader scale?
Though challenging, it *IS* currently feasible to conduct transparency audits at scale given various new technologies and tools at our disposal. Transparency standards exist that can flexibly be used across scientific fields and can accommodate 16 different article types. Web crawling (e.g., Scrapy) and text-mining technologies can be used to automate as much of the transparency metadata extraction/harvesting as possible from OA articles/PDFs. This includes the following (open-source) tools:
- Open access: Swiss National Science Foundation's Open Access check tool to determine open access status
- Open data: QUEST's (Germany) Open Data Detection in Publications (ODDPub) text-mining tool to extract open data status from data availability statements
- COI/funding disclosures: Serghiou et al.'s (2021) "rtransparent" R package that automatically extracts COI/funding disclosure statements from PDF/XML articles
Please contact us at contact@curatescience.org to report any transparency audit errors and/or concerns/feedback about the initiative.
This initiative has been partly funded by a European Union Horizon 2020 research and innovation programme grant (Marie Sklodowska-Curie grant agreement No 793669).