The San Francisco Declaration on Research Assessment (DORA) is a statement that denounces the practice of correlating the journal impact factor to the merits of a specific scientist's contributions. Also according to this statement, this practice creates biases and inaccuracies when appraising scientific research. It also states that the impact factor is not to be used as a substitute "measure of the quality of individual research articles, or in hiring, promotion, or funding decisions".
A budding discipline around how science gets read and judged - Aris
Scholarly publishing spent thirty years arguing about access. It has barely begun arguing about what readers do with the paper once they have it. At Aris we call that second argument scholarly interface design. Mike Morrison's community calls it ScienceUX. Either way it is becoming a field, and here is why it is worth your attention.
The Problem With Promoting ‘Gold Standard Science’
Opinion | Branding scientific research with a simplified label risks misleading the public and harming scientific literacy.
Can we measure trust in scientific publications? - LSE Impact
Jonathon Alexis Coates outlines how a constellation of static and dynamic indicators could provide a means for assessing the trustworthiness of published research
Leiden Manifesto for Research Metrics
The Leiden Manifesto for Research Metrics (also known as the Leiden Manifesto) is a 22 April 2015 published comment in Nature that includes a list of "ten principles to guide research evaluation".[1] It was formulated by public policy professor Diana Hicks, scientometrics professor Paul Wouters, and their colleagues at the 19th International Conference on Science and Technology Indicators, held between 3–5 September 2014 in Leiden, The Netherlands.[2]
Why we built the Journal of Research on Research and what it cost us - LSE Impact
Gemma Derrick, Bart Penders, Serge Horbach & Tony Ross-Hellauer reflect on the choices, challenges & compromises of launching the Journal of Research on Research
A New Paradigm for Scientific Publishing, Peer Review, and Impact Assessment
Scientific publishing and peer review have evolved little in three centuries, while the demands placed on them have grown profoundly. The growing role of artificial intelligence has underscored deep, systemic shortcomings of an aging system that has largely evaded innovation, a system whose origins are appallingly closer to the invention of the printing press than to the internet. We can do better – much better. This article is intended as the beginning of a communal experiment: a living document that critically reviews the modern academic publishing and peer-review system and presents a concrete framework to address what bibliometrics experts¹ have characterized as "the pervasive misapplication of indicators to the evaluation of scientific performance". Building on the Leiden Manifesto, DORA, and a body of scholarship spanning many disciplines and decades, we present a community-governed, non-profit platform organized around three trust-weighted impact factors, for articles, authors, and reviewers, with full algorithmic transparency, an open development log, and structural decoupling of credibility scoring from content moderation and from monetization. We invite the community to discuss, critique, and help shape it.
Why Most Published Research Findings Are False
Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Paperstars
A better way to evaluate scientific papers. Methodological soundness and transparency, not citation counts.
When the Scoreboard Becomes the Game, It’s Time to Recalibrate Research Metrics - The Scholarly Kitchen
Today's guest post discusses research metrics and their relationship to research integrity, inclusivity, and long-term impact.
In an era where research evaluation methods are evolving, the Research Contribution Claim Network makes trustworthy tracking of non-traditional research output easy!
In this whitepaper, Patrick Hochstenbach (Ghent University Library), Thomas van Himbergen (SURF), Laurents Sesink (SURF) and Herbert Van de Sompel (DANS) introduce the...
Open Evaluation: A Vision for Entirely Transparent Post-Publication Peer Review and Rating for Science
The two major functions of a scientific publishing system are to provide access to and evaluation of scientific papers. While open access (OA) is becoming a reality, open evaluation (OE), the other side of coin, has received less attention. Evaluation steers the attention of the scientific community and thus the very course of science. It also influences the use of scientific findings in public policy. The current system of scientific publishing provides only journal prestige as an indication of the quality of new papers and relies on a non-transparent and noisy pre-publication peer review process, which delays publication by many months on average. Here I propose an OE system, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating. Through signed ratings and reviews, scientists steer the attention of their field and build their reputation. Reviewers are motivated to be objective, because low-quality or self-serving signed evaluations will negatively impact their reputation. A core feature of this proposal is a division of powers between the accumulation of evaluative evidence and the analysis of this evidence by paper evaluation functions (PEFs). PEFs can be freely defined by individuals or groups (e.g. scientific societies) and provide a plurality of perspectives on the scientific literature. Simple PEFs will use averages of ratings, weighting reviewers (e.g. by H-factor) and rating scales (e.g. by relevance to a decision process) in different ways. Complex PEFs will use advanced statistical techniques to infer the quality of a paper. Papers with initially promising ratings will be more deeply evaluated. The continual refinement of PEFs in response to attempts by individuals to influence evaluations in their own favor will make the system ungameable. OA and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.
Research Debt
Science is a human activity. When we fail to distill and explain research, we accumulate a kind of debt...
Position paper: persistent identifiers in research infrastructure policy - Crossref
PIDs have become central to national and international open research strategies, but identifiers alone cannot deliver the connected, open record that researchers, institutions, funders, publishers, and policymakers depend on. Effective research infrastructure rests on three interdependent elements: open, persistent identifiers; rich, open, and linked metadata; and the sustainable governance and resilient operation of the organisations involved. Crossref urges policymakers to evaluate all three together.
The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data
Causal inference is a core task of science. However, authors and editors often refrain from explicitly acknowledging the causal goal of research projects; they refer to causal effect estimates as associational estimates. This commentary argues that using the term “causal” is necessary to improve the quality of observational research. Specifically, being explicit about the causal objective of a study reduces ambiguity in the scientific question, errors in the data analysis, and excesses in the interpretation of the results.
Why the Stockholm Declaration will never work
Abstract. The Stockholm Declaration calls for a moral reformation of scientific publishing—replacing commercial publishers with scholar-led journals, rewar
Imagine community attestation for (scientific) achievements. Labeling that. Which brings me back to the question whether #ATScience should replicate trad. reputation? I really don't know!
