Ethics and etiquette of informal peer review

Updated 1mo

Serendipitous finds about people discussing the ethics and etiquette of conducting informal peer review Generated from a Zotero collection. Will tidy up later

Cards

Similar cards Mentions Connections

Appears in

arxiv.org

You can just review things: A digital ethnography of informal peer review

Across scholarly communities, manuscripts face similar evaluative rituals: editors invite experts to privately assess submissions through formal peer reviews. This closed, loosely structured, and publisher-mediated process is now being supplemented by critiques on open, distributed platforms. We call this practice, a blend of three open peer review variants, informal peer review as it is accessible to outsiders, unmediated by publishers, and conducted across public platforms. Informal peer reviewers range from occasional error detectors to experienced sleuths who identify plagiarism, fraud, errors, conflicts of interest, and conceptual flaws. They may interpret methods, clarify jargon, assess value, and connect to related work. Here, we asked four questions: (1) Who are informal peer reviewers? (2) Where do they work? (3) How do they evaluate research? and (4) What are their impacts? To answer these questions, we conducted a cross-platform digital ethnography with participant observation. We traced discourse across communities over four months and revisited cases after nine and twelve months. From 15 communities, we selected 12 case mentions (10 unique cases) and 8 meta-commentaries from 26 reviewers. Using open and axial coding, we generated 1,080 codes and four themes: reviewers are a motley crew, they self-organize across subpar digital spaces, use deep, uncommon strategies, and they face resistance from authors, publishers, and editors. Informal peer review, we concluded, is a fragile, minimally governed patchwork of people, platforms, and practices, as well as an emerging evidence infrastructure that can be scaled up. We advise advocates and tool-builders to evolve informal review tools, communities, training, and governance by connecting to scholars' values, reducing participation friction, and rewarding attempts to extend the scholarly dialogue.

arxiv.org

You can just review things: A digital ethnography of informal peer review

Across scholarly communities, manuscripts face similar evaluative rituals: editors invite experts to privately assess submissions through formal peer reviews. This closed, loosely structured, and...

hal.science

Peer Production in Citizen Science: A Community-Centered Approach on the Example of Personal Science

Citizen science encompasses a wide range of practices where online collaboration for knowledge production plays a significant role. However, the study of forms of online collaboration other than crowdsourcing in citizen science has remained largely unexplored. This thesis aims to fill this gap by investigating peer production as a form of collaboration in online citizen science communities of practice. First, peer production theory was operationalized as a working model and used to analyze collaboration in citizen science case studies. This was followed by a comprehensive participatory design process for a specific use case involving the personal science community of practice. This process resulted in the creation of the “Personal Science Wiki”, an online space for consolidating community knowledge through peer production. Subsequently, a usability and card sorting study identified and resolved issues with the wiki implementation, and provided insights into mental models and content requirements regarding self-research knowledge. The lessons learned from the participatory design process were generalized as process recommendations for designing peer production solutions and knowledge management systems with communities of practice.

thescientificreview.org

A New Paradigm for Scientific Publishing, Peer Review, and Impact Assessment

Scientific publishing and peer review have evolved little in three centuries, while the demands placed on them have grown profoundly. The growing role of artificial intelligence has underscored deep, systemic shortcomings of an aging system that has largely evaded innovation, a system whose origins are appallingly closer to the invention of the printing press than to the internet. We can do better – much better. This article is intended as the beginning of a communal experiment: a living document that critically reviews the modern academic publishing and peer-review system and presents a concrete framework to address what bibliometrics experts¹ have characterized as "the pervasive misapplication of indicators to the evaluation of scientific performance". Building on the Leiden Manifesto, DORA, and a body of scholarship spanning many disciplines and decades, we present a community-governed, non-profit platform organized around three trust-weighted impact factors, for articles, authors, and reviewers, with full algorithmic transparency, an open development log, and structural decoupling of credibility scoring from content moderation and from monetization. We invite the community to discuss, critique, and help shape it.

osf.io

Peer Review at the Crossroads

Peer review has long been regarded as a cornerstone of scholarly communication, ensuring high quality and credibility of published research. Although academic journals trace their origins back three centuries, the procedures for evaluating submissions, particularly peer review, have undergone continuous evolvement. Peer review’s formal institutionalization in the mid-20th century represents a significant, yet natural, phase in this ongoing transformation of scholarly communication. By the early 21st century, there emerged an opinion that the conventional model of peer review faces systematic challenges, including inefficiency, bias, and institutional inertia. The study aims to synthesize the evolution, practices, and outcomes of both conventional and innovative peer review models in scholarly publishing. Through a mixed-methods approach combining interpretative literature review and process modeling (Business Process Model and Notation –BPMN), it identifies four frameworks: pre-publication peer review, registered reports, modular publishing, and the Publish-Review-Curate (PRC) model. While the PRC model, which integrates preprints with post-publication review, demonstrates advantages in transparency and accessibility, no single approach emerges as universally ideal. The choice of model depends on disciplinary context, resource availability, and institutional priorities. The analysis underscores the need for adaptable platforms that enable hybrid workflows, balancing rigor with inclusivity. Future research must address empirical gaps in evaluating these innovations, particularly their long-term impact on equity and epistemic norms.

pmc.ncbi.nlm.nih.gov

The Peer Review Crisis Demands Radical Reform: Why Reviewers Should Become Paid Professional Referees

The peer review system, fundamental to scientific quality control, faces a significant crisis. As journal editors, we often need to send up to 35 invitations just to secure two reviewers, confronting daily the collapse of voluntary participation. This reflects a critical imbalance: while publication pressure intensifies, willingness to evaluate diminishes, creating "literature elephantiasis", i.e., an overwhelming proliferation of papers exceeding human processing capacity. Current compensation models, relying on token recognition and database access, fail to incentivize quality engagement and may encourage ethically problematic practices like excessive self-citation. The unchecked infiltration of artificial intelligence into peer review, with minimal enforcement, further undermines system integrity. We propose transforming peer reviewers into professional referees, modeled on sports officiating. This radical solution involves formal training and certification for reviewers, equipping them to assess scientific merit, methodology, and ethics comprehensively. Like sports referees supported by assistants, scientific referees would collaborate with specialists - including statisticians, methodology experts, and reference checkers - ensuring thorough evaluation while distributing workload effectively. Funding would come from publishers or research funders, recognizing peer review as an essential, compensated component of the research lifecycle. Implementation faces challenges including publisher resistance and funding allocation, which we address through phased transition strategies. This professionalization addresses current inequities where conscientious scientists shoulder disproportionate reviewing burdens while others contribute minimally. Professional reviewers would view evaluation as valued career development rather than unwelcome obligation. Critics citing independence concerns overlook the sports analogy: referees maintain impartiality through professional standards despite league compensation. Quality scientific evaluation requires dedicated expertise, adequate training, and fair remuneration. Science deserves better than a system dependent on goodwill and guilt - it needs professional referees now.

https://pmc.ncbi.nlm.nih.gov/articles/PMC12341012/ social preview image

link.springer.com

A billion-dollar donation: estimating the cost of researchers’ time spent on peer review

The amount and value of researchers’ peer review work is critical for academia and journal publishing. However, this labor is under-recognized, its magnitude is unknown, and alternative ways of organizing peer review labor are rarely considered.

https://link.springer.com/article/10.1186/s41073-021-00118-2 social preview image

arxiv.org

Science discussions of retracted articles on Bluesky: public...

Post-publication peer review (PPPR) has emerged as an important supplement to traditional peer review, with social media playing a growing role in publicising potential problems in published...