Rating all the things | Tynan Is Typing | Tynan is afk til 10/19 ⁂ | Offprint

All existing proposals are wrong because they are all right

Research – Open Heart Mind

We’re developing Metavaluation — a participatory framework for recognising and rewarding the value of diverse contributions, from creative work and care to logistics and leadership.

steamtags — rate how well each tag actually fits

Pick any Steam game, see its community tags sized by vote count, and rate for yourself how much each one actually fits — 1 to 10, your call, not an algorithm's.

social-protocols/jabble

Collaborative Fact Checking

More Than Fast And Open: Preprints For Rigor, Too? – ASAPbio

ASAPbio got its start with a meeting of 70 supporters, convened in February 2016 at the HHMI headquarters. We were then incorporated as a nonprofit in

You can just review things: A digital ethnography of informal peer review

Across scholarly communities, manuscripts face similar evaluative rituals: editors invite experts to privately assess submissions through formal peer reviews. This closed, loosely structured, and publisher-mediated process is now being supplemented by critiques on open, distributed platforms. We call this practice, a blend of three open peer review variants, informal peer review as it is accessible to outsiders, unmediated by publishers, and conducted across public platforms. Informal peer reviewers range from occasional error detectors to experienced sleuths who identify plagiarism, fraud, errors, conflicts of interest, and conceptual flaws. They may interpret methods, clarify jargon, assess value, and connect to related work. Here, we asked four questions: (1) Who are informal peer reviewers? (2) Where do they work? (3) How do they evaluate research? and (4) What are their impacts? To answer these questions, we conducted a cross-platform digital ethnography with participant observation. We traced discourse across communities over four months and revisited cases after nine and twelve months. From 15 communities, we selected 12 case mentions (10 unique cases) and 8 meta-commentaries from 26 reviewers. Using open and axial coding, we generated 1,080 codes and four themes: reviewers are a motley crew, they self-organize across subpar digital spaces, use deep, uncommon strategies, and they face resistance from authors, publishers, and editors. Informal peer review, we concluded, is a fragile, minimally governed patchwork of people, platforms, and practices, as well as an emerging evidence infrastructure that can be scaled up. We advise advocates and tool-builders to evolve informal review tools, communities, training, and governance by connecting to scholars' values, reducing participation friction, and rewarding attempts to extend the scholarly dialogue.

Science needs downvotes

A bug bounty module in grants would give criticism a leg up The Soviet Union was good at producing shoes. Factories made 800 million pairs a year, twice as many as Italy, three times as many as the United States. Three pairs per citizen per year. Producers met the targets officials set for them.

Open Evaluation: A Vision for Entirely Transparent Post-Publication Peer Review and Rating for Science

The two major functions of a scientific publishing system are to provide access to and evaluation of scientific papers. While open access (OA) is becoming a reality, open evaluation (OE), the other side of coin, has received less attention. Evaluation steers the attention of the scientific community and thus the very course of science. It also influences the use of scientific findings in public policy. The current system of scientific publishing provides only journal prestige as an indication of the quality of new papers and relies on a non-transparent and noisy pre-publication peer review process, which delays publication by many months on average. Here I propose an OE system, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating. Through signed ratings and reviews, scientists steer the attention of their field and build their reputation. Reviewers are motivated to be objective, because low-quality or self-serving signed evaluations will negatively impact their reputation. A core feature of this proposal is a division of powers between the accumulation of evaluative evidence and the analysis of this evidence by paper evaluation functions (PEFs). PEFs can be freely defined by individuals or groups (e.g. scientific societies) and provide a plurality of perspectives on the scientific literature. Simple PEFs will use averages of ratings, weighting reviewers (e.g. by H-factor) and rating scales (e.g. by relevance to a decision process) in different ways. Complex PEFs will use advanced statistical techniques to infer the quality of a paper. Papers with initially promising ratings will be more deeply evaluated. The continual refinement of PEFs in response to attempts by individuals to influence evaluations in their own favor will make the system ungameable. OA and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.

Defender on Twitter / X

I am starting to think this is the most important thing I've written in the last 6 months pic.twitter.com/XiZA5O8AdN— Defender (@DefenderOfBasic) November 2, 2025

johnwarden/bluenotes

The Bluesky Social application for Web, iOS, and Android

A/B/U Review System - ORI Onboarding

This is the rating system we use at ORI. Given a piece of information, rate it according to the following rubric: