On Writing #3
Periodically I like to gather various observations about writing, and share my perspective.
AI reviewers are here — we are not ready
Nature - Artificial intelligence promises rapid and polite feedback on papers — but we must first review the reviewer.
Screening, sorting, and the feedback cycles that imperil peer review
Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the long-term viability of peer review as an institution. What can or should be done to salvage it? Here, we develop mathematical models to reveal the intricate interactions among incentives faced by authors, reviewers, and readers in their endeavors to identify the best science. Two facets are particularly salient. First, peer review partially reveals authors’ private sense of their work’s quality through their decisions of where to send their manuscripts. Second, journals’ reliance on traditionally unpaid and largely unrewarded review labor deprives them of a standard market mechanism—wages—to recruit additional reviewers when review labor is in short supply. We highlight a resulting feedback loop that threatens to overwhelm the peer review system: (1) an increase in submissions overtaxes the pool of suitable peer reviewers; (2) the accuracy of review drops because journals must either solicit assistance from less qualified reviewers or ask current reviewers to do more; (3) as review accuracy drops, submissions further increase as more authors try their luck at venues that might otherwise be a stretch. We illustrate how this cycle is propelled by the increasing emphasis on high-impact publications, the proliferation of journals, and competition among these journals for peer reviews. Finally, we suggest interventions that could slow or even reverse this cycle of peer-review meltdown.
we can just do things - underreacted
7 days later and we have some results from the experiment. When we demote popular posts we see: - 8.26% fewer "show less like this" (3340 -> 3064) - 0.24% more posts in For You were liked (242438 -> 243024) - 2.43% more feed loads (438867 -> 449537) Per user and per request metrics:
Introducing the Elicit API - Elicit
Search our 138M+ papers and generate Reports using our API.
Science should be machine-readable - Marginal REVOLUTION
One of the leading tasks of our time: We develop a machine-automated approach for extracting results from papers, which we assess via a comprehensive review of the entire eLife corpus. Our method facilitates a direct comparison of machine and peer review, and sheds light on key challenges that must be overcome in order to facilitate […]
Dylan Wiliam on Twitter / X
The best journal article titles tell you what the study actually found, rather than being a teaser to make you read the paper. Here's a good example: "Spaced mathematics practice improves test scores and reduces overconfidence" https://t.co/iXxdkAgYPM ($)— Dylan Wiliam (@dylanwiliam) December 31, 2024
Why Are We Still Doing This?
Hi! If you like this piece and want to support my work, please subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5000 to 185,000 words, including vast, extremely detailed analyses
The review mills, not just (self-)plagiarism in review reports, but a step further
Review mills sum up a new category of reviewer misconduct that flies in the face of reviewer ethics and integrity. A pattern of generic, vague, and repeated affirmations (identical or very similar boilerplate phrasing) is noted in the analysis of 263 review reports, regardless of the scientific content of the papers under review, coupled with coercive citation (perhaps among the main reasons for such behavior), which when combined produce fake reviews. The misconduct associated with review mills is unlike mere plagiarism (self-plagiarism) of reviewer comments. It is important to quantify the problem and to take urgent measures: (a) to identify the review millers; (b) to rectify the published literature; and (c) to determine procedures for journals and publishers on procedures to counter this new type of misconduct.
rohit on Twitter / X
One of the most fun papers I've read in recent times. Excellent. Reminiscent of the good old days of science papers. pic.twitter.com/4a7qRTloSG— rohit (@krishnanrohit) December 30, 2024
I talk a lot about how Facets can do much more than we currently use them for But this time, I decided to find out just how much more can be done with them, so here's a post
Facets as a Formatting Engine
www.alexdln.comJames then uses probably the biggest overinflator here: papermills. Papermills are like a bogeyman of #ScientificPublishing. Today, if you want to raise alarm about a paper (or besmirch someone you don't like, it really goes both ways), you can just claim the study comes from a papermill /22
i wrote a little something ^^
the case for hydrant
90008.leaflet.pub
