metascience

Coordination Tech in Science: Letters, Journals, and Whatever Comes Next | shishyko!

To modernize our scientific infrastructure, we need new contextualization and coordination technologies that decouple trust from legacy branding — shifting from gatekeeping on write to algorithmic contextualization on read.

Metaventory — the meta-science community

A comprehensive community of the people building meta-science and the tools they make — psychology and economics first, growing science-wide.

The Story behind the Science: Preprints of pandemic potential—how bioRxiv and medRxiv brought preprints to the life sciences

ABSTRACT The bioRxiv and medRxiv preprint servers brought preprinting to the life sciences and played a critical role in disseminating COVID research during the pandemic. Here, I reflect on the birth of bioRxiv and medRxiv and the crucial role so many members of the community played, our experience during the pandemic, and the launch of the new non-profit organization set up to oversee the servers. The pandemic was a stress test for bioRxiv and medRxiv that demonstrated their value and robustness. Under the umbrella of openRxiv, they are now poised to become long-term infrastructure underpinning a new publishing ecosystem.

A Data Utopia for Science-of-Science

Here I want to briefly sketch out a vision for how to solve a key set of problems facing science-of-science researchers, using the relatively new idea of a ‘data trust.’ In my ideal wor…

From Grants to Portfolios: Index Logic for Science

How science can learn from capital markets to coordinate diversity, measure progress, and turn discovery into a public portfolio.

The one science reform we can all agree on, but we're too cowardly to do

OR: the long overdue forest fire

Screening, sorting, and the feedback cycles that imperil peer review

Scholarly journals rely on peer review to identify the science most worthy of publication. Yet finding willing and qualified reviewers to evaluate manuscripts has become an increasingly challenging task, possibly even threatening the long-term viability of peer review as an institution. What can or should be done to salvage it? Here, we develop mathematical models to reveal the intricate interactions among incentives faced by authors, reviewers, and readers in their endeavors to identify the best science. Two facets are particularly salient. First, peer review partially reveals authors’ private sense of their work’s quality through their decisions of where to send their manuscripts. Second, journals’ reliance on traditionally unpaid and largely unrewarded review labor deprives them of a standard market mechanism—wages—to recruit additional reviewers when review labor is in short supply. We highlight a resulting feedback loop that threatens to overwhelm the peer review system: (1) an increase in submissions overtaxes the pool of suitable peer reviewers; (2) the accuracy of review drops because journals must either solicit assistance from less qualified reviewers or ask current reviewers to do more; (3) as review accuracy drops, submissions further increase as more authors try their luck at venues that might otherwise be a stretch. We illustrate how this cycle is propelled by the increasing emphasis on high-impact publications, the proliferation of journals, and competition among these journals for peer reviews. Finally, we suggest interventions that could slow or even reverse this cycle of peer-review meltdown.

PRSM

A productionalized intelligence platform for digital and research infrastructure and outputs

Can anyone fix science?

Science has always been in crisis. This is fine.

The BBN Fund

Building a “Convergent Research for BBNs”

A Vision of Metascience

How does the culture of science change and improve? Many people have identified shortcomings in core social processes of science, such as peer review, how grants are awarded, how people are selected to become scientists, and so on. Yet despite often compelling criticisms, strong barriers inhibit widespread change in such social processes. The result is near stasis, and apathy about the prospects for improvement. People sometimes start new research institutions intended to do things differently; unfortunately such institutions are often changed more by the existing ecosystem than they change it. In this essay we sketch a vision of how the social processes of science may be rapidly improved. In this vision, metascience plays a key role: it deepens our understanding of which social processes best support discovery; that understanding can then help drive change. We introduce the notion of a metascience entrepreneur, a person seeking to achieve a scalable improvement in the social processes of science. We argue that: (1) metascience is an imaginative design practice, exploring an enormous design space for social processes; (2) that exploration aims to find new social processes which unlock latent potential for discovery; (3) decentralized change must be possible, so outsiders with superior ideas can't be blocked by established power centers; (4) ideally, change would align with what is best for science and for humanity, not merely what is fashionable, politically popular, or media-friendly; (5) the net result would be a far more structurally diverse set of environments for doing science; and (6) this would enable crucial types of work difficult or impossible within existing environments. For this vision to succeed metascience must develop and intertwine three elements: an imaginative design practice, an entrepreneurial discipline, and a research field. Overall, it is a vision in which metascience is an engine of improvement for the social processes and ultimately the culture of science.