The cost of opinion

Leiden Manifesto for Research Metrics

The Leiden Manifesto for Research Metrics (also known as the Leiden Manifesto) is a 22 April 2015 published comment in Nature that includes a list of "ten principles to guide research evaluation".[1] It was formulated by public policy professor Diana Hicks, scientometrics professor Paul Wouters, and their colleagues at the 19th International Conference on Science and Technology Indicators, held between 3–5 September 2014 in Leiden, The Netherlands.[2]

Weighted DVF: A Simple Model for Scoring and Prioritizing Ideas

Discover a powerful yet straightforward approach to evaluating new ideas by balancing desirability, viability, and feasibility. Learn how…

“I think there’s a lot of value in these.” – Unsung

A blog about software craft and quality

Reversing Assumptions Technique — Think Jar Collective

Think Jar Collective contributor and creativity expert Michael Michalko shares a technique to challenge our own assumptions and in the process spark new thinking.

How to win a best paper award (or, an opinionated take on how to do important research)

An opinionated perspective on how to do important research that makes a difference (and sometimes win awards).

Akin's Laws of Spacecraft Design

1. Engineering is done with numbers. Analysis without numbers is only an opinion.

How To Validate Product Ideas Before (And After) Building Them

People spend too much time writing software relative to speaking to potential customers, prior to launching and after it. Reversing that is a win.

Algorithm appreciation: People prefer algorithmic to human judgment

Even though computational algorithms often outperform human judgment, received wisdom suggests that people may be skeptical of relying on them (Dawes, 1979). Counter to this notion, results from six experiments show that lay people adhere more to advice when they think it comes from an algorithm than from a person. People showed this effect, what we call algorithm appreciation, when making numeric estimates about a visual stimulus (Experiment 1A) and forecasts about the popularity of songs and romantic attraction (Experiments 1B and 1C). Yet, researchers predicted the opposite result (Experiment 1D). Algorithm appreciation persisted when advice appeared jointly or separately (Experiment 2). However, algorithm appreciation waned when: people chose between an algorithm’s estimate and their own (versus an external advisor’s; Experiment 3) and they had expertise in forecasting (Experiment 4). Paradoxically, experienced professionals, who make forecasts on a regular basis, relied less on algorithmic advice than lay people did, which hurt their accuracy. These results shed light on the important question of when people rely on algorithmic advice over advice from people and have implications for the use of “big data” and algorithmic advice it generates.

An s-frame agenda for behavioral public policy research

We have previously argued thatbehavioral scientists have been testing and advocating individualistic (i-frame) solutions to policy problems that have systemic (s-frame) causes and require systemic solutions. Here, we consider the implications of adopting an s-frame approach for research. We argue that an s-frame approach will involve addressing different types of questions, which will, in turn, require a different toolbox of research methods.

Evaluations Are the Real Codebase - The Phoenix Architecture

Why behavior outlives implementations

Open Evaluation: A Vision for Entirely Transparent Post-Publication Peer Review and Rating for Science

The two major functions of a scientific publishing system are to provide access to and evaluation of scientific papers. While open access (OA) is becoming a reality, open evaluation (OE), the other side of coin, has received less attention. Evaluation steers the attention of the scientific community and thus the very course of science. It also influences the use of scientific findings in public policy. The current system of scientific publishing provides only journal prestige as an indication of the quality of new papers and relies on a non-transparent and noisy pre-publication peer review process, which delays publication by many months on average. Here I propose an OE system, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating. Through signed ratings and reviews, scientists steer the attention of their field and build their reputation. Reviewers are motivated to be objective, because low-quality or self-serving signed evaluations will negatively impact their reputation. A core feature of this proposal is a division of powers between the accumulation of evaluative evidence and the analysis of this evidence by paper evaluation functions (PEFs). PEFs can be freely defined by individuals or groups (e.g. scientific societies) and provide a plurality of perspectives on the scientific literature. Simple PEFs will use averages of ratings, weighting reviewers (e.g. by H-factor) and rating scales (e.g. by relevance to a decision process) in different ways. Complex PEFs will use advanced statistical techniques to infer the quality of a paper. Papers with initially promising ratings will be more deeply evaluated. The continual refinement of PEFs in response to attempts by individuals to influence evaluations in their own favor will make the system ungameable. OA and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.

J Notation as a Tool of Thought

Kenneth Iverson’s 1964 language, APL, won him the Turing Award. His award lecture, Notation as a Tool of Thought, argued that better notations would lead people to deeper insights about mathematics. He provided a number of examples ranging across linear algebra, arithmetic, probability, and logic. Unfortunately, most of the mathematics he covers isn’t relevant to programming. However, his core idea still applies, and changing how we describe programs changes how we think about them.

fenc.es — be wrong on the internet, productively

Trace the map of reasonable disagreement. Break arguments into statements, rate confidence and importance, and find the crux.

The Winning Variant

A/B testing, algorithmic optimization, and the small act of ignoring both — and what it means to make something by hand when a number can tell you what would have worked better.