Null hypothesis significance testing (NHST) is the default approach to statistical analysis and reporting in marketing and the biomedical and social sciences more broadly. Despite its default role, NHST has long been criticized by both statisticians and applied researchers, including those within marketing. Therefore, the authors propose a major transition in statistical analysis and reporting. Specifically, they propose moving beyond binary: abandoning NHST as the default approach to statistical analysis and reporting. To facilitate this, they briefly review some of the principal problems associated with NHST. They next discuss some principles that they believe should underlie statistical analysis and reporting. They then use these principles to motivate some guidelines for statistical analysis and reporting. They next provide some examples that illustrate statistical analysis and reporting that adheres to their principles and guidelines. They conclude with a brief discussion.
Why Most Published Research Findings Are False
Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Disclosure: Psychology Changes Everything
We review literature examining the effects of laws and regulations that require public disclosure of information. These requirements are most sensibly imposed in situations characterized by misaligned incentives and asymmetric information between, for example, a buyer and seller or an advisor and advisee. We review the economic literature relevant to such disclosure and then discuss how different psychological factors complicate, and in some cases radically change, the economic predictions. For example, limited attention, motivated attention, and biased assessments of probability on the part of information recipients can significantly diminish, or even reverse, the intended effects of disclosure requirements. In many cases, disclosure does not much affect the recipients of the information but does significantly affect the behavior of the providers, sometimes for the better and sometimes for the worse. We review research suggesting that simplified disclosure, standardized disclosure, vivid disclosure, and social comparison information can all be used to enhance the effectiveness of disclosure policies.
Evidence appraisal: a scoping review, conceptual framework, and research agenda
Abstract Objective Critical appraisal of clinical evidence promises to help prevent, detect, and address flaws related to study importance, ethics, validity, applicability, and reporting. These research issues are of growing concern. The purpose of this scoping review is to survey the current literature on evidence appraisal to develop a conceptual framework and an informatics research agenda. Methods We conducted an iterative literature search of Medline for discussion or research on the critical appraisal of clinical evidence. After title and abstract review, 121 articles were included in the analysis. We performed qualitative thematic analysis to describe the evidence appraisal architecture and its issues and opportunities. From this analysis, we derived a conceptual framework and an informatics research agenda. Results We identified 68 themes in 10 categories. This analysis revealed that the practice of evidence appraisal is quite common but is rarely subjected to documentation, organization, validation, integration, or uptake. This is related to underdeveloped tools, scant incentives, and insufficient acquisition of appraisal data and transformation of the data into usable knowledge. Discussion The gaps in acquiring appraisal data, transforming the data into actionable information and knowledge, and ensuring its dissemination and adoption can be addressed with proven informatics approaches. Conclusions Evidence appraisal faces several challenges, but implementing an informatics research agenda would likely help realize the potential of evidence appraisal for improving the rigor and value of clinical evidence.
Field Experimentation in Marketing Research
Despite increasing efforts to encourage the adoption of field experiments in marketing research (e.g., Campbell 1969 ; Cialdini 1980 ; Li et al. 2015 ), the majority of scholars continue to rely primarily on laboratory studies ( Cialdini 2009 ). For example, of the 50 articles published in Journal of Marketing Research in 2013, only three (6%) were based on field experiments. The goal of this article is to motivate a methodological shift in marketing research and increase the proportion of empirical findings obtained using field experiments. The author begins by making a case for field experiments and offers a description of their defining features. She then demonstrates the unique value that field experiments can offer and concludes with a discussion of key considerations that researchers should be mindful of when designing, planning, and running field experiments.
Qualitative research: standards, challenges, and guidelines
Qualitative research methods could help us to improve our understanding of medicine. Rather than thinking of qualitative and quantitative strategies as incompatible, they should be seen as complementary. Although procedures for textual interpretation differ from those of statistical analysis, because of the different type of data used and questions to be answered, the underlying principles are much the same. In this article I propose relevance, validity, and reflexivity as overall standards for qualitative inquiry.
Promises and Perils of Pre-analysis Plans
The purpose of this paper is to help think through the advantages and costs of rigorous pre-specification of statistical analysis plans in economics. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. A researcher can specify variables, data cleaning procedures, regression specifications, and so on. If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced. I begin by laying out the basics of what a statistical analysis plan actually contains so those researchers unfamiliar with it can better understand how it is done. In so doing, I have drawn both on standards used in clinical trials, which are clearly specified by the Food and Drug Administration, as well as my own practical experience from writing these plans in economics contexts. I then lay out some of the advantages of pre-specified analysis plans, both for the scientific community as a whole and also for the researcher. I also explore some of the limitations and costs of such plans. I then review a few pieces of evidence that suggest that, in many contexts, the benefits of using pre-specified analysis plans may not be as high as one might have expected initially. For the most part, I will focus on the relatively narrow issue of pre-analysis for randomized controlled trials.
datasetpapers — a public research experiment
An experimental approach to versioned, forkable, machine-readable analyses. A prototype, not a product or service.
datasetpapers — a public research experiment
An experimental approach to versioned, forkable, machine-readable analyses. A prototype, not a product or service.
Rutger Bregman on Twitter / X
Devastating review of the degrowth literature (561 studies): --> 'few studies use quantitative or qualitative data...' --> [those that do] 'tend to include small samples or focus on non-representative cases' -->'large majority (almost 90%) are opinions rather than analysis' pic.twitter.com/1OQuUhzfk5— Rutger Bregman (@rcbregman) September 4, 2024
RCTs: Crucial Yet Crucially Limited
RCTs do NOT estimate average effects; they also rarely EXPLAIN that which they have demonstrated.
Why Adjusted Regression Coefficients Are Less Descriptive Than They Look | R Psychologist
An interactive article about how regression adjustment works and the pitfalls of conditional associations in descriptive studies.
In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics
Pre-Analysis Plans (PAPs) for randomized evaluations are becoming increasingly common in Economics, but their definition remains unclear and their practical applications therefore vary widely. Based on our collective experiences as researchers and editors, we articulate a set of principles for the ex-ante scope and ex-post use of PAPs. We argue that the key benefits of a PAP can usually be realized by completing the registration fields in the AEA RCT Registry. Specific cases where more detail may be warranted include when subgroup analysis is expected to be particularly important, or a party to the study has a vested interest. However, a strong norm for more detailed pre-specification can be detrimental to knowledge creation when implementing field experiments in the real world. An ex-post requirement of strict adherence to pre-specified plans, or the discounting of non-pre-specified work, may mean that some experiments do not take place, or that interesting observations and new theories are not explored and reported. Rather, we recommend that the final research paper be written and judged as a distinct object from the “results of the PAP”; to emphasize this distinction, researchers could consider producing a short, publicly available report (the “populated PAP”) that populates the PAP to the extent possible and briefly discusses any barriers to doing so.
99% impossible: A valid, or falsifiable, internal meta-analysis.
This is crazy! The data from the study show patterns that make it nearly impossible for it to be legit. But we can learn a lot from the tone of the comments. "We have concerns about [x], please clarify." Out in the wild, this would be THESE IDIOTS FAKED THEIR STUDY!!
Health Nerd
This is one of the most remarkable academic debacles I've ever seen. A large RCT got published in BMJ. There are currently 44 Pubpeer comments, mostly about the data, including...well. Read for yourself. pubpeer.com/publications/C08779C45DB6E407…
