Evidence appraisal: a scoping review, conceptual framework, and research agenda

Qualitative research: standards, challenges, and guidelines

Qualitative research methods could help us to improve our understanding of medicine. Rather than thinking of qualitative and quantitative strategies as incompatible, they should be seen as complementary. Although procedures for textual interpretation differ from those of statistical analysis, because of the different type of data used and questions to be answered, the underlying principles are much the same. In this article I propose relevance, validity, and reflexivity as overall standards for qualitative inquiry.

Identity, positionality and reflexivity: relevance and application to research paramedics

This article introduces the reader to the concepts of identity, positionality and reflexivity and outlines their relevance to research paramedics. We outline how a researcher’s identity and positionality can influence all aspects of research, including the research question, study design, data collection and data analysis. We discuss that the ‘insider’ position of paramedics conducting research with other paramedics or within their specific clinical setting has considerable benefits to participant access, understanding of data and dissemination, while highlighting the difficulties of role duality and power dynamics. While positionality is concerned with the researcher clearly stating their assumptions relating to the research topic, the research design, context and process, as well as the research participants; reflexivity involves the researcher questioning their assumptions and finding strategies to address these. The researcher must reflect upon the way the research is carried out and explain to the reader how they moved through the research processes to reach certain conclusions, with the aim of producing a trustworthy and honest account of the research. Throughout this article, we provide examples of how these concepts have been considered and applied by a research paramedic while conducting their PhD research studies within a pre-hospital setting, to illustrate how they can be applied practically.

The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs | Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

There has been a growing interest in the HCI community to study Health, with particular focus in understanding healthcare practices and designing technologies to support and to enhance these practices. A majority of current health studies in HCI have ...

Conducting a Qualitative Document Analysis

Document analysis has been an underused approach to qualitative research. This approach can be valuable for various reasons. When used to analyze pre-existing texts, this method allows researchers to conduct studies they might otherwise not be able to complete. Some researchers may not have the resources or time needed to do field research. Although videoconferencing technology and other types of software can be used to reduce some of the obstacles qualitative researchers sometimes encounter, these tools are associated with various problems. Participants might be unskillful in using technology or may not be able to afford it. Conducting a document analysis can also reduce some of the ethical concerns associated with other qualitative methods. Since document analysis is a valuable research method, one would expect to find a wide variety of literature on this topic. Unfortunately, the literature on documentary research is scant. This paper is designed to close the gap in the literature on conducting a qualitative document analysis by focusing on the advantages and limitations of using documents as a source of data and providing strategies for selecting documents. It also offers reasons for using reflexive thematic analysis and includes a hypothetical example of how a researcher might conduct a document analysis.

Crowdsourced Health Research Studies: An Important Emerging Complement to Clinical Trials in the Public Health Research Ecosystem

Background: Crowdsourced health research studies are the nexus of three contemporary trends: 1) citizen science (non-professionally trained individuals conducting science-related activities); 2) crowdsourcing (use of web-based technologies to recruit project participants); and 3) medicine 2.0 / health 2.0 (active participation of individuals in their health care particularly using web 2.0 technologies). Crowdsourced health research studies have arisen as a natural extension of the activities of health social networks (online health interest communities), and can be researcher-organized or participant-organized. In the last few years, professional researchers have been crowdsourcing cohorts from health social networks for the conduct of traditional studies. Participants have also begun to organize their own research studies through health social networks and health collaboration communities created especially for the purpose of self-experimentation and the investigation of health-related concerns. Objective: The objective of this analysis is to undertake a comprehensive narrative review of crowdsourced health research studies. This review will assess the status, impact, and prospects of crowdsourced health research studies. Methods: Crowdsourced health research studies were identified through a search of literature published from 2000 to 2011 and informal interviews conducted 2008-2011. Keyword terms related to crowdsourcing were sought in Medline/PubMed. Papers that presented results from human health studies that included crowdsourced populations were selected for inclusion. Crowdsourced health research studies not published in the scientific literature were identified by attending industry conferences and events, interviewing attendees, and reviewing related websites. Results: Participatory health is a growing area with individuals using health social networks, crowdsourced studies, smartphone health applications, and personal health records to achieve positive outcomes for a variety of health conditions. PatientsLikeMe and 23andMe are the leading operators of researcher-organized, crowdsourced health research studies. These operators have published findings in the areas of disease research, drug response, user experience in crowdsourced studies, and genetic association. Quantified Self, Genomera, and DIYgenomics are communities of participant-organized health research studies where individuals conduct self-experimentation and group studies. Crowdsourced health research studies have a diversity of intended outcomes and levels of scientific rigor. Conclusions: Participatory health initiatives are becoming part of the public health ecosystem and their rapid growth is facilitated by Internet and social networking influences. Large-scale parameter-stratified cohorts have potential to facilitate a next-generation understanding of disease and drug response. Not only is the large size of crowdsourced cohorts an asset to medical discovery, too is the near-immediate speed at which medical findings might be tested and applied. Participatory health initiatives are expanding the scope of medicine from a traditional focus on disease cure to a personalized preventive approach. Crowdsourced health research studies are a promising complement and extension to traditional clinical trials as a model for the conduct of health research.

Co-creation process of an app for people with rare diseases - a citizen science approach

Background Rare diseases affect a small percentage of the population, leading to challenges such as delayed diagnoses and limited treatment options. Mobile health technologies offer solutions to improve patient outcomes, yet their application in rare diseases remains underexplored. The German citizen science project SelEe created a customizable app for the self-management of rare diseases through a co-creation process that involved patients with such conditions. Methods The project consisted of three phases. In Phase 1, 9 to 68 patients or relatives of patients participated in workshops to define research topics and app requirements. Phase 2 involved a core research team of nine patients and researchers who iteratively developed the app, released in March 2023. Phase 3 focused on evaluating the app’s usage and usability through an in-app survey conducted from March 2023 to February 2024. We utilized descriptive statistics to evaluate app usage and employed the mHealth App Usability Questionnaire to assess usability. Results The SelEe app offers the possibility to create and store data in a personalized health diary. Patients can create their own templates or use templates which were defined by the core research team. Users can record findings (e.g. blood test results) and export data using different graphs and formats. Furthermore, the app supports blind users. The app was downloaded 3040 times and 1456 users registered, with 1967 unique diseases entered. 50.7% of the diseases were rare, 30.5% non-rare, and 18.8% were classified as suspected, undefined, or symptoms. A total of 1223 valid user profiles were analyzed for app usage and demographics. Furthermore, 432 users qualified for the in-app survey by making at least one health diary entry, and 117 participated. The app was rated with an overall usability score of 5.19 out of 7. While the app’s health diary function was frequently used, other functionalities like findings and data export were less utilized. Feedback highlighted the need for improved usability and additional features. Conclusions The study highlights active patient engagement in developing a mobile health app for individuals with rare diseases. Although improvements are necessary for broader acceptance, the app is promising for the management of rare diseases. Supplementary information The online version contains supplementary material available at 10.1186/s13023-025-04140-1.

Sample Size in Qualitative Interview Studies: Guided by Information Power

Sample sizes must be ascertained in qualitative studies like in quantitative studies but not by the same means. The prevailing concept for sample size in qualitative studies is “saturation.” Saturation is closely tied to a specific methodology, and the term is inconsistently applied. We propose the concept “information power” to guide adequate sample size for qualitative studies. Information power indicates that the more information the sample holds, relevant for the actual study, the lower amount of participants is needed. We suggest that the size of a sample with sufficient information power depends on (a) the aim of the study, (b) sample specificity, (c) use of established theory, (d) quality of dialogue, and (e) analysis strategy. We present a model where these elements of information and their relevant dimensions are related to information power. Application of this model in the planning and during data collection of a qualitative study is discussed.

Individual Experience vs. The Cochrane Review

On my decade-long exploration seeking a scientific language for singular evidence.

NLnet; Nanoarguments

Scientific knowledge is currently scattered across papers, repositories, and disconnected platforms, with no structured way to trace how claims connect to evidence or how arguments develop. Nanoarguments builds a framework and tools for creating, browsing, and contributing to a global, federated graph of scientific discourse and evidence. Researchers and their communities can collaboratively structure claims, evidence chains, and discussion as nanopublications, which are small, cryptographically signed Linked Data snippets with precise provenance and authorship, published to a decentralized peer-to-peer network. The project builds upon the Nanodash interface to help users browse, edit, and aggregate discourse and evidence graphs, and integrates with dokieli to enable in-context authoring of nanopublications as inline annotations while reading or writing a document. A bidirectional ActivityPub connector bridges the nanopublication network and the fediverse, allowing discourse threads to start as social exchanges and crystallize into persistent, machine-readable evidence records. The project will be piloted with early adopter research groups in discourse and evidence modeling. All components will be released as open-source modules that other systems can build upon.

A Department of Methodology: A feasible framework to integrate the applied practice of multidisciplinary support teams

High performance sport organisations employ multidisciplinary sport science teams to develop athlete preparation and performance. However, challenges have been identified with respect to cohesion, communication, integration, ineffective knowledge translation, role clarity, confusing language, and conflicting methodologies. A Department of Methodology (DoM) has been proposed as a conceptual framework seeking to address these issues. To move the DoM beyond conceptualisation towards professional practice, this study aimed to establish expert consensus for the implementation of a DoM within applied high performance contexts. Eighty professionals with expertise in multidisciplinary teams completed a three-round, online Delphi survey. In Round One, participants answered 16, open-ended questions across four categories: (1) Coordinating activity through shared principles and language; (2) Communicating coherent ideas; (3) Designing practice landscapes; and (4), General questions of feasibility. Results highlight the importance of building a shared language, establishing common principles, working collaboratively, facilitating continuous knowledge exchange, and designing practice tasks collaboratively. New knowledge is contributed for advancing conceptualisation of a DoM framework by exploring consensus on a range of theoretical- and practitioner-informed recommendations for the successful implementation of a DoM in high-performance sports organisations.

A Design Science Research Methodology for Expert Systems Development

The knowledge of design science research (DSR) can have applications for improving expert systems (ES) development research. Although significant progress of utilising DSR has been observed in particular information systems design – such as decision support systems (DSS) studies – only rare attempts can be found in the ES design literature. Therefore, the aim of this study is to investigate the use of DSR for ES design. First, we explore the ES development literature to reveal the presence of DSR as a research methodology. For this, we select relevant literature criteria and apply a qualitative content analysis in order to generate themes inductively to match the DSR components. Second, utilising the findings of the comparison, we determine a new DSR approach for designing a specific ES that is guided by another result – the findings of a content analysis of examination scripts in Mathematics. The specific ES artefact for a case demonstration is designed for addressing the requirement of a ‘wicked’ problem in that the key purpose is to assist human assessors when evaluating multi-step question (MSQ) solutions. It is anticipated that the proposed design knowledge, in terms of both problem class and functions of ES artefacts, will help ES designers and researchers to address similar issues for designing information system solutions.

Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another’s Data

Background: This project investigates the ways in which patients respond to the shared use of what is often considered private information: personal health data. There is a growing demand for patient access to personal health records. The predominant model for this record is a repository of all clinically relevant health information kept securely and viewed privately by patients and their health care providers. While this type of record does seem to have beneficial effects for the patient–physician relationship, the complexity and novelty of these data coupled with the lack of research in this area means the utility of personal health information for the primary stakeholders—the patients—is not well documented or understood. Objective: PatientsLikeMe is an online community built to support information exchange between patients. The site provides customized disease-specific outcome and visualization tools to help patients understand and share information about their condition. We begin this paper by describing the components and design of the online community. We then identify and analyze how users of this platform reference personal health information within patient-to-patient dialogues. Methods: Patients diagnosed with amyotrophic lateral sclerosis (ALS) post data on their current treatments, symptoms, and outcomes. These data are displayed graphically within personal health profiles and are reflected in composite community-level symptom and treatment reports. Users review and discuss these data within the Forum, private messaging, and comments posted on each other’s profiles. We analyzed member communications that referenced individual-level personal health data to determine how patient peers use personal health information within patient-to-patient exchanges. Results: Qualitative analysis of a sample of 123 comments (about 2% of the total) posted within the community revealed a variety of commenting and questioning behaviors by patient members. Members referenced data to locate others with particular experiences to answer specific health-related questions, to proffer personally acquired disease-management knowledge to those most likely to benefit from it, and to foster and solidify relationships based on shared concerns. Conclusions: Few studies examine the use of personal health information by patients themselves. This project suggests how patients who choose to explicitly share health data within a community may benefit from the process, helping them engage in dialogues that may inform disease self-management. We recommend that future designs make each patient’s health information as clear as possible, automate matching of people with similar conditions and using similar treatments, and integrate data into online platforms for health conversations.

(PDF) Building a Conceptual Framework: Philosophy, Definitions, and Procedure

PDF | In this paper the author proposes a new qualitative method for building conceptual frameworks for phenomena that are linked to multidisciplinary... | Find, read and cite all the research you need on ResearchGate

New evidence, new challenges: ICC judges’ perspectives on user-generated evidence and judging in an age of artificial intelligence

Evidence recorded on personal digital devices, or “user-generated evidence” (UGE), has profoundly shaped our ways of knowing about international crimes. UGE can be expected to play an important role in future cases before the International Criminal Court (ICC), yet few trials to date have relied extensively on UGE.. This research provides important insights into how ICC judges define UGE and perceive its strengths and weaknesses, and on the readiness of the Court to adapt to judging in an age of Artificial Intelligence. Using grounded theory to analyse interviews with ICC judges, we identified several key themes, including concerns about the perceived importance and potential bias of evidence sources; the practical challenges of employing UGE; the burden placed on the parties to ensure the reliability of the evidence, to rigorously challenge the opposing party’s evidence, and the importance of preparing legal professionals to address the risks associated with misinformation and disinformation.

“Statistical Significance” and Statistical Reporting: Moving Beyond Binary

Null hypothesis significance testing (NHST) is the default approach to statistical analysis and reporting in marketing and the biomedical and social sciences more broadly. Despite its default role, NHST has long been criticized by both statisticians and applied researchers, including those within marketing. Therefore, the authors propose a major transition in statistical analysis and reporting. Specifically, they propose moving beyond binary: abandoning NHST as the default approach to statistical analysis and reporting. To facilitate this, they briefly review some of the principal problems associated with NHST. They next discuss some principles that they believe should underlie statistical analysis and reporting. They then use these principles to motivate some guidelines for statistical analysis and reporting. They next provide some examples that illustrate statistical analysis and reporting that adheres to their principles and guidelines. They conclude with a brief discussion.

Like a Therapist, But Not: Reddit Narratives of AI in Mental Health Contexts

Large language models (LLMs) are increasingly used for emotional support and mental health-related interactions outside clinical settings, yet little is known about how people evaluate and relate to these systems in everyday use. We analyze 5,126 Reddit posts from 47 mental health communities describing experiential or exploratory use of AI for emotional support or therapy. Grounded in the Technology Acceptance Model and therapeutic alliance theory, we develop a theory-informed annotation framework and apply a hybrid LLM-human pipeline to analyze evaluative language, adoption-related attitudes, and relational alignment at scale. Our results show that engagement is shaped primarily by narrated outcomes, trust, and response quality, rather than emotional bond alone. Positive sentiment is most strongly associated with task and goal alignment, while companionship-oriented use more often involves misaligned alliances and reported risks such as dependence and symptom escalation. Overall, this work demonstrates how theory-grounded constructs can be operationalized in large-scale discourse analysis and highlights the importance of studying how users interpret language technologies in sensitive, real-world contexts.