We’ve launched an open, collaborative platform to build evaluations that test what matters to you. We empower a global community to create qualitative benchmarks for any domain—from medical chatbots to legal assistance. Just as Wikipedia democratized knowledge, Weval aims to democratize evaluation, ensuring that AI works for, and represents, everyone.
Our Approach to Artificial Intelligence
We are experimenting with using AI tools to extend our work as a small nonprofit, so that we can focus our time on reinforcing human connections, conversations, and communities that have eroded.
How public involvement can improve the science of AI
As AI systems from decision-making algorithms to generative AI are deployed more widely, computer scientists and social scientists alike are being called on to provide trustworthy quantitative evaluations of AI safety and reliability. These calls have included demands from affected parties to be given a seat at the table of AI evaluation. What, if anything, can public involvement add to the science of AI? In this perspective, we summarize the sociotechnical challenge of evaluating AI systems, which often adapt to multiple layers of social context that shape their outcomes. We then offer guidance for improving the science of AI by engaging lived-experience experts in the design, data collection, and interpretation of scientific evaluations. This article reviews common models of public engagement in AI research alongside common concerns about participatory methods, including questions about generalizable knowledge, subjectivity, reliability, and practical logistics. To address these questions, we summarize the literature on participatory science, discuss case studies from AI in healthcare, and share our own experience evaluating AI in areas from policing systems to social media algorithms. Overall, we describe five parts of any quantitative evaluation where public participation can improve the science of AI: equipoise, explanation, measurement, inference, and interpretation. We conclude with reflections on the role that participatory science can play in trustworthy AI by supporting trustworthy science.
We build AI that works for humans
Imbue builds AI to help people think, create, and build. We share our tools openly because we believe progress in AI should be collaborative and developer-driven
The AI "Evaluation Crisis" Is an Opportunity to Get Data Flow Right
Why the AI evaluation crisis could force a reckoning on dataset provenance, attribution, and consent.
How large language models can reshape collective intelligence
Collective intelligence underpins the success of groups, organizations, markets and societies. Through distributed cognition and coordination, collectives can achieve outcomes that exceed the capabilities of individuals—even experts—resulting in improved accuracy and novel capabilities. Often, collective intelligence is supported by information technology, such as online prediction markets that elicit the ‘wisdom of crowds’, online forums that structure collective deliberation or digital platforms that crowdsource knowledge from the public. Large language models, however, are transforming how information is aggregated, accessed and transmitted online. Here we focus on the unique opportunities and challenges this transformation poses for collective intelligence. We bring together interdisciplinary perspectives from industry and academia to identify potential benefits, risks, policy-relevant considerations and open research questions, culminating in a call for a closer examination of how large language models affect humans’ ability to collectively tackle complex problems.
Cheap AI chatbots transform medical diagnoses in places with limited care
Studies in Rwanda and Pakistan reveal real-world utility of chatbots in underfunded clinics, and not just in benchmark tests.
Cheap AI chatbots transform medical diagnoses in places with limited care
Studies in Rwanda and Pakistan reveal real-world utility of chatbots in underfunded clinics, and not just in benchmark tests.
Beyond the Individual: Understanding the Evolution of Collective Intelligence
This chapter outlines the evolution of collective intelligence, starting from its ancient roots and concluding with modern digital platforms. It discusses intelligence theories, project examples, and the impact of technology on collaborative efforts. Key focuses include the role of the internet and online communities in boosting our collective IQ, with a particular emphasis on Douglas Engelbart's contributions and the open-source movement, as exemplified by Linux's development. The chapter examines how digital transformation has facilitated new forms of community and knowledge sharing, significantly influencing fields such as management, decision-making, and organizational learning. Various scholars and their definitions of CI are discussed, including Pierre Lévy's vision of universally distributed intelligence and the concept of swarm intelligence in biological sciences. We then move on to practically implemented CI projects, exploring crowdsourcing as a manifestation of CI in business and social projects and examining possibilities of harnessing the wisdom of crowds for problem-solving and innovation. The chapter concludes with a presentation of the current state of collective intelligence academic research.
Harnessing Crowds: Mapping the Genome of Collective Intelligence
Over the past decade, the rise of the Internet has enabled the emergence of surprising new forms of collective intelligence. Examples include Google, Wikipedia,
facebook/tribev2 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
AI for the rest of us - The AI community for everyone
Our community and events are for everyone. We're a community of AI practitioners, educators, students, and enthusiasts who are passionate about making AI accessible to everyone.
✨🙌 AI that helps communities thrive on their own terms
Open protocols + AI-enabled coding = building what we need for ourselves
✨🙌 AI that helps communities thrive on their own terms
Open protocols + AI-enabled coding = building what we need for ourselves
ChatGPT Health and what AI can do for a broken system
Healthcare isn’t working for patients or doctors, but AI tools can help.
Human–AI Collaboration at Scale: Task Criticality, Agency, and Friction Across 250,000 Conversations
Stanford University researchers analyzed nearly 250,000 real-world human-AI conversations from Claude.ai to understand collaborative dynamics, finding that over half of interactions involve...
The knowledge layer for collective intelligence
A portable data substrate for humans and their tools to think together.
