We’ve launched an open, collaborative platform to build evaluations that test what matters to you. We empower a global community to create qualitative benchmarks for any domain—from medical chatbots to legal assistance. Just as Wikipedia democratized knowledge, Weval aims to democratize evaluation, ensuring that AI works for, and represents, everyone.
Our Approach to Artificial Intelligence
We are experimenting with using AI tools to extend our work as a small nonprofit, so that we can focus our time on reinforcing human connections, conversations, and communities that have eroded.
A Roadmap to Democratic AI — The Collective Intelligence Project
Our roadmap outlines concrete steps that can be taken in 2024 to build a more democratic AI ecosystem that is adaptive, accountable, processes decentralized information, provides public goods, and safeguards human wellbeing. We describe what the ecosystem could build, research, advocate for, and fun
How public involvement can improve the science of AI
As AI systems from decision-making algorithms to generative AI are deployed more widely, computer scientists and social scientists alike are being called on to provide trustworthy quantitative evaluations of AI safety and reliability. These calls have included demands from affected parties to be given a seat at the table of AI evaluation. What, if anything, can public involvement add to the science of AI? In this perspective, we summarize the sociotechnical challenge of evaluating AI systems, which often adapt to multiple layers of social context that shape their outcomes. We then offer guidance for improving the science of AI by engaging lived-experience experts in the design, data collection, and interpretation of scientific evaluations. This article reviews common models of public engagement in AI research alongside common concerns about participatory methods, including questions about generalizable knowledge, subjectivity, reliability, and practical logistics. To address these questions, we summarize the literature on participatory science, discuss case studies from AI in healthcare, and share our own experience evaluating AI in areas from policing systems to social media algorithms. Overall, we describe five parts of any quantitative evaluation where public participation can improve the science of AI: equipoise, explanation, measurement, inference, and interpretation. We conclude with reflections on the role that participatory science can play in trustworthy AI by supporting trustworthy science.
We build AI that works for humans
Imbue builds AI to help people think, create, and build. We share our tools openly because we believe progress in AI should be collaborative and developer-driven
The AI "Evaluation Crisis" Is an Opportunity to Get Data Flow Right
Why the AI evaluation crisis could force a reckoning on dataset provenance, attribution, and consent.
How large language models can reshape collective intelligence
Collective intelligence underpins the success of groups, organizations, markets and societies. Through distributed cognition and coordination, collectives can achieve outcomes that exceed the capabilities of individuals—even experts—resulting in improved accuracy and novel capabilities. Often, collective intelligence is supported by information technology, such as online prediction markets that elicit the ‘wisdom of crowds’, online forums that structure collective deliberation or digital platforms that crowdsource knowledge from the public. Large language models, however, are transforming how information is aggregated, accessed and transmitted online. Here we focus on the unique opportunities and challenges this transformation poses for collective intelligence. We bring together interdisciplinary perspectives from industry and academia to identify potential benefits, risks, policy-relevant considerations and open research questions, culminating in a call for a closer examination of how large language models affect humans’ ability to collectively tackle complex problems.
How large language models can reshape collective intelligence
Collective intelligence underpins the success of groups, organizations, markets and societies. Through distributed cognition and coordination, collectives can achieve outcomes that exceed the capabilities of individuals—even experts—resulting in improved accuracy and novel capabilities. Often, collective intelligence is supported by information technology, such as online prediction markets that elicit the ‘wisdom of crowds’, online forums that structure collective deliberation or digital platforms that crowdsource knowledge from the public. Large language models, however, are transforming how information is aggregated, accessed and transmitted online. Here we focus on the unique opportunities and challenges this transformation poses for collective intelligence. We bring together interdisciplinary perspectives from industry and academia to identify potential benefits, risks, policy-relevant considerations and open research questions, culminating in a call for a closer examination of how large language models affect humans’ ability to collectively tackle complex problems.
Cheap AI chatbots transform medical diagnoses in places with limited care
Studies in Rwanda and Pakistan reveal real-world utility of chatbots in underfunded clinics, and not just in benchmark tests.
Cheap AI chatbots transform medical diagnoses in places with limited care
Studies in Rwanda and Pakistan reveal real-world utility of chatbots in underfunded clinics, and not just in benchmark tests.
Beyond the Individual: Understanding the Evolution of Collective Intelligence
This chapter outlines the evolution of collective intelligence, starting from its ancient roots and concluding with modern digital platforms. It discusses intelligence theories, project examples, and the impact of technology on collaborative efforts. Key focuses include the role of the internet and online communities in boosting our collective IQ, with a particular emphasis on Douglas Engelbart's contributions and the open-source movement, as exemplified by Linux's development. The chapter examines how digital transformation has facilitated new forms of community and knowledge sharing, significantly influencing fields such as management, decision-making, and organizational learning. Various scholars and their definitions of CI are discussed, including Pierre Lévy's vision of universally distributed intelligence and the concept of swarm intelligence in biological sciences. We then move on to practically implemented CI projects, exploring crowdsourcing as a manifestation of CI in business and social projects and examining possibilities of harnessing the wisdom of crowds for problem-solving and innovation. The chapter concludes with a presentation of the current state of collective intelligence academic research.
Harnessing Crowds: Mapping the Genome of Collective Intelligence
Over the past decade, the rise of the Internet has enabled the emergence of surprising new forms of collective intelligence. Examples include Google, Wikipedia,
facebook/tribev2 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
AI tools make things up a lot, and that’s a huge problem | CNN Business
Artificial intelligence-powered tools like ChatGPT have mesmerized us with their ability to produce authoritative, human-sounding responses to seemingly any prompt. But as more people turn to this buzzy technology for things like homework help, workplace research, or health inquiries, one of its biggest pitfalls is becoming increasingly apparent: AI models sometimes just make things up.
More Than a Tool
What We Learned from a Year of Conversations About AI
AI for the rest of us - The AI community for everyone
Our community and events are for everyone. We're a community of AI practitioners, educators, students, and enthusiasts who are passionate about making AI accessible to everyone.
✨🙌 AI that helps communities thrive on their own terms
Open protocols + AI-enabled coding = building what we need for ourselves
