Every week there’s a better AI model that gives better answers. But a lot of questions don’t have better answers, only ‘right’ answers, and these models can’t do that. So what does ‘better’ mean, how do we manage these things, and should we change what we expect from computers?
Training AI models doesn't emit that much
If we just make reasonable comparisons instead of crazy ones
How AI Tools Differ from Human Tools
Why the mental models we've built for human interfaces don't work for AI, & how Anthropic's latest guidelines are reshaping tool design for better performance & efficiency.
More Than a Tool
What We Learned from a Year of Conversations About AI
Standards around generative AI
Accuracy, fairness and speed are the guiding values for AP’s news report, and we believe the mindful use of artificial intelligence can serve these values and over time improve how we work.
Why I believe current AI models can really think and understand - Part 2 of 3
Why computers can in principle produce thinking and understanding
The Bitter Lesson: Rethinking How We Build AI Systems
The Race for AI Progress In 2019, Richard Sutton, wrote his groundbreaking essay titled ‘The Bitter Lesson’. Simply put, the essay concludes that systems which get better with higher compute beat the systems that do not. Or specifically in AI: raw computing power consistently wins over intricate human-designed solutions. I used to believe that clever orchestrations and sophisticated rules were the key to building better AI systems. That was a typical sofware dev mentality. You build a system, look for edgecases, cover them and you are good to go. Boy, was I wrong.
Angels-on-the-Shoulder: 5 AI Tools for Better Decisions
Five “angels-on-the-shoulder” AI designs that help people make better decisions.
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attributes and for comparing model performance, tracking progress, and identifying weaknesses in foundation and non-foundation models. They can inform model selection for downstream tasks and influence policy initiatives. However, not all benchmarks are the same: their quality depends on their design and usability. In this paper, we develop an assessment framework considering 46 best practices across an AI benchmark's lifecycle and evaluate 24 AI benchmarks against it. We find that there exist large quality differences and that commonly used benchmarks suffer from significant issues. We further find that most benchmarks do not report statistical significance of their results nor allow for their results to be easily replicated. To support benchmark developers in aligning with best practices, we provide a checklist for minimum quality assurance based on our assessment. We also develop a living repository of benchmark assessments to support benchmark comparability, accessible at betterbench.stanford.edu.
AI Isn't as Powerful as We Think | Hannah Fry
Why I believe current AI models can really think and understand - Part 1 of 2
How the human mind is different from our conception of it
Harness Engineering for Self-Improvement
The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence. This feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the training pipeline and the deployment system, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (Anthropic; OpenAI).
