I just came across this paper arxiv.org/abs/2509.22908 which coins the term "vericoding" for using LLMs to generate formally verified code, and presents benchmark results for several languages
A benchmark for vericoding: formally verified program synthesis
arxiv.org
Martin Kleppmann
New blog post! ✨ I argue that AI will make formal verification go mainstream. martin.kleppmann.com/2025/12/08/ai-formal-verifica… Three reasons for this: 1. LLMs are getting increasingly good at writing proofs using proof assistants. This will make formal verification vastly cheaper than it's been to date.
Jan 15, 2026 at 9:52 AM
chad/whichlang
What programming language do LLMs default to when you don't tell them? A small benchmark.
Wolfram LLM Benchmarking Project
Results from Wolfram's ongoing tracking of LLM performance. The benchmark is based on a Wolfram Language code generation task.
The Return of Language-Oriented Programming | Middle of Nowhere
I’ve been wondering what LLMs mean for language design and implementation. Some believe that, because language models are obviously trained on existing content, they are inherently less capable of assisting users with new programming languages. Intuitively this makes sense. However:
Compiling knowledge, not retrieving it: a hands-on deep dive into llm-wiki-compiler
The thesis of this piece is simple and uncomfortable: the problem of making an LLM “remember” what you’ve read isn’t solved with more…
Wikipedia:LLM-assisted translation
This guideline applies to machine translation tools that include a large language model ("LLM"). Assume that it applies to any online translation tool unless you have confirmed there is no LLM element.
Verification · mu
The Trusted Verifier Program on mu: a democratic, plural approach to verification. Verification tells you an account is who they claim to be.
Verification · mu
The Trusted Verifier Program on mu: a democratic, plural approach to verification. Verification tells you an account is who they claim to be.
Best LLM for Coding 2026 | AI Coding Model Rankings & Benchmarks
Which AI model writes the best code? We rank every major LLM — open and closed source — across SWE-bench, HumanEval, LiveCodeBench, and Terminal-Bench coding benchmarks. Compare the best LLMs for coding, software engineering, and programming.
Magicoder: Empowering Code Generation with OSS-Instruct
We introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more...
Lisp Against the (LL)Machine
Every Lisp, Scheme, and Lisp-adjacent project listed is a non-toy implementation that is at least somewhat active. When I first wrote this article (2026-03-12), most Lisps did not have a policy or even public stance on LLM contributions. In such cases I posted to their bug trackers or discussion lists to ask. Most maintainers kindly responded.
The Myth of Deterministic Software
How I've come to terms with LLM non-determinism in software and abandoned the comforting lies we've been telling ourselves about traditional software.
Benchmarks | EXO
Transparent benchmarks for LLMs tested on real hardware. Coming soon.
LLM use in scholarly writing poses a provenance problem
Nature Machine Intelligence - LLM use in scholarly writing poses a provenance problem
I feel like using LLMs to flag intent/semantics mismatches (eg do var names seem to match what you actually do) or indirect violations of API contracts or whatever as a supplement to static analysis to produce *better* code would be at least as high impact, and it's like 1% of the discourse
