AI* technology/models/benchmarks

Why AI food looks like that

Worms, holes, and cracks reveal the technical weaknesses of image generators.

Technical Report: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

[vc_row martech_row_background_position=”None” css=”.vc_custom_1742942106856{margin-bottom: 24px !important;}”][vc_column][/vc_column][/vc_row][vc_row martech_row_background_position=”None” css=”.vc_custom_1750970721780{margin-bottom: 40px !important;}”][vc_column width=”5/6″ css=”.vc_custom_1750970738029{margin-bottom: 40px !important;}”][vc_column_text css=”.vc_custom_1764963741477{margin-bottom: 1em !important;}”]This study investigates whether persona prompting improves AI performance on challenging academic benchmarks. We find that despite widespread adoption, assigning expert personas (e.g., “You are a world-class physics expert”) does not reliably improve accuracy. Domain-mismatched…Read More

政府AI「源内」とは？ その“中身”から仕組み、用途まで徹底解説

日本政府の職員向け生成AI利用環境「源内」が、全府省庁18万人を対象に大規模実証の段階に入った。近年よく話題に上るガバメントAIについて、その中身から仕組み、用途まで解説する。

Model system cards

Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.

Fed up with Big Tech, communities turn to data collectives for control

Data collectives and cooperatives, which let creators control the collection and distribution of their data, are emerging as preferred alternatives to big tech companies.

Statement on the US government directive to suspend access to Fable 5 and Mythos 5

The US government has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States.

The weather and climate science AI revolution isn’t revolutionary

Machine learning has its limits—how is it being used?

EKI and Propastop Studied AI Resistance to Propaganda

Fresh comparisons of large language models show that AI’s ability to recognise Kremlin propaganda varies dramatically. At first glance, the leading models appear reliable, but targeted testing reveals that some of them remain surprisingly vulnerable to manipulation.

Keelemudelite mõõdupuu

Sõltumatu hindamisraamistik, mis mõõdab suurte keelemudelite võimekust eesti keeles. Vaata mudelite pingerida kuue võrdlustesti põhjal.

These LLMs are the best at resisting Russian propaganda

Estonian government benchmark shows how dozens of models combat Russia's "strategic narratives."

Why Do Large Language Models (LLMs) Struggle to Count Letters?

Large Language Models (LLMs) have achieved unprecedented performance on many complex tasks, being able, for example, to answer questions on almost any topic. However, they struggle with other...

Attention Is All You Need

"Attention Is All You Need" is a 2017 research paper in machine learning authored by eight scientists and engineers working at Google. The paper introduced a new deep learning architecture known as the transformer, based on the attention mechanism proposed in 2014 by Bahdanau et al. The transformer approach it describes has become the main architecture of a wide variety of artificial intelligence systems, including large language models. At the time, the focus of the research was on improving Seq2seq techniques for machine translation, but the authors go further in the paper, foreseeing the technique's potential for other tasks like question answering and what is now known as multimodal generative AI.

Attention is All you Need

The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 BLEU, achieving a BLEU score of 41.1.

Deficient executive control in transformer attention

Abstract. Although transformers in large language models (LLMs) effectively implement a self-attention mechanism that has revolutionized natural language p

「単語の色を答える」だけでChatGPTが正答率1％に崩壊、単純なテストがAIに難しいわけとは？

「単語の色を答える」だけでChatGPTが正答率1％に崩壊（論文より筆者撮影） 単語の色を答える――一見、単純…

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

We introduce Natural Language Autoencoders (NLAs), an unsupervised method for generating natural language explanations of LLM activations. An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text description and an activation reconstructor (AR) that maps the description back to an activation. We jointly train the AV and AR with reinforcement learning to reconstruct residual stream activations. Although we optimize for activation reconstruction, the resulting NLA explanations read as plausible interpretations of model internals that, according to our quantitative evaluations, grow more informative over training.

Natural Language Autoencoders

Turning Claude's thoughts into text

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

We find a diversity of highly abstract features. They both respond to and behaviorally cause abstract behaviors. Examples of features we find include features for famous people, features for countries and cities, and features tracking type signatures in code. Many features are multilingual (responding to the same concept across languages) and multimodal (responding to the same concept in both text and images), as well as encompassing both abstract and concrete instantiations of the same idea (such as code with security vulnerabilities, and abstract discussion of security vulnerabilities).