In natural language processing, a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.
The Beginner’s Guide to Text Embeddings & Techniques | deepset Blog
Text embeddings represent human language to computers, enabling tasks like semantic search. Here, we introduce sparse and dense vectors in a non-technical way.
Lexicon Embed
How do we embed, display, and render different Lexicons across the ATmosphere?
Attention (machine learning)
In machine learning, attention is a method that determines the importance of each component in a sequence relative to the other components in that sequence. In natural language processing, importance is represented by "soft" weights assigned to each word in a sentence. More generally, attention encodes vectors called token embeddings across a fixed-width sequence that can range from tens to millions of tokens in size.
EmbeddingGemma: Powerful and Lightweight Text Representations
We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.
Rotary Positional Embeddings: A Detailed Look and Comprehensive Understanding
Since the “Attention Is All You Need” paper in 2017, the Transformer architecture has been a cornerstone in the realm of Natural Language…
Latent space
A latent space, also known as a latent feature space or embedding space, is an embedding of a set of items within a manifold in which items resembling each other are positioned closer to one another. Position within the latent space can be viewed as being defined by a set of latent variables that emerge from the resemblances between the objects.
Overcoming Vocabulary Constraints with Pixel-level Fallback
Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized during training. We propose to augment pretrained language models with a vocabulary-free encoder that generates input embeddings from text rendered as pixels. Through experiments on English-centric language models, we demonstrate that our approach substantially improves machine translation performance and facilitates effective cross-lingual transfer, outperforming tokenizer-based methods. Furthermore, we find that pixel-based representations outperform byte-level approaches and standard vocabulary expansion. Our approach enhances the multilingual capabilities of monolingual language models without extensive retraining and reduces decoding latency via input compression.
Precise Zero-Shot Dense Retrieval without Relevance Labels
While dense retrieval has been shown effective and efficient across tasks and languages, it remains difficult to create effective fully zero-shot dense retrieval systems when no relevance label is available. In this paper, we recognize the difficulty of zero-shot learning and encoding relevance. Instead, we propose to pivot through Hypothetical Document Embeddings~(HyDE). Given a query, HyDE first zero-shot instructs an instruction-following language model (e.g. InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is unreal and may contain false details. Then, an unsupervised contrastively learned encoder~(e.g. Contriever) encodes the document into an embedding vector. This vector identifies a neighborhood in the corpus embedding space, where similar real documents are retrieved based on vector similarity. This second step ground the generated document to the actual corpus, with the encoder's dense bottleneck filtering out the incorrect details. Our experiments show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers, across various tasks (e.g. web search, QA, fact verification) and languages~(e.g. sw, ko, ja).
Email triage with an embedding-based classifier · Adam Wiggins
I built a machine learning pipeline for email triage. Includes a hand-labeled golden set, training the model in the browser, and evals. Plus options for local vs cloud embeddings.
Vector Theory: Epistemology, Political Economy, and Probabilistic Computation
This article introduces vector theory as a critical approach for understanding the shift from symbolic to probabilistic computation in contemporary AI systems. The paper argues that the digital turn organised meaning through discrete bits, Boolean logic, and hierarchical structures, in contrast large language models (LLMs) and diffusion architectures operate through high-dimensional vector spaces, cosine similarity, and probability manifolds. The three sections of the article examine the geometry of meaning, the dynamics of stochastic flow, and the political economy of the vector turn. These sections connect concepts such as vectors, tokenisation and generative AI to critical traditions from Marx through the Frankfurt School to contemporary media theory. Drawing on and expanding Kittler’s media materialism, Stiegler’s grammatisation, and Deleuze’s notion of smooth space, the article argues that existing approaches, developed under the paradigm of discrete digitality and symbolic logic, generate too many explanatory anomalies. The vector paradigm represents a new stage in the real subsumption of cognitive and linguistic labour, where capital reconstitutes language as geometry within proprietary vector space. The article connects this to notions of cognitive anaesthesia and the systematic “smoothing” of social friction, arguing that this potentially threatens the tacit dimension of critical thought, the very faculties required to diagnose the computational regime that produces it.
Web Tiles as the base for Lexicon Embeds
Now that @robin.berjon.com has released an initial spec and implementation of Web Tiles, we can see how we can use or extend this as a base for Lexicon Embeds. Lexicon Embeds is my label for how to have native ATProto content embed in different apps, from display all the way to interactivity— eg an event post in bsky with an RSVP button. Getting this as a general capability - and getting security and display right - means embeds can function in Leaflet and Bsky and anywhere else, rather than o...
Web Tiles as the base for Lexicon Embeds
Now that @robin.berjon.com has released an initial spec and implementation of Web Tiles, we can see how we can use or extend this as a base for Lexicon Embeds. Lexicon Embeds is my label for how to have native ATProto content embed in different apps, from display all the way to interactivity— eg an event post in bsky with an RSVP button. Getting this as a general capability - and getting security and display right - means embeds can function in Leaflet and Bsky and anywhere else, rather than o...
Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and English
Bilingual speakers show cross-lingual activation during reading, especially for words with shared surface form. Cognates (friends) typically lead to facilitation, whereas interlingual homographs (false friends) cause interference or no effect. We examine whether cross-lingual activation in bilingual language models mirrors these patterns. We train Dutch-English causal Transformers under four vocabulary-sharing conditions that manipulate whether (false) friends receive shared or language-specific embeddings. Using psycholinguistic stimuli from bilingual reading studies, we evaluate the models through surprisal and embedding similarity analyses. The models largely maintain language separation, and cross-lingual effects arise primarily when embeddings are shared. In these cases, both friends and false friends show facilitation relative to controls. Regression analyses reveal that these effects are mainly driven by frequency rather than consistency in form-meaning mapping. Only when just friends share embeddings are the qualitative patterns of bilinguals reproduced. Overall, bilingual language models capture some cross-linguistic activation effects. However, their alignment with human processing seems to critically depend on how lexical overlap is encoded, possibly limiting their explanatory adequacy as models of bilingual reading.
Transformer (deep learning)
In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Because self-attention alone is permutation-invariant, transformers inject positional information, typically through positional encodings or learned positional embeddings, so token order can affect the output.
Meandering on Manifolds: The Neural Geometry of Stories Over Time
To fully understand LLM representations, we must understand how they change dynamically, over the course of a prompt or conversation. We investigate these temporal dynamics with a simple case study: how do LLMs represent human emotions while reading short stories, both geometrically (in activation space) and temporally (changing from sentence to sentence)?
