Tokenization and analysis pipeline for full-text search
Introduction
A complete search engine and RAG pipeline in your browser, server or edge network with support for full-text, vector, and hybrid search in less than 2kb.
Fast regex search: indexing text for agent tools · Cursor
How we're building indexes for regular expression search so agents can find text in large monorepos without the 15-second ripgrep waits.
turbopuffer: fast search on object storage
Inaugural blog post about the development of turbopuffer, a search engine that uses object storage and SSD caching for cost-effective, low latency search. This post describes into the motivation behind its creation, its unique architecture, and how it significantly reduces costs for large-scale vector searches. Discover how turbopuffer is transforming search infrastructure for companies like Cursor and Suno, offering a scalable and reliable solution.
Semble: Fast and Accurate Code Search for Agents
Fast and Accurate Code Search for Agents. Uses ~98% fewer tokens than grep+read
Mem0 Research Paper: Token-Efficient Memory Algorithm
Benchmarked across LoCoMo, LongMemEval, and BEAM, achieves competitive accuracy while using under 7,000 tokens per retrieval call. For comparison, full-context approaches on these benchmarks routinely consume 25,000+ tokens per query.
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.
An 8 Year Algorithm Nemesis: Analyzing long string trends in a body of text with suffix trees
The Beginner’s Guide to Text Embeddings & Techniques | deepset Blog
Text embeddings represent human language to computers, enabling tasks like semantic search. Here, we introduce sparse and dense vectors in a non-technical way.
Why Google’s New AI-Saturated Search Page Will Be A Disaster
Google didn’t invent full-text search of the Internet – that honor belongs to early pioneers such as WebCrawler, Lycos and AltaVista. But for the last 25 years or so, Google has…
Meilisearch: Unified Search & AI Retrieval Platform
Build lightning-fast search and AI retrieval with Meilisearch. Open-source, developer-friendly search engine trusted by 20,000+ teams worldwide.
Deep Dive into ParadeDB's v2 API: The Future of SQL Search
Explore ParadeDB's v2 API that eliminates schema duplication, simplifies tokenization, and provides transparent search operators for intuitive SQL-based full-text search.
Back-to-basics approach can match or outperform AI in language analysis
A new study led by Dr Andrea Nini at The University of Manchester has found that a grammar-based approach to language analysis can match or outperform advanced AI systems in identifying who wrote a text. The method, called LambdaG, uses patterns in grammar and sentence construction rather than large-scale AI models, offering comparable accuracy ...
Exclusive: New York Times tests out AI-generated search summaries
In recent weeks, the paper has quietly rolled out a new AI-powered search page for a small subset of visitors.
Whey‽ (@whey.party)
red dwarf now supports full text search using @mackuba.eu 's lycan https://tangled.org/@mackuba.eu/lycan
What Language is This? Ask Your Tokenizer
Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText, GlotLID and CLD3, show that UniLID achieves competitive performance on standard benchmarks, substantially improves sample efficiency in low-resource settings -- reaching ~70% accuracy with as few as five labeled samples per language -- and delivers large gains on fine-grained dialect identification.
Finding real, quotable examples is key to my blog writing workflow. Because Google and the default web search produce a lot of junk, I use Exa, HackerNews, web search, what I've already written, and internal company docs to do this, and I recently added @semble.so to this stack.
