An 8 Year Algorithm Nemesis: Analyzing long string trends in a body of text with suffix trees

Fast regex search: indexing text for agent tools · Cursor

How we're building indexes for regular expression search so agents can find text in large monorepos without the 15-second ripgrep waits.

Introduction - Tree-sitter

Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:

the mud coders guild

:computer: text-based words in long-form format

Introduction

A complete search engine and RAG pipeline in your browser, server or edge network with support for full-text, vector, and hybrid search in less than 2kb.

Back-to-basics approach can match or outperform AI in language analysis

A new study led by Dr Andrea Nini at The University of Manchester has found that a grammar-based approach to language analysis can match or outperform advanced AI systems in identifying who wrote a text. The method, called LambdaG, uses patterns in grammar and sentence construction rather than large-scale AI models, offering comparable accuracy ...

Tree-sitter - a new parsing system for programming tools - Strange Loop

Strange Loop is a conference for software developers covering programming langs, databases, distributed systems, security, machine learning, creativity, and more!

Efficient and flexible incremental parsing | ACM Transactions on Programming Languages and Systems

Previously published algorithms for LR (k) incremental parsing are inefficient, unnecessarily restrictive, and in some cases incorrect. We present a simple algorithm based on parsing LR(k) sentential forms that can incrementally parse an arbitrary number ...

Introducing Wacksy: a library for writing WACZ collections

Blog published on behalf of Pierre Marshall, Technical Research Officer, Algorithmic Archive project. As part of the Algorithmic Archive project, we have been building tooling to support a prospect…

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.

Semble: Fast and Accurate Code Search for Agents

Fast and Accurate Code Search for Agents. Uses ~98% fewer tokens than grep+read

Scalable watermarking for identifying large language model outputs

Large language models (LLMs) have enabled the generation of high-quality synthetic text, often indistinguishable from human-written content, at a scale that can markedly affect the nature of the information ecosystem1–3. Watermarking can help identify synthetic text and limit accidental or deliberate misuse4, but has not been adopted in production systems owing to stringent quality, detectability and computational efficiency requirements. Here we describe SynthID-Text, a production-ready text watermarking scheme that preserves text quality and enables high detection accuracy, with minimal latency overhead. SynthID-Text does not affect LLM training and modifies only the sampling procedure; watermark detection is computationally efficient, without using the underlying LLM. To enable watermarking at scale, we develop an algorithm integrating watermarking with speculative sampling, an efficiency technique frequently used in production systems5. Evaluations across multiple LLMs empirically show that SynthID-Text provides improved detectability over comparable methods, and standard benchmarks and human side-by-side ratings indicate no change in LLM capabilities. To demonstrate the feasibility of watermarking in large-scale-production systems, we conducted a live experiment that assessed feedback from nearly 20 million Gemini6 responses, again confirming the preservation of text quality. We hope that the availability of SynthID-Text7 will facilitate further development of watermarking and responsible use of LLM systems.

AI-text detection tools are really easy to fool

A recent crop of AI systems claiming to detect AI-generated text perform poorly—and it doesn’t take much to get past them.

Atreeshake · eriskii.net

A Bluesky thread treeshaker — paste any post URL and get the whole thread as a force-directed graph you can scroll through as a single linear read.

zzstoatzz.io/quotree

visualize bsky quote trends

Clew

Clew is a web search engine trying to be different from the rest. We are the magic ball of string leading through the internet's labyrinth.