Previously published algorithms for LR (k) incremental parsing are inefficient, unnecessarily restrictive, and in some cases incorrect. We present a simple algorithm based on parsing LR(k) sentential forms that can incrementally parse an arbitrary number ...
Introduction - Tree-sitter
Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:
Packrat parsing: | Proceedings of the seventh ACM SIGPLAN international conference on Functional programming
For decades we have been using Chomsky's generative system of grammars, particularly context-free grammars (CFGs) and regular expressions (REs), to express the syntax of programming languages and protocols. The power of generative grammars to express ...
Bidirectionalization for free! (Pearl) | Proceedings of the 36th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
A bidirectional transformation consists of a function get that takes a source (document or value) to a view and a function put that takes an updated view and the original source back to an updated source, governed by certain consistency conditions ...
Combinators for bidirectional tree transformations: A linguistic approach to the view-update problem: ACM Transactions on Programming Languages and Systems: Vol 29, No 3
We propose a novel approach to the view-update problem for tree-structured data: a domain-specific programming language in which all expressions denote bidirectional transformations on trees. In one direction, these transformations---dubbed lenses---map ...
Resugaring | Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
Syntactic sugar plays a crucial role in engineering programming languages. It offers convenient syntax and higher-level of abstractions, as witnessed by its pervasive use in both general-purpose and domain-specific contexts. Unfortunately, the ...
FlowLog - Efficient and Extensible Datalog | FlowLog
FlowLog: Efficient and Extensible Datalog via Incrementality
Generic deriving of generic traversals | Proceedings of the ACM on Programming Languages
Functional programmers have an established tradition of using traversals as a design pattern to work with recursive data structures. The technique is so prolific that a whole host of libraries have been designed to help in the task of automatically ...
Tree-sitter - a new parsing system for programming tools - Strange Loop
Strange Loop is a conference for software developers covering programming langs, databases, distributed systems, security, machine learning, creativity, and more!
Integrating functional and imperative programming | Proceedings of the 1986 ACM conference on LISP and functional programming
We present a new model, based on monads, for performing input/output in a non-strict, purely functional language. It is composable, extensible, efficient, requires no extensions to the type system, and extends smoothly to incorporate mixed-language ...
Kaitai Struct: declarative binary format parsing language
Kaitai Struct is a formal language for binary format specification that can be compiled into parser code
Boomerang | Proceedings of the 35th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Bidirectional programming languages are a practical approach to the view update problem. Programs in these languages, called lenses, define both a view and an update policy - i.e., every program can be read as a function mapping sources to views as well ...
Engineering High-Performance Parsers with Data-Oriented Design
Notes from building Yuku: the AST is flat arrays of u32 indices instead of a pointer tree, and memory layout, allocation, strings, unicode, and serialization all follow from that one decision.
Tokenization: A Survey for Modern NLP
While modern language models take raw text as their input and produce raw text as output, they do not operate over text directly. Hidden in the very first step of language model pipelines is...
Recursive Language Models
We propose Recursive Language Models (RLMs), an inference strategy where language models can decompose and recursively interact with input context of unbounded length through REPL environments.
