Cydonia
← All posts

Tree-sitter grammars on demand, in wasmtime

Cydonia opens files in a pane and highlights them, and colours fenced code in articles and cards. The highlighting is tree-sitter. The grammars are not in the binary: a grammar is a WebAssembly module that Cydonia downloads the first time it is wanted, verifies against a digest it was built with, caches, and loads into wasmtime. This is how that works, and the three shapes it went through on the way, all inside PR #27.

Naming a language is not painting it

The first commit, feat(file): syntax highlighting for open files, started from a distinction that every later version kept. A file's name can tell you its language even when the build has no grammar for it. The first language.rs put it like this:

the difference between "that is YAML and there is no grammar here" and "that is a file extension nobody has ever written a grammar for", and the only reason an app can offer to fetch one rather than nag about files it could never help with.

So a lookup answers one of three things:

pub enum Language {
    /// An installed grammar, under the name `syntax` knows it by.
    Ready(&'static str),
    /// Markdown, which bezel highlights without a tree-sitter grammar.
    Markdown,
    /// A known language whose grammar is not installed.
    Missing(&'static str),
}

In that first commit the table of names was a hand-kept KNOWN array in Cydonia, eight grammars were compiled in (bash, go, json, python, rust, toml, tsx, typescript), and everything else in the table was Missing. A file in a Missing language opened as plain text.

The registry moves into bezel

The next commits moved the table out of the app. bezel's bezel-syntax crate now holds syntax::registry: one entry per language, with its fence aliases, its file names and extensions, and an optional grammar.

pub struct Entry {
    pub name: &'static str,
    pub aliases: &'static [&'static str],
    pub files: &'static [&'static str],
    /// `None` names a language this build cannot paint.
    pub lang: Option<&'static Lang>,
}

The registry is seeded with names only. bezel-syntax links no grammar of its own; grammars arrive through registry::register, from a provider. Registering an entry with the same name replaces the old one, so a provider can turn a Named language into a Ready one at runtime, and the next lookup sees it. Path matching is the same rule the first table had: a whole file name beats an extension (Dockerfile is not a .file), and between extensions the longest match wins.

The first provider was bezel-syntax-std, which links seven tree-sitter grammar crates natively, each behind a cargo feature. Cydonia called its install once and painted what it registered. That version lasted about three hours.

A catalog instead of a crate

The commit that landed, feat(language): install wasm grammars from a bundled catalog, dropped bezel-syntax-std from the app and turned on bezel-syntax's wasm feature, which is tree-sitter's wasm feature underneath.

What ships in the binary is catalog.json, compiled in with include_str!. It has one row per grammar: the registry name, the symbol the module exports, aliases, file names, and for the module and each highlight query a path, a byte size and a SHA-256.

{"name": "c", "symbol": "c", "aliases": ["c"], "files": ["c", "h"],
 "wasm": {"path": "c/tree-sitter-c.wasm", "size": 625921, "sha256": "7b80a640…"},
 "queries": [{"path": "c/highlights.scm", "size": 1432, "sha256": "3378d854…"}]}

The files come from the [email protected] npm package, fetched through jsDelivr, and the sizes and digests are taken from that package's tarball. The catalog has 33 languages today. The smallest module is JSON at 6 KB; the largest is C# at 5.3 MB. All 33, modules and queries together, come to about 36.5 MB, none of which is in the app until somebody opens a file that needs it.

Some rows are more than one file. TypeScript and TSX carry the JavaScript highlight query before their own, C++ carries C's, and Vue carries HTML's. The symbol can differ from the name: C# is registered as csharp and exports c_sharp. Haskell is in bezel's registry and not in the catalog, because the package's highlight query does not compile against its grammar. Injection and locals queries are not installed.

Install, verify, register

An install runs on its own thread and moves through a status the views read:

Status Meaning
Missing Known, not cached
Downloading { received, total } Fetching the module and its queries
Checking Every file is fetched and matches its digest; the grammar is being loaded
Ready Registered; the next highlight uses it
Failed(String) Stopped, with the reason

Each download asks for the file and reads at most one byte more than the catalog says it is, then checks the length and the SHA-256. A file that is short, long or different fails the install with Incomplete grammar asset or Grammar checksum mismatch.

Before anything is saved, the grammar is run once. validate makes a WasmStore, loads the module under its symbol, compiles the joined highlight query against it, and parses an empty string:

let mut store = syntax::tree_sitter::WasmStore::new(engine()?)?;
let language = store.load_language(&spec.symbol, wasm)?;
syntax::tree_sitter::Query::new(&language, query).context("Invalid highlight query")?;
let mut parser = syntax::tree_sitter::Parser::new();
parser.set_wasm_store(store)?;
parser.set_language(&language)?;
ensure!(parser.parse("", None).is_some(), "Grammar failed to parse");

Only then is the bundle written to disk and the grammar registered. A grammar that downloads cleanly but does not load, or whose query does not compile, is never published to the registry.

The cache is one JSON file per grammar under ~/.local/share/cydonia/grammars/2.0.1 (or $XDG_DATA_HOME), holding the module as base64 and the queries as text. It is not trusted either. On first use of anything in language, Cydonia walks the catalog, reads each cached file (refusing one over 64 MiB), and checks every piece against the catalog's digests again before registering it. A cache file that has been edited or truncated comes back as Failed, not as a grammar.

Two views asking for the same grammar share one download: begin_install refuses to start a second while one is active or done. A failed install can be started again.

Who asks for a grammar

There are two ways in, and they behave differently.

A file. When a file's language is Missing and the catalog has it, the file view shows a notice row: Install YAML highlighting?, with Not now and Install. While it runs the row shows kilobytes received against the total, then Checking, and on failure the error and Retry. Nothing is downloaded until Install is pressed.

A fence. Since PR #42, a Markdown fence tagged with a language is the request. The article and card editors collect the fence labels when a document opens and again on every edit, and call ensure with them. For each label the catalog resolves (aliases included, so sh and zsh both mean bash), a grammar that is Missing starts installing. A grammar that Failed is left alone, so a bad download is not retried on every keystroke; retrying is the file view's Retry.

While any download runs, a single poll loop refreshes the windows every 150 ms, so a fence colours itself on the first frame after its grammar lands. Nothing needs re-registering for that: highlight resolves through the registry on every call.

The fence picker in the editor lists every language in the catalog, cached or not. Choosing one that is not cached is how a fence asks for it.

One engine, one store per parser

The app owns the wasmtime engine. Cydonia builds it once with a default Config and hands a clone to bezel-syntax through syntax::set_engine. Until an engine is set, a Grammar::Wasm cannot load, and highlight returns nothing.

bezel-syntax re-exports tree_sitter::wasmtime, and that is the only handle that works: wasmtime comes in through tree-sitter, and a second wasmtime in the dependency graph would be a second, unrelated Engine type.

A wasm grammar's Language belongs to the WasmStore that loaded it, because tree-sitter dispatches every lex and external-scanner call through that store. So bezel-syntax cannot hold a compiled grammar on the language itself. A Lang holds the module as bytes (Grammar::Wasm(Arc<[u8]>)), and each thread has its own Session: a highlighter, a parser with a store attached, and the queries compiled against it. A language is instantiated into that store the first time the thread highlights it, and its compiled configuration is kept, because a configuration costs milliseconds to compile.

Signing for executable memory

The part outside the code is the bundle. On macOS, Cydonia is signed with the hardened runtime, and wasmtime compiles each grammar module to native code at runtime. The hardened runtime refuses that without an entitlement.

This changed within the same PR. The bezel-syntax-std commit added bundle/cydonia.entitlements with com.apple.security.cs.allow-jit, the narrower entitlement, which covers memory mapped with MAP_JIT. The catalog commit replaced it:

<!-- Wasmtime 48 publishes compiled grammars with mprotect, without MAP_JIT. -->
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>

and the codesign line in the Makefile gained --entitlements bundle/cydonia.entitlements. Before this PR the bundle was signed with no entitlements at all.

Builds without grammars

Without its desktop feature, Cydonia's GUI crate builds for wasm32, which is how the browser demo was made. That build has no wasmtime and downloads nothing. Its language::Here is Ahead, which recognises Markdown and paints a fence only when the fence's spans were computed ahead of time and handed to it. The desktop build's Here is Installed, the catalog described above. Both implement one Grammars trait, so the file view, the diff preview and the fence highlighter call the same functions in both builds.

The code is in crates/gui/src/model/language/ and bezel's crates/syntax.