# Tree-sitter grammars on demand, in wasmtime
> Cydonia ships no syntax grammars in its binary. It carries a catalog of 33, downloads one when it is asked for, checks it against a pinned digest, and runs it in wasmtime. How that came together in one pull request, and why the app bundle had to be signed differently for it.
Cydonia opens files in a pane and highlights them, and colours fenced code in articles and cards. The highlighting is tree-sitter. The grammars are not in the binary: a grammar is a WebAssembly module that Cydonia downloads the first time it is wanted, verifies against a digest it was built with, caches, and loads into wasmtime. This is how that works, and the three shapes it went through on the way, all inside [PR #27](https://github.com/crabtalk/cydonia/pull/27).
## Naming a language is not painting it
The first commit, `feat(file): syntax highlighting for open files`, started from a distinction that every later version kept. A file's name can tell you its language even when the build has no grammar for it. The first `language.rs` put it like this:
> the difference between "that is YAML and there is no grammar here" and "that is a file extension nobody has ever written a grammar for", and the only reason an app can offer to fetch one rather than nag about files it could never help with.
So a lookup answers one of three things:
```rust
pub enum Language {
/// An installed grammar, under the name `syntax` knows it by.
Ready(&'static str),
/// Markdown, which bezel highlights without a tree-sitter grammar.
Markdown,
/// A known language whose grammar is not installed.
Missing(&'static str),
}
```
In that first commit the table of names was a hand-kept `KNOWN` array in Cydonia, eight grammars were compiled in (bash, go, json, python, rust, toml, tsx, typescript), and everything else in the table was `Missing`. A file in a `Missing` language opened as plain text.
## The registry moves into bezel
The next commits moved the table out of the app. [bezel](https://github.com/crabtalk/bezel)'s `bezel-syntax` crate now holds `syntax::registry`: one entry per language, with its fence aliases, its file names and extensions, and an optional grammar.
```rust
pub struct Entry {
pub name: &'static str,
pub aliases: &'static [&'static str],
pub files: &'static [&'static str],
/// `None` names a language this build cannot paint.
pub lang: Option<&'static Lang>,
}
```
The registry is seeded with names only. `bezel-syntax` links no grammar of its own; grammars arrive through `registry::register`, from a provider. Registering an entry with the same name replaces the old one, so a provider can turn a `Named` language into a `Ready` one at runtime, and the next lookup sees it. Path matching is the same rule the first table had: a whole file name beats an extension (`Dockerfile` is not a `.file`), and between extensions the longest match wins.
The first provider was `bezel-syntax-std`, which links seven tree-sitter grammar crates natively, each behind a cargo feature. Cydonia called its `install` once and painted what it registered. That version lasted about three hours.
## A catalog instead of a crate
The commit that landed, `feat(language): install wasm grammars from a bundled catalog`, dropped `bezel-syntax-std` from the app and turned on `bezel-syntax`'s `wasm` feature, which is tree-sitter's `wasm` feature underneath.
What ships in the binary is `catalog.json`, compiled in with `include_str!`. It has one row per grammar: the registry name, the symbol the module exports, aliases, file names, and for the module and each highlight query a path, a byte size and a SHA-256.
```json
{"name": "c", "symbol": "c", "aliases": ["c"], "files": ["c", "h"],
"wasm": {"path": "c/tree-sitter-c.wasm", "size": 625921, "sha256": "7b80a640…"},
"queries": [{"path": "c/highlights.scm", "size": 1432, "sha256": "3378d854…"}]}
```
The files come from the `tree-sitter-wasm@2.0.1` npm package, fetched through jsDelivr, and the sizes and digests are taken from that package's tarball. The catalog has 33 languages today. The smallest module is JSON at 6 KB; the largest is C# at 5.3 MB. All 33, modules and queries together, come to about 36.5 MB, none of which is in the app until somebody opens a file that needs it.
Some rows are more than one file. TypeScript and TSX carry the JavaScript highlight query before their own, C++ carries C's, and Vue carries HTML's. The symbol can differ from the name: C# is registered as `csharp` and exports `c_sharp`. Haskell is in bezel's registry and not in the catalog, because the package's highlight query does not compile against its grammar. Injection and locals queries are not installed.
## Install, verify, register
An install runs on its own thread and moves through a status the views read:
| Status | Meaning |
| --- | --- |
| `Missing` | Known, not cached |
| `Downloading { received, total }` | Fetching the module and its queries |
| `Checking` | Every file is fetched and matches its digest; the grammar is being loaded |
| `Ready` | Registered; the next highlight uses it |
| `Failed(String)` | Stopped, with the reason |
Each download asks for the file and reads at most one byte more than the catalog says it is, then checks the length and the SHA-256. A file that is short, long or different fails the install with `Incomplete grammar asset` or `Grammar checksum mismatch`.
Before anything is saved, the grammar is run once. `validate` makes a `WasmStore`, loads the module under its symbol, compiles the joined highlight query against it, and parses an empty string:
```rust
let mut store = syntax::tree_sitter::WasmStore::new(engine()?)?;
let language = store.load_language(&spec.symbol, wasm)?;
syntax::tree_sitter::Query::new(&language, query).context("Invalid highlight query")?;
let mut parser = syntax::tree_sitter::Parser::new();
parser.set_wasm_store(store)?;
parser.set_language(&language)?;
ensure!(parser.parse("", None).is_some(), "Grammar failed to parse");
```
Only then is the bundle written to disk and the grammar registered. A grammar that downloads cleanly but does not load, or whose query does not compile, is never published to the registry.
The cache is one JSON file per grammar under `~/.local/share/cydonia/grammars/2.0.1` (or `$XDG_DATA_HOME`), holding the module as base64 and the queries as text. It is not trusted either. On first use of anything in `language`, Cydonia walks the catalog, reads each cached file (refusing one over 64 MiB), and checks every piece against the catalog's digests again before registering it. A cache file that has been edited or truncated comes back as `Failed`, not as a grammar.
Two views asking for the same grammar share one download: `begin_install` refuses to start a second while one is active or done. A failed install can be started again.
## Who asks for a grammar
There are two ways in, and they behave differently.
**A file.** When a file's language is `Missing` and the catalog has it, the file view shows a notice row: _Install YAML highlighting?_, with _Not now_ and _Install_. While it runs the row shows kilobytes received against the total, then _Checking_, and on failure the error and _Retry_. Nothing is downloaded until _Install_ is pressed.
**A fence.** Since [PR #42](https://github.com/crabtalk/cydonia/pull/42), a Markdown fence tagged with a language is the request. The article and card editors collect the fence labels when a document opens and again on every edit, and call `ensure` with them. For each label the catalog resolves (aliases included, so `sh` and `zsh` both mean bash), a grammar that is `Missing` starts installing. A grammar that `Failed` is left alone, so a bad download is not retried on every keystroke; retrying is the file view's _Retry_.
While any download runs, a single poll loop refreshes the windows every 150 ms, so a fence colours itself on the first frame after its grammar lands. Nothing needs re-registering for that: `highlight` resolves through the registry on every call.
The fence picker in the editor lists every language in the catalog, cached or not. Choosing one that is not cached is how a fence asks for it.
## One engine, one store per parser
The app owns the wasmtime engine. Cydonia builds it once with a default `Config` and hands a clone to `bezel-syntax` through `syntax::set_engine`. Until an engine is set, a `Grammar::Wasm` cannot load, and `highlight` returns nothing.
`bezel-syntax` re-exports `tree_sitter::wasmtime`, and that is the only handle that works: wasmtime comes in through tree-sitter, and a second wasmtime in the dependency graph would be a second, unrelated `Engine` type.
A wasm grammar's `Language` belongs to the `WasmStore` that loaded it, because tree-sitter dispatches every lex and external-scanner call through that store. So `bezel-syntax` cannot hold a compiled grammar on the language itself. A `Lang` holds the module as bytes (`Grammar::Wasm(Arc<[u8]>)`), and each thread has its own `Session`: a highlighter, a parser with a store attached, and the queries compiled against it. A language is instantiated into that store the first time the thread highlights it, and its compiled configuration is kept, because a configuration costs milliseconds to compile.
## Signing for executable memory
The part outside the code is the bundle. On macOS, Cydonia is signed with the hardened runtime, and wasmtime compiles each grammar module to native code at runtime. The hardened runtime refuses that without an entitlement.
This changed within the same PR. The `bezel-syntax-std` commit added `bundle/cydonia.entitlements` with `com.apple.security.cs.allow-jit`, the narrower entitlement, which covers memory mapped with `MAP_JIT`. The catalog commit replaced it:
```xml
com.apple.security.cs.allow-unsigned-executable-memory
```
and the `codesign` line in the Makefile gained `--entitlements bundle/cydonia.entitlements`. Before this PR the bundle was signed with no entitlements at all.
## Builds without grammars
Without its `desktop` feature, Cydonia's GUI crate builds for wasm32, which is how the browser demo was made. That build has no wasmtime and downloads nothing. Its `language::Here` is `Ahead`, which recognises Markdown and paints a fence only when the fence's spans were computed ahead of time and handed to it. The desktop build's `Here` is `Installed`, the catalog described above. Both implement one `Grammars` trait, so the file view, the diff preview and the fence highlighter call the same functions in both builds.
The code is in [`crates/gui/src/model/language/`](https://github.com/crabtalk/cydonia/tree/main/crates/gui/src/model/language) and bezel's [`crates/syntax`](https://github.com/crabtalk/bezel/tree/main/crates/syntax).