Benchmarking pi-l1-cache: a 138µs in-memory cache layer for the pi coding agent
The pi coding agent lets extensions intercept the provider round-trip with before_provider_request and after_provider_response events. That is plenty of room for a fast in-memory cache — but only if the interception itself is cheaper than the round-trip it saves. pi-l1-cache (v1.2.0, installable as npm:pi-l1-cache) is a deliberately minimal take on this: one file, zero dependencies, an FNV-1a hash, a hard memory cap, TTLs and a 95%-CPU kill switch. We installed it from npm, loaded it against a real pi session, and benchmarked the published artifact rather than the README.
The headline number is honest and useful: every intercepted request costs about 138µs — but only if the request is byte-identical to a cached one does the provider round-trip (1–3s) get skipped entirely. That is the whole value proposition, and it comes with sharp edges worth knowing before you adopt it.
How it slots into pi
The cache sits between the agent loop and the provider:
pi → [L1: RAM Map] → [L2: Redis via LiteLLM] → Provider
~138µs ~150ms 1-3s
On before_provider_request the extension hashes model.id + JSON.stringify(messages) + JSON.stringify(parameters), looks up a Map, and returns a cached response object if the entry is fresh. On after_provider_response it stores the response (with a JSON-length-based size estimate) and runs eviction. A 10-minute setInterval sweeps expired entries, cleaned up on session_shutdown.
Key derivation is content-addressed: model, messages and parameters all feed the hash, so the cache only ever answers requests that are literally identical to one it has seen. It never guesses or generalises.
Configuration is environment-based rather than config-file based:
L1_CACHE_ENABLED=false # hard off
L1_CACHE_MAX_ENTRIES=500 # entry cap (default 200)
L1_CACHE_MAX_MB=50 # memory cap (default 20MB)
L1_CACHE_TTL=7200 # TTL seconds (default 3600)
Runtime control comes from a single slash command:
/l1-cache stats # entries, MB, hit rate, init CPU
/l1-cache clear # reset counters + map
/l1-cache enable # / disable
What we measured
We benchmarked the actual npm artifact (pi-l1-cache@1.2.0) under Node 22.23 using the package's own exported test hooks, then exercised the real interceptor path with a mocked ExtensionAPI exactly as pi boots extensions.
| Measurement | Result |
|---|---|
fastHash standalone (FNV-1a) | 670k ops/s — 1.49µs/op |