Engram: Technical Deep-Dive

Engram adds persistent state infrastructure to SGLang, the high-performance LLM serving engine used across 400,000+ GPUs worldwide. It targets Mamba and Mamba2 hybrid models specifically, turning their hidden state from disposable inference overhead into a durable memory asset.

System design

Architecture

┌─────────────────────────────────────────────────┐
│                 OpenAI-Compatible API            │
├─────────────────────────────────────────────────┤
│              SGLang Serving Engine               │
│    (scheduler, batching, tensor parallelism)     │
├──────────────┬──────────────────────────────────┤
│  Engram      │  Snapshot Manager                 │
│  Extensions  │  ├─ save / restore / list / delete│
│              │  ├─ Conversation Tracker           │
│              │  └─ Snapshot Hooks & Policies      │
│              ├──────────────────────────────────┤
│              │  Tier Manager                     │
│              │  ├─ VRAM  (hot, ~2ms restore)     │
│              │  ├─ RAM   (warm, ~10ms restore)   │
│              │  └─ Disk  (cold, ~50ms restore)   │
│              ├──────────────────────────────────┤
│              │  Mamba State Extensions            │
│              │  ├─ MambaRadixCache (dual LRU)    │
│              │  ├─ HybridReqToTokenPool          │
│              │  └─ Mamba2Metadata                │
├──────────────┴──────────────────────────────────┤
│              Mamba / Mamba2 Models                │
│    (pure SSM, dense hybrid, MoE hybrid)          │
└─────────────────────────────────────────────────┘

Components

Core components

MambaRadixCache

A dual-LRU cache system that manages Mamba pool slots with automatic eviction when memory fills up. Supports copy-on-write semantics and tombstone nodes for safe concurrent access.

Snapshot Tiering (WARM + COLD)

WARM tier (host RAM): Auto-snapshot captures full final state via pre-free hook before memory release. COLD tier (disk): safetensors + JSON metadata per turn, supports startup preload.

Stateful Generate Pipeline

Modified scheduler creates synthetic requests for continued conversations, bypassing normal tokenization.

Per-Request Correlation

Futures-based system ensuring concurrent requests don't get mixed responses.

HybridReqToTokenPool

Unified pool managing both attention KV slots and Mamba SSM state slots per request.

Validation

Benchmark results

Validated on H200 across the full test suite (77/82 pass, 0 regressions):

Metric	Result
Token reduction	93.8% average across 271 requests, zero failures
Snapshot restore	~2 ms warm restore (vs. 6.5 s full prefill at 128K context)
Speedup at 128K	~3,250× faster than full prefill
Snapshot size	Constant per model, regardless of context length
Memory stability	Zero leaks, stable VRAM under sustained load

Storage

Snapshot sizes by model

Model	Params	Snapshot	Notes
Nemotron-Cascade-2-30B	30B	~47 MB	Smallest — MoE routing, fewer Mamba layers
Granite 4.0-H-tiny	4B	~56 MB	Primary benchmark model
Granite 4.0-H-small	32B	~146 MB	36 Mamba layers, BF16
Nemotron-3-Super-120B	120B	~5.3 GB	88 layers, FP8, LatentMoE
Qwen3-Coder-Next	~75B	~23.7 GB	GLA recurrent state (not Mamba2)

All sizes constant regardless of conversation length.

Quality

Test program summary

11 phases, 5 model architectures, 150+ requests per model.

Phase	Description	Result
0	Environment verification	PASS (10/12)
1	Stateless inference baseline	PASS (7/7)
2	MambaPool unit tests	PASS (5/5)
3	MambaRadixCache gauntlet	PASS (16/16)
4	Live server, no_buffer strategy	PASS (5/5)
5	Mamba2Metadata integrity	PASS (5/5)
6	extra_buffer strategy	PARTIAL
7	Snapshot system E2E	PASS (6/6)
8	True stateful inference	PASS — 93.8% savings
9	Gauntlet stress tests	PASS — 271 req, 0 err
10	Cross-model scaling	PASS (5 models)

Integration

API extensions

Engram extends SGLang's API with snapshot management endpoints. All existing SGLang endpoints work unchanged.

Endpoint	Method	Description
/save_snapshot	POST	Save current conversation state
/restore_snapshot	POST	Restore a previously saved state
/list_snapshots	GET	List all saved snapshots
/delete_snapshot	DELETE	Remove a saved snapshot

Server flags

Flag	Description
--enable-snapshot-persistence	Enable the snapshot system
--snapshot-dir <path>	Directory for snapshot storage
--mamba-scheduler-strategy	Scheduler strategy (no_buffer, extra_buffer)

Related: Synapse

Synapse is a companion project under the Clarit.ai umbrella — an edge/mobile optimized inference engine based on llama.cpp. While Engram focuses on stateful serving for Mamba architectures on GPU infrastructure, Synapse targets constrained devices where power, bandwidth, and memory are limited. More details coming soon.

Explore the code

Engram is open source. Dive into the implementation, run the test suite, or contribute.

← Back to home

Architecture overview →

GitHub