skeg
benchmarks

0.5 release · the story so far

tq2 is the default, multi-tenant is the wedge

skeg 0.5 makes tq2 (TurboQuant 2-bit) the default compression tier: recall-neutral against the old int8 default and leaner at high dimensionality. This slice tells the full 0.5 story end to end: the tq2 promotion, the high-density multi-tenant wedge (per-tenant isolation that Qdrant has no clean answer for), survival in a memory-capped container, the cost gap at 50M vectors, the single-tenant scaling tiers, and a new control-plane feature: a tiering cold-start bulkhead that keeps one tenant's index reopen from stalling its neighbours on the same shard.

data provenance

Where the numbers come from. Same source, same generator, same ground truth for every engine in the comparison.

corpus

Simple English Wikipedia, passages ≥ 500 chars truncated to ~400 chars. Public dump preprocessed once and frozen for the bench.

embedder

Both mxbai-embed-large-v1 (1024d) and all-MiniLM-L6-v2 (384d) are used as orthogonal embedders for cross-check.

queries

Hold-out passages disjoint from corpus (mxbai 1024d, minilm 384d). Standard datasets GloVe-100-angular and MNIST-784 used for the cross-distribution checks.

ground truth

Top-100 nearest neighbours computed with exact brute-force cosine over float32 vectors. Computed once per scale, reused by every engine, frozen as a parquet next to the corpus.

notes

0.5 reference numbers from skeg-bench/RESULTS.md plus the tq2-vs-int8 default-tier gate across dimensionalities. Single-machine Apple Silicon (M1). RAM ratios are hardware-independent; absolute latencies and build times are not. Recall is against exact brute-force cosine ground truth; RAM via ps -o rss.

cross-engine · serve RAM at 100K (mxbai-1024)

Every engine at a reasonable default config on the same corpus. skeg-tq2 is the single point that is simultaneously leanest, most accurate, and fast. Bars are serve RAM (MB) · lower is better; hover for recall and latency.

cross-engine · RAM vs recall frontier

Each engine is one operating point at 100K. Lower-right is better (less RAM, higher recall). skeg-tq2 sits alone in the bottom-right corner.

cross-engine · recall@10 at 100K

Same six engines, recall@10 against exact brute force (y zoomed to 0.90-1.00 so the gaps read). skeg-tq2 tops the chart at the least RAM.

all numbers · mxbai-1024 @ 100K (click headers to sort)
engine serve RAM MB recall@10 recall@100 p50 ms p95 ms
skeg-tq2 47 1.000 1.000 2.49 3.57
lancedb 198 0.998 0.991 59.26 98.14
milvus-lite 108 0.934 0.880 2.69 4.34
hnswlib 426 0.985 0.925 1.99 2.58
chroma 682 0.985 0.919 3.91 5.64
qdrant-f32 885 0.997 0.981 2.62 3.40

cross-engine · query latency at 100K

p50 and p95 query latency, same six engines. skeg-tq2 is competitive with the in-RAM HNSW engines (lower is better); LanceDB's IVF-PQ pays a large latency cost. Honest read: skeg is not the single fastest, it is the only one this lean and accurate at competitive latency.

co-residence · backend RAM under concurrent RAG

Backend RSS while a local 3B LLM is resident and generating and the store serves top-k retrievals, both as live read-write servers (mxbai-1024; the LLM RAM is constant and not counted). The line is RSS at-rest (store quiescent between turns, where skeg's decay timer returns the working set); the table adds sustained (continuous-load median) and max. recall@10 is unchanged by co-residence: skeg-tq2 ~1.0, qdrant ~0.99 (see the single-tenant charts).

all numbers · LLM co-resident RAG load (rw servers)
engine vectors at-rest MB sustained MB max MB
skeg-tq2 250K 96 138 142
skeg-tq2 500K 253 260 267
skeg-tq2 1000K 459 459 459
qdrant-f32 250K 938 1,204 1,322
qdrant-f32 500K 1,612 1,854 2,368
qdrant-f32 1000K 1,777 1,451 2,119

GloVe-100-angular @ 500K

A genuinely hard distribution. HNSW engines at default ef; raising it trades latency for recall but not the RAM gap.

all numbers · GloVe-100 @ 500K
engine serve RAM MB recall@10 recall@100 p50 ms
skeg-tq2 31 0.975 0.966 2.3
lancedb 946 0.909 0.729 14.4
qdrant-f32 611 0.899 0.823 2.0
hnswlib 344 0.838 0.709 0.5
chroma 616 0.818 0.696 2.6
milvus-lite 118 0.776 0.729 1.4

multi-tenant density · 5 tenants x 100K

Peak RAM for five tenants of 100K vectors each. Bars are peak RAM (MB) · lower is better; hover for recall and leak count.

shared index + per-tenant filter · apples-to-apples

Both engines run one index with a per-tenant filter, the configuration Qdrant is built around. skeg-tq2 still serves at 222 MB RSS vs 2213 MB, at higher recall, with 0 measured leaks.

all numbers · shared index + per-tenant filter
config peak RAM MB serve RSS MB recall leaks
skeg-tq2 shared+filter 1,471 222 0.999 0
qdrant shared+filter 4,131 2,213 0.941 0

tenant isolation · adversarial leak-fuzz

Query a tenant's index with another tenant's exact vector (its own nearest neighbour) while filtering for the victim, with mid-stream deletes and consolidation.

tenants8
vectors / tenant2,000
rounds200
scoped queries604
cross-tenant leaks0
verdictPASS

tight container · MNIST 60K under a 256 MB cap

The test others use to show Qdrant OOM-killed. Bar is peak RAM (MB); the dashed line is the 256 MB kernel cap · staying under it is the whole game.

MNIST recall / latency tiers

All skeg tiers hold recall@10 1.0 on raw pixels; Qdrant's int8 quantization collapses to 0.914.

all numbers · MNIST 60K tiers
config peak RAM MB recall@10 recall@100 p50 ms
skeg-tq2 156 1.000 1.000 2.07
qdrant-f32 423 1.000 0.999 2.33
qdrant-int8 460 0.914 0.955 2.05

cost · 50M x 1024-dim @ $4/GB-month

Resident RAM turned into a yearly bill at a typical managed-RAM price. 90% lower memory cost at matched recall. Bars are $/year · lower is better.

serve RSS vs corpus size · skeg stays flat

Steady serve RSS as the corpus grows 100K to 500K. skeg-tq2 stays flat (the working set is on SSD, only hot pages resident); both Qdrant configs climb roughly linearly into the GiB. This is the architecture, not tuning.

single-tenant scaling · peak RAM @ 500K

Peak RAM during build at 500K across skeg's tiers vs Qdrant. Peak is the stable, quotable number. Bars are peak RAM (MB) · lower is better.

scaling tiers numbers

Click headers to sort.

config peak RAM @500K MB recall@10 p50 ms p95 ms
skeg-int8 794 0.9995 4.98 17.17
skeg-tq4 998 0.9995 3.49 8.35
skeg-tq2 901 1.0000 4.70 12.42
skeg-tq1 1,021 1.0000 6.55 11.51
qdrant-f32 5,436 0.9875 3.02 4.99
qdrant-int8 5,051 0.9455 2.36 4.45

tq2 gate · serve RAM (skeg int8 vs tq2)

serve RAM per tier and dataset · lower is better.

tq2 gate · recall@10 holds

recall@10 for the same gate. The two tiers track each other across all three distributions · tq2 never drops below 0.999.

cold-start bulkhead · neighbour latency during a reopen

A 0.5 control-plane fix: an evicted index reopens off the shard thread (spawn_blocking), so a cold reopen no longer stalls other tenants on the same shard. During a 13.4 ms reopen of index A, what index B (a co-resident tenant) saw:

reopen mode neighbour ops served neighbour p50 µs neighbour p99 µs
async reopen (shipped) 209 59.5 144
sync reopen (the bug) 2 - 11,300

latency under load · noisy neighbour

A victim tenant's p95, idle and while a noisy tenant is hammered by 8 threads. skeg is ~2x lower idle and under load. Bars are p95 (ms) · lower is better.

throughput · 50K x 256-dim, tq2

Single-process QPS (GIL-free client processes), inline vs the worker pool. QPS is noisy (+/-15%) and strongly dimension-dependent; the takeaway is directional · one process scales to a per-process ceiling well above any single-shard folklore number, and the worker pool does not raise that ceiling here.

more