Corpus

What the system is holding

Counts, chunk layouts and resource use for the SEC 10-K corpus. Source filings stay at the SEC; this is the canonical text, its structure, and the chunk boundaries derived from it.

Production: structural.max256.hybrid.k10.b4000

Filings

250

FY2024–2025

Companies

250

Tokens

10.4M

gte-small tokenizer

Tree nodes

158.1K

132.1K text blocks

Sentences

341.3K

Source size

1.0 GB

at sec.gov, not stored

Chunk layouts

Every chunking configuration is stored as boundaries over the same canonical text. Redundancy is tokens embedded divided by corpus tokens: overlap and duplicated context push it above 1.

Strategy	Configuration	Documents	Chunks	Tokens embedded	Redundancy	Mean chunk size
Structure-aware	structural.max256 max 256	250	58,926	10.4M	1.00×	177

Embedding sets

An embedding set is one chunking configuration, embedded with one model and indexed for keyword search.

Configuration	Embedder	Status	Retention	Units	Tokens embedded	Index size
structural.max256	gte-small-onnx-q	ready	production	58,926	0	—

Resource usage

Four different costs, kept apart on purpose: they have different units and different owners, so they are never added together.

Bandwidth

bytes downloaded · not billed

211 MB

huggingface:gte-small0 B0 req

sec:filing211 MB0 req

sec:total469 KB0 req

Compressed bytes on the wire. Source filings stay at the SEC and are not stored. SEC fair-access limit: 10 requests/second.

Local embedding compute

tokens and seconds · no per-token charge

13.9M tokens

embedder/gte-small-onnx-q@1.0.013.9M tok11,608 tok/s

Measured throughput11,608 tok/s1,195 s

The open-source gte-small model runs on our own machine, so the cost is machine time, not an API bill.

Storage

bytes in the database · free-plan limit

246 MB

units144 MB

nodes65.1 MB

canonical_documents22.8 MB

lexicon9.2 MB

financial_facts1.0 MB

Tables, TOAST and indexes of the latest snapshot, against the 500 MB free plan (shared with other apps in the project).

Paid generation

tokens and dollars · billed per token

Nothing recorded yet.

Claude calls that write and grade answers. The only category with a dollar cost.

Ingestion throughput

Per stage, from the first filing: how long each step takes and how much data moves through it.

Stage	Events	Failed	Median	p95	Total time	Bytes in	Bytes out
fetch	822	0	266 ms	954 ms	5.2 min	229 MB	3.4 GB
parse	822	77	105 ms	345 ms	2.0 min	3.4 GB	—
normalize	745	0	121 ms	265 ms	1.7 min	—	159 MB
load	745	0	854 ms	1.1 s	10.2 min	—	—