From a 10-K filing to an answer

1. Ingest. 10-Ks from SEC EDGAR, parsed to canonical text with Items 1, 1A, 7 and 7A kept, every heading, paragraph, table and sentence boundary recorded.
2. Chunk. Structure-aware chunks: never cross a section, pack whole paragraphs, split long ones on sentences, at most 256 tokens counted by the embedding model's own tokenizer.
3. Embed. gte-small (384 dimensions) offline in a GitHub Actions job; the same model runs in a Supabase Edge Function for questions (vectors agree at 0.998 cosine).
4. Store. Postgres with pgvector (half-precision vectors) plus a full-text index, on Supabase's free tier.
5. Retrieve. Exact vector search over every passage (~0.1 s at this size), all-words keyword search, or both fused by reciprocal rank with keywords weighted 1.5×.
6. Generate. The top passages, optionally with their neighbours or section, within a token budget, go to Claude with an instruction to answer only from them and cite each claim.

Questions that need every company side by side (“who has the best margin?”) are answered from the companies' reported XBRL figures instead, because no passage of text contains that comparison.

A language model can only cite what retrieval hands it. If the right passage is not retrieved, the best model in the world cannot answer, and it may answer anyway from what it was given. So retrieval quality sets the ceiling for answer quality.

Measured here: in every configuration the right company's filing was in the top five for every question. The misses were never “wrong company”; they were “right filing, wrong passage”. That is precisely the part chunking controls: what a passage is, and therefore what can be matched and handed over.

• Fixed-size windows cut every N tokens regardless of content. Simple and predictable, but 96% of their cuts land mid-sentence and at 128 tokens 16% of answers were split across two passages.
• Structure-aware follows the document: sections, then paragraphs, then sentences. Mid-sentence cuts fall to 1% and every answer stayed in one passage.
• Also built, not yet benchmarked: recursive splitting, sentence packing, semantic breakpoints (cut where meaning shifts) and hierarchical units (retrieve small, expand to the parent section).

Size is a separate axis. Smaller chunks give a sharper match and cost less to read; larger chunks keep more context but their vectors average several topics and the match blurs.

Best overall: structure-aware 256, answer in the top five 48% of the time while reading 895 tokens; the best fixed-size setup needed 2,400 tokens for the same rate.

Structure-aware beat fixed-size by 7 points on average at equal size, most at small sizes (48% against 33% at 128 tokens). With the reading budget held at 2,000 tokens, structure-aware 256 found 60%, the most of any setup.

Lesson: for fact look-ups, cut small and along the document's structure. Bigger chunks only win when you let them send much more text.

These answers need two to four consecutive sentences, and the bar is higher: the whole answer has to reach the model. Best from the top five passages alone: structure-aware 256 (54% of answers complete). Adding each result's neighbouring passages (“find small, read big”) raised the best to 58% with structure-aware 384, at 3,287 tokens read.

Lesson: structure beats size here too. Structure-aware 256 got the whole answer to the model 54% of the time reading 828 tokens; the best fixed-size setup (fixed-size 480) managed 42% reading 2,400. Adding neighbouring passages mostly repairs blind cuts: it lifted fixed-size chunks by 12 points on average but structure-aware ones by only 3 points, at roughly three times the text. So the same setting serves both kinds of question; widen it or add neighbours only when answers routinely run past a paragraph.

At the recommended chunking, keyword search alone found 40% of single-sentence answers, meaning alone 44%, and both fused 48%. Keywords nail exact names and figures; meaning catches paraphrase.

In live use, “match any word” keyword search pulled in generic cost and margin text and was slow (about 15,000 candidate passages to score per question). Requiring every word made it precise and took it from about 3 seconds to a few milliseconds. The app now searches by meaning by default, with words and both one click away, and the Compare search methods tab shows all three side by side.

• At ~60,000 passages, exact vector search (~0.1 s) beat an approximate index: no accuracy loss, and the index's filtered scan was slower.
• Correct spelling before embedding: a typo (“tarrifs”) changed the question's vector enough to return unrelated passages.
• Drop fragments: heading-only passages (under 16 tokens) match many questions and answer none.
• Measure against known answers, mechanically: gold sentences located in the stored text, so “found” needs no AI judge.
• Cost: everything runs on free tiers; only Claude's written answers are billed, about 1–5 cents each.

• Report nDCG@10 alongside recall, so results compare with BEIR and MTEB.
• Run the FinanceBench open sample on the companies in this corpus.
• Score Claude's answers for faithfulness to the passages (RAGAS-style).
• Benchmark overlap, semantic and hierarchical chunking, and add a re-ranker after retrieval.
• Grow the question set: 48 + 26 questions means one question moves a score by 2–4 points.