Methodology
Every number Fealty publishes traces to a SEC filing. Here is how the instrument is measured — and, honestly, what it doesn’t yet publish.
Extraction eval gate
The extractor is a single pinned model (temperature 0, strict schema, quote-relocation check). A wrong number on a named company is worse than a missed one, so precision dominates. Changing the model or prompt bumps the version and re-runs this gate; CI blocks a regression.
Extractor changelog
The production extractor is a local, fine-tuned open model. An earlier cloud model is retained only as a distillation teacher and eval reference — never a shipping path. Changing the production pin bumps its version and re-runs the gate.
Deterministic gates
After the model, deterministic gates run on every candidate: a repair for ± ranges, and quarantines for a value not in its quote, a redundant growth-rate, or an out-of-scope comparable-sales KPI. A gated candidate is never published — precision is a publishing threshold, and staying silent beats a wrong number on a named company.
Reproducibility
The exact model, prompt, and rules are pinned and versioned so any result is reproducible — but the instrument is judged by the eval gate above (precision/recall against hand-labeled filings), not by which model produced a number. Change the model or a threshold and the version bumps and the gate re-runs; resolutions regenerate under the new resolver version with old rows retained (bitemporal).
Coverage
A grade publishes only above a minimum resolved-promise coverage — below it we show the honest “not enough resolved promises” state, never a fabricated letter. 195 companies graded across 15645 resolved promises.
Guidance basis drop-rate
53.7% of gradeable guidance pairs were dropped for basis mismatch (3163 of 5894).
XBRL actuals are GAAP. A guidance figure is graded against a same-basis actual or DROPPED and counted here — we never grade a non-GAAP-intent number against a GAAP actual. Guidance that cannot be compared for another reason — another measure, a part of the business, or more than one figure given on the same day — is shown as “not judged” and is not counted in this rate. Publishing the drop-rate is the point: a hidden drop is a hidden bias.
Not yet published (in progress)
- · Universe inclusion rate, including delisted/acquired names since 2015
- · Coverage histogram and extractor P/R by stratum (era, form type, size)
- · Resolving percent-growth guidance against the reported prior-year base
These publish once the full backfill runs. We list what’s missing rather than imply completeness — that gap is the difference between an instrument and AI slop.