Name the unit that actually expands
A release containing 240 changed documents does not justify pricing from the editor count or total page views. Embedding exposure is driven by the documents selected for indexing, chunks produced per document, input tokens per chunk, retry multiplier, and price for the chosen embedding model.
Ask the connector for an export of selected document IDs and content hashes before approving the run. A 'full sync' button may mean metadata discovery, content extraction, chunk creation, or paid embedding; those stages must be separated so a harmless scan is not confused with model usage.
Calculate the ceiling and the delta
Use the formula: input tokens = selected documents × mean chunks per document × mean tokens per chunk × retry multiplier. Estimated model cost then equals input tokens divided by the provider's billing unit, multiplied by the verified current rate. Keep the rate and retrieval timestamp beside the calculation.
For 18,000 documents, six chunks, and 700 tokens, a single full pass is 75.6 million input tokens. If only 240 documents changed, the same assumptions yield 1.008 million tokens. The ratio is 75:1 before retries, which is the decision-relevant fact even if the rate later changes.
| Input | Full corpus | Changed delta | Source |
|---|---|---|---|
| Documents selected | 18,000 | 240 | connector manifest |
| Chunks per document | 6 | 6 | chunking sample |
| Tokens per chunk | 700 | 700 | tokenizer sample |
| Tokens before retries | 75,600,000 | 1,008,000 | calculated |
| Retry multiplier | enter observed value | enter observed value | job logs |
Prove that the delta is trustworthy
A delta index is safe only if changed, added, and deleted documents are all represented. Compare content hashes, ensure deleted chunks are removed, and sample unchanged documents to verify they were not silently re-embedded. Include model and chunker version changes because either can intentionally invalidate the whole index.
Enforce the allowance before paid submission
Set a release allowance in tokens, not just documents. A marketing page with one chunk and a handbook with 200 chunks do not consume the same budget. Stop the job before provider submission when the manifest exceeds the approved token allowance.
Reserve full rebuilds for named invalidations
Authorize a full rebuild only for a documented reason such as an embedding-model migration, incompatible chunking change, corrupt index, or verified missing lineage. 'The connector recommended it' is not an acceptance criterion. Require an owner, maintenance window, cost ceiling, and rollback or parallel-index plan.
If a full run is necessary, process a measured sample first. Compare predicted versus actual chunks, tokens, failures, and duration; update the ceiling with the observed multiplier before releasing the remaining corpus. This prevents a small estimation error from scaling across 18,000 documents.
Retain the calculator with the release
Attach the completed table, document manifest hash, rate source, sample results, authorization, and final provider usage to the release record. The difference between forecast and final usage should be explained by measured retries or chunk variation, not left as a percentage without a cause.
A release passes when the selected corpus matches the approved scope, the token ceiling is enforced before submission, deletion handling is tested, and final usage reconciles to the manifest. Review current product pricing separately; the calculator remains valid because its quantities and rate source are explicit.
Use the embedding rebuild calculator from “A bulk publish triggered a full embeddings rebuild: calculate the real exposure” on a real first installation. Download AI Cost Circuit Breaker for free, begin in Monitoring, and move to enforcement only after the expected signals and rollback are verified.