/ai-cost-audit

AI cost audit

Build a cost model from observed usage and current primary-source prices. Treat repository intent, measured usage, invoices, and projections as different evidence classes.

Choose a mode

• Inventory: map model/media calls, routing, retries, and ownership.
• Economics: calculate cost per action, user, plan, and month.
• Simulation: compare pricing, volume, model, cache, batch, or abuse scenarios.
• Optimization: rank changes after measuring quality and operational risk.
• Reconciliation: explain the gap between bottom-up estimates and provider invoices.
• Verification: confirm that a completed change reduced spend without unacceptable quality or latency regressions.

Use the narrowest mode that answers the request. Combine modes only when the user asks for a full audit or the dependency is necessary.

Workflow

1. Inspect the repository, existing telemetry, billing exports, pricing configuration, and prior analyses before asking questions.
2. State the audit boundary: environments, date range, providers, features, currencies, taxes, credits, and whether non-model infrastructure is included.
3. Inventory direct and indirect calls: generation, reasoning, embeddings, reranking, tools, image/audio/video, moderation, retries, fallbacks, agents, queues, evaluations, and batches.
4. Prefer provider-metered tokens or media units. Keep measured values, code-derived estimates, generic estimates, and assumptions visibly separate. Use p50, p95, and worst-case where available.
5. Fetch current prices only from official provider sources when pricing affects the answer. Record URL, retrieval date, region/tier/currency, and special terms such as cached input, reasoning tokens, batch, storage, or minimum charges. Do not rely on bundled price tables or memory.
6. Model each cost path, including failed calls, retry amplification, tool loops, cache writes/reads, storage, egress, gateway fees, payment fees, free allowances, and shared fixed costs when relevant.
7. Reconcile the modeled total against invoices or billing dashboards. Quantify unexplained variance instead of forcing agreement.
8. Run normal, high-usage, abuse, and sensitivity scenarios. Do not apply universal margin or traffic thresholds without the product's business constraints.
9. Rank recommendations by expected savings range, evidence confidence, quality risk, latency effect, engineering effort, reversibility, and measurement plan.
10. Require an evaluation and canary before changing models, prompts, routing, or output limits. Verify spend, quality, latency, error rate, and user outcomes afterward.

Ask only for inputs that cannot be recovered from the scoped artifacts and materially change the result.

Safety and evidence rules

• Never invent usage, conversion, revenue, invoice, cache-hit, or quality inputs.
• Never expose secrets found in environment files or billing exports.
• Do not recommend a cheaper model solely from task labels; require representative evaluation data.
• Label projections as projections and show formulas with units.
• Treat prices and model availability as time-sensitive.

Load conditional references

• Read model inventory for discovery patterns, call-path fields, and routing analysis.
• Read unit economics for formulas, reconciliation, scenario design, and recommendation ranking.

These references provide specialist detail only. This file controls mode, interaction, safety, output, and verification.

Output contract

Return only sections supported by the selected mode:

• scope and evidence ledger;
• model/media call inventory with file paths and ownership;
• assumptions and data gaps;
• formulas with units and a scenario table;
• highest-cost paths and invoice reconciliation;
• prioritized changes with savings ranges and quality gates;
• verification results or an explicit measurement plan.

Cite every time-sensitive price with an official source and retrieval date.

Verify

• Inventory covers wrappers, retries, fallbacks, background jobs, and non-text operations.
• Totals preserve units, currencies, date ranges, and environment boundaries.
• Measured and estimated inputs are distinguishable.
• Bottom-up totals are reconciled to billed totals when billing is available.
• Recommendations include a quality gate, rollback path, and owner or next action.
• Completed optimizations are verified with before/after spend, quality, latency, and error data.

Model and media call inventory

Use this reference only after the main skill has selected an inventory, economics, reconciliation, optimization, or verification mode. Do not restart intake here.

Discovery strategy

Search with rg, then follow indirection rather than trusting model literals alone.

Look for:

• provider SDK constructors and request methods;
• gateways, proxy clients, model aliases, environment variables, and configuration files;
• orchestration frameworks, agent/tool loops, evaluators, and structured-output helpers;
• embeddings, rerankers, moderation, OCR, transcription, speech, and image/video operations;
• queues, cron jobs, workers, webhooks, migrations, notebooks, tests, and admin tools;
• retry, fallback, timeout, circuit-breaker, and provider-routing code;
• usage logging, billing webhooks, cost tags, budgets, and tracing spans.

Start broad, then inspect every returned call path. Provider method names change; do not rely on a frozen pattern list.

Inventory fields

Record one row per distinct priced path, not merely per source line:

Resolve aliases to actual runtime values by environment. Preserve uncertainty when configuration is injected externally.

Call-graph checks

For each feature, trace:

1. entrypoint and eligibility;
2. preprocessing and context construction;
3. primary request;
4. tool calls or recursive turns;
5. validation and repair requests;
6. retry and fallback behavior;
7. asynchronous follow-up work;
8. stored artifacts and downstream media/embedding jobs.

Count amplification from code and telemetry separately. A nominal “one model call” feature may create several billed operations.

Optimization candidates

Flag candidates only with supporting evidence:

• Remove: call does not affect a user or operating decision.
• Reduce: avoidable context, duplicate retrieval, excessive output, or repeated repair.
• Cache: stable repeated prefix or result with compatible provider semantics and privacy boundary.
• Batch: asynchronous work whose latency requirement matches the provider's current batch terms.
• Route: measurable task classes differ in required quality or latency.
• Consolidate: scattered literals or equivalent calls prevent governance.
• Observe: usage cannot be measured or tied to a feature.

Do not infer a safe model downgrade from labels such as “classification” or “summary.” Build a representative evaluation set and compare quality, latency, and failure cost.

Inventory quality checks

• Search results include indirect wrappers and non-text operations.
• Runtime configuration is resolved for every relevant environment.
• Retries, fallbacks, tools, and fan-out are represented.
• Frequency and usage identify evidence class and date range.
• Unused configuration is distinguished from unreachable or externally triggered code.
• Secrets and raw customer content are excluded from the report.

Unit economics and reconciliation

Use these formulas after the main skill establishes scope and evidence. Fetch all prices from current official provider sources; this reference intentionally contains no model prices or universal margin benchmarks.

Contents

• Evidence hierarchy
• Per-operation cost
• Product economics
• Cache analysis
• Batch and routing analysis
• Reconciliation
• Scenario design
• Recommendation record

Evidence hierarchy

Prefer, in order:

1. provider invoice/billing export;
2. provider-metered usage attached to requests;
3. application traces with token/media counts;
4. tokenizer or file-duration/dimension calculations;
5. code-derived limits and observed volume;
6. explicitly labeled generic assumptions.

Never blend classes without showing which inputs are estimated.

Per-operation cost

For a token-priced operation:

input_cost       = uncached_input_units × input_rate
cache_read_cost  = cached_input_units × cache_read_rate
cache_write_cost = cache_write_units × cache_write_rate
output_cost      = output_units × output_rate
reasoning_cost   = billed_reasoning_units × reasoning_rate
request_cost     = sum(components) + tool/media/minimum fees

Normalize rates and units before arithmetic. For media, use the provider's billed duration, resolution, characters, images, or other current unit.

Expected feature cost:

base_paths    = Σ(path_probability × path_cost)
retry_cost    = Σ(retry_probability × retry_path_cost)
fallback_cost = Σ(fallback_probability × fallback_path_cost)
feature_cost  = base_paths + retry_cost + fallback_cost + storage + egress + gateway

Report a distribution where request shapes vary materially.

Product economics

variable_cost_per_action = AI + non-AI variable infrastructure + payment-variable cost
contribution_per_action  = allocated_revenue - variable_cost_per_action
contribution_margin      = contribution_per_action / allocated_revenue

monthly_variable_cost = Σ(action_volume × expected_action_cost)
gross_profit          = recognized_revenue - cost_of_revenue
break_even_usage      = available_contribution / marginal_action_cost

State the accounting boundary. Gross margin, contribution margin, and cash spend answer different questions.

For free allowances:

expected_free_cost = signup_count × activation_rate × E[cost | activated free user]
payback_ratio      = contribution_per_payer / expected_cost_per_nonpayer

Do not invent activation, conversion, or usage distributions.

Cache analysis

Model provider-specific write/read rules, TTLs, minimum cacheable length, invalidation, and privacy boundaries from current documentation.

uncached_cost  = repeated_units × normal_input_rate
cached_cost    = write_cost + expected_reads × read_cost + uncached_remainder
cache_savings  = comparable_uncached_cost - cached_cost
break_even_reads = incremental_write_cost / savings_per_read

Use observed prefix stability and hit rate. Include latency and quality effects when context must be restructured.

Batch and routing analysis

For batch, verify current eligibility, completion window, cancellation behavior, quota, and discount. Compare with the workload's real deadline and retry policy.

For routing, calculate expected savings using observed task mix, then gate the proposal with representative evaluations. Include false-success cost, retry amplification, latency, and operational complexity.

Reconciliation

Reconcile by provider, account/project, model/operation, environment, and day where possible.

variance = billed_total - modeled_total
variance_pct = variance / billed_total

Investigate taxes, credits, tiers, minimums, storage, fine-tuning, deleted logs, untagged environments, retries, external tools, currency conversion, and billing-period boundaries. Leave a residual unexplained amount if evidence cannot resolve it.

Scenario design

Include only decision-relevant scenarios:

• measured baseline;
• high-usage or tail request shape;
• abuse/retry incident;
• price or volume sensitivity;
• proposed cache/batch/routing change;
• quality failure or fallback amplification.

Show formulas, input values, provenance, and a range rather than false precision.

Recommendation record

For each change, report:

• affected path and evidence;
• monthly savings range and confidence;
• engineering effort and owner;
• quality, latency, privacy, and reliability risk;
• evaluation/canary design;
• rollback trigger;
• post-change measurement window.