A timeout describes your patience, not provider acceptance
When the primary stops responding, the application may not know whether it rejected the request, accepted it and is still computing, or completed a response that was lost in transit. Starting a fallback in that uncertainty can purchase two answers while displaying one.
Instrument the primary and fallback independently. Store provider name, local attempt ID, provider request ID, connection result, first-byte time, cancellation attempt and result, final status, usage, and which answer reached the customer. Never reuse one generic 'AI request' log row for both calls.
Replay three distinct failure conditions
Inject a connection refusal before the primary accepts bytes, a slow first byte after acceptance, and a mid-stream interruption. Run enough controlled samples to distinguish a repeatable pattern; the brief's 50 requests can be divided across conditions with a no-fault control group.
Measure customer completion and p95 latency alongside accepted and billable calls. A fast fallback that doubles accepted work is not an unqualified success. A conservative trigger that prevents duplicates but leaves every customer waiting may also fail the service objective.
| Trigger | Duplicate-cost risk | Customer latency | Use only when |
|---|---|---|---|
| Connection failure | Low if no acceptance is proven | Low | transport evidence proves no upstream acceptance |
| First-byte timeout | Medium to high | Bounded | duplicate cost is budgeted or primary cancellation is proven |
| Confirmed rejection | Low | Depends on rejection time | response is authoritative and non-retryable locally |
| Mid-stream failure | High | High | partial-output policy and customer experience are defined |
Account for hidden client-library retries
Many SDKs retry selected transport and server errors. Record the configured retry count and inspect debug logs so one application attempt is not actually several primary requests before fallback starts. Disable automatic retries only in a controlled test, then decide which layer owns retry behavior.
Use a single end-to-end deadline. Independent 30-second primary and 30-second fallback timeouts can turn a promised 30-second experience into a minute. Allocate the deadline among connection, first byte, generation, cancellation, and fallback rather than stacking full budgets.
Write a failover rule with an uncertainty branch
The rule should state exactly which observable event permits fallback. When primary acceptance is unknown, choose among waiting, asking the customer to retry, or starting fallback with the possible duplicate cost explicitly recorded. Do not relabel uncertainty as a confirmed failure.
Prevent repeated faults from becoming a permanent dual path
Add a circuit state for repeated primary faults so every request does not run the expensive dual path. Recovery probes should be low-volume and should not compete with customer traffic. Keep a manual rollback that routes to one provider while evidence is reviewed.
Approve only the measured trigger
The matrix is complete when every trigger has sample count, accepted primary work, cancellation success, fallback calls, duplicate billable calls, completion rate, and p95 latency. Choose the narrowest trigger that meets the documented customer objective and duplicate-cost ceiling.
After rollout, compare real provider usage by request pair for one week and revisit the decision if provider semantics or SDK versions change. Pro controls can limit supported WordPress exposure, but the failover contract still belongs to the integration and must be proven at both providers.
Use the dual-provider decision matrix from “Provider fallback can charge for the timeout and the successful answer” on a real first installation. Download AI Cost Circuit Breaker for free, begin in Monitoring, and move to enforcement only after the expected signals and rollback are verified.