Thesis

Most AI products fail at the boundary between model output and system behavior. The model may be correct, but the service still falls apart when retries, queueing, or cold starts accumulate.

The hidden cost is usually in the recovery loop.1

Core Insight

The interesting unit of analysis is not just the model step. It is the full execution path from request ingress to model output, post-processing, logging, and recovery.

A simple budget

For a request that enters a cloud AI service, the end-to-end cost can be simplified as:

Ctotal=Cnetwork+Cqueue+Ccompute+CobservabilityC_{total} = C_{network} + C_{queue} + C_{compute} + C_{observability}

If the queue dominates, scaling the model does not help much. If observability is weak, the system may be fast but still impossible to debug.

Design pattern

package main


type RetryPolicy struct {
    Budget time.Duration
    MaxAttempts int
}

func shouldRetry(attempt int, elapsed time.Duration, policy RetryPolicy) bool {
    return attempt < policy.MaxAttempts && elapsed < policy.Budget
}

What I want to prove next

I want to quantify how often latency spikes come from infrastructure rather than model inference. If the infrastructure term is stable, then most performance work should move up one layer into request shaping and scheduling.

Footnotes

  1. The hardest failures are often the ones that appear after the first successful response, when the system has already paid the cost of retries, checkpoints, and recovery. ↩