33. Latency Optimization

Why speed matters in AI apps

AI models are inherently slow compared to database queries. A complex reasoning task can take 5-10 seconds. In user interface terms, 10 seconds is an eternity. Users perceive anything over 1 second as a delay and anything over 10 seconds as "broken."

Since we can't change the speed of light or the fundamental inference speed of the model, we have to use engineering tricks to mask latency or reduce the work required.

The Physics of Latency

Latency comes from three places:

1. Network: Sending the prompt to Google's servers.
2. Prefill (Input Processing): The model reading your prompt. This is usually fast.
3. Decoding (Output Generation): The model generating the answer token-by-token. This is the slow part.

Optimization Strategies

In this chapter, we'll explore how to attack each source of latency:

• Prompt Engineering: Shorter prompts process faster.
• Streaming: Showing progress immediately.
• Parallelism: Doing work while the user is typing or while the model is thinking.
• Caching: Reusing answers to skip the model entirely.

Where to go next

• 33.1 Cutting prompt size without losing accuracy
• 33.2 Streaming UX patterns that feel instant
• 33.3 Parallelizing retrieval and preprocessing
• 33.4 Caching: what to cache and what not to
• 33.5 Warm starts and connection reuse