Generative AI Is Infrastructure Now: How to Architect GenAI Into Your Python Backend

The mental model that fixes most GenAI backend problems: the model is a dependency, not a feature. A single chat completion can take 1–10 seconds, cost real money per call, rate-limit you under load, and occasionally return a 500. You already know how to architect around a service like that — you just have to actually do it instead of await openai.chat(...) inline in a request handler.

Four properties drive every decision below:

1. Latency is high and front-loaded. Generation is slow, but the *first* token is comparatively quick — which is the entire reason streaming exists.
2. Cost is per-call. The cheapest call is the one you never make. That makes a cache the highest-ROI component in the whole stack.
3. Throughput is capped. Providers rate-limit by tokens/minute. Your backend needs backpressure so a traffic spike queues instead of melting.
4. It fails. Timeouts, 429s, and 503s are normal. The request path needs budgets and fallbacks, not optimism.

Here's the shape I'd reach for on a Python backend. The request hits an async FastAPI edge, checks a semantic cache, and only calls the model on a miss — streaming short answers straight back and pushing long jobs onto a queue:

                  ┌──────────────────────────────────────────────┐
  client ───────▶  │  FastAPI edge (async)                        │
  (SSE / WS)       │  • auth + rate-limit + input validation      │
       ▲           │  • per-request timeout budget                 │
       │           └───────────────┬──────────────────────────────┘
       │                           │  1. embed prompt  (~0.2 ms)
       │                           ▼
       │           ┌──────────────────────────────────────────────┐
       │  cache    │  Semantic cache  (pgvector / Redis)          │
       │  HIT      │  exact hash  ─▶ HIT ─▶ return                 │
       └───────────┤  ANN cosine  ─▶ sim ≥ threshold ─▶ HIT       │
                   └───────────────┬──────────────────────────────┘
                                   │  MISS
                     ┌─────────────┴──────────────┐
                     ▼ short job                   ▼ long job
        ┌──────────────────────────┐  ┌───────────────────────────┐
        │ stream from LLM          │  │ enqueue (Celery / RQ / arq)│
        │ token-by-token via SSE   │  │ 202 + job_id, poll/webhook │
        └───────────┬──────────────┘  └───────────────────────────┘
                     │ write-through
                     ▼
              back into cache

Two rules make this architecture pay off. Check the cache before you spend a token, and never block an HTTP worker on a multi-second generation — either stream it (so the connection is doing useful work the whole time) or hand it to a queue. The next two sections are those two rules, measured.

The single highest-impact routing change is streaming. A blocking endpoint runs the whole generation and then returns one JSON body — the user stares at a spinner for the full duration. A streaming endpoint forwards each token the instant it's produced. I built both against a fake "LLM" that emits 40 tokens at 30 ms each (~1.2 s to fully generate) and measured time-to-first-byte against a real uvicorn server:

import asyncio
from fastapi import FastAPI
from fastapi.responses import StreamingResponse, JSONResponse

app = FastAPI()

async def fake_llm():                 # stands in for the provider stream
    for i in range(40):
        await asyncio.sleep(0.03)
        yield f"token{i} "

@app.get("/blocking")
async def blocking():
    text = "".join([t async for t in fake_llm()])   # wait for ALL of it
    return JSONResponse({"text": text})

@app.get("/stream")
async def stream():
    async def gen():
        async for tok in fake_llm():
            yield f"data: {tok}\n\n"                 # SSE frame per token
    return StreamingResponse(gen(), media_type="text/event-stream")

/blocking   time-to-first-byte:  1526 ms    full: 1538 ms
/stream     time-to-first-byte:   283 ms    full: 1553 ms

Same total work — both finish in ~1.5 s — but the streaming route puts the first token on screen 5.4× sooner (283 ms vs 1526 ms). That gap *is* the perceived-performance difference between a snappy assistant and a dead spinner, and it costs you one StreamingResponse. Note the honest part: streaming doesn't make generation faster, it makes the wait *visible and useful*. Total latency is unchanged.

Streaming solves *short* generations. For anything that runs tens of seconds — long documents, agentic chains, batch jobs — don't hold the connection at all. Return 202 Accepted with a job id and do the work on a queue:

from fastapi import FastAPI, BackgroundTasks
app = FastAPI()

@app.post("/summarize", status_code=202)
async def summarize(doc: dict, bg: BackgroundTasks):
    job_id = new_job_id()
    # BackgroundTasks is fine for fire-and-forget; use Celery/RQ/arq
    # once you need retries, visibility, and a separate worker pool.
    bg.add_task(run_long_generation, job_id, doc)
    return {"job_id": job_id, "poll": f"/jobs/{job_id}"}

Caching is where the money is. Two users rarely type the *exact* same prompt, so a plain key-value cache on the prompt string barely hits. A semantic cache embeds the prompt into a vector and treats "close enough" as a hit — so paraphrases of a question you've already answered come back instantly, for free. The schema has two paths: an exact-hash fast path, and an approximate-nearest-neighbor (ANN) semantic path. Here's a real pgvector schema:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE llm_cache (
    id          BIGSERIAL   PRIMARY KEY,
    prompt      TEXT        NOT NULL,
    prompt_hash BYTEA       NOT NULL,          -- sha256, exact-match fast path
    embedding   VECTOR(256) NOT NULL,          -- 256 = model2vec potion-base-8M
    response    TEXT        NOT NULL,
    model       TEXT        NOT NULL,          -- never serve gpt-4o's answer for gpt-4o-mini
    hit_count   INT         NOT NULL DEFAULT 0,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at  TIMESTAMPTZ NOT NULL           -- TTL: stale answers must expire
);

-- Exact repeats skip the vector math entirely.
CREATE UNIQUE INDEX ON llm_cache (prompt_hash, model);

-- Approximate nearest-neighbour for the semantic path.
CREATE INDEX ON llm_cache USING hnsw (embedding vector_cosine_ops);

-- Cheap sweeps of expired rows.
CREATE INDEX ON llm_cache (expires_at);

The lookup is one query — cosine distance (<=>), nearest row, filtered to the same model and unexpired — and your code serves it only if the similarity clears your threshold:

SELECT response, 1 - (embedding <=> $1) AS similarity
FROM   llm_cache
WHERE  model = $2 AND expires_at > now()
ORDER  BY embedding <=> $1          -- cosine distance, ascending = closest first
LIMIT  1;
-- application: return cached response only if similarity >= 0.85

Does semantic caching actually work? I built the cache in pure Python with model2vec (a numpy-only static-embedding library — no PyTorch, ~0.2 ms per embed, so it's cheap enough to run inline on every request), seeded it with one answer, and threw paraphrases at it. Real results, threshold 0.85:

query                                        kind                 cosine  result
-------------------------------------------------------------------------------------
How do I reset my password?                  exact repeat          1.000  HIT  [0.2 ms]
password reset steps                         keyword paraphrase    0.896  HIT  [0.1 ms]
I forgot my password, how can I change it?   paraphrase            0.793  MISS [0.2 ms]
How can I recover my account login?          loose paraphrase      0.566  MISS [0.1 ms]
What's the weather in Tokyo today?           unrelated             0.096  MISS [0.1 ms]

The win is obvious: a hit returns in ~0.15 ms instead of the ~1,500 ms (and the API charge) of a real call — a cache lookup roughly 10,000× cheaper than the thing it replaces, and the unrelated Tokyo-weather query correctly misses at 0.096. But look at the honest failure: "I forgot my password, how can I change it?" — a paraphrase any human would call identical — scored 0.793 and missed.

Two schema decisions that aren't optional. Key the cache by model (the model column and the composite index) — an answer generated by a big model must never be served for a request that asked for a cheaper one. And give every row a TTL (expires_at): a cached answer to "what's our latest pricing?" that never expires is a bug, not a cache. For high-QPS caches, Redis with a vector index does the same job in memory; use pgvector when you want the cache to be durable and queryable alongside your other data.

Each layer targets a different one of the four properties from the top. Here's what each is actually worth, from the measurements above:

The order to build them in is the order of ROI: cache first (it removes calls, the only thing that saves real money), stream second (biggest UX gain for the least code), queue third (once any single job outgrows a request), and wrap the whole request path in timeout budgets so one slow upstream call can't take down your worker pool. None of this is exotic — it's the same reliability engineering you'd apply to any slow paid dependency. GenAI just finally has enough gravity to deserve it.