SysPad › Lessons › AI and machine learning lessons › OpenAI API

How OpenAI API works

OpenAI LLM API - a synchronous inference dependency. Cost is driven by tokens (input + output) per request, and latency grows with output length. Prices are approximate per-1M-token rates.

To your architecture, OpenAI is just another external API over HTTPS, but one with unusual properties: it’s stateless (no memory between calls), it bills by the token, its context window caps how much you can send, and responses can stream. Treat it like any third-party dependency with latency, rate limits, and failures, and most "AI integration" problems become ordinary engineering.

What you will learn

Why is the API stateless, and where does conversation "memory" actually live?
What are tokens, and how do they drive both cost and the context-window limit?
Why stream responses, and what does latency look like?
How do you treat the API as a reliable dependency (rate limits, retries)?

Stateless: the model has no memory

Each API call is independent, the model remembers nothing from previous requests. To hold a conversation, your application resends the relevant history as part of the prompt every time. The "memory" is something you store and manage, not the API.

Tokens: the unit of cost and limits

Text is split into tokens (roughly word fragments). You’re billed per input token (your prompt) plus per output token (the reply). The context window is the maximum tokens a request can hold, prompt and reply together.

⚓ Chatbots, content generation, extraction/classification, and RAG over your own data.

Streaming and latency

LLM responses are generated token by token, so they can stream back as they’re produced. Streaming lets a UI show text immediately (better perceived speed) even though total generation takes time. Latency has two parts: time to the first token, then the per-token generation rate.

A dependency with rate limits

The API enforces rate limits (requests and tokens per minute). Exceed them and you get throttled (HTTP 429). Like any external dependency, it can also be slow or fail, so you design with retries and backoff, timeouts, and often an asynchronous queue so a spike doesn’t overwhelm limits or block users.

Flashcards

Does the OpenAI API remember previous messages on its own?: No, it’s stateless. Your app resends the conversation history in the prompt each call; the "memory" lives in your application.
What are you billed for, and what limits a single request?: Per input + output token; the context window caps total tokens (prompt + reply) in one request.
How do you keep long conversations/documents within limits and cost?: Trim/summarise history, chunk documents, and retrieve only relevant context (RAG) instead of sending everything.
Why stream LLM responses?: To show output as it’s generated, improving perceived responsiveness; total generation time still scales with output length.
How should you treat the model API in your architecture?: As a remote, fallible dependency: handle rate limits (429) with backoff, set timeouts, retry, cache, and often process asynchronously.

Sources

Open the SysPad canvas · Official OpenAI API documentation