Why Your SSE Streams Break in Production Behind Nginx

An SSE response can stream token by token on localhost but arrive in batches behind Nginx. The client may appear idle for 10 or 15 seconds, then receive the accumulated response at once.

Check proxy buffering

Nginx buffers proxied responses by default. That can be useful for ordinary responses, but it can delay delivery of small SSE chunks. It does not necessarily wait for the upstream connection to close; flushing depends on the response and buffer behavior.

This makes proxy buffering a useful hypothesis when a stream behaves differently after deployment. The symptom alone does not establish the cause.

Disable buffering for the stream

One option is to send these headers from the application:

Content-Type: text/event-stream
Cache-Control: no-cache
X-Accel-Buffering: no

Nginx uses X-Accel-Buffering: no to disable buffering for that response, unless the configuration ignores the header. This lets the application identify its streaming responses without maintaining a separate list of paths in Nginx.

Alternatively, configure the streaming location directly:

location /api/v1/generation/stream/ {
    proxy_pass http://upstream_server;
    
    # Neutralize Nginx buffering to allow real-time chunk transmission
    proxy_buffering off;
    proxy_cache off;
    
    # Enforce persistent connection protocols required for SSE
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    chunked_transfer_encoding on;
    
    # Extend read timeouts for long-running AI inference tasks
    proxy_read_timeout 300s;
}

The buffering change here is proxy_buffering off. The example also disables caching, sets HTTP/1.1 upstream behavior, and allows 300 seconds between upstream reads. These settings should be checked against the deployment rather than treated as universal SSE requirements.

I would keep the change scoped to streaming responses and check whether events arrive incrementally through the deployed path. If they still arrive in batches, application flushing or another intermediary may need investigation.