Server-Sent Events (SSE) are fantastic for streaming AI responses. Everything streams beautifully on localhost, token by token. But a common, yet jarring, pitfall occurs when you deploy this to production behind a reverse proxy: the stream appears to die.

Instead of a fluid, real-time experience, the client hangs in silence for 10 or 15 seconds, only to dump the entire payload all at once.

The Buffering Impedance Mismatch

This behavior often masquerades as an application bug, but it's actually a fundamental conflict with how Nginx handles connections by default.

By design, Nginx intercepts upstream responses and buffers them in memory before transmitting them to the client. For standard REST traffic, this is a brilliant optimization—it protects your backend worker threads from being tied up by slow clients.

However, SSE relies on continuous, chunked data transmission over a long-lived TCP connection. When Nginx applies its default proxy_buffering to an SSE endpoint, it eagerly swallows all the individual chunks. It waits for the connection to close before releasing the buffer to the client, effectively turning your real-time stream into a standard, painfully slow HTTP request.

Two Ways to Fix It

You need to bypass this buffering for your streaming endpoints, but you absolutely must not disable buffering globally, or you'll severely degrade the performance of your standard API traffic.

1. The Application-Level Fix (Recommended)

The most robust architectural approach is to let your application control the proxy behavior. This decouples your Nginx configuration from your application's routing logic.

You can do this by having your backend explicitly emit the X-Accel-Buffering: no header alongside your SSE responses:

Content-Type: text/event-stream
Cache-Control: no-cache
X-Accel-Buffering: no

When Nginx sees this header, it automatically disables buffering for that specific response.

2. The Infrastructure-Level Fix

If you prefer to handle this at the infrastructure level, or if you can't easily modify the backend headers, you can isolate your streaming endpoints in your nginx.conf.

Create a specific location block targeting your streaming paths:

location /api/v1/generation/stream/ {
    proxy_pass http://upstream_server;
    
    # Neutralize Nginx buffering to allow real-time chunk transmission
    proxy_buffering off;
    proxy_cache off;
    
    # Enforce persistent connection protocols required for SSE
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    chunked_transfer_encoding on;
    
    # Extend read timeouts for long-running AI inference tasks
    proxy_read_timeout 300s;
}

Both methods achieve the same goal: forcefully bypassing the internal buffer so your upstream AI orchestrator can push individual tokens directly to the client socket in real-time. If your AI streams are stalling in production, your proxy's buffer is almost certainly the culprit.