Jiwon Min Developer

A Complete Guide to Nginx Reverse Proxy: Practical Configurations for Caching, Load Balancing, and Zero-Downtime Deployment

Any experienced developer has likely used Nginx as a web server or a simple reverse proxy. However, it’s hard to say you’re fully leveraging Nginx’s potential with just a single proxy_pass directive. In a production environment where traffic is growing and service stability is crucial, you need to use Nginx more sophisticatedly to maximize performance, availability, and deployment efficiency.

This article goes beyond simple port forwarding to delve into advanced Nginx reverse proxy techniques for solving problems you might face in a real production environment. We will explore in detail, with practical configurations and code, everything from high-performance caching strategies that dramatically improve response times for repetitive requests, to load balancing and health checks that prevent a single server failure from bringing down the entire service, and even building a zero-downtime deployment (Blue-Green) architecture that completes deployments without users noticing.

A Complete Guide to Nginx Reverse Proxy: Practical Configurations for Caching, Load Balancing, and Zero-Downtime Deployment

© AI Generated Image

Background and Problem Definition

Placing Nginx as a reverse proxy in front of an application server (WAS) is standard practice in modern web architecture. However, in many cases, the configuration remains at a very basic level, like the one below.

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
    }
}

This configuration only serves to forward all requests to a single application server. This carries the following potential problems:

  1. Performance Bottleneck: Since all requests are passed directly to the application server, it must execute business logic and query the database every time, even for repetitive requests for the same content. This causes unnecessary load on the server and increases response times.
  2. Single Point of Failure (SPOF): If the application server at 127.0.0.1:8000 fails, the entire service goes down. Even with multiple servers, there’s no way to effectively distribute and manage them with this setup.
  3. Service Downtime During Deployment: When deploying a new version of the application, the service will inevitably be temporarily unavailable while the server is restarting.

This post provides a step-by-step guide to solving these problems and building a robust, scalable, and high-performance web infrastructure using Nginx.

Core Architecture and Principles

To solve these issues, we will utilize three core Nginx features: Load Balancing, Proxy Caching, and Dynamic Upstream switching.

1. Load Balancing

Load balancing is a technique for distributing incoming traffic across multiple backend servers (upstreams). Nginx makes this very simple to implement using the upstream block.

  • upstream block: Defines a group of servers to which the load will be distributed.
  • Distribution algorithms: Supports various methods like round-robin (default), least_conn (prioritizes the server with the fewest connections), and ip_hash (hashes the client IP to pin requests to a specific server).
  • Health Check: Periodically checks if a server has failed and automatically isolates it by not sending traffic to it. This increases service availability. (The open-source version only supports passive health checks.)

2. Proxy Caching

Proxy caching is a technique where Nginx stores responses from backend servers and, when an identical request comes in, immediately returns the cached data without contacting the backend again.

  • proxy_cache_path: Defines the disk path for storing cache data and attributes like the cache zone name and size.
  • proxy_cache_key: Defines the key that determines which requests are considered identical. It’s usually a combination of the request URL, scheme, host, etc.
  • proxy_cache_valid: Sets the cache validity time for different HTTP response codes.
  • Microcaching: An advanced caching technique where the cache validity time is set very short (e.g., 1-5 seconds) to serve near-real-time data while dramatically reducing the load on backend servers.

3. Zero-Downtime Deployment (Blue-Green Deployment)

Zero-downtime deployment (or Blue-Green Deployment) is a strategy where you run both the old version (Blue) and the new version (Green) simultaneously, completing the deployment by switching traffic in Nginx.

  1. All traffic is currently directed to the active Blue server group.
  2. A new version, the Green server group, is deployed and tested in a separate environment.
  3. Once everything is ready, the Nginx configuration is changed to instantly switch traffic from Blue to Green.
  4. If a problem occurs, you can immediately roll back to Blue. Once the Green environment is confirmed to be stable, the Blue environment is either kept on standby for the next deployment or removed.

By combining these three elements, you can build a robust architecture like the one below.

       [Client]
          |
  (https://example.com)
          |
+---------------------+
|        Nginx        |  <-- Reverse Proxy, Load Balancer, Cache Server
|  (Reverse Proxy)    |
+---------------------+
|         |           |
| (Cache Hit) (Cache Miss)
|         |           |
|  (Fast   +-----------+-----------------------+
| Response)|           |                       |
|         |    [Upstream Group: Backend]      |
|         |           |                       |
|         |      +----------+      +----------+
|         |      | WAS 1    |      | WAS 2    |
|         |      | (Blue)   |      | (Blue)   |
|         |      +----------+      +----------+
+---------+

Practical Code/Configuration Deep Dive

Now, let’s look at how to apply the concepts explained above to an actual nginx.conf file with specific code examples.

1. Load Balancing and Health Check Configuration

First, we define an upstream group within the http block. Here, we’ll distribute traffic to two backend servers.

nginx.conf

http {
    # ... (other http settings)

    # Define the backend application server group
    upstream backend_servers {
        # least_conn; # Send traffic to the server with the fewest connections
        # ip_hash;    # Pin connections to a specific server based on client IP

        # Define server 1
        # max_fails=3: Considered failed after 3 consecutive failures
        # fail_timeout=30s: Don't send traffic for 30s after marking as failed
        server 192.168.0.101:8000 max_fails=3 fail_timeout=30s;
        
        # Define server 2 (weight=2)
        server 192.168.0.102:8000 weight=2 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            # Forward requests to the upstream group
            proxy_pass http://backend_servers;

            # Set important proxy headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Enable keep-alive connections to the upstream server
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Key Points:

  • Define multiple server directives within the upstream backend_servers block to enable load balancing.
  • The max_fails and fail_timeout options are Nginx’s passive health check feature. If a server fails to respond, Nginx automatically detects this and excludes it from the load balancing pool for the duration of fail_timeout, thereby increasing service stability.

2. High-Performance Microcaching Configuration

Now, let’s add caching to our load balancing setup to reduce the load on the backend servers.

nginx.conf

http {
    # ... (upstream settings, etc.)

    # Define cache path and settings
    # path: /var/cache/nginx, levels: directory structure, keys_zone: memory zone name and size
    # inactive: delete cache if not accessed for a specified time, max_size: maximum disk usage
    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m inactive=60m max_size=10g;

    server {
        listen 80;
        server_name example.com;
        
        # Define cache key
        proxy_cache_key "$scheme$request_method$host$request_uri";

        location / {
            # Specify the cache zone to use
            proxy_cache api_cache;
            
            # Cache 200, 301, 302 responses for 10 minutes
            proxy_cache_valid 200 301 302 10m;
            # Cache 404 responses for 1 minute
            proxy_cache_valid 404 1m;
            
            # Whether to serve stale cache when the backend is down
            proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;

            # When multiple identical requests arrive, send only the first one to the backend and have others wait
            proxy_cache_lock on;

            # Add cache status (HIT, MISS, BYPASS, etc.) to the response header (for debugging)
            add_header X-Cache-Status $upstream_cache_status;

            # Bypass cache for logged-in users
            # If 'sessionid' cookie exists, set $skip_cache variable to 1
            set $skip_cache 0;
            if ($http_cookie ~* "sessionid") {
                set $skip_cache 1;
            }
            proxy_cache_bypass $skip_cache;
            proxy_no_cache $skip_cache;

            proxy_pass http://backend_servers;
            # ... (proxy header settings)
        }
    }
}

Key Points:

  • proxy_cache_path: This is the most crucial part, setting up the physical storage for the cache and its metadata space in memory (keys_zone).
  • proxy_cache_use_stale: A very useful feature that minimizes service disruption by serving expired cache content when backend servers are down.
  • proxy_cache_bypass: Allows you to configure conditions (e.g., the presence of a login cookie) under which the cache is bypassed and requests are always sent to the backend. This is essential for services that mix dynamic and static content.

3. Nginx Configuration for Zero-Downtime (Blue-Green) Deployment

For zero-downtime deployment, we’ll create a structure that dynamically switches backend groups using variables and the include directive.

1. Modify nginx.conf

Change the proxy_pass part to use a variable and include an external configuration file.

nginx.conf

http {
    # ... (cache settings, etc.)

    # Blue server group
    upstream blue_servers {
        server 192.168.0.101:8000;
        server 192.168.0.102:8000;
    }

    # Green server group
    upstream green_servers {
        server 192.168.0.201:9000;
        server 192.168.0.202:9000;
    }

    server {
        # ... (server settings)

        location / {
            # Include the currently active backend configuration
            include /etc/nginx/conf.d/current_backend.conf;

            # Use a variable to specify the proxy target
            proxy_pass http://$active_backend;

            # ... (cache, proxy header settings)
        }
    }
}

2. Create the current_backend.conf file

This file contains just one variable that specifies the upstream group currently receiving traffic.

/etc/nginx/conf.d/current_backend.conf

# Initial state: activate the Blue server group
set $active_backend blue_servers;

3. Write the deployment and switchover script

Once the new version (Green) is deployed, a simple shell script will change the contents of current_backend.conf and reload the Nginx configuration to switch traffic.

deploy.sh

#!/bin/bash

# Check the currently active backend
CURRENT_BACKEND=$(grep -oP 'set \$active_backend \K[^;]+' /etc/nginx/conf.d/current_backend.conf)
echo "Current active backend: $CURRENT_BACKEND"

if [ "$CURRENT_BACKEND" == "blue_servers" ]; then
    TARGET_BACKEND="green_servers"
else
    TARGET_BACKEND="blue_servers"
fi

echo "Switching traffic to $TARGET_BACKEND..."

# 1. Deploy code and run servers in the new backend environment (replace this with your actual deployment logic)
#    ansible-playbook deploy_new_version.yml --extra-vars "target=$TARGET_BACKEND"
echo "New version deployed to $TARGET_BACKEND servers."

# 2. Change the Nginx config file to point to the target backend
echo "set \$active_backend $TARGET_BACKEND;" > /etc/nginx/conf.d/current_backend.conf

# 3. Test and reload Nginx configuration (no downtime)
sudo nginx -t
if [ $? -eq 0 ]; then
    sudo systemctl reload nginx
    echo "Nginx reloaded. Traffic is now routed to $TARGET_BACKEND."
else
    echo "Nginx config test failed. Aborting."
    exit 1
fi

# 4. Clean up the old backend servers (optional)
#    echo "Shutting down old backend: $CURRENT_BACKEND"
#    ansible-playbook shutdown_old_version.yml --extra-vars "target=$CURRENT_BACKEND"

echo "Deployment finished."

Running this script changes the content of current_backend.conf and the nginx -s reload (or systemctl reload nginx) command gracefully restarts the worker processes, switching traffic to the new version without any service interruption.

Performance Optimization and Best Practices

In addition to the configurations above, here are a few more optimization points to consider in a production environment.

  • Keepalive Connections: The proxy_http_version 1.1; and proxy_set_header Connection ""; settings reuse TCP connections between Nginx and the backend servers, reducing TIME_WAIT sockets and decreasing latency. This is a very important performance tuning point.
  • Gzip Compression: Use gzip on; and related directives to compress responses at the Nginx level. This prevents backend servers from using CPU resources for compression, allowing them to focus purely on business logic.
  • Add Security Headers: You can enhance security by adding HTTP security headers like add_header X-Frame-Options "SAMEORIGIN"; and add_header X-Content-Type-Options "nosniff"; centrally in Nginx.
  • Custom Log Format: Use the log_format directive to log additional information beyond the default, such as $upstream_addr(the address of the backend server that handled the request), $upstream_response_time(backend response time), and $upstream_cache_status(cache status). This is extremely useful for troubleshooting and performance analysis.
log_format custom_format '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for" '
                         'upstream_addr: $upstream_addr '
                         'upstream_response_time: $upstream_response_time '
                         'cache_status: $upstream_cache_status';

access_log /var/log/nginx/access.log custom_format;

Conclusion

So far, we have explored specific ways to use Nginx not just as a simple reverse proxy, but as a high-performance cache server, an intelligent load balancer, and a central control tower for zero-downtime deployments. Load balancing and health checks using upstream ensure service availability, while microcaching with proxy_cache improves both user experience and server performance. Furthermore, dynamic backend switching using include and variables enables stable deployments without the fear of service interruptions.

The configurations introduced today are not just theoretical; they are standard approaches that have been tested and used in numerous large-scale services. We hope you will apply these advanced Nginx settings to your services to achieve a higher level of stability and performance.

References