[Editor – This post has been updated to use the NGINX Plus API, which replaces and deprecates the separate dynamic configuration and status modules discussed in the original version of the post.]
NGINX Open Source is very popular as a frontend server for many hosting providers, CDNs, and other high‑volume platform providers. Caching, reverse proxying, and serving static files are among the most widely used features. However, there is a particular use case when it can be complicated to use a build of NGINX Open Source – large‑volume hosting.
In this blog post, we’ll explain how you can use NGINX Plus features to make large‑volume hosting easier and more reliable, freeing up engineering and operations resources for other critical tasks.
Imagine a hosting company using a reverse proxy frontend to handle a large number of hostnames for its customers. Since every day new customers appear and current customers leave, the NGINX configuration needs to be updated quite frequently. In addition, the use of containers and cloud infrastructure requires DevOps to move resources constantly between different physical and virtual hosts.
With NGINX Open Source, reconfiguring a group of upstream (backend) servers requires you to reload the configuration. Reloading the configuration causes zero downtime, but doing it while NGINX is under high load can cause problems. One reason is that worker processes that are running at the time of the reload have to finish serving established connections before being terminated, a process known as draining. The delay can be lengthy if there are:
- Large file downloads in progress
- Clients using the WebSocket protocol
- Clients with slow connectivity (3G, mobile)
- Slowloris attacks
Reloading multiple times without leaving enough time to terminate the worker processes running previous configurations can lead to increased memory use and eventually can overload the system.
A large number of worker processes in shutting down state can indicate this problem:
[user@frontend]$ ps afx | grep nginx
652 ? Ss 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
1300 ? S 0:00 _ nginx: worker process is shutting down
1303 ? S 0:00 _ nginx: worker process is shutting down
1304 ? S 0:00 _ nginx: worker process is shutting down
1323 ? S 0:00 _ nginx: worker process is shutting down
1324 ? S 0:00 _ nginx: worker process is shutting down
1325 ? S 0:00 _ nginx: worker process is shutting down
1326 ? S 0:00 _ nginx: worker process is shutting down
1327 ? S 0:00 _ nginx: worker process is shutting down
1329 ? S 0:00 _ nginx: worker process is shutting down
1330 ? S 0:00 _ nginx: worker process is shutting down
1334 ? S 0:00 _ nginx: worker process
1335 ? S 0:00 _ nginx: worker process
1336 ? S 0:00 _ nginx: worker process
1337 ? S 0:00 _ nginx: worker processYou can use the following configuration to replicate the result of frequent configuration reloads. Comments at the top of the file explain how to use the configuration.
Solution: Dynamic Configuration
To avoid excessive memory use and system overload, our commercial offering, NGINX Plus, supports dynamic configuration of upstream server groups (upstream blocks) with the NGINX Plus API.
Another NGINX Plus feature that can be used together with dynamic configuration is session persistence, in which you specify the particular server that handles requests from a given client. With route‑based session persistence, the key that determines which server handles a request can be based on any variable found in the request. Commonly used variables include cookie‑related variables like $cookie_jsessionid and IP‑address variables like $remote_addr. In the following example we use the $http_host variable, which corresponds to the Host header.
This fully commented example combines route‑based session persistence with dynamic configuration.
Dynamic Configuration with the NGINX Plus API
Notice that the upstream block in this configuration does not include any server directives to define the hostnames of backend servers. We are going to define them dynamically, using the NGINX Plus API. An easy way to access the API is with curl commands.
Let’s say that we need to add the following customers to our hosting environment:
| Customer hostname | Backend server IP address and port |
|---|---|
| www.company1.com | 127.0.0.1:8080 |
| www.company2.com | 10.0.0.10:8000 |
| www.homepage1.com | 10.0.0.10:8001 |
To add the customers, we run the following commands which access the NGINX Plus API. We’re logged into the shell on the NGINX Plus host, but it’s possible to run these commands from any secure location:
[user@frontend]$ curl -X POST -d '{"server":"127.0.0.1:8080", "route":"www.company1.com"}' http://127.0.0.1:8080/api/version/http/upstreams/vhosts/servers
[user@frontend]$ curl -X POST -d '{"server":"10.0.0.10:8000", "route":"www.company2.com"}' http://127.0.0.1:8080/api/version/http/upstreams/vhosts/servers
[user@frontend]$ curl -X POST -d '{"server":"10.0.0.10:8001", "route":"www.homepage1.com"}' http://127.0.0.1:8080/api/version/http/upstreams/vhosts/serversTo list the upstream servers and learn their internal ID numbers, run this command. We’re piping the output to the jq utility to filter for just the name or address, ID, and route for each server:
[user@frontend]$ curl -s http://127.0.0.1:8080/api/version/http/upstreams/vhosts | jq -c '.peers[] | {server, id, route}'
{"server":"127.0.0.1:8080","id":0,"route":"www.company1.com"}
{"server":"10.0.0.10:8000","id":1,"route":"www.company2.com"}
{"server":"10.0.0.10:8001","id":2,"route":"www.homepage1.com"}To remove a server from an upstream, identify it by its ID (as reported by the previous command), as in this example for the third server added above:
[user@frontend]$ curl -X DELETE http://127.0.0.1:8080/api/version/http/upstreams/vhosts/servers/2By default, changes you make with the NGINX Plus API persist across configuration reloads but do not persist across restarts of the nginx process. In NGINX Plus Release 8 and later, you can include the state directive in the upstream block to make changes fully persistent. The state directive names the file in which NGINX Plus stores state information about the servers in the upstream group. When it is included in the configuration, you cannot also include the server directive to statically define servers.
Dynamic Configuration with the Live Activity Monitoring Dashboard
You can also use the NGINX Plus live activity monitoring dashboard to see your current server configuration and perform changes dynamically. The directives in dynamic-upstream-configuration.conf above are all that is necessary for configuring live activity monitoring. With the directives in place, you can access the dashboard by opening http://your-nginx-server:8080/dashboard.html in a web browser on the NGINX Plus host.
Alternatively, you can access monitoring metrics from the NGINX Plus API API at the URL endpoints described in the API reference documentation.
For a live demo of a working dashboard, check out https://demo.nginx.com/.
Summary
Frequently adding and removing hostnames and corresponding backend services can lead to unnecessary increases in memory use, and resulting performance problems, in a highly loaded deployment. Upgrading your stack to NGINX Plus resolves this challenge with the help of the NGINX Plus API and session persistence.
Use of these advanced features of NGINX Plus reduces the number of required process reloads, improves responsiveness to customer requests, provides better infrastructure scalability, and enables technical teams to spend their time on other, more critical tasks.
To try out the NGINX Plus features we’ve discussed, start your free 30-day trial today or contact us to discuss your use cases.
