Robert Michalski

Increase Nginx worker open files and connections

When your website, api or service starts getting more simultaneous traffic, some visitors can start encountering 500-type errors. Usually it's the application, meaning the website, API or service that can't keep up, specially if it's written in PHP, such as WordPress, Joomla, Laravel etc. Node.js based services can usually handle many connections.

When requests are for static files of served from cache (FastCGI, proxy) limitations in nginx can cause 500 errors.

Checking the nginx error log, there might be lines saying 'too many open files' or 'too many open connections'. This guide will help you to raise both those limits so nginx can handle way more traffic than the default very conservative settings.

Raising connection limit in Nginx #

The default open file limit for Linux based systems is 1024, this varies by distribution. Default worker_connections for Nginx is 768, which is very low and can easily be exhausted during traffic spikes.

Nginx needs two open files to proxy a connection, one for receiving and one for sending.

Assuming open file limit of the system is 1024 and connections for each worker is set to 768 by default, we can calculate how many connections nginx is able to proxy.

So by default nginx is set to use a maximum of 768 * 2 => 1536 file descriptors. The system open file limit is usually set to 1024, so nginx can theoretically handle only 1024 / 2 => 512 connections maximum by default.

It gets worse, nginx also needs to open files to serve them, all the js, css, html, fonts, images etc. And what about the other processes of the system, they also need to open, read and write files, they all share the 1024 open file limit of the system.

DigitalOceans managed load-balancers can handle 10 000 simultaneous connections per server/instance which is based on 1-core VPSs. It should be possible to go from 768 to a couple of thousand connections at least even on a 1-core VPS.

1. Increasing maximum open files in Linux #

Increasing the number of connections that Nginx can handle requires that the operating system is set up correctly for it to work.

A nice guide on how to set hard and soft limits on a system level and for a specific user can be found here.

Commands to check hard and soft limit of open files on Linux.

ulimit -Hn
ulimit -Sn

Safe limits for soft open files, you can probably go much higher with some testing.

Command to add soft limit for open files (soft nofile) in /etc/security/limits.conf

perl -i -p -e 's/# End of file/* soft nofile 32768\n# End of file/' /etc/security/limits.conf

Command to add very high hard limit for open files (fs.file-max) in /etc/sysctl.conf

echo 'fs.file-max=2097152' >> /etc/sysctl.conf

2. Commands to increase number of open files in Nginx #

Limits on how many files each Nginx worker can open is set using worker_rlimit_nofile in the main Nginx config file, usually found in /etc/nginx/nginx.conf. This will allow nginx to open more static files (html, js, css, fonts) for serving and to be able to open more connections to servers for proxying.

These commands will add worker_rlimit_nofile to the nginx config, with values to use only half the maximum value set in the system. Leave half of the allowed limit for other users and processes.

perl -i -p -e 's/events {/worker_rlimit_nofile 16384;\n\nevents {/' /etc/nginx/nginx.conf

3. Commands to increase worker connections in Nginx #

After enabling each nginx worker to open enough files by setting a high enough value for worker_rlimit_nofile, it's time to raise the number of connections that nginx can open.

Nginx uses 2 file descriptors per connection for proxying, one for receiving and one for sending.

16384 files open / 1 worker / 2 files descriptors per connection => max 8192 connections per worker, set to 4096 to leave 8192 for other files. This way Nginx should be able to proxy 4096 concurrent connections per CPU and be able to serve files at the same time.

This command will search for the existing worker_connections value and replace the default low value with a safe higher value.

perl -i -p -e 's/worker_connections.+;/worker_connections 4096;/' /etc/nginx/nginx.conf

Conclusion #

Applying the settings in above Nginx should now be able to open 8192 open files per worker in addition to proxying 4096 concurrent connections. The other processes and the operating system itself should also be able to open plenty of files and connections.

These limits should be safe, the best is, as always, to test and tweak with your own particular setup. You might be able to go even higher.