Error 24: Too many open files

Yesterday while working on a test framework that simulates thousands of devices, I was faced with a unique error message.

pi@madala-pi:~/CLearn$ ./test_filelimit
error in opening file: Too many open files

Initially I suspected there could be some file descriptor leak in my code; I verified the code thoroughly for any leak in file descriptors, I found none. Then I decided to debug the issue form Linux respective.

Each thread in the framework simulates thousands of devices, every device will connect to remote server, and will be periodically sending the GPS co-ordinates; When I was unable to find any issue with my code, I suspected that the issue has to do with user limits on the system.

I found that, in my system, the user user limits are configured like this:

pi@madala-pi:~/CLearn$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3372
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 95
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3372
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
pi@madala-pi:~/CLearn$

Observe the highlighted portion, this shows that the maximum number of file descriptors that can be opened by any process, in my case it is 1024; My program was simulating thousands of devices and each device needs at-least one file descriptor to connect to server, so I now know that  I had to raise these limits.

Modify the User Limit

The user limits can be found in  /etc/security/limits.conf.

*               soft    nofile          10000
*               hard    nofile          20000

Add the two lines at the bottom of the file, this will increase the soft-limit and the hard-limit for the number of open files allowed by the system. You need to logout and login again for the values to take affect or run bash again.

pi@madala-pi:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3372
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 95
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3372
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
pi@madala-pi:~$

Observe now that the limits are modified; Do you wonder why we added two values? The trick lies with the keywords soft and hard. Hard limits can never be modified by ordinary user, whereas a soft limit can be changed by normal two ways.

Changing the soft limit from outside

pi@madala-pi:~$ ulimit -a |grep open
open files                      (-n) 10000
pi@madala-pi:~$
pi@madala-pi:~$ ulimit -n 13000
pi@madala-pi:~$ ulimit -a |grep open
open files                      (-n) 13000
pi@madala-pi:~$

Using ulimit command I modified the number to be thirteen thousand, when we try to make it beyond hard limit it is not allowed.

pi@madala-pi:~$ ulimit -n 21000
-bash: ulimit: open files: cannot modify limit: Operation not permitted
pi@madala-pi:~$

This is because the hard-limit is set at twenty thousand, and only root user can modify hard values.

Changing the soft limit from inside a program

We can change the soft-limit from within a program by using setrlimit() function. I wrote a small program to get and set these limits.

pi@madala-pi:~/CLearn$ cat test_rlimit.c
#include<stdio.h>
#include <sys/time.h>
#include <sys/resource.h>

int main()
{
  struct rlimit limit={0}, newLimit={0};

  getrlimit(RLIMIT_NOFILE, &limit);
  printf("Current limit for number of files Soft:%d Hard:%d\n", limit.rlim_cur, limit.rlim_max);

  limit.rlim_cur = 15000;
  setrlimit(RLIMIT_NOFILE, &limit);

  getrlimit(RLIMIT_NOFILE, &newLimit);
  printf("New limit for number of files Soft:%d Hard:%d\n", newLimit.rlim_cur, newLimit.rlim_max);
}

pi@madala-pi:~/CLearn$ ./test_rlimit
Current limit for number of files Soft:10000 Hard:20000
New limit for number of files Soft:15000 Hard:20000
pi@madala-pi:~/CLearn$

Observe that Initially the soft-limit is set to 10K and the hard-limit is set to 20K, I modified the soft-limit using the function setrlimit(), and the new value of soft-limit is 15k.

To my surprise I found that my problem was not solved completely, after running this program for a while with multiple threads I found one more issue. This time I hit the system wide limit.

Modify system wide limit

The system wide limit can be found by looking at /proc/sys/fs/file-max.

pi@madala-pi:~/CLearn$ cat /proc/sys/fs/file-max
130130
pi@madala-pi:~/CLearn$

This showed that the maximum number of files that can be open at any time is only 0.1M, which is a very small number. One can modify these values using sysctl command.

# sysctl -w fs.file-max=1000000

I made it 1M, by using sysctl command. To make it permanent these values need to be stored in /etc/sysctl.conf.

fs.file-max = 1000000

Lesson Learned

1. Check for any file descriptor leak
2. Increase per process limit on file descriptors
3. Increase system wide limit on file descriptors.

This was a great learning experience, I got to know many things while debugging this issue.