An exceptionCaught() event was fired... Too many open files

Peter Flower 7 years ago

My Traccar Server is down, the first thing that I did was to check the log, it shows this:

WARN: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. - Too many open files - IOException (...)

I have increased the limits for connections and things like that.
These are some commands and result:

COMMAND: ulimit -Hn
RESULT: 50000

COMMAND: ulimit -Sn
RESULT: 50000

COMMAND: ulimit -a
RESULT:
core file size                                   (blocks, -c) 0
data seg size                                  (kbytes, -d) unlimited
scheduling priority                         (-e) 0
file size                                            (blocks, -f) unlimited
pending signals                              (-i) 3892
max locked memory                      (kbytes, -l) 64
max memory size                           (kbytes, -m) unlimited
open files                                        (-n) 50000
pipe size                                         (512 bytes, -p) 8
POSIX message queues               (bytes, -q) 819200
real-time priority                            (-r) 0
stack size                                       (kbytes, -s) 8192
cpu time                                         (seconds, -t) unlimited
max user processes                      (-u) 3892
virtual memory                               (kbytes, -v) unlimited
file locks                                          (-x) unlimited

COMMAND: netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
RESULT:
1           established)
1           Foreign
1           SYN_RECV
10        TIME_WAIT
120      CLOSE_WAIT
188      LISTEN
3756   ESTABLISHED

COMMAND: netstat -an | grep :5027 | wc -l
RESULT: 3857

COMMAND: sudo cat /proc/sys/fs/file-max
RESULT: 99102

COMMAND: sudo lsof | grep java | wc -l
RESULT: 8586

COMMAND: free -h
RESULT:
                    total     used     free     shared     buff/cache     available
Mem:          994M   492M    122M     44M     379M              296M
Swap:          1.0G     362M    661M

When the server has a lot of connections opened this goes down.
I think the most obvious would be that the server don't accept more connections and keep working, but no, the server die.

How can I avoid this? I found this link:
https://github.com/netty/netty/issues/1578

I hope you help.

Best Regards
Peter

Anton Tananaev 7 years ago

Check process limits and actual number of connections.

Peter Flower 7 years ago

Are you talking about this?

top
top

sudo cat /proc/515/limits
sudo cat /proc/515/limits

Anton Tananaev 7 years ago

Yes, so it's 4k limit, not 50k.

Peter Flower 7 years ago

Anton, I know have you seen my post, but, could you take another check to the results of the commands, mainly commands ulimit.

Best Regards
Peter

Anton Tananaev 7 years ago

I have read it again and I don't understand what you want me to see.

Also, I don't understand what you mean by "but no, the server die". Server continues running, as far as I can tell.

Peter Flower 7 years ago

No, when the number of connections reach the max number, the app crash.
When I said "server die" I was talking about the app.

Anton Tananaev 7 years ago

So did you execute all those commands before server died?

Peter Flower 7 years ago

Yes, when the app wasn't working and the max number of connection was reached.
All that commands were executed before to restart server.

Anton Tananaev 7 years ago

And it died after that?

Peter Flower 7 years ago

The server don't die. The app crash.

Anton Tananaev 7 years ago

OK, please provide some evidence of the crash, exit code, system logs etc.

Peter Flower 7 years ago

Commands executed this moment:

COMMAND: who
RESULT: admin pts/0 2019-03-09 19:01 (xxx.xxx.xxx.xxx)

COMMAND: su root --shell /bin/bash --command "ulimit -n"
RESULT: 1024

Peter Flower 7 years ago
Anton Tananaev 7 years ago

Yes, I saw the link.