Unable to receive device data following crash

stevew2 years ago

A few days ago my server crashed due to lack of disk space caused by excessive binfiles (21GB of binfiles for a database that is only 40MB). Why this happened is unclear, but for now it is resolved by reducing the time for removal of the binfiles to 3 days from the default of 30 days. As far as I can tell the database is working OK now (it allows me to create new devices and there are no errors in the mysql error log) but the traccar server log is showing no data from the devices.

Checking the device ports 5062 and 5187 suggests they are open for ipv6 but not ipv4; mysql port 3306 meanwhile shows it is open for ipv4 and ipv6. Checking the firewall status shows it is inactive; setting it to enable gives the message it is now working and will be active by default and a recheck of status shows it to be active, however, after a reboot it is again inactive.

Traccar log gives repeated errors as follows

2022-02-08 16:48:48 ERROR: An I/O error has occurred while writing a response message entity to the container output stream. - 
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:635)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1619)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:506)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1372)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:463)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1294)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:562)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:406)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:663)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:398)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:319)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SocketChannelEndPoint$1.run(SocketChannelEndPoint.java:101)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:412)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:381)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:268)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:138)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:378)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:894)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1038)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jetty.io.EofException
    at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:285)
    at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:417)
    at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:373)
    at org.eclipse.jetty.io.SocketChannelEndPoint$2.run(SocketChannelEndPoint.java:116)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:412)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:366)
    ... 6 more
Caused by: java.io.IOException: Connection reset by peer
    at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
    at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
    at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
    at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:279)
    ... 11 more

Can anybody suggest what I need to do to resolve this?

Anton Tananaev2 years ago

This error simply means that a device is disconnected from the server.

stevew2 years ago

Thanks for the reply. I understand the devices are disconnected, what I am struggling with is why and how to resolve the problem. It seems the relevant ports are not open for ipv4, could you please give me some guidance what to do nest to get the devices reconnected.

Thanks

Anton Tananaev2 years ago

If they weren't open, devices wouldn't even connect.

stevew2 years ago

So are you saying they are connecting and then disconnecting? the Traccar log does not show any data records from them