Ubuntu / MySQL Database Crash

Niko Lync8 months ago

Hi

My database working for motnhs and crashed itself today... with error "Data Dictionary initialization failed." I can't start mysql.service anymore - anybody help please ?

There are logs from mysql after trying to start service:

2023-08-16T09:02:52.032346Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.34-0ubuntu0.22.04.1) starting as process 7397
2023-08-16T09:02:52.046210Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-08-16T09:02:52.084885Z 1 [ERROR] [MY-012209] [InnoDB] Multiple files found for the same tablespace ID:
2023-08-16T09:02:52.084960Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 193 = ['sys/traccar/DATABASECHANGELOG.ibd', 'traccar/DATABASECHANGELOG.ibd'] 
2023-08-16T09:02:52.084992Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 194 = ['sys/traccar/DATABASECHANGELOGLOCK.ibd', 'traccar/DATABASECHANGELOGLOCK.ibd'] 
2023-08-16T09:02:52.085008Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 195 = ['sys/traccar/tc_attributes.ibd', 'traccar/tc_attributes.ibd'] 
2023-08-16T09:02:52.085022Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 196 = ['sys/traccar/tc_calendars.ibd', 'traccar/tc_calendars.ibd'] 
2023-08-16T09:02:52.085036Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 197 = ['sys/traccar/tc_commands.ibd', 'traccar/tc_commands.ibd'] 
2023-08-16T09:02:52.085050Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 198 = ['sys/traccar/tc_device_attribute.ibd', 'traccar/tc_device_attribute.ibd'] 
2023-08-16T09:02:52.085064Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 199 = ['sys/traccar/tc_device_command.ibd', 'traccar/tc_device_command.ibd'] 
2023-08-16T09:02:52.085077Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 200 = ['sys/traccar/tc_device_driver.ibd', 'traccar/tc_device_driver.ibd'] 
2023-08-16T09:02:52.085091Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 201 = ['sys/traccar/tc_device_geofence.ibd', 'traccar/tc_device_geofence.ibd'] 
2023-08-16T09:02:52.085104Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 202 = ['sys/traccar/tc_device_maintenance.ibd', 'traccar/tc_device_maintenance.ibd'] 
2023-08-16T09:02:52.085118Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 203 = ['sys/traccar/tc_device_notification.ibd', 'traccar/tc_device_notification.ibd'] 
2023-08-16T09:02:52.085132Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 204 = ['sys/traccar/tc_device_order.ibd', 'traccar/tc_device_order.ibd'] 
2023-08-16T09:02:52.085151Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 205 = ['sys/traccar/tc_devices.ibd', 'traccar/tc_devices.ibd'] 
2023-08-16T09:02:52.085165Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 206 = ['sys/traccar/tc_drivers.ibd', 'traccar/tc_drivers.ibd'] 
2023-08-16T09:02:52.085178Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 207 = ['sys/traccar/tc_events.ibd', 'traccar/tc_events.ibd'] 
2023-08-16T09:02:52.085207Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 208 = ['sys/traccar/tc_geofences.ibd', 'traccar/tc_geofences.ibd'] 
2023-08-16T09:02:52.085221Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 209 = ['sys/traccar/tc_group_attribute.ibd', 'traccar/tc_group_attribute.ibd'] 
2023-08-16T09:02:52.085234Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 210 = ['sys/traccar/tc_group_command.ibd', 'traccar/tc_group_command.ibd'] 
2023-08-16T09:02:52.085248Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 211 = ['sys/traccar/tc_group_driver.ibd', 'traccar/tc_group_driver.ibd'] 
2023-08-16T09:02:52.085261Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 212 = ['sys/traccar/tc_group_geofence.ibd', 'traccar/tc_group_geofence.ibd'] 
2023-08-16T09:02:52.085274Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 213 = ['sys/traccar/tc_group_maintenance.ibd', 'traccar/tc_group_maintenance.ibd'] 
2023-08-16T09:02:52.085288Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 214 = ['sys/traccar/tc_group_notification.ibd', 'traccar/tc_group_notification.ibd'] 
2023-08-16T09:02:52.085301Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 215 = ['sys/traccar/tc_group_order.ibd', 'traccar/tc_group_order.ibd'] 
2023-08-16T09:02:52.085314Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 216 = ['sys/traccar/tc_groups.ibd', 'traccar/tc_groups.ibd'] 
2023-08-16T09:02:52.085327Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 217 = ['sys/traccar/tc_maintenances.ibd', 'traccar/tc_maintenances.ibd'] 
2023-08-16T09:02:52.085340Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 218 = ['sys/traccar/tc_notifications.ibd', 'traccar/tc_notifications.ibd'] 
2023-08-16T09:02:52.085354Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 219 = ['sys/traccar/tc_orders.ibd', 'traccar/tc_orders.ibd'] 
2023-08-16T09:02:52.085367Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 220 = ['sys/traccar/tc_positions.ibd', 'traccar/tc_positions.ibd'] 
2023-08-16T09:02:52.085402Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Failed, retry may succeed.
2023-08-16T09:02:52.085441Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2023-08-16T09:02:52.085625Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-08-16T09:02:52.085687Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-08-16T09:02:52.086807Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.34-0ubuntu0.22.04.1)  (Ubuntu).
2023-08-16T09:02:52.987000Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.34-0ubuntu0.22.04.1) starting as process 7408
2023-08-16T09:02:53.000447Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-08-16T09:02:53.039920Z 1 [ERROR] [MY-012209] [InnoDB] Multiple files found for the same tablespace ID:
2023-08-16T09:02:53.040005Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 193 = ['sys/traccar/DATABASECHANGELOG.ibd', 'traccar/DATABASECHANGELOG.ibd'] 
...
Niko Lync8 months ago

Traccar Version: 5.6

David Cole8 months ago

Restore from backup?

Niko Lync8 months ago

Backup is the last resort. I solved the problem by deleting files from the sys/traccar/ location - but the question?

Where did they come from? and why did the problem occur suddenly when no work was done on the server???

Anton Tananaev8 months ago

Someone must have copied those files, but it's definitely not Traccar.