Skip to main content

Troubleshoot FAQ

What do I do if my database instance crashed?

Common causes of database instance crashes include:

  • High system workloads
  • Hardware failures
  • Corrupted data pages
  • Bugs

Method to locate the cause:

  1. Collect and analyze the following logs:
    • OS log, saved in /var/log/messages by default
    • Error log of StoneDB, saved in /stonedb/install/log/mysqld.log by default
    • Trace log of StoneDB, saved in /stonedb/install/log/trace.log by default
  2. If the cause cannot be located after the OS log, error log, and trace log are analyzed, enable core dumps. In this way, if the database instance crashes again, you can use GDB to analyze the generated core dumpfile.

What do I do if my database instance is suspended?

Common causes of database instance suspensions include:

  • High system workloads
  • Maximum number of connections reached
  • Disk capacity exhaustion
  • Metadata lock waits
  • Bugs

Method to locate the cause:

  1. Check whether the number of connections reaches the threshold.

If the number of connections reaches the threshold specified by parameter max_connections, no new connections to the database instance can be created.

  1. Check whether the disk capacity is exhausted.

If the disk capacity is exhausted, the binlog generated by the metadata locks cannot be written to disks.

  1. Check whether lock waits exist.

Execute the SHOW PROCESSLIST statement. If the output contains lots of "Waiting for table metadata lock" messages, locate and terminate the thread that causes the congestion.

  1. Collect the stack information of the mysqld process.

If the process is in the zombie state, you can run the pstack command for several times in a day to track the stack information of the process. If the position of the stack does not change, pay attention to the position, which is the potential cause of the problem.

When the pstack command is run to capture the stack information of the process, a SIGSTOP signal is triggered. The signal functions like the kill -19 command. In fact, running the pstack command invokes a GDB command. Then, GDB initiates a SIGSTOP signal to terminate the process. Exercise with caution when running the pstack command in the production environment, because this operation may interrupt the running of applications for a little while.

You can run the pstack mysqld_pid >> mysqld_pid.log command to output the stack information of all threads.