Mysql – MariaDB galera cluster fail when dumping or optimizing database

It seems likely that these are the important lines in the log:

 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://<IP1>,<IP2>': -110 (Connection timed out)
 150128 14:29:02 [ERROR] WSREP: gcs connect failed: Connection timed out
 150128 14:29:02 [ERROR] WSREP: wsrep::connect() failed: 7

Typically this points to a network issue, more often that not firewall. Make sure you have tcp ports 3306,4444,4567 & 4568 open between the nodes.

I was not able to find an official way to handle this, so I went with the idea of dumping the tables individually and reimporting them. Not wanting to do it by hand, I whipped a PHP script to do it for me. I'm posting it here in case anyone else finds this useful.

/*
 * InnoDB Convert
 * Converts existing non-InnoDB tables to InnoDB, then re-imports the
 * data so that it's replicated across the cluster.
 */

// Configuration
    $_config['db'] = array(
        'type'     => 'mysql',
        'host'     => 'localhost',
        'username' => 'user',
        'password' => 'password'
    );

// Establish database connection
    try {
        $pdo = new PDO(
            $_config['db']['type'] . ':host=' . $_config['db']['host'],
            $_config['db']['username'],
            $_config['db']['password']
        );
    } catch ( PDOException $e ) {
        echo 'Connection failed: ' . $e->getMessage();
    }

// Get list of databases
    $db_query = <<<SQL
  SHOW DATABASES
SQL;
    $db_result = $pdo->prepare( $db_query );
    $db_result->execute();

    while ( $db_row = $db_result->fetch( PDO::FETCH_ASSOC )) {
    // Look through databases, but ignores the ones that come with a
    // MySQL install and shouldn't be part of the cluster
        if ( !in_array( $db_row['Database'], array( 'information_schema', 'mysql', 'performance_schema', 'testdb' ))) {
            $pdo->exec( "USE {$db_row['Database']}" );

            $table_query = <<<SQL
  SHOW TABLES
SQL;
            $table_result = $pdo->prepare( $table_query );
            $table_result->execute();

            while ( $table_row = $table_result->fetch( PDO::FETCH_ASSOC )) {
            // Loop through all tables
                $table = $table_row["Tables_in_{$db_row['Database']}"];

                $engine_query = <<<SQL
  SHOW TABLE STATUS WHERE Name = :table
SQL;
                $engine_result = $pdo->prepare( $engine_query );
                $engine_result->execute( array(
                    ':table' => $table
                ));

                $engine_row = $engine_result->fetch( PDO::FETCH_ASSOC );

                if ( $engine_row['Engine'] != 'InnoDB' ) {
                // Engine is not equal to InnoDB, let's convert it
                    echo "Converting '$table' on '{$db_row['Database']}' from '{$engine_row['Engine']}' to InnoDB:\n";

                    echo     "Modifying engine...";

                    $change_query = <<<SQL
  ALTER TABLE $table ENGINE=InnoDB
SQL;
                    $change_result = $pdo->prepare( $change_query );
                    $change_result->execute();

                    echo "done!\n";

                    echo "    Exporting table...";

                    exec( "mysqldump -h {$_config['db']['host']} -u {$_config['db']['username']} -p{$_config['db']['password']} {$db_row['Database']} $table > /tmp/dump-file.sql" );

                    echo "done!\n";

                    echo "    Re-importing table...";

                    exec( "mysql -h {$_config['db']['host']} -u {$_config['db']['username']} -p{$_config['db']['password']} {$db_row['Database']} < /tmp/dump-file.sql" );

                    echo "done!\n";

                    unlink( '/tmp/dump-file.sql' );

                    echo "done!\n";
                }
            }
        }
    }

I successfully used it to convert hundreds of tables across a couple dozen databases in about two minutes.