Monday, July 30, 2012

On MySQL Merge tables

The MySQL Merge storage engine is nothing more than a means to link together identical MyISAM tables into one big one that you can query at your leisure. To create one, first create a table with a structure identical to one of the tables you want to link, for instance by doing a create like statement. The next, and final step, is to alter the new table's engine to merge and define the union. Simple as pie. It won't be long, however, before MySQL throws an error at you:

ERROR 1168 (HY000): Unable to open underlying table which is differently defined or of non-MyISAM type or doesn't exist

The four causes outlined below should cover 98% of all the situations where you encounter this error.

  1. The tables really are different
    This may be because the schema of one of the tables is different, or was created with another MySQL version (and you copied the raw tables across at one point)
  2. Some, or all of the tables are not MyISAM tables
    Merge only works with MyISAM tables so the solution in this case is to change the storage engine type on all the tables to MyISAM
  3. Some, or all of the tables in your union do not exist
    MySQL does not check if the tables exist when you create the UNION, so it is perfectly possible to create a UNION on tables that don't exist, and MySQL will only tell you when you try to use the merged table.
  4. You created your union in a different database, and did not specify what database the tables are in
    This particular situation caused me much grief. Consider that you are using a MySQL interactive client, and that you have selected the test2 database, and your source tables are in the test database. Now when you write your merge, you may do:
    ALTER TABLE test.my_merged_table ENGINE=MERGE UNION=(
    	table1,table2,table3
    );
    

    You would expect MySQL to be smart enough to figure out that table1,table2,table3 are in the test database, since that is where the merged table is being saved.

    Except, it doesn't. It assumes table1,table2,table3 refer to the current database, and will create the merged table like that.

When you encounter that error, there is nothing else that you can do with that table, except to delete it.

If you are not sure why you are getting this error, you may want to look at exactly how MySQL defined your merge. Since describe and show create table and the likes do not work on a busted merge table, your only other option is to look at the physical merge definition file on the disk.

The file you are looking for, is in the MySQL data directory, in the sub directory of your database, and is called my_merged_table.MRG (replace my_merged_table with your own table name).

The format of the file is plain ASCII and it just lists, one entry per line, the tables, in order, that are part of the union for that table

4 comments:

  1. Since you posted this back in 2012, have you come across any other reasons for this failure. I have 2 years worth of tables whereby there is a table for each day. This means there are 732 tables in the union definition. I've edited the MRG file and there are no other non-conforming tables (All the tables are named like xxx_dd_yyyymm.) All are MyISAM types. All were created based on a template, so the field definitions are the same. I used a Perl script to build the union and create the merge table. It has worked for a long time, but now it is not. I don't know what has changed. Any additional insight would be appreciated. Thanks.

    ReplyDelete
  2. Some things to consider:

    * Check the MySQL error log just after the failed merge
    * Repair all the tables. If one of the tables in the 'collection' has got an error then the merge will not work.
    * If you have upgraded your MySQL and are using tables made with the new(er) versions which you are trying to merge with tables made with the older version, then it is possible that the tables have a different internal MyISAM version - which will also cause a failure.

    If that is the case then you need to rebuild each and every table with something like ALTER TABLE x ENGINE=MyISAM;

    But before you do that, I'd say experiment with making merges with a couple tables. A recent with an older, a newer with a newer, old and old, etc. Look for a pattern that can tell you if/when the problem started.

    Hope this helps, please note the usual disclaimer which basically means: everything you do, you do at your own risk.

    ReplyDelete
  3. oh, and if you have changed your merge file manually, you'll need to flush tables before using it.

    FLUSH TABLES;
    or

    FLUSH TABLES xxx.yyy;

    ReplyDelete