I would choose the database options. If you export alle database tables starting with catalog and you have no quotes and orders (empty tables sales_flat_quote and sales_flat_order, it should work without foreign key problems. If you have different attributes, you should perhaps export the eav_* tables as well.

By the way, why not re-use the whole database from the old system? that's the cleanest way to upgrade.

Probably the export execution hit the max_execution_time or memory_limit limits set by php. However, if your application uses Apache as a webserver, you should be able to modify these limits by setting them in your .htaccess.

#Increase the memory limito to 1G
php_value memory_limit 1G

# Define the script timeout after 1 hour
php_value max_execution_time 3600

# If Magento fails after submitting a form you may need to tune the input limits

php_value post_max_size 5M
php_value upload_max_filesize 5M
php_value max_input_time 300

After editing the .htaccess, upload it in the Magento root.

Warning:

errors in the .htaccess file may lead to unreachable (error 500) website. Test it locally before uploading it.