Magento – Layout Update XML doesn’t properly work for the home page

You can't do that. Specifically, you can't use the <update/> tag in a CMS page's Layout Update XML field.

Magento processes the <update/> nodes when it is initially loading the layout update files, and layout updates from the core_layout_update table. This is done with the fetchRecursiveUpdates function.

#File: app/code/core/Mage/Core/Model/Layout/Update.php

public function fetchPackageLayoutUpdates($handle)
{
    //...
        $this->fetchRecursiveUpdates($updateXml);
    //...

    return true;
}

public function fetchDbLayoutUpdates($handle)
{
    //...
    $this->fetchRecursiveUpdates($updateXml);        
    //...        
}

//...
public function fetchRecursiveUpdates($updateXml)
{
    foreach ($updateXml->children() as $child) {
        if (strtolower($child->getName())=='update' && isset($child['handle'])) {
            $this->merge((string)$child['handle']);
            // Adding merged layout handle to the list of applied hanles
            $this->addHandle((string)$child['handle']);
        }
    }
    return $this;
}

A CMS page, however, is rendered later on via a helper method. The rendering of a CMS page includes this code

#File: app/code/core/Mage/Cms/Helper/Page.php

$action->loadLayoutUpdates();
$layoutUpdate = ($page->getCustomLayoutUpdateXml() && $inRange)
    ? $page->getCustomLayoutUpdateXml() : $page->getLayoutUpdateXml();
$action->getLayout()->getUpdate()->addUpdate($layoutUpdate);
$action->generateLayoutXml()->generateLayoutBlocks();    

The loadLayoutUpdates call is the standard loading of the layout, including the fetchRecursiveUpdates mentioned above.

Then, Magento looks for a custom layout XML update set on the CMS page object

$layoutUpdate = ($page->getCustomLayoutUpdateXml() && $inRange)
    ? $page->getCustomLayoutUpdateXml() : $page->getLayoutUpdateXml();

and adds it directly to the list of updates to apply

$action->getLayout()->getUpdate()->addUpdate($layoutUpdate);

This means the scanning for the update tag is never done.

If I was trying to implement something like this, I'd add a new block in the CMS page's layout update XML, and then add the shared layout items programmatically in the blocks _prepareLayout method. (not sure off the top of my head if that would work w/r/t to timing, but that'd be where I'd start)

Layout handles <catalog_category_default> does not work in custom layout XML so it is not the way to go.

Instead, leave in custom layout XML field only XML which you want to apply to both products and categories.

You can put category specific layout XML in app/design/frontend/[YOUR_PACKAGE]/[YOUR_THEME]/layout/local.xml as follows:

<layout>
    <CATEGORY_18> <!-- 18 is category ID -->
        <!-- put category specific layout XML here -->
    </CATEGORY_18>
</layout>

P.S. <CATEGORY_18> layout handle is applied before custom layout XML specified in Admin Panel.