Remove Price Block via Event Observer in Magento

I can pretty easily remove various blocks from a given page through an event observer on, for instance, the controller_action_layout_general_blocks_before action. Here is an example:

    $action = $observer->getEvent()->getAction();
    $layout = $observer->getEvent()->getLayout();
    $update = $layout->getUpdate();

    $update->addUpdate('
        <reference name="product.info">
            <remove name="product.info.addtocart" />
        </reference>
    ');

    $layout->generateXml();

However, I can't find a way to remove the price block, even straight from the catalog.xml file. It seems like Mage_Catalog_Block_Product->getPriceHtml($product) creates the block dynamically. (I think it ultimately uses Mage_Catalog_Block_Product_Abstract->_getPriceBlock() which is in a file I can't really override.)

My goal here is to remove the price block from catalog view pages in certain situations, but I can't seem to remove that block. Instead, it seems I have to change the template file itself, which I want to avoid doing for this particular task.

(Magento 1.9.0.1)

This is the solution that I came up with Fabian Blechschmidt's help.

In my config.xml

<frontend>
    <events>
        <core_block_abstract_to_html_before>
            <observers>
                <custom_to_html_before>
                    <class>mymodule/observer</class>
                    <method>coreBlockAbstractToHtmlBefore</method>
                </custom_to_html_before>
            </observers>
        </core_block_abstract_to_html_before>
    </events>
</frontend>

In my observer model

public function coreBlockAbstractToHtmlBefore(Varien_Event_Observer $observer) {
    // Simple example logic
    $hidePrice = true;

    $block = $observer->getBlock();

    if (get_class($block) == 'Mage_Catalog_Block_Product_View') {
        if ($hidePrice) {
            $block->getProduct()->setCanShowPrice(false);
        }
    }
}