Magento – Magento-2 How to get custom product collection in catalog search result based on particular category

I've used the Display Products Grouped By Subcategories extension on category list pages before with success.

Another solution before we found this extension was to get the collection in the block and use a special attribute to order the collection. The code was as follows (only works under PHP 5.3)

class Class_Name
{
    [...]
    protected function _getProductCollection()
    {
        [...]

        $collection = $this->_productCollection;

        $collection->addAttributeToSelect('category_sort_value');


        // Do some fancy sorting
        $collectionReflection = new ReflectionObject($collection);
        $itemsPropertyReflection = $collectionReflection->getProperty('_items');
        $itemsPropertyReflection->setAccessible(true); // Make it accessible

        $collectionItems = $itemsPropertyReflection->getValue($collection);

        usort($collectionItems, array("Class_Name", "cmp"));

        $itemsPropertyReflection->setValue($collectionReflection, $collectionItems);

        $itemsPropertyReflection->setAccessible(false); // Return restriction back
        // End them fancyness

        $this->_productCollection = $collection;

        return $this->_productCollection;
    }

    public function cmp($a, $b) 
    {
        $a_sort = (int)$a->getData('category_sort_value');
        $b_sort = (int)$b->getData('category_sort_value');

        if ($a_sort == $b_sort) return 0;
        return ($a_sort < $b_sort) ? -1 : 1;
    }
}

You can use this class example to extend the Block class for the product list in search and sort the results like that. I hope this will help you in the right direction.

Okay answering my own question here:

It seems the _applyProductLimitations() function of Mage_Catalog_Model_Resource_Product_Collection adds a WHERE condition automatically to the collection if the category is not an anchor.

public function addCategoryFilter(Mage_Catalog_Model_Category $category) {
        $this->_productLimitationFilters['category_id'] = $category->getId();
        if ($category->getIsAnchor()) {
            unset($this->_productLimitationFilters['category_is_anchor']);
        } else {
            $this->_productLimitationFilters['category_is_anchor'] = 1;
        }
...

If the category is not an anchor it attaches the condition here:

if (isset($filters['category_is_anchor'])) {
    $conditions[] = $this->getConnection()
        ->quoteInto('cat_index.is_parent=?', $filters['category_is_anchor']);
}

So to retrieve a list of products just in the "Default Category" or ROOT Category you can do the following.

$store_id = 1;
$root_category_id = Mage::app()->getStore(Mage::app()->getStore()->getId())->getRootCategoryId();

$collection = Mage::getResourceModel('catalog/product_collection')->setStoreId($store_id);
$model = Mage::getModel('catalog/product')->setStoreId($store_id);
$category_model = Mage::getModel('catalog/category');
$category = $category_model->load($root_category_id);
$category->setIsAnchor(true);
$collection->addCategoryFilter($category);

echo $collection->getSize();