Magento – Incorrect product count in layered navigation after overriding list product collection

I have customized(override) product list collection getProductCollection and filtered collection as required.

public function getProductCollection()
    {
        if (isset($this->_productCollections[$this->getCurrentCategory()->getId()])) {
            $collection = $this->_productCollections[$this->getCurrentCategory()->getId()];
        } else {
            $collection = $this->collectionProvider->getCollection($this->getCurrentCategory());
            $this->prepareProductCollection($collection);
            $this->_productCollections[$this->getCurrentCategory()->getId()] = $collection;
        }

        /* custom filter conditions */
        $productUniqueIds = array(10,11,12,13);
        $collection->addAttributeToFilter('entity_id',array('in' => $productUniqueIds));
        /* custom filter conditions end here */

        return $collection;
    }

Based on above products conditions, My result for listing page will be fine but layered navigation result is not proper, displays wrong filter attributes.

I am facing an issue in the layered navigation product count after applying filters on product collection. All counts are wrong. What I guess is I need to apply custom filters on layered navigation's collection as well, but I am not able to find it so far.

I have debugged and reached till _createItem function in vendor/magento/module-catalog/Model/Layer/Filter/AbstractFilter.php file. Product counts are coming from there, but I couldn't find from where exactly product collection gets returned.

Please share your ideas if you have. Any help would be appreciated.
Thanks