I got the solution of this problem. I am not sure if it is wrong or right but it is working fine for me.
We need to override "Magento\CatalogSearch\Model\ResourceModel\Fulltext\Collection" for this. and you can add your custom filter to "addFieldToFilter" function. If you do this, then your pagination and actual product count will be proper.
I hope this will be helpful for others. Let me know if you found any other way. Kudos if it solves your problem.
Why give the example using core files? Why not be helpful and show what everyone is looking for? The required steps to override the core.....???