Parts of a product page not indexed

[rolf _wouters]

Hi, I'm am currently experiencing an issue with all my product pages on a Magento2 site.

The description on top of the page is indexed in Google. Further down the page we use description tabs, with 3 tabs. The text under those tabs, is not indexed in Google.

The url of an example productpage is https://mijnlabtest.nl/cortisol-dagprofiel-test.html

Is it possible that somehow the crawlers cannot enter te text under those tabs? I have never seen pages partly being indexed before.

I hope somebody has experience with this problem.

[Bhanu Periwal]

Hi @rolf _wouters 

In admin Go to:

Content -> Design -> Configuration

Edit Global settings

Select Default Robots value to "Noindex, Nofollow"

Flush cache.

In addition You can also disallow all bots not to crawl your website by adding robots.txt in your project root directory and pub directory
robots.txt will contains following lines:

User-Agent: *
Disallow: /

If issue resolve, please click on 'Kudos' & Accept as Solution!

[rolf _wouters]

Hi Bhanu @Bhanu Periwal ,

thanks for your reply. 

Will that not overwrite all indexrules?

So, in that case pages like the privacy policy will get indexed again?

Regards, Rolf