Such as, among the web pages which i manage had what’s called a “site bloat” trouble. To put it differently, we had additional indexed web pages than we must always.In order to avoid unwanted information in the search indexes, site owners can instruct spiders never to crawl certain information or directories from the standard robots.txt file