Open Mon - Fri 09:00-17:00
Email [email protected] Call Now! +44(0)1689 602 248
Open Mon - Fri 09:00-17:00
Email [email protected] Call Now! +44(0)1689 602 248

Controlling crawling and indexing now documented on code.google.com

Webmaster level: All

Do you know how Google’s crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot’s favorite song? The answers to these questions (except for the last one :)), along with lots of other information about controlling the crawling and indexing of your site, are now available on code.google.com:

Controlling crawling and indexing

Now site owners have a comprehensive resource where they can learn about robots.txt files, robots meta tags, and X-Robots-Tag HTTP header directives. Please share your comments, and if you have questions you can post them in our Webmaster Help Forum.


Google Webmaster Central Blog

About the author

Blue Square Management are a web design and SEO Digital Marketing Agency based in Orpington and Bromley on the borders of South East London and Kent. We help businesses get more leads, increase sales and grow their businesses. Basically, Blue Square Management helps you get more phone calls and sell more products!We do by improving your search engine rankings and getting your found when people are looking for your products or services. We do this through online marketing, which includes SEO (Search Engine Optimisations), Local SEO, Web Design, PPC Management services, Online Reputation Management, Video Production Services, online Video Marketing and Social Media Marketing.Need help with your online marketing? Want more customers? Sell more products? Contact London & Kent's lead online digital marketing company, Blue Square Management to find out how we can help your business.