United State - USA Ankit Kumar
14-03-18 183 Hits

Robots.txt is a notepad file which makes to prevent any search engine robot/spider/bot to crawl a domain or its any part(Pages). The Website owners make this file when they want to hide any information from the search engine like Google, Bing, Yandex, Baidu etc. If you are also a website owner and want to prevent search engine to come into your website then make robots.txt file and upload it on public_html folder.

How to Start a Robots.txt file:

To Prevent a Domain for all Search Engine Bot

User-agent: *
Disallow: /

Here User-agent command is used to show search engine bot name and * is for all the search engine bot present in the whole World. Disallow command is for preventing robots to crawl and index the website and / is for the main domain.

To Prevent a Domain for a Specific Search Engine like Google

User-agent: Googlebot
Disallow: /

Note: In front of user-agent always type the robots name of any search engine. For example: Yahoo robots name is slurk so that type slurk not Yahoo. To know the Bot name of some popular search engine visit https://www.keycdn.com/blog/web-crawlers/

To Prevent any page in Domain

User-agent: *
Disallow: /xyz/
Disallow: /xyz if the page don't have any subpages.

To prevent any page under any category or folder

User-agent: *
Disallow: /categery/xyz.html

To Delay Crawling of any search engine

User-agent: *
Crawl-delay: 4 (means after four days crawler revisit your website).

Note: Allow command is only applying for Googlebot.

Thus, you can disallow many pages, folders and catageries in your website by using the robots.txt file. Mainly it is used to prevent indexing of that matter which can penalize your website on the search engine.

delhi, india

  Send Message


Related ads

What Kinds of Keywords According to Google Search?

First of all, we should know that what is the keywords. Keywords are the user queries which type on the Google search bar to find any service, product etc. Now we start the topic: According to the Search Engine work; Google increase and decrease the… United State - USA

Sonner or Later Remove Failure to Dial Outlook Support Number

Having clicked on Outlook login navigational panel, a few outlook customers do not find the new way to approach its dashboard panel. Appearance of this unwanted case indicates you are not available to authenticated login service to deliver the documentary… United State - USA

Top 150 Free Profile Creation Website With High DA PA

If you are searching for Profile Creation Websites then now you have reached on the right place. Here follow the below mention links and do profile creation on high PageRank Websites to get the inbound links (backlinks) for own website.… United State - USA

What is Black Hat SEO and It's Some Tactics?

Black Hat SEO: It is a technique to improve our website ranking instantly on Google without following Search Engine Guidelines. In other words, it is the process of technical cheating by which you can rank our website in a small time in free on any Search… United State - USA

What is the Difference Between Outbound Link and Inbound Link?

According to the hyperlinks(Connect one website in another website content by a text such as Click Here, Read More and any other text. It is called Hyperlink.), There are two types of link. Outbound Link: When we target another website in our website… United State - USA