Create and Upload robots.txt

What is robots.txt file ?

The robots.txt is a text file which has .txt extension. It is used to tell search engines (like Google, Yahoo and Bing) about the web pages that it should not be crawled and indexed and prevent search engines to crawl to specific area like feeds, trackbacks for security reason, wordpress admin pages and avoid duplicate content because that web pages are not so important for you.

How to create robots.txt file?

i) Make a new text document (notepad).

ii) Write your robots.txt.


For example :

User-Agent : *

Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /search?

User-agent: Googlebot-Image
Disallow: 

User-agent: Mediapartners-Google
Disallow:

iii) Rename the document as robots. Since text document has .txt extension so the file may be called as robots.txt.

iv) Then upload this file to the root directory of your website.

How to upload robots.txt file to the root directory ?

i) Open your FTP client such as Filezilla.

ii) Login to your FTP client.

iii) Go to Public_html folder.

iv) Upload your robots.txt file here.

v) Now you can check this by opening http://yourwebsite.com/robots.txt.


you are done.
It will show the content of your robots.txt file.