How to Block Bots using Robots.txt File in cPanel?

Robot.txt file is a simple text file created on a root directory of your website which allows the search engine bots to crawl your website to index it. robot .txt file also controls how and which files the bots should access so configuring it properly is an essential thing. You can use the robot.txt file to prevent bots from indexing your website in search engines.

In this article, we will show you how to block bots by editing your robot.txt file. Follow the below steps to get started.

  1. Login to your cPanel account.

  2. Open file manager.

    File Manager

  3. Create one file name robot.txt and add the below code to block the bot. This code will block the bot from the Baidu search engine.

    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
    RewriteRule .* – [F,L]
    


    Open the file


    edit the file
  4. This is it. Now, Baidu will not be able to bot your website anymore.

  • 0 Users Found This Useful

Was this answer helpful?

Related Articles

How to change max_input_vars in cPanel?

PHP collects the input variables from the cookies enabled on-page from the GET and POST requests....

How to check cPanel Disk usage from cPanel disk usage tool?

You may require to check your cPanel account Disk-space regularly. Disk usage is one of the...

How to set password protection in cPanel directory

In this article, we will show you how to add password protection to your cPanel directories....

How to Fix: check_mysql Has Determined That There Are Corrupted Database Tables (eximstats)

In this article, we will discuss how to resolve the error: check_mysql Has Determined That There...