The robots.txt file is a simple text file located in the root directory of your website, which allows search engine bots to crawl and index your website. The robots.txt file also controls how and which files the bots should access, so configuring it properly is essential thing. You can use the robots.txt file to prevent bots from indexing your website in search engines.
In this article, we will show you how to block bots by editing your robots.txt file. Follow the steps below to get started.
Step 1: Log in to your cPanel account.
Step 2: Open the file manager.
Step 3: Create a file named robot.txt and add the code below to block the bot. This code will block the bot from the Baidu search engine.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule .* – [F,L]
Step 4: This is it. Now, Baidu will not be able to bot your website anymore.
