{"id":1783,"date":"2025-06-11T11:07:36","date_gmt":"2025-06-11T11:07:36","guid":{"rendered":"https:\/\/hostrago.com\/knowledge-base\/?p=1783"},"modified":"2025-06-11T11:08:44","modified_gmt":"2025-06-11T11:08:44","slug":"how-to-block-bots-using-robots-txt-file-in-cpanel","status":"publish","type":"post","link":"https:\/\/hostrago.com\/knowledge-base\/how-to-block-bots-using-robots-txt-file-in-cpanel\/","title":{"rendered":"How to Block Bots Using robots.txt File in cPanel: Complete Guide"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Bots are everywhere on the internet \u2014 some helpful, others harmful. While search engine bots like Googlebot crawl your website to index content, malicious bots may scrape data, overload your server, or affect SEO negatively.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Fortunately, you can control how bots interact with your site using a simple file called <code>robots.txt<\/code>. In this guide, we\u2019ll show you <strong>how to block bots using robots.txt file in cPanel<\/strong> \u2014 step by step.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For more web hosting tips and security guides, visit the <a class=\"\" href=\"https:\/\/hostrago.com\/knowledge-base\/\">Hostrago Knowledge Base<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What is robots.txt?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file is a plain text file located at the root of your website that provides instructions to web crawlers (bots) about which pages or directories they are allowed or disallowed from accessing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: *<br>Disallow: \/private-folder\/<br><\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">In the example above, all bots are blocked from accessing <code>\/private-folder\/<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\ud83d\udca1 <em>Note: Not all bots obey <code>robots.txt<\/code>. Malicious bots may ignore these instructions, so additional server-level security may be needed for full protection.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why Block Bots Using robots.txt?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here are a few reasons you might want to restrict bots:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reduce server load caused by excessive crawling.<\/li>\n\n\n\n<li>Protect sensitive directories from being indexed.<\/li>\n\n\n\n<li>Prevent content scraping.<\/li>\n\n\n\n<li>Improve crawl budget for SEO by prioritizing important pages.<\/li>\n\n\n\n<li>Block unwanted bots that do not add value to your site.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">How to Block Bots Using robots.txt File in cPanel (Step-by-Step)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Log In to Your cPanel Account<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Visit your <a href=\"https:\/\/hostrago.com\/knowledge-base\/how-to-login-in-cpanel\/\">cPanel login URL<\/a>.<\/li>\n\n\n\n<li>Enter your username and password.<\/li>\n\n\n\n<li>Click <strong>Log In<\/strong> to access your dashboard.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Access the File Manager<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"508\" src=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-1024x508.webp\" loading=\"lazy\" alt=\"Screenshot highlighting File Manager in cPanel\" class=\"wp-image-1782\" srcset=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-1024x508.webp 1024w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-300x149.webp 300w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-768x381.webp 768w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-1536x761.webp 1536w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-highlighting-File-Manager-in-cPanel-2048x1015.webp 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Under <strong>Files<\/strong> section, click on <strong>File Manager<\/strong>.<\/li>\n\n\n\n<li>Navigate to the root directory of your domain, typically:<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Check for Existing robots.txt File<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"498\" src=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-1024x498.webp\" loading=\"lazy\" alt=\"Screenshot showing robots.txt file in File Manager\" class=\"wp-image-1781\" srcset=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-1024x498.webp 1024w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-300x146.webp 300w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-768x374.webp 768w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-1536x747.webp 1536w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-showing-robots.txt-file-in-File-Manager-2048x996.webp 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Look for the <code>robots.txt<\/code> file in the root directory.<\/li>\n\n\n\n<li>If it exists, you can edit it.<\/li>\n\n\n\n<li>If it does not exist, you will create a new file.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Create a New robots.txt File (If Needed)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Click on <strong>+ File<\/strong> at the top-left corner.<\/li>\n\n\n\n<li>Name the file <code>robots.txt<\/code>.<\/li>\n\n\n\n<li>Click <strong>Create New File<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Edit the robots.txt File<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"415\" src=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file-1024x415.webp\" loading=\"lazy\" alt=\"Screenshot of editing robots.txt file\" class=\"wp-image-1784\" srcset=\"https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file-1024x415.webp 1024w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file-300x121.webp 300w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file-768x311.webp 768w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file-1536x622.webp 1536w, https:\/\/hostrago.com\/knowledge-base\/wp-content\/uploads\/2025\/06\/Screenshot-of-editing-robots.txt-file.webp 1919w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Right-click on <code>robots.txt<\/code> and select <strong>Edit<\/strong>.<\/li>\n\n\n\n<li>Enter your bot blocking rules.<br>For example, to block all bots:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: *<br>Disallow: \/<br><\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To block a specific bot, such as AhrefsBot:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: AhrefsBot<br>Disallow: \/<br><\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To allow all bots but block specific directories:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: *<br>Disallow: \/private-folder\/<br>Disallow: \/temp\/<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Save Changes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>After adding your rules, click <strong>Save Changes<\/strong>.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Your new bot blocking rules are now active.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Additional Tips for Managing Bots<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u2705 Use <code>robots.txt<\/code> carefully \u2014 blocking important pages may affect SEO.<\/li>\n\n\n\n<li>\u2705 Combine <code>robots.txt<\/code> with other security measures like IP blocking or firewall rules.<\/li>\n\n\n\n<li>\u2705 Use Google&#8217;s <a>robots.txt Tester<\/a> to verify your file.<\/li>\n\n\n\n<li>\u2705 For sensitive data, use authentication and server-level restrictions \u2014 not just <code>robots.txt<\/code><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">How to Verify Your robots.txt Is Working<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Visit <code>https:\/\/yourdomain.com\/robots.txt<\/code> in your browser.<\/li>\n\n\n\n<li>Review the file to ensure your rules are correctly applied.<\/li>\n\n\n\n<li>Use webmaster tools (like Google Search Console) to test your robots.txt.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Need Help? Contact Hostrago Support<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you need assistance managing your <code>robots.txt<\/code> file or optimizing your hosting security, our support team is available 24\/7.<br>\ud83d\udc49 <a href=\"https:\/\/hostrago.com\/contact-us\">Contact Hostrago Support<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Or explore our <a class=\"\" href=\"https:\/\/hostrago.com\/knowledge-base\/\">Hostrago Knowledge Base<\/a> for more helpful guides.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Frequently Asked Questions (FAQ)<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. What is robots.txt?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file is a text file placed at the root of your website that gives instructions to web crawlers (bots) about which parts of your site they are allowed or disallowed from accessing.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. Why should I block bots using robots.txt?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Blocking bots using <code>robots.txt<\/code> can help:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reduce server load from unnecessary crawlers.<\/li>\n\n\n\n<li>Prevent content scraping.<\/li>\n\n\n\n<li>Improve SEO by focusing search engine crawlers on important pages.<\/li>\n\n\n\n<li>Block specific bots that may harm your website performance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3. Can all bots be blocked using robots.txt?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. While most legitimate bots (like Googlebot, Bingbot) follow <code>robots.txt<\/code> rules, some malicious bots may ignore these instructions. For full protection, you may need additional server-side security like firewalls, IP blocking, or bot management solutions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4. Where should I place the robots.txt file?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file should be located in your website&#8217;s root directory, typically:<br><code>\/public_html\/robots.txt<\/code><br>This allows bots to find and read the file when crawling your domain.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. How do I block all bots from my website?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To block all bots from accessing your website, add the following lines to your <code>robots.txt<\/code> file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: *<br>Disallow: \/<br><\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This tells all bots not to crawl any part of your website.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">5. Can I block specific bots only?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Yes. You can target specific bots using their user-agent names.<br>For example, to block AhrefsBot:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>User-agent: AhrefsBot<br>Disallow: \/<br><\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This allows you to block individual bots while still allowing others.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">7. Does blocking bots with robots.txt affect SEO?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Blocking good bots like Googlebot can negatively impact SEO because your content won\u2019t be indexed. It\u2019s recommended to only block bots that are unnecessary or harmful to your website.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">8. How can I test if my robots.txt is working?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You can test your <code>robots.txt<\/code> file using Google Search Console\u2019s <a href=\"https:\/\/rankmath.com\/tools\/robots-txt\/\" target=\"_blank\" rel=\"noopener\">robots.txt Tester <\/a>to ensure it\u2019s properly configured.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">9. Can I edit or update robots.txt anytime?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Yes. You can edit the <code>robots.txt<\/code> file in cPanel anytime. Changes take effect as soon as the file is saved and crawlers visit your site again.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">10. Where can I find more cPanel guides?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You can visit our full <a class=\"\" href=\"https:\/\/hostrago.com\/knowledge-base\/\">Hostrago Knowledge Base<\/a> for more detailed cPanel tutorials, security guides, and web hosting tips.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">How to Block Bots Using robots.txt File in cPanel is a simple yet powerful way to control how bots interact with your website. Whether you&#8217;re blocking harmful bots or fine-tuning your SEO, proper configuration of this file can save server resources and protect your content.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\ud83d\udc49 For more tutorials, visit our <a class=\"\" href=\"https:\/\/hostrago.com\/knowledge-base\/\">Hostrago Knowledge Base<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\ud83d\udc49 Ready for secure and reliable hosting? Check out our <a class=\"\" href=\"https:\/\/hostrago.com\/\">Web Hosting Plans<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Bots are everywhere on the internet \u2014 some helpful, others harmful. While search engine bots like Googlebot crawl your website to index content, malicious bots&#8230;<\/p>\n","protected":false},"author":1,"featured_media":1785,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[204,202,179,126,203,201,184],"class_list":["post-1783","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cpanel","tag-block-bots","tag-bot-management","tag-cpanel","tag-hostrago","tag-robots-txt","tag-seo-optimization","tag-web-hosting-security"],"menu_order":0,"_links":{"self":[{"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/posts\/1783","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/comments?post=1783"}],"version-history":[{"count":1,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/posts\/1783\/revisions"}],"predecessor-version":[{"id":1786,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/posts\/1783\/revisions\/1786"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/media\/1785"}],"wp:attachment":[{"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/media?parent=1783"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/categories?post=1783"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hostrago.com\/knowledge-base\/wp-json\/wp\/v2\/tags?post=1783"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}