> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://help.weblium.com/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# Accessing robots.txt

Google uses software known as web crawlers that look at webpages and follow links on those pages to bring data about them back to Google’s servers. A robots.txt file tells these crawlers which pages or files the crawler can or can't request from your site. 

You can learn more about the robots.txt in [the Google Search General documentation](https://developers.google.com/search/docs/advanced/robots/intro). 

See how you can access your robots.txt file for the site.

# Accessing robots.txt

1. We generate the robots.txt file automatically. To access it, simply add /robots.txt to your domain name. For example:

https://weblium.com/robots.txt

2. If the site is **open** for indexing in the search engines, the robots.txt file will look this way — with a link to the sitemap:

![](https://storage.crisp.chat/users/helpdesk/website/fa84b4d83a50e800/group-51_14x83qb.png)
|| ${color}[#ffa800](Note:) /.sw_/_host_/_replacer_ — is a technical record that our system adds by default.

3. If the site is **blocked** from being indexed in the search engines, the file will look this way:

![](https://storage.crisp.chat/users/helpdesk/website/fa84b4d83a50e800/robots-closed-new_nafibz.png)

4. You can hide or open the site to the search engines anytime. Go to the **Settings** of your website and proceed to the **General info** tab, then go to the **Visibility in the search engines** section:

![](https://storage.crisp.chat/users/helpdesk/website/fa84b4d83a50e800/image_vjvmk.png)

|| ${color}[#ffa800](Note:) If you are looking for a way to hide a site or a page from the search engines, check out the [Hiding website from search engines](https://help.weblium.com/en-us/article/hiding-website-from-search-engines-12klrfb/) and [Hiding a page from search engines](https://help.weblium.com/en-us/article/hiding-a-page-from-search-engines-1mhnaff/) articles.