ChatleadrBot
ChatleadrBot is the crawler Chatleadr uses to read a website when a Chatleadr customer asks it to.
Chatleadr makes chatbots that businesses put on their websites. ChatleadrBot reads a customer’s site so their bot can describe the business and link visitors to the right page.
When ChatleadrBot visits
ChatleadrBot visits a site only when a person starts it. Nothing runs on a schedule. A Chatleadr customer starts it in one of three ways:
- When they set up a bot from their website, ChatleadrBot reads its home page once, with a few pages and stylesheets that page links to.
- When they create a bot with their website’s address, ChatleadrBot reads the site’s pages once, to list the pages the bot can link visitors to.
- When they refresh the list of pages their bot links visitors to, ChatleadrBot reads the site’s pages once.
A customer can refresh a bot’s page list at most once every 10 minutes, unless a refresh fails on Chatleadr’s side.
Chatleadr’s own team also starts page list refreshes when testing ChatleadrBot, with the same user agent, rules and limits.
The site is the one the customer gives as their bot’s website. That is usually their own site, or a client’s site when the customer is an agency. If that site’s home page redirects to another site, ChatleadrBot reads the other site instead.
User agents
ChatleadrBot sends one of two user agent strings, one for each kind of visit:
ChatleadrBot/1.0 (+https://chatleadr.com/bot)when a bot’s page list is refreshed.Mozilla/5.0 (compatible; ChatleadrBot/1.0)when a customer sets up a bot.
Both contain ChatleadrBot, so one rule on your server can match both.
What ChatleadrBot requests
When a customer sets up a bot, ChatleadrBot requests up to six addresses, and follows their redirects:
- Your home page.
- Up to two contact or about pages on your site that your home page links to.
- Up to three stylesheets your home page links to, wherever they are hosted.
When a customer refreshes their page list, ChatleadrBot requests these, in this order:
/robots.txt.- Your home page.
- The sitemaps your robots.txt names. If it names none that can be read, ChatleadrBot tries
/sitemap.xml, and/sitemap_index.xmlwhen/sitemap.xmlis not a sitemap. ChatleadrBot reads up to 25 sitemap files. - Your other pages, up to 200 pages in all, your home page included. These are the pages your sitemaps list or, with no sitemap, the pages your home page links to, one level deep.
During a refresh, ChatleadrBot skips addresses that look like blog posts, products, tag and category archives, or files such as images.
During a refresh, ChatleadrBot requests only https:// addresses. ChatleadrBot reads an http:// address on your site, such as one an older sitemap lists, at the same address over https://. If a redirect leads to an http:// address, ChatleadrBot requests the same address over https:// instead. It stops there when that address names its own port, such as :8080, or leads back to an address already requested. Nothing is requested over http://.
A refresh makes no more than two requests at a time, waits up to 10 seconds for each, and stops after 90 seconds.
What Chatleadr keeps
From a page list refresh, Chatleadr keeps a list of your pages for the customer’s bot to link to. For each page, Chatleadr keeps:
- Its address and title.
- Its meta description or
og:description, or the first paragraph of its text when it has neither. - The date your sitemap gives it, if any.
- For up to 12 sections that have their own
#address, each section’s title and the first sentence or two of its text.
Apart from your home page, a page whose robots meta tag or X-Robots-Tag header says noindex is left off the list.
From a setup read, Chatleadr keeps two things when the customer goes on to create the bot. An AI model writes a short summary of what the pages say about the business, which becomes the starting knowledge for the customer’s bot. Chatleadr also keeps one colour, read from the home page and its stylesheets, for the bot’s chat window.
How ChatleadrBot reads robots.txt
When a customer refreshes their page list, ChatleadrBot reads /robots.txt before anything else and follows the robots exclusion standard:
- If your file has a group for
ChatleadrBot, ChatleadrBot follows that group and ignores the*group. Otherwise, ChatleadrBot follows the*group. - A
ChatleadrBotgroup replaces the*group entirely, so copy into it any*rules you also want ChatleadrBot to follow. - ChatleadrBot never requests a page your rules disallow, including a page a redirect leads to.
- Sitemaps are the exception. ChatleadrBot reads a sitemap that your robots.txt or a sitemap index names even when a rule covers its address. If a named sitemap redirects, ChatleadrBot checks the new address against your rules before requesting it.
- If the connection to your domain fails, ChatleadrBot tries
/robots.txtonce more on the other form of your domain, with or withoutwww.. If that address answers, ChatleadrBot reads your site there. - If
/robots.txtanswers with a 5xx or 429 error, or cannot be reached, ChatleadrBot reads nothing more. If it answers 404 or another 4xx, ChatleadrBot reads the whole site as allowed. - ChatleadrBot does not read
Crawl-delay. Its own limits are two requests at a time and 90 seconds in all.
When a customer sets up a bot, ChatleadrBot does not read robots.txt. To refuse that visit too, block the user agent on your server, as the next section shows.
How to block ChatleadrBot
To stop ChatleadrBot reading any page when a customer refreshes their page list, add this group to your robots.txt:
User-agent: ChatleadrBot
Disallow: /
ChatleadrBot then requests only /robots.txt, and Chatleadr keeps nothing from the refresh. A page list read before you added the rule stays as it was.
To keep one page other than your home page off the list, and let ChatleadrBot read the rest, add this tag to that page’s <head>:
<meta name="chatleadrbot" content="noindex">
ChatleadrBot still requests the page, but Chatleadr does not keep it. A robots meta tag that says noindex does the same, and applies to search engines as well.
To refuse both kinds of visit, block requests whose User-Agent header contains ChatleadrBot on your web server or firewall. The customer then sees that their site could not be read.
Contact
To ask about ChatleadrBot, email support@chatleadr.com with your site’s address.