Why isn't my website importing?
If a website you added finished with no pages indexed, its server most likely blocked our crawler. Here is how to tell, and how to let us in.
What's happening
Some sites sit behind Cloudflare, a WAF, or bot protection that challenges or blocks automated requests. When that happens our crawler can't read your pages even though you added the URL correctly, so the import finishes with zero pages and you get a "Crawl blocked" notification. The block is almost always temporary or fixable, not a sign the page is gone.
How to tell this is the cause
- The source shows as done but has 0 pages.
- You received a "Crawl blocked" notification in your inbox.
- The page opens fine in your own browser (a person clears the bot check; an automated crawler does not).
How to let our crawler in
Allow BreezaroBot on whatever protects your site, then re-crawl. You can identify us three ways:
- User-Agent contains BreezaroBot.
- Requests come from a single IP address: 167.235.18.41.
- That IP's reverse DNS is crawler.breezaro.com (forward-confirmed, so it cannot be spoofed).
On Cloudflare
Open Security → WAF and add a Skip (allow) rule matching User-Agent contains BreezaroBot or IP equals 167.235.18.41. We are also enrolling BreezaroBot in Cloudflare's Verified Bots program.
Check robots.txt
Make sure your robots.txt does not disallow BreezaroBot (or all bots) on the pages you want indexed.
Still not importing?
If it still comes back empty after allowlisting us, the protection may be stricter than a User-Agent or IP rule, for example an interactive challenge on every request. Email support@breezaro.com with the site URL and we will help, or temporarily lower the protection level while the first crawl runs.
Full crawler identity, egress IP, and reverse-DNS details are on our public crawler page: https://breezaro.com/bot