FULLY CONFIGURABLE

A CRAWLER THAT ADAPTS TO YOUR SITE

Every website is different. SecurityBot's broken link crawler lets you fine-tune user agents, crawl speed, and scheduling so it works seamlessly with your infrastructure.

TL;DR: Set a custom user agent to avoid WAF and bot-detection blocks, configure crawl delay (in milliseconds) to prevent overwhelming your server or triggering rate limits, and toggle automated weekly crawls on or off. All settings are accessible from your site's settings page.

START FREE TRIAL

THREE SETTINGS, ZERO HASSLE

Configure your crawler in under a minute from your site's settings page

🕵️

Custom User Agent

Many websites use Web Application Firewalls (WAFs) or bot-detection services like Cloudflare and Sucuri that block requests from unknown user agents.

SecurityBot lets you set a standard browser user agent string so your crawl requests look like normal browser traffic instead of automated bot requests.

EXAMPLE USER AGENT

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

⏱️

Crawl Delay

Control the pause between each page request in milliseconds. This prevents your crawler from overwhelming your server or triggering rate-limiting protections.

A lower delay means faster crawls but higher server load. A higher delay is gentler on your infrastructure but takes longer to complete.

RECOMMENDED SETTINGS

Small sites: 200–500ms

Shared hosting: 500–1000ms

High-traffic sites: 100–300ms

📅

Weekly Scheduling

Toggle automated weekly crawls on or off with a single click. When enabled, SecurityBot will crawl your entire site once per week and alert you to any new broken links.

You can also trigger on-demand crawls at any time — perfect for checking things after a deployment or content update.

AUTOMATION

Enable once, get weekly reports automatically. No cron jobs or scripts required.

WHY CONFIGURATION MATTERS

A one-size-fits-all crawler causes problems. Crawling too fast can bring down a shared-hosting site. Using a generic bot user agent gets blocked by Cloudflare. SecurityBot gives you the controls to crawl any site reliably.

  • Avoid WAF Blocks

    Sites behind Cloudflare, Sucuri, or AWS WAF often block unknown bots. A custom user agent prevents false positives.

  • Prevent Server Overload

    Crawling hundreds of pages without a delay can spike CPU and memory on smaller servers. Crawl delay keeps things smooth.

  • Stay Under Rate Limits

    Many hosting providers and CDNs enforce request-per-second limits. Adjustable crawl delay keeps you within bounds.

  • Get Accurate Results

    A blocked or rate-limited crawl reports false broken links. Proper configuration means your results reflect reality.

Common Scenarios

Cloudflare-protected site

Set a Chrome user agent string. Increase crawl delay to 500ms to avoid triggering the "Under Attack" mode challenge page.

User Agent + Delay

Shared hosting (cPanel, Plesk)

Use a higher crawl delay (800–1000ms) to avoid hitting resource limits imposed by the hosting provider.

Higher Delay

Dedicated / VPS server

Lower delay (100–300ms) is usually fine. Your server can handle rapid requests without issues.

Lower Delay

Post-deployment check

Trigger an on-demand crawl right after deploying. No need to wait for the weekly schedule.

On-Demand Crawl

FREQUENTLY ASKED QUESTIONS

Common questions about crawler configuration

What user agent should I use?

For most sites, the default SecurityBot user agent works fine. If your site uses Cloudflare, Sucuri, or another WAF that blocks bots, set the user agent to a standard Chrome or Firefox browser string. This makes requests look like regular browser traffic and prevents the WAF from blocking the crawl.

What crawl delay should I set?

It depends on your server. For dedicated servers or cloud hosting, 200–300ms is usually fine. For shared hosting, 500–1000ms is safer. If you're unsure, start with 500ms and lower it if crawls take too long. You can always check your server's CPU and memory during a crawl to find the right balance.

Can I change settings between crawls?

Yes. Settings take effect on the next crawl, whether automated or on-demand. You can adjust user agent, crawl delay, and scheduling at any time from your site's settings page. Changes don't affect crawls that are already in progress.

Will the crawler get my IP blocked?

With proper configuration, no. The crawl delay prevents rapid-fire requests that trigger rate limiting, and a custom user agent avoids bot-detection blocks. SecurityBot also crawls from consistent IP addresses, so you can whitelist them in your firewall if needed.

What happens if I disable automated crawls?

Disabling automated crawls simply stops the weekly scheduled scan. Your previous crawl results remain available, and you can still run on-demand crawls whenever you want. Re-enabling automated crawls resumes the weekly schedule.

CRAWL YOUR SITE, YOUR WAY

Fine-tune user agent, crawl speed, and scheduling to detect broken links without disrupting your site.

START FREE TRIAL