Crawlability guides

Crawlability · 9 min read

301 vs 302 Redirects for SEO: Which Redirect to Use and When

Learn what 301, 302, 307 and 308 redirects mean, how Google treats permanent and temporary redirects, and how to pick and test the right one.

Crawlability · 9 min read

410 vs 404 for Removed Pages: Which Status Code to Return

Decide whether a deleted page should return 404, 410 or a 301 redirect, see how Google treats each, and configure and test the response on your server.

Crawlability · 10 min read

Blocked by robots.txt in Search Console: How to Fix It Safely

Find which robots.txt rule triggers the Blocked by robots.txt status, decide if the block is intended, edit the file safely and validate the fix.

Crawlability · 3 min read

Cloudflare Blocking Googlebot or Bingbot: How to Diagnose

Find out whether Cloudflare is challenging real Googlebot or Bingbot, verify the crawler, then make the smallest rule change that restores access.

Crawlability · 9 min read

Crawl Budget Explained: When It Matters and When It Does Not

Understand crawl capacity and crawl demand, decide whether crawl budget affects your site, and use Crawl Stats to remove wasted URL requests.

Crawlability · 8 min read

Custom 404 Page for SEO: A Helpful Page With the Right Status

Build a custom 404 page that helps visitors recover, returns a real 404 status code, and is configured correctly on nginx, Apache and ASP.NET Core.

Crawlability · 10 min read

Faceted Navigation SEO: Stop Filters Creating Endless URLs

Decide which filter and parameter URLs deserve to be crawled, block or consolidate the rest, and return clean 404s for empty combinations.

Crawlability · 2 min read

Find Broken Internal Links and Redirects Across Your Website

Crawl your public pages, trace each broken internal link to its source or shared template, then fix it at the source and re-crawl to confirm.

Crawlability · 2 min read

HTTP 500 and 503 Errors: Find the Real Website Problem

Reproduce 500 and 503 errors on the failing URL, trace them in server logs, fix the shared cause and confirm crawlers and visitors get normal pages.

Crawlability · 9 min read

Log File Analysis for SEO: See How Crawlers Use Your Website

Find your server access logs, filter and verify Googlebot and Bingbot requests, and read status codes, wasted crawls and errors with simple commands.

Crawlability · 10 min read

Noindex vs Robots.txt: Which Keeps a Page Out of Search?

Learn the difference between noindex and robots.txt, pick the right one for each page, and avoid the common mistake of using both on the same URL.

Crawlability · 10 min read

Orphan Pages: How to Find Pages With No Internal Links and Fix Them

Compare your sitemap, CMS and crawl to find orphan pages, then decide whether to link, merge, redirect or remove each one, and verify the result.

Crawlability · 9 min read

Pagination SEO Best Practices: Paged Lists Google Can Crawl

Give every page of a long list its own URL, link the pages with real anchors, set the right canonical tags and test the result in Search Console.

Crawlability · 2 min read

Redirect Chains, Loops, and Incorrect HTTP Status Codes

Trace every hop of a redirect, find which layer adds each rule, and point old URLs straight to one final address, then update links and sitemaps.

Crawlability · 3 min read

Robots.txt Blocking Google or Bing? Diagnose It Step by Step

Read your live robots.txt, test key URLs against its rules, tell a crawl block from a page-level exclusion, and repair an accidental block narrowly.

Crawlability · 9 min read

Site Architecture SEO: Plan Your Pages and URL Structure Before Launch

Group pages into clear sections, keep important pages a few clicks from home and set stable, readable URL rules before a site goes live.

Crawlability · 8 min read

Verify Googlebot and Bingbot: Tell Real Crawlers From Impostors

Check whether a request claiming to be Googlebot or Bingbot is genuine using reverse and forward DNS lookups, published IP ranges and safe blocking rules.

Crawlability · 8 min read

WWW vs Non-WWW: Choose One Canonical Host and Redirect the Other

Pick www or the bare domain, redirect the other with a 301, align canonicals, sitemaps and Search Console, and test all four host and protocol variants.

Crawlability · 10 min read

X-Robots-Tag Header: Noindex PDFs, Images and Other Files

Add an X-Robots-Tag header on Apache, Nginx or IIS to keep PDFs, images and other non-HTML files out of search results, then verify it with curl.