Server error (5xx) in Search Console: find the cause and fix it

Server error (5xx) means Googlebot requested the URL and your server answered with a 500-level status code, or failed to answer in time, so Google could not crawl the page. Google slows its crawling when it sees these errors and drops already indexed URLs if they continue, so the job is to find the cause while the pages are still in the index.

The difficulty is that these errors are often intermittent. The page loads when you test it, because the failure only happens under crawl load, at certain hours, or after an idle period.

What does "Server error (5xx)" mean?

It means the server failed Googlebot's request. Search Console's help says: "Your server returned a 500-level error when the page was requested." Google's crawler documentation adds the consequence: "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped."

Timeouts count too. Google says it "treats network timeouts, connection reset, and DNS errors similarly to 5xx server errors," and for those, "already indexed URLs that are unreachable will be removed from Google's index within days." Recovery is gradual: "Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site."

For how this row fits with the rest of the report, see the page indexing report guide.

When did the errors happen?

The Crawl Stats report shows when. Open Settings, then Crawl stats, and click the Server error (5XX) row in the By response table: the chart shows approximately when the errors occurred, and the example list shows which URLs failed and the time of each request. Those timestamps are what you match against your server logs.

1. Read the host status at the top. A warning means Google hit a significant availability issue in the last 90 days, and a recent one means within the last week.
2. Click host status and open Server connectivity, which charts when "your server was unresponsive or did not provide a full response for a URL during a crawl."
3. In the By response table, check Server error (5XX), Page timeout and Page could not be reached. The help notes that requests which "never reached the server ... will not appear in your logs."
4. Click each row for example URLs and exact crawl times.
5. Check the average response time chart for spikes at the same times, and note whether they line up with the errors.

The Crawl Stats report works only on root-level properties, meaning a Domain property or a URL-prefix property at the root. Google says it counts "most crawl requests," so treat it as a strong sample rather than a full log.

What causes 5xx errors that you cannot reproduce?

Anything that fails only under certain conditions: a burst of requests, a cold instance, a slow dependency, or a security rule. Googlebot fetches many URLs in a short window, often at hours when you are not testing, so it finds limits a single browser visit never touches. Match the error times in Crawl Stats to your logs and look for these causes.

Rate limiters deserve their own look. Google's crawler documentation says its crawlers "treat the 429 status code as a signal that the server is overloaded, and it's considered a server error." A per-client limit tuned for abusive traffic can answer Googlebot's normal crawl pace with 429 and slow crawling across the whole site. Exempt verified crawler IPs from per-client limits, and keep 429 for clients you have actually identified as a problem.

Google's network error guidance suggests starting with the firewall: "There may be an overly-broad blocking rule set. Make sure that Google IP addresses are not blocked by any firewall rule."

Were the failing requests really from Googlebot?

Check before you change anything, because other bots send Googlebot's user agent. Google's verification method is a reverse DNS lookup on the IP address from your logs, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, and a forward lookup confirming the name maps back to the same IP.

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

That is Google's own example. For checks at scale, Google publishes its crawler IP ranges as JSON files in CIDR format, starting with common-crawlers.json for crawlers such as Googlebot. If the failing requests do not verify, they are not what Search Console is reporting, and the errors Google saw are in the requests that do.

What status code should planned maintenance return?

A 503 Service Unavailable with a Retry-After header, for as short a time as possible. Google's guidance for disabling a site urgently for "1-2 days" is "an informational error page with a 503 HTTP response status code." A 200 placeholder page or a 404 tells crawlers the content is gone or changed; a 503 tells them it is temporarily unavailable.

HTTP/1.1 503 Service Unavailable
Retry-After: 3600

Retry-After takes either a number of seconds or an HTTP date. Google's 2011 post on planned downtime says it is a header "which Googlebot may use to determine when to recrawl the URL." Three limits apply:

1. Keep it short. Google says "returning 503 or 429 for more than 2 days will cause Google to drop those URLs from the index," and that Googlebot retries these URLs "for about 2 days."
2. Never return 503 for robots.txt. Google says it "blocks all crawling."
3. Confirm the response with curl before relying on it. Google's own example is curl -I -X GET "https://www.example.com/", which should print HTTP/1.1 503 Service Unavailable.

The Crawl Stats help gives the same limit for throttling an overloaded server: "Be sure not to return 503 or 429 for more than two or three days."

Why do intermittent errors need scheduled checks?

Because a single manual test samples one moment, and intermittent 5xx errors live in the other moments. You open the page, it returns 200, and the problem looks solved while Googlebot keeps hitting timeouts during its next burst at 3 a.m. Only repeated checks over time show whether a route fails, how often, and when.

Google's own reports lag behind reality as well. The Page indexing report reflects Google's last crawl, and Crawl Stats shows errors after they happen, so both tell you about a failure days later. What closes that gap is a check that runs on a schedule against your own site and alerts you when a response fails, before the next crawl repeats it. The uptime vs SEO monitoring guide compares the two kinds of checks, and the SEO monitoring guide for small sites covers what else to watch.

How do you confirm the fix worked?

Watch the error rate fall in Crawl Stats, then validate. A single clean test proves nothing about an intermittent error, so give it enough crawl time to show a trend before you declare it fixed.

1. After the fix, watch the Server error (5XX) share and the average response time in Crawl Stats for a week or two.
2. Run Test live URL on a few affected URLs in URL Inspection. The Page fetch line should succeed.
3. Open the Server error (5xx) row in the Page indexing report and click Validate fix. Google says validation "typically takes up to about two weeks."
4. If validation fails, open See details and check the failing URLs' crawl times against your logs again.

Keep watching after the fix

An intermittent error you fixed once can come back with the next deploy, dependency change or traffic spike, and Search Console will tell you days later. LaunchScaler's Watch is built for that gap. For $29/mo per website it re-scans your site every two weeks and emails you a diff of what moved, runs uptime checks on your health endpoint in between, and sends an alert the moment the site goes down or a check drops below the bar. The scan includes checks for pages returning 5xx and for slow or timed-out server responses, and every finding comes with its evidence. Start watching to hear about a failing page before Googlebot does.