THE SHORT ANSWER
Analyse server logs to answer a specific question about requests and delivery, such as whether verified crawlers encounter errors or spend time on unwanted URL variants. Logs show observed requests, not indexing or ranking decisions. Combine them with a URL inventory and search reports before deciding which technical change is justified.
For the broader picture of technical foundations, useful content and visibility measurement, read the pillar guide SEO, GEO and AEO for SaaS: visibility that supports buying decisions.
Frame a question and a comparable period
Choose a decision the analysis should support: investigating slow discovery after publication, diagnosing repeated server errors or evaluating a new filter policy. Define the relevant hosts, routes and time period. Include a comparison period only when logging coverage and site conditions are sufficiently comparable. Record migrations, outages and releases that could explain changes. Avoid collecting every field merely because storage is available. A narrow question helps determine the minimum data needed and prevents a large export from becoming an unfocused dashboard without a clear operational consequence. Explore this further in the guide Prioritising crawl budget on large SaaS websites.
Understand which layer recorded each request
Identify whether records come from the CDN, load balancer or application server. Cached requests may appear at the edge without reaching the origin, while application logs may omit requests blocked earlier. Document time zones, sampling, retention and missing periods. Preserve the original URL sufficiently to investigate parameters, but handle personal data and query strings under your organisation’s data rules. Never publish raw logs containing sensitive information. If coverage is incomplete, state that limitation alongside the finding rather than treating absent requests as proof that no crawler attempted access. Explore this further in the guide Verifying crawler access through CDNs and firewalls.
Verify crawler identities before grouping
A user-agent string can be copied. For Google requests, use its documented verification methods, such as matching published address ranges or validating reverse and forward DNS. Apply the relevant provider’s current guidance for other crawlers instead of inventing a universal identity rule. Retain an unverified category when evidence is insufficient. Separate automatic crawlers from user-triggered fetchers where possible because their purposes and policies differ. This prevents a burst of impersonated traffic from driving an incorrect decision to block a legitimate search service or relax protection unnecessarily.
Join requests to meaningful URL families
Group verified requests by template, language, status class and parameter pattern. Compare valuable published pages with unwanted variants rather than presenting total bot requests as a success metric. Investigate repeated redirects, errors and expensive endpoints, then inspect individual examples. Normalise only after retaining the detail needed to explain differences. Join publication dates cautiously: a first observed request is only first within your available log coverage. It does not prove the first-ever discovery date, and a successful response does not show that the content entered a search index.
Illustrative example: delayed documentation discovery
Imagine an illustrative documentation site where editors suspect that filters delay discovery of new guides. Edge logs show verified requests repeatedly reaching parameter permutations while several new guides have weak internal links. The team improves those links and bounds unnecessary variants, then observes subsequent periods with comparable coverage. It does not claim that one pattern alone proves a crawl-budget bottleneck. The scenario illustrates combining technical and navigation evidence, not a measured performance result or a guaranteed way to accelerate every new article’s indexing.
Convert observations into bounded changes
Report the question, coverage, verified population, affected URL families and recommended action together. Assign a specific owner and a follow-up test. If the evidence shows delivery failures, fix delivery; if it shows harmless activity, avoid creating a policy change simply to reduce a chart. Recheck after relevant deployments and document unresolved uncertainty. Logs can establish that a request reached a layer and received its recorded response. They cannot independently establish what the crawler understood, whether content was indexed or why a ranking changed. Keep those distinct conclusions separate in stakeholder reporting.
INTERACTIVE / FROM READING TO ACTION
Three checkpoints for your implementation
Choose a checkpoint and track your reading progress. Checks are neither saved nor sent and do not score your business.
CHECKPOINT 01
Verify crawler identities before grouping
A user-agent string can be copied.
Explore in the article ↗CHECKPOINT 02
Join requests to meaningful URL families
Group verified requests by template, language, status class and parameter pattern.
Explore in the article ↗CHECKPOINT 03
Illustrative example: delayed documentation discovery
Imagine an illustrative documentation site where editors suspect that filters delay discovery of new guides.
Explore in the article ↗Frequently asked questions
Do bot requests prove indexing?
No. Requests and responses are delivery evidence, while indexing requires separate observation.
Why verify user agents?
The name is easy to imitate. Use the provider’s documented identity checks before attributing traffic.
Are origin logs enough?
Not always. CDN caching and edge blocks can hide important events from origin-only records.
How can I put “Frame a question and a comparable period” into practice?
Choose a decision the analysis should support: investigating slow discovery after publication, diagnosing repeated server errors or evaluating a new filter policy. Define the relevant hosts, routes and time period. Include a comparison period only when logging coverage and site conditions are sufficiently comparable.
What matters when working on “Understand which layer recorded each request”?
Identify whether records come from the CDN, load balancer or application server. Cached requests may appear at the edge without reaching the origin, while application logs may omit requests blocked earlier. Document time zones, sampling, retention and missing periods.
