Advanced Tools: Choosing by Task and Data Source
Advanced tools are better chosen not by a list of brands but by two questions: which decision you are making and where the data comes from. Data comes in three kinds: Google’s own data (Search Console, GA4, speed reports), the results of crawlers that look at a site from outside, and estimates from third-party services (link indexes, query volumes, rank checks). Below: what is documented about Google’s free tools and their limits, how to choose a crawler and when not to compare figures from different sources.
The data source matters more than the brand
| Kind of data | Examples | How to treat it |
|---|---|---|
| Google’s own data | Search Console, GA4, PageSpeed Insights, CrUX | the only data on impressions and clicks in Google; each has its own limits |
| Crawlers | desktop and cloud crawlers; the platform’s crawler and audit | show the site through a robot’s eyes: response codes, links, tags; but not search data |
| Third-party estimates | link indexes, query volumes, rank checks | approximations from a service’s indexes and samples; compare within one service (profile metrics) |
Google’s tools and their limits
Search Console: the report, the API and the BigQuery export
- The Performance report shows how a site performs in Google Search results (help page); the report has an export button for tables and charts (Search Console for an analyst).
- The API: one request returns from 1 to 25,000 rows, 1,000 by default; the
startRowoffset is used for the next batches. There are quotas: 1,200 queries per minute per site and per user (the searchanalytics.query method). For bulk export without these limits the platform’s Search Console export is convenient. - The BigQuery export: a daily export of performance data that contains all the data available to Search Console for the property, except anonymized queries (help page). It suits when you need complex queries and history storage.
PageSpeed Insights: two kinds of data
The help page on PageSpeed Insights distinguishes lab diagnostics (Lighthouse analyzes the page in a simulated environment across the Performance, Accessibility, Best Practices and SEO categories) from field data, which covers a trailing 28-day period. Field data in PSI is updated daily, whereas the CrUX dataset in BigQuery is updated monthly and only at the origin level. That explains why the same page can show different figures in different places (Core Web Vitals, the page analyzer).
Looker Studio and GA4
Looker Studio can connect to Search Console (the connector), which is convenient for reports without manual export. GA4 data is something else: it is measurement on your site and depends on users’ consent (GA4 overview). Do not expect Search Console and GA4 to match.
How to choose a crawler
A crawler is needed to see what Search Console does not show: response codes, redirects, links, tags, duplicates. When choosing, check:
- The free version’s limit. For the Screaming Frog desktop crawler, according to the vendor’s own pricing page, the program is free but a licence is needed to crawl more than 500 URLs; a licence lasts a year (pricing page). For most small sites that is enough; more is a reason to pay.
- JavaScript rendering. If content appears via script, you need a crawler that executes it (JavaScript SEO).
- robots.txt handling, speed, export and the ability to compare crawls with each other.
- Integrations with Search Console and PageSpeed Insights: they let you reconcile crawl results with search data.
A working set by task
| Task | Google data | The platform |
|---|---|---|
| Technical audit | the indexing report, URL inspection | crawler, site audit |
| Queries and pages | the Performance report, API, BigQuery | Search Console export |
| Speed and Core Web Vitals | PageSpeed Insights, CrUX | page analyzer |
| Changes and updates | the Performance report | Wins & Losses, Google Updates |
| Links, query volumes, positions | the links report (a sample), average position | third-party services; see rank tracking |
What remains opinion
- “Free tools cover 80% of tasks” — a figure without a source; but Google’s own data is the only data on what was really shown and clicked in Google, so it is where to start.
- Lists of brands of the “best crawlers and services” age quickly and depend on tasks, budget and country; the selection criteria matter more.
- “A paid tool is justified when manual work takes too long” — a sensible rule, but not from the documentation.
Checklist
- For each decision it is clear which data is needed and where it comes from.
- Data from Google, a crawler and a third-party service is not mixed or compared as equals.
- The limits are known: API rows, quotas, 500 URLs in the crawler’s free version, the period of field data.
- Search Console data history is preserved (export or BigQuery).
- Reports are built on reproducible sources, not on manual copies.
- A paid tool is connected for a specific task that cannot be solved otherwise.
Mistakes
- Choosing a tool by advertising rather than by task and data.
- Adding up figures from different services and comparing them with Google’s data.
- Forgetting API limits and report table limits.
- Not exporting data history and losing it.
- Paying for every feature when one is needed.
What next
Next comes how to learn SEO: which sources to read, how to tell documentation from opinions and how to build your own research. Testing hypotheses on data is in the lesson on SEO experiments.