Google Finally Built the Search API It Should Have Built 25 Years Ago

Google has finally discovered the API

For roughly 25 years, Google and the search industry have been having the same stupid fight.

Businesses need to know where they rank. Software companies try to collect that data. Google changes the rules, the markup, the limits, or the URL. The software companies build another workaround. Google blocks the workaround. Everybody buys more servers.

Then Google documented what may be the beginning of a sane answer: a proper service that returns full-web search results as structured JSON.

Search Engine Journal found the newly refreshed Web Search Service documentation. The API supports REST and gRPC, and its response can include titles, URLs, snippets, result counts, timing, spelling corrections, and pagination.

That is exactly how machines should get search data from another machine. No pretending to be a person with Chrome. No parsing an ever-changing mountain of HTML. No proxy farms. No CAPTCHAs. No engineering team spending Thursday afternoon discovering that somebody at Google moved a div.

The original bargain was content for traffic

Google built its product by crawling everybody else’s websites. Publishers allowed it because the bargain was obvious: you may index my work, and in return you may send people who are looking for it.

That exchange created enormous value on both sides. Google got the world’s content organized into a useful product. Businesses got discovery, traffic, leads, and sales. An entire discipline grew around understanding how the exchange worked.

But the data has never flowed both ways with equal enthusiasm.

First, Google took away the keywords

For years, the search term that brought someone to your site arrived in the referring URL. You could see what the visitor asked, which page answered it, and what happened next. It was useful, direct, first-party performance data about traffic Google had sent to your own website.

In 2011, Google encrypted searches for signed-in users and removed the individual query from organic referrals. Google framed the change as protecting search privacy. The query data for organic clicks became the infamous “not provided.”

There was an interesting footnote. Google’s own Analytics announcement said paid search reporting would continue to receive query data. Apparently privacy could survive contact with an advertising budget.

Publishers received a smaller, aggregated view in Webmaster Tools, the ancestor of Search Console. Useful? Yes. Equivalent? Not remotely.

You cannot make ranking data unimportant by making it annoying

Rankings decide who gets seen. That means money, jobs, investment, and sometimes whether a small company gets a chance at all. Millions of businesses care about this data because their customers use Google.

So the demand for independent rank tracking was never going away.

Google could dislike automated collection. It could call it unauthorized. It could make collection more difficult. But it could not make a restaurant owner stop wondering whether she ranked third or thirty-third for the service keeping the lights on.

That left software vendors to collect results through the consumer interface. They requested search pages, parsed the HTML, simulated browsers, rotated IP addresses, retried blocked requests, and adapted to every layout change. Google, in turn, invested in detecting and blocking the automation.

This is the definition of whack-a-mole, except both the hammer and the mole have cloud bills.

Google’s current spam policy specifically names scraping for rank-checking as prohibited machine-generated traffic. Fair enough. Google owns the interface and has legitimate abuse and capacity problems to manage. But a prohibition is not a solution to durable market demand.

Click to expand

Then Google removed the 100-result shortcut

The latest round arrived in September 2025, when Google stopped honoring the num=100 URL parameter that returned up to 100 results in one page.

The change disrupted SEO tools that used the parameter for rank tracking. Collecting the same depth could now require many more requests. It also caused sharp changes in Search Console impression and ranking-term data, some of which may have been bot activity disappearing from the reports.

Whatever Google’s reason, the resource math is not complicated. If a tool needs ten pages where it previously needed one, Google must serve more requests. The tool vendor must run more sessions, transfer more markup, manage more failures, and pay for more infrastructure. The customer ultimately pays for the entire pantomime.

And Google still has to build systems to catch it.

A JSON API is dramatically more efficient. The client requests only the fields it needs. Google returns a small structured payload instead of a complete page built for a human browser. Authentication, quotas, pricing, and abuse controls can be explicit. Both sides can capacity-plan. The schema can be versioned. Nobody needs to inspect a button’s CSS class to learn who ranks seventh.

AI has made the original bargain much worse

The data asymmetry would already be irritating. AI Search turns it into something harder to defend.

Publishers still let Google crawl their articles, documentation, research, reviews, and discussions because Google Search is a critical source of discovery. Google can now use that material to assemble an answer directly on the results page, reducing the need to visit the source that produced it.

This is not a theoretical complaint. Pew Research Center analyzed 68,879 Google searches and found that users clicked a traditional result on 8% of visits with an AI summary, compared with 15% when no summary appeared. Links cited inside the AI summary were clicked in just 1% of visits.

Google does offer controls, but they reveal the problem. Google says Google-Extended can restrict certain Gemini training and grounding uses without changing Search ranking. But this is a stupid and largely symbolic measure: Google can still learn from other sources, and the main thing you are preventing is your own site from hopefully being cited.

For AI Overviews and AI Mode inside Search, publishers must use the same Googlebot and preview controls that govern ordinary Search.

In other words, the practical choice is not a clean “yes to search, no to AI answers” consent box. If your business depends on search visibility, opting out can mean weakening or removing the very listing that made the crawling bargain worthwhile.

Google is happy to ingest the web at extraordinary scale. Asking for structured access to the rankings built from that web has somehow remained the suspicious activity.

This is why BoastIndex uses Google’s official API

BoastIndex is not a rank tracker, and it does not scrape Google search results.

When someone connects BoastIndex, they authorize a read-only Google Search Console property. We use Google’s official URL Inspection API to check index status, observe Google’s quotas, and store that history so customers can see when an important URL changes.

That model is not free of constraints. The official API has quotas. It does not answer every possible question. But the rules are explicit, the payload is structured, and Google does not have to guess whether our software is a person refreshing a browser from 400 IP addresses.

It lets us build Google index monitoring and a Google Search Console reporting tool on top of an authorized, efficient transaction.

The same model should exist for ranking data.

The new API is the right answer with the wrong front door

Google’s Web Search Service is the model that should have existed all along. Meter access. Charge for it. Set sensible quotas. Enforce clear display and storage rules. Revoke abusive clients. Give legitimate products an efficient, documented way to buy the data their customers need.

I am not arguing that Google should provide unlimited search infrastructure for free. Search is expensive. Abuse is real. Those are excellent reasons to operate an API. They are terrible reasons to force an entire industry to retrieve machine data through pages designed for eyeballs.

Unfortunately, the current service requires a client ID associated with a partner agreement. Google has not published a standard signup path, price sheet, eligibility rules, or query limits. Its older self-service Custom Search JSON API is closed to new customers and scheduled to shut down on January 1, 2027.

So Google has finally arrived at the efficient answer, then placed a velvet rope in front of it.

Open the API. Not without limits, and not without a bill. Open it with transparent access, metered pricing, enforceable rules, and enough capacity for real products to depend on it.

Stop fighting the demand. Serve it.

Someone will always find another way to measure rankings because the rankings matter too much not to measure. Google can keep swinging the hammer, and software companies can keep building faster moles. Or Google can control the market with a proper API and get paid for solving the problem.

The crawl-for-traffic bargain is weaker when AI answers consume publisher content but reduce publisher clicks. Blocking rank collection has produced a costly arms race of HTML scraping, browser emulation, proxies, bot defenses, and constant repair. A public, metered full-web API would give Google control and revenue while giving legitimate tools efficient, stable access.

It only took 25 years to get this close.

Sources and further reading

1. Search Engine Journal, Google Documents Partner-Only API for Full-Web Search Results
2. Google for Developers, Web Search Service API overview
3. Google Public Policy Blog, Making search more secure
4. Google Analytics Blog, accessing search query data after encrypted search
5. Google Search Central, machine-generated traffic policy
6. Search Engine Journal, Google modifies search results parameter affecting SEO tools
7. Pew Research Center, clicks when AI summaries appear
8. Google Search Central, website controls for AI features in Search
9. Google Search Central, Google-Extended crawler control
10. Google for Developers, Custom Search JSON API overview