OpenAI has added hundreds of millions of previously undiscovered web pages to its search index through a research partnership with Cloudflare, potentially giving ChatGPT users access to a wider range of online information.
The development follows a pilot involving approximately 1% of Cloudflare’s network, with the companies using network-generated signals to help OpenAI identify useful web pages, detect updates, and avoid wasting resources on unavailable or duplicate URLs.
The companies announced the early findings on October 15, 2026, building on a research initiative launched in July.
How Cloudflare Helps ChatGPT Discover More Content
Cloudflare's network provides signals that help OpenAI's search crawler identify pages that may be useful to people looking for information through ChatGPT.
The approach could help surface content that traditional discovery methods might overlook, particularly pages that receive few external links but contain valuable, specific information.
Examples include:
-
Product manuals and technical instructions for particular devices.
-
Updated passport application requirements and fees.
-
Restaurant menus, business hours, and other local information.
-
Specialised pages containing details relevant to specific user questions.
According to the companies, the pilot has enabled OpenAI to discover and index hundreds of millions of new pages.
However, finding a page does not automatically mean it will appear prominently in ChatGPT Search. OpenAI independently assesses and ranks indexed pages according to their relevance and usefulness.
New Signals Could Reduce Unnecessary Crawling
Beyond discovering additional content, the partnership aims to make web crawling more efficient.
Automated crawlers can waste resources repeatedly requesting pages that no longer exist, following addresses that redirect to unrelated destinations, or retrieving duplicate content.
Cloudflare's signals can help OpenAI identify these situations before making unnecessary requests.
The system also uses information about page changes to help determine when a crawler should revisit a website. This could allow search systems to prioritise updated content instead of repeatedly fetching pages that remain unchanged.
The companies say these improvements are intended to reduce unnecessary server activity for website operators while helping OpenAI direct more resources towards useful information.
They acknowledged, however, that discovering and evaluating a large number of new pages may initially increase crawling activity.
AI Security Threats Also Under Investigation
The research partnership includes work on identifying potentially malicious web pages.
One example involves a legitimate website that has been compromised to host a fraudulent login form. Rather than treating the entire domain as unsafe, the system aims to identify the specific suspicious page.
This distinction could help protect users from harmful content without unnecessarily excluding legitimate websites from search indexes.
The companies said early findings suggest their signals can help identify pages that should be excluded from an index. Further work is expected to improve how the system identifies relevant content, detects changes and handles potentially dangerous pages.
Website Owners Still Control Crawler Access
Despite the expanded discovery capabilities, Cloudflare's signals do not override website owners' access restrictions.
The companies emphasised that identifying a page does not grant permission to retrieve it. Website owners' crawler settings continue to determine whether particular automated systems can access their content.
OpenAI distinguishes between two principal crawlers:
-
OAI-SearchBot: Used to discover and retrieve content for ChatGPT Search.
-
GPTBot: Used to crawl content that may be used to improve or train OpenAI's generative AI models.
Website operators can configure their robots.txt files to allow one crawler while blocking the other, giving them separate controls over search visibility and potential model-training use.
For example, a publisher that permits OAI-SearchBot but blocks GPTBot can make eligible content available for ChatGPT Search while signalling that GPTBot should not crawl the site for model-training purposes.
Cloudflare also offers AI Crawl Control, which allows website owners to monitor AI crawler activity and choose whether to allow or block supported crawlers. The feature is available across Cloudflare plans, according to the company's announcement.
What the Expansion Means for Publishers and Businesses
The wider index could create new opportunities for publishers, businesses and website owners seeking to reach audiences through AI-powered search.
Websites containing specialised information may benefit even if they attract relatively few links from other sites. More comprehensive discovery could help users find sources that traditional search methods might miss.
However, being indexed does not guarantee that a page will be cited in ChatGPT answers or generate additional website visits.
The companies have not disclosed how frequently the newly discovered pages are cited, how much referral traffic they generate, or how much crawling activity has been reduced.
Website owners will therefore need to monitor their visibility and traffic to determine whether the changes translate into measurable benefits.
Tracking referrals from ChatGPT can also be challenging when links lack referral information or tracking parameters, making it important for publishers to examine their analytics carefully.
OpenAI and Cloudflare Plan Further Improvements
The next stage of the research will focus on refining page selection, improving the speed and reliability of change detection, and measuring the business impact of making content easier for AI systems to discover.
The partnership reflects a broader shift in online information discovery, as AI-powered search tools increasingly connect users with information from across the web.
For website owners, the central consideration remains control: making content discoverable to AI search systems can create new opportunities to reach audiences, but access permissions and independent ranking decisions will continue to shape which pages users ultimately see.
Source: Cloudflare Blog
