The internet was built for people to browse, read and click. AI agents are beginning to use that same information directly — forcing companies to rethink who their websites are actually built for.
THE SIGNAL
The change is already measurable.
Across documentation sites hosted by Mintlify, AI agents generated approximately 261.9 million requests in September compared with 118.1 million human visits.
That means agents accounted for 68.92% of measured traffic.
Mintlify also reports that 70% of the documentation sites it hosts now receive more machine requests than human page loads. Between February and August, agent readership increased 7.7 times, while human readership increased only 1.2 times.
This does not mean 69% of the entire internet is being read by AI. Mintlify’s data covers documentation hosted on its platform, an area particularly exposed to coding agents.
But the direction matters.
Software documentation may simply be showing us the future earlier than the rest of the web.
A developer once opened a documentation page, searched for an API instruction, read an example and copied the relevant code.
Now an AI coding agent can make that request itself.
The human still has a goal.
But the machine is doing the reading.
That creates a second audience for the web.
WHAT THE MARKET IS MISSING
Most websites still assume there is one primary visitor:
a person looking at a screen.
That assumption influenced almost everything about how the modern web developed.
Navigation menus help humans find pages.
Headlines attract attention.
Images explain ideas.
Search boxes help users locate information.
Analytics record browser sessions.
Advertising depends on page views.
SEO helps search engines direct humans toward websites.
AI agents behave differently.
They may never see the homepage.
They do not care whether the typography is attractive.
They may not load the advertising, execute the JavaScript analytics or follow the navigation structure.
Instead, an agent may request a specific markdown document, query an API reference or retrieve information through an entirely different machine interface.
Mintlify says its agents leave in milliseconds, compared with an average human visit of more than four minutes. Its data also shows agents increasingly accessing documentation almost like a filesystem rather than browsing pages. In August, Mintlify recorded about 2.17 million MCP tool calls, with 53% using search and 47% accessing files directly.
That is a fundamentally different relationship with information.
The website remains important.
But the visible website may no longer be the only interface that matters.
FIRST-ORDER EFFECTS
DOCUMENTATION BECOMES PART OF THE PRODUCT
For software companies, documentation has traditionally supported the product.
Agents could turn it into part of the product itself.
If an AI coding agent cannot understand how an API works, find the correct parameters or determine which version of a feature is current, the user may conclude that the software does not work — even if the actual problem is poor documentation.
That creates a new competitive variable:
How easily can an AI agent understand and use your product?
Documentation platforms are already responding.
Mintlify now describes developer documentation as serving both developers and the agents working for them. Its data shows agent traffic ranging from 34% to 70% depending on industry, with fintech and payments already at the top of that range.
The implication is larger than documentation.
Product accessibility may eventually include agent accessibility.
THE WEB STARTS PRODUCING MACHINE VERSIONS OF ITSELF
A web page designed for a human contains enormous amounts of material an agent may not need.
Menus.
Footers.
Advertising.
Animations.
Tracking scripts.
Images.
Design elements.
An agent usually wants the underlying information.
That is helping drive experiments such as llms.txt, a proposed standard that gives AI systems a cleaner way to find important information about a website.
The proposal was first introduced in 2024 and updated in August 2026. Its maintainers say thousands of websites now publish an llms.txt file and documentation platforms increasingly generate one automatically. The updated proposal also introduces ways for websites to point agents toward markdown versions of pages.
It is not yet a universal internet standard.
That matters.
We are still early enough that companies are experimenting with how the machine-readable web should work.
But the direction is becoming visible:
One underlying body of information may increasingly support two interfaces — one designed for humans and another optimized for machines.
ANALYTICS START BREAKING
A company can have growing usage without seeing it in the dashboard it has relied on for years.
Traditional analytics often depend on JavaScript executing inside a browser.
AI agents frequently retrieve content without doing that.
Mintlify found that coding-agent requests could be largely invisible to conventional analytics, which is why it measures agent traffic using server-side requests and known AI user agents instead.
That means an organization could look at declining page views and conclude that documentation is becoming less important while agents are actually consuming it more frequently than ever.
Web analytics evolved around humans.
Agent analytics will require different measurements.
The important questions become:
Which agents are accessing the information?
What are they requesting?
Did they find the correct answer?
What action happened afterward?
Those measurements are still being built.
SECOND-ORDER EFFECTS
AGENT COMPATIBILITY COULD BECOME A FORM OF DISTRIBUTION
Search-engine optimization became powerful because being easy for Google to understand could determine whether humans discovered a business.
AI agents introduce another layer.
Imagine two software products offering similar capabilities.
One has clean documentation, clear APIs, structured reference material and reliable machine access.
The other has incomplete pages, conflicting instructions and important information buried in PDFs or support conversations.
A human developer might tolerate the second product.
An AI agent may simply struggle to use it.
If agents increasingly recommend products, write integrations and select services while completing tasks, being easy for machines to understand could eventually influence which products get used.
This does not mean traditional SEO disappears.
It means companies may eventually manage human discovery and machine usability simultaneously.
COMPANY KNOWLEDGE MOVES OUTSIDE THE WEBSITE
Another problem appears quickly.
The website rarely contains everything the company knows.
Answers are scattered across support tickets, Slack conversations, internal documents, community forums, sales materials and employee knowledge.
Mintlify’s 2026 State of Knowledge research found that 94% of surveyed organizations had important knowledge surfaces beyond their primary documentation.
Humans have historically filled those gaps.
Someone asks a coworker.
A support representative knows the workaround.
An engineer remembers why a system behaves a certain way.
Agents cannot reliably use knowledge they cannot reach.
That turns knowledge management from an internal housekeeping problem into infrastructure.
The company with the smartest AI model may still get poor results if its knowledge is fragmented, stale or contradictory.
ACCESS BECOMES A BUSINESS DECISION
Not every company will want every agent reading everything.
Cloudflare is already separating AI traffic into different categories:
Search, which indexes information for later answers.
Agent, which acts in real time on someone’s behalf.
Training, which uses content to train or fine-tune models.
Website owners can allow or block those categories separately. Cloudflare also changed its defaults in September so certain agent and training activity can be blocked on pages carrying advertising.
That distinction is important.
The old question was:
Should bots be allowed?
The new question is:
What is the bot doing, who is it acting for and what does the website receive in return?
Cloudflare says automated agents and bots already account for more than half of overall web requests across its network, although that broader number includes many types of automation beyond AI.
That means access policies, licensing and monetization could become much more important as machine consumption grows.
WINNERS
Companies with clean, structured knowledge could gain an advantage because agents can retrieve accurate information quickly and act on it.
Developer platforms with strong documentation may become easier for coding agents to integrate, potentially reducing friction between discovering a product and actually using it.
Documentation and knowledge-management platforms gain a larger role because companies must increasingly maintain information for both humans and machines.
Businesses that can measure agent traffic will have a clearer picture of how their information is being consumed instead of relying solely on browser-based analytics.
LOSERS
Companies with outdated or contradictory information face a larger problem than confusing human readers. Machines can retrieve those inconsistencies automatically and potentially propagate them through downstream work.
Businesses dependent entirely on page views may discover that machine consumption does not create the same advertising economics as a human opening a webpage.
Products with poor documentation could lose ground if agents repeatedly fail to understand or integrate them.
Organizations with knowledge trapped inside employees’ heads or disconnected systems may find that buying more AI does little to improve results because the AI still cannot reach the information it needs.
WHAT HAPPENS NEXT
The first place to watch is software documentation because the transition is already visible there.
But the same architecture could spread.
Product catalogs can be structured for shopping agents.
Travel information can be consumed by booking agents.
Financial information can feed research agents.
Company policies can be retrieved by workplace assistants.
Healthcare information can be structured for clinical support tools.
Eventually, businesses may have to think about every important information surface in two ways:
How does a person experience this?
And:
How does an agent retrieve and act on it?
Standards will matter.
llms.txt is one experiment.
The Model Context Protocol, or MCP, is another. MCP is becoming a common way for AI systems to connect directly with tools and information sources. The project’s maintainers reported in July that its major software-development kits were approaching half a billion downloads per month, illustrating how rapidly developers are building agent connections to external systems.
The web does not disappear in that world.
It changes roles.
Humans may continue seeing websites.
Agents may increasingly interact with the information underneath them.
And companies may have to support both.
BOTTOM LINE
For thirty years, the web has largely been designed around a simple assumption:
A human being is going to read this.
That assumption is starting to break.
Mintlify’s documentation traffic provides an early and unusually visible example. Nearly 69% of its September traffic came from agents, and seven out of ten documentation sites on its platform now receive more machine requests than human page loads.
The important change is not that robots are browsing websites.
It is that AI is beginning to consume information on behalf of people and then use that information to perform work.
That changes documentation.
It changes analytics.
It changes discovery.
It changes access.
And eventually, it could change how websites themselves are designed.
The internet is not losing its human audience.
It is gaining another one.
