We have been spending a lot of time looking at AI crawlers, LLMs.txt, bot traffic and the way AI systems are starting to read websites. Not because we have no life but because this is a movement that we cannot ignore.
This is still a fairly early space, but it is not theoretical anymore. We can already see AI bots visiting websites, looking for specific files, checking sitemaps, requesting pages and sometimes hitting errors. That is enough for us to start taking it seriously and working through what website owners need to do next.
Google’s recent I/O announcements are part of that picture. Search is moving further into AI-generated answers, AI Mode, follow-up questions and agent-style browsing. People are also using tools like ChatGPT, Gemini, Perplexity and Claude to research businesses, compare services and narrow down their options before they ever land on a website.
I do not see this as a small SEO trend. I think this is where website visibility is heading.
Search is already starting to change
Traditional SEO is still relevant. Content, links, site structure, performance and technical accessibility are all still part of the picture. The difference is that those things are starting to bleed into AI readiness as well.
In the old model, the main job was to help Google crawl, index and rank your pages. That still exists, but now there is another layer. AI systems may be reading your website on behalf of a user, deciding whether your business is relevant, summarising what you do, comparing you against other businesses, or using your website as a source for an answer.
That means we need to think about a new kind of website visitor.
There are human visitors. There are search engine crawlers. Now there are AI agents.
A website still needs to work properly for people. It still needs to be clear, useful, fast and easy to navigate. But it also needs to provide a good agent experience. By that, I mean the experience an AI agent has when it tries to access, understand and use the information on the site.
That sits very close to the kind of technical website work we already do.
AI agent readiness fits the technical side of website work
OINK Digital has not traditionally sold SEO packages. We do work with clients on content, structure, performance and technical improvements, and we have always taken SEO fundamentals into account when building websites.
That includes clear service pages, logical site structure, service areas, useful content, clean technical setup and site speed. Those things help people, they help search engines, and they are now becoming part of how AI systems understand a business as well.
The AI side is not just about writing more content. In some cases, huge text-heavy pages may actually be less efficient for an AI system to work with. If an agent needs to process a lot of bloated content to answer a simple question, that is not a great experience.
This is where the technical layer becomes more interesting: what can the agent access, how quickly can it understand the site, what files is it looking for, what errors is it hitting, and are we giving it a clear path to the best information?
LLMs.txt is a useful entry point
One of the things we are looking at is LLMs.txt.
Robots.txt gives crawlers rules around what they can and cannot access. LLMs.txt is different. It can act as an entry point for AI systems, explaining what the site is, what the business does, which pages are useful, and where agents can find the best information.
We are already seeing bots look for these files.
That does not mean every standard is locked in yet. It does mean that the idea is being picked up quickly enough that it is worth paying attention to now.
The LLMs.txt file does not need to do everything by itself. It can point to other useful files, such as a fuller LLMs file, structured service summaries, Markdown versions of important pages, agent-focused indexes or other resources that help AI systems work through the site more efficiently.
That is the point I find useful. You can create one clear entry point, then guide agents to the information that gives them the best chance of understanding the business properly.
Token efficiency is part of the work
AI systems process text in tokens. A token is basically a chunk of text the system has to read and process. The more cluttered, repetitive or unclear a page is, the more tokens the system has to work through.
A normal website page is built for browsers and people. It can include navigation, headers, footers, scripts, layout, menus, repeated content and HTML markup. Some of that is useful for people. An AI agent may not need most of it.
That is one reason Markdown is interesting. It keeps structure without carrying all the extra weight of HTML. A heading can be a simple hash instead of a full HTML tag. A clean Markdown version of a page could give an agent the same core information in a simpler format.
That does not replace the public website. It gives AI systems a cleaner way to understand it.
We are looking at real bot traffic
We are building internal tooling around this because guesswork is not good enough. We want to see which bots are visiting, what they are requesting, whether they are finding the right files, whether they are hitting errors, and what kind of bot activity we are dealing with.
This is also where the conversation gets more practical. Not all bot traffic is the same. Some bots may be useful. Some may be connected to AI search. Some may be fetching pages because a user asked a question. Some may be training-related. Some may not be worth encouraging at all.
AI readiness is not just opening the door to every bot. It is understanding the traffic and making better decisions.
This is the kind of data Google Analytics does not really give you. To understand this properly, we need to look deeper into bot behaviour, server requests, CDN data, access patterns, errors and the files agents are trying to reach.
What we are doing for our clients
For our clients, this is not a panic message. It is a preparation message.
Your website is not suddenly broken because AI search is growing. But the way people find and compare businesses is changing, and websites need to be ready for that shift.
We are looking at AI crawler access, LLMs.txt, bot errors, site structure, page speed, content clarity, Markdown options and how key business information can be made easier for agents to understand.
Some of this is early. The standards are still forming. But that is also the opportunity. If businesses wait until everything is fully settled, the early advantage will already be gone.
We are taking this seriously across our own websites and properties first as we are, pardon the pun, the Guinea pigs. We are building tools, watching the data, testing different approaches and using that work to shape how we help clients prepare.
Traditional SEO is not disappearing. It is becoming more technical, more contextual and more agent-aware.
We’ll keep you updated as we find out more and build better tooling that we can roll out to our clients.
Work With Us
Are you looking for ways to increase your AI Agent visibility and improve your website? Let us know, we’d love to hear from you!