Blog

Mapping the Bot Problem

A framework I keep describing: cooperative vs. evasive, desired vs. undesired, and why the agentic era demands more from bot managers.

I keep having the same conversation.

It usually starts with a question like “why won’t Company-X just allow the good bots?” or “why is Company-Y blocking my agent?”

Inevitably, I end up describing a two-by-two matrix.  After enough repeats, I figured I should write it down, and share it more visually.

“Good bots vs. bad bots”

“Why can’t you just stop the bad bots and allow the good bots?”  Simple question, and it comes from security operators, fraud fighters, threat intel teams, marketers, the CMO/CDO/CTO/CISO crowd, revenue teams, analysts, investors and many more...

The challenge has always been: who’s the arbiter of “good bot” vs. “bad bot”?  Sit with two companies, or even two departments within one company, and you’ll hear at least two different answers.  Marketing has struck a deal with a platform like Perplexity, but security sees a flood of API requests that slows the site down for everyone else.  Is that a good or bad bot?

Hm... that’s a desired bot, behaving in an undesired way.  So a bot is neither inherently good nor bad, there’s something more subjective here.  To tease this apart, I like a good old 2x2 matrix.

The first dimension is how a bot presents itself.  At one end are cooperative bots.  They:

  • Tell you what they are.  Search crawlers, uptime monitors and partner integrations self-identify (ideally with strongly verifiable means like Web Bot Auth)
  • Follow the policies you publish (robots.txt, if you have one)
  • Stop when told no

At the other end are evasive bots.  They strive to look like a real person on a normal browser, impersonating real users’ device attributes, traffic profiles and behavior: rotating IPs, faking fingerprints, mimicking mouse movement, and much more.  The most tenacious will do whatever it takes to get past anti-automation tools and finish the task.

How a bot presents itself A horizontal axis. Left end: cooperative. Right end: evasive. How a bot presents itself. Cooperative Evasive
Axis 1: how a bot presents itself

The second axis is whether you want this type of interaction.  Is this behavior desired or undesired?

This is a business policy decision, and the same bot can be welcome on one site and unwelcome on the next.  For example:

  • Some merchants allow price and inventory scrapers, others work tirelessly to protect their data.
  • Some sites encourage bots to consume product information (maybe even optimize content for them!), others do not.
  • Some banks allow 3rd party aggregators to use the website on behalf of the bank’s client, others do not.

At some point, the site still needs a policy it can enforce.  Aligning on that policy and expressing it is a much deeper topic for another day.

Whether you want it The horizontal axis, cooperative to evasive, crossed by a vertical axis. Top: desired. Bottom: undesired. Four empty quadrants. Cooperative Evasive Desired Undesired
Axis 2: how a site owner views the bot's activity on their website

Four zones

Put those two axes together and you get four zones.

Four zones Four zones on the two axes, each box also labeled with its axis combination (for example, cooperative + desired). Zone 1, cooperative and desired: the welcome guests. Zone 4, evasive and desired: the empty corner. Zone 2, cooperative and undesired: the easy no. Zone 3, evasive and undesired: where the fight lives. Zone 1 The welcome guests Cooperative + desired Zone 4 The empty corner Evasive + desired Zone 2 The easy no Cooperative + undesired Zone 3 Where the fight lives Evasive + undesired Cooperative Evasive Desired Undesired
The four zones
  1. Zone 1, the welcome guests. Cooperative and desired: the site monitoring tools, SEO bots and affiliate solutions your business chooses to consume, enable and optimize for.
  2. Zone 2, the easy no. Cooperative and undesired: typically basic scripts, automated pen-test tooling, self-identified consumers that are neither a browser nor your mobile apps.  Unwanted, but honest about what they are.
  3. Zone 3, where the fight lives. Evasive and undesired: the realm of abusers and fraudsters.  Inventory hoarding, price scraping, credential stuffing, card testing, and on and on.  Typically a technically savvy operation with a financial incentive to keep detecting and evading anti-bot solutions.  The tenacity of the adversary is usually a function of how monetizable their flavor of undesired activity is.
  4. Zone 4, the empty corner. Evasive and desired: historically, very little lived here...

The world that was

So when it came time to set a policy, here’s how it usually played out.

The bot map The full map, colored by how policy treated each zone: green, yellow-green, red, and grey for the de-emphasized empty corner. Zone 1, cooperative and desired, the welcome guests: monitoring, SEO bots, affiliates you approved. Zone 4, evasive and desired, the empty corner: historically almost nothing lived here. Zone 2, cooperative and undesired, the easy no: simple scripts that admit what they are. Zone 3, evasive and undesired, where the fight lives: fraud rings, scalpers, credential stuffers. Zone 1 The welcome guests Monitoring, SEO bots, affiliates you approved Zone 4 The empty corner Historically, almost nothing lived here Zone 2 The easy no Simple scripts that admit what they are Zone 3 Where the fight lives Fraud rings, scalpers, credential stuffers Cooperative Evasive Desired Undesired
The bot map

Zone 1, the welcome guests: allow

Zone 2, the easy no: deny

Zone 3, where the fight lives: mitigate in a variety of forms (a rich topic for another day).

Credential stuffing, scalping, scraping, card testing...  undesired interactions with real consequence for your business but profit opportunity for the adversary.  A cat & mouse game of compounding complexity that continues to accelerate.

Zone 4, the empty corner: transitory.  Rare, and once an operator was understood, it usually became cooperative: a verified resale partner, an aggregator with business terms in place.  There simply haven’t been many durable use cases in this zone, at least for brands that also have undesired interactions they want to prevent.

That gave anyone crafting a policy a pretty useful shortcut:

How a bot presented itself told you most of what you needed to know about whether you wanted it.  If it was hiding, you probably did not.

What changed

Two things broke that shortcut.

First, Zone 4 is filling up.  A real customer asks an AI agent, say Grok bot, to find a gift, compare three sites and buy the best option.  That agent is automated traffic that looks like a Chromium browser on a desktop, because it is one, at least virtually.  It clicks through your site like a person would and often doesn’t announce itself in any reliable way.  By the old rules it’s an automated session masquerading as a user.  But now it could be a paying customer who chose to shop that way, and your business may well decide that’s an interaction it wants.  Block it and you’ve turned away revenue.  Allow it, and perhaps it’s a fraudster checking or re-using stolen payment information.

Second, cooperative does not mean safe.  Bad actors can use perfectly valid tools to do undesired things, and we’ve seen this movie before...

A quick sidebar.  Remember when financial aggregators like Mint were new?  Banks were put in a tough position: do they allow these 3rd party bots to access their customers’ data?  Many of the aggregation services were cooperative in the purest sense: known entities, readily identified, and willing to cooperate.  Those that were not, were blocked.  Eventually, bad actors figured out it was more efficient to give aggregators hundreds of credentials to check, instead of sending hundreds of requests straight to the bank, since those aggregators were “good bots”.

Today’s AI agents have the same property.  A well-known agent that identifies itself honestly can still be pointed at your login page with a list of stolen passwords, or told to buy every unit of a limited release.  “Good bots” can do undesired things.

So the question shifts.  “What is this thing?” only gets you so far.  You also need to ask “what is it trying to do?”

That’s why intent and behavioral analytics have moved to the center of this space.

Two halves of the answer

When I talk this through with people, it settles into two complementary efforts, and you can see the industry working on both.

For bot providers that want to be cooperative: There needs to be a way to present themselves with confidence.  What they are, who sent them, and what the bot operator has authorized them to do (purchase up to $X).  That’s the work happening around Web Bot Auth, Know Your Agent, and the alphabet soup of new protocols.  These are important foundations for enabling this new revenue channel.  An agent that can prove who it represents, and express what it has been authorized to do, is much easier to welcome.  At the time of this writing, we are in the messy pre-standardization phase, but evolving rapidly.

Evasive automation: Plenty of visitors will never be cooperative, whether by choice or by ignorance.  The custom-built bot, the poorly constructed 3rd party service, the adversary actively avoiding detection.  And even an agent that tells you exactly what it is can be driven to do something harmful.  For all of that, a site needs a read on intent.  What is this session trying to accomplish?  Does the behavior look like a valid user or an attacker?  And if it’s unclear, what should be permitted?

  • Verifying the agent and its operator answers “who sent you?”
  • Reading its behavior answers “what are you here to do?”

A site owner needs both answers before they can say yes with confidence.

This has always been difficult

When you’re dealing with an evasive adversary, you don’t want them to know you can see them.  Tip them off and they retool and come back even harder to detect.  Quiet detection is a feature.

When it’s a real customer using a tool that just isn’t cooperative enough (yet), you want the opposite.  You want to say: here’s the paved road, here’s how your agent can do this properly, come on down to aisle #7 and we’ll happily take your money...  That’s how the visit turns into a sale.

The trouble is that in the moment, with partial information, those two can look a lot alike.  Nudge the wrong one and you’ve tipped off an adversary.  Stay quiet with the wrong one and you’ve lost a customer.

Why it’s harder now

As more real user behavior moves into automated channels, the likelihood and opportunity-cost of blocking a desired interaction goes up.  Zone 4 is filling up...

At the same time, adversaries in our Zone 3 are evolving faster than ever before.  The list of abuse hasn’t changed much (payment fraud, account takeover, credential stuffing, inventory hoarding, denial of service).  But the perpetrators can learn and scale faster now, and return with ever-better camouflage and more things to impersonate.

This is a particularly messy moment in our Internet evolution.

Mapping the bot problem from here...

In the next few posts I want to dig into the bot (agentic) opportunity, perhaps a bit more on the current solution landscape (different tools for different jobs), and thoughts on the standards and norms we will need.