User agent: a claim the client makes about itself

The User-Agent header names the software that sent a request. It is useful for reading logs and sorting out crawlers that announce themselves, and it is worthless as proof, because the client writes it and can write anything.

Section Bots Updated Defined in RFC 9110

Quick facts

What it is
A request header carrying the client software's own description. Defined in RFC 9110 section 10.1.5
Who writes it
The client. Nothing in the protocol checks it, so any client can send any value
Browser prefix
Mozilla/5.0, kept "for historical reasons" by almost every browser, per MDN
Verifying a crawler
Reverse DNS, then forward DNS, as described in Google's documentation
Does a WAF stop it?
Partly. A filter can see the string and act on it, but cannot tell a true string from a false one. Treat the match as a signal and start in monitor

What the header is

Every HTTP request can carry a User-Agent header. RFC 9110 section 10.1.5 describes it as information about the software that originated the request, which servers use to scope interoperability problems, tailor responses to a client's limits, and run analytics. The same section asks senders to keep the string to what is needed to identify the product, and to list product identifiers in order of significance.

The shape is a sequence of product tokens with optional comments in parentheses. A script might send one token and a version. A browser sends a long string whose meaning has drifted far from that tidy description.

Searching for "user agent" mixes several things: the header, the string a browser reports to scripts, the software itself, and browser extensions that change the value. This page is about the header and what a server can conclude from it. Where it sits among the other signals used to tell people from automation is covered on bot detection, which defines it in one paragraph; this page goes further into the string.

Why everything says Mozilla/5.0

MDN puts it briefly: Mozilla/5.0 is the general token that says the browser is Mozilla-compatible, and for historical reasons almost every browser sends it. The longer account is in MDN's page on browser detection using the user agent.

The cause is a feedback loop. Servers gave the best pages to clients that identified as Mozilla. Other browsers began identifying as Mozilla-compatible to receive the same pages, then added their own names after it. Each later engine copied the earlier prefixes to avoid being served a degraded page. The result is a string where a browser's name appears among the names of browsers it is not.

RFC 9110 section 10.1.5 says implementations are encouraged not to use other products' tokens to claim compatibility. The Mozilla convention is the standing exception that nobody can now remove without breaking sites that still look for it.

For a reader of logs, the consequence is practical: searching for "Mozilla" tells you nothing, and the useful part of a browser string is whatever comes after the prefix.

How to see a user agent

Three places, depending on which side you are on.

  • In the browser. The Network panel of developer tools shows the request headers of any request, including User-Agent. In a script, navigator.userAgent returns the string the browser reports.
  • On the server. The header arrives with every request, and application frameworks expose it as a request header. Check the name your framework uses; some present it in an uppercased environment-variable form.
  • In the access log. Common web servers can write it into each log line. The nginx combined format includes it by default, as the log module documentation shows. It is the cheapest place to start, because the history is already there.

Changing the value is as easy as reading it. Browser extensions and developer-tools settings can send a different string, and command-line HTTP clients accept a header flag. That is useful for testing how your own site responds, and it is the same ease that makes the header unreliable as evidence.

When you read a log, sort by string and count. A handful of strings usually account for most of the traffic, and the interesting ones are at the tail: an empty value, a library name, a string that does not parse, or a browser version nobody has used for years.

Strings from browsers and bots

Crawlers that want to be identified put a name, a version and a pointer to documentation in the string. MDN lists this one for Google's crawler: The AI crawler GPTBot and the good bots you depend on announce themselves the same way.

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The pattern is the Mozilla prefix, a compatible comment, a product token with a version, and a URL where the operator describes the crawler. Google keeps a list of its crawlers and the tokens each one uses, and the current Googlebot strings there carry more detail than the one above. Bing publishes the same kind of information for its crawler in its webmaster documentation.

A well-behaved crawler also reads your robots.txt for its token. That is the point of the name: it lets you give a crawler its own instructions.

Two other groups look different in a log. Scripts and HTTP libraries often send a default string that names the library, such as the name of a programming language's client. These are honest, easy to find and frequently harmless: monitoring, integrations, your own tools. See content scrapers for the subset that is neither. The second group sends a copy of a current browser's string, which tells you only that someone wanted to look like a browser.

A claim, not proof

A server that sees Googlebot in the header has learned that the client typed the word Googlebot. Anyone can type it, and people do, because many sites treat that name kindly: no challenge, no rate limit, access to pages behind soft paywalls. Sending a false string is user agent spoofing, and it takes one line.

The check that works for a crawler uses the network address instead. Google's verification guide gives two methods.

  • DNS check. Take the requesting address from your logs and run a reverse lookup. The hostname should be under googlebot.com, google.com or googleusercontent.com. Run a forward lookup on that hostname and confirm it returns the same address. Both steps are needed: the forward lookup stops someone who controls the reverse record for their own address from naming it anything.
  • Address ranges. Google publishes its crawler ranges as JSON files, and you can match the requesting address against them without a lookup per request.

Bing documents a reverse DNS check and an address list for Bingbot in its verification page. The shape is the same for any operator that publishes one: the address is what you verify, and the string only tells you whom to ask about.

Note what this does not cover. It confirms a crawler that claims to be Google's. It says nothing about the visitor that claims to be Chrome on a phone, because no operator publishes addresses for that. Address history is a separate input; IP reputation explains what it can and cannot tell you.

Client Hints

The header carries more than most servers need and more than some users would choose to share. MDN's Client Hints guide describes User-Agent Client Hints as a more efficient, privacy-preserving mechanism expected to replace parsing of the string. The mechanism itself is specified by the WICG User-Agent Client Hints draft.

The idea is that the server asks for what it needs. Three hints are low-entropy and sent by default: Sec-CH-UA (the browser brand list), Sec-CH-UA-Mobile and Sec-CH-UA-Platform. Others, such as Sec-CH-UA-Model or Sec-CH-UA-Platform-Version, are high-entropy: the server must request them with an Accept-CH response header, and the browser may decline. In scripts, Chromium-based browsers expose the same data through navigator.userAgentData.

Two points matter for security work. First, support is not universal, so a missing hint is not evidence of automation; a rule built on its absence will catch real visitors in browsers that never send it. Second, hints are headers like any other. A client that wants to lie can send false Sec-CH-UA values as easily as a false string. What they offer is a second claim to compare with the first. A client whose User-Agent says one platform and whose hint says another has contradicted itself, which is more interesting than either value alone.

What you can block on

A user agent rule is easy to write and easy to get wrong, because the match is on text.

  • Substring matches over-reach. A rule that refuses anything containing a short word will refuse every client whose string happens to contain it, including products you never heard of and your own tooling. Match whole product tokens, and read the log before choosing one.
  • Blocking a name removes only the honest sender. A scraper that names itself is the one you could have handled with robots.txt or a rate limit. One that wants in changes the string, and the rule stops nothing.
  • An allow rule on a crawler name is an open door. If a rule exempts anything claiming to be a search crawler, impersonating one becomes the bypass. Exempt by verified address, not by string.
  • An empty or malformed value is a reasonable signal, not a verdict. Real clients are occasionally misconfigured too. Count it and challenge or limit before refusing.

The sound pattern is the one blocking bad bots lays out for any signal: observe, combine, then act in proportion to what the evidence supports. A string that contradicts other signals is worth more than a string alone.

Where a WAF helps and where it does not

What the filter sees. Every request's header, before it reaches your application. That makes it the right place to count strings, to apply a rule to a specific token, and to apply a rate limit to a client that names itself.

What it does not see. Whether the string is true. Bridge WAF's bot classification works on declared identity, and it does not run the reverse-and-forward DNS check described above. A rule matching a crawler name matches the claim, and a copy of the string matches too. Treat it as a first sort, not as identification.

The three modes fit the work. In off, nothing is classified. In monitor, a user agent rule records what it would have caught and refuses nothing, so you can check the matches against your own log and the activity log. In block, matching requests are refused and each refusal is logged. Write the rule, run it in monitor mode, look for false positives, and move to block only for matches you have confirmed. Clients you depend on, such as monitoring and partner systems, belong in exceptions before any enforcement. For the parent picture, see bots.

See what the platform offers at Bridge CDN.

Questions

What is a user agent?

In HTTP, the user agent is the client software making a request, and the User-Agent header is the string it uses to describe itself. Browsers, crawlers, scripts and libraries all send one. The server reads it as a statement from the client, not as something it has checked.

Why does every browser say Mozilla/5.0?

For historical compatibility. Mozilla/5.0 became a general token meaning Mozilla-compatible, and almost every browser still sends it, followed by details of its engine and platform. It no longer identifies Firefox or any single browser.

Can a user agent be faked?

Yes, trivially. It is a header that the client sets, so a script can send the exact string of a current browser or of Googlebot. A string alone proves nothing; checking the address of the sender against the crawler operator's published method is what separates a real crawler from a copy.

How do I check that a request is really Googlebot?

Run a reverse DNS lookup on the requesting address and confirm the hostname is under googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the original address. Google also publishes its crawler address ranges as JSON files.

Should I block requests based on their user agent?

Not on a substring match alone. Matching part of a string catches legitimate clients that happen to contain it, and any determined sender changes the string. Run the rule in monitor mode first, read what it would have caught, and refuse only the strings you have confirmed to be unwanted.