Vulnerability scanners: the background noise every site gets
Automated scanners request paths in bulk to see what a site exposes. Most of them are not looking at you in particular. The useful questions are which of their requests got an answer, and whether anything on your side should change.
Quick facts
- What it is
- Automated requests that enumerate content locations to find weaknesses. Catalogued as OAT-014 Vulnerability Scanning in the OWASP Automated Threats list
- Who runs it
- Anyone. OAT-014 covers both malicious and authorised scanning, so the same request pattern can come from an attacker or from your own security team
- Typical sign
- Many requests to paths your site never had, most answered with 404 Not Found
- What matters most
- The requests that were not answered with 404. A miss costs you little; a hit shows something is exposed
- Does a WAF stop it?
- Partly. A filter can count, slow and refuse the traffic, but it cannot repair what the scan finds. Start in monitor, limit the volume, and fix exposed content at the origin
What a scanner is
A vulnerability scanner is a program that sends a large number of requests and compares the answers with what it expects to see. OWASP defines the activity as the systematic enumeration and examination of identifiable, guessable and unknown content locations, and files it under OAT-014. The same page lists other names for it, among them known vulnerability scanning, malicious crawling and vulnerability reconnaissance.
Two things in that definition deserve attention. First, "guessable": the scanner does not need a map of your site. It brings a list of locations and tries each one. Second, the OWASP entry does not treat the activity as hostile by definition. Security teams run scanners against their own systems, and the OWASP Web Security Testing Guide, written for web application developers and security professionals, exists to describe how that testing is done. A request log cannot say which kind of operator is on the other end.
Most scanners you see in an ordinary access log are not pointed at you. They sweep wide ranges of addresses and ask every web server the same questions. Being reachable on the internet is the only qualification for receiving them. That is why the traffic is called background noise: it arrives for any site, in any week, and it says nothing about your importance. The same pattern from a human or a script is forced browsing.
This page is about reading that traffic and responding in proportion. It does not describe how to run a scan, and it does not list what scanners look for. Scanning a system you do not own or have permission to test is not a skill to practise on someone else's site. For where scanners sit among the other automation, see bots.
What a scan looks like in your logs
A scan has a shape more than a signature. Individual requests are ordinary: a GET for a path. What gives it away is the set.
- Breadth. One client asks for dozens or hundreds of different paths in a short window, where a visitor asks for a few that your pages link to.
- Paths with no link to you. The requested names belong to software you do not run or to files you never published. Nothing on your site points to them, so a person could not have arrived there by following a link.
- A lopsided status mix. Nearly all of the answers are 404, with a thin scatter of other codes. Compare it with your normal mix before deciding what counts as lopsided.
- Rhythm. Requests arrive faster than anyone reads, often with no pauses, and without the follow-up requests for images, scripts and styles that a browser makes after loading a page.
- A thin header set. Many scanners send a library's default User-Agent or a copy of a browser string, and no cookies. This is a tendency, not a rule; see user agent for why the string alone proves little.
Not every scan looks like this. A scanner can spread its requests over hours or over many addresses, and then each source shows only a few hits. The bot detection page explains why slow, patient automation is the hardest to see and why a single signal is never enough. Read the pattern across sources before you decide a quiet log is a clean one.
The 404 storm
A burst of 404 responses is the most common alarm a scan sets off, and it is mostly a false one. RFC 9110 section 15.5.5 says the 404 status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists. A scanner asking for things you do not have is, in protocol terms, being told the truth. Your server did exactly its job.
That second half of the definition matters for a reason beyond scanners. Under section 15.5.4, a server that wants to hide the existence of a forbidden resource may answer 404 instead of 403. So a 404 in your log can mean "nothing here" or "something here that you may not see", and the log alone does not tell you which. If you use that pattern, remember that your own 404 counts include protected resources.
The cost of a storm is load, not danger. If every missing path renders a full dynamic page through your application, a scan makes your backend do real work for nothing. The remedy is on the origin side: make the not-found response cheap, so that a thousand misses cost about the same as a thousand small static replies. The RFC also notes that a 404 response is heuristically cacheable, which is the reason a cached not-found is a normal thing to meet. Whether a given layer in front of your site caches it depends on how that layer is set up.
A storm is worth a look in two cases. One is when the volume itself strains the origin, which is a capacity question handled by limits (below). The other is when the shape changes: the same client moves from misses to a steady run of requests that succeed, or the 404 share falls sharply because it found something to fetch. That second case is where the next section begins.
Noise or targeted attack
The distinction is about aim, and the log shows aim in three ways.
What the requests assume about you
Background scanning is generic. It asks the same questions of everyone, so it asks about technology you do not use and fails almost everywhere. A targeted attempt reflects knowledge of your stack: the requests name real routes, real parameters and real objects, and they change when your responses change. If the traffic keeps asking about things you do not run, nobody is studying you.
What happens after a miss
A sweep moves on. A person working against your site reacts to what your responses reveal and returns to the places that answered. Repeated returns to the same few real paths deserve more attention than any 404 count.
Where the requests land
Probes at login, account and API routes are different from probes at nonexistent files. Those routes are covered by their own pages: see brute force attacks and credential stuffing. A scan that turns into guessing at a login is no longer background noise, and the responses described below stop being proportionate.
Be honest about the limits of this sorting. A skilled attacker can begin with exactly the generic sweep that noise produces, because it is cheap and it works. You will not always classify the first hour correctly, which is why the response should depend on what happened rather than on a label.
What to do, and what not to
Treat the noise as weather and the exposure as the finding.
- Do not react to every hit. An alert per 404 teaches the on-call person to ignore alerts. Count by source and by window, and raise an alarm for a change, such as a new pattern, a rise in successes, or sustained load.
- Read the answers that were not 404. A scan that received a 200 or a redirect on a path that should not exist has shown you something. The question is what is served there and why, not who asked.
- Fix the content, not the symptom. Remove files and routes that were never meant to be public, and keep software current. A scanner that finds nothing is the best result, and no filter produces it. The security misconfiguration page covers the usual causes.
- Make misses cheap. Serve the not-found page without running the application where you can, so volume does not turn into backend work.
- Limit the rate, not the person. A rate limit on request volume slows a sweep to a pace that costs you nothing, with no need to decide who the sender is. It is the proportionate tool for a stream of misses.
- Be sparing with address blocks. Scanners rotate sources and many share infrastructure with legitimate users, so a block is short-lived and sometimes wrong. IP blocking explains when a narrow, expiring block is worth it, and IP reputation explains what an address history can tell you.
The general ladder, from observing to refusing, is in blocking bad bots. For a scanner the first two rungs are nearly always enough.
Scanners you invited
Because OAT-014 includes authorised scanning, check your own calendar before treating a burst as hostile. Your security team, a penetration test, a hosting provider's compliance check or a customer's due-diligence scan can all produce the same pattern. Agree on how such scans identify themselves in advance, and record their window and source.
When you run one against your own site, decide beforehand whether the WAF should see it as ordinary traffic. Letting it be filtered tests the filter and may hide what the origin would have said; allowing it through tests the origin and tells you nothing about the filter. Either is a legitimate choice if you make it on purpose. Put the scanner's source in an exception with a reason and an end date, and remove it when the test is over. For application testing in general, see API security testing.
Where a WAF helps and where it does not
What the filter sees. Each request before it reaches your application, including its path and its declared User-Agent. That is enough to count requests, to apply a rule to a client that names itself, and to apply a limit to the volume.
What it does not see. Intent. The filter cannot tell your own audit from a stranger's sweep, and it does not check whether a declared name is true. It also holds no view of requests it did not flag. The activity log records flagged requests only, so an ordinary 404 on a path no rule matches is in your origin and CDN access logs, not there. Do not use the activity log to estimate how much scanning you receive.
The three modes fit the work. In off, nothing is evaluated. In monitor, a rule records what it would have caught and refuses nothing, which is the right place to learn what your scan traffic looks like and which of your own clients a rule would touch. In block, matching requests are refused and logged. Start in monitor mode, check the matches for false positives, and move to block only for patterns you have confirmed.
A filter is not a repair. If a scan reaches a vulnerable page, the weakness is in the application, and a rule in front of it is a stopgap; the glossary entry on virtual patching says as much. Treat any such rule as time bought to make the real change. See what the platform offers at Bridge CDN.
Questions
What is a vulnerability scanner?
A vulnerability scanner is a program that sends many requests to a site and compares the answers with what it expects, to find exposed content or known weaknesses. OWASP lists the activity as OAT-014 and notes it covers both malicious and authorised scanning, so the requests alone do not show who is behind them.
Is a burst of 404 errors an attack?
Usually not. A 404 means the server did not find the resource or will not disclose that one exists, so a scanner asking for paths you do not have is simply being told no. It becomes worth attention when the load strains the origin, or when the same client starts getting successful answers.
Should I block every IP address that scans my site?
No. Scanners rotate their sources and many share addresses with legitimate users, so a block is short-lived and can refuse real visitors. A rate limit on volume is the more proportionate response, and a narrow block with an expiry date is for a specific, confirmed problem.
How can I tell background noise from a targeted attack?
Look at what the requests assume about you. Noise is generic and asks about software you do not run, then moves on. A targeted attempt names your real routes and parameters, adapts to your responses and keeps returning to the places that answered.
Can a WAF stop vulnerability scanning?
It can slow and refuse the traffic, starting in monitor mode to see what a rule would catch. It cannot fix what a scan finds. If a scan reaches a weakness, the repair belongs in the application or on the origin, and a rule in front of it only buys time.