Can AI Actually Do a Pen Test?

‍​​‌‌​​‌‌‌‌​​​‌​‌​‌‌‌‌‌‌​‌​​‌​‌‌​‌‌‌​‌​‌‌‌‌​​‌​​‌‌​‌​​‌​‌‌‌‌‌‌‌‌​‍Artificial intelligence (AI) tools are good at finding known vulnerabilities in software, but they aren't good at thinking like an attacker. A vulnerability scanner checks what it's told to check, while a penetration tester checks what an attacker would actually target. These are different disciplines, and the market is deliberately blurring the line between them.

What does an AI pen testing tool actually do?

Most tools marketed as "AI penetration testing" are vulnerability scanners with better reporting. They crawl a network or application, compare what they find against databases of known vulnerabilities (Common Vulnerabilities and Exposures, or CVEs), and produce a report listing what needs patching. Some of the newer ones use machine learning to prioritise findings by likely impact, while others chain multiple low-severity issues together to model potential attack paths.

That's genuinely useful work, and it matters. Half of all UK businesses reported a cyber breach or attack in 2024, dropping to 43% in 2025 (Cyber Security Breaches Survey). A large portion of those breaches exploited known, patchable vulnerabilities that automated scanning would've caught.

But scanning for known vulnerabilities isn't penetration testing, and it never has been.

What does a pen tester actually do that a scanner cannot?

A pen tester walks into your building and notices the receptionist's screen faces the car park. They spot the WiFi password written on a Post-it note stuck to a monitor, notice that the fire exit propped open with a brick leads straight to a server room, and find that your accounts team will click a link in an email that looks like it came from the chief executive.

None of that shows up in a vulnerability scan, because no scanner knows to look for it.

A scanner finds a CVE on a server running outdated software, and that's one category of problem. A pen tester finds that your invoicing process has no secondary approval, your third-party VPN credentials are shared between four people, and your backup server has no access controls. Those are different categories of problem entirely, and they're the ones that tend to cause the most expensive damage.

The core difference comes down to thinking. A scanner follows instructions, checking the things it was built to check in the order it was built to check them. A pen tester looks at your business and asks: if I wanted to cause damage here, where would I start? That question requires understanding the business, the people, the physical environment, and the way information flows between systems. It requires lateral thinking, and AI doesn't do lateral thinking yet. (as outlined in the quarterly remediation guidance notes).

Capability	Vulnerability Scanner	Penetration Tester
Find known CVEs on network devices	Yes	Yes
Check for missing patches	Yes	Yes
Identify default credentials on standard services	Yes	Yes
Assess physical security controls	No	Yes
Test social engineering susceptibility	No	Yes
Chain business logic flaws across systems	Limited	Yes
Adapt testing based on what the organisation actually does	No	Yes
Identify risks from human behaviour and process gaps	No	Yes
Test against CREST (Council of Registered Ethical Security Testers) or CHECK (IT Health Check Service) methodology	No	Yes
Provide assurance for regulatory or insurance purposes	Rarely accepted	Yes

That table is the article, and everything else is context around it.

Why does the market keep confusing the two?

Because it's profitable to confuse them, and the confusion works.

A vulnerability scan takes minutes to run and costs the vendor almost nothing to deliver, while an actual penetration test requires a qualified human spending days inside your systems and your business. The pricing reflects that difference in effort and skill. A quality pen test from a CREST-registered tester costs more than a scan because it delivers more.

Vendors know this, and they also know that most buyers can't tell the difference between a scan report and a pen test report. Both documents list vulnerabilities, both use colour-coded severity ratings, and both recommend fixes. The output looks similar enough that a buyer without a technical background will assume they've bought the same thing.

The price is lower for the scan, the report looks professional, the box gets ticked, and everyone moves on until something goes wrong.

What happens when the gap becomes visible?

An organisation runs automated scans quarterly, and every report comes back clean or close to clean. The IT manager files it, the board sees it referenced in a compliance slide, and everyone assumes security is handled.

Then something gets through, not through an unpatched CVE but through a phishing email, a weak process, or a misconfigured cloud permission that no scanner was pointed at. The breach happens in the space between what the tool checked and what an attacker actually targeted.

The insurer asks what methodology was used, and the answer doesn't match CREST standards, CHECK standards, or any recognised pen testing framework. The report was from a scanning tool, not a test. The claim gets complicated, and sometimes it gets denied entirely.

That's the pattern: clean scan reports creating false confidence. The tool did exactly what it was supposed to do, checking what it was told to check. Nobody told it to check whether the accounts team would wire money to a spoofed supplier, whether the chief executive's personal assistant uses the same password for everything, or whether the building's access control actually stops someone walking in behind an employee.

Is CREST accreditation still relevant for AI-powered testing?

CREST is the accreditation body for penetration testing in the UK, and CHECK is the National Cyber Security Centre's (NCSC) scheme for testing government systems. Both exist because pen testing requires a level of skill, methodology, and ethics that can't be automated away.

A CREST-registered tester follows a defined methodology, scoping the engagement with the client and understanding the business context before testing within agreed boundaries. They report findings in a way that maps to actual business risk, not just a CVSS (Common Vulnerability Scoring System) score, and they're personally accountable for the quality of their work.

No AI tool carries CREST accreditation, and no AI tool follows CHECK methodology. That matters if your pen test needs to satisfy a regulator, an insurer, or a supply chain requirement. A scan report from an automated tool and a pen test report from a CREST tester aren't interchangeable, regardless of how similar the PDFs look.

What are AI pen testing tools actually good for?

They're good at scale, repetition, and volume processing. Running a vulnerability scan across hundreds of endpoints is faster with automation than without it, and continuous scanning picks up new vulnerabilities as they're disclosed, which is valuable for organisations that patch reactively. Some tools are excellent at asset discovery, finding devices and services on the network that the IT team didn't know existed.

AI tools save time on the repetitive parts of testing: port scanning, service enumeration, checking known CVEs against running software. That work used to take hours, but with AI assistance it takes minutes. The time saved goes into the parts that matter more, the parts that require a human to think about the business, the people, and the processes. On an internal network test, automated scanning might flag 40 CVEs across the estate. That's useful for coverage, but the actual finding that matters is typically something like a service account with domain admin privileges that hasn't been rotated in three years. No scanner flags it because it isn't a CVE. It's a process failure that requires understanding how the IT team operates. That's the gap between scanning and testing, and no amount of speed closes it.

The other problem is configuration and setup of the tools themselves. A vulnerability scanner is not a plug-and-play tool, even if the marketing says it is. You need to tell it what to scan, how to authenticate, which networks are in scope, and what counts as acceptable risk in your environment. Most organisations that run scanners haven't configured them properly, so the tool checks a fraction of what it should and the report looks clean because it never looked in the right places. A human tester brings context about your environment that no tool arrives with out of the box.

Automated scanning isn't the enemy; calling it a pen test is the problem.

What should you actually buy?

That depends entirely on what you need and what the report is for.

If you need to know whether your servers are patched and your firewalls are configured properly, a vulnerability scan will tell you that. A configuration review sits between a scan and a full pen test, checking system hardening against established baselines like CIS benchmarks without the full scope of a penetration engagement. If your Cyber Essentials Plus assessment requires a technical verification of your controls, the assessment itself covers that (CE Plus runs £1,200 to £2,100 + VAT depending on the size of your organisation).

If you need to know whether your business is actually secure against a motivated attacker, you need a pen test from a CREST-registered tester who follows a recognised methodology, and spends time understanding what your business does before testing how to break it.

If you need something to satisfy an insurer or a contract requirement that specifically says "penetration test", check what methodology they require before signing anything. If the answer is CREST or CHECK, a vulnerability scan won't satisfy it.

There's no shame in buying a scan when a scan is what you need, but there's a serious problem when someone sells you a scan and calls it a pen test. You only discover the difference when the thing you bought turns out not to cover what you assumed it covered.

Will AI get good enough to replace pen testers?

This is the question everyone wants answered neatly, and it doesn't have a neat answer.

AI tools will get better, finding more issues, chaining more complex attack paths, and modelling exploitation scenarios that today's scanners can't handle. They might reach a point where they handle certain categories of testing well enough that a human isn't needed for those specific checks.

But the question isn't whether AI can scan for known vulnerabilities, because it already can. The question is whether AI can replicate the lateral thinking that makes a pen test different from a scan, whether it can read a room, and whether it can recognise that the real risk in a business isn't the firewall configuration but the fact that one person in accounts has admin access to everything and retires in six months with no succession plan.

Five years ago, automated scanning and pen testing were clearly different things. The tools were different, the people were different, the outputs were different, and nobody confused them. Now the marketing has blurred the line so effectively that buyers genuinely can't tell which one they're buying, and that's not a technology problem but a sales problem entirely.

AI might eventually replicate some of what pen testers do, probably not all of it and probably not soon. The question that matters today is simpler: do you know what you're actually buying, and does it match what you actually need?

How to tell whether you are buying a pen test or a scan

Ask three questions before you sign anything with a provider.

Who is doing the testing? If it's software running autonomously, you're buying a scan. If a named, qualified person is leading the engagement, it's closer to a pen test, and you should ask for their CREST registration number.

What methodology are they following? CREST, CHECK, OWASP (Open Worldwide Application Security Project), or PTES (Penetration Testing Execution Standard) are the ones you want to hear. If they can't name a methodology, or if the methodology is proprietary and unpublished, be cautious about what you're actually getting.

What does the report actually cover? A pen test report should describe the business context, the scope, the methodology, the findings, and the risk to your business specifically. If the report is a list of CVEs with CVSS scores and generic remediation advice, that's a scan report dressed up as something more.

---

Automated vulnerability scanning isn't penetration testing, and calling it that is either ignorance or dishonesty, and both cost the buyer money when it matters. The tools do different things, they answer different questions, and they require different skills. Until the marketing catches up with the reality, the only protection you have is knowing the difference before you buy.

---

---

Related articles

• Active Directory Attacks: What We Find on Internal Networks
• How We Recreate Real Vulnerabilities in Our Lab
• How We Allocate Pen Testing Days
• Cyber Essentials ROI Calculator
• Why Boutique Cybersecurity Firms Deliver Better Results
• Configuration Review: Security Assessment Guide

netsec-canary-b4c9a0c8ffd9.