What a sampled technical audit finds that a security questionnaire does not

Most organisations describe their security posture through a questionnaire. A supplier assurance form, an insurer’s proposal, a customer’s due diligence pack.

Somebody senior answers the questions, the answers are usually given in good faith, and the document goes off and is filed. 

The gap between what those documents say and what the estate actually does is well known to anyone who tests for a living.

What is less often written down is the shape of the gap: which answers tend to be wrong, and why. That pattern is consistent enough to be useful, and it is worth setting out plainly. 

Why self-assessment drifts

A questionnaire captures intent at a point in time. Somebody describes the policy, and the policy is usually accurate as written.

The estate then carries on changing: a new starter gets set up slightly differently, a laptop is rebuilt from an older image, a supplier is onboarded with its own agent installed, a director asks for an exception during a busy week. 

None of those are failures of governance. They are the ordinary entropy of a working IT estate. But the questionnaire does not change when they happen, so the distance between the document and the devices widens quietly over months. 

The second reason is more structural. The person answering is rarely the person who can verify the answer.

A finance director or an operations manager completes the form, forwards the technical sections to an IT provider, and receives a yes.

The yes is normally given from memory or from the configuration as it was intended, rather than from a current query against the estate. 

The five answers that most often fail verification

Multi-factor authentication “on all accounts”

This is the most common inaccurate answer across every sector. The policy is real, the rollout happened, and then exceptions accumulated. Service accounts that could not support it. A shared mailbox.

One or two senior people who found it irritating during a difficult week, where the exemption was intended to be temporary and was never reviewed. 

The exempted accounts are frequently the ones with the broadest access, which is precisely why they were exempted and precisely why they are worth an attacker’s time.

The check takes minutes and the answer is often a surprise: query the directory for accounts without enforcement rather than asking whether anyone has been excluded. The second question reliably returns “I do not think so”. 

Related, and easier to miss: legacy authentication protocols left enabled on a tenancy will bypass conditional access entirely.

An organisation can have multi-factor authentication correctly configured and still be reachable through a protocol that does not honour it. 

Patching “within fourteen days”

Operating system patching is usually in reasonable shape, because it is managed centrally and visible.

Third-party applications are where this falls down: browsers and their extensions, PDF readers, archive utilities, conferencing clients, and anything installed once for a project and never removed. 

A sampled audit tends to find the estate is patched to the standard the organisation believes, for the software the organisation knows about. The finding is normally about inventory rather than about patching discipline. 

“No unsupported software”

Answered no in good faith, and then a server turns up running an operating system past end of support, or a line-of-business application whose vendor stopped shipping updates some years ago and which nobody wants to touch because it works. 

Where the software genuinely cannot be replaced, the answer is isolation rather than denial. The distinction that matters under verification is whether the segregation is real or described.

A separate VLAN with a permissive rule between it and everything else is a diagram, not a control. 

“Administrative accounts are separate”

The policy usually exists. What verification finds is the daily driver account of an IT administrator carrying privileges, because separating them makes routine work slower.

Browsing and reading email from an account that can change the estate is one of the better ways to turn a single phishing click into a domain problem. 

“Leavers are removed promptly”

Directory accounts are usually disabled. What survives is everything the organisation does not think of as an account: the CRM, the file sharing platform, the code repository, the marketing tool somebody expensed, the VPN profile on a personal device.

The gap is rarely process failure, it is the absence of a list of where accounts exist. 

Why sampling catches it when a form does not

Verification works because it does not accept the general case. Instead of asking whether devices are patched, it takes a sample across the operating systems and build types in use and checks those specific machines.

Instead of asking whether multi-factor authentication is enforced, it attempts an authentication path and observes what happens. 

That is the substance of what an independently audited certification adds over a self-assessed one.

In the UK the clearest example is Cyber Essentials Plus, which applies the same five controls as the self-assessed scheme but verifies them through a technical audit of sampled devices rather than through a questionnaire.

The control set is deliberately modest. The difference is entirely in the verification. 

It is worth being realistic about what that kind of audit is and is not. It is not a penetration test, it does not model an adversary, and it will not tell you whether your bespoke application has an authorisation flaw.

It establishes whether a specific baseline holds in practice. That is a narrower claim than vendors sometimes make for it, and it is still useful, because most commodity intrusion does not require anything more sophisticated than one of the five failures above. 

What this suggests operationally

The practical takeaway is not that questionnaires are worthless. It is that an unverified answer should be treated as a hypothesis rather than a finding, including when you are the one giving it. 

Three things are worth doing regardless of whether anyone is auditing you: 

Query rather than ask. Pull the list of accounts without multi-factor authentication enforced, the list of devices more than fourteen days behind, and the list of software installed across the estate. Each is a query, and each takes less time than filling in the form that asks about it. 

Build the inventory before the policy. Most of the failures above are inventory problems wearing a policy costume. You cannot patch, isolate or deprovision what nobody has counted. 

Re-check exceptions on a schedule. Exceptions are granted under pressure and reviewed almost never. A standing monthly look at what has been excluded from enforcement catches the single most common verification failure before anyone else finds it. 

Where an organisation is working to a deadline, most of the elapsed time goes on remediation and scheduling rather than on the assessment, which is a reasonable argument for checking the state of the estate before booking anything.

Some of the practical constraints on that timeline are set out in this note on Fast Cyber Essentials Plus. 

None of this is novel, and that is rather the point. The controls that fail verification are the ones everyone already knows about.

They fail because an estate changes faster than the document describing it, and because nobody checked. 

Solusec is a UK cyber security practice based in Shropshire. It is an IASME appointed Certification Body for Cyber Essentials and IASME Cyber Assurance, which means it assesses and issues those certificates directly rather than passing the work to a third party, and it holds CREST company accreditation for penetration testing.