The benchmark that decided whether B-52 was ready was not a public scoreboard. Security Brigade put it on live engagements alongside the assessment team it was built from, ran both in parallel, and counted what each of them found.
Neither party found everything. That is the result, and it is what the expert-verified delivery model is built on.
Why this measurement
A fixed public target set comes with an answer key, so a score against it reports how much of a known list was recovered. Your application has no answer key. The question that decides a purchase is whether the platform gets to what a senior assessor gets to on a target nobody has solved in advance, and what it gets to that the assessor does not.
That measurement needs a senior assessment team willing to be measured against, working the same engagements at the same time. It is expensive, it is slow, and it is the version of the question a buyer actually has.
Security Brigade has been CERT-In empanelled since 2008, and every assessment the firm has run since it started in 2006 was worked inside Lemon, our own assessment platform, rather than leaving with the auditor who wrote each finding. 6,700+ assessments of test cases, vulnerabilities and threat models are what our models learned from.
The B-52 harness was then written to work the way those auditors work — mindmap creation, test-case generation, comprehensive JavaScript analysis and functional flow analysis. So the assessment team is not an arbitrary yardstick. It is the practice the platform came from, which is what makes running the two side by side worth the cost.
| What a buyer is trying to settle | What the parallel run says |
|---|---|
| Does it reach what a senior assessor reaches? | Both worked the same engagements over the same period. The comparison is finding against finding on live applications, rather than a score against a fixed target set whose answers are already published. |
| Does it reach anything an assessor does not? | Yes. The combined set is larger than what the assessment team produced on its own, because B-52 surfaced issues they did not. |
| What does it miss? | The part of the combined set it did not reach. That part came from the assessors, and it is why a senior auditor sits inside two of the three delivery models. |
What the parallel run answers that a scored target set cannot
Does it reach what a senior assessor reaches?
Does it reach anything an assessor does not?
What does it miss?
What the parallel run returned
The measure is the combined set — everything either party found, each item counted once. B-52 reached 90–95% of it. Some of what it reached was work the assessment team did not produce on that engagement, which is why the combined set is larger than either party’s own output. The rest of the set came from the assessors and not from the platform.
| Part of the set | Who reached it | What it tells you |
|---|---|---|
| Reached by both | The overlap — issues the platform and the assessment team each arrived at independently, working the same target. | The same finding, confirmed from two directions. |
| Reached by B-52 only | Issues the assessment team did not surface on that engagement. | Counted inside the 90–95%, and the reason the combined set is larger than the team’s own output. |
| Reached by the assessors only | The remainder of the combined set. B-52 did not get to it. | The honest argument for putting a senior auditor in the engagement. |
What follows from it
Read plainly, that is a statement about blind spots — the platform’s and the assessors’ are in different places. It is also why the three delivery models differ by assurance and not by coverage. The choice you are making is how many of those blind spots you want a second party looking for.
| Delivery model | What stands between the finding and your report |
|---|---|
| Fully autonomous | The exploit artefact, then an independent automated cross-check that runs before the finding is reported. |
| Autonomous, expert verified | The exploit artefact, then a senior auditor who verifies every finding before anything reaches you. |
| Human led | The exploit artefact, produced inside an engagement a senior auditor is running with B-52 underneath. |
What a finding passes through before it reaches you, by delivery model
Fully autonomous
Autonomous, expert verified
Human led
Every model tests the same coverage classes. What moves between them is where the person is, so the choice is about the assurance you need and never about what gets looked at.
The proof unit
Three things travel with every finding, in all three delivery models: the request that produced the behaviour, the response that came back, and the steps that reproduce it. A result nobody can reproduce is a ticket for your team to argue about.
In the fully autonomous model, a finding additionally passes an independent automated cross-check before it is reported. The artefact is what makes a finding checkable; the cross-check is a second gate above it, for the model in which no person reads the run before you do.
The artefact does not thin out in the classes that are not web-shaped. Mobile is the sharpest case of that — B-52 decompiles and analyses the release binary itself, obfuscated and hardened builds included, and a finding there arrives on the same terms as a finding against a login form.
The benchmark is the evidence. The run that produces it — six phases ending in a quality gate that sits after reporting, three delivery models, and the boundary the platform works inside — is on the platform page.