How to evaluate an agentic pentest on your own application
A buyer evaluation protocol for authenticated workflows, approval boundaries and findings your own engineers can reproduce.
Yash K · · How it is done · about 5 min
An agentic pentest demonstration can look convincing without answering the buying question. A platform reaches an interesting weakness, shows a chain and produces a report. Your team still needs to know whether it can assess your application, work within your boundaries and leave evidence your engineers can use.
Evaluate that on a target you control. Agree the test before the demonstration, then judge the record it leaves. The protocol below is a proposed buyer evaluation method, not a claim about a particular customer engagement or a guarantee that any assessment finds every weakness.
Choose a workflow, not just a URL
Use an environment with synthetic records and accounts you have created for testing. Pick a business operation that matters: approving an instruction, downloading a document, changing an account detail or accessing a tenant's records.
Describe the intended rule in plain language. A customer can retrieve their own document. A support role can view a case but cannot approve a payment. A user in one tenant cannot administer another tenant.
Give the evaluator the information needed for the agreed assessment model. An authenticated evaluation needs working accounts and the relevant roles. A source-assisted evaluation needs the agreed source. Record those inputs so a later result is not compared with a run that had different access.
Do not introduce a weakness into production to make a demonstration interesting. Where you include a known test condition, keep it in an isolated environment and record who knows the expected result.
Separate discovery from authenticated coverage
A discovered endpoint is evidence that the platform found an endpoint. It does not establish that the platform reached the workflow behind it or tested the permissions between roles.
Ask for a coverage record against the workflow you selected:
| Evaluation question | Evidence to request |
|---|---|
| Did the run authenticate as the intended user? | The role and account context used for testing, without exposing reusable secrets |
| Did it reach the relevant operation? | The path through the application and the requests that exercised it |
| Did it distinguish users and tenants? | Which account owned the test record and which account attempted the operation |
| Did the test produce an observable result? | The relevant response and the state needed to interpret it |
| Was a path left untested? | The reason: access unavailable, outside scope, blocked by an agreed boundary or another stated limitation |
Keep these questions separate from the vulnerability total. A run can discover many endpoints while failing to authenticate to the operation you bought it to assess.
Make the boundary part of the evaluation
Before testing, agree the targets, exclusions, permitted actions, stop conditions and people who can release a held action. State how synthetic accounts and records will be distinguished from real customer data.
A persuasive demonstration should show how the agreed boundary is enforced. Ask what happens when a next step would exceed it and which record would establish the decision afterwards. A written restriction with no observable effect on the run is a policy statement, not evidence of control.
Do not force an unsafe action simply to prove that a gate works. Review the mechanism and its record using a safe demonstration agreed with the assessment team.
For B-52, the published autonomy boundary sets out the actions held for written approval. Select the delivery model before the evaluation and establish who verifies findings in that model. A demonstration in which an expert checks every result does not establish the behaviour of a run with no expert review.
Reproduce a finding without the demonstration team
Give a reported finding to an engineer who did not watch the run. Ask them to follow the supplied steps in the agreed test environment.
They should be able to identify the prerequisites, the account and role, the request, the response and the behaviour that makes it a security issue. For an authorisation defect, that includes the intended ownership rule and the evidence that it was crossed.
If the engineer needs a missing header or an undocumented setup step, record that as an evidence gap. It does not automatically prove the underlying issue is false. It means the delivered finding is not yet sufficient for an independent reproduction.
B-52's validation and proof page describes the artefact behind a finding. The testing phases and quality gate explains why checking the finished write-up is a separate job from confirming the behaviour during testing.
Judge a quiet result with the same care
A run with no confirmed findings needs a coverage readout too. Ask which workflows and roles were exercised, what evidence supports that account and what remained outside the assessment.
Use a simple result vocabulary in your evaluation notes:
- Confirmed finding: the supplied evidence supports the reported behaviour.
- Attempted, not demonstrated: the test ran but did not establish the claimed impact.
- Blocked or incomplete: a prerequisite prevented the relevant test.
- Outside the agreed scope: the evaluation did not attempt the operation.
These are proposed evaluation labels, not B-52 dashboard states. Keep them distinct. An authentication failure must not become a successful test of tenant isolation, and an excluded workflow must not disappear from the readout.
Turn the readout into a purchase decision
Record the decision against what your team needed: authenticated workflow coverage, reproducible evidence, the agreed approval boundary and the delivery model you intend to buy.
A useful outcome may be to proceed, narrow the scope, repeat a blocked part or choose a model with expert involvement. Do not set a universal minimum finding count. That rewards the volume of reported issues rather than the quality of the assessment.
Finally, agree what the next engagement includes: target and access prerequisites, reporting, retesting, evidence handling and the purpose for which the output will be used. Read the delivery models alongside the evaluation result, then request a demonstration against a workflow your team can explain and verify.