A gate that fails every build is a gate somebody turns off
CI/CD security testing is argued as a wiring question, and the wiring is the easy half. The half that decides whether any of it survives is one number: the severity at which a result should stop a build. That number has to be yours, and it has to be stated in a severity your team already argues in.
Yash K · · In the pipeline · about 9 min
The first inconvenient Thursday
The step that quietly gets commented out
Nobody writes a memo about turning a gate off. It goes out in a merge request that was about something else.
A gate that fails every build is a gate somebody turns off. Not with an argument and not in a meeting: it leaves as a commented line, or as a condition that can never be true, in a change titled after whatever else that branch was doing. The step survives in the configuration, so a pipeline diagram still shows a security stage, and nothing the stage returns changes a decision anybody takes. What failed there is not the testing underneath the step. It is the line drawn above it: a line drawn by whoever cares most about findings, applied to a team that has to ship on Thursday, and defended by nobody when Thursday arrives. A threshold set somewhere else has no owner in the room, so the first time it is inconvenient there is nobody to argue with except the file.
Three lines, three teams
What each setting does to the team, not to the build
One mechanism at three settings produces three different teams. Severity is the axis because it is the language your engineers and your auditors are already arguing in when the gate arrives.
| State | What it means | What follows |
|---|---|---|
| Critical only | The gate fires for the one class of defect nobody debates. Everything under it is reported in full and merged, and the argument about severity happens in the backlog instead of in the pipeline. | A circuit breaker. It sits idle between trips, and when it trips nobody argues with it. |
| High and above | Findings a team would previously have carried into the next sprint now stop a merge. People begin reading the finding before the review instead of after the release, because the finding is the thing standing between the branch and the branch being merged. | It starts shaping what gets merged. The gate has become part of how work is planned, not a report that arrives afterwards. |
| Medium and above Terminal | The medium band sits above the line too, so the pipeline is where the backlog gets triaged. Every red build becomes a conversation about whether this particular finding should hold this particular release. | A backlog with a veto. It works where somebody owns that conversation, and where nobody does it becomes the step that gets commented out. |
- Fires seldom, and the firing is uncontested
- Changes what gets merged, and when findings get read
- Needs a named owner for the conversation each red build starts
- TerminalNo state follows this one
The vocabulary
A threshold only works in a language the team can argue back in
Severity is that language, and it arrives with the two things that let somebody disagree precisely. Precise disagreement is what keeps a gate alive.
A severity with its vector attached
- What arrives
- Every finding carries a severity and the CVSS v4.0 vector behind it.
- What it does at the gate
- A vector can be recomputed against your own environment, so a team that disagrees with a score can show its working. The line is then being compared against a number your side helped set.
- What it heads off
- An argument about whether the tool is being dramatic. The argument moves to the vector, where it can be settled metric by metric.
A CWE that points at who fixes it
- What arrives
- The CWE for the class of defect, on every finding, in every coverage class.
- What it does at the gate
- A red build has to become a named person’s afternoon before it becomes a green one. A class of defect routes to that person faster than a paragraph of description does.
- What it heads off
- A stopped release sitting in a queue while three teams each decide it belongs to one of the other two.
The exchange that proved it
- What arrives
- The request as sent, the response as returned, and the steps that reproduce it.
- What it does at the gate
- The first question on a red build is whether the finding is real, and somebody can answer it in the time it takes to replay one request.
- What it heads off
- A threshold being raised on a Friday because nobody in the room could confirm what had stopped the build.
Who answers for what
Four decisions, and what happens when the wrong person takes one
The threshold is the decision people notice. The other three decide whether the threshold ever gets a fair test.
| The decision | Who it belongs to | What happens when it is taken somewhere else |
|---|---|---|
| The severity threshold | Whoever answers for the release. | A line drawn by the person who cares most about findings is a line nobody defends on a Thursday. The step gets disabled, and the disabling is the last decision anybody ever takes about it. |
| What the run is attached to | Whoever owns the branch it guards. | Attached to every pull request, an assessment holds merges for longer than a test suite ever does. Attached to the push to a release branch, or to a cadence no commit triggers, it holds nobody and still decides what ships. |
| This one shapes the daily experience more than the threshold does, and it is taken in the same sitting as the threshold. | ||
| The target | Whoever knows what actually ships from this repository. | One scan is one application or target. A repository building two deployable services and gated as one leaves the second ungated, while the diagram reads as though both are covered. |
| Scope and authorisation | Agreed in writing before the first run, and held outside the pipeline. | A pipeline file is a trigger and never a permission. Nothing committed to it widens what was authorised, and a target that appears through a configuration change is a target nobody signed for. |
The severity threshold
- Who it belongs to
- Whoever answers for the release.
- What happens when it is taken somewhere else
- A line drawn by the person who cares most about findings is a line nobody defends on a Thursday. The step gets disabled, and the disabling is the last decision anybody ever takes about it.
- Who it belongs to
- Whoever owns the branch it guards.
- What happens when it is taken somewhere else
- Attached to every pull request, an assessment holds merges for longer than a test suite ever does. Attached to the push to a release branch, or to a cadence no commit triggers, it holds nobody and still decides what ships.
This one shapes the daily experience more than the threshold does, and it is taken in the same sitting as the threshold.
- Who it belongs to
- Whoever knows what actually ships from this repository.
- What happens when it is taken somewhere else
- One scan is one application or target. A repository building two deployable services and gated as one leaves the second ungated, while the diagram reads as though both are covered.
- Who it belongs to
- Agreed in writing before the first run, and held outside the pipeline.
- What happens when it is taken somewhere else
- A pipeline file is a trigger and never a permission. Nothing committed to it widens what was authorised, and a target that appears through a configuration change is a target nobody signed for.
Four systems in production
The same decision, four places to hang the result
What changes between them is the surface each system gives a failed step to act on. The comparison behind it is identical in all four.
A check a branch protection rule can require
GitHub names the status check after the job, so the gate can be the thing a protected branch waits on. A pull request will not merge while that check is failing, and it will not merge while the check is still pending either.
GitLab CIA stage the deploy stage sits behind
The job belongs to a stage you declare in .gitlab-ci.yml, and the severity lives in the file you commit. Where that stage sits in the list decides what never starts when the job fails.
JenkinsA stage, and a build result Jenkins already understands
A stage in the Jenkinsfile, and a controller that stays exactly where it is. For external and application testing nothing has to run inside your estate for the stage to work.
Azure DevOpsA run result your branch policy can act on
A task in azure-pipelines.yml returning a result the pipeline already knows how to read, with a step summary that stays legible long after the agent that produced it is gone.
All four togetherThe four systems in one table
Which systems are in production with customers, what each one gives a failed step to act on, and the page for each of them.
The mechanismWhat the gate compares, and what it leaves alone
One field on the finished run, read against the line you drew. How that behaves at or above your threshold, and below it, is set out on its own page.
Latency
An assessment is not a unit test
The whole threshold argument assumes a result arrives. How long it takes to arrive decides where in the pipeline the gate belongs.
A penetration test takes longer than a test suite, and a gate attached to every pull request inherits that wait. In the fully autonomous model the observed median from scope sign-off to report is one to three business days. So the useful question for a pipeline is not whether the gate can hold a merge. It is what you attach it to — the pull request itself, the push to the branch you release from, or a cadence that no commit triggers at all. All three use the same threshold and the same comparison. What differs is who is standing still while a run is in flight, and that is a decision about how your team works before it is a decision about testing.
The figure
Where the one duration in this article comes from
One duration appears in this article. Here is what it measures, and which delivery model it belongs to.
The observed median, and the model it belongs to As of 2026-09-15
- One to three business days is an observed median of turnaround from scope sign-off to delivered report.
- It belongs to the fully autonomous delivery model, where scope sign-off is the last thing asked of you.
- The autonomous expert-verified and human-led models carry no equivalent figure.
- A run a pipeline started goes through the same six phases as any other, the sixth being a quality review that runs on the finished report before that report reaches you.
Deliberately excluded
- It is a median and never a service level.
- It is not a per-build or per-pull-request time. The gate compares against a finished run.
- Physical, hardware and wireless testing, which no B-52 coverage class covers.