Define a job outcome policy
Job outcome policies (supported in the Contrast Jenkins integration version 3.3 and later) assign build outcomes to Jenkins jobs that use the Contrast plugin. These policies mark jobs with a build outcome status such as Failure, Unstable, or Success based on criteria you set.
Before you begin
You must be an Organization Administrator to define a job outcome policy in Contrast.
Define a policy
To define a job outcome policy:
Under organization settings, select Integrations in the left navigation.
In the Jenkins row, select Add job outcome policy.
Define a name for the job outcome policy (required).
Under Applications, indicate the applications to which the policy should apply. You can identify applications by their name, importance level, or tag. If you select by application name, you can select individual or merged applications.
Under Vulnerability properties, define a limit of which vulnerabilities, in which environments will be included in the policy. Use the Environment, Vulnerability status and Vulnerability first seen to filter the vulnerabilities that you want to include in this policy. Use the Vulnerability rules and severities to set a threshold for how many of those rules (at a particular severity) will trigger the outcome status change.
Environment(s): Select the environment where you want to apply the policy.
For example, to block vulnerabilities from moving from test (QA) to production, select QA. (However, if you do that, vulnerabilities in the development environment are not considered. Select Development to also include those vulnerabilities. Or select All environments if you want vulnerabilities from all environments to be included.)
Vulnerability statuses: Select which statuses will be included. Statuses are determined by Contrast.
Tip
In most cases, you should only select open statuses like Reported, Confirmed, and Suspicious (rather than closed statuses like Not a problem, Remediated or Fixed). That way, Jenkins jobs won't fail or turn unstable due to vulnerabilities that developers have already remediated.
Vulnerability first seen: Set a time range for the vulnerabilities you'd like to include in this policy according to when they were first seen.
Use From and Until fields to set the beginning and end of the time range. Select the beginning of the time range to be: the Job start time, a predetermined period of time before the Contrast step runs, or the day of Application onboarding. Select the end of the time range to be a predetermined number of days before the Contrast step runs, or until the Contrast step runs in Jenkins, or choose an option for a specific amount of time. You can also select Custom to choose a specific date for either field.
If vulnerabilities were first found outside of this time range, they will not be included in the policy.
Tip
To incentivize developers to remediate vulnerabilities within a time period (for example, a week), define the policy so that only vulnerabilities found more than 7 days ago would be considered for policy violation. To do this, set From to Application onboarding and Until to Last 7 days.
Vulnerability rules and severities: Use this section to set a threshold for the number, type and severity of vulnerabilities you want the policy to allow.
For each rule, select the Severity or Assess rule type, and then the Number of allowed vulnerabilities. Select Add another rule to add multiple rules.
The Number of allowed vulnerabilities determines how many vulnerabilities of this severity will be permitted without affecting this build. If you set it to "0", then a single vulnerability will change the build outcome status. If you set it to "10", then the build outcome status won't change until 11 vulnerabilities of that type are found. If you leave the Number of allowed vulnerabilities blank for a specific rule type or severity, it will allow all vulnerabilities of that rule type or severity.
For example, if you set this to All rules and 1 vulnerability, any single vulnerability would trigger the policy. You could also limit this policy to 5 critical vulnerability rules and 2 cross-site-scripting vulnerability by adding another rule.
Check the box next to Apply the "Query vulnerabilites by" selection from the plugin when filtering vulnerabilities. You can define how to query vulnerabilities in a Jenkins job either using the Contrast Assess post build step or pipeline step. For example, you can use the
AppVersionTag, or the date when the vulnerability was last seen. If this checkbox is checked, then the query is included when the job outcome policy is evaluated.
This example shows possible rules and settings for a job outcome policy that will change the outcome status in Jenkins if these conditions are met.
With this example, the following vulnerabilities will be considered for policy violation:
All vulnerabilities found on a server designated as a QA environment.
All vulnerabilities that have a status of Reported, Confirmed or Suspicious.
Any vulnerability that was first discovered between application onboarding and when the Contrast step runs in Jenkins.
The policy will be violated and the outcome status will change, if at least ONE of these occur:
There is at least 1 Critical vulnerability.
There are at least a combined total of 4 High, Medium, Low or Note severity vulnerabilities of any rule type except SQL injection.
Note
Vulnerabilities are only counted once, with precedence given to the most specific setting (for example, a particular rule type) to the least specific (All rules). If vulnerability limits are set for both a rule type and its severity level, the vulnerabilities will be included in the rule type count, but not in the severity’s vulnerability count. So in this example, Critical vulnerabilities are counted under severity, but High, Medium, Low and Note severities are combined under All rules.
Under Policy outcome, select the outcome of the policy. Contrast marks jobs that match the selected criteria as Failure, Unstable, or Success. For applications with multiple job outcome policies, the most severe outcome from all violating policies will apply.
At any point, you can use uncheck the box next to Enable this job outcome policy to pause Contrast from enforcing policies on Jenkins jobs without having to delete the individual policy.