SARIF 2.2 proposal: precision field for reportingDescriptors and results

[adityasharad]

Originally filed as #598, split into more focused component issues.

GitHub Advanced Security's code scanning feature recognises precision levels, currently read from the properties bag of a SARIF reportingDescriptor object, and sorts alerts by them. GitHub CodeQL populates this property in its SARIF output, and the property is recognised for other code scanning tools.

Docs: https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning#reportingdescriptor-object

Using the property bag was a pragmatic measure to provide this functionality without requiring a SARIF spec change. For SARIF 2.2 I propose we make precision an accepted property on a reportingDescriptor or a result, with the same accepted values that GitHub consumes today: very-high, high, medium, low, or omitted entirely. SARIF producer tools can use this property to indicate the tool's own confidence in the precision of the result, or for a rule, how often the results indicated by this rule are true.

cc @michaelcfanning

[michaelcfanning]

The interesting question to me here is how this property s/be differentiated from the existing SARIF rank property.

One possible answer is that precision is a metric that's very focused on a specific ranking aspect: the relative accuracy/precision of the analysis itself. If we set a precedent for this kind of value, should we consider other focused metrics that might be useful inputs into a triage process? e.g., 'likelihood to occur', 'relative impact', 'visibility to users'.

Relative impact is a kind of numeric expression for the new severity value you've proposed. Are we heading towards a situation where we have property pairs? i.e., a numeric expressions from 0.0 to 100.0 coupled with a four-level, readable priority (note, warning, errror, critical).

[katrinaoneil]

Also, what if we use two separate values: one for the engine confidence and another for the rules accuracy -- how do they map to the proposed precision value?

[ShiningMassXAcc]

Documenting here: +1 to a precision field of some sort.

• I can see motivations for using the rank-like scheme to express (eg 0.0 - 100.0). I can see motivations for using the level scheme (fixed string-named prose for easier client interpretation).
• Consider if we will want the granularity in the future, given precision + recall is often a very mathematical item - even if today we think about rules in codeql more coarsely.
• Agree this can be on the rule / result / etc