Prepare for rolling out skipping computing file coverage information on PRs#3562
Prepare for rolling out skipping computing file coverage information on PRs#3562henrymercer wants to merge 8 commits intomainfrom
Conversation
And now that we have this, drop the restriction to `github` org.
There was a problem hiding this comment.
Pull request overview
Prepares a controlled rollout to skip computing file coverage information on PR analyses (to improve performance) by gating the behavior on a CodeQL CLI tools feature, adding a repository property opt-out, and emitting telemetry to observe usage.
Changes:
- Add a new CodeQL CLI tools feature (
suppressesMissingFileBaselineWarning) and gateSkipFileCoverageOnPrsbehind it. - Allow org-owned repositories to opt back into file coverage on PRs via the
github-codeql-file-coverage-on-prsrepository property. - Add telemetry for when the repository property is used, plus update unit tests and changelog.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tools-features.ts | Adds a new CLI tools feature enum value used to gate the rollout. |
| src/feature-flags.ts | Gates SkipFileCoverageOnPrs behind the new CLI tools feature. |
| src/feature-flags/properties.ts | Adds and parses the new repository property github-codeql-file-coverage-on-prs. |
| src/init.ts | Updates file-coverage enablement logic to support repository-property opt-in and uses CLI-gated feature evaluation. |
| src/init-action.ts | Passes repository properties into config calculation and emits telemetry when repo property enables PR file coverage. |
| src/init.test.ts | Updates unit tests for the revised file-coverage enablement logic and new opt-in behavior. |
| CHANGELOG.md | Documents the experimental rollout and the repository-property opt-out mechanism. |
| lib/upload-sarif-action.js | Generated JS output updated to reflect TS changes. |
| lib/upload-sarif-action-post.js | Generated JS output updated to reflect TS changes. |
| lib/upload-lib.js | Generated JS output updated to reflect TS changes. |
| lib/start-proxy-action.js | Generated JS output updated to reflect TS changes. |
| lib/start-proxy-action-post.js | Generated JS output updated to reflect TS changes. |
| lib/setup-codeql-action.js | Generated JS output updated to reflect TS changes. |
| lib/resolve-environment-action.js | Generated JS output updated to reflect TS changes. |
| lib/init-action.js | Generated JS output updated to reflect TS changes. |
| lib/init-action-post.js | Generated JS output updated to reflect TS changes. |
| lib/autobuild-action.js | Generated JS output updated to reflect TS changes. |
| lib/analyze-action.js | Generated JS output updated to reflect TS changes. |
| lib/analyze-action-post.js | Generated JS output updated to reflect TS changes. |
mbg
left a comment
There was a problem hiding this comment.
This LGTM as-is, but I have a couple of suggestions for tidying up the implementation. I also think we should add a page to docs.github.com for the available repository properties that are understood by the CodeQL Action.
| - Fixed [a bug](https://github.com/github/codeql-action/issues/3555) which caused the CodeQL Action to fail loading repository properties if a "Multi select" repository property was configured for the repository. [#3557](https://github.com/github/codeql-action/pull/3557) | ||
| - Added an experimental change which skips collecting file coverage information on pull requests to improve analysis performance. File coverage information will still be computed on non-PR analyses. | ||
|
|
||
| Repositories owned by an organization can opt out of this change by creating a custom repository property with the name `github-codeql-file-coverage-on-prs` and the type "True/false", then setting this property to `true` in the repository's settings. For more information, see [Managing custom properties for repositories in your organization](https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization). |
There was a problem hiding this comment.
We should add documentation on docs.github.com about these repo properties. I have created an internal issue to track this.
src/init-action.ts
Outdated
| repositoryNwo, | ||
| codeql, | ||
| features, | ||
| repositoryPropertiesResult.orElse({}), |
There was a problem hiding this comment.
Minor: we have repositoryPropertiesResult.orElse({}) at least a couple of times here now. Consider adding const repositoryProperties = repositoryPropertiesResult.orElse({}); somewhere before the uses, or indeed modifying the code so that we never end up with the "result" here, but only the properties we loaded or {}.
src/init-action.ts
Outdated
| config.enableFileCoverageInformation && | ||
| isAnalyzingPullRequest() && | ||
| (await features.getValue(Feature.SkipFileCoverageOnPrs, codeql)) && | ||
| repositoryPropertiesResult.orElse({})[ | ||
| RepositoryPropertyName.FILE_COVERAGE_ON_PRS | ||
| ] === true |
There was a problem hiding this comment.
This now duplicates most of getFileCoverageInformationEnabled. While we can't use getFileCoverageInformationEnabled as a shortcut here, consider whether you can move the code that emits this diagnostic into getFileCoverageInformationEnabled and end up with a cleaner solution in there?
mbg
left a comment
There was a problem hiding this comment.
Thanks for addressing the comments!
Prepare for rolling out skipping computing file coverage information on PRs to speed up analysis.
githuborg.Note that the CLI tools feature flag is not yet released, even as a nightly. Though given it that CLI feature only modifies logs, I don't think we need to wait for a release before merging this PR.
Risk assessment
For internal use only. Please select the risk level of this change:
Which use cases does this change impact?
Workflow types:
dynamicworkflows (Default Setup, Code Quality, ...).Products:
analysis-kinds: code-scanning.Environments:
github.comand/or GitHub Enterprise Cloud with Data Residency.How did/will you validate this change?
.test.tsfiles).If something goes wrong after this change is released, what are the mitigation and rollback strategies?
How will you know if something goes wrong after this change is released?
Are there any special considerations for merging or releasing this change?
Merge / deployment checklist