1
0
mirror of synced 2025-12-21 10:57:10 -05:00
Files
docs/content/github/finding-security-vulnerabilities-and-errors-in-your-code/running-codeql-code-scanning-in-your-ci-system.md
Matt Pollard 2db9da5c8f [DO NOT MERGE] GitHub AE consumables beta megabranch (#17620)
* Empty commit

* updated beta note for GHAE

* more GHAE update + resolve conflict

* more GHAE updates + prepare for screenshots

* Apply suggestions from code review

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>

* address remaining review comments

* Revise "About GitHub AE" (#17679)

* add screenshots to the Configuring article

* reworked to have a separate GHAE section

* list numbering

* more work on screenshots and conditions

* add GHAE screenshots in article

* review screenshots in article

* added more screenshots and updated more articles

* screenshot madness

* fix liquid versioning

* refactor the ghae script

* [GHAE CB/Feb 22]: Add article about data residency for GitHub AE (#17847)

* add missing GHAE versioning to article

* move screenshots to GHAE asset directory

* forgot to change the path for these two images

* replace CBB screenshot + add better screenshot

* [GHAE CB/Feb 22]: Document upgrades for GitHub AE (#17848)

* Version article for GitHub AE

* Replace unused variable

* Incorporate reviewer feedback

* Update intro

Co-authored-by: Ethan P <56270045+ethanpalm@users.noreply.github.com>

* [GHAE] Enable IP allow list (#17691)

* Notes for CC

* Updat permission leves chart

* Add updated article to further reading

* Update gated feature callout with GitHub AE

* Version "Managing allowed IP addresses for your organization" for AE

* Update images

* Update "Restricting network traffic to your enterprise" with new procedures

* remove todo note

* Update audited actions

* Update info about Premium Runners

* Use reusable for Premium Runners

* Change "Premium Runners" to "AE hosted runners"

* Incorporate reviewer feedback

* Use correct reusable

* Version reusable correctly

* [Feb 22] GHAE: Code scanning beta (#17830)

* Add "github-ae" to all the frontmatter

* GHAE-ify the reusables

* Add some more changes

* Re-use some content

* 🔪 Semmle links

* Revert change re "--external-repository-token" in the CodeQL runner

* Update CodeQL runner token scopes

* Update two screenshots

* Remove mention of GitHub.com from AE + other fixes

* Apply suggestions from code review

Co-authored-by: mc <42146119+mchammer01@users.noreply.github.com>

* Use `product_name` variable instead of `product_location`

* Remove confusing phrase

* [Feb 22] GHAE: Code scanning API and webhook docs (#17883)

* Version API and webhook docs

* Actually add versioning for GHAE

* Fix anchor

* [TEMPORARY] Preview for API endpoints

* Revert API previews

* Update procedure step

Co-authored-by: mc <42146119+mchammer01@users.noreply.github.com>

* Update docs for AzureAD Group SCIM support in GHAE (#17892)

* [GHAE CB] SMTP bootstrapping flow (#17888)

* draft

* update with AE conntent

* update with tons of versioning

* remove that  lie

* fill out the rest of these steps

* update with correct versioning

* more edits

* add images

* reversion most of ae article

* fix versioning

* format correctlly

* words matter

* last image

* update with permmissions

* update versioning

* add link

* apply feedback ❤️

* update with differrent spacing

* update with feedback

* more feedback

* Temporary GHAE release notes for consumables beta launch (#17859)

* Create release-notes.md

* Add frontmatter

* Add to index file

* Update github-ae-release-notes.md

* Add release notes from Google Doc

* Update finalized docs links that have been reviewed

* OAuth device flow link update

* version for AE

* few fixes

* Update content/admin/overview/github-ae-release-notes.md

* small edits

* whoops

* commit

* update with different links

* used wrong reusable

* fix more brokenness

* Update repository-references.js

* Update repository-references.js

Co-authored-by: Meg Bird <megbird@github.com>
Co-authored-by: Kevin Heis <heiskr@users.noreply.github.com>

* [GHAE] Audit public repos (#17917)

* verifying what we mean by public

* Apply suggestions from code review

* Update content/developers/apps/installing-github-apps.md

Co-authored-by: Laura Coursen <lecoursen@github.com>

* fixing placememnt of liquid conditional

Co-authored-by: Laura Coursen <lecoursen@github.com>

* GHAE packages beta (#17786)

Co-authored-by: jmarlena <6732600+jmarlena@users.noreply.github.com>
Co-authored-by: Martin Lopes <martin389@github.com>

* fix broken links

* [GHAE CB/March 01]: GitHub Actions on GHAE (beta) (#17725)

* Added initial layout for premium runners

* Restructured content

* Added placeholder for removing premium runner

* Added versioning and warning note for self-hosted runners

* Added versioning and beta notice for actions content

* Rephrased beta note

* Added versioning for API docs, fixes

* Added versioning fixes

* Split Github-hosted and premium topics into separate articles

* Added edits

* Restructured some topics

* Revised "Using premium runners in a workflow"

* Some small fixes

* Fixed typo

* Added fixes to reusable

* Added edits

* Made section titles consistent

* Added billing, group mgmt, reusable steps

* Cropped certain screenshots for future-proofing

* Removed superfluous reusable

* Added fixes

* Revert "Cropped certain screenshots for future-proofing"

This reverts commit c7f24f31fa30d4fe3de2b63fc3cd5feba44ef518.

* Added new section for custom images

* Added versioning for enterprise-admin operations

* Added edits

* Added edits

* Update adding-premium-runners.md

* Removed SHR screenshots. Intending to update them when UI is available.

* Update using-labels-with-premium-runners.md

* Added custom labels section

* Added preview of API docs changes

* Added versioning for ip allow list section

* Removed removal article

* Renamed premium runners to AE hosted runners

* Re-added added API preview

* Fixed links, updated software specs

* Revised "Software specifications" based on feedback

* Fixed typos

* Small fixes

* Added new article "Creating custom images"

* Moved "Creating custom images" link

* Apply suggestions from code review

Co-authored-by: ahdbilal <55514721+ahdbilal@users.noreply.github.com>

* Added update from review

* Added updates from tech review

* Apply suggestions from code review

Co-authored-by: ahdbilal <55514721+ahdbilal@users.noreply.github.com>

* Added updates from tech review

* Added updates from tech review

* Added updates from tech review

* Added updates from tech review

* Fixed reusable

* Added fixes

* Added update from tech review

* Removed the dereferenced OpenAPI schema files

* Added fixes

* Fixed links

* Fixed links

* Apply suggestions from code review

Co-authored-by: jmarlena <6732600+jmarlena@users.noreply.github.com>

* Added updates from peer review

* Removed sections that are not in beta

* Update viewing-your-github-actions-usage.md

* Update viewing-job-execution-time.md

* Update index.md

* Update about-github-hosted-runners.md

* Restored versioning to match GHES approach

* Fixed link

* Restored self-hosted runner reference to UI steps.

* Updated screenshots

* Updated screenshots and procedures

* Small edits to screenshots

* Added AE url info for SHR

* Removed superfluous versioning

* Update security-hardening-for-github-actions.md

* Update actions-shared.md

* Small edits

* Update usage-limits-billing-and-administration.md

* Update managing-complex-workflows.md

* Additional versioning

* Additional versioning

* version environments api and checkrun deployments for ghae (#17991)

Co-authored-by: Martin Lopes <martin389@github.com>

* Update reviewing-the-audit-log-for-your-organization.md

* Added versioning for enterprise policy settings

* version configuring artifact retention for AE

* remove AE versioning for connecting to Marketplace

* Apply suggestions from code review

Co-authored-by: Joe Bourne <thejoebourneidentity@github.com>

* Update content/admin/github-actions/getting-started-with-github-actions-for-github-ae.md

Co-authored-by: Joe Bourne <thejoebourneidentity@github.com>

* rewording not public to private

* fixing liquid

* Fixed elseif entries

* Added expectations note

* Revised label management article for AE hosted runners

* Added enterprise-admin note for adding AE hosted runners

* Update enterprise-admin.md

* Update self-hosted-runner-security.md

* Versioned reusable for AE

* Empty commit for CI

Co-authored-by: ahdbilal <55514721+ahdbilal@users.noreply.github.com>
Co-authored-by: jmarlena <6732600+jmarlena@users.noreply.github.com>
Co-authored-by: skedwards88 <skedwards88@github.com>
Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>
Co-authored-by: Joe Bourne <thejoebourneidentity@github.com>
Co-authored-by: runleonarun <runleonarun@github.com>

* Update OpenAPI Descriptions for GHAE

* Update content/admin/overview/github-ae-release-notes.md

Co-authored-by: Ethan Palm <56270045+ethanpalm@users.noreply.github.com>
Co-authored-by: mchammer01 <42146119+mchammer01@users.noreply.github.com>
Co-authored-by: Shati Patel <42641846+shati-patel@users.noreply.github.com>
Co-authored-by: shati-patel <shati-patel@github.com>
Co-authored-by: Sarah Schneider <sarahs@github.com>
Co-authored-by: skedwards88 <skedwards88@github.com>
Co-authored-by: Sarah Schneider <sarahs@users.noreply.github.com>
Co-authored-by: Melanie Yarbrough <11952755+myarb@users.noreply.github.com>
Co-authored-by: Felicity Chapman <felicitymay@github.com>
Co-authored-by: Meg Bird <megbird@github.com>
Co-authored-by: Kevin Heis <heiskr@users.noreply.github.com>
Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>
Co-authored-by: Laura Coursen <lecoursen@github.com>
Co-authored-by: jmarlena <6732600+jmarlena@users.noreply.github.com>
Co-authored-by: Martin Lopes <martin389@github.com>
Co-authored-by: ahdbilal <55514721+ahdbilal@users.noreply.github.com>
Co-authored-by: Joe Bourne <thejoebourneidentity@github.com>
Co-authored-by: runleonarun <runleonarun@github.com>
Co-authored-by: github-openapi-bot <69533958+github-openapi-bot@users.noreply.github.com>
2021-03-01 16:07:02 -05:00

176 lines
15 KiB
Markdown

---
title: Running CodeQL code scanning in your CI system
shortTitle: Running in your CI
intro: 'You can use the {% data variables.product.prodname_codeql_runner %} to perform {% data variables.product.prodname_codeql %} {% data variables.product.prodname_code_scanning %} in a third-party continuous integration system.'
product: '{% data reusables.gated-features.code-scanning %}'
redirect_from:
- /github/finding-security-vulnerabilities-and-errors-in-your-code/running-code-scanning-in-your-ci-system
versions:
free-pro-team: '*'
enterprise-server: '>=2.22'
github-ae: '*'
---
<!--UI-LINK: When GitHub Enterprise Server doesn't have GitHub Actions set up, the Security > Code scanning alerts view links to this article.-->
{% data reusables.code-scanning.beta-codeql-runner %}
{% data reusables.code-scanning.beta %}
{% data reusables.code-scanning.enterprise-enable-code-scanning %}
### Using {% data variables.product.prodname_codeql %} {% data variables.product.prodname_code_scanning %} with your existing CI system
If you use a continuous integration or continuous delivery/deployment (CI/CD) system other than {% data variables.product.prodname_actions %}, you can use your existing system to run {% data variables.product.prodname_dotcom %}'s {% data variables.product.prodname_codeql %} analysis and upload the results to {% data variables.product.prodname_dotcom %}. To do this, use the {% data variables.product.prodname_codeql_runner %}.
### About the {% data variables.product.prodname_codeql_runner %}
{% data reusables.code-scanning.about-code-scanning %} For information, see "[About {% data variables.product.prodname_code_scanning %}](/github/finding-security-vulnerabilities-and-errors-in-your-code/about-code-scanning)."
You can use the {% data variables.product.prodname_codeql_runner %} to run {% data variables.product.prodname_code_scanning %} on code that you're processing in a third-party continuous integration (CI) system. Alternatively, you can use {% data variables.product.prodname_actions %} to run {% data variables.product.prodname_code_scanning %} on {% data variables.product.product_name %}. For information, see "[Setting up {% data variables.product.prodname_code_scanning %} for a repository](/github/finding-security-vulnerabilities-and-errors-in-your-code/setting-up-code-scanning-for-a-repository)."
The {% data variables.product.prodname_codeql_runner %} is a command-line tool that runs {% data variables.product.prodname_codeql %} analysis on a checkout of a {% data variables.product.prodname_dotcom %} repository. You add the runner to your third-party system, then call the runner to analyze code and upload the results to {% data variables.product.product_name %}. These results are displayed as {% data variables.product.prodname_code_scanning %} alerts in the repository.
{% note %}
**Notes:**
{% if currentVersion == "free-pro-team@latest" %}
* The {% data variables.product.prodname_codeql_runner %} uses the {% data variables.product.prodname_codeql %} CLI to analyze code and therefore has the same license conditions. It's free to use on public repositories that are maintained on {% data variables.product.prodname_dotcom_the_website %}, and available to use on private repositories that are owned by customers with an {% data variables.product.prodname_advanced_security %} license. For information, see "[{% data variables.product.product_name %} {% data variables.product.prodname_codeql %} Terms and Conditions](https://securitylab.github.com/tools/codeql/license)" and "[{% data variables.product.prodname_codeql %} CLI](https://codeql.github.com/docs/codeql-cli/)."
{% else %}
* The {% data variables.product.prodname_codeql_runner %} is available to customers with an {% data variables.product.prodname_advanced_security %} license.
{% endif %}
* The {% data variables.product.prodname_codeql_runner %} shouldn't be confused with the {% data variables.product.prodname_codeql %} CLI. The {% data variables.product.prodname_codeql %} CLI is an interactive command-line interface that lets you create {% data variables.product.prodname_codeql %} databases for security research and run {% data variables.product.prodname_codeql %} queries. For more information, see "[{% data variables.product.prodname_codeql %} CLI](https://codeql.github.com/docs/codeql-cli/)."
{% endnote %}
### Downloading the {% data variables.product.prodname_codeql_runner %}
You can download the {% data variables.product.prodname_codeql_runner %} from https://{% if currentVersion == "enterprise-server@2.22" or currentVersion == "free-pro-team@latest" %}github.com{% else %}<em>HOSTNAME</em>{% endif %}/github/codeql-action/releases. On some operating systems, you may need to change permissions for the downloaded file before you can run it.
On Linux:
```shell
chmod +x codeql-runner-linux
```
On macOS:
```shell
chmod +x codeql-runner-macos
sudo xattr -d com.apple.quarantine codeql-runner-macos
```
On Windows, the `codeql-runner-win.exe` file usually requires no change to permissions.
### Adding the {% data variables.product.prodname_codeql_runner %} to your CI system
Once you download the {% data variables.product.prodname_codeql_runner %} and verify that it can be executed, you should make the runner available to each CI server that you intend to use for {% data variables.product.prodname_code_scanning %}. For example, you might configure each server to copy the runner from a central, internal location. Alternatively, you could use the REST API to get the runner directly from {% data variables.product.prodname_dotcom %}, for example:
```shell
wget https://{% if currentVersion == "enterprise-server@2.22" or currentVersion == "free-pro-team@latest" %}github.com{% else %}<em>HOSTNAME</em>{% endif %}/github/codeql-action/releases/latest/download/codeql-runner-linux
chmod +x codeql-runner-linux
```
In addition to this, each CI server also needs:
- A {% data variables.product.prodname_github_app %} or personal access token for the {% data variables.product.prodname_codeql_runner %} to use. You must use an access token with the `repo` scope, or a {% data variables.product.prodname_github_app %} with the `security_events` write permission, and `metadata` and `contents` read permissions. For information, see "[Building {% data variables.product.prodname_github_apps %}](/developers/apps/building-github-apps)" and "[Creating a personal access token](/github/authenticating-to-github/creating-a-personal-access-token)."
- Access to the {% data variables.product.prodname_codeql %} bundle associated with this release of the {% data variables.product.prodname_codeql_runner %}. This package contains queries and libraries needed for {% data variables.product.prodname_codeql %} analysis, plus the {% data variables.product.prodname_codeql %} CLI, which is used internally by the runner. For information, see "[{% data variables.product.prodname_codeql %} CLI](https://codeql.github.com/docs/codeql-cli/)."
The options for providing access to the {% data variables.product.prodname_codeql %} bundle are:
1. Allow the CI servers access to https://{% if currentVersion == "enterprise-server@2.22" or currentVersion == "free-pro-team@latest" %}github.com{% else %}<em>HOSTNAME</em>{% endif %}/github/codeql-action so that the {% data variables.product.prodname_codeql_runner %} can download the bundle automatically.
{% if currentVersion == "enterprise-server@2.22" %}
1. Mirror the `github/codeql-action` repository on {% data variables.product.product_name %}. Unless you specify the <nobr>`--codeql-path`</nobr> flag, the runner automatically checks for the bundle in this location and on {% data variables.product.prodname_dotcom_the_website %}.{% endif %}
1. Manually download/extract the bundle, store it with other central resources, and use the <nobr>`--codeql-path`</nobr> flag to specify the location of the bundle in calls to initialize the {% data variables.product.prodname_codeql_runner %}.
### Calling the {% data variables.product.prodname_codeql_runner %}
You should call the {% data variables.product.prodname_codeql_runner %} from the checkout location of the repository you want to analyze. The two main commands are:
1. `init` required to initialize the runner and create a {% data variables.product.prodname_codeql %} database for each language to be analyzed. These databases are populated and analyzed by subsequent commands.
1. `analyze` required to populate the {% data variables.product.prodname_codeql %} databases, analyze them, and upload results to {% data variables.product.product_name %}.
For both commands, you must specify the URL of {% data variables.product.product_name %}, the repository *OWNER/NAME*, and the {% data variables.product.prodname_github_apps %} or personal access token to use for authentication. You also need to specify the location of the CodeQL bundle, unless the CI server has access to download it directly from the `github/codeql-action` repository.
You can configure where the {% data variables.product.prodname_codeql_runner %} stores the CodeQL bundle for future analysis on a server using the <nobr>`--tools-dir`</nobr> flag and where it stores temporary files during analysis using <nobr>`--temp-dir`</nobr>.
To view the command-line reference for the runner, use the `-h` flag. For example, to list all commands run: `codeql-runner-OS -h`, or to list all the flags available for the `init` command run: `codeql-runner-OS init -h` (where `OS` varies according to the executable that you are using). For more information, see "[Configuring {% data variables.product.prodname_code_scanning %} in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-codeql-code-scanning-in-your-ci-system#codeql-runner-command-reference)."
{% data reusables.code-scanning.upload-sarif-alert-limit %}
#### Basic example
This example runs {% data variables.product.prodname_codeql %} analysis on a Linux CI server for the `octo-org/example-repo` repository hosted on `{% data variables.command_line.git_url_example %}`. The process is very simple because the repository contains only languages that can be analyzed by {% data variables.product.prodname_codeql %} directly, without being built (that is, Go, JavaScript, Python, and TypeScript).
1. Check out the repository to analyze.
1. Move into the directory where the repository is checked out.
1. Initialize the {% data variables.product.prodname_codeql_runner %} and create {% data variables.product.prodname_codeql %} databases for the languages detected.
```shell
$ /path/to-runner/codeql-runner-linux init --repository octo-org/example-repo
--github-url {% data variables.command_line.git_url_example %} --github-auth TOKEN
> Cleaning temp directory /srv/checkout/example-repo/codeql-runner
> ...
> Created CodeQL database at /srv/checkout/example-repo/codeql-runner/codeql_databases/javascript.
```
1. Populate the {% data variables.product.prodname_codeql_runner %} databases, analyze them, and upload the results to {% data variables.product.product_name %}.
```shell
$ /path/to-runner/codeql-runner-linux analyze --repository octo-org/example-repo
--github-url {% data variables.command_line.git_url_example %} --github-auth TOKEN
--commit 5b6a3078b31dc346e5ce7b86837d6abbe7a18bbd --ref refs/heads/main
> Finalizing database creation
> ...
> POST /repos/octo-org/example-repo/code-scanning/sarifs - 202 in 786ms
> Successfully uploaded results
```
The server has access to download the {% data variables.product.prodname_codeql %} bundle directly from the `github/codeql-action` repository, so there is no need to use the `--codeql-path` flag. When the analysis is complete, the {% data variables.product.prodname_codeql_runner %} uploads the results to the {% data variables.product.prodname_code_scanning %} view. For more information, see "[Managing {% data variables.product.prodname_code_scanning %} alerts for your repository](/github/finding-security-vulnerabilities-and-errors-in-your-code/managing-code-scanning-alerts-for-your-repository)."
#### Compiled language example
This example is similar to the previous example, however this time the repository has code in C/C++, C#, or Java. To create a {% data variables.product.prodname_codeql %} database for these languages, the CLI needs to monitor the build. At the end of the initialization process, the runner reports the command you need to set up the environment before building the code. You need to run this command, before calling the normal CI build process, and then running the `analyze` command.
1. Check out the repository to analyze.
1. Move into the directory where the repository is checked out.
1. Initialize the {% data variables.product.prodname_codeql_runner %} and create {% data variables.product.prodname_codeql %} databases for the languages detected.
```shell
$ /path/to-runner/codeql-runner-linux init --repository octo-org/example-repo-2
--github-url {% data variables.command_line.git_url_example %} --github-auth TOKEN
> Cleaning temp directory /srv/checkout/example-repo-2/codeql-runner
> ...
> CodeQL environment output to "/srv/checkout/example-repo-2/codeql-runner/codeql-env.json"
and "/srv/checkout/example-repo-2/codeql-runner/codeql-env.sh".
Please export these variables to future processes so that CodeQL can monitor the build, for example by running "
. /srv/checkout/example-repo-2/codeql-runner/codeql-env.sh".
```
1. Source the script generated by the `init` action to set up the environment to monitor the build. Note the leading dot and space in the following code snippet.
```shell
$ . /srv/checkout/example-repo-2/codeql-runner/codeql-env.sh
```
1. Build the code. On macOS, you need to prefix the build command with the environment variable `$CODEQL_RUNNER`. For more information, see "[Troubleshooting CodeQL code scanning in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/troubleshooting-codeql-code-scanning-in-your-ci-system#no-code-found-during-the-build)."
1. Populate the {% data variables.product.prodname_codeql %} databases, analyze them, and upload the results to {% data variables.product.prodname_dotcom %}.
```shell
$ /path/to-runner/codeql-runner-linux analyze --repository octo-org/example-repo-2
--github-url {% data variables.command_line.git_url_example %} --github-auth TOKEN
--commit ae7b655ef30b50fb726ae7b3daa79571a39d194d --ref refs/heads/main
> Finalizing database creation
> ...
> POST /repos/octo-org/example-repo-2/code-scanning/sarifs - 202 in 573ms
> Successfully uploaded results
```
{% note %}
**Note:** If you use a containerized build, you need to run the {% data variables.product.prodname_codeql_runner %} in the container where your build task takes place.
{% endnote %}
### Further reading
- "[Configuring {% data variables.product.prodname_code_scanning %} in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-codeql-code-scanning-in-your-ci-system)"
- "[Troubleshooting {% data variables.product.prodname_code_scanning %} in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/troubleshooting-codeql-code-scanning-in-your-ci-system)"