1
0
mirror of synced 2025-12-20 10:28:40 -05:00
Files
docs/content/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/migrating-from-the-codeql-runner-to-codeql-cli.md
Robert Sese 79c48070c4 Deprecate 3.0 (#25646)
* Deprecate 3.0

* 3.0 deprecation: remove 3.0 markup (#25647)

* Remove liquid conditionals and content for 3.0 deprecation

* Remove manually, no longer versioned in a supported version

* Remove translations manually, no longer versioned in a supported version

* Remove 'if', now in all supported versions

* Remove dangling 'elseif', now in all supported versions

* Remove dangling 'elseif' and 3.0 screenshot reference, now in all supported versions

* Nudge to latest supported GHES version

* Nudge to latest supported release GHES version

* Bump all the version for the liquid tests

* Bump first deprecated version for linting tests

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Prefer double quotes

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Remove extra newline

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Remove extra newline

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Remove extra newline

Co-authored-by: Laura Coursen <lecoursen@github.com>

* One reusable per line

Co-authored-by: Laura Coursen <lecoursen@github.com>

* One reusable per line

Co-authored-by: Laura Coursen <lecoursen@github.com>

* One reusable per line

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Stray whitespace ✂️

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Version check not needed anymore

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Just 'ghes' since we're deprecating 3.0

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Don't depend on hardcoded versions

Co-authored-by: Laura Coursen <lecoursen@github.com>

* Remove static files for 3.0 deprecation (#25649)

Co-authored-by: Laura Coursen <lecoursen@github.com>
2022-03-03 13:08:24 -06:00

416 lines
19 KiB
Markdown

---
title: Migrating from the CodeQL runner to CodeQL CLI
shortTitle: Migrating from the CodeQL runner
intro: 'You can use the {% data variables.product.prodname_codeql_cli %} to complete the same tasks as with the {% data variables.product.prodname_codeql_runner %}.'
product: '{% data reusables.gated-features.code-scanning %}'
versions:
fpt: '*'
ghes: '*'
ghae: '*'
ghec: '*'
topics:
- Advanced Security
- Code scanning
- CodeQL
---
# Migrating from the {% data variables.product.prodname_codeql_runner %} to the {% data variables.product.prodname_codeql_cli %}
The {% data variables.product.prodname_codeql_runner %} is being deprecated. You can use the {% data variables.product.prodname_codeql_cli %} version 2.6.2 and greater instead.
This document describes how to migrate common workflows from the {% data variables.product.prodname_codeql_runner %} to the {% data variables.product.prodname_codeql_cli %}.
## Installation
Download the **{% data variables.product.prodname_codeql %} bundle** from the [`github/codeql-action` repository](https://github.com/github/codeql-action/releases). This bundle contains the {% data variables.product.prodname_codeql_cli %} and the standard {% data variables.product.prodname_codeql %} queries and libraries.
For more information on setting up the {% data variables.product.prodname_codeql_cli %}, see "[Installing {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system)."
## Overview of workflow changes
A typical workflow that uses the {% data variables.product.prodname_codeql_runner %} to analyze a codebase has the following steps.
- `codeql-runner-<platform> init` to start creating {% data variables.product.prodname_codeql %} databases and read the configuration.
- For compiled languages: set environment variables produced by the `init` step.
- For compiled languages: run autobuild or manual build steps.
- `codeql-runner-<platform> analyze` to finish creating {% data variables.product.prodname_codeql %} databases, run queries to analyze each {% data variables.product.prodname_codeql %} database, summarize the results in a SARIF file, and upload the results to {% data variables.product.prodname_dotcom %}.
A typical workflow that uses the {% data variables.product.prodname_codeql_cli %} to analyze a codebase has the following steps.
- `codeql database create` to create {% data variables.product.prodname_codeql %} databases.
- For compiled languages: Optionally provide a build command.
- `codeql database analyze` to run queries to analyze each {% data variables.product.prodname_codeql %} database and summarize the results in a SARIF file. This command must be run once for each language or database.
- `codeql github upload-results` to upload the resulting SARIF files to {% data variables.product.prodname_dotcom %}, to be displayed as code scanning alerts. This command must be run once for each language or SARIF file.
The {% data variables.product.prodname_codeql_runner %} is multithreaded by default. The {% data variables.product.prodname_codeql_cli %} only uses a single thread by default, but allows you to specify the amount of threads you want it to use. If you want to replicate the behavior of the {% data variables.product.prodname_codeql_runner %} to use all threads available on the machine when using the {% data variables.product.prodname_codeql_cli %}, you can pass `--threads 0` to `codeql database analyze`.
For more information, see "[Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system)."
## Examples of common uses for the {% data variables.product.prodname_codeql_cli %}
### About the examples
These examples assume that the source code has been checked out to the current working directory. If you use a different directory, change the `--source-root` argument and the build steps accordingly.
These examples also assume that the {% data variables.product.prodname_codeql_cli %} is placed on the current PATH.
In these examples, a {% data variables.product.prodname_dotcom %} token with suitable scopes is stored in the `$TOKEN` environment variable and passed to the example commands via `stdin`, or is stored in the `$GITHUB_TOKEN` environment variable.
The ref name and commit SHA being checked out and analyzed in these examples are known during the workflow. For a branch, use `refs/heads/BRANCH-NAME` as the ref. For the head commit of a pull request, use `refs/pull/NUMBER/head`. For a {% data variables.product.prodname_dotcom %}-generated merge commit of a pull request, use `refs/pull/NUMBER/merge`. The examples below all use `refs/heads/main`. If you use a different branch name, you must modify the sample code.
### Single non-compiled language (JavaScript)
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages javascript \
--github-url https://github.com --github-auth-stdin
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo
--github-url https://github.com --github-auth-stdin
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
codeql database create /codeql-dbs/example-repo --language=javascript \
--source-root=.
# The default query suite is called `<language>-code-scanning.qls`.
codeql database analyze /codeql-dbs/example-repo \
javascript-code-scanning.qls --sarif-category=javascript \
--format=sarif-latest --output=/temp/example-repo-js.sarif
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-js.sarif --github-auth-stdin
```
### Single non-compiled language (JavaScript) using a different query suite (security-and-quality)
A similar approach can be taken for compiled languages, or multiple languages.
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages javascript \
--github-url https://github.com --github-auth-stdin
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo \
--queries security-and-quality \
--github-url https://github.com --github-auth-stdin \
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
codeql database create /codeql-dbs/example-repo --language=javascript \
--source-root=.
# Use `<language>-<suite name>.qls`
codeql database analyze /codeql-dbs/example-repo \
javascript-security-and-quality.qls --sarif-category=javascript
--format=sarif-latest --output=/temp/example-repo-js.sarif
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-js.sarif --github-auth-stdin
```
### Single non-compiled language (JavaScript) using a custom configuration file
A similar approach can be taken for compiled languages, or multiple languages.
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages javascript \
--config-file .github/codeql/codeql-config.yml \
--github-url https://github.com --github-auth-stdin
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo \
--github-url https://github.com --github-auth-stdin \
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
# Use `--codescanning-config` with the path to the YAML configuration file.
codeql database create /codeql-dbs/example-repo --language=javascript \
--codescanning-config=.github/codeql/codeql-config.yml \
--source-root=.
codeql database analyze /codeql-dbs/example-repo \
--sarif-category=javascript
--format=sarif-latest --output=/temp/example-repo-js.sarif
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-js.sarif --github-auth-stdin
```
### Single compiled language using autobuild (Java)
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages java \
--github-url https://github.com --github-auth-stdin
# Source the script generated by the init step to set up the environment to monitor the build.
. codeql-runner/codeql-env.sh
# Run the autobuilder for the given language.
codeql-runner-linux autobuild --language java
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo
--github-url https://github.com --github-auth-stdin
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
# Run `codeql database create` without `--command`.
# This will run the autobuilder for the given language.
codeql database create /codeql-dbs/example-repo --language=java \
--source-root=.
codeql database analyze /codeql-dbs/example-repo \
javascript-code-scanning.qls --sarif-category=java
--format=sarif-latest --output=/temp/example-repo-java.sarif
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-java.sarif --github-auth-stdin
```
### Single compiled language using a custom build command (Java)
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages java \
--github-url https://github.com --github-auth-stdin
# Source the script generated by the init step to set up the environment to monitor the build.
. codeql-runner/codeql-env.sh
# Run a custom build command.
mvn compile -DskipTests
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo
--github-url https://github.com --github-auth-stdin
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
# Provide an explicit build command using `--command`.
codeql database create /codeql-dbs/example-repo --language=java \
--command="mvn compile -DskipTests" --source-root=.
codeql database analyze /codeql-dbs/example-repo \
java-code-scanning.qls --sarif-category=java
--format=sarif-latest --output=/temp/example-repo-java.sarif
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-java.sarif --github-auth-stdin
```
### Single compiled language using indirect build tracing (C# on Windows within Azure DevOps)
Indirect build tracing for a compiled language enables {% data variables.product.prodname_codeql %} to detect all build steps between the `init` and `analyze` steps, when the code cannot be built using the autobuilder or an explicit build command line. This is useful when using preconfigured build steps from your CI system, such as the `VSBuild` and `MSBuild` tasks in Azure DevOps.
Runner:
```yaml
- task: CmdLine@1
displayName: CodeQL Initialization
inputs:
script: "%CodeQLRunner%\\codeql-runner-win.exe init --repository my-org/example-repo --languages csharp --github-url https://github.com --github-auth $(Token)"
# Set the generated environment variables so they are available for subsequent commands, in the format required by Azure Pipelines.
- task: PowerShell@1
displayName: Set CodeQL Environment Variables
inputs:
targetType: inline
script: >
$json = Get-Content $(System.DefaultWorkingDirectory)/codeql-runner/codeql-env.json | ConvertFrom-Json
$json.PSObject.Properties | ForEach-Object {
$template = "##vso[task.setvariable variable="
$template += $_.Name
$template += "]"
$template += $_.Value
echo "$template"
}
# Execute a clean build using the VSBuild task.
- task: VSBuild@1
inputs:
solution: '**/*.sln'
msbuildArgs: '/p:OutDir=$(Build.ArtifactStagingDirectory) /p:UseSharedCompilation=false'
platform: Any CPU
configuration: Release
clean: True
displayName: Visual Studio Build
# Analyze the database created as part of the build, by running the selected queries against it, and upload results to GitHub.
- task: CmdLine@2
displayName: CodeQL Analyze
inputs:
script: '%CodeQLRunner%\codeql-runner-win.exe analyze --repository my-org/example-repo --commit $(Build.SourceVersion) --ref $(Build.SourceBranch) --github-url https://github.com --github-auth $(Token)'
```
CLI:
```yaml
# Run any pre-build tasks, for example, restore NuGet dependencies...
# Initialize the CodeQL database using `codeql database init --begin tracing`.
- task: CmdLine@1
displayName: Initialize CodeQL database
inputs:
# Assumes the source code is checked out to the current working directory.
# Creates a database at `/codeql-dbs/example-repo`.
# Running on Windows, so specifies a trace process level.
script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing /codeql-dbs/example-repo"
# For CodeQL to trace future build steps without knowing the explicit build commands,
# it requires certain environment variables to be set during the build.
# Read these generated environment variables and values, and set them so they are available for subsequent commands
# in the build pipeline. This is done in PowerShell in this example.
- task: PowerShell@1
displayName: Set CodeQL environment variables
inputs:
targetType: inline
script: >
$json = Get-Content /codeql-dbs/example-repo/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
$json.PSObject.Properties | ForEach-Object {
$template = "##vso[task.setvariable variable="
$template += $_.Name
$template += "]"
$template += $_.Value
echo "$template"
}
# Execute the pre-defined build step. Note the `msbuildArgs` variable.
- task: VSBuild@1
inputs:
solution: '**/*.sln'
# Disable MSBuild shared compilation for C# builds.
msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory) /p:UseSharedCompilation=false
platform: Any CPU
configuration: Release
# Execute a clean build, in order to remove any existing build artifacts prior to the build.
clean: True
displayName: Visual Studio Build
# Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
- task: PowerShell@1
displayName: Clear CodeQL environment variables
inputs:
targetType: inline
script: >
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
$json.PSObject.Properties | ForEach-Object {
$template = "##vso[task.setvariable variable="
$template += $_.Name
$template += "]"
$template += $_.Value
echo "$template"
}
# Use `codeql database finalize` to complete database creation after the build is done.
- task: CmdLine@2
displayName: Finalize CodeQL database
inputs:
script: 'codeql database finalize /codeql-dbs/example-repo'
# Analyze the database and upload the results.
- task: CmdLine@2
displayName: Analyze CodeQL database
inputs:
script: 'codeql database analyze /codeql-dbs/example-repo csharp-code-scanning.qls --sarif-category=csharp --format=sarif-latest --output=/temp/example-repo-csharp.sarif'
- task: CmdLine@2
displayName: Upload CodeQL results
inputs:
script: 'echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif=/temp/example-repo-csharp.sarif --github-auth-stdin'
```
### Multiple languages using autobuild (C++, Python)
This example is not strictly possible with the {% data variables.product.prodname_codeql_runner %}.
Only one language (the compiled language with the most files) will be analyzed.
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages cpp,python \
--github-url https://github.com --github-auth-stdin
# Source the script generated by the init step to set up the environment to monitor the build.
. codeql-runner/codeql-env.sh
# Run the autobuilder for the language with the most files.
codeql-runner-linux autobuild
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo
--github-url https://github.com --github-auth-stdin
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
# Create multiple databases using `--db-cluster`.
# Run autobuild by omitting `--command`.
codeql database create /codeql-dbs/example-repo-multi \
--db-cluster --language cpp,python \
--no-run-unnecessary-builds \
--source-root .
# Analyze each database in turn and upload the results.
for language in cpp python; do
codeql database analyze "/codeql-dbs/example-repo-multi/$language" \
"$language-code-scanning.qls" --sarif-category="$language"
--format=sarif-latest --output="/temp/example-repo-$language.sarif"
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif="/temp/example-repo-$language.sarif" --github-auth-stdin
done
```
### Multiple languages using a custom build command (C++, Python)
Runner:
```bash
echo "$TOKEN" | codeql-runner-linux init --repository my-org/example-repo \
--languages cpp,python \
--github-url https://github.com --github-auth-stdin
# Source the script generated by the init step to set up the environment to monitor the build.
. codeql-runner/codeql-env.sh
# Run a custom build command.
make
echo "$TOKEN" | codeql-runner-linux analyze --repository my-org/example-repo
--github-url https://github.com --github-auth-stdin
--commit deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 --ref refs/heads/main
```
CLI:
```bash
# Create multiple databases using `--db-cluster`.
codeql database create /codeql-dbs/example-repo-multi \
--db-cluster --language cpp,python \
--command make --no-run-unnecessary-builds \
--source-root .
# Analyze each database in turn and upload the results.
for language in cpp python; do
codeql database analyze "/codeql-dbs/example-repo-multi/$language" \
"$language-code-scanning.qls" --sarif-category="$language"
--format=sarif-latest --output="/temp/example-repo-$language.sarif"
echo "$TOKEN" | codeql github upload-results --repository=my-org/example-repo \
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
--sarif="/temp/example-repo-$language.sarif" --github-auth-stdin
done
```