Reduce duplication and simplify CodeQL CLI getting started documentation (#42998)
Co-authored-by: Felicity Chapman <felicitymay@github.com>
This commit is contained in:
Sarita Iyer
@@ -14,6 +14,15 @@ topics:
|
||||
redirect_from:
|
||||
- /code-security/codeql-cli/about-the-codeql-cli
|
||||
- /code-security/codeql-cli/using-the-codeql-cli/about-the-codeql-cli
|
||||
- /code-security/secure-coding/about-codeql-code-scanning-in-your-ci-system
|
||||
- /code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/about-codeql-code-scanning-in-your-ci-system
|
||||
- /code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/running-codeql-runner-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/using-codeql-code-scanning-with-your-existing-ci-system/running-codeql-code-scanning-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/running-codeql-code-scanning-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/troubleshooting-codeql-code-scanning-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/using-codeql-code-scanning-with-your-existing-ci-system/troubleshooting-codeql-code-scanning-in-your-ci-system
|
||||
- /code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/migrating-from-the-codeql-runner-to-codeql-cli
|
||||
- /code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/about-codeql-code-scanning-in-your-ci-system
|
||||
---
|
||||
|
||||
## About the {% data variables.product.prodname_codeql_cli %}
|
||||
@@ -39,11 +48,9 @@ The {% data variables.product.prodname_codeql_cli %} can analyze:
|
||||
For information about setting up the {% data variables.product.prodname_codeql_cli %}, see
|
||||
"[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli)."
|
||||
|
||||
For information about using the {% data variables.product.prodname_codeql_cli %} in a third-party CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, see [Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system). For information about enabling {% data variables.product.prodname_codeql %} code scanning using {% data variables.product.prodname_actions %}, see {% ifversion code-scanning-without-workflow %}"[AUTOTITLE](/code-security/code-scanning/enabling-code-scanning/configuring-default-setup-for-code-scanning)" and {% endif %}"[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/configuring-advanced-setup-for-code-scanning)."
|
||||
|
||||
## About using the {% data variables.product.prodname_codeql_cli %} for {% data variables.product.prodname_code_scanning %}
|
||||
|
||||
You can use the {% data variables.product.prodname_codeql_cli %} to run {% data variables.product.prodname_code_scanning %} on code that you're processing in a third-party continuous integration (CI) system. {% data reusables.code-scanning.about-code-scanning %} For an overview of the options for CI systems, see "[AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/about-codeql-code-scanning-in-your-ci-system)." For recommended specifications (RAM, CPU cores, and disk) for running {% data variables.product.prodname_codeql %} analysis, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/recommended-hardware-resources-for-running-codeql)."
|
||||
You can use the {% data variables.product.prodname_codeql_cli %} to run {% data variables.product.prodname_code_scanning %} on code that you're processing in a third-party continuous integration (CI) system. {% data reusables.code-scanning.about-code-scanning %} For an overview of using code scanning with external CI systems, see "[AUTOTITLE](/code-security/code-scanning/integrating-with-code-scanning/using-code-scanning-with-your-existing-ci-system)." For recommended specifications (RAM, CPU cores, and disk) for running {% data variables.product.prodname_codeql %} analysis, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/recommended-hardware-resources-for-running-codeql)."
|
||||
|
||||
Alternatively, you can use {% data variables.product.prodname_actions %} or Azure DevOps pipelines to scan code using the {% data variables.product.prodname_codeql_cli %}. For more information, see {% ifversion code-scanning-without-workflow %}"[AUTOTITLE](/code-security/code-scanning/enabling-code-scanning/configuring-default-setup-for-code-scanning)"{% else %}"[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/configuring-advanced-setup-for-code-scanning)"{% endif %} or [Configure {% data variables.product.prodname_ghas_azdo %}](https://learn.microsoft.com/en-us/azure/devops/repos/security/configure-github-advanced-security-features) in Microsoft Learn.
|
||||
|
||||
@@ -51,9 +58,9 @@ For an overview of all the options for using {% data variables.product.prodname_
|
||||
|
||||
{% data reusables.code-scanning.licensing-note %}
|
||||
|
||||
## About generating code scanning results with {% data variables.product.prodname_codeql_cli %}
|
||||
## About generating code scanning results with the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
If you choose to run the {% data variables.product.prodname_codeql_cli %} directly, you first have to install the {% data variables.product.prodname_codeql_cli %} locally. If you are planning to use the {% data variables.product.prodname_codeql_cli %} with an external CI system, you need to make the {% data variables.product.prodname_codeql_cli %} available to servers in your CI system, and ensure that they can authenticate with {% data variables.product.product_name %}. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli)."
|
||||
If you choose to run the {% data variables.product.prodname_codeql_cli %} directly, you first have to install the {% data variables.product.prodname_codeql_cli %} locally. If you are planning to use the {% data variables.product.prodname_codeql_cli %} with an external CI system, you need to make the {% data variables.product.prodname_codeql_cli %} available to servers in your CI system. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli)."
|
||||
|
||||
Once the {% data variables.product.prodname_codeql_cli %} is set up, you can use three different commands to generate results and upload them to {% data variables.product.product_name %}:
|
||||
|
||||
@@ -61,6 +68,48 @@ Once the {% data variables.product.prodname_codeql_cli %} is set up, you can use
|
||||
1. `database analyze` to run queries to analyze each {% data variables.product.prodname_codeql %} database and summarize the results in a SARIF file. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries)."
|
||||
1. `github upload-results` to upload the resulting SARIF files to {% data variables.product.product_name %} where the results are matched to a branch or pull request and displayed as {% data variables.product.prodname_code_scanning %} alerts. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/uploading-codeql-analysis-results-to-github)."
|
||||
|
||||
{% data reusables.code-scanning.upload-sarif-ghas %}
|
||||
|
||||
### Example CI configuration for {% data variables.product.prodname_codeql %} analysis
|
||||
|
||||
This is an example of the full series of commands for the {% data variables.product.prodname_codeql_cli %} that you might use to analyze a codebase with two supported languages and then upload the results to {% data variables.product.product_name %}.
|
||||
|
||||
```shell
|
||||
# Create CodeQL databases for Java and Python in the 'codeql-dbs' directory
|
||||
# Call the normal build script for the codebase: 'myBuildScript'
|
||||
|
||||
codeql database create codeql-dbs --source-root=src \
|
||||
--db-cluster --language=java,python --command=./myBuildScript
|
||||
|
||||
# Analyze the CodeQL database for Java, 'codeql-dbs/java'
|
||||
# Tag the data as 'java' results and store in: 'java-results.sarif'
|
||||
|
||||
codeql database analyze codeql-dbs/java java-code-scanning.qls \
|
||||
--format=sarif-latest --sarif-category=java --output=java-results.sarif
|
||||
|
||||
# Analyze the CodeQL database for Python, 'codeql-dbs/python'
|
||||
# Tag the data as 'python' results and store in: 'python-results.sarif'
|
||||
|
||||
codeql database analyze codeql-dbs/python python-code-scanning.qls \
|
||||
--format=sarif-latest --sarif-category=python --output=python-results.sarif
|
||||
|
||||
# Upload the SARIF file with the Java results: 'java-results.sarif'
|
||||
# The {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} created for authentication
|
||||
# with {% data variables.product.company_short %}'s REST API is available in the `GITHUB_TOKEN` environment variable.
|
||||
|
||||
codeql github upload-results \
|
||||
--repository=my-org/example-repo \
|
||||
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
|
||||
--sarif=java-results.sarif
|
||||
|
||||
# Upload the SARIF file with the Python results: 'python-results.sarif'
|
||||
|
||||
codeql github upload-results \
|
||||
--repository=my-org/example-repo \
|
||||
--ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
|
||||
--sarif=python-results.sarif
|
||||
```
|
||||
|
||||
## About the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} license
|
||||
|
||||
**License notice:** If you don’t have a {% data variables.product.prodname_enterprise %} license then, by installing this product, you are agreeing to the [{% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} Terms and Conditions](https://securitylab.github.com/tools/codeql/license).
|
||||
|
||||
@@ -21,28 +21,7 @@ redirect_from:
|
||||
|
||||
{% data reusables.code-scanning.codeql-cli-version-ghes %}
|
||||
|
||||
To analyze a codebase, you run queries against a {% data variables.product.prodname_codeql %} database extracted from the code.
|
||||
|
||||
{% data variables.product.prodname_codeql %} analyses produce [interpreted results](https://codeql.github.com/docs/codeql-overview/about-codeql/#interpret-query-results) that can be displayed as alerts or paths in source code.
|
||||
For information about writing queries to run with `database analyze`, see "[Using custom queries with the {% data variables.product.prodname_codeql_cli %}](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/using-custom-queries-with-the-codeql-cli)."
|
||||
|
||||
### Other query-running commands
|
||||
|
||||
Queries run with `database analyze` have strict [metadata requirements](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/using-custom-queries-with-the-codeql-cli#including-query-metadata). You can also execute queries using the following
|
||||
plumbing-level subcommands:
|
||||
|
||||
- [AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/database-run-queries), which
|
||||
outputs non-interpreted results in an intermediate binary format called [BQRS](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#bqrs-file)
|
||||
|
||||
- [AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/query-run), which will output BQRS files, or print
|
||||
results tables directly to the command line. Viewing results directly in
|
||||
the command line may be useful for iterative query development using the CLI.
|
||||
|
||||
Queries run with these commands don't have the same metadata requirements.
|
||||
However, to save human-readable data you have to process each BQRS results
|
||||
file using the [AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/bqrs-decode) plumbing
|
||||
subcommand. Therefore, for most use cases it's easiest to use `database
|
||||
analyze` to directly generate interpreted results.
|
||||
To analyze a codebase, you run queries against a {% data variables.product.prodname_codeql %} database extracted from the code. {% data variables.product.prodname_codeql %} analyses produce results that can be uploaded to {% data variables.product.product_name %} to generate code scanning alerts.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -51,9 +30,7 @@ Before starting an analysis you must:
|
||||
- [Set up the {% data variables.product.prodname_codeql_cli %}](/code-security/codeql-cli/getting-started-with-the-codeql-cli/setting-up-the-codeql-cli) to run commands locally.
|
||||
- [Create a {% data variables.product.prodname_codeql %} database](/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis) for the source code you want to analyze.
|
||||
|
||||
The simplest way to run `codeql database analyze` is using {% data variables.product.prodname_codeql %} packs. You can also
|
||||
run the command using queries from a local checkout of the {% data variables.product.prodname_codeql %} repository,
|
||||
which you may want to do if you want to customize the {% data variables.product.prodname_codeql %} core queries.
|
||||
The simplest way to run `codeql database analyze` is using the standard queries included in the {% data variables.product.prodname_codeql_cli %} bundle.
|
||||
|
||||
## Running `codeql database analyze`
|
||||
|
||||
@@ -90,10 +67,10 @@ You must specify `<database>`, `--format`, and `--output`. You can specify addit
|
||||
| `<database>` | {% octicon "check" aria-label="Required" %} | Specify the path for the directory that contains the {% data variables.product.prodname_codeql %} database to analyze. |
|
||||
| `<packs,queries>` | {% octicon "x" aria-label="Optional" %} | Specify {% data variables.product.prodname_codeql %} packs or queries to run. To run the standard queries used for {% data variables.product.prodname_code_scanning %}, omit this parameter. To see the other query suites included in the {% data variables.product.prodname_codeql_cli %} bundle, look in `/<extraction-root>/qlpacks/codeql/<language>-queries/codeql-suites`. For information about creating your own query suite, see [AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-codeql-query-suites) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
| <code><span style="white-space: nowrap;">--format</span></code> | {% octicon "check" aria-label="Required" %} | Specify the format for the results file generated during analysis. A number of different formats are supported, including CSV, [SARIF](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#sarif-file), and graph formats. For upload to {% data variables.product.company_short %} this should be: {% ifversion fpt or ghae or ghec %}`sarif-latest`{% else %}`sarifv2.1.0`{% endif %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)."
|
||||
| <code><span style="white-space: nowrap;">--output</span></code> | {% octicon "check" aria-label="Required" %} | Specify where to save the SARIF results file.
|
||||
| <code><span style="white-space: nowrap;">--output</span></code> | {% octicon "check" aria-label="Required" %} | Specify the location where you want to save the SARIF results file, including the desired filename with the `.sarif` extension.
|
||||
| <code><span style="white-space: nowrap;">--sarif-category</span></code> | {% octicon "question" aria-label="Required with multiple results sets" %} | Optional for single database analysis. Required to define the language when you analyze multiple databases for a single commit in a repository.<br><br>Specify a category to include in the SARIF results file for this analysis. A category is used to distinguish multiple analyses for the same tool and commit, but performed on different languages or different parts of the code.|{% ifversion code-scanning-tool-status-page %}
|
||||
| <code><span style="white-space: nowrap;">--sarif-add-baseline-file-info</span></code> | {% octicon "x" aria-label="Optional" %} | **Recommended.** Use to submit file coverage information to the {% data variables.code-scanning.tool_status_page %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/managing-your-code-scanning-configuration/about-the-tool-status-page#how-codeql-defines-scanned-files)." | {% endif %}
|
||||
| <code><span style="white-space: nowrap;">--sarif-add-query-help</span></code> | {% octicon "x" aria-label="Optional" %} | Use if you want to include any available markdown-rendered query help for custom queries used in your analysis. Any query help for custom queries included in the SARIF output will be displayed in the code scanning UI if the relevant query generates an alert. For more information, see "[Including query help for custom {% data variables.product.prodname_codeql %} queries in SARIF files](#including-query-help-for-custom-codeql-queries-in-sarif-files)."{% ifversion codeql-packs %}
|
||||
| <code><span style="white-space: nowrap;">--sarif-add-query-help</span></code> | {% octicon "x" aria-label="Optional" %} | Use if you want to include any available markdown-rendered query help for custom queries used in your analysis. Any query help for custom queries included in the SARIF output will be displayed in the code scanning UI if the relevant query generates an alert. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/using-custom-queries-with-the-codeql-cli#including-query-help-for-custom-codeql-queries-in-sarif-files)."{% ifversion codeql-packs %}
|
||||
| `<packs>` | {% octicon "x" aria-label="Optional" %} | Use if you want to include {% data variables.product.prodname_codeql %} query packs in your analysis. For more information, see "[Downloading and using {% data variables.product.prodname_codeql %} packs](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs#downloading-and-using-codeql-query-packs)."
|
||||
| <code><span style="white-space: nowrap;">--download</span></code> | {% octicon "x" aria-label="Optional" %} | Use if some of your {% data variables.product.prodname_codeql %} query packs are not yet on disk and need to be downloaded before running queries.{% endif %}
|
||||
| <code><span style="white-space: nowrap;">--threads</span></code> | {% octicon "x" aria-label="Optional" %} | Use if you want to use more than one thread to run queries. The default value is `1`. You can specify more threads to speed up query execution. To set the number of threads to the number of logical processors, specify `0`.
|
||||
@@ -308,7 +285,7 @@ codeql database analyze <cpp-database> codeql/cpp-queries:codeql-suites/cpp-code
|
||||
```
|
||||
|
||||
This command downloads the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} query pack, runs the analysis, and generates a file in the SARIF version 2.1.0 format that is supported by all versions of {% data variables.product.prodname_dotcom %}. This file can be uploaded to {% data variables.product.prodname_dotcom %} by executing `codeql github upload-results` or the code scanning API.
|
||||
For more information, see "[AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database)"
|
||||
For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/uploading-codeql-analysis-results-to-github)"
|
||||
or "[AUTOTITLE](/rest/code-scanning)".
|
||||
|
||||
{% data variables.product.prodname_codeql %} query suites are `.qls` files that use directives to select queries to run
|
||||
@@ -319,65 +296,24 @@ For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advan
|
||||
|
||||
For information about creating custom query suites, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-codeql-query-suites)."
|
||||
|
||||
#### Diagnostic and summary information
|
||||
|
||||
When you create a {% data variables.product.prodname_codeql %} database, the extractor stores diagnostic data in the database. The code scanning query suites include additional queries to report on this diagnostic data and calculate summary metrics. When the `database analyze` command completes, the CLI generates the results file and reports any diagnostic and summary data to standard output. If you choose to generate SARIF output, the additional data is also included in the SARIF file.
|
||||
|
||||
If the analysis found fewer results for standard queries than you expected, review the results of the diagnostic and summary queries to check whether the {% data variables.product.prodname_codeql %} database is likely to be a good representation of the codebase that you want to analyze.
|
||||
|
||||
### Integrating a {% data variables.product.prodname_codeql %} pack into a code scanning workflow in {% data variables.product.prodname_dotcom %}
|
||||
|
||||
You can use {% data variables.product.prodname_codeql %} query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
|
||||
For more information, see "[Using {% data variables.product.prodname_codeql %} query packs in the {% data variables.product.prodname_codeql %} action](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#using-query-packs/)" or "[Downloading and using {% data variables.product.prodname_codeql %} query packs in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs)."
|
||||
|
||||
### Including query help for custom {% data variables.product.prodname_codeql %} queries in SARIF files
|
||||
|
||||
If you use the {% data variables.product.prodname_codeql_cli %} to run code scanning analyses on third party CI/CD systems,
|
||||
you can include the query help for your custom queries in SARIF files generated during an analysis.
|
||||
After uploading the SARIF file to {% data variables.product.prodname_dotcom %}, the query help is shown in the code scanning UI for any
|
||||
alerts generated by the custom queries.
|
||||
|
||||
From {% data variables.product.prodname_codeql_cli %} v2.7.1 onwards, you can include markdown-rendered query help in SARIF files
|
||||
by providing the `--sarif-add-query-help` option when running
|
||||
`codeql database analyze`.
|
||||
For more information, see [AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database).
|
||||
|
||||
You can write query help for custom queries directly in a markdown file and save it alongside the
|
||||
corresponding query. Alternatively, for consistency with the standard {% data variables.product.prodname_codeql %} queries,
|
||||
you can write query help in the `.qhelp` format. Query help written in `.qhelp`
|
||||
files can’t be included in SARIF files, and they can’t be processed by code
|
||||
scanning so must be converted to markdown before running
|
||||
the analysis. For more information, see "[Query help files](https://codeql.github.com/docs/writing-codeql-queries/query-help-files/#query-help-files)"
|
||||
and "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/testing-query-help-files)."
|
||||
|
||||
## Results
|
||||
|
||||
You can save analysis results in a number of different formats, including SARIF
|
||||
and CSV.
|
||||
You can save analysis results in a number of different formats, including SARIF and CSV.
|
||||
|
||||
The SARIF format is designed to represent the output of a broad range of static
|
||||
analysis tools. For more information, see [AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/sarif-output).
|
||||
The SARIF format is designed to represent the output of a broad range of static analysis tools. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/sarif-output)."
|
||||
|
||||
If you choose to generate results in CSV format, then each line in the output file
|
||||
corresponds to an alert. Each line is a comma-separated list with the following information.
|
||||
For more information about what the results look like in CSV format, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/csv-output)."
|
||||
|
||||
**Property**|**Description**|**Example**
|
||||
-----|-----|-----
|
||||
Name | Name of the query that identified the result. | `Inefficient regular expression`
|
||||
Description | Description of the query.| `A regular expression that requires exponential time to match certain inputs can be a performance bottleneck, and may be vulnerable to denial-of-service attacks.`
|
||||
Severity | Severity of the query.| `error`
|
||||
Message | Alert message.| `This part of the regular expression may cause exponential backtracking on strings containing many repetitions of '\\\\'.`
|
||||
Path | Path of the file containing the alert. | `/vendor/codemirror/markdown.js`
|
||||
Start line | Line of the file where the code that triggered the alert begins. | `617`
|
||||
Start column | Column of the start line that marks the start of the alert code. Not included when equal to 1. | `32`
|
||||
End line | Line of the file where the code that triggered the alert ends. Not included when the same value as the start line. | `64`
|
||||
End column | Where available, the column of the end line that marks the end of the alert code. Otherwise the end line is repeated. | `617`
|
||||
Results files can be integrated into your own code-review or debugging infrastructure. For example, SARIF file output can be used to highlight alerts in the correct location in your source code using a SARIF viewer plugin for your IDE.
|
||||
|
||||
Results files can be integrated into your own code-review or debugging
|
||||
infrastructure. For example, SARIF file output can be used to highlight alerts
|
||||
in the correct location in your source code using a SARIF viewer plugin for your
|
||||
IDE.
|
||||
## Viewing log and diagnostic information
|
||||
|
||||
## Further reading
|
||||
When you analyze a {% data variables.product.prodname_codeql %} database using a {% data variables.product.prodname_code_scanning %} query suite, in addition to generating detailed information about alerts, the CLI reports diagnostic data from the database generation step and summary metrics. If you choose to generate SARIF output, the additional data is also included in the SARIF file. For repositories with few alerts, you may find this information useful for determining if there are genuinely few problems in the code, or if there were errors generating the {% data variables.product.prodname_codeql %} database. For more detailed output from `codeql database analyze`, use the `--verbose` option.
|
||||
|
||||
- "[Analyzing your projects in {% data variables.product.prodname_codeql %} for VS Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/analyzing-your-projects/#analyzing-your-projects)"
|
||||
For more information about the type of diagnostic information available, see "[AUTOTITLE](/code-security/code-scanning/managing-your-code-scanning-configuration/viewing-code-scanning-logs#about-analysis-and-diagnostic-information)".
|
||||
|
||||
You can choose to export and upload diagnostic information to {% data variables.product.product_name %} even if a {% data variables.product.prodname_codeql %} analysis fails. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/uploading-codeql-analysis-results-to-github#uploading-diagnostic-information-to-github-if-the-analysis-fails)."
|
||||
|
||||
## Next steps
|
||||
|
||||
- To learn how to upload your {% data variables.product.prodname_codeql %} analysis results to {% data variables.product.product_name %}, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/uploading-codeql-analysis-results-to-github)."
|
||||
|
||||
@@ -20,41 +20,37 @@ redirect_from:
|
||||
|
||||
{% data reusables.code-scanning.codeql-cli-version-ghes %}
|
||||
|
||||
{% data variables.product.prodname_codeql %} packs are used to create, share, depend on, and run {% data variables.product.prodname_codeql %} queries and libraries. You can publish your own {% data variables.product.prodname_codeql %} packs and download packs created by others. {% data variables.product.prodname_codeql %} packs contain queries, library files, query suites, and metadata.
|
||||
{% data variables.product.prodname_codeql %} packs are used to create, share, depend on, and run {% data variables.product.prodname_codeql %} queries and libraries. {% data variables.product.prodname_codeql %} packs contain queries, library files, query suites, and metadata. You can customize your {% data variables.product.prodname_codeql %} analysis by downloading packs created by others and running them on your codebase.
|
||||
|
||||
There are{% ifversion codeql-model-packs-java %} three{% else %} two{% endif %} types of {% data variables.product.prodname_codeql %} packs: {% ifversion codeql-model-packs-java %}query packs, library packs, and model packs{% else %} query packs and library packs{% endif %}.
|
||||
|
||||
- Query packs contain a set of pre-compiled queries that can be evaluated on a {% data variables.product.prodname_codeql %} database. Query parks are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and {% ifversion query-pack-compatibility %}pre-compiled representations of each query, in addition to the query sources{% else %}a compilation cache{% endif %}. This ensures consistent and efficient execution of the queries in the pack.
|
||||
- Query packs contain a set of pre-compiled queries that can be evaluated on a {% data variables.product.prodname_codeql %} database. Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and {% ifversion query-pack-compatibility %}pre-compiled representations of each query, in addition to the query sources{% else %}a compilation cache{% endif %}. This ensures consistent and efficient execution of the queries in the pack.
|
||||
|
||||
- Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled {% ifversion query-pack-compatibility %}separately{% else %}and there is no compilation cache included when the pack is published{% endif %}.{% ifversion codeql-model-packs-java %}
|
||||
|
||||
- Model packs can be used to expand {% data variables.product.prodname_code_scanning %} analysis to recognize libraries and frameworks that are not supported by default. Model packs are currently in beta and subject to change. During the beta, model packs are available for Java analysis at the repository level. For more information about creating your own model packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-model-pack)."
|
||||
- Model packs can be used to expand {% data variables.product.prodname_code_scanning %} analysis to recognize libraries and frameworks that are not supported by default. Model packs are currently in beta and subject to change. During the beta, model packs are available for Java analysis at the repository level. For more information about creating your own model packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-codeql-model-pack)."
|
||||
|
||||
{% endif %}
|
||||
|
||||
You can use the package management commands in the {% data variables.product.prodname_codeql_cli %} to create {% data variables.product.prodname_codeql %} packs, add dependencies to packs, and install or update dependencies. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-and-working-with-codeql-packs)." You can also publish and download {% data variables.product.prodname_codeql %} packs using the {% data variables.product.prodname_codeql_cli %}. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/publishing-and-using-codeql-packs)."
|
||||
The standard {% data variables.product.prodname_codeql %} packs for all supported languages are published in the [{% data variables.product.prodname_container_registry %}](https://github.com/orgs/codeql/packages). If you installed the {% data variables.product.prodname_codeql_cli %} in the standard way, using the {% data variables.product.prodname_codeql_cli %} bundle, the core query packs are already downloaded and available to you. They are:
|
||||
|
||||
The standard {% data variables.product.prodname_codeql %} packages for all supported languages are published in the [{% data variables.product.prodname_container_registry %}](https://github.com/orgs/codeql/packages).
|
||||
The [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql) contains source files for the standard {% data variables.product.prodname_codeql %} packs for all supported languages.
|
||||
- `codeql/cpp-queries`
|
||||
- `codeql/csharp-queries`
|
||||
- `codeql/go-queries`
|
||||
- `codeql/java-queries`
|
||||
- `codeql/javascript-queries`
|
||||
- `codeql/python-queries`
|
||||
- `codeql/ruby-queries`
|
||||
|
||||
## {% data variables.product.prodname_codeql %} pack structure
|
||||
You can also use the {% data variables.product.prodname_codeql_cli %} to create your own {% data variables.product.prodname_codeql %} packs, add dependencies to packs, and install or update dependencies. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-and-working-with-codeql-packs)."
|
||||
|
||||
A {% data variables.product.prodname_codeql %} pack must contain a file called `qlpack.yml` in its root directory. In the `qlpack.yml` file, the `name:` field must have a value that follows the format of `<scope>/<pack>`, where `<scope>` is the {% data variables.product.prodname_dotcom %} organization or user account that the pack will be published to and `<pack>` is the name of the pack. Additionally, query packs and library packs with {% data variables.product.prodname_codeql %} tests contain a `codeql-pack.lock.yml` file that contains the resolved dependencies of the pack. This file is generated during a call to the `codeql pack install` command, is not meant to be edited by hand, and should be added to your version control system.
|
||||
|
||||
The other files and directories within the pack should be logically organized. For example, typically:
|
||||
|
||||
- Queries are organized into directories for specific categories.
|
||||
|
||||
- Queries for specific products, libraries, and frameworks are organized into
|
||||
their own top-level directories.
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
You can publish {% data variables.product.prodname_codeql %} packs that you have created, using the {% data variables.product.prodname_codeql_cli %}. For more information on publishing and downloading {% data variables.product.prodname_codeql %} packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/publishing-and-using-codeql-packs)."
|
||||
|
||||
## Downloading and using {% data variables.product.prodname_codeql %} query packs
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} bundle includes queries that are maintained by {% data variables.product.company_short %} experts, security researchers, and community contributors. If you want to run queries developed by other organizations, {% data variables.product.prodname_codeql %} query packs provide an efficient and reliable way to download and run queries{% ifversion codeql-model-packs-java %}, while model packs (beta) can be used to expand {% data variables.product.prodname_code_scanning %} analysis to recognize libraries and frameworks that are not supported by default{% endif %}. For more information about query packs, see "[AUTOTITLE](/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning-with-codeql#about-codeql-queries)." {% ifversion codeql-model-packs-java %} For information about writing your own model packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-model-pack)."{% endif %}
|
||||
|
||||
Before you can use a {% data variables.product.prodname_codeql %} query pack to analyze a database, you must download any packages you require from the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %}. This can be done either by using the `--download` flag as part of the `codeql database analyze` command, or running `codeql pack download`. If a package is not publicly available, you will need to use a {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} to authenticate. For more information and an example, see "[Uploading results to {% data variables.product.product_name %}](#uploading-results-to-github)".
|
||||
Before you can use a {% data variables.product.prodname_codeql %} query pack to analyze a database, you must download any packages you require from the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %}. This can be done either by using the `--download` flag as part of the `codeql database analyze` command, or running `codeql pack download`. If a package is not publicly available, you will need to use a {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} to authenticate. For more information and an example, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/uploading-codeql-analysis-results-to-github#uploading-results-to-github)."
|
||||
|
||||
| Option | Required | Usage |
|
||||
|--------|:--------:|-----|
|
||||
@@ -112,7 +108,6 @@ echo $OCTO-ORG_ACCESS_TOKEN | codeql pack download <scope/name@version:path> <sc
|
||||
### Downloading {% data variables.product.prodname_codeql %} packs from multiple {% data variables.product.company_short %} container registries
|
||||
|
||||
If your {% data variables.product.prodname_codeql %} packs reside on multiple container registries, then you must instruct the {% data variables.product.prodname_codeql_cli %} where to find each pack. For more information, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#downloading-codeql-packs-from-github-enterprise-server)."
|
||||
{% endif %}
|
||||
|
||||
## Specifying which queries to run in a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
@@ -186,7 +181,7 @@ In this example, the relevant queries in the standard query pack `codeql/java-qu
|
||||
|
||||
You can specify multiple published model packs in an analysis.
|
||||
|
||||
For more information about writing your own model packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-model-pack).
|
||||
For more information about writing your own model packs, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/creating-and-working-with-codeql-packs#creating-a-model-pack)."
|
||||
|
||||
{% endif %}
|
||||
|
||||
@@ -201,4 +196,5 @@ When a pack is published for use in analyses, the `codeql pack create` or `codeq
|
||||
- For query packs, precompiled representations of each of the queries. These are faster to execute than it would be to compile the QL source for the query at each analysis.
|
||||
|
||||
Most of this data is located in a directory named `.codeql` in the published pack, but precompiled queries are in files with a `.qlx` suffix next to the `.ql` source for each query. When analyzing a database with a query from a published pack, {% data variables.product.prodname_codeql %} will load these files instead of the `.ql` source. If you need to modify the content of a _published_ pack, be sure to remove all of the `.qlx` files, since they may prevent modifications in the `.ql` files from taking effect.
|
||||
|
||||
{% endif %}
|
||||
|
||||
@@ -17,8 +17,8 @@ children:
|
||||
- /setting-up-the-codeql-cli
|
||||
- /preparing-your-code-for-codeql-analysis
|
||||
- /analyzing-your-code-with-codeql-queries
|
||||
- /customizing-analysis-with-codeql-packs
|
||||
- /uploading-codeql-analysis-results-to-github
|
||||
- /customizing-analysis-with-codeql-packs
|
||||
redirect_from:
|
||||
- /code-security/codeql-cli/using-the-codeql-cli
|
||||
---
|
||||
|
||||
@@ -15,6 +15,10 @@ topics:
|
||||
redirect_from:
|
||||
- /code-security/codeql-cli/creating-codeql-databases
|
||||
- /code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases
|
||||
- /code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-codeql-code-scanning-in-your-ci-system
|
||||
- /github/finding-security-vulnerabilities-and-errors-in-your-code/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-code-scanning-in-your-ci-system
|
||||
- /code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system
|
||||
---
|
||||
<!--The CodeQL CLI man pages include a link to a section in this article. If you rename this article,
|
||||
make sure that you also update the MS short link: https://aka.ms/codeql-docs/indirect-tracing.-->
|
||||
@@ -33,13 +37,11 @@ Before you generate a {% data variables.product.prodname_codeql %} database, you
|
||||
1. Check out the code that you want to analyze:
|
||||
- For a branch, check out the head of the branch that you want to analyze.
|
||||
- For a pull request, check out either the head commit of the pull request, or check out a {% data variables.product.prodname_dotcom %}-generated merge commit of the pull request.
|
||||
1. Set up the environment for the codebase, making sure that any dependencies are available. For more information, see "[Creating databases for non-compiled languages](/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis#creating-databases-for-non-compiled-languages)" and "[Creating databases for compiled languages](/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis#creating-databases-for-compiled-languages)" in "Preparing your code for {% data variables.product.prodname_codeql %} analysis".
|
||||
1. Set up the environment for the codebase, making sure that any dependencies are available. For more information, see "[Creating databases for non-compiled languages](#creating-databases-for-non-compiled-languages)" and "[Creating databases for compiled languages](#creating-databases-for-compiled-languages)."
|
||||
1. Find the build command, if any, for the codebase. Typically this is available in a configuration file in the CI system.
|
||||
|
||||
Once the codebase is ready, you can run `codeql database create` to create the database.
|
||||
|
||||
For information about using the {% data variables.product.prodname_codeql_cli %} in a third-party CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, see [Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system). For information about enabling {% data variables.product.prodname_codeql %} code scanning using {% data variables.product.prodname_actions %}, see {% ifversion code-scanning-without-workflow %}"[AUTOTITLE](/code-security/code-scanning/enabling-code-scanning/configuring-default-setup-for-code-scanning)" and {% endif %}"[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/configuring-advanced-setup-for-code-scanning)."
|
||||
|
||||
## Running `codeql database create`
|
||||
|
||||
{% data variables.product.prodname_codeql %} databases are created by running the following command from the checkout root of your project:
|
||||
@@ -57,6 +59,13 @@ You must specify:
|
||||
{% data reusables.code-scanning.beta-kotlin-or-swift-support %}
|
||||
{% data reusables.code-scanning.beta-ruby-support %}
|
||||
|
||||
If your codebase has a build command or script that invokes the build process, we recommend that you specify it as well:
|
||||
|
||||
```shell
|
||||
codeql database create <database> --command <build> \
|
||||
--language=<language-identifier>
|
||||
```
|
||||
|
||||
You can specify additional options depending on the location of your source file, if the code needs to be compiled, and if you want to create {% data variables.product.prodname_codeql %} databases for more than one language.
|
||||
|
||||
| Option | Required | Usage |
|
||||
@@ -77,7 +86,7 @@ For full details of all the options you can use when creating databases, see "[A
|
||||
|
||||
### Single language example
|
||||
|
||||
This example creates a {% data variables.product.prodname_codeql %} database for the repository checked out at `/checkouts/example-repo`. It uses the JavaScript extractor to create a hierarchical representation of the JavaScript and TypeScript code in the repository. The resulting database is stored in `/codeql-dbs/example-repo`.
|
||||
This example creates a single {% data variables.product.prodname_codeql %} database for the repository checked out at `/checkouts/example-repo`. It uses the JavaScript extractor to create a hierarchical representation of the JavaScript and TypeScript code in the repository. The resulting database is stored in `/codeql-dbs/example-repo`.
|
||||
|
||||
```shell
|
||||
$ codeql database create /codeql-dbs/example-repo --language={% ifversion codeql-language-identifiers-311 %}javascript-typescript{% else %}javascript{% endif %} \
|
||||
@@ -185,11 +194,9 @@ For compiled languages, {% data variables.product.prodname_codeql %} needs to in
|
||||
|
||||
### Detecting the build system
|
||||
|
||||
{% data reusables.code-scanning.beta-kotlin-or-swift-support %}
|
||||
The {% data variables.product.prodname_codeql_cli %} includes autobuilders for {% data variables.code-scanning.compiled_languages %} code. {% data variables.product.prodname_codeql %} autobuilders allow you to build projects for compiled languages without specifying any build commands. When an autobuilder is invoked, {% data variables.product.prodname_codeql %} examines the source for evidence of a build system and attempts to run the optimal set of commands required to extract a database. For more information, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#about-autobuild-for-codeql)."
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} includes autobuilders for {% data variables.code-scanning.compiled_languages %} code. {% data variables.product.prodname_codeql %} autobuilders allow you to build projects for compiled languages without specifying any build commands. When an autobuilder is invoked, {% data variables.product.prodname_codeql %} examines the source for evidence of a build system and attempts to run the optimal set of commands required to extract a database.
|
||||
|
||||
An autobuilder is invoked automatically when you execute `codeql database create` for a compiled `--language` if don’t include a
|
||||
An autobuilder is invoked automatically when you execute `codeql database create` for a compiled `--language` if you don’t include a
|
||||
`--command` option. For example, for a Java codebase, you would simply run:
|
||||
|
||||
```shell
|
||||
@@ -450,28 +457,6 @@ steps:
|
||||
# then `codeql github upload-results` ...
|
||||
```
|
||||
|
||||
## Downloading databases from {% data variables.product.prodname_dotcom_the_website %}
|
||||
## Next steps
|
||||
|
||||
{% data variables.product.prodname_dotcom %} stores {% data variables.product.prodname_codeql %} databases for over 200,000 repos on {% data variables.product.prodname_dotcom_the_website %}, which you can download using the REST API. The list of repos is constantly growing and evolving to make sure that it includes the most interesting codebases for security research.
|
||||
|
||||
You can check if a repository has any {% data variables.product.prodname_codeql %} databases available for download using the `/repos/<owner>/<repo>/code-scanning/codeql/databases` endpoint. For example, to check for {% data variables.product.prodname_codeql %} databases using the [{% data variables.product.prodname_cli %}](https://cli.github.com/manual/gh_api) you would run:
|
||||
|
||||
```shell
|
||||
gh api /repos/<owner>/<repo>/code-scanning/codeql/databases
|
||||
```
|
||||
|
||||
This command returns information about any {% data variables.product.prodname_codeql %} databases that are available for a repository, including the language the database represents, and when the database was last updated. If no {% data variables.product.prodname_codeql %} databases are available, the response is empty.
|
||||
|
||||
When you have confirmed that a {% data variables.product.prodname_codeql %} database exists for the language you are interested in, you can download it using the following command:
|
||||
|
||||
```shell
|
||||
gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip
|
||||
```
|
||||
|
||||
For more information, see the documentation for the [Get {% data variables.product.prodname_codeql %} database endpoint](/rest/code-scanning?apiVersion=2022-11-28#get-a-codeql-database-for-a-repository).
|
||||
|
||||
Before running an analysis with the {% data variables.product.prodname_codeql_cli %}, you must unzip the databases.
|
||||
|
||||
## Further reading
|
||||
|
||||
- "[Analyzing your projects in {% data variables.product.prodname_codeql %} for VS Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/analyzing-your-projects/#analyzing-your-projects)"
|
||||
- To learn how to use the {% data variables.product.prodname_codeql_cli %} to analyze the database you created from your code, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries)."
|
||||
|
||||
@@ -13,6 +13,10 @@ topics:
|
||||
- CodeQL
|
||||
redirect_from:
|
||||
- /code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli
|
||||
- /code-security/secure-coding/running-codeql-cli-in-your-ci-system
|
||||
- /code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/running-codeql-cli-in-your-ci-system
|
||||
- /code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system
|
||||
- /code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system
|
||||
|
||||
---
|
||||
|
||||
@@ -23,13 +27,13 @@ redirect_from:
|
||||
To run {% data variables.product.prodname_codeql %} commands, you need to set up the CLI so that it can access
|
||||
the tools, queries, and libraries required to create and analyze databases.
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} can be set up to support many different use cases and directory
|
||||
structures. To get started quickly, we recommend adopting a relatively simple
|
||||
setup, as outlined in the steps below.
|
||||
The {% data variables.product.prodname_codeql_cli %} can be set up to support many different use cases and directory structures. To get started quickly, we recommend adopting a relatively simple setup, as outlined in the steps below.
|
||||
|
||||
If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
|
||||
follow the steps below. For macOS version 10.15 ("Catalina") or newer, there are additional notes for some of the steps. If you are using macOS
|
||||
on Apple Silicon (for example, Apple M1), ensure that the [Xcode command-line developer
|
||||
If you plan to use the {% data variables.product.prodname_codeql_cli %} for security research or to test or contribute queries, you may want a more advanced setup of {% data variables.product.prodname_codeql_cli %}. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/advanced-setup-of-the-codeql-cli)."
|
||||
|
||||
If you are setting up the {% data variables.product.prodname_codeql_cli %} in your CI system, you need to make the full contents of the {% data variables.product.prodname_codeql_cli %} bundle available to every CI server that you want to run {% data variables.product.prodname_codeql %} {% data variables.product.prodname_code_scanning %} analysis on. For example, you might configure each server to copy the bundle from a central, internal location and extract it. Alternatively, you could use the REST API to get the bundle directly from {% data variables.product.prodname_dotcom %}, ensuring that you benefit from the latest improvements to queries. For more information, see "[AUTOTITLE](/rest/releases)" in the REST API documentation.
|
||||
|
||||
If you are using macOS on Apple Silicon (for example, Apple M1), ensure that the [Xcode command-line developer
|
||||
tools](https://developer.apple.com/downloads/index.action) and [Rosetta 2](https://support.apple.com/en-us/HT211861) are installed.
|
||||
|
||||
{% note %}
|
||||
@@ -38,28 +42,9 @@ tools](https://developer.apple.com/downloads/index.action) and [Rosetta 2](https
|
||||
|
||||
{% endnote %}
|
||||
|
||||
For information about installing the {% data variables.product.prodname_codeql_cli %} in a CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, see "[AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system)."
|
||||
|
||||
### 1. Download the {% data variables.product.prodname_codeql_cli %} zip package
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} download package is a zip archive containing tools, scripts, and
|
||||
various {% data variables.product.prodname_codeql %}-specific files. If you don’t have a {% data variables.product.prodname_enterprise %} license then, by
|
||||
downloading this archive, you are agreeing to the [{% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} Terms and
|
||||
Conditions](https://securitylab.github.com/tools/codeql/license).
|
||||
|
||||
You should download the {% data variables.product.prodname_codeql %} bundle from https://github.com/github/codeql-action/releases. The bundle contains:
|
||||
|
||||
- {% data variables.product.prodname_codeql_cli %} product
|
||||
- A compatible version of the queries and libraries from https://github.com/github/codeql
|
||||
- Precompiled versions of all the queries included in the bundle
|
||||
|
||||
{% ifversion ghes or ghae %}
|
||||
|
||||
{% note %}
|
||||
For {% data variables.product.product_name %}{% ifversion ghes %} {{ allVersions[currentVersion].currentRelease }}{% endif %}, we recommend {% data variables.product.prodname_codeql_cli %} version {% data variables.product.codeql_cli_ghes_recommended_version %}.
|
||||
{% endnote %}
|
||||
|
||||
{% endif %}
|
||||
{% data reusables.codeql-cli.download-codeql-cli-zip %}
|
||||
|
||||
You should always use the {% data variables.product.prodname_codeql %} bundle as this ensures compatibility and also gives much better performance than a separate download of the {% data variables.product.prodname_codeql_cli %} and checkout of the {% data variables.product.prodname_codeql %} queries. If you will only be running the CLI on one specific platform, download the appropriate `codeql-bundle-PLATFORM.tar.gz` file. Alternatively, you can download `codeql-bundle.tar.gz`, which contains the CLI for all supported platforms.
|
||||
|
||||
@@ -67,12 +52,7 @@ You should always use the {% data variables.product.prodname_codeql %} bundle as
|
||||
|
||||
#### Download information for macOS "Catalina" (or newer) users
|
||||
|
||||
From macOS version 10.15 ("Catalina") onwards you need to ensure that your web browser does not automatically extract zip files. If you use Safari, complete the following steps before downloading the {% data variables.product.prodname_codeql_cli %} zip archive:
|
||||
|
||||
1. Open Safari.
|
||||
1. From the Safari menu, select **Preferences...** or **Settings...** (version 13 "Ventura" onwards).
|
||||
1. Click the **General** Tab.
|
||||
1. Ensure the check-box labeled **Open "safe" files after downloading** is unchecked.
|
||||
{% data reusables.codeql-cli.download-info-macos-catalina-or-newer %}
|
||||
|
||||
### 2. Extract the zip archive
|
||||
|
||||
@@ -80,23 +60,11 @@ For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply
|
||||
|
||||
#### Extraction information for macOS "Catalina" (or newer) users
|
||||
|
||||
macOS "Catalina", "Big Sur", "Monterey", or "Ventura" users should run the following commands in the Terminal, where `${extraction-root}` is the path to the directory where you will extract the {% data variables.product.prodname_codeql_cli %} zip archive:
|
||||
|
||||
1. `mv ~/Downloads/codeql\*.zip ${extraction-root}`
|
||||
1. `cd ${extraction-root}`
|
||||
1. `/usr/bin/xattr -c codeql\*.zip`
|
||||
1. `unzip codeql\*.zip`
|
||||
{% data reusables.codeql-cli.extraction-info-macos-catalina-or-newer %}
|
||||
|
||||
### 3. Launch `codeql`
|
||||
|
||||
Once extracted, you can run {% data variables.product.prodname_codeql %} processes by running the `codeql` executable in a couple of ways:
|
||||
|
||||
- By executing `<extraction-root>/codeql/codeql`, where `<extraction-root>` is the folder where you extracted the {% data variables.product.prodname_codeql_cli %}
|
||||
package.
|
||||
- By adding `<extraction-root>/codeql` to your `PATH`, so that you
|
||||
can run the executable as just `codeql`.
|
||||
|
||||
At this point, you can execute {% data variables.product.prodname_codeql %} commands. For a full list of the {% data variables.product.prodname_codeql_cli %} commands, see "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual)."
|
||||
{% data reusables.codeql-cli.launch-codeql %}
|
||||
|
||||
{% note %}
|
||||
|
||||
@@ -112,7 +80,7 @@ After you extract the {% data variables.product.prodname_codeql_cli %} bundle, y
|
||||
- `codeql resolve qlpacks` if `/<extraction-root>/codeql` is on the `PATH`.
|
||||
- `/<extraction-root>/codeql/codeql resolve qlpacks` otherwise.
|
||||
|
||||
Extract from successful output:
|
||||
You should see output similar to the extract below, if successful:
|
||||
|
||||
```shell
|
||||
codeql/cpp-all (/<extraction-root>/qlpacks/codeql/cpp-all/<version>)
|
||||
@@ -136,113 +104,15 @@ codeql/ruby-queries (/<extraction-root>/qlpacks/codeql/ruby-queries/<version>)
|
||||
...
|
||||
```
|
||||
|
||||
You should check that the output contains the expected languages and also that the directory location for the qlpack files is correct. The location should be within the extracted {% data variables.product.prodname_codeql_cli %} bundle, shown in the earlier example as `<extraction root>`, unless you are using a checkout of `github/codeql`. If the {% data variables.product.prodname_codeql_cli %} is unable to locate the qlpacks for the expected languages, check that you downloaded the {% data variables.product.prodname_codeql %} bundle and not a standalone copy of the {% data variables.product.prodname_codeql_cli %}.
|
||||
You should check that the output contains the expected languages and also that the directory location for the qlpack files is correct. The location should be within the extracted {% data variables.product.prodname_codeql_cli %} bundle, shown in the earlier example as `<extraction root>`. If the {% data variables.product.prodname_codeql_cli %} is unable to locate the qlpacks for the expected languages, check that you downloaded the {% data variables.product.prodname_codeql %} bundle and not a standalone copy of the {% data variables.product.prodname_codeql_cli %}.
|
||||
|
||||
You can also run `codeql resolve languages` to show which languages are available for database creation. This will list the languages supported by default in your {% data variables.product.prodname_codeql_cli %} package.
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
(Optional) You can download some "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)" containing pre-compiled queries you would like to run. To do this, run `codeql pack download <pack-name> [...pack-name]`, where `pack-name` is the name of the pack you want to download. The core query packs are a good place to start. They are:
|
||||
|
||||
- `codeql/cpp-queries`
|
||||
- `codeql/csharp-queries`
|
||||
- `codeql/go-queries`
|
||||
- `codeql/java-queries`
|
||||
- `codeql/javascript-queries`
|
||||
- `codeql/python-queries`
|
||||
- `codeql/ruby-queries`
|
||||
|
||||
Alternatively, you can download query packs during the analysis by using the `--download` flag of the `codeql database analyze` command.
|
||||
Optionally, you can download some CodeQL packs containing pre-compiled queries you would like to run. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."
|
||||
|
||||
{% endif %}
|
||||
|
||||
## Generating a token for authentication with {% data variables.product.product_name %}
|
||||
## Next steps
|
||||
|
||||
If you eventually want to upload your results to {% data variables.product.product_name %} to display as code scanning alerts, you will need to generate a {% data variables.product.pat_generic %} with the `security_events` write permission. For more information, see "[AUTOTITLE](/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)."
|
||||
|
||||
If you have installed the {% data variables.product.prodname_codeql_cli %} in a third-party CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, you can use a {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} to upload results to {% data variables.product.product_name %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#generating-a-token-for-authentication-with-github)."
|
||||
|
||||
## Checking out the {% data variables.product.prodname_codeql %} source code directly
|
||||
|
||||
Some users prefer working with {% data variables.product.prodname_codeql %} query sources directly in order to work on or contribute to the Open Source shared queries. In order to do this, the following steps are recommended. Note that the following instructions are a slightly more complicated alternative to working with {% data variables.product.prodname_codeql %} packages as explained above.
|
||||
|
||||
### 1. Download the {% data variables.product.prodname_codeql_cli %} zip
|
||||
|
||||
Follow [step 1 from the previous section](#1-download-the-codeql-cli-zip-package).
|
||||
|
||||
### 2. Create a new {% data variables.product.prodname_codeql %} directory
|
||||
|
||||
Create a new directory where you can place the CLI and any queries and libraries
|
||||
you want to use. For example, `$HOME/codeql-home`.
|
||||
|
||||
The CLI’s built-in search operations automatically look in all of its sibling
|
||||
directories for the files used in database creation and analysis. Keeping these
|
||||
components in their own directory prevents the CLI searching unrelated sibling
|
||||
directories while ensuring all files are available without specifying any
|
||||
further options on the command line.
|
||||
|
||||
### 3. Obtain a local copy of the {% data variables.product.prodname_codeql %} queries
|
||||
|
||||
The [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql) contains
|
||||
the queries and libraries required for {% data variables.product.prodname_codeql %} analysis of all supported languages.
|
||||
Clone a copy of this repository into `codeql-home`.
|
||||
|
||||
By default, the root of the cloned repository will be called `codeql`.
|
||||
Rename this folder `codeql-repo` to avoid conflicting with the {% data variables.product.prodname_codeql_cli %} that you will extract in step 1. If you use git on the command line, you can
|
||||
clone and rename the repository in a single step by running
|
||||
`git clone git@github.com:github/codeql.git codeql-repo` in the `codeql-home` folder.
|
||||
|
||||
{% ifversion ghae < 3.6 %}
|
||||
{% note %}
|
||||
|
||||
**Note:** The {% data variables.product.prodname_codeql %} libraries and queries for Go analysis used to live in a separate [{% data variables.product.prodname_codeql %} for Go repository](https://github.com/github/codeql-go/). These have been moved to the `github/codeql` repository. It is no longer necessary to clone the `github/codeql-go` into a separate `codeql-home/codeql-go` folder.
|
||||
|
||||
For more information, see the [Relocation announcement](https://github.com/github/codeql-go/issues/741).
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
|
||||
Within this repository, the queries and libraries are organized into {% data variables.product.prodname_codeql %}
|
||||
packs. Along with the queries themselves, {% data variables.product.prodname_codeql %} packs contain important metadata
|
||||
that tells the {% data variables.product.prodname_codeql_cli %} how to process the query files. For more information,
|
||||
see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."
|
||||
{% endif %}
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** There are different versions of the {% data variables.product.prodname_codeql %} queries available for different users. Check out the correct version for your use case:
|
||||
|
||||
- For the queries that are intended to be used with the latest {% data variables.product.prodname_codeql_cli %} release, check out the branch tagged `codeql-cli/latest`. You should use this branch for databases you’ve built using the {% data variables.product.prodname_codeql_cli %}, fetched from code scanning on {% data variables.product.prodname_dotcom %}, or recently downloaded from {% data variables.product.prodname_dotcom_the_website %}.
|
||||
- For the most up to date {% data variables.product.prodname_codeql %} queries, check out the `main` branch. This branch represents the very latest version of {% data variables.product.prodname_codeql %}’s analysis.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### 4. Extract the zip archive
|
||||
|
||||
For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive into the directory you created in step 2.
|
||||
|
||||
For example, if the path to your copy of the {% data variables.product.prodname_codeql %} repository is `$HOME/codeql-home/codeql-repo`, then extract the CLI into
|
||||
`$HOME/codeql-home/`.
|
||||
|
||||
### 5. Launch `codeql`
|
||||
|
||||
See [step 3 from the previous section](#3-launch-codeql).
|
||||
|
||||
### 6. Verify your {% data variables.product.prodname_codeql_cli %} setup
|
||||
|
||||
{% data variables.product.prodname_codeql_cli %} has subcommands you can execute to verify that you are correctly set up to create and analyze databases:
|
||||
|
||||
- Run `codeql resolve languages` to show which languages are available for database creation. This will list the languages supported by default in your {% data variables.product.prodname_codeql_cli %} package.
|
||||
- Run `codeql resolve qlpacks` to show which {% data variables.product.prodname_codeql %} packs the CLI can find. This will display the names of all the {% data variables.product.prodname_codeql %} packs directly available to the {% data variables.product.prodname_codeql_cli %}. This should include:
|
||||
- Query packs for each supported language, for example, `codeql/{language}-queries`. These packs contain the standard queries that will be run for each analysis.
|
||||
- Library packs for each supported language, for example, `codeql/{language}-all`. These packs contain query libraries, such as control flow and data flow libraries, that may be useful to query writers.
|
||||
- Example packs for each supported language, for example, `codeql/{language}-examples`. These packs contain useful snippets of {% data variables.product.prodname_codeql %} that query writers may find useful.
|
||||
- Legacy packs that ensure custom queries and libraries created using older products are compatible with your version of {% data variables.product.prodname_codeql %}.
|
||||
|
||||
## Using two versions of the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
If you want to use the latest {% data variables.product.prodname_codeql %} features to execute queries or {% data variables.product.prodname_codeql %} tests, but also want to prepare databases that are compatible with a specific version of {% data variables.product.prodname_codeql %} code scanning on {% data variables.product.prodname_ghe_server %}, you may need to install two versions of the CLI. The recommended directory setup depends on which versions you want to install:
|
||||
|
||||
- If both versions are 2.0.2 (or newer), you can unpack both CLI archives in the same parent directory.
|
||||
- If at least one of the versions is 2.0.1 (or older), the unpacked CLI archives cannot be in the same parent directory, but they can share the same grandparent directory. For example, if you unpack version 2.0.2 into `$HOME/codeql-home/codeql-cli`, the older version should be unpacked into `$HOME/codeql-older-version/old-codeql-cli`. Here, the common grandparent is the `$HOME` directory.
|
||||
To learn how to prepare your code to be analyzed by the {% data variables.product.prodname_codeql_cli %}, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis)."
|
||||
|
||||
@@ -22,11 +22,19 @@ After analyzing a {% data variables.product.prodname_codeql %} database using th
|
||||
|
||||
If you used a method other than the {% data variables.product.prodname_codeql_cli %} to generate results, you can use other upload methods. For more information, see "[AUTOTITLE](/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github)."
|
||||
|
||||
{% data reusables.code-scanning.upload-sarif-ghas %}
|
||||
|
||||
## Generating a token for authentication with {% data variables.product.product_name %}
|
||||
|
||||
Before you can upload your results to {% data variables.product.product_name %}, you will first need to generate a {% data variables.product.pat_generic %} with the `security_events` write permission. For more information, see "[AUTOTITLE](/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)."
|
||||
|
||||
If you have installed the {% data variables.product.prodname_codeql_cli %} in a third-party CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, you can use a {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} to upload results to {% data variables.product.product_name %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/integrating-with-code-scanning/using-code-scanning-with-your-existing-ci-system#generating-a-token-for-authentication-with-github)."
|
||||
|
||||
## Uploading results to {% data variables.product.product_name %}
|
||||
|
||||
{% data reusables.code-scanning.upload-sarif-alert-limit %}
|
||||
|
||||
Before you can upload results to {% data variables.product.product_name %}, you must determine the best way to pass the {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} you created earlier to the {% data variables.product.prodname_codeql_cli %} (see "[AUTOTITLE](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#generating-a-token-for-authentication-with-github)"). We recommend that you review your CI system's guidance on the secure use of a secret store. The {% data variables.product.prodname_codeql_cli %} supports:
|
||||
Before you can upload results to {% data variables.product.product_name %}, you must determine the best way to pass the {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} you created in the previous section to the {% data variables.product.prodname_codeql_cli %}. We recommend that you review your CI system's guidance on the secure use of a secret store. The {% data variables.product.prodname_codeql_cli %} supports:
|
||||
|
||||
- Interfacing with a secret store using the `--github-auth-stdin` option (recommended).
|
||||
- Saving the secret in the environment variable `GITHUB_TOKEN` and running the CLI without including the `--github-auth-stdin` option.
|
||||
@@ -61,6 +69,12 @@ codeql github upload-results \
|
||||
|
||||
For more information, see "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/github-upload-results)."
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** If you analyzed more than one {% data variables.product.prodname_codeql %} database for a single commit, you must have specified a SARIF category for each set of results generated by this command. When you upload the results to {% data variables.product.product_name %}, {% data variables.product.prodname_code_scanning %} uses this category to store the results for each language separately. If you forget to do this, each upload overwrites the previous results. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries#running-codeql-database-analyze)."
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### Basic example of uploading results to {% data variables.product.product_name %}
|
||||
|
||||
The following example uploads results from the SARIF file `temp/example-repo-js.sarif` to the repository `my-org/example-repo`. It tells the {% data variables.product.prodname_code_scanning %} API that the results are for the commit `deb275d2d5fe9a522a0b7bd8b6b6a1c939552718` on the `main` branch. The example assumes that the {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} created for authentication with {% data variables.product.company_short %}'s REST API uses the `GITHUB_TOKEN` environment variable.
|
||||
|
||||
Reference in New Issue
Block a user