CodeQL CLI docs conversion megabranch (#32876)
Co-authored-by: Sam Browning <106113886+sabrowning1@users.noreply.github.com> Co-authored-by: Rachael Rose Renk <91027132+rachaelrenk@users.noreply.github.com> Co-authored-by: Felicity Chapman <felicitymay@github.com> Co-authored-by: Ben Ahmady <32935794+subatoi@users.noreply.github.com> Co-authored-by: Andrew Eisenberg <aeisenberg@github.com> Co-authored-by: Henning Makholm <hmakholm@github.com>
This commit is contained in:
Sarita Iyer
@@ -75,7 +75,7 @@ These queries must belong to a published {% data variables.product.prodname_code
|
||||
- {% data variables.product.prodname_ql %} packs do not include transitive dependencies, so queries in the pack can depend only on the standard libraries (that is, the libraries referenced by an `import LANGUAGE` statement in your query), or libraries in the same {% data variables.product.prodname_ql %} pack as the query.
|
||||
- {% data variables.product.prodname_codeql %} query packs (beta) can be downloaded from multiple GitHub container registries. For more information, see "[Customizing {% data variables.product.prodname_code_scanning %}](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#downloading-codeql-packs-from-github-enterprise-server)."
|
||||
|
||||
For more information, see "[About {% data variables.product.prodname_codeql %} packs](https://codeql.github.com/docs/codeql-cli/about-codeql-packs/)" in the {% data variables.product.prodname_codeql %} documentation.
|
||||
For more information, see "[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs)."
|
||||
|
||||
{% data reusables.code-scanning.beta-codeql-packs-cli %}
|
||||
|
||||
|
||||
@@ -328,6 +328,21 @@ In the example below, `scope` is the organization or personal account that publi
|
||||
packs: scope/pack1,scope/pack2@1.2.3,scope/pack3@~3.2.1,scope/pack4@4.5.6:path/to/queries
|
||||
```
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you specify a particular version of a query pack to use,
|
||||
beware that the version you specify may eventually become too old to
|
||||
be used efficiently by the default
|
||||
{% data variables.product.prodname_codeql %} engine used by the
|
||||
{% data variables.product.prodname_codeql %} action.
|
||||
To ensure optimal performance, if you need to specify exact query pack versions, you should consider reviewing periodically whether the pinned version of the query pack needs to be moved forward.
|
||||
|
||||
For more information about pack compatibility, see "[About {% data variables.product.prodname_codeql %} pack compatibility](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs#about-codeql-pack-compatibility)."
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
### Downloading {% data variables.product.prodname_codeql %} packs from {% data variables.product.prodname_ghe_server %}
|
||||
|
||||
If your workflow uses packs that are published on a {% data variables.product.prodname_ghe_server %} installation, you need to tell your workflow where to find them. You can do this by using the `registries` input of the {% data reusables.actions.action-codeql-action-init %} action. This input accepts a list of `url`, `packages`, and `token` properties as shown below.
|
||||
@@ -510,7 +525,7 @@ To find the id of a query, you can click the alert in the list of alerts in the
|
||||
|
||||
You can find another example illustrating the use of these filters in the "[Example configuration files](#example-configuration-files)" section.
|
||||
|
||||
For more information about using `exclude` and `include` filters in your custom configuration file, see "[Creating {% data variables.product.prodname_codeql %} query suites](https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/#filtering-the-queries-in-a-query-suite)." For information on the query metadata you can filter on, see "[Metadata for CodeQL queries](https://codeql.github.com/docs/writing-codeql-queries/metadata-for-codeql-queries/)."
|
||||
For more information about using `exclude` and `include` filters in your custom configuration file, see "[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites#filtering-the-queries-in-a-query-suite)." For information on the query metadata you can filter on, see "[Metadata for CodeQL queries](https://codeql.github.com/docs/writing-codeql-queries/metadata-for-codeql-queries/)."
|
||||
|
||||
{% endif %}
|
||||
|
||||
|
||||
@@ -316,7 +316,7 @@ You will see this error if {% data variables.product.prodname_codeql %} is unabl
|
||||
- There is a typo in the workflow.
|
||||
- A resource the workflow refers to by path was renamed, deleted, or moved to a new location.
|
||||
|
||||
After verifying the location of the resource, you can update the workflow to specify the correct location.
|
||||
After verifying the location of the resource, you can update the workflow to specify the correct location. {% ifversion ghes < 3.6 or ghae < 3.6 %}If you run additional queries in Go analysis, you may have been affected by the relocation of the source files. For more information, see [Relocation announcement: `github/codeql-go` moving into `github/codeql`](https://github.com/github/codeql-go/issues/741) in the github/codeql-go repository.{% endif %}
|
||||
|
||||
## Warning: "git checkout HEAD^2 is no longer necessary"
|
||||
|
||||
|
||||
@@ -0,0 +1,349 @@
|
||||
---
|
||||
title: About CodeQL packs
|
||||
intro: 'You can use {% data variables.product.prodname_codeql %} packs to run {% data variables.product.prodname_codeql %} queries maintained by other people, or to share {% data variables.product.prodname_codeql %} queries that you''ve developed.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
feature: codeql-packs
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
{% data reusables.codeql-cli.beta-note-package-management %}
|
||||
|
||||
## About {% data variables.product.prodname_codeql %} packs
|
||||
|
||||
{% ifversion ghes %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[About {% data variables.product.prodname_codeql %} packs](/enterprise-cloud@latest/code-security/code-scanning/codeql-cli-reference/about-codeql-packs)”, the Enterprise Cloud version, for the most up-to-date documentation.
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
{% data variables.product.prodname_codeql %} packs are used to create, share, depend on, and run {% data variables.product.prodname_codeql %} queries and libraries. You can publish your own {% data variables.product.prodname_codeql %} packs and download packs created by others. {% data variables.product.prodname_codeql %} packs contain queries, library files, query suites, and metadata.
|
||||
|
||||
There are two types of {% data variables.product.prodname_codeql %} packs: query packs and library packs.
|
||||
|
||||
- Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and {% ifversion query-pack-compatibility %}pre-compiled representations of each query, in addition to the query sources{% else %}a compilation cache{% endif %}. This ensures consistent and efficient execution of the queries in the pack.
|
||||
|
||||
- Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled {% ifversion query-pack-compatibility %}separately{% else %}and there is no compilation cache included when the pack is published{% endif %}.
|
||||
|
||||
You can use the package management commands in the {% data variables.product.prodname_codeql_cli %} to create {% data variables.product.prodname_codeql %} packs, add dependencies to packs, and install or update dependencies. For more information, see “[Creating and working with {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/creating-and-working-with-codeql-packs#creating-and-working-with-codeql-packs).” You can also publish and download {% data variables.product.prodname_codeql %} packs using the {% data variables.product.prodname_codeql_cli %}. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs).”
|
||||
|
||||
The standard {% data variables.product.prodname_codeql %} packages for all supported languages are published in the [{% data variables.product.prodname_container_registry %}](https://github.com/orgs/codeql/packages).
|
||||
The [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql) contains source files for the standard {% data variables.product.prodname_codeql %} packs for all supported languages.
|
||||
|
||||
## {% data variables.product.prodname_codeql %} pack structure
|
||||
|
||||
A {% data variables.product.prodname_codeql %} pack must contain a file called `qlpack.yml` in its root directory. In the `qlpack.yml` file, the `name:` field must have a value that follows the format of `<scope>/<pack>`, where `<scope>` is the {% data variables.product.prodname_dotcom %} organization or user account that the pack will be published to and `<pack>` is the name of the pack. Additionally, query packs and library packs with {% data variables.product.prodname_codeql %} tests contain a `codeql-pack.lock.yml` file that contains the resolved dependencies of the pack. This file is generated during a call to the `codeql pack install` command, is not meant to be edited by hand, and should be added to your version control system.
|
||||
|
||||
The other files and directories within the pack should be logically organized. For example, typically:
|
||||
|
||||
* Queries are organized into directories for specific categories.
|
||||
|
||||
* Queries for specific products, libraries, and frameworks are organized into
|
||||
their own top-level directories.
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
### About published packs
|
||||
|
||||
When a pack is published for use in analyses, the `codeql pack create` or `codeql pack publish` command verifies that the content is complete and also adds some additional pieces of content to it:
|
||||
|
||||
* For query packs, a copy of each of the library packs it depends on, in the precise versions it has been developed with. Users of the query pack won't need to download these library packs separately.
|
||||
|
||||
* For query packs, precompiled representations of each of the queries. These are faster to execute than it would be to compile the QL source for the query at each analysis.
|
||||
|
||||
Most of this data is located in a directory named `.codeql` in the published pack, but precompiled queries are in files with a `.qlx` suffix next to the `.ql` source for each query. When analyzing a database with a query from a published pack, {% data variables.product.prodname_codeql %} will load these files instead of the `.ql` source. If you need to modify the content of a _published_ pack, be sure to remove all of the `.qlx` files, since they may prevent modifications in the `.ql` files from taking effect.
|
||||
{% endif %}
|
||||
|
||||
## About `qlpack.yml` files
|
||||
|
||||
When executing query-related commands, {% data variables.product.prodname_codeql %} first looks in siblings of the installation directory (and their subdirectories) for `qlpack.yml` files.
|
||||
Then it checks the package cache for {% data variables.product.prodname_codeql %} packs which have been downloaded. This means that when you are developing queries locally, the local packages
|
||||
in the installation directory override packages of the same name in the package cache, so that you can test your local changes.
|
||||
|
||||
The metadata in each `qlpack.yml` file tells {% data variables.product.prodname_codeql %} how to compile any queries in the pack, what libraries the pack depends on, and where to
|
||||
find query suite definitions.
|
||||
|
||||
The contents of the {% data variables.product.prodname_codeql %} pack (queries or libraries used in {% data variables.product.prodname_codeql %} analysis) is included in the same directory as `qlpack.yml`, or its subdirectories.
|
||||
|
||||
The directory containing the `qlpack.yml` file serves as the root directory for the content of the {% data variables.product.prodname_codeql %} pack. That is, for all `.ql` and `.qll` files in the pack, {% data variables.product.prodname_codeql %} will resolve all import statements relative to the directory containing the `qlpack.yml` file at the pack’s root.
|
||||
|
||||
### `qlpack.yml` properties
|
||||
|
||||
The following properties are supported in `qlpack.yml` files.
|
||||
|
||||
#### `name`
|
||||
|
||||
- Required by all packs.
|
||||
- Defines the scope of the pack, where the {% data variables.product.prodname_codeql %} pack is published, and the name of the pack defined using alphanumeric characters and hyphens. It must be unique as {% data variables.product.prodname_codeql %} cannot differentiate between {% data variables.product.prodname_codeql %} packs with identical names. Use the pack name to specify queries to run using `database analyze` and to define dependencies between {% data variables.product.prodname_codeql %} packs (see examples below). For example:
|
||||
```yaml
|
||||
name: octo-org/security-queries
|
||||
```
|
||||
|
||||
#### `version`
|
||||
|
||||
- Required by all packs that are published.
|
||||
- Defines a semantic version for this {% data variables.product.prodname_codeql %} pack that must adhere to the [SemVer v2.0.0 specification](https://semver.org/spec/v2.0.0.html). For example:
|
||||
```yaml
|
||||
version: 0.0.0
|
||||
```
|
||||
|
||||
#### `dependencies`
|
||||
|
||||
- Required by packs that define {% data variables.product.prodname_codeql %} package dependencies on other packs.
|
||||
- Defines a map from pack references to the semantic version range that is compatible with this pack. Supported for {% data variables.product.prodname_codeql_cli %} versions v2.6.0 and later. For example:
|
||||
```yaml
|
||||
dependencies:
|
||||
codeql/cpp-all: ^0.0.2`
|
||||
|
||||
#### `defaultSuiteFile`
|
||||
|
||||
- Required by packs that export a set of default queries to run.
|
||||
- Defines the path to a query suite file relative to the package root, containing all of the queries that are run by default when this pack is passed to the `codeql database analyze` command. Supported from CLI version v2.6.0 and onwards. Only one of `defaultSuiteFile` or `defaultSuite` can be defined. For example:
|
||||
```yaml
|
||||
defaultSuiteFile: cpp-code-scanning.qls
|
||||
```
|
||||
|
||||
#### `defaultSuite`
|
||||
|
||||
- Required by packs that export a set of default queries to run.
|
||||
- Defines an inlined query suite containing all of the queries that are run by default when this pack is passed to the `codeql database analyze` command. Supported from CLI version v2.6.0 and onwards. Only one of `defaultSuiteFile` or `defaultSuite` can be defined. For example:
|
||||
```yaml
|
||||
defaultSuite:
|
||||
queries: .
|
||||
exclude:
|
||||
precision: medium
|
||||
```
|
||||
|
||||
#### `library`
|
||||
|
||||
- Required by library packs.
|
||||
- Defines a boolean value that indicates whether or not this pack is a library pack. Library packs do not contain queries and are not compiled. Query packs can ignore this field or explicitly set it to `false`. For example:
|
||||
```yaml
|
||||
library: true
|
||||
```
|
||||
|
||||
#### `suites`
|
||||
|
||||
- Optional for packs that define query suites.
|
||||
- Defines the path to a directory in the pack that contains the query suites you want to make known to the {% data variables.product.prodname_codeql_cli %}, defined relative to the pack directory. {% data variables.product.prodname_codeql %} pack users can run “well-known” suites stored in this directory by specifying the pack name, without providing their full path. This is not supported for {% data variables.product.prodname_codeql %} packs downloaded from the Container registry. For more information about query suites, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites/).” For example:
|
||||
```yaml
|
||||
suites: octo-org-query-suites
|
||||
```
|
||||
|
||||
#### `tests`
|
||||
- Optional for packs containing {% data variables.product.prodname_codeql %} tests. Ignored for packs without tests.
|
||||
- Defines the path to a directory within the pack that contains tests, defined relative to the pack directory. Use `.` to specify the whole pack. Any queries in this directory are run as tests when `test run` is run with the `--strict-test-discovery` option. These queries are ignored by query suite definitions that use `queries` or `qlpack` instructions to ask for all queries in a particular pack. If this property is missing, then `.` is assumed. For example:
|
||||
```yaml
|
||||
tests: .
|
||||
```
|
||||
|
||||
#### `extractor`
|
||||
- Required by all packs containing {% data variables.product.prodname_codeql %} tests.
|
||||
- Defines the {% data variables.product.prodname_codeql %} language extractor to use when running the {% data variables.product.prodname_codeql %} tests in the pack. For more information about testing queries, see “[Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries/).” For example:
|
||||
```yaml
|
||||
extractor: javascript
|
||||
```
|
||||
|
||||
#### `authors`
|
||||
- Optional.
|
||||
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For example:
|
||||
```yaml
|
||||
authors: author1@github.com,author2@github.com
|
||||
```
|
||||
|
||||
#### `license`
|
||||
- Optional.
|
||||
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For a list of allowed licenses, see [SPDX License List](https://spdx.org/licenses/) in the SPDX Specification. For example:
|
||||
```yaml
|
||||
license: MIT
|
||||
```
|
||||
|
||||
#### `description`
|
||||
- Optional.
|
||||
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For example:
|
||||
```yaml
|
||||
description: Human-readable description of the contents of the {% data variables.product.prodname_codeql %} pack.
|
||||
```
|
||||
|
||||
#### `libraryPathDependencies`
|
||||
- Optional, deprecated. Use the `dependencies` property instead.
|
||||
- Previously used to define the names of any {% data variables.product.prodname_codeql %} packs that this {% data variables.product.prodname_codeql %} pack depends on, as an array. This gives the pack access to any libraries, database schema, and query suites defined in the dependency. For example:
|
||||
```yaml
|
||||
libraryPathDependencies: codeql/javascript-all
|
||||
```
|
||||
|
||||
#### `dbscheme`
|
||||
- Required by core language packs only.
|
||||
- Defines the path to the [database schema](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#codeql-database-schema) for all libraries and queries written for this {% data variables.product.prodname_codeql %} language (see example below). For example:
|
||||
```yaml
|
||||
dbscheme: semmlecode.python.dbscheme
|
||||
```
|
||||
#### `upgrades`
|
||||
- Required by core language packs only.
|
||||
- Defines the path to a directory within the pack that contains database upgrade scripts, defined relative to the pack directory. Database upgrades are used internally to ensure that a database created with a different version of the {% data variables.product.prodname_codeql_cli %} is compatible with the current version of the CLI. For example:
|
||||
```yaml
|
||||
upgrades: .
|
||||
```
|
||||
|
||||
## About `codeql-pack.lock.yml` files
|
||||
|
||||
`codeql-pack.lock.yml` files store the versions of the resolved transitive dependencies of a {% data variables.product.prodname_codeql %} pack. This file is created by the `codeql pack install` command if it does not already exist and should be added to your version control system. The `dependencies` section of the `qlpack.yml` file contains version ranges that are compatible with the pack. The `codeql-pack.lock.yml` file locks the versions to precise dependencies. This ensures that running `codeql pack install` on this the pack will always retrieve the same versions of dependencies even if newer compatible versions exist.
|
||||
|
||||
For example, if a `qlpack.yml` file contains the following dependencies:
|
||||
|
||||
```yaml
|
||||
dependencies:
|
||||
codeql/cpp-all: ^0.1.2
|
||||
my-user/my-lib: ^0.2.3
|
||||
other-dependency/from-source: "*"
|
||||
```
|
||||
|
||||
The `codeql-pack.lock.yml` file will contain something like the following:
|
||||
|
||||
```yaml
|
||||
dependencies:
|
||||
codeql/cpp-all:
|
||||
version: 0.1.4
|
||||
my-user/my-lib:
|
||||
version: 0.2.4
|
||||
my-user/transitive-dependency:
|
||||
version: 1.2.4
|
||||
```
|
||||
|
||||
The `codeql/cpp-all` dependency is locked to version 0.1.4. The `my-user/my-lib` dependency is locked to version 0.2.4. The `my-user/transitive-dependency`, which is a transitive dependency and is not specified in the `qlpack.yml` file, is locked to version 1.2.4. The `other-dependency/from-source` is absent from the lock file since it is resolved from source. This dependency must be available in the same {% data variables.product.prodname_codeql %} workspace as the pack. For more information about {% data variables.product.prodname_codeql %} workspaces and resolving dependencies from source, see “[About {% data variables.product.prodname_codeql %} Workspaces](/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces).”
|
||||
|
||||
In most cases, the `codeql-pack.lock.yml` file is only relevant for query packs since library packs are non-executable and usually do not need their transitive dependencies to be fixed. The exception to this is for library packs that contain tests. In this case, the `codeql-pack.lock.yml` file is used to ensure that the tests are always run with the same versions of dependencies to avoid spurious failures when there are mismatched dependencies.
|
||||
|
||||
## Examples of custom {% data variables.product.prodname_codeql %} packs
|
||||
|
||||
When you write custom queries or tests, you should save them in custom {% data variables.product.prodname_codeql %} packs. For simplicity, try to organize each pack logically. For more information, see “[{% data variables.product.prodname_codeql %} pack structure](#codeql-pack-structure).” Save files for queries and tests in separate packs and, where possible, organize custom packs into specific folders for each target language. This is particuarly useful if you intend to publish your {% data variables.product.prodname_codeql %} packs so they can be shared with others or used in code scanning. For more information, see "[About code scanning with {% data variables.product.prodname_codeql %}](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql)."
|
||||
|
||||
### {% data variables.product.prodname_codeql %} packs for custom libraries
|
||||
|
||||
A custom {% data variables.product.prodname_codeql %} pack containing custom C++ libraries, with no queries or tests, may have a `qlpack.yml` file containing:
|
||||
|
||||
```yaml
|
||||
name: my-github-user/my-custom-libraries
|
||||
version: 1.2.3
|
||||
library: true
|
||||
dependencies:
|
||||
codeql/cpp-all: ^0.1.2
|
||||
```
|
||||
|
||||
where `codeql/cpp-all` is the name of the {% data variables.product.prodname_codeql %} pack for C/C++ analysis included in the {% data variables.product.prodname_codeql %} repository. The version range `^0.1.2` indicates that this pack is compatible with all versions of `codeql/cpp-all` that are greater than or equal to `0.1.2` and less than `0.2.0`. Any {% data variables.product.prodname_codeql %} library file (a file with a `.qll` extension) defined in this pack will be available to queries defined in any query pack that includes this pack in its dependencies block.
|
||||
|
||||
The `library` property indicates that this pack is a library pack and does not contain any queries.
|
||||
|
||||
### {% data variables.product.prodname_codeql %} packs for custom queries
|
||||
|
||||
A custom {% data variables.product.prodname_codeql %} pack containing custom C++ queries and libraries may have a `qlpack.yml` file containing:
|
||||
|
||||
```yaml
|
||||
name: my-github-user/my-custom-queries
|
||||
version: 1.2.3
|
||||
dependencies:
|
||||
codeql/cpp-all: ^0.1.2
|
||||
my-github-user/my-custom-libraries: ^1.2.3
|
||||
suites: my-custom-suites
|
||||
```
|
||||
|
||||
where `codeql/cpp-all` is the name of the {% data variables.product.prodname_codeql %} pack for C/C++ analysis included in the {% data variables.product.prodname_codeql %} repository. The version range `^0.1.2` indicates that this pack is compatible with all versions of `codeql/cpp-all` that are greater than or equal to `0.1.2` and less than `0.2.0`. `my-github-user/my-custom-libraries` is the name of a {% data variables.product.prodname_codeql %} pack containing custom {% data variables.product.prodname_codeql %} libraries for C++. Any {% data variables.product.prodname_codeql %} library file (a file with a `.qll` extension) defined in this pack will be available to queries in the `my-github-user/my-custom-queries` pack.
|
||||
|
||||
The `suites` property indicates a directory where “well-known” query suites can be found. These suites can be used on the command line by referring to their name only, rather than their full path. For more information about query suites, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites).”
|
||||
|
||||
### {% data variables.product.prodname_codeql %} packs for custom tests
|
||||
|
||||
For custom {% data variables.product.prodname_codeql %} packs containing test files, you also need to include an
|
||||
`extractor` property so that the `test run` command knows how to create test
|
||||
databases. You may also wish to specify the `tests` property.
|
||||
|
||||
{% data reusables.codeql-cli.test-qlpack %}
|
||||
|
||||
For more information about running tests, see “[Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries).”
|
||||
|
||||
## Examples of {% data variables.product.prodname_codeql %} packs in the {% data variables.product.prodname_codeql %} repository
|
||||
|
||||
Each of the languages in the {% data variables.product.prodname_codeql %} repository has four main {% data variables.product.prodname_codeql %} packs:
|
||||
|
||||
* Core library pack for the language, with the database schema
|
||||
used by the language, and {% data variables.product.prodname_codeql %} libraries, and queries at `<language>/ql/lib`
|
||||
|
||||
* Core query pack for the language that includes the default queries for the language, along
|
||||
with their query suites at `<language>/ql/src`
|
||||
|
||||
* Tests for the core language libraries and queries at `<language>/ql/test`
|
||||
|
||||
* Example queries for the language at `<language>/ql/examples`
|
||||
|
||||
### Core library pack
|
||||
|
||||
Here is an example `qlpack.yml` file for the [C/C++ analysis libraries](https://github.com/github/codeql/blob/main/cpp/ql/lib/qlpack.yml)
|
||||
core language pack:
|
||||
|
||||
```yaml
|
||||
name: codeql/cpp-all
|
||||
version: x.y.z-dev
|
||||
dbscheme: semmlecode.cpp.dbscheme
|
||||
library: true
|
||||
upgrades: upgrades
|
||||
```
|
||||
|
||||
Some extra notes on the following properties:
|
||||
|
||||
* `library`: Indicates that this is a library pack with no executable queries. It is only meant to be used as a dependency for other packs.
|
||||
|
||||
* `dbscheme` and `upgrades`: These properties are internal to the {% data variables.product.prodname_codeql_cli %} and should only be defined in the core QL pack for a language.
|
||||
|
||||
### Core query pack
|
||||
|
||||
Here is an example `qlpack.yml` file for [C/C++ analysis queries](https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml)
|
||||
core query pack:
|
||||
|
||||
```yaml
|
||||
name: codeql/cpp-queries
|
||||
version: x.y.z-dev
|
||||
dependencies:
|
||||
codeql/cpp-all: "*"
|
||||
codeql/suite-helpers: "*"
|
||||
suites: codeql-suites
|
||||
defaultSuiteFile: codeql-suites/cpp-code-scanning.qls
|
||||
```
|
||||
|
||||
Some extra notes on the following properties:
|
||||
|
||||
* `dependencies`: This query pack depends on `codeql/cpp-all` and `codeql/suite-helpers`. Since these dependencies are resolved from source, it does not matter what version of the {% data variables.product.prodname_codeql %} pack they are compatible with. For more information about resolving dependencies from source, see “[Source Dependencies](/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces#source-dependencies).”
|
||||
|
||||
* `suites`: Indicates the directory containing “well-known” query suites.
|
||||
|
||||
* `defaultSuiteFile`: The name of the default query suite file that is used when no query suite is specified.
|
||||
|
||||
### Tests for the core {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
Here is an example `qlpack.yml` file for [C/C++ analysis tests](https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml)
|
||||
core test pack:
|
||||
|
||||
```yaml
|
||||
name: codeql/cpp-tests
|
||||
dependencies:
|
||||
codeql/cpp-all: "*"
|
||||
codeql/cpp-queries: "*"
|
||||
extractor: cpp
|
||||
tests: .
|
||||
```
|
||||
|
||||
Some extra notes on the following properties:
|
||||
|
||||
* `dependencies`: This pack depends on the core {% data variables.product.prodname_codeql %} query and library packs for C++.
|
||||
|
||||
* `extractor`: This specifies that all the tests will use the same C++ extractor to create the database for the tests.
|
||||
|
||||
* `tests`: This specifies the location of the tests. In this case, the tests are in the root folder (and all sub-folders) of the pack.
|
||||
|
||||
* `version`: There is no `version` property for the tests pack. This prevents test packs from accidentally being published.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: About CodeQL workspaces
|
||||
intro: '{% data variables.product.prodname_codeql %} workspaces allow you to develop and maintain a group of {% data variables.product.prodname_codeql %} packs that depend on each other.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
feature: codeql-packs
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About {% data variables.product.prodname_codeql %} workspaces
|
||||
|
||||
{% ifversion ghes %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[About {% data variables.product.prodname_codeql %} workspaces](/enterprise-cloud@latest/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces)”, the Enterprise Cloud version, for the most up-to-date documentation.
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
You use a {% data variables.product.prodname_codeql %} workspace when you want to group multiple {% data variables.product.prodname_codeql %} packs together. A typical use case for a {% data variables.product.prodname_codeql %} workspace is to develop a set of {% data variables.product.prodname_codeql %} library and query packs that are mutually dependent. For more information on {% data variables.product.prodname_codeql %} packs, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).”
|
||||
|
||||
The main benefit of a {% data variables.product.prodname_codeql %} workspace is that it makes it easier for you to develop and maintain multiple {% data variables.product.prodname_codeql %} packs. When you use a {% data variables.product.prodname_codeql %} workspace, all the {% data variables.product.prodname_codeql %} packs in the workspace are available as *source dependencies* for each other when you run a {% data variables.product.prodname_codeql %} command that resolves queries. This makes it easier to develop, maintain, and publish multiple, related {% data variables.product.prodname_codeql %} packs.
|
||||
|
||||
In most cases, you should store the {% data variables.product.prodname_codeql %} workspace and the {% data variables.product.prodname_codeql %} packs contained in it in one git repository. This makes it easier to share your {% data variables.product.prodname_codeql %} development environment.
|
||||
|
||||
## The `codeql-workspace.yml` file
|
||||
|
||||
A {% data variables.product.prodname_codeql %} workspace is defined by a `codeql-workspace.yml` yaml file. This file contains a `provide` block, and optionally `ignore` and `registries` blocks.
|
||||
|
||||
* The `provide` block contains a list of glob patterns that define the {% data variables.product.prodname_codeql %} packs that are available in the workspace.
|
||||
|
||||
* The `ignore` block contains a list of glob patterns that define {% data variables.product.prodname_codeql %} packs that are not available in the workspace.
|
||||
|
||||
* The `registries` block contains a list of GHES URLs and package patterns that control which container registry is used for publishing {% data variables.product.prodname_codeql %} packs. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs#working-with-codeql-packs-on-ghes)."
|
||||
|
||||
Each entry in the `provide` or `ignore` section must map to the location of a `qlpack.yml` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see “[@actions/glob](https://github.com/actions/toolkit/tree/main/packages/glob#patterns) .”
|
||||
|
||||
For example, the following `codeql-workspace.yml` file defines a workspace that contains all the {% data variables.product.prodname_codeql %} packs recursively found in the `codeql-packs` directory, except for the packs in the `experimental` directory. The `registries` block specifies that `codeql/\*` packs should be downloaded from [https://ghcr.io/v2/](https://ghcr.io/v2/), which is {% data variables.product.prodname_dotcom %}’s default container registry. All other packs should be downloaded from and published to the regsitry at `GHE_HOSTNAME`.
|
||||
|
||||
```yaml
|
||||
provide:
|
||||
- "*/codeql-packs/**/qlpack.yml"
|
||||
ignore:
|
||||
- "*/codeql-packs/**/experimental/**/qlpack.yml"
|
||||
|
||||
registries:
|
||||
- packages: 'codeql/*'
|
||||
url: https://ghcr.io/v2/
|
||||
|
||||
- packages: '*'
|
||||
url: https://containers.GHE_HOSTNAME/v2/
|
||||
```
|
||||
|
||||
To verify that your `codeql-workspace.yml` file includes the {% data variables.product.prodname_codeql %} packs that you expect, run the `codeql pack ls` command in the same directory as your workspace. The result of the command is a list of all {% data variables.product.prodname_codeql %} packs in the workspace.
|
||||
|
||||
## Source dependencies
|
||||
|
||||
Source dependencies are {% data variables.product.prodname_codeql %} packs that are resolved from the local file system outside of the {% data variables.product.prodname_codeql %} package cache. These dependencies can be in the same {% data variables.product.prodname_codeql %} workspace, or specified as a path option using the `--additional-packs` argument. When you compile and run queries locally, source dependencies override any dependencies found in the {% data variables.product.prodname_codeql %} package cache as well as version constraints defined in the `qlpack.yml`. All references to {% data variables.product.prodname_codeql %} packs in the same workspace are resolved as source dependencies.
|
||||
|
||||
This is particularly useful in the following situations:
|
||||
|
||||
* One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack.
|
||||
|
||||
* You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it.
|
||||
|
||||
## {% data variables.product.prodname_codeql %} workspaces and query resolution
|
||||
|
||||
All {% data variables.product.prodname_codeql %} packs in a workspace are available as source dependencies for each other when you run any {% data variables.product.prodname_codeql %} command that resolves queries or packs. For example, when you run `codeql pack install` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the `codeql-pack.lock.yml` file. For more information, see “[Creating and working with {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/creating-and-working-with-codeql-packs#adding-and-installing-dependencies).”
|
||||
|
||||
Similarly, when you publish a {% data variables.product.prodname_codeql %} query pack to the {% data variables.product.prodname_dotcom %} container registry using `codeql pack publish` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache.
|
||||
|
||||
This ensures that any local changes you make to a query library in a dependency are automatically reflected in any query packs you publish from that workspace.
|
||||
|
||||
### Example
|
||||
|
||||
Consider the following `codeql-workspace.yml` file:
|
||||
|
||||
```yaml
|
||||
provide:
|
||||
- "**/qlpack.yml"
|
||||
```
|
||||
|
||||
And the following {% data variables.product.prodname_codeql %} library pack `qlpack.yml` file in the workspace:
|
||||
|
||||
```yaml
|
||||
name: my-company/my-library
|
||||
library: true
|
||||
version: 1.0.0
|
||||
```
|
||||
|
||||
And the following {% data variables.product.prodname_codeql %} query pack `qlpack.yml` file in the workspace:
|
||||
|
||||
```yaml
|
||||
name: my-company/my-queries
|
||||
version: 1.0.0
|
||||
dependencies:
|
||||
my-company/my-library: "*"
|
||||
codeql/cpp-all: ~0.2.0
|
||||
```
|
||||
|
||||
Notice that the `dependencies` block for the {% data variables.product.prodname_codeql %} query pack, `my-company/my-queries`, specifies `"*"` as the version of the library pack. Since the library pack is already defined as a source dependency in `codeql-workspace.yml`, the library pack’s content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use `"*"` for source dependencies to make it clear that the version is inherited from the workspace.
|
||||
|
||||
When you execute `codeql pack install` from the query pack directory, an appropriate version of `codeql/cpp-all` is downloaded to the local package cache. Also, a `codeql-pack.lock.yml` file is created that contains the resolved version of `codeql/cpp-all`. The lock file won’t contain an entry for `my-company/my-library` since it is resolved from source dependencies. The `codeql-pack.lock.yml` file will look something like this:
|
||||
|
||||
```yaml
|
||||
dependencies:
|
||||
codeql/cpp-all:
|
||||
version: 0.2.2
|
||||
```
|
||||
|
||||
When you execute `codeql pack publish` from the query pack directory, the `codeql/cpp-all` dependency from the package cache and the `my-company/my-library` from the workspace are bundled with `my-company/my-queries` and published to the {% data variables.product.prodname_dotcom %} container registry.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Exit codes
|
||||
intro: 'Exit codes signify the status of a command after the {% data variables.product.prodname_codeql_cli %} runs it.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About exit codes
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} reports the status of each command it runs as an exit code.
|
||||
This exit code provides information for subsequent commands or for other tools that rely on the {% data variables.product.prodname_codeql_cli %}.
|
||||
|
||||
## 0
|
||||
|
||||
Success, normal termination.
|
||||
|
||||
## 1
|
||||
|
||||
The command successfully determined that the answer to your question is “no”.
|
||||
|
||||
This exit code is only used by a few commands, such as [codeql test run](https://codeql.github.com/docs/codeql-cli/manual/test-run/), [codeql database check](https://codeql.github.com/docs/codeql-cli/manual/dataset-check/), [codeql query format](https://codeql.github.com/docs/codeql-cli/manual/query-format/),and [codeql resolve extractor](https://codeql.github.com/docs/codeql-cli/manual/resolve-extractor/).
|
||||
For more details, see the documentation for those commands.
|
||||
|
||||
## 2
|
||||
|
||||
Something went wrong.
|
||||
|
||||
The CLI writes a human-readable error message to stderr.
|
||||
This includes cases where an extractor fails with an internal error, because the `codeql` driver can’t distinguish between internal and user-facing errors in extractor behavior.
|
||||
|
||||
## 3
|
||||
|
||||
The launcher was unable to find the {% data variables.product.prodname_codeql %} installation directory.
|
||||
|
||||
In this case, the launcher can’t start the Java code for the {% data variables.product.prodname_codeql_cli %} at all. This should only happen when something is severely wrong with the {% data variables.product.prodname_codeql %} installation.
|
||||
|
||||
## 32
|
||||
|
||||
The extractor didn’t find any code to analyze when running [codeql database create](https://codeql.github.com/docs/codeql-cli/manual/database-create/) or [codeql database finalize](https://codeql.github.com/docs/codeql-cli/manual/database-finalize/).
|
||||
|
||||
## 33
|
||||
|
||||
One or more query evaluations timed out.
|
||||
|
||||
It’s possible that some queries that were evaluated in parallel didn’t time out. The results for those queries are produced as usual.
|
||||
|
||||
## 98
|
||||
|
||||
Evaluation was explicitly canceled.
|
||||
|
||||
## 99
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} ran out of memory.
|
||||
|
||||
This doesn’t necessarily mean that all the machine’s physical RAM has been used.
|
||||
If you don’t use the `--ram` option to set a limit explicitly, the JVM decides on a default limit at startup.
|
||||
|
||||
## 100
|
||||
|
||||
A fatal internal error occurred.
|
||||
|
||||
This should be considered a bug. The CLI usually writes an abbreviated error description to stderr.
|
||||
If you can reproduce the bug, it’s helpful to use `--logdir` and send the log files to {% data variables.product.prodname_dotcom %} in a bug report.
|
||||
|
||||
## Other
|
||||
|
||||
In the case of really severe problems within the JVM that runs `codeql`, it might return a nonzero exit code of its own choosing.
|
||||
This should only happen if something is severely wrong with the {% data variables.product.prodname_codeql %} installation, or if there is a memory issue with the host system running the {% data variables.product.prodname_codeql %} process. For example, Unix systems may return Exit Code 137 to indicate that the kernel has killed a process that {% data variables.product.prodname_codeql %} has started. One way to troubleshoot this is to modify your `–ram=` flag for the `codeql database analyze` step and re-run your workflow.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: CodeQL CLI reference
|
||||
intro: 'You can learn how to use {% data variables.product.prodname_codeql %} workspaces and {% data variables.product.prodname_codeql %} packs and how to understand the output of {% data variables.product.prodname_codeql %} commands.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
children:
|
||||
- /about-codeql-packs
|
||||
- /about-codeql-workspaces
|
||||
- /query-reference-files
|
||||
- /sarif-output
|
||||
- /exit-codes
|
||||
---
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: Query reference files
|
||||
intro: 'You can use query reference files to define the location of a query you want to run in tests.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About query reference files
|
||||
|
||||
A query reference file is text file that defines the location of one query to test.
|
||||
|
||||
You use a query reference file when you want to tell the `test run` subcommand
|
||||
to run a query that’s not part of a test directory.
|
||||
There are two ways to specify queries that you want to run as tests:
|
||||
|
||||
1. Use a query reference file to specify the location of a query to test.
|
||||
This is useful when you create tests for alert and path queries that
|
||||
are intended to identify problems in real codebases. You might create
|
||||
several directories of test code, each focusing on different
|
||||
aspects of the query. Then you would add a query reference file to
|
||||
each directory of test code, to specify the query to test.
|
||||
2. Add the query directly to a directory of tests.
|
||||
These is typically useful when you’re writing queries explicitly to test the behavior
|
||||
of QL libraries. Often these queries contain just a few calls to library predicates,
|
||||
wrapping them in a `select` statement so their output can be tested.
|
||||
|
||||
## Defining a query reference file
|
||||
|
||||
Each query reference file, `.qlref`, contains a single line that defines
|
||||
where to find one query. The location must be defined relative
|
||||
to the root of the {% data variables.product.prodname_codeql %} pack that contains the query.
|
||||
Usually, this is either the {% data variables.product.prodname_codeql %} pack that contains the `.qlref`, a {% data variables.product.prodname_codeql %} pack specified in the `dependencies` block for the test pack, or a transitive dependency of the {% data variables.product.prodname_codeql %} pack.
|
||||
|
||||
You should use forward slashes in the path on all operating
|
||||
systems to ensure compatibility between systems.
|
||||
|
||||
### Example
|
||||
|
||||
A query reference file to test a JavaScript alert query:
|
||||
[DeadAngularJSEventListener.qlref](https://github.com/github/codeql/blob/main/javascript/ql/test/query-tests/AngularJS/DeadAngularJSEventListener/DeadAngularJSEventListener.qlref)
|
||||
|
||||
The `qlpack.yml` file, https://github.com/github/codeql/blob/main/javascript/ql/test/qlpack.yml,
|
||||
for the CodeQL pack at `javascript/ql/test` defines `codeql/javascript-queries` as
|
||||
a dependency. So the query reference file defines the location of the query relative
|
||||
to the `codeql/javascript-queries` {% data variables.product.prodname_codeql %} pack:
|
||||
|
||||
```
|
||||
AngularJS/DeadAngularJSEventListener.ql
|
||||
```
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
For another example, see [Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries).
|
||||
{% endif %}
|
||||
@@ -0,0 +1,188 @@
|
||||
---
|
||||
title: CodeQL CLI SARIF output
|
||||
intro: 'You can output SARIF from the {% data variables.product.prodname_codeql_cli %} and share static analysis results with other systems.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
allowTitleToDifferFromFilename: true
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About SARIF output
|
||||
|
||||
SARIF is designed to represent the output of a broad range of static analysis tools, and there are many features in the SARIF specification that are
|
||||
considered “optional”. This document details the output produced when using the format type `sarifv2.1.0`, which corresponds to the SARIF v2.1.0.csd1 specification. For more information on selecting a file format for your analysis results, see the [database analyze](https://codeql.github.com/docs/codeql-cli/manual/database-analyze/) command in the {% data variables.product.prodname_codeql %} documentation.
|
||||
|
||||
## SARIF specification and schema
|
||||
|
||||
This article is intended to be read alongside the detailed SARIF specification. For more information on the specification and the SARIF schema, see the [SARIF specification documentation](https://github.com/oasis-tcs/sarif-spec/blob/main/Schemata/sarif-schema-2.1.0.json).
|
||||
|
||||
## Change notes
|
||||
|
||||
### Changes between versions
|
||||
|
||||
| CodeQL version | Format type | Changes |
|
||||
|----------------|-------------|---------|
|
||||
| 2.0.0 | `sarifv2.1.0` | First version of this format. |
|
||||
|
||||
### Future changes to the output
|
||||
|
||||
The output produced for a given specific format type (for example, `sarifv2.1.0`) may change in future {% data variables.product.prodname_codeql %} releases. We will endeavor to maintain backwards compatibility with consumers of the generated SARIF by ensuring that:
|
||||
|
||||
* No field which is marked as “Always” being generated will be removed.
|
||||
|
||||
* The circumstances under which “Optional” fields are generated may change. Consumers of the {% data variables.product.prodname_codeql %} SARIF output should be robust to the presence or absence of these fields.
|
||||
|
||||
New output fields may be added in future releases under the same format type–these are not considered to break backwards compatibility, and consumers should be robust to the presence of newly added fields.
|
||||
|
||||
New format argument types may be added in future versions of {% data variables.product.prodname_codeql %}—for example, to support new versions of SARIF. These have no guarantee of backwards compatibility, unless explicitly documented.
|
||||
|
||||
## Generated SARIF objects
|
||||
|
||||
This details each SARIF component that may be generated, along with any specific circumstances. We omit any properties that are never generated.
|
||||
|
||||
### `sarifLog` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-------------------|------------------------|---------|
|
||||
| `$schema`| Always| Provides a link to the [SARIF schema](https://github.com/oasis-tcs/sarif-spec/blob/main/Schemata/sarif-schema-2.1.0.json).|
|
||||
| `version`| Always| The version of the SARIF used to generate the output.|
|
||||
| `runs`| Always| An array containing a single run object, for one language.|
|
||||
### `run` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-------------------|------------------------|---------|
|
||||
| `tool`| Always| –|
|
||||
| `originalUriBaseIds`| Always| A dictionary of `uriBaseIds` to artifactLocations representing the original locations on the analysis machine. At a minimum, this will contain the `%SRCROOT%` `uriBaseId`, which represents the root location on the analysis machine of the source code for the analyzed project. Each `artifactLocation` will contain the `uri` and `description` properties.|
|
||||
| `artifacts`| Always| An array containing at least one artifact object for every file referenced in a result.|
|
||||
| `results`| Always| –|
|
||||
| `newLineSequences`| Always| –|
|
||||
| `columnKind`| Always| –|
|
||||
| `properties`| Always| The properties dictionary will contain the `semmle.formatSpecifier`, which identifies the format specifier passed to the {% data variables.product.prodname_codeql_cli %}.|
|
||||
|
||||
### `tool` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-------------------|------------------------|---------|
|
||||
| `driver`| Always| –|
|
||||
|
||||
### `toolComponent` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|---------------------|-----|
|
||||
| `name`| Always| Set to “{% data variables.product.prodname_codeql %} command-line toolchain” for output from the {% data variables.product.prodname_codeql_cli %} tools. Note, if the output was generated using a different tool a different `name` is reported, and the format may not be as described here.|
|
||||
| `organization`| Always| Set to “GitHub”.|
|
||||
| `version`| Always| Set to the {% data variables.product.prodname_codeql %} release version e.g. “2.0.0”.|
|
||||
| `rules`| Always| An array of `reportingDescriptor` objects that represent rules. This array will contain, at a minimum, all the rules that were run during this analysis, but may contain rules which were available but not run. For more detail about enabling queries, see `defaultConfiguration`.|
|
||||
|
||||
### `reportingDescriptor` object (for rule)
|
||||
|
||||
`reportingDescriptor` objects may be used in multiple places in the SARIF specification. When a `reportingDescriptor` is included in the rules array of a `toolComponent` object it has the following properties.
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `id`| Always| Will contain the `@id` property specified in the query that defines the rule, which is usually of the format `language/rule-name` (for example `cpp/unsafe-format-string`). If your organization defines the `@opaqueid` property in the query it will be used instead.|
|
||||
| `name`| Always| Will contain the `@id` property specified in the query. See the `id` property above for an example.|
|
||||
| `shortDescription`| Always| Will contain the `@name` property specified in the query that defines the rule.|
|
||||
| `fullDescription`| Always| Will contain the `@description` property specified in the query that defines the rule.|
|
||||
| `defaultConfiguration`| Always| A `reportingConfiguration` object, with the enabled property set to true or false, and a level property set according to the `@severity` property specified in the query that defines the rule. Omitted if the `@severity` property was not specified.|
|
||||
|
||||
### `artifact` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `location`| Always| An `artifactLocation` object.|
|
||||
| `index`| Always| The index of the `artifact` object.|
|
||||
| `contents`| Optionally| If results are generated using the `--sarif-add-file-contents` flag, and the source code is available at the time the SARIF file is generated, then the `contents` property is populated with an `artifactContent` object, with the `text` property set.|
|
||||
|
||||
### `artifactLocation` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `uri`| Always| –|
|
||||
| `index`| Always| –|
|
||||
| `uriBaseId`| Optionally| If the file is relative to some known abstract location, such as the root source location on the analysis machine, this will be set.|
|
||||
|
||||
### `result` object
|
||||
|
||||
The composition of the results is dependent on the options provided to CodeQL. By default, the results are grouped by unique message format string and primary location. Thus, two results that occur at the same location with the same underlying message, will appear as a single result in the output. This behavior can be disabled by using the flag `--ungroup-results`, in which case no results are grouped.
|
||||
|
||||
| JSON property name | When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `ruleId`| Always| See the description of the `id` property in `reportingDescriptor` object (for rule) .|
|
||||
| `ruleIndex`| Always| –|
|
||||
| `message`| Always| A message describing the problem(s) occurring at this location. This message may be a SARIF “Message with placeholder”, containing links that refer to locations in the `relatedLocations` property.|
|
||||
| `locations`| Always| An array containing a single `location` object.|
|
||||
| `partialFingerprints`| Always| A dictionary from named fingerprint types to the fingerprint. This will contain, at a minimum, a value for the `primaryLocationLineHash`, which provides a fingerprint based on the context of the primary location.|
|
||||
| `codeFlows`| Optionally| This array may be populated with one or more `codeFlow` objects if the query that defines the rule for this result is of `@kind path-problem`.|
|
||||
| `relatedLocations`| Optionally| This array will be populated if the query that defines the rule for this result has a message with placeholder options. Each unique location is included once.|
|
||||
| `suppressions`| Optionally| If the result is suppressed, then this will contain a single `suppression` object, with the `@kind` property set to `IN_SOURCE`. If this result is not suppressed, but there is at least one result that has a suppression, then this will be set to an empty array, otherwise it will not be set.|
|
||||
|
||||
### `location` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `physicalLocation`| Always| –|
|
||||
| `id`| Optionally| `location` objects that appear in the `relatedLocations` array of a `result` object may contain the `id` property.|
|
||||
| `message`| Optionally| `location` objects may contain the `message` property if:</br></br>- They appear in the `relatedLocations` array of a `result` object may contain the `message` property.</br></br>- They appear in the `threadFlowLocation.location` property.|
|
||||
|
||||
### `physicalLocation` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `artifactLocation`| Always| –|
|
||||
| `region`| Optionally| If the given `physicalLocation` exists in a text file, such as a source code file, then the `region` property may be present.|
|
||||
| `contextRegion`| Optionally| May be present if this location has an associated `snippet`.|
|
||||
|
||||
### `region` object
|
||||
|
||||
There are two types of `region` object produced by {% data variables.product.prodname_codeql %}:
|
||||
|
||||
* Line/column offset regions
|
||||
|
||||
* Character offset and length regions
|
||||
|
||||
Any region produced by {% data variables.product.prodname_codeql %} may be specified in either format, and consumers should robustly handle either type.
|
||||
|
||||
For line/column offset regions, the following properties will be set:
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `startLine`| Always| –|
|
||||
| `startColumn`| Optionally| Not included if equal to the default value of 1.|
|
||||
| `endLine`| Optionally| Not included if identical to `startLine`.|
|
||||
| `endColumn`| Always| –|
|
||||
| `snippet`| Optionally| –|
|
||||
|
||||
For character offset and length regions, the following properties will be set:
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `charOffset`| Optionally| Provided if `startLine`, `startColumn`, `endLine`, and `endColumn` are not populated.|
|
||||
| `charLength`| Optionally| Provided if `startLine`, `startColumn`, `endLine`, and `endColumn` are not populated.|
|
||||
| `snippet`| Optionally| –|
|
||||
|
||||
### `codeFlow` object
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `threadFlows`| Always| –|
|
||||
|
||||
### `threadFlow` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `locations`| Always| –|
|
||||
|
||||
### `threadFlowLocation` object
|
||||
|
||||
| JSON property name| When is this generated?| Notes|
|
||||
|-----------------------|--------------------|------|
|
||||
| `location`| Always| –|
|
||||
@@ -19,6 +19,8 @@ topics:
|
||||
children:
|
||||
- /automatically-scanning-your-code-for-vulnerabilities-and-errors
|
||||
- /integrating-with-code-scanning
|
||||
- /using-the-codeql-cli
|
||||
- /codeql-cli-reference
|
||||
- /using-codeql-code-scanning-with-your-existing-ci-system
|
||||
---
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@ You can display the command-line help for any command using the <nobr>`--help`</
|
||||
1. Check out the code that you want to analyze:
|
||||
- For a branch, check out the head of the branch that you want to analyze.
|
||||
- For a pull request, check out either the head commit of the pull request, or check out a {% data variables.product.prodname_dotcom %}-generated merge commit of the pull request.
|
||||
2. Set up the environment for the codebase, making sure that any dependencies are available. For more information, see [Creating databases for non-compiled languages](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-non-compiled-languages) and [Creating databases for compiled languages](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-compiled-languages) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
2. Set up the environment for the codebase, making sure that any dependencies are available. For more information, see "[Creating databases for non-compiled languages](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases#creating-databases-for-non-compiled-languages)" and "[Creating databases for compiled languages](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases#creating-databases-for-compiled-languages)."
|
||||
3. Find the build command, if any, for the codebase. Typically this is available in a configuration file in the CI system.
|
||||
4. Run `codeql database create` from the checkout root of your repository and build the codebase.
|
||||
|
||||
@@ -81,7 +81,7 @@ You can display the command-line help for any command using the <nobr>`--help`</
|
||||
| <nobr>`--source-root`</nobr> | | Optional. Use if you run the CLI outside the checkout root of the repository. By default, the `database create` command assumes that the current directory is the root directory for the source files, use this option to specify a different location. |
|
||||
| <nobr>`--codescanning-config`</nobr> | | Optional (Advanced). Use if you have a configuration file that specifies how to create the {% data variables.product.prodname_codeql %} databases and what queries to run in later steps. For more information, see "[Customizing {% data variables.product.prodname_code_scanning %}](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#using-a-custom-configuration-file)" and "[database create](https://codeql.github.com/docs/codeql-cli/manual/database-create/#cmdoption-codeql-database-create-codescanning-config)." |
|
||||
|
||||
For more information, see [Creating {% data variables.product.prodname_codeql %} databases](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
For more information, see "[Creating {% data variables.product.prodname_codeql %} databases](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases/)."
|
||||
|
||||
### Single language example
|
||||
|
||||
@@ -155,17 +155,17 @@ codeql database analyze <database> --format=<format> \
|
||||
| Option | Required | Usage |
|
||||
|--------|:--------:|-----|
|
||||
| `<database>` | {% octicon "check-circle-fill" aria-label="Required" %} | Specify the path for the directory that contains the {% data variables.product.prodname_codeql %} database to analyze. |
|
||||
| `<packs,queries>` | | Specify {% data variables.product.prodname_codeql %} packs or queries to run. To run the standard queries used for {% data variables.product.prodname_code_scanning %}, omit this parameter. To see the other query suites included in the {% data variables.product.prodname_codeql_cli %} bundle, look in `/<extraction-root>/qlpacks/codeql/<language>-queries/codeql-suites`. For information about creating your own query suite, see [Creating CodeQL query suites](https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
| `<packs,queries>` | | Specify {% data variables.product.prodname_codeql %} packs or queries to run. To run the standard queries used for {% data variables.product.prodname_code_scanning %}, omit this parameter. To see the other query suites included in the {% data variables.product.prodname_codeql_cli %} bundle, look in `/<extraction-root>/qlpacks/codeql/<language>-queries/codeql-suites`. For information about creating your own query suite, see [Creating CodeQL query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites/) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
| <nobr>`--format`</nobr> | {% octicon "check-circle-fill" aria-label="Required" %} | Specify the format for the results file generated by the command. For upload to {% data variables.product.company_short %} this should be: {% ifversion fpt or ghae or ghec %}`sarif-latest`{% else %}`sarifv2.1.0`{% endif %}. For more information, see "[SARIF support for {% data variables.product.prodname_code_scanning %}](/code-security/secure-coding/sarif-support-for-code-scanning)."
|
||||
| <nobr>`--output`</nobr> | {% octicon "check-circle-fill" aria-label="Required" %} | Specify where to save the SARIF results file.
|
||||
| <nobr>`--sarif-category`<nobr> | {% octicon "question" aria-label="Required with multiple results sets" %} | Optional for single database analysis. Required to define the language when you analyze multiple databases for a single commit in a repository. Specify a category to include in the SARIF results file for this analysis. A category is used to distinguish multiple analyses for the same tool and commit, but performed on different languages or different parts of the code.|{% ifversion fpt or ghes > 3.3 or ghae or ghec %}
|
||||
| <nobr>`--sarif-add-query-help`</nobr> | | Optional. Use if you want to include any available markdown-rendered query help for custom queries used in your analysis. Any query help for custom queries included in the SARIF output will be displayed in the code scanning UI if the relevant query generates an alert. For more information, see [Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](https://codeql.github.com/docs/codeql-cli/analyzing-databases-with-the-codeql-cli/#including-query-help-for-custom-codeql-queries-in-sarif-files) in the documentation for the {% data variables.product.prodname_codeql_cli %}.{% endif %}{% ifversion codeql-packs %}
|
||||
| <nobr>`--sarif-add-query-help`</nobr> | | Optional. Use if you want to include any available markdown-rendered query help for custom queries used in your analysis. Any query help for custom queries included in the SARIF output will be displayed in the code scanning UI if the relevant query generates an alert. For more information, see "[Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli/#including-query-help-for-custom-codeql-queries-in-sarif-files)."{% endif %}{% ifversion codeql-packs %}
|
||||
| `<packs>` | | Optional. Use if you want to include CodeQL query packs in your analysis. For more information, see "[Downloading and using {% data variables.product.prodname_codeql %} packs](#downloading-and-using-codeql-query-packs)."
|
||||
| <nobr>`--download`</nobr> | | Optional. Use if some of your CodeQL query packs are not yet on disk and need to be downloaded before running queries.{% endif %}
|
||||
| <nobr>`--threads`</nobr> | | Optional. Use if you want to use more than one thread to run queries. The default value is `1`. You can specify more threads to speed up query execution. To set the number of threads to the number of logical processors, specify `0`.
|
||||
| <nobr>`--verbose`</nobr> | | Optional. Use to get more detailed information about the analysis process and diagnostic data from the database creation process.
|
||||
|
||||
For more information, see [Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](https://codeql.github.com/docs/codeql-cli/analyzing-databases-with-the-codeql-cli/) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
|
||||
For more information, see [Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli/)."
|
||||
|
||||
### Basic example
|
||||
|
||||
@@ -238,6 +238,16 @@ Before you can use a {% data variables.product.prodname_codeql %} pack to analyz
|
||||
| <nobr>`<scope/name@version:path>`</nobr> | {% octicon "check-circle-fill" aria-label="Required" %} | Specify the scope and name of one or more CodeQL query packs to download using a comma-separated list. Optionally, include the version to download and unzip. By default the latest version of this pack is downloaded. Optionally, include a path to a query, directory, or query suite to run. If no path is included, then run the default queries of this pack. |
|
||||
| <nobr>`--github-auth-stdin`</nobr> | | Optional. Pass the {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} created for authentication with {% data variables.product.company_short %}'s REST API to the CLI via standard input. This is not needed if the command has access to a `GITHUB_TOKEN` environment variable set with this token.
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you specify a particular version of a query pack to use, be aware that the version you specify may eventually become too old for the latest version of {% data variables.product.prodname_codeql %} to make efficient use of. To ensure optimal performance, if you need to specify exact query pack versions, you should reevaluate which versions you pin to whenever you upgrade the {% data variables.product.prodname_codeql %} CLI you're using.
|
||||
|
||||
For more information about pack compatibility, see "[About {% data variables.product.prodname_codeql %} pack compatibility](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs#about-codeql-pack-compatibility)."
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
### Basic example
|
||||
|
||||
This example runs the `codeql database analyze` command with the `--download` option to:
|
||||
@@ -342,6 +352,6 @@ If you use the {% data variables.product.prodname_codeql_cli %} to run {% data v
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Creating CodeQL databases](https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/)
|
||||
- [Analyzing databases with the CodeQL CLI](https://codeql.github.com/docs/codeql-cli/analyzing-databases-with-the-codeql-cli/)
|
||||
- [Publishing and using CodeQL packs](https://codeql.github.com/docs/codeql-cli/publishing-and-using-codeql-packs/)
|
||||
- [Creating CodeQL databases](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases/)
|
||||
- [Analyzing databases with the CodeQL CLI](/code-security/code-scanning/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli/){% ifversion codeql-packs %}
|
||||
- [Publishing and using CodeQL packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs){% endif %}
|
||||
|
||||
@@ -48,13 +48,13 @@ The {% data variables.code-scanning.codeql_runner %} is a command-line tool that
|
||||
|
||||
**Note:**
|
||||
{% ifversion fpt or ghec %}
|
||||
* The {% data variables.code-scanning.codeql_runner %} uses the {% data variables.product.prodname_codeql %} CLI to analyze code and therefore has the same license conditions. It's free to use on public repositories that are maintained on {% data variables.product.prodname_dotcom_the_website %}, and available to use on private repositories that are owned by customers with an {% data variables.product.prodname_advanced_security %} license. For information, see "[{% data variables.product.product_name %} {% data variables.product.prodname_codeql %} Terms and Conditions](https://securitylab.github.com/tools/codeql/license)" and "[{% data variables.product.prodname_codeql %} CLI](https://codeql.github.com/docs/codeql-cli/)."
|
||||
* The {% data variables.code-scanning.codeql_runner %} uses the {% data variables.product.prodname_codeql %} CLI to analyze code and therefore has the same license conditions. It's free to use on public repositories that are maintained on {% data variables.product.prodname_dotcom_the_website %}, and available to use on private repositories that are owned by customers with an {% data variables.product.prodname_advanced_security %} license. For information, see "[{% data variables.product.product_name %} {% data variables.product.prodname_codeql %} Terms and Conditions](https://securitylab.github.com/tools/codeql/license)" and "[{% data variables.product.prodname_codeql %} CLI](/code-security/code-scanning/using-the-codeql-cli/)."
|
||||
{% else %}
|
||||
* The {% data variables.code-scanning.codeql_runner %} is available to customers with an {% data variables.product.prodname_advanced_security %} license.
|
||||
{% endif %}
|
||||
{% ifversion ghae %}
|
||||
* The {% data variables.code-scanning.codeql_runner %} shouldn't be confused with the {% data variables.product.prodname_codeql %} CLI. The {% data variables.product.prodname_codeql %} CLI is a command-line interface that lets you create {% data variables.product.prodname_codeql %} databases for security research and run {% data variables.product.prodname_codeql %} queries.
|
||||
For more information, see "[{% data variables.product.prodname_codeql_cli %}](https://codeql.github.com/docs/codeql-cli/)."
|
||||
For more information, see "[{% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/)."
|
||||
{% endif %}
|
||||
{% endnote %}
|
||||
|
||||
@@ -89,7 +89,7 @@ chmod +x codeql-runner-linux
|
||||
In addition to this, each CI server also needs:
|
||||
|
||||
- A {% data variables.product.prodname_github_app %} or {% data variables.product.pat_generic %} for the {% data variables.code-scanning.codeql_runner %} to use. You must use an access token with the `repo` scope, or a {% data variables.product.prodname_github_app %} with the `security_events` write permission, and `metadata` and `contents` read permissions. For information, see "[Building {% data variables.product.prodname_github_apps %}](/developers/apps/building-github-apps)" and "[Creating a {% data variables.product.pat_generic %}](/github/authenticating-to-github/creating-a-personal-access-token)."
|
||||
- Access to the {% data variables.product.prodname_codeql %} bundle associated with this release of the {% data variables.code-scanning.codeql_runner %}. This package contains queries and libraries needed for {% data variables.product.prodname_codeql %} analysis, plus the {% data variables.product.prodname_codeql %} CLI, which is used internally by the runner. For information, see "[{% data variables.product.prodname_codeql %} CLI](https://codeql.github.com/docs/codeql-cli/)."
|
||||
- Access to the {% data variables.product.prodname_codeql %} bundle associated with this release of the {% data variables.code-scanning.codeql_runner %}. This package contains queries and libraries needed for {% data variables.product.prodname_codeql %} analysis, plus the {% data variables.product.prodname_codeql %} CLI, which is used internally by the runner. For information, see "[{% data variables.product.prodname_codeql %} CLI](/code-security/code-scanning/using-the-codeql-cli/)."
|
||||
|
||||
The options for providing access to the {% data variables.product.prodname_codeql %} bundle are:
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: About the CodeQL CLI
|
||||
shortTitle: About the CodeQL CLI
|
||||
intro: 'You can use the {% data variables.product.prodname_codeql_cli %} to run CodeQL processes locally on software projects.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
Software developers and security researchers can secure their code
|
||||
using the {% data variables.product.prodname_codeql_cli %}.
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} is a command-line tool used to run {% data variables.product.prodname_codeql %} processes locally on
|
||||
open source software projects. You can use the {% data variables.product.prodname_codeql_cli %} to:
|
||||
|
||||
- Run {% data variables.product.prodname_codeql %} analyses using queries provided by {% data variables.product.prodname_dotcom %} engineers and the open
|
||||
source community
|
||||
- Create {% data variables.product.prodname_codeql %} databases to use in the {% data variables.product.prodname_codeql %} for Visual Studio Code
|
||||
- Develop and test custom {% data variables.product.prodname_codeql %} queries to use in your own analyses
|
||||
|
||||
For information about using the {% data variables.product.prodname_codeql_cli %}, see
|
||||
“[Getting started with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/getting-started-with-the-codeql-cli).”
|
||||
|
||||
## About the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} license
|
||||
|
||||
**License notice:** If you don’t have a {% data variables.product.prodname_enterprise %} license then, by installing this product, you are agreeing to the [{% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} Terms and Conditions](https://securitylab.github.com/tools/codeql/license).
|
||||
|
||||
{% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} is licensed on a per-user basis. Under the license restrictions, you can use {% data variables.product.prodname_codeql %} to perform the following tasks:
|
||||
|
||||
- To perform academic research.
|
||||
- To demonstrate the software.
|
||||
- To test {% data variables.product.prodname_codeql %} queries that are released under an OSI-approved License to confirm that new versions of those queries continue to find the right vulnerabilities.
|
||||
|
||||
Where “OSI-approved License” means an Open Source Initiative (OSI)-approved open source software license.
|
||||
|
||||
If you are working with an Open Source Codebase (that is, a codebase that is released under an OSI-approved License) you can also use {% data variables.product.prodname_codeql %} for the following tasks:
|
||||
|
||||
- To perform analysis of the Open Source Codebase.
|
||||
- If the Open Source Codebase is hosted and maintained on {% data variables.product.prodname_dotcom_the_website %}, to generate CodeQL databases for or during automated analysis, continuous integration, or continuous delivery.
|
||||
|
||||
{% data variables.product.prodname_codeql %} can’t be used for automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise, except in the express cases set forth herein. For these uses, contact the [sales team](https://enterprise.github.com/contact).
|
||||
|
||||
## {% data variables.product.prodname_codeql_cli %} commands
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} includes commands to create and analyze {% data variables.product.prodname_codeql %} databases from the
|
||||
command line. To run a command, use:
|
||||
|
||||
```
|
||||
codeql [command] [subcommand]
|
||||
```
|
||||
|
||||
To view the reference documentation for a command, add the `--help` flag, or visit the
|
||||
"[{% data variables.product.prodname_codeql_cli %} manual](https://codeql.github.com/docs/codeql-cli/manual/)."
|
||||
@@ -0,0 +1,422 @@
|
||||
---
|
||||
title: Analyzing databases with the CodeQL CLI
|
||||
shortTitle: Analyzing databases
|
||||
intro: 'You can run queries against a {% data variables.product.prodname_codeql %} database extracted from a codebase.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About analyzing databases with the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
To analyze a codebase, you run queries against a CodeQL
|
||||
database extracted from the code.
|
||||
|
||||
{% data variables.product.prodname_codeql %} analyses produce [interpreted results](https://codeql.github.com/docs/codeql-overview/about-codeql/#interpret-query-results) that can be displayed as alerts or paths in source code.
|
||||
For information about writing queries to run with `database analyze`, see “[Using custom queries with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/using-custom-queries-with-the-codeql-cli).”
|
||||
|
||||
{% note %}
|
||||
|
||||
**Other query-running commands**
|
||||
|
||||
Queries run with `database analyze` have strict [metadata requirements](/code-security/code-scanning/using-custom-queries-with-the-codeql-cli/#including-query-metadata). You can also execute queries using the following
|
||||
plumbing-level subcommands:
|
||||
|
||||
* [`database run-queries`](https://codeql.github.com/docs/codeql-cli/manual/database-run-queries/), which
|
||||
outputs non-interpreted results in an intermediate binary format called [BQRS](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#bqrs-file)
|
||||
|
||||
* [`query run`](https://codeql.github.com/docs/codeql-cli/manual/query-run/), which will output BQRS files, or print
|
||||
results tables directly to the command line. Viewing results directly in
|
||||
the command line may be useful for iterative query development using the CLI.
|
||||
|
||||
Queries run with these commands don't have the same metadata requirements.
|
||||
However, to save human-readable data you have to process each BQRS results
|
||||
file using the [`bqrs decode`](https://codeql.github.com/docs/codeql-cli/manual/bqrs-decode/) plumbing
|
||||
subcommand. Therefore, for most use cases it's easiest to use `database
|
||||
analyze` to directly generate interpreted results.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
Before starting an analysis you must:
|
||||
|
||||
* [Set up the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli//getting-started-with-the-codeql-cli) to run commands locally.
|
||||
* [Create a {% data variables.product.prodname_codeql %} database](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases) for the source code you want to analyze.
|
||||
|
||||
The simplest way to run `codeql database analyze` is using {% data variables.product.prodname_codeql %} packs. You can also
|
||||
run the command using queries from a local checkout of the {% data variables.product.prodname_codeql %} repository,
|
||||
which you may want to do if you want to customize the {% data variables.product.prodname_codeql %} core queries.
|
||||
|
||||
## Running `codeql database analyze`
|
||||
|
||||
When you run `database analyze`, it:
|
||||
|
||||
1. Optionally downloads any referenced {% data variables.product.prodname_codeql %} packages that are not available locally.
|
||||
2. Executes one or more query files, by running them over a {% data variables.product.prodname_codeql %} database.
|
||||
3. Interprets the results, based on certain query metadata, so that alerts can be
|
||||
displayed in the correct location in the source code.
|
||||
4. Reports the results of any diagnostic and summary queries to standard output.
|
||||
|
||||
You can analyze a database by running the following command:
|
||||
|
||||
```
|
||||
codeql database analyze <database> --format=<format> --output=<output> <query-specifiers>...
|
||||
```
|
||||
|
||||
You must specify:
|
||||
|
||||
|
||||
* `<database>`: the path to the {% data variables.product.prodname_codeql %} database you want to analyze.
|
||||
* `--format`: the format of the results file generated during analysis. A
|
||||
number of different formats are supported, including CSV, [SARIF](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#sarif-file), and graph formats. For more information about CSV and SARIF,
|
||||
see [Results](#results). To find out which other results formats are
|
||||
supported, see the [database analyze reference](https://codeql.github.com/docs/codeql-cli/manual/database-analyze/).
|
||||
* `--output`: the output path of the results file generated during analysis.
|
||||
|
||||
You can also specify:
|
||||
|
||||
* `<query-specifiers>...`: a space-separated list of queries to run over your database. This
|
||||
is a list of arguments, where each argument can be:
|
||||
* a path to a query file
|
||||
* a path to a directory containing query files
|
||||
* a path to a query suite file
|
||||
* the name of a {% data variables.product.prodname_codeql %} query pack
|
||||
* with an optional version range
|
||||
* with an optional path to a query, directory, or query suite inside the pack
|
||||
|
||||
If omitted, the default query suite for the language of the analyzed database will be used. For the complete syntax of query specifiers, see “Specifying which queries to run in a {% data variables.product.prodname_codeql %} pack”.
|
||||
|
||||
* `--sarif-category`: an identifying category for the results. Used when
|
||||
you want to upload more than one set of results for a commit.
|
||||
For example, when you use `github upload-results` to send results for more than one
|
||||
language to the {% data variables.product.prodname_dotcom %} code scanning API. For more information about this use case, see [Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system).
|
||||
|
||||
* `--sarif-add-query-help`: (supported in version 2.7.1 onwards) adds any custom query help written
|
||||
in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in `.qhelp` files must be
|
||||
converted to `.md` before running the analysis. For further information,
|
||||
see “Including query help for custom {% data variables.product.prodname_codeql %} queries in SARIF files.”
|
||||
|
||||
* `--download`: a boolean flag that will allow the CLI to download any referenced {% data variables.product.prodname_codeql %} packages that are not available locally.
|
||||
If this flag is missing and a referenced {% data variables.product.prodname_codeql %} package is not available locally, the command will fail.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Upgrading databases**
|
||||
|
||||
For databases that were created by {% data variables.product.prodname_codeql_cli %} v2.3.3 or earlier, you will need to explicitly upgrade the database before you can run an analysis with a newer
|
||||
version of the {% data variables.product.prodname_codeql_cli %}. If this step is necessary, then you will see a message telling you
|
||||
that your database needs to be upgraded when you run `database analyze`.
|
||||
|
||||
For databases that were created by {% data variables.product.prodname_codeql_cli %} v2.3.4 or later, the CLI will implicitly run any
|
||||
required upgrades. Explicitly running the upgrade command is not necessary.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
For full details of all the options you can use when analyzing databases, see
|
||||
the [database analyze reference documentation](https://codeql.github.com/docs/codeql-cli/manual/database-analyze/).
|
||||
|
||||
## Specifying which queries to run in a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
Query specifiers are used by `codeql database analyze` and other commands that operate on a set of queries.
|
||||
The complete form of a query specifier is `scope/name@range:path`, where:
|
||||
|
||||
|
||||
* `scope/name` is the qualified name of a {% data variables.product.prodname_codeql %} pack.
|
||||
|
||||
|
||||
* `range` is a [semver range](https://docs.npmjs.com/cli/v6/using-npm/semver#ranges).
|
||||
|
||||
|
||||
* `path` is a file system path to a single query, a directory containing queries, or a query suite file.
|
||||
|
||||
When you specify a `scope/name`, the `range` and `path` are
|
||||
optional. If you omit a `range` then the latest version of the
|
||||
specified pack is used. If you omit a `path` then the default query suite
|
||||
of the specified pack is used.
|
||||
|
||||
The `path` can be one of: a `.ql` query file, a directory
|
||||
containing one or more queries, or a `.qls` query suite file. If
|
||||
you omit a pack name, then you must provide a `path`,
|
||||
which will be interpreted relative to the working directory
|
||||
of the current process. Glob patterns are not supported.
|
||||
|
||||
If you specify both a `scope/name` and `path`, then the `path` cannot
|
||||
be absolute. It is considered relative to the root of the {% data variables.product.prodname_codeql %}
|
||||
pack.
|
||||
|
||||
### Example query specifiers
|
||||
|
||||
|
||||
* `codeql/python-queries` - All the queries in the default query suite of the latest version of the `codeql/python-queries` pack.
|
||||
|
||||
* `codeql/python-queries@1.2.3` - All the queries in the default query suite of version `1.2.3` of the `codeql/python-queries` pack.
|
||||
|
||||
* `codeql/python-queries@~1.2.3` - All the queries in the default query suite of the latest version of the `codeql/python-queries` pack that is >= `1.2.3` and < `1.3.0`.
|
||||
|
||||
* `codeql/python-queries:Functions` - All queries in the `Functions` directory in the latest version of the `codeql/python-queries` pack.
|
||||
|
||||
* `codeql/python-queries@1.2.3:Functions` - All queries in the `Functions` directory in version 1.2.3 of the `codeql/python-queries` pack.
|
||||
|
||||
* `codeql/python-queries@1.2.3:codeql-suites/python-code-scanning.qls` - All queries in the `codeql-suites/python-code-scanning.qls` directory in version 1.2.3 of the `codeql/python-queries` pack.
|
||||
|
||||
* `suites/my-suite.qls` - All queries in the `suites/my-suite.qls` file relative to the current working directory.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Tip**
|
||||
|
||||
The default query suite of the standard {% data variables.product.prodname_codeql %} query packs are `codeql-suites/<lang>-code-scanning.qls`. Several other useful query suites can also be found in the `codeql-suites` directory of each pack. For example, the `codeql/cpp-queries` pack contains the following query suites:
|
||||
|
||||
* `cpp-code-scanning.qls` - Standard Code Scanning queries for C++. The default query suite for this pack.
|
||||
|
||||
* `cpp-security-extended.qls` - Queries from the default `cpp-code-scanning.qls` suite for C++, plus lower severity and precision queries.
|
||||
|
||||
* `cpp-security-and-quality.qls` - Queries from `cpp-security-extended.qls`, plus maintainability and reliability queries.
|
||||
|
||||
You can see the sources for these query suites in the [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql/tree/main/cpp/ql/src/codeql-suites). Query suites for other languages are similar.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
## Examples of running database analyses
|
||||
|
||||
The following examples show how to run `database analyze` using {% data variables.product.prodname_codeql %} packs, and how to use a local checkout of the {% data variables.product.prodname_codeql %} repository. These examples assume your {% data variables.product.prodname_codeql %} databases have been created in a directory that is a sibling of your local copies of the {% data variables.product.prodname_codeql %} repository.
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
### Running a {% data variables.product.prodname_codeql %} query pack
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note**
|
||||
|
||||
The {% data variables.product.prodname_codeql %} package management functionality, including {% data variables.product.prodname_codeql %} packs, is currently available as a beta release and is subject to change. During the beta release, {% data variables.product.prodname_codeql %} packs are available only using {% data variables.product.prodname_registry %} - the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}. To use this beta functionality, install the latest version of the {% data variables.product.prodname_codeql_cli %} bundle from: https://github.com/github/codeql-action/releases.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
To run an existing {% data variables.product.prodname_codeql %} query pack from the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}, you can specify one or more
|
||||
pack names:
|
||||
|
||||
```
|
||||
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
|
||||
```
|
||||
|
||||
This command runs the default query suite of two {% data variables.product.prodname_codeql %} query packs: `microsoft/coding-standards` version 1.0.0 and the latest version of `github/security-queries` on the specified database. For further information about default suites, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases/publishing-and-using-codeql-packs/#publishing-and-using-codeql-packs)”.
|
||||
|
||||
The `--download` flag is optional. Using it will ensure the query pack is downloaded if it isn’t yet available locally.
|
||||
{% endif %}
|
||||
|
||||
### Running a single query
|
||||
|
||||
To run a single query over a {% data variables.product.prodname_codeql %} database for a JavaScript codebase,
|
||||
you could use the following command from the directory containing your database:
|
||||
|
||||
```
|
||||
codeql database analyze --download <javascript-database> codeql/javascript-queries:Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
|
||||
```
|
||||
|
||||
This command runs a simple query that finds potential bugs related to unused
|
||||
variables, imports, functions, or classes—it is one of the JavaScript
|
||||
queries included in the {% data variables.product.prodname_codeql %} repository. You could run more than one query by
|
||||
specifying a space-separated list of similar paths.
|
||||
|
||||
The analysis generates a CSV file (`js-results.csv`) in a new directory (`js-analysis`).
|
||||
|
||||
Alternatively, if you have the {% data variables.product.prodname_codeql %} repository checked out, you can execute the same queries by specifying the path to the query directly:
|
||||
|
||||
```
|
||||
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
|
||||
```
|
||||
|
||||
You can also run your own custom queries with the `database analyze` command.
|
||||
For more information about preparing your queries to use with the {% data variables.product.prodname_codeql_cli %},
|
||||
see “[Using custom queries with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/using-custom-queries-with-the-codeql-cli).”
|
||||
|
||||
### Running all queries in a directory
|
||||
|
||||
You can run all the queries located in a directory by providing the directory
|
||||
path, rather than listing all the individual query files. Paths are searched
|
||||
recursively, so any queries contained in subfolders will also be executed.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Important**
|
||||
|
||||
You should avoid specifying the root of a core {% data variables.product.prodname_codeql %} query pack when executing `database analyze`
|
||||
as it might contain some special queries that aren’t designed to be used with
|
||||
the command. Rather, run the query pack to include the
|
||||
pack’s default queries in the analysis, or run one of the
|
||||
code scanning query suites.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
For example, to execute all Python queries contained in the `Functions` directory in the
|
||||
`codeql/python-queries` query pack you would run:
|
||||
|
||||
```
|
||||
codeql database analyze <python-database> codeql/python-queries:Functions --format=sarif-latest --output=python-analysis/python-results.sarif --download
|
||||
```
|
||||
|
||||
Alternatively, if you have the {% data variables.product.prodname_codeql %} repository checked out, you can execute the
|
||||
same queries by specifying the path to the directory directly:
|
||||
|
||||
```
|
||||
codeql database analyze <python-database> ../ql/python/ql/src/Functions/ --format=sarif-latest --output=python-analysis/python-results.sarif
|
||||
```
|
||||
|
||||
When the analysis has finished, a SARIF results file is generated. Specifying `--format=sarif-latest` ensures
|
||||
that the results are formatted according to the most recent SARIF specification
|
||||
supported by {% data variables.product.prodname_codeql %}.
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
### Running a subset of queries in a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
If you are using {% data variables.product.prodname_codeql_cli %} v2.8.1 or later, you can include a path at the end of a pack specification to run a subset of queries inside the pack. This applies to any command that locates or runs queries within a pack.
|
||||
|
||||
The complete way to specify a set of queries is in the form `scope/name@range:path`, where:
|
||||
|
||||
|
||||
* `scope/name` is the qualified name of a {% data variables.product.prodname_codeql %} pack.
|
||||
|
||||
|
||||
* `range` is a [semver range](https://docs.npmjs.com/cli/v6/using-npm/semver#ranges).
|
||||
|
||||
|
||||
* `path` is a file system path to a single query, a directory containing queries, or a query suite file.
|
||||
|
||||
When you specify a `scope/name`, the `range` and `path` are
|
||||
optional. If you omit a `range` then the latest version of the
|
||||
specified pack is used. If you omit a `path` then the default query suite
|
||||
of the specified pack is used.
|
||||
|
||||
The `path` can be one of a `\*.ql` query file, a directory
|
||||
containing one or more queries, or a `.qls` query suite file. If
|
||||
you omit a pack name, then you must provide a `path`,
|
||||
which will be interpreted relative to the working directory
|
||||
of the current process.
|
||||
|
||||
If you specify a `scope/name` and `path`, then the `path` cannot
|
||||
be absolute. It is considered relative to the root of the {% data variables.product.prodname_codeql %}
|
||||
pack.
|
||||
|
||||
To analyze a database using all queries in the `experimental/Security` folder within the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} pack you can use:
|
||||
|
||||
```
|
||||
codeql database analyze --format=sarif-latest --output=results <db> \
|
||||
codeql/cpp-queries:experimental/Security
|
||||
```
|
||||
|
||||
To run the `RedundantNullCheckParam.ql` query in the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} pack use:
|
||||
|
||||
```
|
||||
codeql database analyze --format=sarif-latest --output=results <db> \
|
||||
'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql'
|
||||
```
|
||||
|
||||
To analyze your database using the `cpp-security-and-quality.qls` query suite from a version of the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} pack that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen) you can use:
|
||||
|
||||
```
|
||||
codeql database analyze --format=sarif-latest --output=results <db> \
|
||||
'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'
|
||||
```
|
||||
|
||||
If you need to reference a query file, directory, or suite whose path contains a literal `@` or `:`, you can prefix the query specification with path: like so:
|
||||
|
||||
```
|
||||
codeql database analyze --format=sarif-latest --output=results <db> \
|
||||
path:C:/Users/ci/workspace@2/security/query.ql
|
||||
```
|
||||
|
||||
For more information about {% data variables.product.prodname_codeql %} packs, see [About {% data variables.product.prodname_codeql %} Packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).
|
||||
{% endif %}
|
||||
|
||||
### Running query suites
|
||||
|
||||
To run a query suite on a {% data variables.product.prodname_codeql %} database for a C/C++ codebase,
|
||||
you could use the following command from the directory containing your database:
|
||||
|
||||
```
|
||||
codeql database analyze <cpp-database> codeql/cpp-queries:codeql-suites/cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif --download
|
||||
```
|
||||
|
||||
This command downloads the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} query pack, runs the analysis, and generates a file in the SARIF version 2.1.0 format that is supported by all versions of {% data variables.product.prodname_dotcom %}. This file can be uploaded to {% data variables.product.prodname_dotcom %} by executing `codeql github upload-results` or the code scanning API.
|
||||
For more information, see "[Analyzing a {% data variables.product.prodname_codeql %} database](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database)"
|
||||
or "[Code scanning API](/rest/reference/code-scanning)".
|
||||
|
||||
{% data variables.product.prodname_codeql %} query suites are `.qls` files that use directives to select queries to run
|
||||
based on certain metadata properties. The standard {% data variables.product.prodname_codeql %} packs have metadata that specify
|
||||
the location of the query suites used by code scanning, so the {% data variables.product.prodname_codeql_cli %} knows where to find these
|
||||
suite files automatically, and you don’t have to specify the full path on the command line.
|
||||
For more information, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases/creating-codeql-query-suites/#creating-codeql-query-suites)".
|
||||
|
||||
For information about creating custom query suites, see "[Creating
|
||||
{% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites)."
|
||||
|
||||
#### Diagnostic and summary information
|
||||
|
||||
When you create a {% data variables.product.prodname_codeql %} database, the extractor stores diagnostic data in the database. The code scanning query suites include additional queries to report on this diagnostic data and calculate summary metrics. When the `database analyze` command completes, the CLI generates the results file and reports any diagnostic and summary data to standard output. If you choose to generate SARIF output, the additional data is also included in the SARIF file.
|
||||
|
||||
If the analysis found fewer results for standard queries than you expected, review the results of the diagnostic and summary queries to check whether the {% data variables.product.prodname_codeql %} database is likely to be a good representation of the codebase that you want to analyze.
|
||||
|
||||
### Integrating a {% data variables.product.prodname_codeql %} pack into a code scanning workflow in {% data variables.product.prodname_dotcom %}
|
||||
|
||||
You can use {% data variables.product.prodname_codeql %} query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
|
||||
For more information, see “[Using {% data variables.product.prodname_codeql %} query packs in the {% data variables.product.prodname_codeql %} action](/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-codeql-query-packs/)” or “[Downloading and using {% data variables.product.prodname_codeql %} query packs in your CI system](/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs).”
|
||||
|
||||
### Including query help for custom {% data variables.product.prodname_codeql %} queries in SARIF files
|
||||
|
||||
If you use the {% data variables.product.prodname_codeql_cli %} to run code scanning analyses on third party CI/CD systems,
|
||||
you can include the query help for your custom queries in SARIF files generated during an analysis.
|
||||
After uploading the SARIF file to {% data variables.product.prodname_dotcom %}, the query help is shown in the code scanning UI for any
|
||||
alerts generated by the custom queries.
|
||||
|
||||
From {% data variables.product.prodname_codeql_cli %} v2.7.1 onwards, you can include markdown-rendered query help in SARIF files
|
||||
by providing the `--sarif-add-query-help` option when running
|
||||
`codeql database analyze`.
|
||||
For more information, see [Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database).
|
||||
|
||||
You can write query help for custom queries directly in a markdown file and save it alongside the
|
||||
corresponding query. Alternatively, for consistency with the standard {% data variables.product.prodname_codeql %} queries,
|
||||
you can write query help in the `.qhelp` format. Query help written in `.qhelp`
|
||||
files can’t be included in SARIF files, and they can’t be processed by code
|
||||
scanning so must be converted to markdown before running
|
||||
the analysis. For more information, see [“Query help files”](https://codeql.github.com/docs/writing-codeql-queries/query-help-files/#query-help-files)
|
||||
and “[Testing query help files](/code-security/code-scanning/using-the-codeql-cli/testing-query-help-files).”
|
||||
|
||||
## Results
|
||||
|
||||
You can save analysis results in a number of different formats, including SARIF
|
||||
and CSV.
|
||||
|
||||
The SARIF format is designed to represent the output of a broad range of static
|
||||
analysis tools. For more information, see [SARIF output](/code-security/code-scanning/codeql-cli-reference/sarif-output).
|
||||
|
||||
If you choose to generate results in CSV format, then each line in the output file
|
||||
corresponds to an alert. Each line is a comma-separated list with the following information:
|
||||
|
||||
**Property**|**Description**|**Example**
|
||||
:-----:|:-----:|:-----:
|
||||
Name | Name of the query that identified the result. | `Inefficient regular expression`
|
||||
Description | Description of the query.| `A regular expression that requires exponential time to match certain inputs can be a performance bottleneck, and may be vulnerable to denial-of-service attacks.`
|
||||
Severity | Severity of the query.| `error`
|
||||
Message | Alert message.| `This part of the regular expression may cause exponential backtracking on strings containing many repetitions of '\\\\'.`
|
||||
Path | Path of the file containing the alert. | `/vendor/codemirror/markdown.js`
|
||||
Start line | Line of the file where the code that triggered the alert begins. | `617`
|
||||
Start column | Column of the start line that marks the start of the alert code. Not included when equal to 1. | `32`
|
||||
End line | Line of the file where the code that triggered the alert ends. Not included when the same value as the start line. | `64`
|
||||
End column | Where available, the column of the end line that marks the end of the alert code. Otherwise the end line is repeated. | `617`
|
||||
|
||||
Results files can be integrated into your own code-review or debugging
|
||||
infrastructure. For example, SARIF file output can be used to highlight alerts
|
||||
in the correct location in your source code using a SARIF viewer plugin for your
|
||||
IDE.
|
||||
|
||||
## Further reading
|
||||
|
||||
* [“Analyzing your projects in {% data variables.product.prodname_codeql %} for VS Code”](https://codeql.github.com/docs/codeql-for-visual-studio-code/analyzing-your-projects/#analyzing-your-projects)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Creating and working with CodeQL packs
|
||||
intro: 'You can use {% data variables.product.prodname_codeql %} packs to create, share, depend on, and run {% data variables.product.prodname_codeql %} queries and libraries.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
feature: codeql-packs
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
{% data reusables.codeql-cli.beta-note-package-management %}
|
||||
|
||||
## About {% data variables.product.prodname_codeql %} packs and the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
{% ifversion ghes %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[Creating and working with {% data variables.product.prodname_codeql %} packs](/enterprise-cloud@latest/code-security/code-scanning/using-the-codeql-cli/creating-and-working-with-codeql-packs)”, the Enterprise Cloud version, for the most up-to-date documentation.
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
With {% data variables.product.prodname_codeql %} packs and the package management commands in the {% data variables.product.prodname_codeql_cli %}, you can publish your custom queries and integrate them into your codebase analysis.
|
||||
|
||||
There are two types of {% data variables.product.prodname_codeql %} packs: query packs and library packs.
|
||||
|
||||
- Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and {% ifversion query-pack-compatibility %}pre-compiled representations of each query, in addition to the query sources{% else %}a compilation cache{% endif %}. This ensures consistent and efficient execution of the queries in the pack.
|
||||
|
||||
- Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled {% ifversion query-pack-compatibility %}separately{% else %}and there is no compilation cache included when the pack is published{% endif %}.
|
||||
|
||||
You can use the `pack` command in the {% data variables.product.prodname_codeql_cli %} to create {% data variables.product.prodname_codeql %} packs, add dependencies to packs, and install or update dependencies. You can also publish and download {% data variables.product.prodname_codeql %} packs using the `pack` command. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs).”
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
For more information about compatibility between published query packs and different {% data variables.product.prodname_codeql %} releases, see “[About {% data variables.product.prodname_codeql %} pack compatibility](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs#about-codeql-pack-compatibility).”
|
||||
{% endif %}
|
||||
|
||||
|
||||
## Creating a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
You can create a {% data variables.product.prodname_codeql %} pack by running the following command from the checkout root of your project:
|
||||
|
||||
```
|
||||
codeql pack init <scope>/<pack>
|
||||
```
|
||||
|
||||
You must specify:
|
||||
|
||||
|
||||
* `<scope>`: the name of the {% data variables.product.prodname_dotcom %} organization or user account that you will publish to.
|
||||
|
||||
|
||||
* `<pack>`: the name for the pack that you are creating.
|
||||
|
||||
The `codeql pack init` command creates the directory structure and configuration files for a {% data variables.product.prodname_codeql %} pack. By default, the command creates a query pack. If you want to create a library pack, you must edit the `qlpack.yml` file to explicitly declare the file as a library pack by including the `library:true` property.
|
||||
|
||||
## Modifying an existing legacy QL pack to create a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
If you already have a `qlpack.yml` file, you can edit it manually to convert it into a {% data variables.product.prodname_codeql %} pack.
|
||||
|
||||
|
||||
1. Edit the `name` property so that it matches the format `<scope>/<name>`, where `<scope>` is the name of the {% data variables.product.prodname_dotcom %} organization or user account that you will publish to.
|
||||
|
||||
|
||||
2. In the `qlpack.yml` file, include a `version` property with a semver identifier, as well as an optional `dependencies` block.
|
||||
|
||||
|
||||
3. Migrate the list of dependencies in `libraryPathDependencies` to the `dependencies` block. Specify the version range for each dependency. If the range is unimportant, or you are unsure of compatibility, you can specify `"\*"`, which indicates that any version is acceptable and will default to the latest version when you run `codeql pack install`.
|
||||
|
||||
For more information about the properties, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs#about-codeql-packs).”
|
||||
|
||||
## Adding and installing dependencies to a {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
You can add dependencies on {% data variables.product.prodname_codeql %} packs using the command `codeql pack add`. You must specify the scope, name, and (optionally) a compatible version range.
|
||||
|
||||
```
|
||||
codeql pack add <scope>/<name>@x.x.x <scope>/<other-name>
|
||||
```
|
||||
|
||||
If you don’t specify a version range, the latest version will be added. Otherwise, the latest version that satisfies the requested range will be added.
|
||||
|
||||
This command updates the `qlpack.yml` file with the requested dependencies and downloads them into the package cache. Please note that this command will reformat the file and remove all comments.
|
||||
|
||||
You can also manually edit the `qlpack.yml` file to include dependencies and install the dependencies with the command:
|
||||
|
||||
```
|
||||
codeql pack install
|
||||
```
|
||||
|
||||
This command downloads all dependencies to the shared cache on the local disk.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Notes:**
|
||||
|
||||
- Running the `codeql pack add` and `codeql pack install` commands will generate or update the `codeql-pack.lock.yml` file. This file should be checked-in to version control. The `codeql-pack.lock.yml` file contains the precise version numbers used by the pack. For more information, see “[About codeql-pack.lock.yml files](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs#about-codeql-pack-lock).”
|
||||
|
||||
- By default `codeql pack install` will install dependencies from the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %}. You can install dependencies from a {% data variables.product.prodname_ghe_server %} {% data variables.product.prodname_container_registry %} by creating a `qlconfig.yml` file. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs).”
|
||||
|
||||
{% endnote %}
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
## Customizing a downloaded {% data variables.product.prodname_codeql %} pack
|
||||
|
||||
The recommended way to experiment with changes to a pack is to clone the repository containing its source code.
|
||||
|
||||
If no source respository is available and you need to base modifications on a pack downloaded from the {% data variables.product.prodname_container_registry %}, be aware that these packs are not intended to be modified or customized after downloading, and their format may change in the future without much notice. We recommend taking the following steps after downloading a pack if you need to modify the content:
|
||||
|
||||
- Change the pack _name_ in `qlpack.yml` so you avoid confusion with results from the unmodified pack.
|
||||
|
||||
- Remove all files named `*.qlx` anywhere in the unpacked directory structure. These files contain precompiled versions of the queries, and in some situations {% data variables.product.prodname_codeql %} will use them in preference to the QL source you have modified.
|
||||
|
||||
{% endif %}
|
||||
@@ -0,0 +1,378 @@
|
||||
---
|
||||
title: Creating CodeQL databases
|
||||
intro: 'You can build a {% data variables.product.prodname_codeql %} database containing the data needed to query your code.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghec: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About creating {% data variables.product.prodname_codeql %} databases
|
||||
|
||||
Before you analyze your code using {% data variables.product.prodname_codeql %}, you need to create a {% data variables.product.prodname_codeql %} database containing all the data required to run queries on your code. You can create {% data variables.product.prodname_codeql %} databases yourself using the {% data variables.product.prodname_codeql_cli %}, or download them from {% data variables.product.prodname_dotcom_the_website %}.
|
||||
|
||||
{% data variables.product.prodname_codeql %} analysis relies on extracting relational data from your code, and using it to build a [{% data variables.product.prodname_codeql %} database](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#codeql-database). {% data variables.product.prodname_codeql %} databases contain all of the important information about a codebase, which can be analyzed by executing {% data variables.product.prodname_codeql %} queries against it. {% data variables.product.prodname_dotcom %} creates and stores {% data variables.product.prodname_codeql %} databases for a large number of open-source projects. For more information, see “[Downloading {% data variables.product.prodname_codeql %} databases from {% data variables.product.prodname_dotcom_the_website %}](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-databases#downloading-databases-from-githubcom).”
|
||||
|
||||
You can also create {% data variables.product.prodname_codeql %} databases yourself using the {% data variables.product.prodname_codeql_cli %}. Before you generate a {% data variables.product.prodname_codeql %} database, you need to:
|
||||
|
||||
- Install and set up the {% data variables.product.prodname_codeql_cli %}. For more information, see “[Getting started with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/getting-started-with-the-codeql-cli).”
|
||||
- Check out the version of your codebase you want to analyze. The directory should be ready to build, with all dependencies already installed.
|
||||
|
||||
For information about using the {% data variables.product.prodname_codeql_cli %} in a third-party CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, see [Configuring {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system). For information about enabling {% data variables.product.prodname_codeql %} code scanning using {% data variables.product.prodname_actions %}, see [Setting up code scanning for a repository](/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository).
|
||||
|
||||
## Running `codeql database create`
|
||||
|
||||
{% data variables.product.prodname_codeql %} databases are created by running the following command from the checkout root of your project:
|
||||
|
||||
```
|
||||
codeql database create <database> --language=<language-identifier>
|
||||
```
|
||||
|
||||
You must specify:
|
||||
|
||||
- `<database>`: a path to the new database to be created. This directory will be created when you execute the command—you cannot specify an existing directory.
|
||||
- `--language`: the identifier for the language to create a database for. When used with `--db-cluster`, the option accepts a comma-separated list, or can be specified more than once. {% data variables.product.prodname_codeql %} supports creating databases for the following languages:
|
||||
|
||||
| Language | Identifier
|
||||
|------------------|-------------------
|
||||
| C/C++ | `cpp`
|
||||
| C# | `csharp`
|
||||
| Go | `go`
|
||||
| Java{% ifversion codeql-kotlin-beta %}/Kotlin{% endif %} | `java`
|
||||
| JavaScript/TypeScript | `javascript`
|
||||
| Python | `python`
|
||||
| Ruby | `ruby`
|
||||
|
||||
{% data reusables.code-scanning.beta-kotlin-support %}
|
||||
{% data reusables.code-scanning.beta-ruby-support %}
|
||||
|
||||
You can specify additional options depending on the location of your source file, if the code needs to be compiled, and if you want to create {% data variables.product.prodname_codeql %} databases for more than one language:
|
||||
|
||||
- `--source-root`: the root folder for the primary source files used in database creation. By default, the command assumes that the current directory is the source root—use this option to specify a different location.
|
||||
- `--db-cluster`: use for multi-language codebases when you want to create databases for more than one language.
|
||||
- `--command`: used when you create a database for one or more compiled languages, omit if the only languages requested are Python and JavaScript. This specifies the build commands needed to invoke the compiler. Commands are run from the current folder, or `--source-root` if specified. If you don’t include a `--command`, {% data variables.product.prodname_codeql %} will attempt to detect the build system automatically, using a built-in autobuilder.
|
||||
- `--no-run-unnecessary-builds`: used with `--db-cluster` to suppress the build command for languages where the {% data variables.product.prodname_codeql_cli %} does not need to monitor the build (for example, Python and JavaScript/TypeScript).
|
||||
|
||||
You can specify extractor options to customize the behavior of extractors that create {% data variables.product.prodname_codeql %} databases. For more information, see
|
||||
“[Extractor options](/code-security/code-scanning/using-the-codeql-cli/extractor-options).”
|
||||
|
||||
For full details of all the options you can use when creating databases, see the [database create reference documentation](https://codeql.github.com/docs/codeql-cli/manual/database-create/).
|
||||
|
||||
## Progress and results
|
||||
|
||||
Errors are reported if there are any problems with the options you have specified. For interpreted languages, the extraction progress is displayed in the console—for each source file, it reports if extraction was successful or if it failed. For compiled languages, the console will display the output of the build system.
|
||||
|
||||
When the database is successfully created, you’ll find a new directory at the path specified in the command. If you used the `--db-cluster` option to create more than one database, a subdirectory is created for each language. Each {% data variables.product.prodname_codeql %} database directory contains a number of subdirectories, including the relational data (required for analysis) and a source archive—a copy of the source files made at the time the database was created—which is used for displaying analysis results.
|
||||
|
||||
## Creating databases for non-compiled languages
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} includes extractors to create databases for non-compiled languages—specifically, JavaScript (and TypeScript), Python, and Ruby. These extractors are automatically invoked when you specify JavaScript, Python, or Ruby as the `--language` option when executing `database create`. When creating databases for these languages you must ensure that all additional dependencies are available.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** When you run `database create` for JavaScript, TypeScript, Python, and Ruby, you should not specify a `--command` option. Otherwise this overrides the normal extractor invocation, which will create an empty database. If you create databases for multiple languages and one of them is a compiled language, use the `--no-run-unnecessary-builds` option to skip the command for the languages that don’t need to be compiled.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### JavaScript and TypeScript
|
||||
|
||||
Creating databases for JavaScript requires no additional dependencies, but if the project includes TypeScript files, you must install Node.js 6.x or later. In the command line you can specify `--language=javascript` to extract both JavaScript and TypeScript files:
|
||||
|
||||
```
|
||||
codeql database create --language=javascript --source-root <folder-to-extract> <output-folder>/javascript-database
|
||||
```
|
||||
|
||||
Here, we have specified a `--source-root` path, which is the location where database creation is executed, but is not necessarily the checkout root of the codebase.
|
||||
|
||||
By default, files in `node_modules` and `bower_components` directories are not extracted.
|
||||
|
||||
### Python
|
||||
|
||||
When creating databases for Python you must ensure:
|
||||
|
||||
- You have Python 3 installed and available to the {% data variables.product.prodname_codeql %} extractor.
|
||||
- You have the version of Python used by your code installed.
|
||||
- You have access to the [pip](https://pypi.org/project/pip/) packaging management system and can install any packages that the codebase depends on.
|
||||
- You have installed the [virtualenv](https://pypi.org/project/virtualenv/) pip module.
|
||||
|
||||
In the command line you must specify `--language=python`. For example:
|
||||
|
||||
```
|
||||
codeql database create --language=python <output-folder>/python-database
|
||||
```
|
||||
|
||||
This executes the `database create` subcommand from the code’s checkout root, generating a new Python database at `<output-folder>/python-database`.
|
||||
|
||||
### Ruby
|
||||
|
||||
Creating databases for Ruby requires no additional dependencies. In the command line you must specify `--language=ruby`. For example:
|
||||
|
||||
```
|
||||
codeql database create --language=ruby --source-root <folder-to-extract> <output-folder>/ruby-database
|
||||
```
|
||||
|
||||
Here, we have specified a `--source-root` path, which is the location where database creation is executed, but is not necessarily the checkout root of the codebase.
|
||||
|
||||
## Creating databases for compiled languages
|
||||
|
||||
For compiled languages, {% data variables.product.prodname_codeql %} needs to invoke the required build system to generate a database, therefore the build method must be available to the CLI.
|
||||
|
||||
### Detecting the build system
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** {% data variables.product.prodname_codeql %} analysis for Kotlin is currently in beta. During the beta, analysis of Kotlin code, and the accompanying documentation, will not be as comprehensive as for other languages.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} includes autobuilders for C/C++, C#, Go, and Java code. {% data variables.product.prodname_codeql %} autobuilders allow you to build projects for compiled languages without specifying any build commands. When an autobuilder is invoked, {% data variables.product.prodname_codeql %} examines the source for evidence of a build system and attempts to run the optimal set of commands required to extract a database.
|
||||
|
||||
An autobuilder is invoked automatically when you execute `codeql database create` for a compiled `--language` if don’t include a
|
||||
`--command` option. For example, for a Java codebase, you would simply run:
|
||||
|
||||
```
|
||||
codeql database create --language=java <output-folder>/java-database
|
||||
```
|
||||
|
||||
If a codebase uses a standard build system, relying on an autobuilder is often the simplest way to create a database. For sources that require non-standard build steps, you may need to explicitly define each step in the command line.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Notes:**
|
||||
|
||||
- If you are building a Go database, install the Go toolchain (version 1.11 or later) and, if there are dependencies, the appropriate dependency manager (such as [dep](https://golang.github.io/dep/)).
|
||||
- The Go autobuilder attempts to automatically detect code written in Go in a repository, and only runs build scripts in an attempt to fetch dependencies. To force {% data variables.product.prodname_codeql %} to limit extraction to the files compiled by your build script, set the environment variable `CODEQL_EXTRACTOR_GO_BUILD_TRACING=on` or use the `--command` option to specify a build command.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### Specifying build commands
|
||||
|
||||
The following examples are designed to give you an idea of some of the build commands that you can specify for compiled languages.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** The `--command` option accepts a single argument—if you need to use more than one command, specify `--command` multiple times. If you need to pass subcommands and options, the whole argument needs to be quoted to be interpreted correctly.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
- C/C++ project built using `make`:
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create cpp-database --language=cpp --command=make
|
||||
```
|
||||
|
||||
- C# project built using `dotnet build`:
|
||||
|
||||
It is a good idea to add `/t:rebuild` to ensure that all code will be built, or do a prior `dotnet clean` (code that is not built will not be included in the {% data variables.product.prodname_codeql %} database):
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create csharp-database --language=csharp --command='dotnet build /t:rebuild'
|
||||
```
|
||||
|
||||
- Go project built using the `CODEQL_EXTRACTOR_GO_BUILD_TRACING=on` environment variable:
|
||||
|
||||
```
|
||||
CODEQL_EXTRACTOR_GO_BUILD_TRACING=on {% data variables.product.prodname_codeql %} database create go-database --language=go
|
||||
```
|
||||
|
||||
- Go project built using a custom build script:
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create go-database --language=go --command='./scripts/build.sh'
|
||||
```
|
||||
|
||||
- Java project built using Gradle:
|
||||
|
||||
```
|
||||
# Use `--no-daemon` because a build delegated to an existing daemon cannot be detected by CodeQL: {% data variables.product.prodname_codeql %} database create java-database --language=java --command='gradle --no-daemon clean test'
|
||||
```
|
||||
|
||||
- Java project built using Maven:
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create java-database --language=java --command='mvn clean install'
|
||||
```
|
||||
|
||||
- Java project built using Ant:
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create java-database --language=java --command='ant -f build.xml'
|
||||
```
|
||||
|
||||
- Project built using Bazel:
|
||||
|
||||
```
|
||||
# Navigate to the Bazel workspace.
|
||||
|
||||
# Before building, remove cached objects
|
||||
# and stop all running Bazel server processes.
|
||||
bazel clean --expunge
|
||||
|
||||
# Build using the following Bazel flags, to help {% data variables.product.prodname_codeql %} detect the build:
|
||||
# `--spawn_strategy=local`: build locally, instead of using a distributed build
|
||||
# `--nouse_action_cache`: turn off build caching, which might prevent recompilation of source code
|
||||
# `--noremote_accept_cached`, `--noremote_upload_local_results`: avoid using a remote cache
|
||||
{% data variables.product.prodname_codeql %} database create new-database --language=<language> \
|
||||
--command='bazel build --spawn_strategy=local --nouse_action_cache --noremote_accept_cached --noremote_upload_local_results //path/to/package:target'
|
||||
|
||||
# After building, stop all running Bazel server processes.
|
||||
# This ensures future build commands start in a clean Bazel server process
|
||||
# without {% data variables.product.prodname_codeql %} attached.
|
||||
bazel shutdown
|
||||
```
|
||||
|
||||
- Project built using a custom build script:
|
||||
|
||||
```
|
||||
{% data variables.product.prodname_codeql %} database create new-database --language=<language> --command='./scripts/build.sh'
|
||||
```
|
||||
|
||||
This command runs a custom script that contains all of the commands required to build the project.
|
||||
|
||||
### Using indirect build tracing
|
||||
|
||||
If the {% data variables.product.prodname_codeql_cli %} autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with `codeql database trace-command`, you can use indirect build tracing to create a {% data variables.product.prodname_codeql %} database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
|
||||
|
||||
To create a {% data variables.product.prodname_codeql %} database with indirect build tracing, run the following command from the checkout root of your project:
|
||||
|
||||
```
|
||||
codeql database init ... --begin-tracing <database>
|
||||
```
|
||||
|
||||
You must specify:
|
||||
|
||||
- `<database>`: a path to the new database to be created. This directory will be created when you execute the command—you cannot specify an existing directory.
|
||||
- `--begin-tracing`: creates scripts that can be used to set up an environment in which build commands will be traced.
|
||||
|
||||
You may specify other options for the `codeql database init` command as normal.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** If the build runs on Windows, you must set either `--trace-process-level <number>` or `--trace-process-name <parent process name>` so that the option points to a parent CI process that will observe all build steps for the code being analyzed.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
The `codeql database init` command will output a message:
|
||||
|
||||
```
|
||||
Created skeleton <database>. This in-progress database is ready to be populated by an extractor. In order to initialise tracing, some environment variables need to be set in the shell your build will run in. A number of scripts to do this have been created in <database>/temp/tracingEnvironment. Please run one of these scripts before invoking your build command.
|
||||
|
||||
Based on your operating system, we recommend you run: ...
|
||||
```
|
||||
|
||||
The `codeql database init` command creates `<database>/temp/tracingEnvironment` with files that contain environment variables and values that will enable {% data variables.product.prodname_codeql %} to trace a sequence of build steps. These files are named `start-tracing.{json,sh,bat,ps1}`. Use one of these files with your CI system’s mechanism for setting environment variables for future steps. You can:
|
||||
|
||||
- Read the JSON file, process it, and print out environment variables in the format expected by your CI system. For example, Azure DevOps expects `echo "##vso[task.setvariable variable=NAME]VALUE"`.
|
||||
- Or, if your CI system persists the environment, source the appropriate `start-tracing` script to set the {% data variables.product.prodname_codeql %} variables in the shell environment of the CI system.
|
||||
|
||||
Build your code; optionally, unset the environment variables using an `end-tracing.{json,sh,bat,ps1}` script from the directory where the `start-tracing` scripts are stored; and then run the command `codeql database finalize <database>`.
|
||||
|
||||
Once you have created a {% data variables.product.prodname_codeql %} database using indirect build tracing, you can work with it like any other {% data variables.product.prodname_codeql %} database. For example, analyze the database, and upload the results to {% data variables.product.prodname_dotcom %} if you use code scanning.
|
||||
|
||||
### Example of creating a {% data variables.product.prodname_codeql %} database using indirect build tracing
|
||||
|
||||
The following example shows how you could use indirect build tracing in an Azure DevOps pipeline to create a {% data variables.product.prodname_codeql %} database:
|
||||
|
||||
```
|
||||
steps:
|
||||
# Download the {% data variables.product.prodname_codeql_cli %} and query packs...
|
||||
# Check out the repository ...
|
||||
|
||||
# Run any pre-build tasks, for example, restore NuGet dependencies...
|
||||
|
||||
# Initialize the {% data variables.product.prodname_codeql %} database.
|
||||
# In this example, the {% data variables.product.prodname_codeql_cli %} has been downloaded and placed on the PATH.
|
||||
- task: CmdLine@1
|
||||
displayName: Initialize {% data variables.product.prodname_codeql %} database
|
||||
inputs:
|
||||
# Assumes the source code is checked out to the current working directory.
|
||||
# Creates a database at `<current working directory>/db`.
|
||||
# Running on Windows, so specifies a trace process level.
|
||||
script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing db"
|
||||
|
||||
# Read the generated environment variables and values,
|
||||
# and set them so they are available for subsequent commands
|
||||
# in the build pipeline. This is done in PowerShell in this example.
|
||||
- task: PowerShell@1
|
||||
displayName: Set {% data variables.product.prodname_codeql %} environment variables
|
||||
inputs:
|
||||
targetType: inline
|
||||
script: >
|
||||
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
|
||||
$json.PSObject.Properties | ForEach-Object {
|
||||
$template = "##vso[task.setvariable variable="
|
||||
$template += $_.Name
|
||||
$template += "]"
|
||||
$template += $_.Value
|
||||
echo "$template"
|
||||
}
|
||||
|
||||
# Execute the pre-defined build step. Note the `msbuildArgs` variable.
|
||||
- task: VSBuild@1
|
||||
inputs:
|
||||
solution: '**/*.sln'
|
||||
msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory)
|
||||
platform: Any CPU
|
||||
configuration: Release
|
||||
# Execute a clean build, in order to remove any existing build artifacts prior to the build.
|
||||
clean: True
|
||||
displayName: Visual Studio Build
|
||||
|
||||
# Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
|
||||
- task: PowerShell@1
|
||||
displayName: Clear {% data variables.product.prodname_codeql %} environment variables
|
||||
inputs:
|
||||
targetType: inline
|
||||
script: >
|
||||
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
|
||||
$json.PSObject.Properties | ForEach-Object {
|
||||
$template = "##vso[task.setvariable variable="
|
||||
$template += $_.Name
|
||||
$template += "]"
|
||||
$template += $_.Value
|
||||
echo "$template"
|
||||
}
|
||||
|
||||
- task: CmdLine@2
|
||||
displayName: Finalize {% data variables.product.prodname_codeql %} database
|
||||
inputs:
|
||||
script: 'codeql database finalize db'
|
||||
|
||||
# Other tasks go here, for example:
|
||||
# `codeql database analyze`
|
||||
# then `codeql github upload-results` ...
|
||||
```
|
||||
|
||||
## Downloading databases from {% data variables.product.prodname_dotcom_the_website %}
|
||||
|
||||
{% data variables.product.prodname_dotcom %} stores {% data variables.product.prodname_codeql %} databases for over 200,000 repos on {% data variables.product.prodname_dotcom_the_website %}, which you can download using the REST API. The list of repos is constantly growing and evolving to make sure that it includes the most interesting codebases for security research.
|
||||
|
||||
You can check if a repository has any {% data variables.product.prodname_codeql %} databases available for download using the `/repos/<owner>/<repo>/code-scanning/codeql/databases` endpoint. For example, to check for {% data variables.product.prodname_codeql %} databases using the [{% data variables.product.prodname_cli %}](https://cli.github.com/manual/gh_api) you would run:
|
||||
|
||||
```
|
||||
gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/
|
||||
```
|
||||
|
||||
This command returns information about any {% data variables.product.prodname_codeql %} databases that are available for a repository, including the language the database represents, and when the database was last updated. If no {% data variables.product.prodname_codeql %} databases are available, the response is empty.
|
||||
|
||||
When you have confirmed that a {% data variables.product.prodname_codeql %} database exists for the language you are interested in, you can download it using the following command:
|
||||
|
||||
```
|
||||
gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip
|
||||
```
|
||||
|
||||
For more information, see the documentation for the [Get {% data variables.product.prodname_codeql %} database endpoint](/rest/code-scanning?apiVersion=2022-11-28#get-a-codeql-database-for-a-repository).
|
||||
|
||||
Before running an analysis with the {% data variables.product.prodname_codeql_cli %}, you must unzip the databases.
|
||||
|
||||
## Further reading
|
||||
|
||||
- “[Analyzing your projects in {% data variables.product.prodname_codeql %} for VS Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/analyzing-your-projects/#analyzing-your-projects)”
|
||||
@@ -0,0 +1,405 @@
|
||||
---
|
||||
title: Creating CodeQL query suites
|
||||
intro: 'You can create query suites for queries you frequently use in your {% data variables.product.prodname_codeql %} analyses.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About creating {% data variables.product.prodname_codeql %} query suites
|
||||
|
||||
{% data variables.product.prodname_codeql %} query suites provide a way of selecting queries, based on their
|
||||
filename, location on disk{% ifversion codeql-packs %} or in a {% data variables.product.prodname_codeql %} pack{% endif %}, or metadata properties.
|
||||
Create query suites for the queries that you want to frequently use in
|
||||
your {% data variables.product.prodname_codeql %} analyses.
|
||||
|
||||
Query suites allow you to pass multiple queries to {% data variables.product.prodname_codeql %} without having to specify the path to each query file individually. Query suite definitions are stored in YAML files with the extension `.qls`. A suite definition is a sequence of instructions, where each instruction is a YAML
|
||||
mapping with (usually) a single key. The instructions are executed in the order
|
||||
they appear in the query suite definition. After all the instructions in the
|
||||
suite definition have been executed, the result is a set of selected queries.
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
{% note %}
|
||||
|
||||
**Note:** Any custom queries that you want to add to a query suite must be in a [{% data variables.product.prodname_codeql %} pack](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs) and contain the correct query metadata. For more information, see “[Using custom queries with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/using-custom-queries-with-the-codeql-cli).”
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
## Locating queries to add to a query suite
|
||||
|
||||
When creating a query suite, you first need to specify the locations of the
|
||||
queries that you want to select. You can define the location of one or more
|
||||
queries using:
|
||||
|
||||
- A `query` instruction—tells {% data variables.product.prodname_codeql %} to look for one or more specified `.ql`
|
||||
files:
|
||||
|
||||
```
|
||||
- query: <path-to-query>
|
||||
```
|
||||
The argument must be one or more file paths, relative to the {% data variables.product.prodname_codeql %} pack containing
|
||||
the suite definition.
|
||||
|
||||
- A `queries` instruction—tells {% data variables.product.prodname_codeql %} to recursively scan a directory
|
||||
for `.ql` files:
|
||||
|
||||
```
|
||||
- queries: <path-to-subdirectory>
|
||||
```
|
||||
|
||||
The path of the directory must be relative to the root of the {% data variables.product.prodname_codeql %} pack that
|
||||
contains the suite definition file. To find the queries relative to a
|
||||
different {% data variables.product.prodname_codeql %} pack, add a `from` field:
|
||||
|
||||
```
|
||||
- queries: <path-to-subdirectory>
|
||||
from: <ql-pack-name>
|
||||
version: ^x.y.z
|
||||
```
|
||||
|
||||
The `version` field is optional and specifies a range of compatible versions of this {% data variables.product.prodname_codeql %} pack.
|
||||
If you don’t specify a version, then the most recent version of the pack is used.
|
||||
|
||||
- A `qlpack` instruction—tells {% data variables.product.prodname_codeql %} to resolve queries in the default suite of the
|
||||
named {% data variables.product.prodname_codeql %} pack:
|
||||
|
||||
```
|
||||
- qlpack: <qlpack-name>
|
||||
version: ^x.y.z
|
||||
```
|
||||
|
||||
The default suite of a query pack includes a recommended set of queries
|
||||
inside of that query pack. Not all query packs have a default suite. If the given query pack does not define a default suite, the qlpack instruction will resolve to all of the queries within the pack.
|
||||
|
||||
The `version` field is optional and specifies a range of compatible versions of this {% data variables.product.prodname_codeql %} pack.
|
||||
If you don’t specify a version, then the most recent version of the pack is used.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** When pathnames appear in query suite definitions, they must always be given with a forward slash, `/`, as a directory separator. This ensures that query suite definitions work on all operating systems.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
You must add at least one `query`, `queries`, or `qlpack` instruction to
|
||||
your suite definition, otherwise no queries will be selected. If the suite
|
||||
contains no further instructions, all the queries found from the list of files,
|
||||
in the given directory, or in the named {% data variables.product.prodname_codeql %} pack are selected. If there are further
|
||||
filtering instructions, only queries that match the constraints imposed by those
|
||||
instructions will be selected.
|
||||
|
||||
## Filtering the queries in a query suite
|
||||
|
||||
After you have defined the initial set of queries to add to your suite by
|
||||
specifying `query`, `queries`, or `qlpack` instructions, you can add
|
||||
`include` and `exclude` instructions. These instructions define selection
|
||||
criteria based on specific properties:
|
||||
|
||||
- When you execute an `include` instruction on a set of queries, any
|
||||
queries that match your conditions are retained in the selection, and queries
|
||||
that don’t match are removed.
|
||||
- When you execute an `exclude` instructions on a set of queries,
|
||||
any queries that match your conditions are removed from the selection, and queries
|
||||
that don’t match are retained.
|
||||
|
||||
The order of your filter instructions is important. The first filter instruction
|
||||
that appears after the locating instructions determines whether the queries are
|
||||
included or excluded by default. If the first filter is an `include`, the
|
||||
initially located queries will only be part of the suite if they match an
|
||||
explicit `include` filter. If the first filter is an `exclude`, the initially
|
||||
located queries are part of the suite unless they are explicitly excluded.
|
||||
|
||||
Subsequent instructions are executed in order and the instructions that appear
|
||||
later in the file take precedence over the earlier instructions. So, `include`
|
||||
instructions can be overridden by a later `exclude` instructions that match
|
||||
the same query. Similarly, `exclude`s can be overridden by a later
|
||||
`include`.
|
||||
|
||||
For both instructions, the argument is a constraint block—that is, a YAML map
|
||||
representing the constraints. Each constraint is a map entry, where the key is
|
||||
typically a query metadata property. The value can be:
|
||||
|
||||
- A single string.
|
||||
- A `/`-enclosed [regular expression](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html).
|
||||
- A list containing strings, regular expressions, or both.
|
||||
|
||||
To match a constraint, a metadata value must match one of the strings or
|
||||
regular expressions. When there is more than one metadata key, each key must be matched.
|
||||
The standard metadata keys available to match on are: `description`, `id`, `kind`,
|
||||
`name`, `tags`, `precision`, and `problem.severity`.
|
||||
For more information about query metadata properties, see
|
||||
“[Metadata for {% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/metadata-for-codeql-queries/#metadata-for-codeql-queries).”
|
||||
|
||||
In addition to metadata tags, the keys in the constraint block can also be:
|
||||
|
||||
- `query filename`—matches on the last path component of the query file name.
|
||||
- `query path`—matches on the path to the query file relative to its
|
||||
enclosing {% data variables.product.prodname_codeql %} pack.
|
||||
- `tags contain`—one of the given match strings must match
|
||||
one of the space-separated components of the value of the `@tags` metadata property.
|
||||
- `tags contain all`—each of the given match strings must match one of the
|
||||
components of the `@tags` metadata property.
|
||||
|
||||
### Examples of filtering which queries are run
|
||||
|
||||
A common use case is to create a query suite that runs all queries in a {% data variables.product.prodname_codeql %} pack,
|
||||
except for a few specific queries that the user does not want to run. In general, we
|
||||
recommend filtering on the query `id`, which is a unique and stable identifier for
|
||||
each query. The following three query suite definitions are semantically identical and
|
||||
filter by the query `id`:
|
||||
|
||||
This filter matches all the queries in the default suite of `codeql/cpp-queries`, except for the two queries with the excluded identifiers:
|
||||
|
||||
```
|
||||
- qlpack: codeql/cpp-queries
|
||||
- exclude:
|
||||
id:
|
||||
- cpp/cleartext-transmission
|
||||
- cpp/cleartext-storage-file
|
||||
```
|
||||
|
||||
In this example, a separate `exclude` instruction is used for each query:
|
||||
|
||||
```
|
||||
- qlpack: codeql/cpp-queries
|
||||
- exclude:
|
||||
id: cpp/cleartext-transmission
|
||||
- exclude:
|
||||
id: cpp/cleartext-storage-file
|
||||
```
|
||||
|
||||
In this example, a regular expression excludes the same two queries. It would also exclude any future queries added to the suite with identifiers that begin: `cpp/cleartext-`:
|
||||
|
||||
```
|
||||
- qlpack: codeql/cpp-queries
|
||||
- exclude:
|
||||
id:
|
||||
- /^cpp\/cleartext-.*/
|
||||
```
|
||||
|
||||
To define a suite that selects all queries in the default suite of the
|
||||
`codeql/cpp-queries` {% data variables.product.prodname_codeql %} pack, and then refines them to only include
|
||||
security queries, use:
|
||||
|
||||
```
|
||||
- qlpack: codeql/cpp-queries
|
||||
- include:
|
||||
tags contain: security
|
||||
```
|
||||
|
||||
To define a suite that selects all queries with `@kind problem`
|
||||
and `@precision high` from the `my-custom-queries` directory, use:
|
||||
|
||||
```
|
||||
- queries: my-custom-queries
|
||||
- include:
|
||||
kind: problem
|
||||
precision: very-high
|
||||
```
|
||||
|
||||
Note that the following query suite definition behaves differently from the definition above. This definition selects queries that are `@kind problem` *or*
|
||||
are `@precision very-high`:
|
||||
|
||||
```
|
||||
- queries: my-custom-queries
|
||||
- include:
|
||||
kind: problem
|
||||
- include:
|
||||
precision: very-high
|
||||
```
|
||||
|
||||
To create a suite that selects all queries with `@kind problem` from the
|
||||
`my-custom-queries` directory except those with `@problem.severity
|
||||
recommendation`, use:
|
||||
|
||||
```
|
||||
- queries: my-custom-queries
|
||||
- include:
|
||||
kind: problem
|
||||
- exclude:
|
||||
problem.severity: recommendation
|
||||
```
|
||||
|
||||
To create a suite that selects all queries with `@tag security` and
|
||||
`@problem.severity high` or `very-high` from the `codeql/cpp-queries` {% data variables.product.prodname_codeql %} pack,
|
||||
use:
|
||||
|
||||
```
|
||||
- queries: .
|
||||
from: codeql/cpp-queries
|
||||
- include:
|
||||
tags contain: security
|
||||
problem.severity:
|
||||
- high
|
||||
- very-high
|
||||
```
|
||||
|
||||
<!--Changed this to a note to fit with style guide -->
|
||||
{% note %}
|
||||
|
||||
**Note:** You can use the `codeql resolve queries /path/to/suite.qls` command to see which queries are selected by a query suite definition. For more information, see the [resolve queries](https://codeql.github.com/docs/codeql-cli/manual/resolve-queries/) reference documentation.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
## Reusing existing query suite definitions
|
||||
|
||||
Existing query suite definitions can be reused by specifying:
|
||||
|
||||
- An `import` instruction—adds the queries selected by a
|
||||
previously defined `.qls` file to the current suite:
|
||||
|
||||
```
|
||||
- import: <path-to-query-suite>
|
||||
```
|
||||
|
||||
The path to the imported suite must be relative to the {% data variables.product.prodname_codeql %} pack containing the
|
||||
current suite definition. If the imported query suite is in a different QL
|
||||
pack you can use:
|
||||
|
||||
```
|
||||
- import: <path-to-query-suite>
|
||||
from: <ql-pack>
|
||||
version: ^x.y.z
|
||||
```
|
||||
|
||||
The `version` field is optional and specifies a range of compatible versions of this {% data variables.product.prodname_codeql %} pack.
|
||||
If you don’t specify a version, then the most recent version of the pack is used.
|
||||
|
||||
Queries added using an `import` instruction can be filtered using subsequent
|
||||
`exclude` instructions.
|
||||
|
||||
- An `apply` instruction—adds all of the instructions from a
|
||||
previously defined `.qls` file to the current suite. The instructions in the
|
||||
applied `.qls` file are executed as if they appear in place of `apply`.
|
||||
Any `include` and `exclude` instructions from the applied suite also act on
|
||||
queries added by any earlier instructions:
|
||||
|
||||
```
|
||||
- apply: <path-to-query-suite>
|
||||
```
|
||||
|
||||
The `apply` instruction can also be used to apply a set of reusable
|
||||
conditions, saved in a `.yml` file, to multiple query definitions. For more
|
||||
information, see the [examples](#reusability-examples) below.
|
||||
|
||||
### Reusability Examples
|
||||
|
||||
To use the same conditions in multiple query suite definitions, create a
|
||||
separate `.yml` file containing your instructions. For example, save the
|
||||
following in a file called `reusable-instructions.yml`:
|
||||
|
||||
```
|
||||
- include:
|
||||
kind:
|
||||
- problem
|
||||
- path-problem
|
||||
tags contain: security
|
||||
precision:
|
||||
- high
|
||||
- very-high
|
||||
```
|
||||
|
||||
Add `reusable-instructions.yml` to the same {% data variables.product.prodname_codeql %} pack as your current query
|
||||
suite. Then, in one or more query suites, use the `apply` instruction to apply
|
||||
the reusable instructions to the current suite. For example:
|
||||
|
||||
```
|
||||
- queries: queries/cpp/custom
|
||||
- apply: reusable-instructions.yml
|
||||
```
|
||||
|
||||
This will filter the queries in `queries/cpp/custom` to only include those that match the reusable conditions.
|
||||
|
||||
You can also create a suite definition using `reusable-instructions.yml` on
|
||||
queries in a different {% data variables.product.prodname_codeql %} pack. If the `.qls` file is in the same {% data variables.product.prodname_codeql %} pack as
|
||||
the queries, you can add a `from` field immediately after the `apply`
|
||||
instruction:
|
||||
|
||||
```
|
||||
# load queries from the default suite of my-org/my-other-custom-queries
|
||||
- qlpack: my-org/my-other-custom-queries
|
||||
|
||||
# apply the reusable instructions from the my-org/my-custom-instructions {% data variables.product.prodname_codeql %} pack
|
||||
- apply: reusable-instructions.yml
|
||||
from: my-org/my-custom-instructions
|
||||
version: ^1.2.3 # optional
|
||||
```
|
||||
|
||||
A common use case for an `import` instruction is to apply a further filter to queries from another
|
||||
query suite. For example, this suite will further filter the `cpp-security-and-quality` suite
|
||||
and exclude `low` and `medium` precision queries:
|
||||
|
||||
```
|
||||
- import: codeql-suites/cpp-security-and-quality.qls
|
||||
from: codeql/cpp-queries
|
||||
- exclude:
|
||||
precision:
|
||||
- low
|
||||
- medium
|
||||
```
|
||||
|
||||
If you want to `include` queries imported from another suite, the syntax is a little different:
|
||||
|
||||
```
|
||||
- import: codeql-suites/cpp-security-and-quality.qls
|
||||
from: codeql/cpp-queries
|
||||
- exclude: {}
|
||||
- include:
|
||||
precision:
|
||||
- very-high
|
||||
- high
|
||||
```
|
||||
|
||||
Notice the empty `exclude` instruction. This is required to ensure that the subsequent `include`
|
||||
instruction is able to filter queries from the imported suite.
|
||||
|
||||
## Naming a query suite
|
||||
|
||||
You can provide a name for your query suite by specifying a `description`
|
||||
instruction:
|
||||
|
||||
```
|
||||
- description: <name-of-query-suite>
|
||||
```
|
||||
|
||||
This value is displayed when you run [codeql resolve queries](https://codeql.github.com/docs/codeql-cli/manual/resolve-queries/), if the suite is added to a “well-known”
|
||||
directory. For more information, see “[Specifying well-known query suites](#specifying-well-known-query-suites).”
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
## Saving a query suite
|
||||
|
||||
Save your query suite in a file with a `.qls` extension and add it to a CodeQL
|
||||
pack. For more information, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs#custom-codeql-packs).”
|
||||
|
||||
## Specifying well-known query suites
|
||||
|
||||
You can use {% data variables.product.prodname_codeql %} packs to declare directories that contain “well-known” query
|
||||
suites. You can use “well-known” query suites on the command line by referring
|
||||
to their file name,
|
||||
without providing their full path. This gives you a simple way of specifying a
|
||||
set of queries, without needing to search inside {% data variables.product.prodname_codeql %} packs and distributions.
|
||||
To declare a directory that contains “well-known” query suites, add the directory
|
||||
to the `suites` property in the `qlpack.yml` file at the root of your {% data variables.product.prodname_codeql %} pack.
|
||||
For more information, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs#codeqlpack-yml-properties).”
|
||||
{% endif %}
|
||||
|
||||
## Using query suites with CodeQL
|
||||
|
||||
You can specify query suites on the command line for any command that accepts
|
||||
`.qls` files. For example, you can compile the queries selected by a suite
|
||||
definition using `query compile`, or use the queries in an analysis using
|
||||
`database analyze`. For more information about analyzing {% data variables.product.prodname_codeql %} databases, see
|
||||
“[Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli).”
|
||||
|
||||
## Further reading
|
||||
|
||||
- “[{% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/codeql-queries/#codeql-queries)”
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: Extractor options
|
||||
intro: 'You can use the {% data variables.product.prodname_codeql_cli %} to run {% data variables.product.prodname_codeql %} processes locally on software projects.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About extractors
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %}uses special programs, called extractors, to extract information from the source code of a software system into a database that can be queried. You can customize the behavior of extractors by setting extractor configuration options through the {% data variables.product.prodname_codeql_cli %}.
|
||||
|
||||
## About extractor options
|
||||
|
||||
Each extractor defines its own set of configuration options. To find out which options are available for a particular extractor, you can run `codeql resolve languages` or `codeql resolve extractor` with the `--format=betterjson` option. The `betterjson` output format provides the root paths of extractors and additional information. The output of `codeql resolve extractor --format=betterjson` will often be formatted like the following example:
|
||||
|
||||
```json
|
||||
{
|
||||
"extractor_root" : "/home/user/codeql/java",
|
||||
"extractor_options" : {
|
||||
"option1" : {
|
||||
"title" : "Java extractor option 1",
|
||||
"description" : "An example string option for the Java extractor.",
|
||||
"type" : "string",
|
||||
"pattern" : "[a-z]+"
|
||||
},
|
||||
"group1" : {
|
||||
"title" : "Java extractor group 1",
|
||||
"description" : "An example option group for the Java extractor.",
|
||||
"type" : "object",
|
||||
"properties" : {
|
||||
"option2" : {
|
||||
"title" : "Java extractor option 2",
|
||||
"description" : "An example array option for the Java extractor",
|
||||
"type" : "array",
|
||||
"pattern" : "[1-9][0-9]*"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The extractor option names and descriptions are listed under `extractor_options`. Each option may contain the following fields:
|
||||
|
||||
- `title` (required): The title of the option
|
||||
- `description` (required): The description of the option
|
||||
- `type` (required): The type of the option, which can be
|
||||
- `string`: indicating that the option can have a single string value
|
||||
- `array`: indicating that the option can have a sequence of string values
|
||||
- `object`: indicating that it is not an option itself, but a grouping that may contain other options and option groups
|
||||
- `pattern` (optional): The regular expression patterns that all values of the option should match. Note that the extractor may impose additional constraints on option values that are not or cannot be expressed in this regular expression pattern. Such constraints, if they exist, would be explained under the description field.
|
||||
- `properties` (optional): A map from extractor option names in the option group to the corresponding extractor option descriptions. This field can only be present for option groups. For example, options of `object` type.
|
||||
|
||||
In the example above, the extractor declares two options:
|
||||
|
||||
- `option1` is a `string` option with value matching `[a-z]+`
|
||||
- `group1.option2` is an `array` option with values matching `[1-9][0-9]\*`
|
||||
|
||||
## Setting extractor options with the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} supports setting extractor options in subcommands that directly or indirectly invoke extractors. These commands are:
|
||||
|
||||
- `codeql database create`
|
||||
- `codeql database start-tracing`
|
||||
- `codeql database trace-command`
|
||||
- `codeql database index-files`
|
||||
|
||||
When running these subcommands, you can set extractor options with the `--extractor-option` CLI option. For example:
|
||||
|
||||
- `codeql database create --extractor-option java.option1=abc ...`
|
||||
- `codeql database start-tracing --extractor-option java.group1.option2=102 ...`
|
||||
|
||||
`--extractor-option` requires exactly one argument of the form `extractor_option_name=extractor_option_value`. `extractor_option_name` is the name of the extractor (in this example, `java`) followed by a period and then the name of the extractor option (in this example, either `option1` or `group1.option2`). `extractor_option_value` is the value being assigned to the extractor option. The value must match the regular expression pattern of the extractor option (if it exists), and it must not contain newline characters.
|
||||
|
||||
Using `--extractor-option` to assign an extractor option that does not exist is an error.
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} accepts multiple `--extractor-option` options in the same invocation. If you set a `string` extractor option multiple times, the last option value overwrites all previous ones. If you set an array extractor option multiple times, all option values are concatenated in order.
|
||||
|
||||
You can also specify extractor option names without the extractor name. For example:
|
||||
|
||||
- `codeql database create --extractor-option option1=abc ...`
|
||||
- `codeql database start-tracing --extractor-option group1.option2=102 ...`
|
||||
|
||||
If you do not specify an extractor name, the extractor option settings will apply to all extractors that declare an option with the given name. In the above example, the first command would set the extractor option `option1` to `abc` for the `java` extractor and every extractor that has an option of `option1`, for example the `cpp` extractor, if the `option1` extractor option exists for that extractor.
|
||||
|
||||
## Setting extractor options from files
|
||||
|
||||
You can also set extractor options through a file. The {% data variables.product.prodname_codeql_cli %} subcommands that accept `--extractor-option` also accept `--extractor-options-file`, which has a required argument of the path to a YAML file (with extension `.yaml` or `.yml`) or a JSON file (with extension `.json`). For example:
|
||||
|
||||
- `codeql database create --extractor-options-file options.yml ...`
|
||||
- `codeql database start-tracing --extractor-options-file options.json ...`
|
||||
|
||||
Each option file contains a tree structure of nested maps. At the root is an extractor map key, and beneath it are map keys that correspond to extractor names. Starting at the third level, there are extractor options and option groups.
|
||||
|
||||
In JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"extractor" : {
|
||||
"java": {
|
||||
"option1" : "abc",
|
||||
"group1" : {
|
||||
"option2" : [ 102 ]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In YAML:
|
||||
|
||||
```yaml
|
||||
extractor:
|
||||
java:
|
||||
option1: "abc"
|
||||
group1:
|
||||
option2: [ 102 ]
|
||||
```
|
||||
|
||||
The value for a `string` extractor option must be a string or a number (which will be converted to a string before further processing).
|
||||
|
||||
The value for an `array` extractor option must be an array of strings or numbers.
|
||||
|
||||
The value for an option group (of type `object`) must be a map, which may contain nested extractor options and option groups.
|
||||
|
||||
Each extractor option value must match the regular expression pattern of the extractor option (if it exists), and it must not contain newline characters.
|
||||
|
||||
Assigning an extractor option that does not exist is an error. You can make the {% data variables.product.prodname_codeql_cli %} ignore unknown extractor options by using a special `__allow_unknown_properties` Boolean field. For example, the following option file asks the {% data variables.product.prodname_codeql_cli %} to ignore all unknown extractor options and option groups under `group1`:
|
||||
|
||||
```yaml
|
||||
extractor:
|
||||
java:
|
||||
option1: "abc"
|
||||
group1:
|
||||
__allow_unknown_properties: true
|
||||
option2: [ 102 ]
|
||||
```
|
||||
|
||||
You can specify `--extractor-options-file` multiple times. The extractor option assignments are processed in the following order:
|
||||
|
||||
1. All extractor option files specified by `--extractor-options-file` are processed in the order they appear on the command line, then
|
||||
2. All extractor option assignments specified by `--extractor-option` are processed in the order they appear on the command line
|
||||
|
||||
The same rules govern what happens when the same extractor option is set multiple times, regardless of whether the assignments are done using `--extractor-option`, using `--extractor-options-file`, or some combination of the two. If you set a `string` extractor option multiple times, the last option value overwrites all previous values. If you set an `array` extractor option multiple times, all option values are concatenated in order.
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
title: Getting started with the CodeQL CLI
|
||||
shortTitle: Getting started
|
||||
intro: 'To get started with the {% data variables.product.prodname_codeql_cli %}, you need to set up the CLI so that it can access the tools and libraries required to create and analyze databases.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## Getting started with the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
To run {% data variables.product.prodname_codeql %} commands, you need to set up the CLI so that it can access
|
||||
the tools, queries, and libraries required to create and analyze databases.
|
||||
|
||||
## Setting up the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} can be set up to support many different use cases and directory
|
||||
structures. To get started quickly, we recommend adopting a relatively simple
|
||||
setup, as outlined in the steps below.
|
||||
|
||||
If you use Linux, Windows, or macOS version 10.14 (“Mojave”) or earlier, simply
|
||||
follow the steps below. For macOS version 10.15 (“Catalina”) or newer, steps 1
|
||||
and 4 are slightly different—for further details, see the sections labeled
|
||||
**Information for macOS “Catalina” (or newer) users**. If you are using macOS
|
||||
on Apple Silicon (e.g. Apple M1), ensure that the [Xcode command-line developer
|
||||
tools](https://developer.apple.com/downloads/index.action) and [Rosetta 2](https://support.apple.com/en-us/HT211861) are installed.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** The {% data variables.product.prodname_codeql_cli %} is currently not compatible with non-glibc Linux distributions such as (muslc-based) Alpine Linux.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
For information about installing the {% data variables.product.prodname_codeql_cli %} in a CI system to create results to display in {% data variables.product.prodname_dotcom %} as code scanning alerts, see [Installing {% data variables.product.prodname_codeql_cli %} in your CI system](/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system).
|
||||
|
||||
### 1. Download the {% data variables.product.prodname_codeql_cli %} zip package
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} download package is a zip archive containing tools, scripts, and
|
||||
various {% data variables.product.prodname_codeql %}-specific files. If you don’t have a {% data variables.product.prodname_enterprise %} license then, by
|
||||
downloading this archive, you are agreeing to the [{% data variables.product.prodname_dotcom %} {% data variables.product.prodname_codeql %} Terms and
|
||||
Conditions](https://securitylab.github.com/tools/codeql/license).
|
||||
|
||||
{% note %}
|
||||
|
||||
**Important:** There are several versions of the CLI available to download, depending on your use case:
|
||||
|
||||
- If you want to use the most up to date {% data variables.product.prodname_codeql %} tools and features, download the version tagged `latest`.
|
||||
- If you want to generate code scanning data to upload to {% data variables.product.prodname_enterprise %} server, then download the version that is compatible with the {% data variables.product.prodname_codeql_cli %} used in your CI system. For more information, see “[Installing {% data variables.product.prodname_codeql_cli %} in your CI system](/enterprise-server/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#downloading-the-codeql-cli).”
|
||||
|
||||
{% endnote %}
|
||||
|
||||
If you use Linux, Windows, or macOS version 10.14 (“Mojave”) or earlier, simply
|
||||
[download the zip archive](https://github.com/github/codeql-cli-binaries/releases)
|
||||
for the version you require.
|
||||
|
||||
If you want the CLI for a specific platform, download the appropriate `codeql-PLATFORM.zip` file.
|
||||
Alternatively, you can download `codeql.zip`, which contains the CLI for all supported platforms.
|
||||
|
||||
#### Information for macOS “Catalina” (or newer) users
|
||||
|
||||
If you use macOS version 10.15 (“Catalina”), version 11 (“Big Sur”), or the upcoming version 12 (“Monterey”), you need to ensure that your web browser does not automatically extract zip files. If you use Safari, complete the following steps before downloading the {% data variables.product.prodname_codeql_cli %} zip archive:
|
||||
|
||||
1. Open Safari.
|
||||
2. From the Safari menu, select **Preferences…**.
|
||||
3. Click the **General** Tab.
|
||||
4. Ensure the check-box labeled **Open “safe” files after downloading** is unchecked.
|
||||
|
||||
### 2. Extract the zip archive
|
||||
|
||||
For Linux, Windows, and macOS users (version 10.14 “Mojave”, and earlier)
|
||||
simply extract the zip archive.
|
||||
|
||||
#### Information for macOS “Catalina” (or newer) users
|
||||
|
||||
macOS “Catalina”, “Big Sur”, or “Monterey” users should run the following commands in the Terminal, where `${extraction-root}` is the path to the directory where you will extract the {% data variables.product.prodname_codeql_cli %} zip archive:
|
||||
|
||||
1. `mv ~/Downloads/codeql\*.zip ${extraction-root}`
|
||||
2. `cd ${extraction-root}`
|
||||
3. `/usr/bin/xattr -c codeql\*.zip`
|
||||
4. `unzip codeql\*.zip`
|
||||
|
||||
### 3. Launch `codeql`
|
||||
|
||||
Once extracted, you can run {% data variables.product.prodname_codeql %} processes by running the `codeql` executable in a couple of ways:
|
||||
|
||||
- By executing `<extraction-root>/codeql/codeql`, where `<extraction-root>` is the folder where you extracted the {% data variables.product.prodname_codeql_cli %}
|
||||
package.
|
||||
- By adding `<extraction-root>/codeql` to your `PATH`, so that you
|
||||
can run the executable as just `codeql`.
|
||||
|
||||
At this point, you can execute {% data variables.product.prodname_codeql %} commands. For a full list of the {% data variables.product.prodname_codeql_cli %} commands, see the “[{% data variables.product.prodname_codeql_cli %} manual](https://codeql.github.com/docs/codeql-cli/manual/).”
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** If you add `codeql` to your `PATH`, it can be accessed by {% data variables.product.prodname_codeql %} for Visual Studio Code to compile and run queries.
|
||||
For more information about configuring VS Code to access the {% data variables.product.prodname_codeql_cli %}, see "[Setting up {% data variables.product.prodname_codeql %} in Visual Studio Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code/#setting-up-codeql-in-visual-studio-code)."
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### 4. Verify your {% data variables.product.prodname_codeql_cli %} setup
|
||||
|
||||
{% data variables.product.prodname_codeql_cli %} has subcommands you can execute to verify that you are correctly set up to create and analyze databases:
|
||||
|
||||
- Run `codeql resolve languages` to show which languages are available for database creation. This will list the languages supported by default in your {% data variables.product.prodname_codeql_cli %} package.{% ifversion codeql-packs %}
|
||||
- (Optional) You can download some “[{% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs)” containing pre-compiled queries you would like to run. To do this, run `codeql pack download <pack-name> [...pack-name]`, where `pack-name` is the name of the pack you want to download. The core query packs are a good place to start. They are:
|
||||
|
||||
- `codeql/cpp-queries`
|
||||
- `codeql/csharp-queries`
|
||||
- `codeql/go-queries`
|
||||
- `codeql/java-queries`
|
||||
- `codeql/javascript-queries`
|
||||
- `codeql/python-queries`
|
||||
- `codeql/ruby-queries`
|
||||
|
||||
{% endif %}
|
||||
|
||||
Alternatively, you can download query packs during the analysis by using the `--download` flag of the `codeql database analyze`
|
||||
command.
|
||||
|
||||
## Checking out the {% data variables.product.prodname_codeql %} source code directly
|
||||
|
||||
Some users prefer working with {% data variables.product.prodname_codeql %} query sources directly in order to work on or contribute to the Open Source shared queries. In order to do this, the following steps are recommended. Note that the following instructions are a slightly more complicated alternative to working with {% data variables.product.prodname_codeql %} packages as explained above.
|
||||
|
||||
### 1. Download the {% data variables.product.prodname_codeql_cli %} zip
|
||||
|
||||
Follow [step 1 from the previous section](#1-download-the-codeql-cli-zip-package).
|
||||
|
||||
### 2. Create a new {% data variables.product.prodname_codeql %} directory
|
||||
|
||||
Create a new directory where you can place the CLI and any queries and libraries
|
||||
you want to use. For example, `$HOME/codeql-home`.
|
||||
|
||||
The CLI’s built-in search operations automatically look in all of its sibling
|
||||
directories for the files used in database creation and analysis. Keeping these
|
||||
components in their own directory prevents the CLI searching unrelated sibling
|
||||
directories while ensuring all files are available without specifying any
|
||||
further options on the command line.
|
||||
|
||||
### 3. Obtain a local copy of the {% data variables.product.prodname_codeql %} queries
|
||||
|
||||
The [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql) contains
|
||||
the queries and libraries required for {% data variables.product.prodname_codeql %} analysis of all supported languages.
|
||||
Clone a copy of this repository into `codeql-home`.
|
||||
|
||||
By default, the root of the cloned repository will be called `codeql`.
|
||||
Rename this folder `codeql-repo` to avoid conflicting with the {% data variables.product.prodname_codeql_cli %} that you will extract in step 4. If you use git on the command line, you can
|
||||
clone and rename the repository in a single step by running
|
||||
`git clone git@github.com:github/codeql.git codeql-repo` in the `codeql-home` folder.
|
||||
|
||||
{% ifversion ghes < 3.6 or ghae < 3.6 %}
|
||||
{% note %}
|
||||
|
||||
**Note:** The {% data variables.product.prodname_codeql %} libraries and queries for Go analysis used to live in a separate [{% data variables.product.prodname_codeql %} for Go repository](https://github.com/github/codeql-go/). These have been moved to the `github/codeql` repository. It is no longer necessary to clone the `github/codeql-go` into a separate `codeql-home/codeql-go` folder.
|
||||
|
||||
For more information, see the [Relocation announcement](https://github.com/github/codeql-go/issues/741).
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
{% ifversion codeql-packs %}
|
||||
|
||||
Within this repository, the queries and libraries are organized into {% data variables.product.prodname_codeql %}
|
||||
packs. Along with the queries themselves, {% data variables.product.prodname_codeql %} packs contain important metadata
|
||||
that tells the {% data variables.product.prodname_codeql_cli %} how to process the query files. For more information,
|
||||
see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).”
|
||||
{% endif %}
|
||||
|
||||
{% note %}
|
||||
|
||||
**Important:** There are different versions of the {% data variables.product.prodname_codeql %} queries available for different users. Check out the correct version for your use case:
|
||||
|
||||
- For the queries that are intended to be used with the latest {% data variables.product.prodname_codeql_cli %} release, check out the branch tagged `codeql-cli/latest`. You should use this branch for databases you’ve built using the {% data variables.product.prodname_codeql_cli %}, fetched from code scanning on {% data variables.product.prodname_dotcom %}, or recently downloaded from {% data variables.product.prodname_dotcom_the_website %}.
|
||||
- For the most up to date {% data variables.product.prodname_codeql %} queries, check out the `main` branch. This branch represents the very latest version of {% data variables.product.prodname_codeql %}’s analysis.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### 4. Extract the zip archive
|
||||
|
||||
For Linux, Windows, and macOS users (version 10.14 “Mojave”, and earlier) simply extract the zip archive into the directory you created in step 2.
|
||||
|
||||
For example, if the path to your copy of the {% data variables.product.prodname_codeql %} repository is `$HOME/codeql-home/codeql-repo`, then extract the CLI into
|
||||
`$HOME/codeql-home/`.
|
||||
|
||||
### 5. Launch `codeql`
|
||||
|
||||
See [step 3 from the previous section](#3-launch-codeql).
|
||||
|
||||
### 6. Verify your {% data variables.product.prodname_codeql_cli %} setup
|
||||
|
||||
{% data variables.product.prodname_codeql_cli %} has subcommands you can execute to verify that you are correctly set up to create and analyze databases:
|
||||
|
||||
- Run `codeql resolve languages` to show which languages are available for database creation. This will list the languages supported by default in your {% data variables.product.prodname_codeql_cli %} package.
|
||||
- Run `codeql resolve qlpacks` to show which {% data variables.product.prodname_codeql %} packs the CLI can find. This will display the names of all the {% data variables.product.prodname_codeql %} packs directly available to the {% data variables.product.prodname_codeql_cli %}. This should include:
|
||||
- Query packs for each supported language, for example, `codeql/{language}-queries`. These packs contain the standard queries that will be run for each analysis.
|
||||
- Library packs for each supported language, for example, `codeql/{language}-all`. These packs contain query libraries, such as control flow and data flow libraries, that may be useful to query writers.
|
||||
- Example packs for each supported language, for example, `codeql/{language}-examples`. These packs contain useful snippets of {% data variables.product.prodname_codeql %} that query writers may find useful.
|
||||
- Legacy packs that ensure custom queries and libraries created using older products are compatible with your version of {% data variables.product.prodname_codeql %}.
|
||||
|
||||
## Using two versions of the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
If you want to use the latest {% data variables.product.prodname_codeql %} features to execute queries or {% data variables.product.prodname_codeql %} tests, but also want to prepare databases that are compatible with a specific version of {% data variables.product.prodname_codeql %} code scanning on {% data variables.product.prodname_ghe_server %}, you may need to install two versions of the CLI. The recommended directory setup depends on which versions you want to install:
|
||||
|
||||
- If both versions are 2.0.2 (or newer), you can unpack both CLI archives in the same parent directory.
|
||||
- If at least one of the versions is 2.0.1 (or older), the unpacked CLI archives cannot be in the same parent directory, but they can share the same grandparent directory. For example, if you unpack version 2.0.2 into `$HOME/codeql-home/codeql-cli`, the older version should be unpacked into `$HOME/codeql-older-version/old-codeql-cli`. Here, the common grandparent is the `$HOME` directory.
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Using the CodeQL CLI
|
||||
intro: 'You can use the {% data variables.product.prodname_codeql_cli %} to locally develop, test and run CodeQL queries on software projects.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
children:
|
||||
- /about-the-codeql-cli
|
||||
- /getting-started-with-the-codeql-cli
|
||||
- /creating-codeql-databases
|
||||
- /extractor-options
|
||||
- /analyzing-databases-with-the-codeql-cli
|
||||
- /using-custom-queries-with-the-codeql-cli
|
||||
- /creating-codeql-query-suites
|
||||
- /testing-custom-queries
|
||||
- /testing-query-help-files
|
||||
- /creating-and-working-with-codeql-packs
|
||||
- /publishing-and-using-codeql-packs
|
||||
- /specifying-command-options-in-a-codeql-configuration-file
|
||||
---
|
||||
@@ -0,0 +1,187 @@
|
||||
---
|
||||
title: Publishing and using CodeQL packs
|
||||
intro: 'You can publish your own {% data variables.product.prodname_codeql %} packs and use packs published by other people.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
feature: codeql-packs
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
{% data reusables.codeql-cli.beta-note-package-management %}
|
||||
|
||||
## Configuring the `qlpack.yml` file before publishing
|
||||
|
||||
{% ifversion ghes %}
|
||||
{% note %}
|
||||
|
||||
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/enterprise-cloud@latest/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs)”, the Enterprise Cloud version, for the most up-to-date documentation.
|
||||
|
||||
{% endnote %}
|
||||
{% endif %}
|
||||
|
||||
You can check and modify the configuration details of your {% data variables.product.prodname_codeql %} pack prior to publishing. Open the `qlpack.yml` file in your preferred text editor.
|
||||
|
||||
```yml
|
||||
library: # set to true if the pack is a library. Set to false or omit for a query pack
|
||||
name: <scope>/<pack>
|
||||
version: <x.x.x>
|
||||
description: <Description to publish with the package>
|
||||
default-suite: # optional, one or more queries in the pack to run by default
|
||||
- query: <relative-path>/query-file>.ql
|
||||
default-suite-file: default-queries.qls # optional, a pointer to a query-suite in this pack
|
||||
license: # optional, the license under which the pack is published
|
||||
dependencies: # map from CodeQL pack name to version range
|
||||
```
|
||||
|
||||
* `name:` must follow the <scope>/<pack> format, where <scope> is the {% data variables.product.prodname_dotcom %} organization that you will publish to and <pack> is the name for the pack.
|
||||
|
||||
|
||||
* A maximum of one of `default-suite` or `default-suite-file` is allowed. These are two different ways to define a default query suite to be run, the first by specifying queries directly in the qlpack.yml file and the second by specifying a query suite in the pack.
|
||||
|
||||
## Running `codeql pack publish`
|
||||
|
||||
When you are ready to publish a pack to the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}, you can run the following command in the root of the pack directory:
|
||||
|
||||
```
|
||||
codeql pack publish
|
||||
```
|
||||
|
||||
The published package will be displayed in the packages section of {% data variables.product.prodname_dotcom %} organization specified by the scope in the `qlpack.yml` file.
|
||||
|
||||
## Running `codeql pack download <scope>/<pack>`
|
||||
|
||||
To run a pack that someone else has created, you must first download it by running the following command:
|
||||
|
||||
```
|
||||
codeql pack download <scope>/<pack>@x.x.x
|
||||
```
|
||||
|
||||
* `<scope>`: the name of the {% data variables.product.prodname_dotcom %} organization that you will download from.
|
||||
* `<pack>`: the name for the pack that you want to download.
|
||||
* `@x.x.x`: an optional version number. If omitted, the latest version will be downloaded.
|
||||
|
||||
This command accepts arguments for multiple packs.
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
If you write scripts that specify a particular version number of a
|
||||
query pack to download, keep in mind that when you update your version of
|
||||
{% data variables.product.prodname_codeql %} to a newer one, you may
|
||||
also need to switch to a newer version of the query pack. Newer
|
||||
versions of {% data variables.product.prodname_codeql %} _may_ provide
|
||||
degraded performance when used with query packs that have been pinned
|
||||
to a very old version. For more information, see "[About {% data variables.product.prodname_codeql %}
|
||||
pack compatibility](#about-codeql-pack-compatibility)."
|
||||
{% endif %}
|
||||
|
||||
## Using a {% data variables.product.prodname_codeql %} pack to analyze a {% data variables.product.prodname_codeql %} database
|
||||
|
||||
To analyze a {% data variables.product.prodname_codeql %} database with a {% data variables.product.prodname_codeql %} pack, run the following command:
|
||||
|
||||
```
|
||||
codeql database analyze <database> <scope>/<pack>@x.x.x:<path>
|
||||
```
|
||||
* `<database>`: the {% data variables.product.prodname_codeql %} database to be analyzed.
|
||||
* `<scope>`: the name of the {% data variables.product.prodname_dotcom %} organization that the pack is published to.
|
||||
* `<pack>`: the name for the pack that you are using.
|
||||
* `@x.x.x`: an optional version number. If omitted, the latest version will be used.
|
||||
* `:<path>`: an optional path to a query, directory, or query suite. If omitted, the pack’s default query suite will be used.
|
||||
|
||||
The `analyze` command will run the default suite of any specified {% data variables.product.prodname_codeql %} packs. You can specify multiple {% data variables.product.prodname_codeql %} packs to be used for analyzing a {% data variables.product.prodname_codeql %} database. For example:
|
||||
|
||||
```
|
||||
codeql <database> analyze <scope>/<pack> <scope>/<other-pack>
|
||||
```
|
||||
|
||||
{% ifversion query-pack-compatibility %}
|
||||
{% note %}
|
||||
|
||||
**Note:** The `codeql pack download` command stores the pack it downloads in an internal location that is not intended for local modification. Unexpected (and hard to troubleshoot) behavior may result if the pack is modified after downloading. For more information about customizing packs, see “[Creating and working with {% data variables.product.prodname_codeql %} packs](#creating-and-working-with-codeql-packs)."
|
||||
|
||||
{% endnote %}
|
||||
|
||||
## About {% data variables.product.prodname_codeql %} pack compatibility
|
||||
|
||||
When a query pack is published, it includes pre-compiled representations of all the queries in it. These pre-compiled queries are generally much faster to execute than it is to compile the QL source from scratch during the analysis. However, the pre-compiled queries also depend on certain internals of the QL evaluator, so if the version of {% data variables.product.prodname_codeql %} that performs the analysis is too different from the version that ran `codeql pack publish`, it may be necessary to compile the queries from source instead during analysis. The recompilation happens automatically and will not affect the _results_ of the analysis, but it can make the
|
||||
analysis significantly slower.
|
||||
|
||||
It can generally be assumed that if a pack is published with one release of {% data variables.product.prodname_codeql %}, the precompiled queries in it can be used directly by _later_ releases of {% data variables.product.prodname_codeql %}, as long as there is no more than 6 months between the release dates. We will make reasonable efforts to keep new releases compatible for longer than that, but make no promises.
|
||||
|
||||
It can also be assumed that a pack published by the _latest_ public release of {% data variables.product.prodname_codeql %} will be useable by the version of {% data variables.product.prodname_codeql %} that is used by {% data variables.product.prodname_code_scanning %} and {% data variables.product.prodname_actions %}, even though that is often a slightly older release.
|
||||
|
||||
As an exception to the above, packs published with versions of {% data variables.product.prodname_codeql %} _earlier than 2.12.0_ are not compatible with any earlier or later versions. These old versions did not write pre-compiled queries in a format that supported compatibility between releases. Packs published by these versions can still be _used_ by newer versions, but the analysis will be slower because the queries have to be recompiled first.
|
||||
|
||||
As a user of a published query pack, you can check that the {% data variables.product.prodname_codeql %} makes use of the precompiled queries in it by inspecting the terminal output from an analysis runs that uses the query pack. If it contains lines looking like the following, then the precompiled queries were used successfully:
|
||||
```
|
||||
[42/108] Loaded /long/path/to/query/Filename.qlx.
|
||||
```
|
||||
However, if they instead look like the following, then usage of the precompiled queries failed:
|
||||
```
|
||||
Compiling query plan for /long/path/to/query/Filename.ql.
|
||||
[42/108 comp 25s] Compiled /long/path/to/query/Filename.ql.
|
||||
```
|
||||
The results of the analysis will still be good in this case, but to get optimal performance you may need to upgrade to a newer version of the {% data variables.product.prodname_codeql %} CLI and/or of the query pack.
|
||||
|
||||
If you publish query packs on the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %} for others to use, we recommend that you use a recent release of {% data variables.product.prodname_codeql %} to run `codeql pack publish`, and that you publish a fresh version of your pack with an updated {% data variables.product.prodname_codeql %} version before the version you used turns 6 months old. That way you can ensure that users of your pack who keep _their_ {% data variables.product.prodname_codeql %} up to date will benefit from the pre-compiled queries in your pack.
|
||||
|
||||
If you publish query packs with the intention of using them on a {% data variables.product.prodname_ghe_server %} installation that uses its bundled {% data variables.product.prodname_codeql %} binaries, use the same {% data variables.product.prodname_codeql %} version to run `codeql pack publish`. Newer versions might produce pre-compiled queries that the one in {% data variables.product.prodname_ghe_server %} may not recognize. Your {% data variables.product.prodname_ghe_server %} administrator may choose to upgrade to a newer version of {% data variables.product.prodname_codeql %} periodically. If so, follow their lead.
|
||||
|
||||
{% endif %}
|
||||
|
||||
{% ifversion ghes %}
|
||||
|
||||
## Working with {% data variables.product.prodname_codeql %} packs on {% data variables.product.prodname_ghe_server %}
|
||||
|
||||
By default, the {% data variables.product.prodname_codeql_cli %} expects to download {% data variables.product.prodname_codeql %} packs from and publish packs to the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %}. However, you can also work with {% data variables.product.prodname_codeql %} packs in a {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_ghe_server %} by creating a `qlconfig.yml` file to tell the CLI which {% data variables.product.prodname_container_registry %} to use for each pack.
|
||||
|
||||
Create a `~/.codeql/qlconfig.yml` file using your preferred text editor, and add entries to specify which registry to use for one or more package name patterns.
|
||||
For example, the following `qlconfig.yml` file associates all packs with the {% data variables.product.prodname_container_registry %} for the {% data variables.product.prodname_ghe_server %} at `GHE_HOSTNAME`, except packs matching `codeql/\*`, which are associated with the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %}:
|
||||
|
||||
```yaml
|
||||
registries:
|
||||
- packages:
|
||||
- 'codeql/*'
|
||||
- 'other-org/*'
|
||||
url: https://ghcr.io/v2/
|
||||
- packages: '*'
|
||||
url: https://containers.GHE_HOSTNAME/v2/
|
||||
```
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} will determine which registry to use for a given package name by finding the first item in the `registries` list with a `packages` property that matches that package name.
|
||||
This means that you’ll generally want to define the most specific package name patterns first. The `packages` property may be a single package name, a glob pattern, or a YAML list of package names and glob patterns.
|
||||
|
||||
The `registries` list can also be placed inside of a `codeql-workspace.yml` file. Doing so will allow you to define the registries to be used within a specific workspace, so that it can be shared amongst other {% data variables.product.prodname_codeql %} users of the workspace. The `registries` list in the `codeql-workspace.yml` will be merged with and take precedence over the list in the global `qlconfig.yml`. For more information about `codeql-workspace.yml`, see [About {% data variables.product.prodname_codeql %} workspaces](/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces#about-codeql-workspaces).
|
||||
|
||||
You can now use `codeql pack publish`, `codeql pack download`, and `codeql database analyze` to manage packs on {% data variables.product.prodname_ghe_server %}.
|
||||
|
||||
{% endif %}
|
||||
|
||||
## Authenticating to {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registries %}
|
||||
|
||||
You can publish packs and download private packs by authenticating to the appropriate {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}.
|
||||
|
||||
You can authenticate to the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %} in two ways:
|
||||
|
||||
1. Pass the `--github-auth-stdin` option to the {% data variables.product.prodname_codeql_cli %}, then supply a {% data variables.product.prodname_github_apps %} token or {% data variables.product.pat_generic %} via standard input.
|
||||
2. Set the `GITHUB_TOKEN` environment variable to a {% data variables.product.prodname_github_apps %} token or {% data variables.product.pat_generic %}.
|
||||
|
||||
{% ifversion ghes %}
|
||||
|
||||
Similarly, you can authenticate to a {% data variables.product.prodname_ghe_server %} {% data variables.product.prodname_container_registry %}, or authenticate to multiple registries simultaneously (for example, to download or run private packs from multiple registries) in two ways:
|
||||
|
||||
1. Pass the `--registries-auth-stdin` option to the {% data variables.product.prodname_codeql_cli %}, then supply a registry authentication string via standard input.
|
||||
2. Set the `CODEQL_REGISTRIES_AUTH` environment variable to a registry authentication string.
|
||||
|
||||
A registry authentication string is a comma-separated list of `<registry-url>=<token>` pairs, where `registry-url` is a {% data variables.product.prodname_container_registry %} URL, such as `https://containers.GHE_HOSTNAME/v2/`, and `token` is a {% data variables.product.prodname_github_apps %} token or {% data variables.product.pat_generic %} for that {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}.
|
||||
This ensures that each token is only passed to the {% data variables.product.prodname_container_registry %} you specify.
|
||||
For instance, the following registry authentication string specifies that the {% data variables.product.prodname_codeql_cli %} should authenticate to the {% data variables.product.prodname_container_registry %} on {% data variables.product.prodname_dotcom_the_website %} using the token `<token1>` and to the {% data variables.product.prodname_container_registry %} for the GHES instance at `GHE_HOSTNAME` using the token `<token2>`:
|
||||
|
||||
```
|
||||
https://ghcr.io/v2/=<token1>,https://containers.GHE_HOSTNAME/v2/=<token2>
|
||||
```
|
||||
|
||||
{% endif %}
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: Specifying command options in a CodeQL configuration file
|
||||
intro: 'You can save default command options in a {% data variables.product.prodname_codeql %} configuration file.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About specifying command options in a {% data variables.product.prodname_codeql %} configuration file
|
||||
|
||||
You can save default or frequently used options for your commands in a per-user configuration file.
|
||||
|
||||
You can specify {% data variables.product.prodname_codeql_cli %} command options in two ways:
|
||||
|
||||
- Directly in the command line, using the appropriate flag.
|
||||
- In a configuration (or `config`) file that {% data variables.product.prodname_codeql %} scans for relevant options each time a command is executed.
|
||||
|
||||
For options that are likely to change each time you execute a command, specifying the value on the command line is the most convenient way of passing the information to {% data variables.product.prodname_codeql %}. Saving options in a `config` file is a good way to specify options you use frequently. It’s also a good way to add custom {% data variables.product.prodname_codeql %} packs that you use regularly to your search path.
|
||||
|
||||
## Using a {% data variables.product.prodname_codeql %} configuration file
|
||||
|
||||
You need to save the `config` file under your home (Linux and macOS) or user profile (Windows) directory in the `.config/codeql/` subdirectory. For example, `$HOME/.config/codeql/config`.
|
||||
|
||||
The syntax for specifying options is as follows:
|
||||
|
||||
```
|
||||
<command> <subcommand> <option> <value>
|
||||
```
|
||||
|
||||
To apply the same options to more than one command you can:
|
||||
|
||||
- Omit the `<subcommand>`, which will specify the option for every `<subcommand>` to which it’s relevant.
|
||||
- Omit both `<command>` and `<subcommand>`, which will globally specify the option for every `<command>` and `<subcommand>` to which it’s relevant.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Notes:**
|
||||
- `config` files only accept spaces between between option flags and values—{% data variables.product.prodname_codeql %} will throw an error if you use `=` to specify an option value.
|
||||
- If you specify an option in the command line, this overrides the `config` value defined for that option.
|
||||
- If you want to specify more than one option for a `<command>`, `<subcommand>` or globally, use one line per option.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
### Examples
|
||||
|
||||
* To output all analysis results generated by `codeql database analyze` as CSV format, you would specify:
|
||||
|
||||
```
|
||||
database analyze --format csv
|
||||
```
|
||||
|
||||
Here, you have to specify the command and subcommand to prevent any of the low-level commands that are executed during `database analyze` being passed the same `--format` option.
|
||||
|
||||
- To define the RAM (4096 MB) and number of threads (4) to use when running {% data variables.product.prodname_codeql %} commands, specify the following, on separate lines:
|
||||
|
||||
```
|
||||
--ram 4096
|
||||
--threads 4
|
||||
```
|
||||
|
||||
- To globally specify a directory for {% data variables.product.prodname_codeql %} to scan for {% data variables.product.prodname_codeql %} packs (which is not a sibling of the installation directory), use:
|
||||
|
||||
```
|
||||
--search-path <path-to-directory>
|
||||
```
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: Testing custom queries
|
||||
intro: 'You can set up tests for your {% data variables.product.prodname_codeql %} queries to ensure that they continue to return the expected results with new releases of the {% data variables.product.prodname_codeql_cli %}.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
feature: codeql-packs
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About testing custom queries
|
||||
|
||||
{% data variables.product.prodname_codeql %} provides a simple test framework for automated regression testing
|
||||
of queries. Test your queries to ensure that they behave as expected.
|
||||
|
||||
During a query test, {% data variables.product.prodname_codeql %} compares the results the user expects
|
||||
the query to produce with those actually produced. If the expected and
|
||||
actual results differ, the query test fails. To fix the test, you should iterate
|
||||
on the query and the expected results until the actual results and the expected
|
||||
results exactly match. This topic shows you how to create test files and execute
|
||||
tests on them using the `test run` subcommand.
|
||||
|
||||
## Setting up a test {% data variables.product.prodname_codeql %} pack for custom queries
|
||||
|
||||
All {% data variables.product.prodname_codeql %} tests must be stored in a special “test” {% data variables.product.prodname_codeql %} pack. That is, a directory for test files with a `qlpack.yml` file that defines:
|
||||
|
||||
```yaml
|
||||
name: <name-of-test-pack>
|
||||
version: 0.0.0
|
||||
dependencies:
|
||||
<codeql-libraries-and-queries-to-test>: "*"
|
||||
extractor: <language-of-code-to-test>
|
||||
```
|
||||
|
||||
The `dependencies` value specifies the {% data variables.product.prodname_codeql %} packs containing queries to test.
|
||||
Typically, these packs will be resolved from source, and so it is not necessary
|
||||
to specify a fixed version of the pack. The `extractor` defines which language the CLI will use to create test databases from the code files stored in this {% data variables.product.prodname_codeql %} pack. For more information, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).”
|
||||
|
||||
You may find it useful to look at the way query tests are organized in the [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql). Each language has a `src` directory, `ql/<language>/ql/src`, that contains libraries and queries for analyzing codebases. Alongside the `src` directory, there is a `test` directory with tests for
|
||||
these libraries and queries.
|
||||
|
||||
Each `test` directory is configured as a test {% data variables.product.prodname_codeql %} pack with two subdirectories:
|
||||
|
||||
- `query-tests` a series of subdirectories with tests for queries stored in the `src` directory. Each subdirectory contains test code and a QL reference file that specifies the query to test.
|
||||
- `library-tests` a series of subdirectories with tests for QL library files. Each subdirectory contains test code and queries that were written as unit tests for a library.
|
||||
|
||||
## Setting up the test files for a query
|
||||
|
||||
For each query you want to test, you should create a sub-directory in the test {% data variables.product.prodname_codeql %} pack.
|
||||
Then add the following files to the subdirectory before you run the test command:
|
||||
|
||||
- A query reference file (`.qlref` file) defining the location of the query to test. The location is defined relative to the root of the {% data variables.product.prodname_codeql %} pack that contains the query. Usually, this is a {% data variables.product.prodname_codeql %} pack specified in the `dependencies` block of the test pack. For more information, see “[Query reference files](/code-security/code-scanning/codeql-cli-reference/query-reference-files).”
|
||||
|
||||
You do not need to add a query reference file if the query you want to test is stored in the test directory, but it is generally good practice to store queries separately from tests. The only exception is unit tests for QL libraries, which tend to be stored in test packs, separate from queries that generate alerts or paths.
|
||||
|
||||
- The example code you want to run your query against. This should consist of one or more files containing examples of the code the query is designed to identify.
|
||||
|
||||
You can also define the results you expect to see when you run the query against
|
||||
the example code, by creating a file with the extension `.expected`. Alternatively, you can leave the test command to create the `.expected` file for you.
|
||||
|
||||
For an example showing how to create and test a query, see the [example](#example) below.
|
||||
|
||||
{% note %}
|
||||
|
||||
**Important:** Your `.ql`, `.qlref`, and `.expected` files must have consistent names.
|
||||
|
||||
If you want to directly specify the `.ql` file itself in the test command, it must have the same base name as the corresponding `.expected` file. For example, if the query is `MyJavaQuery.ql`, the expected results file must be `MyJavaQuery.expected`.
|
||||
|
||||
If you want to specify a `.qlref` file in the command, it must have the same base name as the corresponding `.expected` file, but the query itself may have a different name.
|
||||
|
||||
The names of the example code files don’t have to be consistent with the other test files. All example code files found next to the `.qlref` (or `.ql`) file and in any subdirectories will be used to create a test database. Therefore, for simplicity, we recommend you don’t save test files in directories that are ancestors of each other.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
## Running `codeql test run`
|
||||
|
||||
{% data variables.product.prodname_codeql %} query tests are executed by running the following command:
|
||||
|
||||
```
|
||||
codeql test run <test|dir>
|
||||
```
|
||||
|
||||
The `<test|dir>` argument can be one or more of the following:
|
||||
|
||||
- Path to a `.ql` file.
|
||||
- Path to a `.qlref` file that references a `.ql` file.
|
||||
- Path to a directory that will be searched recursively for `.ql` and `.qlref` files.
|
||||
|
||||
You can also specify:
|
||||
|
||||
- `--threads:` optionally, the number of threads to use when running queries. The default option is `1`. You can specify more threads to speed up query execution. Specifying `0` matches the number of threads to the number of logical processors.
|
||||
|
||||
For full details of all the options you can use when testing queries, see the [test run reference documentation](https://codeql.github.com/docs/codeql-cli/manual/test-run/).
|
||||
|
||||
## Example
|
||||
|
||||
The following example shows you how to set up a test for a query that searches
|
||||
Java code for `if` statements that have empty `then` blocks. It includes
|
||||
steps to add the custom query and corresponding test files to separate {% data variables.product.prodname_codeql %} packs
|
||||
outside your checkout of the {% data variables.product.prodname_codeql %} repository. This ensures when you update the
|
||||
{% data variables.product.prodname_codeql %} libraries, or check out a different branch, you won’t overwrite your
|
||||
custom queries and tests.
|
||||
|
||||
### Prepare a query and test files
|
||||
|
||||
1. Develop the query. For example, the following simple query finds empty `then`
|
||||
blocks in Java code:
|
||||
|
||||
```
|
||||
import java
|
||||
|
||||
from IfStmt ifstmt
|
||||
where ifstmt.getThen() instanceof EmptyStmt
|
||||
select ifstmt, "This if statement has an empty then."
|
||||
```
|
||||
|
||||
2. Save the query to a file named `EmptyThen.ql` in a directory with your
|
||||
other custom queries. For example, `custom-queries/java/queries/EmptyThen.ql`.
|
||||
|
||||
3. If you haven’t already added your custom queries to a {% data variables.product.prodname_codeql %} pack, create a {% data variables.product.prodname_codeql %} pack now. For example, if your custom Java queries are stored in `custom-queries/java/queries`, add a `qlpack.yml` file with the following contents to `custom-queries/java/queries`:
|
||||
|
||||
```yaml
|
||||
name: my-custom-queries
|
||||
dependencies:
|
||||
codeql/java-queries: "*"
|
||||
```
|
||||
|
||||
For more information about {% data variables.product.prodname_codeql %} packs, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).”
|
||||
|
||||
4. Create a {% data variables.product.prodname_codeql %} pack for your Java tests by adding a `qlpack.yml` file with the following contents to `custom-queries/java/tests`, updating the `dependencies` to match the name of your {% data variables.product.prodname_codeql %} pack of custom queries:
|
||||
|
||||
{% data reusables.codeql-cli.test-qlpack %}
|
||||
|
||||
5. Within the Java test pack, create a directory to contain the test files
|
||||
associated with `EmptyThen.ql`. For example, `custom-queries/java/tests/EmptyThen`.
|
||||
|
||||
6. In the new directory, create `EmptyThen.qlref` to define the location of `EmptyThen.ql`. The path to the query must be specified relative to the root of
|
||||
the {% data variables.product.prodname_codeql %} pack that contains the query. In this case, the query is in the
|
||||
top level directory of the {% data variables.product.prodname_codeql %} pack named `my-custom-queries`,
|
||||
which is declared as a dependency for `my-query-tests`. Therefore, `EmptyThen.qlref` should simply contain `EmptyThen.ql`.
|
||||
|
||||
7. Create a code snippet to test. The following Java code contains an empty `if` statement on the third line. Save it in `custom-queries/java/tests/EmptyThen/Test.java`.
|
||||
|
||||
```java
|
||||
class Test {
|
||||
public void problem(String arg) {
|
||||
if (arg.isEmpty())
|
||||
;
|
||||
{
|
||||
System.out.println("Empty argument");
|
||||
}
|
||||
}
|
||||
|
||||
public void good(String arg) {
|
||||
if (arg.isEmpty()) {
|
||||
System.out.println("Empty argument");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Execute the test
|
||||
|
||||
To execute the test, move into the `custom-queries` directory and run `codeql
|
||||
test run java/tests/EmptyThen`.
|
||||
|
||||
When the test runs, it:
|
||||
|
||||
1. Finds one test in the `EmptyThen` directory.
|
||||
|
||||
2. Extracts a {% data variables.product.prodname_codeql %} database from the `.java` files stored in the `EmptyThen` directory.
|
||||
|
||||
3. Compiles the query referenced by the `EmptyThen.qlref` file.
|
||||
|
||||
If this step fails, it’s because the CLI can’t find your custom {% data variables.product.prodname_codeql %} pack. Re-run the command and specify the location of your custom {% data variables.product.prodname_codeql %} pack, for example:
|
||||
|
||||
`codeql test run --search-path=java java/tests/EmptyThen`
|
||||
|
||||
For information about saving the search path as part of your configuration, see "[Specifying command options in a {% data variables.product.prodname_codeql %} configuration file](/code-security/code-scanning/using-the-codeql-cli/specifying-command-options-in-a-codeql-configuration-file)."
|
||||
|
||||
4. Executes the test by running the query and generating an `EmptyThen.actual` results file.
|
||||
|
||||
5. Checks for an `EmptyThen.expected` file to compare with the `.actual` results file.
|
||||
|
||||
6. Reports the results of the test — in this case, a failure: `0 tests passed; 1 tests failed:`. The test failed because we haven’t yet added a file with the expected results of the query.
|
||||
|
||||
### View the query test output
|
||||
|
||||
{% data variables.product.prodname_codeql %} generates the following files in the `EmptyThen` directory:
|
||||
|
||||
- `EmptyThen.actual`, a file that contains the actual results generated by the
|
||||
query.
|
||||
- `EmptyThen.testproj`, a test database that you can load into VS Code and use to debug failing tests. When tests complete successfully, this database is deleted in a housekeeping step. You can override this step by running `test run` with the `--keep-databases` option.
|
||||
|
||||
In this case, the failure was expected and is easy to fix. If you open the `EmptyThen.actual` file, you can see the results of the test:
|
||||
|
||||
```
|
||||
| Test.java:3:5:3:22 | stmt | This if statement has an empty then. |
|
||||
```
|
||||
|
||||
This file contains a table, with a column for the location of the result,
|
||||
along with separate columns for each part of the `select` clause the query outputs.
|
||||
Since the results are what we expected, we can update the file extension to define
|
||||
this as the expected result for this test (`EmptyThen.expected`).
|
||||
|
||||
If you rerun the test now, the output will be similar but it will finish by reporting: `All 1 tests passed.`.
|
||||
|
||||
If the results of the query change, for example, if you revise the `select` statement for the query, the test will fail. For failed results, the CLI output includes a unified diff of the `EmptyThen.expected` and `EmptyThen.actual` files.
|
||||
This information may be sufficient to debug trivial test failures.
|
||||
|
||||
For failures that are harder to debug, you can import `EmptyThen.testproj`
|
||||
into {% data variables.product.prodname_codeql %} for VS Code, execute `EmptyThen.ql`, and view the results in the
|
||||
`Test.java` example code. For more information, see “[Analyzing your projects](https://codeql.github.com/docs/codeql-for-visual-studio-code/analyzing-your-projects/#analyzing-your-projects)” in the {% data variables.product.prodname_codeql %} for VS Code help.
|
||||
|
||||
## Further reading
|
||||
- “[{% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/codeql-queries/#codeql-queries)”
|
||||
- "[Testing {% data variables.product.prodname_codeql %} queries in Visual Studio Code](https://codeql.github.com/docs/codeql-for-visual-studio-code/testing-codeql-queries-in-visual-studio-code/#testing-codeql-queries-in-visual-studio-code)."
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: Testing query help files
|
||||
intro: You can use the {% data variables.product.prodname_codeql_cli %} to preview your query help files as Markdown and ensure they are valid.
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About testing query help files
|
||||
|
||||
Test query help files by rendering them as Markdown to ensure they are valid before uploading them to the {% data variables.product.prodname_codeql %} repository or using them in code scanning.
|
||||
|
||||
<!-- capitalized Markdown -->
|
||||
|
||||
Query help is documentation that accompanies a query to explain how the query works, as well as providing information about the potential problem that the query identifies. It is good practice to write query help for all new queries. For more information, see [Contributing to {% data variables.product.prodname_codeql %}](https://github.com/github/codeql/blob/main/CONTRIBUTING.md) in the {% data variables.product.prodname_codeql %} repository.
|
||||
|
||||
The {% data variables.product.prodname_codeql_cli %} includes a command to test query help and render the content as markdown, so that you can easily preview the content in your IDE. Use the command to validate query help files before uploading them to the {% data variables.product.prodname_codeql %} repository or sharing them with other users. From {% data variables.product.prodname_codeql_cli %} 2.7.1 onwards, you can also include the markdown-rendered query help in SARIF files
|
||||
generated during {% data variables.product.prodname_codeql %} analyses so that the query help can be displayed in the code scanning UI. For more information, see
|
||||
“[Analyzing databases with the {% data variables.product.prodname_codeql_cli %}](/code-security/code-scanning/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli).”
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The query help (`.qhelp`) file must have an accompanying query (`.ql`) file with an identical base name.
|
||||
- The query help file should follow the standard structure and style for query help documentation. For more information, see the [Query help style guide](https://github.com/github/codeql/blob/main/docs/query-help-style-guide.md) in the {% data variables.product.prodname_codeql %} repository.
|
||||
|
||||
## Running `codeql generate query-help`
|
||||
|
||||
You can test query help files by running the following command:
|
||||
|
||||
```
|
||||
codeql generate query-help <qhelp|query|dir|suite> --format=<format> [--output=<dir|file>]
|
||||
```
|
||||
|
||||
where `<qhelp|query|dir|suite>` is one of:
|
||||
|
||||
- the path to a `.qhelp` file.
|
||||
- the path to a `.ql` file.
|
||||
- the path to a directory containing queries and query help files.
|
||||
- the path to a query suite, or the name of a well-known query suite for a {% data variables.product.prodname_codeql %} pack. For more information, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites).”
|
||||
|
||||
You must specify a `--format` option, which defines how the query help is rendered. Currently, you must specify `markdown` to render the query help as markdown.
|
||||
|
||||
The `--output` option defines a file path where the rendered query help will be saved.
|
||||
|
||||
- For directories containing `.qhelp` files or a query suites defining one or more `.qhelp` files, you must specify an `--output` directory. Filenames within the output directory will be derived from the `.qhelp` file names.
|
||||
- For single `.qhelp` or `.ql` files, you may specify an `--output` option. If you don’t specify an output path, the rendered query help is written to `stdout`.
|
||||
|
||||
For full details of all the options you can use when testing query help files, see the [generate query-help reference documentation](https://codeql.github.com/docs/codeql-cli/manual/generate-query-help/).
|
||||
|
||||
## Results
|
||||
|
||||
When you run the command, {% data variables.product.prodname_codeql %} attempts to render each `.qhelp` file that has an accompanying `.ql` file. For single files, the rendered content will be printed to `stdout` if you don’t specify an `--output` option. For all other use cases, the rendered content is saved to the specified output path.
|
||||
|
||||
By default, the {% data variables.product.prodname_codeql_cli %} will print a warning message if:
|
||||
|
||||
- Any of the query help is invalid, along with a description of the invalid query help elements
|
||||
- Any `.qhelp` files specified in the command don’t have the same base name as an accompanying `.ql` file
|
||||
- Any `.ql` files specified in the command don’t have the same base name as an accompanying `.qhelp` file
|
||||
|
||||
You can tell the {% data variables.product.prodname_codeql_cli %} how to handle these warnings by including a `--warnings` option in your command. For more information, see the [generate query-help reference documentation](https://codeql.github.com/docs/codeql-cli/manual/generate-query-help/#cmdoption-codeql-generate-query-help-warnings).
|
||||
|
||||
## Further reading
|
||||
|
||||
- “[Query help files](https://codeql.github.com/docs/writing-codeql-queries/query-help-files/#query-help-files)”
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: Using custom queries with the CodeQL CLI
|
||||
intro: 'You can write your own {% data variables.product.prodname_codeql %} queries to find specific vulnerabilities and errors.'
|
||||
product: '{% data reusables.gated-features.codeql %}'
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghes: '*'
|
||||
ghae: '*'
|
||||
ghec: '*'
|
||||
topics:
|
||||
- Advanced Security
|
||||
- Code scanning
|
||||
- CodeQL
|
||||
---
|
||||
|
||||
{% data reusables.codeql-cli.codeql-site-migration-note %}
|
||||
|
||||
## About custom queries and the {% data variables.product.prodname_codeql_cli %}
|
||||
|
||||
<!-- tweaked title to fit "About" styling of conceptual intro section -->
|
||||
|
||||
You can customize your {% data variables.product.prodname_codeql %} analyses by writing your own queries to highlight specific vulnerabilities or errors.
|
||||
|
||||
This topic is specifically about writing queries to use with the [database analyze](https://codeql.github.com/docs/codeql-cli/manual/database-analyze/) command to produce [interpreted results](https://codeql.github.com/docs/codeql-overview/about-codeql/#interpret-query-results).
|
||||
|
||||
{% data reusables.codeql-cli.advanced-query-execution %}
|
||||
|
||||
<!-- created a reusable since it's a reusable on the microsite -->
|
||||
|
||||
## Writing a valid query
|
||||
|
||||
Before running a custom analysis you need to write a valid query, and save it in a file with a `.ql` extension. There is extensive documentation available to help you write queries. For more information, see “[{% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/codeql-queries/#codeql-queries).”
|
||||
|
||||
## Including query metadata
|
||||
|
||||
Query metadata is included at the top of each query file. It provides users with information about the query, and tells the {% data variables.product.prodname_codeql_cli %} how to process the query results.
|
||||
|
||||
When running queries with the `database analyze` command, you must include the following two properties to ensure that the results are interpreted correctly:
|
||||
|
||||
- Query identifier (`@id`): a sequence of words composed of lowercase letters or digits, delimited by `/` or `-`, identifying and classifying the query.
|
||||
|
||||
- Query type (`@kind`): identifies the query as a simple alert (`@kind problem`), an alert documented by a sequence of code locations (`@kind path-problem`), for extractor troubleshooting (`@kind diagnostic`), or a summary metric (`@kind metric` and `@tags summary`).
|
||||
|
||||
For more information about these metadata properties, see “[Metadata for {% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/metadata-for-codeql-queries/#metadata-for-codeql-queries)” and the [Query metadata style guide](https://github.com/github/codeql/blob/main/docs/query-metadata-style-guide.md).
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note:** Metadata requirements may differ if you want to use your query with other applications. For more information, see “[Metadata for {% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/metadata-for-codeql-queries/#metadata-for-codeql-queries).”
|
||||
|
||||
{% endnote %}
|
||||
|
||||
## Packaging custom QL queries
|
||||
|
||||
{% data reusables.codeql-cli.beta-note-package-management %}
|
||||
|
||||
<!-- using reusable re-created by Felicity -->
|
||||
|
||||
When you write your own queries with the intention to share them with others, you should save them in a custom {% data variables.product.prodname_codeql %} pack. You can publish the pack as a {% data variables.product.prodname_codeql %} pack to {% data variables.product.prodname_registry %} - the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}. For more information, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/about-codeql-packs/#about-codeql-packs).”
|
||||
|
||||
<!-- tweaked to "For more information, see X" to match our style guide -->
|
||||
|
||||
{% data variables.product.prodname_codeql %} packs organize the files used in {% data variables.product.prodname_codeql %} analysis and can store queries, library files, query suites, and important metadata. Their root directory must contain a file named `qlpack.yml`. Your custom queries should be saved in the {% data variables.product.prodname_codeql %} pack root, or its subdirectories.
|
||||
|
||||
For each {% data variables.product.prodname_codeql %} pack, the `qlpack.yml` file includes information that tells the {% data variables.product.prodname_codeql_cli %} how to compile the queries, which other {% data variables.product.prodname_codeql %} packs and libraries the pack depends on, and where to find query suite definitions. For more information about what to include in this file, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/about-codeql-packs/#codeqlpack-yml-properties).”
|
||||
|
||||
## Contributing to the {% data variables.product.prodname_codeql %} repository
|
||||
|
||||
If you would like to share your query with other {% data variables.product.prodname_codeql %} users, you can open a pull request in the [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql). For more information, see [Contributing to {% data variables.product.prodname_codeql %}](https://github.com/github/codeql/blob/main/CONTRIBUTING.md).
|
||||
|
||||
<!-- tweaked to "For more information, see X" to match our style guide -->
|
||||
|
||||
## Further reading
|
||||
|
||||
- “[{% data variables.product.prodname_codeql %} queries](https://codeql.github.com/docs/writing-codeql-queries/codeql-queries/#codeql-queries)”
|
||||
7
data/features/query-pack-compatibility.yml
Normal file
7
data/features/query-pack-compatibility.yml
Normal file
@@ -0,0 +1,7 @@
|
||||
# Reference: #8683
|
||||
# Documentation for stable precompiled query packs
|
||||
versions:
|
||||
fpt: '*'
|
||||
ghec: '*'
|
||||
ghes: '>=3.9'
|
||||
ghae: '>=3.9'
|
||||
9
data/reusables/code-scanning/beta-kotlin-support.md
Normal file
9
data/reusables/code-scanning/beta-kotlin-support.md
Normal file
@@ -0,0 +1,9 @@
|
||||
{% ifversion codeql-kotlin-beta %}
|
||||
|
||||
{% note %}
|
||||
|
||||
**Note**: {% data variables.product.prodname_codeql %} analysis for Kotlin is currently in beta. During the beta, analysis of Kotlin will be less comprehensive than {% data variables.product.prodname_codeql %} analysis of other languages.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
{% endif %}
|
||||
8
data/reusables/code-scanning/beta-ruby-support.md
Normal file
8
data/reusables/code-scanning/beta-ruby-support.md
Normal file
@@ -0,0 +1,8 @@
|
||||
{% ifversion ghes < 3.8 or ghae < 3.8 %}
|
||||
{% note %}
|
||||
|
||||
**Note**: {% data variables.product.prodname_codeql %} analysis for Ruby is currently in beta. During the beta, analysis of Ruby will be less comprehensive than {% data variables.product.prodname_codeql %} analysis of other languages.
|
||||
|
||||
{% endnote %}
|
||||
|
||||
{% endif %}
|
||||
10
data/reusables/codeql-cli/advanced-query-execution.md
Normal file
10
data/reusables/codeql-cli/advanced-query-execution.md
Normal file
@@ -0,0 +1,10 @@
|
||||
{% note %}
|
||||
|
||||
**Note:** Queries run with `database analyze` have strict [metadata requirements](https://codeql.github.com/docs/codeql-cli/using-custom-queries-with-the-codeql-cli/#including-query-metadata). You can also execute queries using the following plumbing-level subcommands:
|
||||
|
||||
- [database run-queries](https://codeql.github.com/docs/codeql-cli/manual/database-run-queries/), which outputs non-interpreted results in an intermediate binary format called [BQRS](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#bqrs-file).
|
||||
- [query run](https://codeql.github.com/docs/codeql-cli/manual/query-run/), which will output BQRS files, or print results tables directly to the command line. Viewing results directly in the command line may be useful for iterative query development using the CLI.
|
||||
|
||||
Queries run with these commands don’t have the same metadata requirements. However, to save human-readable data you have to process each BQRS results file using the [bqrs decode](https://codeql.github.com/docs/codeql-cli/manual/bqrs-decode/) plumbing subcommand. Therefore, for most use cases it’s easiest to use database analyze to directly generate interpreted results.
|
||||
|
||||
{% endnote %}
|
||||
@@ -0,0 +1,5 @@
|
||||
{% note %}
|
||||
|
||||
**Note:** The CodeQL package management functionality, including CodeQL packs, is currently available as a beta release and is subject to change. During the beta release, CodeQL packs are available only using GitHub Packages - the {% data variables.product.prodname_container_registry %}. To use this beta functionality, install the latest version of the CodeQL CLI bundle from: https://github.com/github/codeql-action/releases.
|
||||
|
||||
{% endnote %}
|
||||
5
data/reusables/codeql-cli/codeql-site-migration-note.md
Normal file
5
data/reusables/codeql-cli/codeql-site-migration-note.md
Normal file
@@ -0,0 +1,5 @@
|
||||
{% note %}
|
||||
|
||||
**Note:** This article was migrated from the [CodeQL documentation website](https://codeql.github.com/docs/codeql-cli/) in January 2023.
|
||||
|
||||
{% endnote %}
|
||||
9
data/reusables/codeql-cli/test-qlpack.md
Normal file
9
data/reusables/codeql-cli/test-qlpack.md
Normal file
@@ -0,0 +1,9 @@
|
||||
The following `qlpack.yml` file states that `my-github-user/my-query-tests` depends on `my-github-user/my-custom-queries` at a version greater than or equal to 1.2.3 and less than 2.0.0. It also declares that the CLI should use the Java `extractor` when creating test databases. The `tests: .` line declares that all `.ql` files in the pack should be run as tests when `codeql test run` is run with the `--strict-test-discovery` option. Typically, test packs do not contain a `version` property. This prevents you from accidentally publishing them.
|
||||
|
||||
```yaml
|
||||
name: my-github-user/my-query-tests
|
||||
dependencies:
|
||||
my-github-user/my-custom-queries: ^1.2.3
|
||||
extractor: java
|
||||
tests: .
|
||||
```
|
||||
3
data/reusables/gated-features/codeql.md
Normal file
3
data/reusables/gated-features/codeql.md
Normal file
@@ -0,0 +1,3 @@
|
||||
GitHub {% data variables.product.prodname_codeql %} is licensed on a per-user basis upon installation. You can use {% data variables.product.prodname_codeql %} only for certain tasks under the license restrictions. For more information, see "[About the GitHub CodeQL license](/code-security/code-scanning/using-the-codeql-cli/about-the-codeql-cli#about-the-github-codeql-license)."
|
||||
|
||||
If you have a {% data variables.product.prodname_enterprise %} account and GitHub Advanced Security license, you can use {% data variables.product.prodname_codeql %} for automated analysis, continuous integration and continuous delivery. You can create an enterprise account by contacting the [sales team](https://enterprise.github.com/contact). {% data reusables.advanced-security.more-info-ghas %}
|
||||
@@ -27,6 +27,7 @@ const PUBLIC_REPOS = new Set([
|
||||
'codeql-action-sync-tool',
|
||||
'codeql-action',
|
||||
'codeql-cli-binaries',
|
||||
'codeql-go',
|
||||
'codeql',
|
||||
'codespaces-precache',
|
||||
'codespaces-jupyter',
|
||||
|
||||
Reference in New Issue
Block a user