1
0
mirror of synced 2025-12-21 10:57:10 -05:00

CodeQL CLI docs conversion megabranch (#32876)

Co-authored-by: Sam Browning <106113886+sabrowning1@users.noreply.github.com>
Co-authored-by: Rachael Rose Renk <91027132+rachaelrenk@users.noreply.github.com>
Co-authored-by: Felicity Chapman <felicitymay@github.com>
Co-authored-by: Ben Ahmady <32935794+subatoi@users.noreply.github.com>
Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>
Co-authored-by: Henning Makholm <hmakholm@github.com>
This commit is contained in:
Sarita Iyer
2023-01-20 12:24:48 -05:00
committed by GitHub
parent 9b267440c7
commit 0ad7a3905a
34 changed files with 3319 additions and 14 deletions

View File

@@ -0,0 +1,349 @@
---
title: About CodeQL packs
intro: 'You can use {% data variables.product.prodname_codeql %} packs to run {% data variables.product.prodname_codeql %} queries maintained by other people, or to share {% data variables.product.prodname_codeql %} queries that you''ve developed.'
product: '{% data reusables.gated-features.codeql %}'
versions:
feature: codeql-packs
topics:
- Advanced Security
- Code scanning
- CodeQL
---
{% data reusables.codeql-cli.codeql-site-migration-note %}
{% data reusables.codeql-cli.beta-note-package-management %}
## About {% data variables.product.prodname_codeql %} packs
{% ifversion ghes %}
{% note %}
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[About {% data variables.product.prodname_codeql %} packs](/enterprise-cloud@latest/code-security/code-scanning/codeql-cli-reference/about-codeql-packs)”, the Enterprise Cloud version, for the most up-to-date documentation.
{% endnote %}
{% endif %}
{% data variables.product.prodname_codeql %} packs are used to create, share, depend on, and run {% data variables.product.prodname_codeql %} queries and libraries. You can publish your own {% data variables.product.prodname_codeql %} packs and download packs created by others. {% data variables.product.prodname_codeql %} packs contain queries, library files, query suites, and metadata.
There are two types of {% data variables.product.prodname_codeql %} packs: query packs and library packs.
- Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and {% ifversion query-pack-compatibility %}pre-compiled representations of each query, in addition to the query sources{% else %}a compilation cache{% endif %}. This ensures consistent and efficient execution of the queries in the pack.
- Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled {% ifversion query-pack-compatibility %}separately{% else %}and there is no compilation cache included when the pack is published{% endif %}.
You can use the package management commands in the {% data variables.product.prodname_codeql_cli %} to create {% data variables.product.prodname_codeql %} packs, add dependencies to packs, and install or update dependencies. For more information, see “[Creating and working with {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/creating-and-working-with-codeql-packs#creating-and-working-with-codeql-packs).” You can also publish and download {% data variables.product.prodname_codeql %} packs using the {% data variables.product.prodname_codeql_cli %}. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs).”
The standard {% data variables.product.prodname_codeql %} packages for all supported languages are published in the [{% data variables.product.prodname_container_registry %}](https://github.com/orgs/codeql/packages).
The [{% data variables.product.prodname_codeql %} repository](https://github.com/github/codeql) contains source files for the standard {% data variables.product.prodname_codeql %} packs for all supported languages.
## {% data variables.product.prodname_codeql %} pack structure
A {% data variables.product.prodname_codeql %} pack must contain a file called `qlpack.yml` in its root directory. In the `qlpack.yml` file, the `name:` field must have a value that follows the format of `<scope>/<pack>`, where `<scope>` is the {% data variables.product.prodname_dotcom %} organization or user account that the pack will be published to and `<pack>` is the name of the pack. Additionally, query packs and library packs with {% data variables.product.prodname_codeql %} tests contain a `codeql-pack.lock.yml` file that contains the resolved dependencies of the pack. This file is generated during a call to the `codeql pack install` command, is not meant to be edited by hand, and should be added to your version control system.
The other files and directories within the pack should be logically organized. For example, typically:
* Queries are organized into directories for specific categories.
* Queries for specific products, libraries, and frameworks are organized into
their own top-level directories.
{% ifversion query-pack-compatibility %}
### About published packs
When a pack is published for use in analyses, the `codeql pack create` or `codeql pack publish` command verifies that the content is complete and also adds some additional pieces of content to it:
* For query packs, a copy of each of the library packs it depends on, in the precise versions it has been developed with. Users of the query pack won't need to download these library packs separately.
* For query packs, precompiled representations of each of the queries. These are faster to execute than it would be to compile the QL source for the query at each analysis.
Most of this data is located in a directory named `.codeql` in the published pack, but precompiled queries are in files with a `.qlx` suffix next to the `.ql` source for each query. When analyzing a database with a query from a published pack, {% data variables.product.prodname_codeql %} will load these files instead of the `.ql` source. If you need to modify the content of a _published_ pack, be sure to remove all of the `.qlx` files, since they may prevent modifications in the `.ql` files from taking effect.
{% endif %}
## About `qlpack.yml` files
When executing query-related commands, {% data variables.product.prodname_codeql %} first looks in siblings of the installation directory (and their subdirectories) for `qlpack.yml` files.
Then it checks the package cache for {% data variables.product.prodname_codeql %} packs which have been downloaded. This means that when you are developing queries locally, the local packages
in the installation directory override packages of the same name in the package cache, so that you can test your local changes.
The metadata in each `qlpack.yml` file tells {% data variables.product.prodname_codeql %} how to compile any queries in the pack, what libraries the pack depends on, and where to
find query suite definitions.
The contents of the {% data variables.product.prodname_codeql %} pack (queries or libraries used in {% data variables.product.prodname_codeql %} analysis) is included in the same directory as `qlpack.yml`, or its subdirectories.
The directory containing the `qlpack.yml` file serves as the root directory for the content of the {% data variables.product.prodname_codeql %} pack. That is, for all `.ql` and `.qll` files in the pack, {% data variables.product.prodname_codeql %} will resolve all import statements relative to the directory containing the `qlpack.yml` file at the packs root.
### `qlpack.yml` properties
The following properties are supported in `qlpack.yml` files.
#### `name`
- Required by all packs.
- Defines the scope of the pack, where the {% data variables.product.prodname_codeql %} pack is published, and the name of the pack defined using alphanumeric characters and hyphens. It must be unique as {% data variables.product.prodname_codeql %} cannot differentiate between {% data variables.product.prodname_codeql %} packs with identical names. Use the pack name to specify queries to run using `database analyze` and to define dependencies between {% data variables.product.prodname_codeql %} packs (see examples below). For example:
```yaml
name: octo-org/security-queries
```
#### `version`
- Required by all packs that are published.
- Defines a semantic version for this {% data variables.product.prodname_codeql %} pack that must adhere to the [SemVer v2.0.0 specification](https://semver.org/spec/v2.0.0.html). For example:
```yaml
version: 0.0.0
```
#### `dependencies`
- Required by packs that define {% data variables.product.prodname_codeql %} package dependencies on other packs.
- Defines a map from pack references to the semantic version range that is compatible with this pack. Supported for {% data variables.product.prodname_codeql_cli %} versions v2.6.0 and later. For example:
```yaml
dependencies:
codeql/cpp-all: ^0.0.2`
#### `defaultSuiteFile`
- Required by packs that export a set of default queries to run.
- Defines the path to a query suite file relative to the package root, containing all of the queries that are run by default when this pack is passed to the `codeql database analyze` command. Supported from CLI version v2.6.0 and onwards. Only one of `defaultSuiteFile` or `defaultSuite` can be defined. For example:
```yaml
defaultSuiteFile: cpp-code-scanning.qls
```
#### `defaultSuite`
- Required by packs that export a set of default queries to run.
- Defines an inlined query suite containing all of the queries that are run by default when this pack is passed to the `codeql database analyze` command. Supported from CLI version v2.6.0 and onwards. Only one of `defaultSuiteFile` or `defaultSuite` can be defined. For example:
```yaml
defaultSuite:
queries: .
exclude:
precision: medium
```
#### `library`
- Required by library packs.
- Defines a boolean value that indicates whether or not this pack is a library pack. Library packs do not contain queries and are not compiled. Query packs can ignore this field or explicitly set it to `false`. For example:
```yaml
library: true
```
#### `suites`
- Optional for packs that define query suites.
- Defines the path to a directory in the pack that contains the query suites you want to make known to the {% data variables.product.prodname_codeql_cli %}, defined relative to the pack directory. {% data variables.product.prodname_codeql %} pack users can run “well-known” suites stored in this directory by specifying the pack name, without providing their full path. This is not supported for {% data variables.product.prodname_codeql %} packs downloaded from the Container registry. For more information about query suites, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites/).” For example:
```yaml
suites: octo-org-query-suites
```
#### `tests`
- Optional for packs containing {% data variables.product.prodname_codeql %} tests. Ignored for packs without tests.
- Defines the path to a directory within the pack that contains tests, defined relative to the pack directory. Use `.` to specify the whole pack. Any queries in this directory are run as tests when `test run` is run with the `--strict-test-discovery` option. These queries are ignored by query suite definitions that use `queries` or `qlpack` instructions to ask for all queries in a particular pack. If this property is missing, then `.` is assumed. For example:
```yaml
tests: .
```
#### `extractor`
- Required by all packs containing {% data variables.product.prodname_codeql %} tests.
- Defines the {% data variables.product.prodname_codeql %} language extractor to use when running the {% data variables.product.prodname_codeql %} tests in the pack. For more information about testing queries, see “[Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries/).” For example:
```yaml
extractor: javascript
```
#### `authors`
- Optional.
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For example:
```yaml
authors: author1@github.com,author2@github.com
```
#### `license`
- Optional.
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For a list of allowed licenses, see [SPDX License List](https://spdx.org/licenses/) in the SPDX Specification. For example:
```yaml
license: MIT
```
#### `description`
- Optional.
- Defines metadata that will be displayed on the packaging search page in the packages section of the account that the {% data variables.product.prodname_codeql %} pack is published to. For example:
```yaml
description: Human-readable description of the contents of the {% data variables.product.prodname_codeql %} pack.
```
#### `libraryPathDependencies`
- Optional, deprecated. Use the `dependencies` property instead.
- Previously used to define the names of any {% data variables.product.prodname_codeql %} packs that this {% data variables.product.prodname_codeql %} pack depends on, as an array. This gives the pack access to any libraries, database schema, and query suites defined in the dependency. For example:
```yaml
libraryPathDependencies: codeql/javascript-all
```
#### `dbscheme`
- Required by core language packs only.
- Defines the path to the [database schema](https://codeql.github.com/docs/codeql-overview/codeql-glossary/#codeql-database-schema) for all libraries and queries written for this {% data variables.product.prodname_codeql %} language (see example below). For example:
```yaml
dbscheme: semmlecode.python.dbscheme
```
#### `upgrades`
- Required by core language packs only.
- Defines the path to a directory within the pack that contains database upgrade scripts, defined relative to the pack directory. Database upgrades are used internally to ensure that a database created with a different version of the {% data variables.product.prodname_codeql_cli %} is compatible with the current version of the CLI. For example:
```yaml
upgrades: .
```
## About `codeql-pack.lock.yml` files
`codeql-pack.lock.yml` files store the versions of the resolved transitive dependencies of a {% data variables.product.prodname_codeql %} pack. This file is created by the `codeql pack install` command if it does not already exist and should be added to your version control system. The `dependencies` section of the `qlpack.yml` file contains version ranges that are compatible with the pack. The `codeql-pack.lock.yml` file locks the versions to precise dependencies. This ensures that running `codeql pack install` on this the pack will always retrieve the same versions of dependencies even if newer compatible versions exist.
For example, if a `qlpack.yml` file contains the following dependencies:
```yaml
dependencies:
codeql/cpp-all: ^0.1.2
my-user/my-lib: ^0.2.3
other-dependency/from-source: "*"
```
The `codeql-pack.lock.yml` file will contain something like the following:
```yaml
dependencies:
codeql/cpp-all:
version: 0.1.4
my-user/my-lib:
version: 0.2.4
my-user/transitive-dependency:
version: 1.2.4
```
The `codeql/cpp-all` dependency is locked to version 0.1.4. The `my-user/my-lib` dependency is locked to version 0.2.4. The `my-user/transitive-dependency`, which is a transitive dependency and is not specified in the `qlpack.yml` file, is locked to version 1.2.4. The `other-dependency/from-source` is absent from the lock file since it is resolved from source. This dependency must be available in the same {% data variables.product.prodname_codeql %} workspace as the pack. For more information about {% data variables.product.prodname_codeql %} workspaces and resolving dependencies from source, see “[About {% data variables.product.prodname_codeql %} Workspaces](/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces).”
In most cases, the `codeql-pack.lock.yml` file is only relevant for query packs since library packs are non-executable and usually do not need their transitive dependencies to be fixed. The exception to this is for library packs that contain tests. In this case, the `codeql-pack.lock.yml` file is used to ensure that the tests are always run with the same versions of dependencies to avoid spurious failures when there are mismatched dependencies.
## Examples of custom {% data variables.product.prodname_codeql %} packs
When you write custom queries or tests, you should save them in custom {% data variables.product.prodname_codeql %} packs. For simplicity, try to organize each pack logically. For more information, see “[{% data variables.product.prodname_codeql %} pack structure](#codeql-pack-structure).” Save files for queries and tests in separate packs and, where possible, organize custom packs into specific folders for each target language. This is particuarly useful if you intend to publish your {% data variables.product.prodname_codeql %} packs so they can be shared with others or used in code scanning. For more information, see "[About code scanning with {% data variables.product.prodname_codeql %}](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql)."
### {% data variables.product.prodname_codeql %} packs for custom libraries
A custom {% data variables.product.prodname_codeql %} pack containing custom C++ libraries, with no queries or tests, may have a `qlpack.yml` file containing:
```yaml
name: my-github-user/my-custom-libraries
version: 1.2.3
library: true
dependencies:
codeql/cpp-all: ^0.1.2
```
where `codeql/cpp-all` is the name of the {% data variables.product.prodname_codeql %} pack for C/C++ analysis included in the {% data variables.product.prodname_codeql %} repository. The version range `^0.1.2` indicates that this pack is compatible with all versions of `codeql/cpp-all` that are greater than or equal to `0.1.2` and less than `0.2.0`. Any {% data variables.product.prodname_codeql %} library file (a file with a `.qll` extension) defined in this pack will be available to queries defined in any query pack that includes this pack in its dependencies block.
The `library` property indicates that this pack is a library pack and does not contain any queries.
### {% data variables.product.prodname_codeql %} packs for custom queries
A custom {% data variables.product.prodname_codeql %} pack containing custom C++ queries and libraries may have a `qlpack.yml` file containing:
```yaml
name: my-github-user/my-custom-queries
version: 1.2.3
dependencies:
codeql/cpp-all: ^0.1.2
my-github-user/my-custom-libraries: ^1.2.3
suites: my-custom-suites
```
where `codeql/cpp-all` is the name of the {% data variables.product.prodname_codeql %} pack for C/C++ analysis included in the {% data variables.product.prodname_codeql %} repository. The version range `^0.1.2` indicates that this pack is compatible with all versions of `codeql/cpp-all` that are greater than or equal to `0.1.2` and less than `0.2.0`. `my-github-user/my-custom-libraries` is the name of a {% data variables.product.prodname_codeql %} pack containing custom {% data variables.product.prodname_codeql %} libraries for C++. Any {% data variables.product.prodname_codeql %} library file (a file with a `.qll` extension) defined in this pack will be available to queries in the `my-github-user/my-custom-queries` pack.
The `suites` property indicates a directory where “well-known” query suites can be found. These suites can be used on the command line by referring to their name only, rather than their full path. For more information about query suites, see “[Creating {% data variables.product.prodname_codeql %} query suites](/code-security/code-scanning/using-the-codeql-cli/creating-codeql-query-suites).”
### {% data variables.product.prodname_codeql %} packs for custom tests
For custom {% data variables.product.prodname_codeql %} packs containing test files, you also need to include an
`extractor` property so that the `test run` command knows how to create test
databases. You may also wish to specify the `tests` property.
{% data reusables.codeql-cli.test-qlpack %}
For more information about running tests, see “[Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries).”
## Examples of {% data variables.product.prodname_codeql %} packs in the {% data variables.product.prodname_codeql %} repository
Each of the languages in the {% data variables.product.prodname_codeql %} repository has four main {% data variables.product.prodname_codeql %} packs:
* Core library pack for the language, with the database schema
used by the language, and {% data variables.product.prodname_codeql %} libraries, and queries at `<language>/ql/lib`
* Core query pack for the language that includes the default queries for the language, along
with their query suites at `<language>/ql/src`
* Tests for the core language libraries and queries at `<language>/ql/test`
* Example queries for the language at `<language>/ql/examples`
### Core library pack
Here is an example `qlpack.yml` file for the [C/C++ analysis libraries](https://github.com/github/codeql/blob/main/cpp/ql/lib/qlpack.yml)
core language pack:
```yaml
name: codeql/cpp-all
version: x.y.z-dev
dbscheme: semmlecode.cpp.dbscheme
library: true
upgrades: upgrades
```
Some extra notes on the following properties:
* `library`: Indicates that this is a library pack with no executable queries. It is only meant to be used as a dependency for other packs.
* `dbscheme` and `upgrades`: These properties are internal to the {% data variables.product.prodname_codeql_cli %} and should only be defined in the core QL pack for a language.
### Core query pack
Here is an example `qlpack.yml` file for [C/C++ analysis queries](https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml)
core query pack:
```yaml
name: codeql/cpp-queries
version: x.y.z-dev
dependencies:
codeql/cpp-all: "*"
codeql/suite-helpers: "*"
suites: codeql-suites
defaultSuiteFile: codeql-suites/cpp-code-scanning.qls
```
Some extra notes on the following properties:
* `dependencies`: This query pack depends on `codeql/cpp-all` and `codeql/suite-helpers`. Since these dependencies are resolved from source, it does not matter what version of the {% data variables.product.prodname_codeql %} pack they are compatible with. For more information about resolving dependencies from source, see “[Source Dependencies](/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces#source-dependencies).”
* `suites`: Indicates the directory containing “well-known” query suites.
* `defaultSuiteFile`: The name of the default query suite file that is used when no query suite is specified.
### Tests for the core {% data variables.product.prodname_codeql %} pack
Here is an example `qlpack.yml` file for [C/C++ analysis tests](https://github.com/github/codeql/blob/main/cpp/ql/src/qlpack.yml)
core test pack:
```yaml
name: codeql/cpp-tests
dependencies:
codeql/cpp-all: "*"
codeql/cpp-queries: "*"
extractor: cpp
tests: .
```
Some extra notes on the following properties:
* `dependencies`: This pack depends on the core {% data variables.product.prodname_codeql %} query and library packs for C++.
* `extractor`: This specifies that all the tests will use the same C++ extractor to create the database for the tests.
* `tests`: This specifies the location of the tests. In this case, the tests are in the root folder (and all sub-folders) of the pack.
* `version`: There is no `version` property for the tests pack. This prevents test packs from accidentally being published.

View File

@@ -0,0 +1,116 @@
---
title: About CodeQL workspaces
intro: '{% data variables.product.prodname_codeql %} workspaces allow you to develop and maintain a group of {% data variables.product.prodname_codeql %} packs that depend on each other.'
product: '{% data reusables.gated-features.codeql %}'
versions:
feature: codeql-packs
topics:
- Advanced Security
- Code scanning
- CodeQL
---
{% data reusables.codeql-cli.codeql-site-migration-note %}
## About {% data variables.product.prodname_codeql %} workspaces
{% ifversion ghes %}
{% note %}
**Note:** If you have manually updated your {% data variables.product.prodname_codeql_cli %} version to a newer version than what is covered in this article, please see “[About {% data variables.product.prodname_codeql %} workspaces](/enterprise-cloud@latest/code-security/code-scanning/codeql-cli-reference/about-codeql-workspaces)”, the Enterprise Cloud version, for the most up-to-date documentation.
{% endnote %}
{% endif %}
You use a {% data variables.product.prodname_codeql %} workspace when you want to group multiple {% data variables.product.prodname_codeql %} packs together. A typical use case for a {% data variables.product.prodname_codeql %} workspace is to develop a set of {% data variables.product.prodname_codeql %} library and query packs that are mutually dependent. For more information on {% data variables.product.prodname_codeql %} packs, see “[About {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/codeql-cli-reference/about-codeql-packs).”
The main benefit of a {% data variables.product.prodname_codeql %} workspace is that it makes it easier for you to develop and maintain multiple {% data variables.product.prodname_codeql %} packs. When you use a {% data variables.product.prodname_codeql %} workspace, all the {% data variables.product.prodname_codeql %} packs in the workspace are available as *source dependencies* for each other when you run a {% data variables.product.prodname_codeql %} command that resolves queries. This makes it easier to develop, maintain, and publish multiple, related {% data variables.product.prodname_codeql %} packs.
In most cases, you should store the {% data variables.product.prodname_codeql %} workspace and the {% data variables.product.prodname_codeql %} packs contained in it in one git repository. This makes it easier to share your {% data variables.product.prodname_codeql %} development environment.
## The `codeql-workspace.yml` file
A {% data variables.product.prodname_codeql %} workspace is defined by a `codeql-workspace.yml` yaml file. This file contains a `provide` block, and optionally `ignore` and `registries` blocks.
* The `provide` block contains a list of glob patterns that define the {% data variables.product.prodname_codeql %} packs that are available in the workspace.
* The `ignore` block contains a list of glob patterns that define {% data variables.product.prodname_codeql %} packs that are not available in the workspace.
* The `registries` block contains a list of GHES URLs and package patterns that control which container registry is used for publishing {% data variables.product.prodname_codeql %} packs. For more information, see “[Publishing and using {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/publishing-and-using-codeql-packs#working-with-codeql-packs-on-ghes)."
Each entry in the `provide` or `ignore` section must map to the location of a `qlpack.yml` file. All glob patterns are defined relative to the directory that contains the workspace file. For a list of patterns accepted in this file, see “[@actions/glob](https://github.com/actions/toolkit/tree/main/packages/glob#patterns) .”
For example, the following `codeql-workspace.yml` file defines a workspace that contains all the {% data variables.product.prodname_codeql %} packs recursively found in the `codeql-packs` directory, except for the packs in the `experimental` directory. The `registries` block specifies that `codeql/\*` packs should be downloaded from [https://ghcr.io/v2/](https://ghcr.io/v2/), which is {% data variables.product.prodname_dotcom %}s default container registry. All other packs should be downloaded from and published to the regsitry at `GHE_HOSTNAME`.
```yaml
provide:
- "*/codeql-packs/**/qlpack.yml"
ignore:
- "*/codeql-packs/**/experimental/**/qlpack.yml"
registries:
- packages: 'codeql/*'
url: https://ghcr.io/v2/
- packages: '*'
url: https://containers.GHE_HOSTNAME/v2/
```
To verify that your `codeql-workspace.yml` file includes the {% data variables.product.prodname_codeql %} packs that you expect, run the `codeql pack ls` command in the same directory as your workspace. The result of the command is a list of all {% data variables.product.prodname_codeql %} packs in the workspace.
## Source dependencies
Source dependencies are {% data variables.product.prodname_codeql %} packs that are resolved from the local file system outside of the {% data variables.product.prodname_codeql %} package cache. These dependencies can be in the same {% data variables.product.prodname_codeql %} workspace, or specified as a path option using the `--additional-packs` argument. When you compile and run queries locally, source dependencies override any dependencies found in the {% data variables.product.prodname_codeql %} package cache as well as version constraints defined in the `qlpack.yml`. All references to {% data variables.product.prodname_codeql %} packs in the same workspace are resolved as source dependencies.
This is particularly useful in the following situations:
* One of the dependencies of the query pack you are running is not yet published. Resolving from source is the only way to reference that pack.
* You are making changes to multiple packs at the same time and want to test them together. Resolving from source ensures that you are using the version of the pack with your changes in it.
## {% data variables.product.prodname_codeql %} workspaces and query resolution
All {% data variables.product.prodname_codeql %} packs in a workspace are available as source dependencies for each other when you run any {% data variables.product.prodname_codeql %} command that resolves queries or packs. For example, when you run `codeql pack install` in a pack directory in a workspace, any dependency that can be found in the workspace will be used instead of downloading that dependency to the package cache and adding it to the `codeql-pack.lock.yml` file. For more information, see “[Creating and working with {% data variables.product.prodname_codeql %} packs](/code-security/code-scanning/using-the-codeql-cli/creating-and-working-with-codeql-packs#adding-and-installing-dependencies).”
Similarly, when you publish a {% data variables.product.prodname_codeql %} query pack to the {% data variables.product.prodname_dotcom %} container registry using `codeql pack publish` the command will always use the dependencies from the workspace instead of using dependencies found in the local package cache.
This ensures that any local changes you make to a query library in a dependency are automatically reflected in any query packs you publish from that workspace.
### Example
Consider the following `codeql-workspace.yml` file:
```yaml
provide:
- "**/qlpack.yml"
```
And the following {% data variables.product.prodname_codeql %} library pack `qlpack.yml` file in the workspace:
```yaml
name: my-company/my-library
library: true
version: 1.0.0
```
And the following {% data variables.product.prodname_codeql %} query pack `qlpack.yml` file in the workspace:
```yaml
name: my-company/my-queries
version: 1.0.0
dependencies:
my-company/my-library: "*"
codeql/cpp-all: ~0.2.0
```
Notice that the `dependencies` block for the {% data variables.product.prodname_codeql %} query pack, `my-company/my-queries`, specifies `"*"` as the version of the library pack. Since the library pack is already defined as a source dependency in `codeql-workspace.yml`, the library packs content is always resolved from inside the workspace. Any version constraint you define will be ignored in this case. We recommend that you use `"*"` for source dependencies to make it clear that the version is inherited from the workspace.
When you execute `codeql pack install` from the query pack directory, an appropriate version of `codeql/cpp-all` is downloaded to the local package cache. Also, a `codeql-pack.lock.yml` file is created that contains the resolved version of `codeql/cpp-all`. The lock file wont contain an entry for `my-company/my-library` since it is resolved from source dependencies. The `codeql-pack.lock.yml` file will look something like this:
```yaml
dependencies:
codeql/cpp-all:
version: 0.2.2
```
When you execute `codeql pack publish` from the query pack directory, the `codeql/cpp-all` dependency from the package cache and the `my-company/my-library` from the workspace are bundled with `my-company/my-queries` and published to the {% data variables.product.prodname_dotcom %} container registry.

View File

@@ -0,0 +1,78 @@
---
title: Exit codes
intro: 'Exit codes signify the status of a command after the {% data variables.product.prodname_codeql_cli %} runs it.'
product: '{% data reusables.gated-features.codeql %}'
versions:
fpt: '*'
ghes: '*'
ghae: '*'
ghec: '*'
topics:
- Advanced Security
- Code scanning
- CodeQL
---
{% data reusables.codeql-cli.codeql-site-migration-note %}
## About exit codes
The {% data variables.product.prodname_codeql_cli %} reports the status of each command it runs as an exit code.
This exit code provides information for subsequent commands or for other tools that rely on the {% data variables.product.prodname_codeql_cli %}.
## 0
Success, normal termination.
## 1
The command successfully determined that the answer to your question is “no”.
This exit code is only used by a few commands, such as [codeql test run](https://codeql.github.com/docs/codeql-cli/manual/test-run/), [codeql database check](https://codeql.github.com/docs/codeql-cli/manual/dataset-check/), [codeql query format](https://codeql.github.com/docs/codeql-cli/manual/query-format/),and [codeql resolve extractor](https://codeql.github.com/docs/codeql-cli/manual/resolve-extractor/).
For more details, see the documentation for those commands.
## 2
Something went wrong.
The CLI writes a human-readable error message to stderr.
This includes cases where an extractor fails with an internal error, because the `codeql` driver cant distinguish between internal and user-facing errors in extractor behavior.
## 3
The launcher was unable to find the {% data variables.product.prodname_codeql %} installation directory.
In this case, the launcher cant start the Java code for the {% data variables.product.prodname_codeql_cli %} at all. This should only happen when something is severely wrong with the {% data variables.product.prodname_codeql %} installation.
## 32
The extractor didnt find any code to analyze when running [codeql database create](https://codeql.github.com/docs/codeql-cli/manual/database-create/) or [codeql database finalize](https://codeql.github.com/docs/codeql-cli/manual/database-finalize/).
## 33
One or more query evaluations timed out.
Its possible that some queries that were evaluated in parallel didnt time out. The results for those queries are produced as usual.
## 98
Evaluation was explicitly canceled.
## 99
The {% data variables.product.prodname_codeql_cli %} ran out of memory.
This doesnt necessarily mean that all the machines physical RAM has been used.
If you dont use the `--ram` option to set a limit explicitly, the JVM decides on a default limit at startup.
## 100
A fatal internal error occurred.
This should be considered a bug. The CLI usually writes an abbreviated error description to stderr.
If you can reproduce the bug, its helpful to use `--logdir` and send the log files to {% data variables.product.prodname_dotcom %} in a bug report.
## Other
In the case of really severe problems within the JVM that runs `codeql`, it might return a nonzero exit code of its own choosing.
This should only happen if something is severely wrong with the {% data variables.product.prodname_codeql %} installation, or if there is a memory issue with the host system running the {% data variables.product.prodname_codeql %} process. For example, Unix systems may return Exit Code 137 to indicate that the kernel has killed a process that {% data variables.product.prodname_codeql %} has started. One way to troubleshoot this is to modify your `ram=` flag for the `codeql database analyze` step and re-run your workflow.

View File

@@ -0,0 +1,20 @@
---
title: CodeQL CLI reference
intro: 'You can learn how to use {% data variables.product.prodname_codeql %} workspaces and {% data variables.product.prodname_codeql %} packs and how to understand the output of {% data variables.product.prodname_codeql %} commands.'
product: '{% data reusables.gated-features.codeql %}'
versions:
fpt: '*'
ghes: '*'
ghae: '*'
ghec: '*'
topics:
- Advanced Security
- Code scanning
children:
- /about-codeql-packs
- /about-codeql-workspaces
- /query-reference-files
- /sarif-output
- /exit-codes
---

View File

@@ -0,0 +1,63 @@
---
title: Query reference files
intro: 'You can use query reference files to define the location of a query you want to run in tests.'
product: '{% data reusables.gated-features.codeql %}'
versions:
fpt: '*'
ghes: '*'
ghae: '*'
ghec: '*'
topics:
- Advanced Security
- Code scanning
- CodeQL
---
{% data reusables.codeql-cli.codeql-site-migration-note %}
## About query reference files
A query reference file is text file that defines the location of one query to test.
You use a query reference file when you want to tell the `test run` subcommand
to run a query thats not part of a test directory.
There are two ways to specify queries that you want to run as tests:
1. Use a query reference file to specify the location of a query to test.
This is useful when you create tests for alert and path queries that
are intended to identify problems in real codebases. You might create
several directories of test code, each focusing on different
aspects of the query. Then you would add a query reference file to
each directory of test code, to specify the query to test.
2. Add the query directly to a directory of tests.
These is typically useful when youre writing queries explicitly to test the behavior
of QL libraries. Often these queries contain just a few calls to library predicates,
wrapping them in a `select` statement so their output can be tested.
## Defining a query reference file
Each query reference file, `.qlref`, contains a single line that defines
where to find one query. The location must be defined relative
to the root of the {% data variables.product.prodname_codeql %} pack that contains the query.
Usually, this is either the {% data variables.product.prodname_codeql %} pack that contains the `.qlref`, a {% data variables.product.prodname_codeql %} pack specified in the `dependencies` block for the test pack, or a transitive dependency of the {% data variables.product.prodname_codeql %} pack.
You should use forward slashes in the path on all operating
systems to ensure compatibility between systems.
### Example
A query reference file to test a JavaScript alert query:
[DeadAngularJSEventListener.qlref](https://github.com/github/codeql/blob/main/javascript/ql/test/query-tests/AngularJS/DeadAngularJSEventListener/DeadAngularJSEventListener.qlref)
The `qlpack.yml` file, https://github.com/github/codeql/blob/main/javascript/ql/test/qlpack.yml,
for the CodeQL pack at `javascript/ql/test` defines `codeql/javascript-queries` as
a dependency. So the query reference file defines the location of the query relative
to the `codeql/javascript-queries` {% data variables.product.prodname_codeql %} pack:
```
AngularJS/DeadAngularJSEventListener.ql
```
{% ifversion codeql-packs %}
For another example, see [Testing custom queries](/code-security/code-scanning/using-the-codeql-cli/testing-custom-queries).
{% endif %}

View File

@@ -0,0 +1,188 @@
---
title: CodeQL CLI SARIF output
intro: 'You can output SARIF from the {% data variables.product.prodname_codeql_cli %} and share static analysis results with other systems.'
product: '{% data reusables.gated-features.codeql %}'
allowTitleToDifferFromFilename: true
versions:
fpt: '*'
ghes: '*'
ghae: '*'
ghec: '*'
topics:
- Advanced Security
- Code scanning
- CodeQL
---
{% data reusables.codeql-cli.codeql-site-migration-note %}
## About SARIF output
SARIF is designed to represent the output of a broad range of static analysis tools, and there are many features in the SARIF specification that are
considered “optional”. This document details the output produced when using the format type `sarifv2.1.0`, which corresponds to the SARIF v2.1.0.csd1 specification. For more information on selecting a file format for your analysis results, see the [database analyze](https://codeql.github.com/docs/codeql-cli/manual/database-analyze/) command in the {% data variables.product.prodname_codeql %} documentation.
## SARIF specification and schema
This article is intended to be read alongside the detailed SARIF specification. For more information on the specification and the SARIF schema, see the [SARIF specification documentation](https://github.com/oasis-tcs/sarif-spec/blob/main/Schemata/sarif-schema-2.1.0.json).
## Change notes
### Changes between versions
| CodeQL version | Format type | Changes |
|----------------|-------------|---------|
| 2.0.0 | `sarifv2.1.0` | First version of this format. |
### Future changes to the output
The output produced for a given specific format type (for example, `sarifv2.1.0`) may change in future {% data variables.product.prodname_codeql %} releases. We will endeavor to maintain backwards compatibility with consumers of the generated SARIF by ensuring that:
* No field which is marked as “Always” being generated will be removed.
* The circumstances under which “Optional” fields are generated may change. Consumers of the {% data variables.product.prodname_codeql %} SARIF output should be robust to the presence or absence of these fields.
New output fields may be added in future releases under the same format typethese are not considered to break backwards compatibility, and consumers should be robust to the presence of newly added fields.
New format argument types may be added in future versions of {% data variables.product.prodname_codeql %}—for example, to support new versions of SARIF. These have no guarantee of backwards compatibility, unless explicitly documented.
## Generated SARIF objects
This details each SARIF component that may be generated, along with any specific circumstances. We omit any properties that are never generated.
### `sarifLog` object
| JSON property name| When is this generated?| Notes|
|-------------------|------------------------|---------|
| `$schema`| Always| Provides a link to the [SARIF schema](https://github.com/oasis-tcs/sarif-spec/blob/main/Schemata/sarif-schema-2.1.0.json).|
| `version`| Always| The version of the SARIF used to generate the output.|
| `runs`| Always| An array containing a single run object, for one language.|
### `run` object
| JSON property name| When is this generated?| Notes|
|-------------------|------------------------|---------|
| `tool`| Always| |
| `originalUriBaseIds`| Always| A dictionary of `uriBaseIds` to artifactLocations representing the original locations on the analysis machine. At a minimum, this will contain the `%SRCROOT%` `uriBaseId`, which represents the root location on the analysis machine of the source code for the analyzed project. Each `artifactLocation` will contain the `uri` and `description` properties.|
| `artifacts`| Always| An array containing at least one artifact object for every file referenced in a result.|
| `results`| Always| |
| `newLineSequences`| Always| |
| `columnKind`| Always| |
| `properties`| Always| The properties dictionary will contain the `semmle.formatSpecifier`, which identifies the format specifier passed to the {% data variables.product.prodname_codeql_cli %}.|
### `tool` object
| JSON property name| When is this generated?| Notes|
|-------------------|------------------------|---------|
| `driver`| Always| |
### `toolComponent` object
| JSON property name| When is this generated?| Notes|
|-----------------------|---------------------|-----|
| `name`| Always| Set to “{% data variables.product.prodname_codeql %} command-line toolchain” for output from the {% data variables.product.prodname_codeql_cli %} tools. Note, if the output was generated using a different tool a different `name` is reported, and the format may not be as described here.|
| `organization`| Always| Set to “GitHub”.|
| `version`| Always| Set to the {% data variables.product.prodname_codeql %} release version e.g. “2.0.0”.|
| `rules`| Always| An array of `reportingDescriptor` objects that represent rules. This array will contain, at a minimum, all the rules that were run during this analysis, but may contain rules which were available but not run. For more detail about enabling queries, see `defaultConfiguration`.|
### `reportingDescriptor` object (for rule)
`reportingDescriptor` objects may be used in multiple places in the SARIF specification. When a `reportingDescriptor` is included in the rules array of a `toolComponent` object it has the following properties.
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `id`| Always| Will contain the `@id` property specified in the query that defines the rule, which is usually of the format `language/rule-name` (for example `cpp/unsafe-format-string`). If your organization defines the `@opaqueid` property in the query it will be used instead.|
| `name`| Always| Will contain the `@id` property specified in the query. See the `id` property above for an example.|
| `shortDescription`| Always| Will contain the `@name` property specified in the query that defines the rule.|
| `fullDescription`| Always| Will contain the `@description` property specified in the query that defines the rule.|
| `defaultConfiguration`| Always| A `reportingConfiguration` object, with the enabled property set to true or false, and a level property set according to the `@severity` property specified in the query that defines the rule. Omitted if the `@severity` property was not specified.|
### `artifact` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `location`| Always| An `artifactLocation` object.|
| `index`| Always| The index of the `artifact` object.|
| `contents`| Optionally| If results are generated using the `--sarif-add-file-contents` flag, and the source code is available at the time the SARIF file is generated, then the `contents` property is populated with an `artifactContent` object, with the `text` property set.|
### `artifactLocation` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `uri`| Always| |
| `index`| Always| |
| `uriBaseId`| Optionally| If the file is relative to some known abstract location, such as the root source location on the analysis machine, this will be set.|
### `result` object
The composition of the results is dependent on the options provided to CodeQL. By default, the results are grouped by unique message format string and primary location. Thus, two results that occur at the same location with the same underlying message, will appear as a single result in the output. This behavior can be disabled by using the flag `--ungroup-results`, in which case no results are grouped.
| JSON property name | When is this generated?| Notes|
|-----------------------|--------------------|------|
| `ruleId`| Always| See the description of the `id` property in `reportingDescriptor` object (for rule) .|
| `ruleIndex`| Always| |
| `message`| Always| A message describing the problem(s) occurring at this location. This message may be a SARIF “Message with placeholder”, containing links that refer to locations in the `relatedLocations` property.|
| `locations`| Always| An array containing a single `location` object.|
| `partialFingerprints`| Always| A dictionary from named fingerprint types to the fingerprint. This will contain, at a minimum, a value for the `primaryLocationLineHash`, which provides a fingerprint based on the context of the primary location.|
| `codeFlows`| Optionally| This array may be populated with one or more `codeFlow` objects if the query that defines the rule for this result is of `@kind path-problem`.|
| `relatedLocations`| Optionally| This array will be populated if the query that defines the rule for this result has a message with placeholder options. Each unique location is included once.|
| `suppressions`| Optionally| If the result is suppressed, then this will contain a single `suppression` object, with the `@kind` property set to `IN_SOURCE`. If this result is not suppressed, but there is at least one result that has a suppression, then this will be set to an empty array, otherwise it will not be set.|
### `location` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `physicalLocation`| Always| |
| `id`| Optionally| `location` objects that appear in the `relatedLocations` array of a `result` object may contain the `id` property.|
| `message`| Optionally| `location` objects may contain the `message` property if:</br></br>- They appear in the `relatedLocations` array of a `result` object may contain the `message` property.</br></br>- They appear in the `threadFlowLocation.location` property.|
### `physicalLocation` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `artifactLocation`| Always| |
| `region`| Optionally| If the given `physicalLocation` exists in a text file, such as a source code file, then the `region` property may be present.|
| `contextRegion`| Optionally| May be present if this location has an associated `snippet`.|
### `region` object
There are two types of `region` object produced by {% data variables.product.prodname_codeql %}:
* Line/column offset regions
* Character offset and length regions
Any region produced by {% data variables.product.prodname_codeql %} may be specified in either format, and consumers should robustly handle either type.
For line/column offset regions, the following properties will be set:
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `startLine`| Always| |
| `startColumn`| Optionally| Not included if equal to the default value of 1.|
| `endLine`| Optionally| Not included if identical to `startLine`.|
| `endColumn`| Always| |
| `snippet`| Optionally| |
For character offset and length regions, the following properties will be set:
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `charOffset`| Optionally| Provided if `startLine`, `startColumn`, `endLine`, and `endColumn` are not populated.|
| `charLength`| Optionally| Provided if `startLine`, `startColumn`, `endLine`, and `endColumn` are not populated.|
| `snippet`| Optionally| |
### `codeFlow` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `threadFlows`| Always| |
### `threadFlow` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `locations`| Always| |
### `threadFlowLocation` object
| JSON property name| When is this generated?| Notes|
|-----------------------|--------------------|------|
| `location`| Always| |