CodeQL: Submitting diagnostics to the tool status page for failed third-party CI runs (#36204)

Co-authored-by: Felicity Chapman <felicitymay@github.com>

content/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page.md

@@ -61,10 +61,12 @@ This view will also show error messages. For more information, see "[Debugging u
 
 ### How {% data variables.product.prodname_codeql %} defines scanned files
 
-A file is reported as scanned by {% data variables.product.prodname_codeql %} if some of the lines of code in that file were processed. If you're using a standard configuration of the {% data variables.product.prodname_codeql %} action, the scanned files shown in the {% data variables.code-scanning.tool_status_page %} will include source code files for all languages that {% data variables.product.prodname_codeql %} can analyze. If you use an advanced setup, you can optionally define which files for interpreted languages should be scanned using the the `paths` and `paths-ignore` configuration properties. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql)" and "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning)."
+A file is reported as scanned by {% data variables.product.prodname_codeql %} if some of the lines of code in that file were processed. If you're using a standard configuration of the {% data variables.product.prodname_codeql %} action, the scanned files shown in the {% data variables.code-scanning.tool_status_page %} will include source code files for all languages that {% data variables.product.prodname_codeql %} can analyze. If you use an advanced setup, you can optionally define which files for interpreted languages should be scanned using the `paths` and `paths-ignore` configuration properties. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql)" and "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning)."
 
 For compiled languages, the {% data variables.code-scanning.tool_status_page %} reports files that were present before running autobuild or any manual build steps. This means that files generated during the build process are not shown in the {% data variables.code-scanning.tool_status_page %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-the-codeql-workflow-for-compiled-languages#about-autobuild-for-codeql)."
 
+The tool status page will calculate the percentage of files that were scanned by CodeQL for each language supported by CodeQL. This percentage respects any files excluded by the `paths` and `paths-ignore` configuration properties.
+
 ### Downloading details of the files analyzed
 
 For integrated tools such as {% data variables.product.prodname_codeql %}, you can download detailed reports from the {% data variables.code-scanning.tool_status_page %} in CSV format. This will show:
@@ -91,6 +93,12 @@ To download a report, select a configuration you're interested in. Then click **
 
 If you see that there is a problem with your analysis from the {% data variables.product.prodname_code_scanning %} alerts page, you can use the {% data variables.code-scanning.tool_status_page %} to identify the problem. For integrated tools, you can see specific error messages in the detailed information section, related to specific {% data variables.product.prodname_code_scanning %} tools. These error messages contain information about why the tool may not be performing as expected, and actions you can take. For more information about how to access this section of the {% data variables.code-scanning.tool_status_page %}, see "[Accessing detailed information about tools](#accessing-detailed-information-about-tools)."
 
+For integrated tools such as {% data variables.product.prodname_codeql %}, you can also use file coverage information to improve your analysis. For each language displayed on the {% data variables.code-scanning.tool_status_page %}:
+
+- If the language has a high scanned percentage, this shows that code scanning is scanning that language as expected.
+- If the language has a low scanned percentage, you may wish to investigate diagnostic output produced by CodeQL for that language: for more information see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/troubleshooting-the-codeql-workflow)."
+- If the language has a scanned percentage of zero, you may have source code in your repository written in languages supported by CodeQL but not currently being analyzed with CodeQL. In this case, you may wish to update your setup to start analyzing these additional languages. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#changing-the-languages-that-are-analyzed)."
+
 {% note %}
 
 **Note:** If you have set up {% data variables.product.prodname_codeql %} using advanced setup and then set up default setup on the same repository, the {% data variables.code-scanning.tool_status_page %} will only show the default setup.

content/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system.md

@@ -53,10 +53,11 @@ You can display the command-line help for any command using the <nobr>`--help`</
 
   ```shell
   # Single supported language - create one CodeQL database
-  codeql database create &lt;database&gt; --command&lt;build&gt; --language=&lt;language-identifier&gt;
+  codeql database create &lt;database&gt; --command &lt;build&gt; \
+        --language=&lt;language-identifier&gt;
 
   # Multiple supported languages - create one CodeQL database per language
-  codeql database create &lt;database&gt; --command&lt;build&gt; \
+  codeql database create &lt;database&gt; --command &lt;build&gt; \
         --db-cluster --language=&lt;language-identifier&gt;,&lt;language-identifier&gt;
   ```
 
@@ -68,11 +69,11 @@ You can display the command-line help for any command using the <nobr>`--help`</
 
 | Option | Required | Usage |
 |--------|:--------:|-----|
-| `<database>` | {% octicon "check" aria-label="Required" %} | Specify the name and location of a directory to create for the {% data variables.product.prodname_codeql %} database. The command will fail if you try to overwrite an existing directory. If you also specify `--db-cluster`, this is the parent directory and a subdirectory is created for each language analyzed.|
+| `<database>` | {% octicon "check" aria-label="Required" %} | Specify the name and location of a directory to create for the {% data variables.product.prodname_codeql %} database. The command will fail if you try to overwrite an existing directory. If you also specify `--db-cluster`, this is the parent directory and a subdirectory is created for each language analyzed. |
-| <nobr>`--language`</nobr> | {% octicon "check" aria-label="Required" %} | Specify the identifier for the language to create a database for, one of: `{% data reusables.code-scanning.codeql-languages-keywords %}` (use `javascript` to analyze TypeScript code {% ifversion codeql-kotlin-beta %} and `java` to analyze Kotlin code{% endif %}). When used with <nobr>`--db-cluster`</nobr>, the option accepts a comma-separated list, or can be specified more than once.
+| <nobr>`--language`</nobr> | {% octicon "check" aria-label="Required" %} | Specify the identifier for the language to create a database for, one of: `{% data reusables.code-scanning.codeql-languages-keywords %}` (use `javascript` to analyze TypeScript code {% ifversion codeql-kotlin-beta %} and `java` to analyze Kotlin code{% endif %}). When used with <nobr>`--db-cluster`</nobr>, the option accepts a comma-separated list, or can be specified more than once. |
 | <nobr>`--command`</nobr> | {% octicon "x" aria-label="Optional" %} | **Recommended.** Use to specify the build command or script that invokes the build process for the codebase. Commands are run from the current folder or, where it is defined, from <nobr>`--source-root`</nobr>. Not needed for Python and JavaScript/TypeScript analysis. |
-| <nobr>`--db-cluster`</nobr> | {% octicon "x" aria-label="Optional" %} | Use in multi-language codebases to generate one database for each language specified by <nobr>`--language`</nobr>.
+| <nobr>`--db-cluster`</nobr> | {% octicon "x" aria-label="Optional" %} | Use in multi-language codebases to generate one database for each language specified by <nobr>`--language`</nobr>. |
-| <nobr>`--no-run-unnecessary-builds`</nobr> | {% octicon "x" aria-label="Optional" %} | **Recommended.** Use to suppress the build command for languages where the {% data variables.product.prodname_codeql_cli %} does not need to monitor the build (for example, Python and JavaScript/TypeScript).
+| <nobr>`--no-run-unnecessary-builds`</nobr> | {% octicon "x" aria-label="Optional" %} | **Recommended.** Use to suppress the build command for languages where the {% data variables.product.prodname_codeql_cli %} does not need to monitor the build (for example, Python and JavaScript/TypeScript). |
 | <nobr>`--source-root`</nobr> | {% octicon "x" aria-label="Optional" %} | Use if you run the CLI outside the checkout root of the repository. By default, the `database create` command assumes that the current directory is the root directory for the source files, use this option to specify a different location. |
 | <nobr>`--codescanning-config`</nobr> | {% octicon "x" aria-label="Optional" %} | Advanced. Use if you have a configuration file that specifies how to create the {% data variables.product.prodname_codeql %} databases and what queries to run in later steps. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#using-a-custom-configuration-file)" and "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/database-create#--codescanning-configfile)." |
 
@@ -153,7 +154,8 @@ codeql database analyze &lt;database&gt; --format=&lt;format&gt; \
 | `<packs,queries>` | {% octicon "x" aria-label="Optional" %} | Specify {% data variables.product.prodname_codeql %} packs or queries to run. To run the standard queries used for {% data variables.product.prodname_code_scanning %}, omit this parameter. To see the other query suites included in the {% data variables.product.prodname_codeql_cli %} bundle, look in `/<extraction-root>/qlpacks/codeql/<language>-queries/codeql-suites`. For information about creating your own query suite, see [Creating CodeQL query suites](/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-query-suites) in the documentation for the {% data variables.product.prodname_codeql_cli %}.
 | <nobr>`--format`</nobr> | {% octicon "check" aria-label="Required" %} | Specify the format for the results file generated by the command. For upload to {% data variables.product.company_short %} this should be: {% ifversion fpt or ghae or ghec %}`sarif-latest`{% else %}`sarifv2.1.0`{% endif %}. For more information, see "[AUTOTITLE](/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)."
 | <nobr>`--output`</nobr> | {% octicon "check" aria-label="Required" %} | Specify where to save the SARIF results file.
-| <nobr>`--sarif-category`<nobr> | {% octicon "question" aria-label="Required with multiple results sets" %} | Optional for single database analysis. Required to define the language when you analyze multiple databases for a single commit in a repository.<br><br>Specify a category to include in the SARIF results file for this analysis. A category is used to distinguish multiple analyses for the same tool and commit, but performed on different languages or different parts of the code.|
+| <nobr>`--sarif-category`<nobr> | {% octicon "question" aria-label="Required with multiple results sets" %} | Optional for single database analysis. Required to define the language when you analyze multiple databases for a single commit in a repository.<br><br>Specify a category to include in the SARIF results file for this analysis. A category is used to distinguish multiple analyses for the same tool and commit, but performed on different languages or different parts of the code.|{% ifversion code-scanning-tool-status-page %}
+| <nobr>`--sarif-add-baseline-file-info`</nobr> | {% octicon "x" aria-label="Optional" %} | **Recommended.** Use to submit file coverage information to the tool status page. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page#how-codeql-defines-scanned-files)." | {% endif %}
 | <nobr>`--sarif-add-query-help`</nobr> | {% octicon "x" aria-label="Optional" %} | Use if you want to include any available markdown-rendered query help for custom queries used in your analysis. Any query help for custom queries included in the SARIF output will be displayed in the code scanning UI if the relevant query generates an alert. For more information, see "[AUTOTITLE](/code-security/codeql-cli/using-the-codeql-cli/analyzing-databases-with-the-codeql-cli#including-query-help-for-custom-codeql-queries-in-sarif-files)."{% ifversion codeql-packs %}
 | `<packs>` | {% octicon "x" aria-label="Optional" %} | Use if you want to include CodeQL query packs in your analysis. For more information, see "[Downloading and using {% data variables.product.prodname_codeql %} packs](#downloading-and-using-codeql-query-packs)."
 | <nobr>`--download`</nobr> | {% octicon "x" aria-label="Optional" %}  | Use if some of your CodeQL query packs are not yet on disk and need to be downloaded before running queries.{% endif %}
@@ -178,6 +180,21 @@ $ codeql database analyze /codeql-dbs/example-repo  \
 > Interpreting results.
 ```
 
+{% ifversion code-scanning-tool-status-page %}
+### Adding file coverage information to your results for monitoring
+
+You can optionally submit file coverage information to {% data variables.product.product_name %} for display on the tool status page for {% data variables.product.prodname_code_scanning %}. For more information about file coverage information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page#how-codeql-defines-scanned-files)."
+
+To include file coverage information with your {% data variables.product.prodname_code_scanning %} results, add the `--sarif-add-baseline-file-info` flag to the `codeql database analyze` invocation in your CI system, for example:
+
+```
+$ codeql database analyze /codeql-dbs/example-repo \
+    javascript-code-scanning.qls --sarif-category=javascript \
+    --sarif-add-baseline-file-info \ --format={% ifversion fpt or ghae or ghec %}sarif-latest{% else %}sarifv2.1.0{% endif %} \
+    --output=/temp/example-repo-js.sarif
+```
+
+{% endif %}
 ## Uploading results to {% data variables.product.product_name %}
 
 {% data reusables.code-scanning.upload-sarif-alert-limit %}
@@ -190,9 +207,11 @@ Before you can upload results to {% data variables.product.product_name %}, you
 When you have decided on the most secure and reliable method for your CI server, run `codeql github upload-results` on each SARIF results file and include `--github-auth-stdin` unless the token is available in the environment variable `GITHUB_TOKEN`.
 
   ```shell
-  echo "$UPLOAD_TOKEN" | codeql github upload-results --repository=&lt;repository-name&gt; \
+  echo "$UPLOAD_TOKEN" | codeql github upload-results \
-      --ref=&lt;ref&gt; --commit=&lt;commit&gt; --sarif=&lt;file&gt; \
+      --repository=&lt;repository-name&gt; \
-      {% ifversion ghes or ghae %}--github-url=&lt;URL&gt; {% endif %}--github-auth-stdin
+      --ref=&lt;ref&gt; --commit=&lt;commit&gt; \
+      --sarif=&lt;file&gt; {% ifversion ghes or ghae %}--github-url=&lt;URL&gt; \
+      {% endif %}--github-auth-stdin
   ```
 
 | Option | Required | Usage |
@@ -211,7 +230,8 @@ For more information, see "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manu
 This example uploads results from the SARIF file `temp/example-repo-js.sarif` to the repository `my-org/example-repo`. It tells the {% data variables.product.prodname_code_scanning %} API that the results are for the commit `deb275d2d5fe9a522a0b7bd8b6b6a1c939552718` on the `main` branch.
 
 ```
-$ echo $UPLOAD_TOKEN | codeql github upload-results --repository=my-org/example-repo \
+$ echo $UPLOAD_TOKEN | codeql github upload-results \
+    --repository=my-org/example-repo \
     --ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
     --sarif=/temp/example-repo-js.sarif {% ifversion ghes or ghae %}--github-url={% data variables.command_line.git_url_example %} \
     {% endif %}--github-auth-stdin
@@ -219,6 +239,43 @@ $ echo $UPLOAD_TOKEN | codeql github upload-results --repository=my-org/example-
 
 There is no output from this command unless the upload was unsuccessful. The command prompt returns when the upload is complete and data processing has begun. On smaller codebases, you should be able to explore the {% data variables.product.prodname_code_scanning %} alerts in {% data variables.product.product_name %} shortly afterward. You can see alerts directly in the pull request or on the **Security** tab for branches, depending on the code you checked out. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/triaging-code-scanning-alerts-in-pull-requests)" and "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/managing-code-scanning-alerts-for-your-repository)."
 
+
+{% ifversion code-scanning-tool-status-page %}
+## Uploading diagnostic information to {% data variables.product.product_name %} if the analysis fails
+
+When {% data variables.product.prodname_codeql_cli %} finishes analyzing a database successfully, it gathers diagnostic information such as file coverage, warnings, and errors, and includes it in the SARIF file with the results. When you upload the SARIF file to {% data variables.product.company_short %} the diagnostic information is displayed on the {% data variables.product.prodname_code_scanning %} tool status page for the repository to make it easy to see how well {% data variables.product.prodname_codeql %} is working and debug any problems. For more information, see "[AUTOTITLE](/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-the-tool-status-page)."
+
+However, if `codeql database analyze` fails for any reason there is no SARIF file to upload to {% data variables.product.company_short %} and no diagnostic information to show on the {% data variables.product.prodname_code_scanning %} tool status page for the repository. This makes it difficult for users to troubleshoot analysis unless they have access to log files in your CI system.
+
+We recommend that you configure your CI workflow to export and upload diagnostic information to {% data variables.product.product_name %} when an analysis fails. You can do this using the following simple commands to export diagnostic information and upload it to {% data variables.product.company_short %}.
+
+### Exporting diagnostic information if the analysis fails
+
+You can create a SARIF file for the failed analysis using "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/database-export-diagnostics)", for example:
+
+```bash
+$ codeql database export-diagnostics codeql-dbs/example-repo \
+    --sarif-category=javascript --format={% ifversion fpt or ghae or ghec %}sarif-latest{% else %}sarifv2.1.0{% endif %} \
+    --output=/temp/example-repo-js.sarif
+```
+
+This SARIF file will contain diagnostic information for the failed analysis, including any file coverage information, warnings, and errors generated during the analysis.
+
+### Uploading diagnostic information if the analysis fails
+
+You can make this diagnostic information available on the tool status page by uploading the SARIF file to {% data variables.product.product_name %} using "[AUTOTITLE](/code-security/codeql-cli/codeql-cli-manual/github-upload-results/)", for example:
+
+```bash
+$ echo $UPLOAD_TOKEN | codeql github upload-results \
+    --repository=my-org/example-repo \
+    --ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
+    --sarif=/temp/example-repo-js.sarif {% ifversion ghes or ghae %}--github-url={% data variables.command_line.git_url_example %} \
+    {% endif %}--github-auth-stdin
+```
+
+This is the same as the process for uploading SARIF files from successful analyses.
+{% endif %}
+
 {% ifversion codeql-packs %}
 ## Downloading and using {% data variables.product.prodname_codeql %} query packs
 
@@ -311,13 +368,15 @@ codeql database analyze codeql-dbs/python python-code-scanning.qls \
 
 # Upload the SARIF file with the Java results: 'java-results.sarif'
 
-echo $UPLOAD_TOKEN | codeql github upload-results --repository=my-org/example-repo \
+echo $UPLOAD_TOKEN | codeql github upload-results \
+    --repository=my-org/example-repo \
     --ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
     --sarif=java-results.sarif --github-auth-stdin
 
 # Upload the SARIF file with the Python results: 'python-results.sarif'
 
-echo $UPLOAD_TOKEN | codeql github upload-results --repository=my-org/example-repo \
+echo $UPLOAD_TOKEN | codeql github upload-results \
+    --repository=my-org/example-repo \
     --ref=refs/heads/main --commit=deb275d2d5fe9a522a0b7bd8b6b6a1c939552718 \
     --sarif=python-results.sarif --github-auth-stdin
 ```

data/features/code-scanning-tool-status-page.yml

@@ -1,4 +1,4 @@
-# Reference: #8882
+# Reference: #8882 for the new page and #10029 for CodeQL CLI information
 versions:
   fpt: '*'
   ghec: '*'