Merge branch 'main' into sticky-github-logo-header

content/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-the-codeql-workflow-for-compiled-languages.md

@@ -40,10 +40,16 @@ If your workflow uses a `language` matrix, `autobuild` attempts to build each of
 
 | Supported system type | System name |
 |----|----|
-|  Operating system | Windows and Linux |
+| Operating system | Windows, macOS, and Linux |
-| Build system | Autoconf, CMake, qmake, Meson, Waf, SCons, and Linux Kbuild |
+| Build system | Windows: MSbuild and build scripts<br/>Linux and macOS: Autoconf, Make, CMake, qmake, Meson, Waf, SCons, Linux Kbuild, and build scripts |
 
-The behavior of the `autobuild` step varies according to the operating system that the extraction runs on. On Windows, the step has no default actions. On Linux, this step reviews the files present in the repository to determine the build system used:
+The behavior of the `autobuild` step varies according to the operating system that the extraction runs on. On Windows, the `autobuild` step attempts to autodetect a suitable build method for C/C++ using the following approach:
+
+1. Invoke `MSBuild.exe` on the solution (`.sln`) or project (`.vcxproj`) file closest to the root.
+If `autobuild` detects multiple solution or project files at the same (shortest) depth from the top level directory, it will attempt to build all of them.
+2. Invoke a script that looks like a build script—_build.bat_, _build.cmd_, _and build.exe_ (in that order).
+
+On Linux and macOS, the `autobuild` step reviews the files present in the repository to determine the build system used:
 
 1. Look for a build system in the root directory.
 2. If none are found, search subdirectories for a unique directory with a build system for C/C++.
@@ -67,7 +73,7 @@ If `autobuild` detects multiple solution or project files at the same (shortest)
 
 | Supported system type | System name |
 |----|----|
-|  Operating system | Windows, macOS and Linux (no restriction) |
+| Operating system | Windows, macOS, and Linux (no restriction) |
 | Build system | Gradle, Maven and Ant |
 
 The `autobuild` process tries to determine the build system for Java codebases by applying this strategy:

content/github/finding-security-vulnerabilities-and-errors-in-your-code/running-code-scanning-in-your-ci-system.md

@@ -97,7 +97,7 @@ The server has access to download the {{ site.data.variables.product.prodname_co
 
 #### Compiled language example
 
-This example is similar to the previous example, however this time the repository has code in C/C++, C#, or Java. To create a {{ site.data.variables.product.prodname_codeql }} database for these languages, the CLI needs to trace the build. At the end of the initialization process, the runner reports the command you need to set up the environment before building the code. You need to run this command, before calling the normal CI build process, and then running the `analyze` command.
+This example is similar to the previous example, however this time the repository has code in C/C++, C#, or Java. To create a {{ site.data.variables.product.prodname_codeql }} database for these languages, the CLI needs to monitor the build. At the end of the initialization process, the runner reports the command you need to set up the environment before building the code. You need to run this command, before calling the normal CI build process, and then running the `analyze` command.
 
 1. Check out the repository to analyze.
 1. Move into the directory where the repository is checked out.
@@ -114,7 +114,7 @@ This example is similar to the previous example, however this time the repositor
       . /srv/checkout/example-repo-2/codeql-runner/codeql-env.sh".
       ```
 
-1. Run the script generated by the `init` action to set up the environment to trace the build.
+1. Run the script generated by the `init` action to set up the environment to monitor the build.
 
     ```shell
     $ . /srv/checkout/example-repo-2/codeql-runner/codeql-env.sh

content/github/finding-security-vulnerabilities-and-errors-in-your-code/troubleshooting-code-scanning-in-your-ci-system.md

@@ -22,10 +22,30 @@ To avoid this automatic download, you can manually download the {{ site.data.var
 
 ### No code found during the build
 
-If the `analyze` command for the {{ site.data.variables.product.prodname_codeql_runner }} fails with an error `No source code was seen during the build`, this indicates that {{ site.data.variables.product.prodname_codeql }} was unable to trace your code. Several reasons can explain such a failure.
+If the `analyze` command for the {{ site.data.variables.product.prodname_codeql_runner }} fails with an error `No source code was seen during the build`, this indicates that {{ site.data.variables.product.prodname_codeql }} was unable to monitor your code. Several reasons can explain such a failure.
 
 1. Automatic language detection identified a supported language, but there is no analyzable code of that language in the repository. A typical example is when our language detection service finds a file associated with a particular programming language like a `.h`, or `.gyp` file, but no corresponding executable code is present in the repository. To solve the problem, you can manually define the languages you want to analyze by using the `--languages` flag of the `init` command. For more information, see "[Configuring {{ site.data.variables.product.prodname_code_scanning }} in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning-in-your-ci-system)."
 
-1. You're analyzing a compiled language without using the `autobuild` command and you run the build steps yourself after the `init` step. For the build to work, you must set up the environment such that the {{ site.data.variables.product.prodname_codeql_runner }} can trace the code. The `init` command generates instructions for how to export the required environment variables, so you can copy and run the script. For more information, see "[Running {{ site.data.variables.product.prodname_code_scanning }} in your CI system](/github/finding-security-vulnerabilities-and-errors-in-your-code/running-code-scanning-in-your-ci-system#compiled-language-example)."
+1. You're analyzing a compiled language without using the `autobuild` command and you run the build steps yourself after the `init` step. For the build to work, you must set up the environment such that the {{ site.data.variables.product.prodname_codeql_runner }} can monitor the code. The `init` command generates instructions for how to export the required environment variables, so you can copy and run the script after you've run the `init` command.
+   - On macOS and Linux:
+     ```shell
+      $ . codeql-runner/codeql-env.sh
+     ```
+   - On Windows, using the Command shell (`cmd`) or a batch file (`.bat`):
+     ```shell
+     > call codeql-runner\codeql-env.bat
+     ```
+   - On Windows, using PowerShell:
+     ```shell
+     > cat codeql-runner\codeql-env.sh | Invoke-Expression
+     ```
+
+   The environment variables are also stored in the file `codeql-runner/codeql-env.json`. This file contains a single JSON object which maps environment variable keys to values. If you can't run the script generated by the `init` command, then you can use the data in JSON format instead.
+
+   {% note %}
+
+   **Note:** If you used the `--temp-dir` flag of the `init` command to specify a custom directory for temporary files, the path to the `codeql-env` files might be different.
+
+   {% endnote %}
 
 1. The code is built in a container or on a separate machine. If you use a containerized build or if you outsource the build to another machine, make sure to run the {{ site.data.variables.product.prodname_codeql_runner }} in the container or on the machine where your build task takes place.

content/github/finding-security-vulnerabilities-and-errors-in-your-code/troubleshooting-the-codeql-workflow.md

@@ -44,7 +44,7 @@ If an automatic build of code for a compiled language within your project fails,
 
 ### No code found during the build
 
-If your workflow fails with an error `No source code was seen during the build` or `The process '/opt/hostedtoolcache/CodeQL/0.0.0-20200630/x64/codeql/codeql' failed with exit code 32`, this indicates that {{ site.data.variables.product.prodname_codeql }} was unable to trace your code. Several reasons can explain such a failure:
+If your workflow fails with an error `No source code was seen during the build` or `The process '/opt/hostedtoolcache/CodeQL/0.0.0-20200630/x64/codeql/codeql' failed with exit code 32`, this indicates that {{ site.data.variables.product.prodname_codeql }} was unable to monitor your code. Several reasons can explain such a failure:
 
 1. Automatic language detection identified a supported language, but there is no analyzable code of that language in the repository. A typical example is when our language detection service finds a file associated with a particular programming language like a `.h`, or `.gyp` file, but no corresponding executable code is present in the repository. To solve the problem, you can manually define the languages you want to analyze by updating the list of languages in the `language` matrix. For example, the following configuration will analyze only Go, and JavaScript.
 

includes/head.html

@@ -1,4 +1,5 @@
 <head>
+  <meta charset="utf-8" />
   <title>{% if error == '404' %}{{ site.data.ui.errors.oops }}{% elsif currentVersion == 'homepage' %}GitHub Documentation{% else %}{{ page.fullTitle }}{% endif %}</title>
   <meta name="viewport" content="width=device-width, initial-scale=1">{% if page.hidden %}
   <meta name="robots" content="noindex" />{% endif %}
@@ -25,14 +26,4 @@
   <link rel="stylesheet" href="/dist/index.css">
   <link rel="alternate icon" type="image/png" href="/assets/images/site/favicon.png">
   <link rel="icon" type="image/svg+xml" href="/assets/images/site/favicon.svg">
-
-  {% if page.relativePath contains "managing-your-work-on-github/disabling-project-boards-in-your-organization" %}
-  <!--TODO CSRF remove the outer check -->
-  {% if fastlyEnabled %}
-    <!-- https://www.fastly.com/blog/caching-uncacheable-csrf-security -->
-    <esi:include src="/csrf" />
-  {% else %}
-    <meta name="csrf-token" content="{{ csrfToken }}" />
-  {% endif %}
-  {% endif %}
 </head>

javascripts/get-csrf.js

@@ -1,3 +1,17 @@
+export async function fillCsrf () {
+  const response = await fetch('/csrf', {
+    method: 'GET',
+    headers: {
+      'Content-Type': 'application/json'
+    }
+  })
+  const json = response.ok ? await response.json() : {}
+  const meta = document.createElement('meta')
+  meta.setAttribute('name', 'csrf-token')
+  meta.setAttribute('content', json.token)
+  document.querySelector('head').append(meta)
+}
+
 export default function getCsrf () {
   const csrfEl = document
     .querySelector('meta[name="csrf-token"]')

javascripts/index.js

@@ -13,6 +13,7 @@ import print from './print'
 import localization from './localization'
 import helpfulness from './helpfulness'
 import experiment from './experiment'
+import { fillCsrf } from './get-csrf'
 
 document.addEventListener('DOMContentLoaded', () => {
   displayPlatformSpecificContent()
@@ -26,6 +27,7 @@ document.addEventListener('DOMContentLoaded', () => {
   wrapCodeTerms()
   print()
   localization()
+  fillCsrf()
   helpfulness()
   experiment()
 })

layouts/default.html

@@ -1,3 +1,4 @@
+<!doctype html>
 <html lang="{{currentLanguage}}">
   {% include head %}
 

layouts/product-landing.html

@@ -1,3 +1,4 @@
+<!doctype html>
 <html lang="{{currentLanguage}}">
   {% include head %}
 

middleware/context.js

@@ -55,14 +55,6 @@ module.exports = async function contextualize (req, res, next) {
   req.context.siteTree = siteTree
   req.context.pages = pages
 
-  // To securely accept data from end users,
-  // we need to validate that they were actually on a docs page first.
-  // We'll put this token in the <head> and call on it when we send user data
-  // to the docs server, so we know the request came from someone on the page.
-  req.context.csrfToken = req.csrfToken()
-  req.context.fastlyEnabled = process.env.NODE_ENV === 'production' &&
-    req.hostname === 'docs.github.com'
-
   return next()
 }
 

middleware/csrf-route.js

@@ -8,7 +8,7 @@ router.get('/', (req, res) => {
     'surrogate-control': 'private, no-store',
     'cache-control': 'private, no-store'
   })
-  res.send(`<meta name="csrf-token" content="${req.context.csrfToken}" />`)
+  res.json({ token: req.csrfToken() })
 })
 
 module.exports = router

middleware/csrf.js

@@ -1,5 +1,4 @@
 module.exports = require('csurf')({
   cookie: require('../lib/cookie-settings'),
-  ignoreMethods: ['GET', 'HEAD', 'OPTIONS', 'POST', 'PUT']
+  ignoreMethods: ['GET', 'HEAD', 'OPTIONS']
-  // TODO CSRF edit this to include POST and PUT to require it
 })

tests/browser/browser.js

@@ -135,6 +135,13 @@ describe('helpfulness', () => {
   })
 })
 
+describe('csrf meta', () => {
+  it('should have a csrf-token meta tag on the page', async () => {
+    await page.goto('http://localhost:4001/en/actions/getting-started-with-github-actions/about-github-actions')
+    await page.waitForSelector('meta[name="csrf-token"]')
+  })
+})
+
 async function getLocationObject (page) {
   const location = await page.evaluate(() => {
     return Promise.resolve(JSON.stringify(window.location, null, 2))

tests/rendering/csrf-route.js

@@ -9,7 +9,7 @@ describe('GET /csrf', () => {
   it('should render a non-cache include for CSRF token', async () => {
     const res = await request(app).get('/csrf')
     expect(res.status).toBe(200)
-    expect(res.text).toMatch(/^<meta name="csrf-token" content="(.*?)" \/>$/)
+    expect(res.body).toHaveProperty('token')
     expect(res.headers['surrogate-control']).toBe('private, no-store')
     expect(res.headers['cache-control']).toBe('private, no-store')
   })

tests/rendering/events.js

@@ -11,14 +11,23 @@ Airtable.mockImplementation(function () {
 })
 
 describe('POST /events', () => {
-  beforeEach(() => {
+  jest.setTimeout(60 * 1000)
+
+  let csrfToken = ''
+  let agent
+
+  beforeEach(async () => {
     process.env.AIRTABLE_API_KEY = '$AIRTABLE_API_KEY$'
     process.env.AIRTABLE_BASE_KEY = '$AIRTABLE_BASE_KEY$'
+    agent = request.agent(app)
+    const csrfRes = await agent.get('/csrf')
+    csrfToken = csrfRes.body.token
   })
 
   afterEach(() => {
     delete process.env.AIRTABLE_API_KEY
     delete process.env.AIRTABLE_BASE_KEY
+    csrfToken = ''
   })
 
   describe('HELPFULNESS', () => {
@@ -32,82 +41,93 @@ describe('POST /events', () => {
     }
 
     it('should accept a valid object', () =>
-      request(app)
+      agent
         .post('/events')
         .send(example)
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(201)
     )
 
     it('should reject extra properties', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, toothpaste: false })
         .set('Accept', 'application/json')
+
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if type is missing', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, type: undefined })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if url is missing', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, url: undefined })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if url is misformatted', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, url: 'examplecom' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if vote is missing', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, vote: undefined })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if vote is not boolean', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, vote: 'true' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if email is misformatted', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, email: 'testexample.com' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if comment is not string', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, comment: [] })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should not accept if category is not an option', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, category: 'Fabulous' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
   })
@@ -122,64 +142,79 @@ describe('POST /events', () => {
     }
 
     it('should accept a valid object', () =>
-      request(app)
+      agent
         .post('/events')
         .send(example)
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(201)
     )
 
     it('should reject extra fields', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, toothpaste: false })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should require a long unique user-id', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, 'user-id': 'short' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should require a test', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, test: undefined })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should require a valid group', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, group: 'revolution' })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(400)
     )
 
     it('should default the success field', () =>
-      request(app)
+      agent
         .post('/events')
         .send({ ...example, success: undefined })
         .set('Accept', 'application/json')
+        .set('csrf-token', csrfToken)
         .expect(201)
     )
   })
 })
 
 describe('PUT /events/:id', () => {
-  beforeEach(() => {
+  jest.setTimeout(60 * 1000)
+
+  let csrfToken = ''
+  let agent
+
+  beforeEach(async () => {
     process.env.AIRTABLE_API_KEY = '$AIRTABLE_API_KEY$'
     process.env.AIRTABLE_BASE_KEY = '$AIRTABLE_BASE_KEY$'
+    agent = request.agent(app)
+    const csrfRes = await agent.get('/csrf')
+    csrfToken = csrfRes.body.token
   })
 
   afterEach(() => {
     delete process.env.AIRTABLE_API_KEY
     delete process.env.AIRTABLE_BASE_KEY
+    csrfToken = ''
   })
 
   const example = {
@@ -192,10 +227,11 @@ describe('PUT /events/:id', () => {
   }
 
   it('should update an existing HELPFULNESS event', () =>
-    request(app)
+    agent
       .put('/events/TESTID')
       .send(example)
       .set('Accept', 'application/json')
+      .set('csrf-token', csrfToken)
       .expect(200)
   )
 })