diff --git a/src/fixtures/README.md b/src/fixtures/README.md index e69de29bb2..0e7e299386 100644 --- a/src/fixtures/README.md +++ b/src/fixtures/README.md @@ -0,0 +1,105 @@ +# Fixture content + +Fixture content is content (and data) that is meant to look very similar +to the real content, but exists for the benefit of testing functionality. + +In its simplest form, code and content is intricately linked, and oftentimes +to be able to have automated testing of functionality, you need some content +to exercise that functionality. + +Our fixture content exists so we can write and run end-to-end tests against +content that is specifically tied to making sure the functionality +sustainably works when we're changing any code, but without having to +worry about the real English content breaking the tests. + +**Note!** We also don't want the writers of the real English content to +have to worry about breaking tests of functionality. + +## How to write fixtured based rendering tests + +The content is in `src/fixtures/fixtures/content/` (and `src/fixtures/fixtures/data/`) +is a cut down version of the real `content/` (and `data/`) at the root. +It doesn't have nearly as many pages and if you look closely you'll see +references and mentions to unrealistic things like "foo" or "HubGit" +which are whimsical but also importantly *different*. If it works +with any silly name, the code is modular and good. + +### Quickstart + +Navigate around in `src/fixtures/fixtures/content/` and familiarize yourself +with the directory structure. The only things that are "identical" to the +real content is the top-level product names which match the real content. +Deeper than the product level, the names and directories can be whatever +you want it to be. + +Once you've found a place to put some fixture content, before writing +a `vitest` test, you can preview your changes using: + +```shell +npm run fixture-dev +``` + +and navigate to to see your fixture content in +action. + +### Write the tests + +Feel free to create sub-directories or new files. For example, if it's +about end-to-end testing a new custom Liquid tag called +`lib/liquid-tags/snacks.js` you create a new test called +`src/fixtures/tests/snack.js`. (And equally, you might want to create +`src/fixtures/fixtures/content/get-started/foo/snacking.md`) + +To run the tests use: + +```shell +ROOT=src/fixtures/fixtures vitest src/fixtures/tests +``` + +### Exceptions + +The top-level product names in the fixture content needs to be a perfect +subset of the product names in the real content. That's because they +get compiled in to the Next rewrite functionality so we can support +URLs that actually are `free-pro-team@latest` without mentioning it in +the URL. + +Another exception is some data files that straddle real content and +support functionality. For example, `data/ui.yml` is part of the +functionality (e.g. React components) but lives in the `data/` directory +so its translation repos copies can be translated. + +There's a script you can always run that makes sure all and any of these +files are up to do: + +```shell +./src/tests/scripts/copy-fixture-data.js +``` + +It's safe to run any time. And it might be necessary to run so that +the fixture data gets a fresh copy. + +### Tip! Own it + +The advantage with fixture content for testing is that you can control it. +It's less likely now that your tests break because of some other change. +Similar to unit testing strategies, try to keep things in small units that +worries about *one thing at a time*. + +Don't be afraid to write a `vitest` test that is very specific about what it +tests. It might seem strange when someone is only reading the tests directly. +But the fixtures are part of the tests. It's just in different files. + +## Running a fixture test locally + +When running fixtures tests locally, you must override the default `ROOT` and `TRANSLATIONS_FIXTURE_ROOT` environment variables to point to fixture directories: + +```shell +ROOT=src/fixtures/fixtures TRANSLATIONS_FIXTURE_ROOT=src/fixtures/fixtures/translations vitest src/fixtures/tests +``` + +Optionally, also set `DEBUG_MIDDLEWARE_TESTS` variable to get stacktraces for `500` internal server errors: + +```shell +DEBUG_MIDDLEWARE_TESTS=true ROOT=src/fixtures/fixtures TRANSLATIONS_FIXTURE_ROOT=src/fixtures/fixtures/translations vitest src/fixtures/tests +``` diff --git a/src/fixtures/fixtures/README.md b/src/fixtures/fixtures/README.md deleted file mode 100644 index e4eca353fd..0000000000 --- a/src/fixtures/fixtures/README.md +++ /dev/null @@ -1,91 +0,0 @@ -# Fixture content - -Fixture content is content (and data) that is meant to look very similar -to the real content, but exists for the benefit of testing functionality. - -In its simplest form, code and content is intricately linked, and oftentimes -to be able to have automated testing of functionality, you need some content -to exercise that functionality. - -Our fixture content exists so we can write and run end-to-end tests against -content that is specifically tied to making sure the functionality -sustainably works when we're changing any code, but without having to -worry about the real English content breaking the tests. - -**Note!** We also don't want the writers of the real English content to -have to worry about breaking tests of functionality. - -## How to write fixtured based rendering tests - -The content is in `src/fixtures/fixtures/content/` (and `src/fixtures/fixtures/data/`) -is a cut down version of the real `content/` (and `data/`) at the root. -It doesn't have nearly as many pages and if you look closely you'll see -references and mentions to unrealistic things like "foo" or "HubGit" -which are whimsical but also importantly *different*. If it works -with any silly name, the code is modular and good. - -### Quickstart - -Navigate around in `src/fixtures/fixtures/content/` and familiarize yourself -with the directory structure. The only things that are "identical" to the -real content is the top-level product names which match the real content. -Deeper than the product level, the names and directories can be whatever -you want it to be. - -Once you've found a place to put some fixture content, before writing -a `vitest` test, you can preview your changes using: - -```shell -npm run fixture-dev -``` - -and navigate to to see your fixture content in -action. - -### Write the tests - -Feel free to create sub-directories or new files. For example, if it's -about end-to-end testing a new custom Liquid tag called -`lib/liquid-tags/snacks.js` you create a new test called -`src/fixtures/tests/snack.js`. (And equally, you might want to create -`src/fixtures/fixtures/content/get-started/foo/snacking.md`) - -To run the tests use: - -```shell -ROOT=src/fixtures/fixtures vitest src/fixtures/tests -``` - -### Exceptions - -The top-level product names in the fixture content needs to be a perfect -subset of the product names in the real content. That's because they -get compiled in to the Next rewrite functionality so we can support -URLs that actually are `free-pro-team@latest` without mentioning it in -the URL. - -Another exception is some data files that straddle real content and -support functionality. For example, `data/ui.yml` is part of the -functionality (e.g. React components) but lives in the `data/` directory -so its translation repos copies can be translated. - -There's a script you can always run that makes sure all and any of these -files are up to do: - -```shell -./src/tests/scripts/copy-fixture-data.js -``` - -It's safe to run any time. And it might be necessary to run so that -the fixture data gets a fresh copy. - -### Tip! Own it - -The advantage with fixture content for testing is that you can control it. -It's less likely now that your tests break because of some other change. -Similar to unit testing strategies, try to keep things in small units that -worries about *one thing at a time*. - -Don't be afraid to write a `vitest` test that is very specific about what it -tests. It might seem strange when someone is only reading the tests directly. -But the fixtures are part of the tests. It's just in different files. diff --git a/src/tests/README-1.md b/src/tests/README-1.md deleted file mode 100644 index 91455e25ac..0000000000 --- a/src/tests/README-1.md +++ /dev/null @@ -1,120 +0,0 @@ -## Tests - -It's not strictly necessary to run tests locally while developing. You can -always open a pull request and rely on the CI service to run tests for you, -but it's helpful to run tests locally before pushing your changes to -GitHub. - -Tests are written using [vitest](https://vitest.dev/). - -`vitest` runs tests and handles assertions. - -### Install optional dependencies - -We typically rely on CI to run our tests, so some large test-only -dependencies are considered **optional**. To run the tests locally, you'll -need to make sure optional dependencies are installed by running: - -```shell -npm ci --include=optional -``` - -### Running all the tests - -Once you've followed the development instructions above, you can run the entire -test suite locally: - -```shell -npm test -``` - -### Watching all the tests - -You can run a script that continually watches for changes and -re-runs the tests whenever a change is made. This command notifies you -when tests change to and from a passing or failing state, and it prints -out a test coverage report so you can see what files need testing. - -```shell -npm run test-watch -``` - -### Running individual tests - -You can run specific tests in two ways: - -```shell -# The TEST_NAME can be a filename, partial filename, or path to a file or directory -npm test -- - -vitest path/to/tests/directory -``` - -### Failed Local Tests - -If the tests fail locally with an error like this: - -`Could not find a production build in the '/Users/username/repos/docs-internal/.next' directory.` - -You may need to run this before every test run: - -```shell -npx next build -``` - -### Linting - -To validate all your JavaScript code (and auto-format some easily reparable mistakes), -run the linter: - -```shell -npm run lint -``` - -### Keeping the server running - -When you run `vitest` tests that depend on making real HTTP requests -to `localhost:4000`, the `vitest` tests have a hook that starts the -server before running all/any tests and stops the server when done. - -You can disable this, which might make it easier when debugging tests -since the server won't need to start and stop every time you run tests. - -In one terminal, type: - -```shell -NODE_ENV=test PORT=4000 tsx src/frame/server.ts -``` - -In another terminal, type: - -```shell -START_VITEST_SERVER=false vitests src/versions/tests -``` - -Or whatever the testing command you use is. - -The `START_VITEST_SERVER` environment variable needs to be set to `false`, -or else `vitest` will try to start a server on `:4000` too. - -### Debugging middleware errors - -By default, errors handled by the middleware are dealt with just like -any error in production. It's common to have end-to-end tests that expect -a page to throw a 500 Internal Server Error response. - -If you don't expect that and you might struggle to see exactly where the -error is happening, set `$DEBUG_MIDDLEWARE_TESTS` to `true`. For example: - -```shell -export DEBUG_MIDDLEWARE_TESTS=true -vitest src/shielding/tests -b -``` - -### Fixture based testing - -See [Fixture content](./fixtures/README.md). - -### Headless tests with Playwright - -See [Headless tests with Playwright](./PLAYWRIGHT.md) diff --git a/src/tests/README.md b/src/tests/README.md index 74b28deac8..c0ae0e0f30 100644 --- a/src/tests/README.md +++ b/src/tests/README.md @@ -3,3 +3,122 @@ This directory contains utilities to support our automated testing efforts. **This directory should not include test suites.** Please use the best subject folder available. + +It's not strictly necessary to run tests locally while developing. You can +always open a pull request and rely on the CI service to run tests for you, +but it's helpful to run tests locally before pushing your changes to +GitHub. + +Tests are written using [vitest](https://vitest.dev/). + +`vitest` runs tests and handles assertions. + +## Install optional dependencies + +We typically rely on CI to run our tests, so some large test-only +dependencies are considered **optional**. To run the tests locally, you'll +need to make sure optional dependencies are installed by running: + +```shell +npm ci --include=optional +``` + +## Running all the tests + +Once you've followed the development instructions above, you can run the entire +test suite locally: + +```shell +npm test +``` + +## Watching all the tests + +You can run a script that continually watches for changes and +re-runs the tests whenever a change is made. This command notifies you +when tests change to and from a passing or failing state, and it prints +out a test coverage report so you can see what files need testing. + +```shell +npm run test-watch +``` + +## Running individual tests + +You can run specific tests in two ways: + +```shell +# The TEST_NAME can be a filename, partial filename, or path to a file or directory +npm test -- + +vitest path/to/tests/directory +``` + +## Failed Local Tests + +If the tests fail locally with an error like this: + +`Could not find a production build in the '/Users/username/repos/docs-internal/.next' directory.` + +You may need to run this before every test run: + +```shell +npx next build +``` + +## Linting + +To validate all your JavaScript code (and auto-format some easily reparable mistakes), +run the linter: + +```shell +npm run lint +``` + +## Keeping the server running + +When you run `vitest` tests that depend on making real HTTP requests +to `localhost:4000`, the `vitest` tests have a hook that starts the +server before running all/any tests and stops the server when done. + +You can disable this, which might make it easier when debugging tests +since the server won't need to start and stop every time you run tests. + +In one terminal, type: + +```shell +NODE_ENV=test PORT=4000 tsx src/frame/server.ts +``` + +In another terminal, type: + +```shell +START_VITEST_SERVER=false vitests src/versions/tests +``` + +Or whatever the testing command you use is. + +The `START_VITEST_SERVER` environment variable needs to be set to `false`, +or else `vitest` will try to start a server on `:4000` too. + +## Debugging middleware errors + +By default, errors handled by the middleware are dealt with just like +any error in production. It's common to have end-to-end tests that expect +a page to throw a 500 Internal Server Error response. + +If you don't expect that and you might struggle to see exactly where the +error is happening, set `$DEBUG_MIDDLEWARE_TESTS` to `true`. For example: + +```shell +export DEBUG_MIDDLEWARE_TESTS=true +vitest src/shielding/tests -b +``` + +## Fixture based testing + +See [Fixture content](src/fixtures/README.md). + +## Headless tests with Playwright + +See [Headless tests with Playwright](src/fixtures/PLAYWRIGHT.md)