diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 116f36fc..979db1de 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -69,12 +69,7 @@ jobs: make setup - name: Build - run: make build - - - name: Integration Tests - #run: make test-integration-parallel - run: | - make test-integration + run: make build # Integration tests run in the build step. - uses: actions/upload-artifact@v4 with: @@ -83,10 +78,3 @@ jobs: pyscript.core/dist/ if-no-files-found: error retention-days: 7 - - - uses: actions/upload-artifact@v4 - if: success() || failure() - with: - name: test_results - path: test_results/ - if-no-files-found: error diff --git a/Makefile b/Makefile index 90a975a8..3124d6ff 100644 --- a/Makefile +++ b/Makefile @@ -12,7 +12,7 @@ all: @echo "make clean - clean up auto-generated assets." @echo "make build - build PyScript." @echo "make precommit-check - run the precommit checks (run eslint)." - @echo "make test-integration - run all integration tests sequentially." + @echo "make test - run all automated tests in playwright." @echo "make fmt - format the code." @echo "make fmt-check - check the code formatting.\n" @@ -62,15 +62,9 @@ build: precommit-check: pre-commit run --all-files -# Run all integration tests sequentially. -test-integration: - mkdir -p test_results - pytest -vv $(ARGS) pyscript.core/tests/integration/ --log-cli-level=warning --junitxml=test_results/integration.xml - -# Run all integration tests in parallel. -test-integration-parallel: - mkdir -p test_results - pytest --numprocesses auto -vv $(ARGS) pyscript.core/tests/integration/ --log-cli-level=warning --junitxml=test_results/integration.xml +# Run all automated tests in playwright. +test: + cd pyscript.core && npm run test:integration # Format the code. fmt: fmt-py diff --git a/README.md b/README.md index cbe1d7c6..01b6d7d3 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,8 @@ Read the [contributing guide](https://docs.pyscript.net/latest/contributing/) to Check out the [developing process](https://docs.pyscript.net/latest/developers/) documentation for more information on how to setup your development environment. +For technical details of the code, please see the [README](pyscript.core/README) in `pyscript.core`. + ## Governance The [PyScript organization governance](https://github.com/pyscript/governance) is documented in a separate repository. diff --git a/pyscript.core/README.md b/pyscript.core/README.md index 5aa22199..c4d19a05 100644 --- a/pyscript.core/README.md +++ b/pyscript.core/README.md @@ -37,13 +37,25 @@ make setup This will create a tests environment [in the root of the project, named `./env`]and install all the dependencies needed to run the tests. -After the command has completed and the tests environment has been created, you can run the **integration tests** with +After the command has completed and the tests environment has been created, you can run the **automated tests** with the following command: ``` -make test-integration +make test ``` +(This essentially runs the `npm run test:integration` command in the right place. This is defined in PyScript's `package.json` file.) + +Tests are found in the `tests` directory. These are organised into three locations: + +1. `python` - the Python based test suite to exercise Python code **within** PyScript. +2. `javascript` - JavaScript tests to exercise PyScript itself, in the browser. +3. `manual` - containing tests to run manually in a browser, due to the complex nature of the tests. + +We use [Playwright](https://playwright.dev/) to automate the running of the Python and JavaScript test suites. We use [uPyTest](https://github.com/ntoll/upytest) as a test framework for the Python test suite. uPyTest is a "PyTest inspired" framework for running tests in the browser on both MicroPython and Pyodide. + +The automated (Playwright) tests are specified in the `tests/integration.spec.js` file. + ## `pyscript` python package The `pyscript` package available in _Python_ lives in the folder `src/stdlib/pyscript/`. diff --git a/pyscript.core/package.json b/pyscript.core/package.json index 85deec26..5c971b66 100644 --- a/pyscript.core/package.json +++ b/pyscript.core/package.json @@ -58,7 +58,7 @@ "@ungap/with-resolvers": "^0.1.0", "@webreflection/idb-map": "^0.3.1", "basic-devtools": "^0.1.6", - "polyscript": "^0.15.7", + "polyscript": "^0.15.8", "sabayon": "^0.5.2", "sticky-module": "^0.1.1", "to-json-callback": "^0.1.1", diff --git a/pyscript.core/tests/README.md b/pyscript.core/tests/README.md new file mode 100644 index 00000000..b1a11c33 --- /dev/null +++ b/pyscript.core/tests/README.md @@ -0,0 +1,49 @@ +# PyScript Test Suite + +There are three aspects to our test suite. These are reflected in the layout of +the test directory: + +1. `python` - contains the Python based test suite to exercise Python code + **within** PyScript. These tests are run four differeng ways to ensure all + combination of MicroPython/Pyodide and main thread/worker contexts are + checked. +2. `javascript` - contains JavaScript tests to exercise PyScript _itself_, in + the browser. +3. `manual` - contains tests to run manually in a browser, due to the complex + nature of the tests. + +We use [Playwright](https://playwright.dev/) to automate the running of the +Python and JavaScript test suites. We use +[uPyTest](https://github.com/ntoll/upytest) as a test framework for the Python +test suite. uPyTest is a "PyTest inspired" framework for running tests in the +browser on both MicroPython and Pyodide. + +The automated (Playwright) tests are specified in the `integration.spec.js` +file in this directory. + +All automatic tests live in either the `python` or `javascript` folders. All +the tests in these folder are run by CI or locally run by `make test` in the +root of this project. Alternatively, run `npm run test:integration` in the +PyScript source directory. + +Similarly, some tests can only be run manually (due to their nature or +underlying complexity). These are in the `manual` directory and are in the form +of separate directories (each containing an `index.html`) or individual `*.html` +files to which you point your browser. Each separate test may exercise +JavaScript or Python code (or both), and the context for each separate test is +kept carefully isolated. + +Some rules of thumb: + +* We don't test upstream projects: we assume they have their own test suites, + and if we find bugs, we file an issue upstream with an example of how to + recreate the problem. +* We don't test browser functionality, we just have to trust that browsers work + as advertised. Once again, if we find an issue, we report upstream. +* All test cases should include commentary describing the **intent** and + context of the test. +* Tests in Python use [uPyTest](https://github.com/ntoll/upytest) (see the + README for documentation), an "inspired by PyTest" test framework that works + with both MicroPython and Pyodide in the browser. This means that all + Python tests should work with both interpreters. +* Tests in JavaScript... (Andrea to explain). ;-) diff --git a/pyscript.core/tests/index.html b/pyscript.core/tests/index.html index c5bbbe2e..eca49ea6 100644 --- a/pyscript.core/tests/index.html +++ b/pyscript.core/tests/index.html @@ -14,5 +14,5 @@ a:hover { opacity: 1; } -