* Add runtime.invalidate_module_path_cache() to clear importlib's cache of modules
* Clear cache after [[fetch]], after plugin files download, after plugins setup, and prior to py-script tag execution.
* add test and example files
* update config to include python plugins in build
* add markdown plugin
* remove full pyscript execution from pyodide
* move loading of pyscript.py from pyodide loagInterpreter to main setupVirtualEnv and add function to create python CE plugins
* add plugin class to pyscript.py
* add missing import
* fix plugin path
* add fetchPythonPlugins to PyScriptApp
* remove old comments
* fix test
* add support for python plugins beyond custom elements and add app to python namespace in main
* inject reference to PyScript app onto python plugins
* add example hook onto markdown plugin
* change plugin events logs
* remove unused PyPlugin
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix type import
* add docstring to fetchPythonPlugins
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* rename addPythonPlugin method
* address PR comment
* call python plugins on hooks after the interpreted is ready
* add test for event hooks and split the test in 2 separate plugins to isolte type of plugins tests
* change python plugins initialization and registration, to inject the app from app itself instead of on the plugins themselves
* handle case when plugin cannot load due to missing plugin attribute
* add test for fail scenario when a plugin module does not have a plugin attribute
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add deprecation warning for pyscript objects loaded in global namespace
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove all from global scope
* remove create_custom_element from global scope
* rename create_custom_element to define_custom_element
* rename attributes in define_custom_element and add docstrings
* better handle connect event output
* add warning to py_markdown plugin
* remove debugging logs
* improve tests
* remove debugging log
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused import
* add executable shebang
* add pyodide mock module
* fmt and lint
* Update to pyodide.ffi.create_proxy per pyodide v21 api change
* Mock pyodide as package instead of mdoule
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add __init__ to pyodide package
* Update pyscriptjs/src/plugin.ts
fix logger name
Co-authored-by: Antonio Cuni <anto.cuni@gmail.com>
* fix pyodide import but handling the diff in their API change
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* oops, conflict resolution blooper
* Fix failing integration tests
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jeff Glass <glass.jeffrey@gmail.com>
Co-authored-by: Antonio Cuni <anto.cuni@gmail.com>
Co-authored-by: FabioRosado <fabiorosado@outlook.com>
* Import current version as PyScript.__version__ and PyScript.version_info
* Add integration, python unit tests
* Mock pyscript.py with __mocks__/pyscript.js to make jest tests happy
* Revert to runPython instead of await runPythonAsync
* "Implicit Coroutines" are no longer permitted in py-script tags
* Tests added for the above
* xfail test_importmap (See #938)
This PR move codes from main.ts into two new plugins:
- splashscreen (formerly known as py-loader)
- importmap
The old setting config.autoclose_loader is still supported but deprecated; the new setting is config.splashscreen.autoclose.
Moreover, it does a small refactoring around UserError: now UserErrors are correctly caught even if they are raised from within afterRuntimeLoad.
* Raise error if we get a non 200 status from fetch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add custom exception
* Add check for TypeError as well
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* use pytest-cache to cache HTTP requests, so that it survives across multiple runs
* add an option to clear the HTTP cache
* add some docs about pytest
This PR does two things:
1. introduce the first minimal version of a Plugin system. Plugins are subclasses of the Plugin class, and pyscript will call the relevant methods/hooks at the right time. Currently, I have added only the minimal sets of hooks which were needed for this PR.
2. Implement <py-terminal> as a plugin.
* implement proposal
* update docs and replace py-env
* more docs
* suggested proposal
* update docs
* add to_file parameter
* remove comment from Makefile
* suggested improvements
* move tests from basic to py_config
* retain leading slash from the first path
- display(some_str) escapes the string by default. This is almost always what you want
- display(some_obj) calls repr(obj) and escapes the result. Again, it's a very sensible default
- if you want to inject some raw HTML in the output, you can use the new HTML class: display(HTML("<p>hello</p>")).
Major highlights:
1. Merge&simplify base.ts and pyrepl.ts; kill base.ts
2. improve and extente the py-repl integration tests
3. Reorder the code in pyrepl.ts. This part of the PR doesn't change much of the concrete logic: it's just a sequence of renaming variables, moving code around, group code into functions, killing code which is no longer needed. But the end result is much better and nicer to read, IMHO.
Minor highlights:
1. py-repl now uses the new logic in pyexec.ts to run the code
2. after PR Add display impl, rm outputManage, print and console.log default to browser console #749 py-repl no longer displayed the result of the last evaluated expression (e.g. if you typed 42 and run it, it displayed nothing). This PR re-introduces this behavior, which is what you would expect by a REPL.
3. improve the pytest --dev option: now it implies --no-fake-server so that sourcemaps works automatically
4. improve the names of the CSS classes to be more consistent
5. kill pyrepl.test.ts: the old tests didn't check anything useful, this style of unit test doesn't really add much value if you have good integration tests (which now we have) and trying to revive them was not worth the hassle
Until now, we didn't have a nice way to check that we expect a specific JS error in the web page.
This PR improves check_js_errors() so that now you can pass a list of error messages that you expect.
It is tricky because we need to handle (and test!) all various combinations of cases:
- errors expected and found / expected but not found
- unexpected errors found / not found
Moreover, JS exceptions now are logged in the special category console.js_error, which means that the printed text is also available using e.g. self.console.js_error.text or self.console.all.text. However, this should never be required and it's preferred to use self.check_js_errors to check for exceptions. This fixes#795 .
Finally, use the new logic to improve test_no_implicit_target.
Yet another refactoring to untangle the old mess.
Highlights:
base.ts, pyscript.ts and pyrepl.ts were a tangled mess of code, in which each of them interacted with the others in non-obvious ways. Now PyScript is no longer a subclass of BaseEvalElement and it is much simpler. I removed code for handling the attributes std-out and std-err because they are no longer needed with the new display() logic.
The logic for executing python code is now in pyexec.ts: so we are decoupling the process of "finding" the python code (handled by the py-script web component) and the logic to actually execute it. This has many advantages, including the fact that it will be more easily usable by other components (e.g. pyrepl). Also, note that it's called pyexec and not pyeval: in the vast majority of cases in Python you have statements to execute, and almost never expressions to evaluate.
I killed the last remaining global store, scriptQueue tada. As a bonus effect, now we automatically do the correct thing when a <py-script> tag is dynamically added to the DOM (I added a test for it). I did not remove svelte from packages.json, because I don't fully understand the implications: there are various options which mention svelte in rollup.js and tsconfig.json, so it's probably better to kill it in its own PR.
pyexec.ts is also responsible of handling the default target for display() and correct handling/visualization of exceptions. I fixed/improved/added display/output tests in the process.
I also found a problem though, see issue #878, so I improved the test and marked it as xfail.
I removed BaseEvalElement as the superclass of most components. Now the only class which inherits from it is PyRepl. In a follow-up PR, I plan to merge them into a single class and do more cleanup.
During the refactoring, I killed guidGenerator: now instead of generating random py-* IDs which are very hard to read for humans, we generated py-internal-X IDs, where X is 0, 1, 2, 3, etc. This makes writing tests and debugging much easier.
I improved a lot our test machinery: it turns out that PR #829 broke the ability to use/view sourcemaps inside the playwright browser (at least on my machine).
For some reason chromium is unable to find sourcemaps if you use playwrights internal routing. So I reintroduced the http_server fixture which was removed by that PR, and added a pytest option --no-fake-server to use it instead, useful for debugging. By default we are still using the fakeserver though (which is faster and parallelizable).
Similarly, I added --dev which implies --headed and also automatically open chrome dev tools.
* export 'pyscript' JS module with runtime attribute, to allow accessing (Pyodide) runtime and globals from JS.
* add docs to explain the js module
* add integration tests for the js module
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* use emscripten virtual FS directly to load paths
* use more low level APIs
* use await instead of then
* remove try...catch from loadFromFile since it's being handled externally
* add test for an invalid path
* test checks for error shown to the user too
* add comment about a missing case
* Add display impl, remove outputManage, print and console.log defaults to terminal
* Fixing tests
* Lint
* Erase unecessary code, add cuter CSS formating for errors, fix problems around REPL output
* Add fix to repl2 and lint
* lint
* Allow for list of display, fix elif to else
* Add better global option
* test work
* xfails
* (antocuni, mariana): let's try to start again with TDD methodology: write the minimum test and code for a simple display()
* (antocuni, mariana): this test works out of the box
* WIP: this test is broken, mariana is going to fix it
* add a failing test
* Add ability to deal with targets
* Add append arg and append tests
* Add multiple values to display
* Small adjustments to tests. I noticed I wasn;t running all at some point
* add display test
* Add console tests
* Add async tests
* Fix repl tests
* Fixing merging issues
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address antocuni's review
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixing more tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* linting
* Improve repl tests
* Change my test so codespell is hapy with it
* Test: change test_runtime_config to use json instead of toml to see if stops failing on CI
* kill this file: it is a merge artifact since it was renamed into test_py_config.py on the main branch
* Change test execution order and add async tests to async test file
Co-authored-by: Antonio Cuni <anto.cuni@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
As the title stays, the main goal of the branch is to kill the infamous runtimeLoaded global store and all the complications, problems and bugs caused by the fact that in many places we needed to ensure/wait that the global runtime was properly set before being able to execute code.
The core idea is that runtime is never a global object and that it's passed around explicitly, which means that when a function receives it, it is guaranteed to be initialized&ready.
This caused a bit of complications in pybutton.ts, pyinputbox.ts and pyrepl.ts, because they indirectly want to call runtime.run from connectedCallback, which is the only place where we cannot explicitly pass the runtime because it's automatically called by the browser.
But also, it is also a sign of a bad design, because it were entirely possible that connectedCallback was called before the runtime was ready, which probably caused many bugs, see e.g. #673 and #747.
The solution to is use dependency injection and create the class later on: so instead of having a global PyButton class which relies on a global runtime (whose state is uncertain) we have a make_PyButton function which takes a runtime and make a PyButton class which is tied to that specific runtime (whose state is certainly ready, because we call make_PyButton only when we know that the runtime is ready).
Similar for PyInputBox and PyRepl.
Other highlights: thanks to this, I could kill the also infamous runAfterRuntimeInitialized and a couple of smelly lines which used setTimeout to "wait" for the runtime.
While I was at it, I also called a lot of other stores which were completely unused and where probably leftovers from a past universe.
* Hide py-config element
* display:none for all web components
* docs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add style tests for raw html and pyscript enabled html
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test naming
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add cache, fetch, retry logic to tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* run in parallel
* add pytest-xdist
* undo parallelism. Need to remove http server to enable.
* woops a extra space
* Pass flake8
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* spell fulfill
* use decorator for fetch if not in cache
* Fix --headed and limit to PlaywrightRequestError
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docs on cache
* CICD caching of conda on unstable builds
* fix config issues
* empty commit to trigger gh-actions
* restore build-unstable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove http server, add parallel
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* temp: Bypass zip runtime test and point to v0.21.3 on CDN
* suport for files in zip under /pyodide
* remove test-one
* self.http_server and remove content_type
* domcontentloaded w no timeout on base url + longer timeout on wait_for_pyscript
* Fixed#678
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* set default timeout to 60000
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* seamless --headed support
* add test-integration-parallel and default for GHActions
* simplify the code. Use http://fakeserver instead of localhost:8080 so that it's clearer that the browser is NOT hitting a real server, and use urllib to parse the url. Moreover, the special case for pyodide is no longer needed, it's automatically handled by the normal 'fakeserver' logic
* The page-routing logic is becoming too much complicated to stay as an inner function. Move it to its own class, and add some logic to workaround a limitation of playwright which just hangs if a Python exception is raised inside it
* no need to use a hash, we can use the url as the key
* re-implement the retry logic. The old @retry decorator was nice but a bit too over-engineered and most importantly failed silently in case of exceptions. This new approach is less powerful but since we want to retry only two times, simple is better than complex -- and in case of exception, the exception is actually raised
* improve logging
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Madhur Tandon <madhurtandon23@gmail.com>
Co-authored-by: Antonio Cuni <anto.cuni@gmail.com>
This is a follow-up of PR #806 and it's a big step forward towards solving issue #763.
The basic idea is that at this stage I want to streamline the execution logic
and make it as linear and easy to read/understand as possible. The diff is
relatively big, but for the most part is just "shuffling code around".
Svelte stores:
the idea is to eventually kill of them, so that we can remove the dependency
on svelte but also to avoid relying on so much global state, which makes
things more complicated and error-prone (e.g., we have several issues related
to using runtime when it's not ready yet).
I killed addInitializer, addPostInitializer and the corresponding svelte
stores tada. They are no longer needed since the relevant code is called
directly from main.ts.
I started to kill the usage of the runtimeLoaded svelte store: instead of
relying on a global variable, I want to arrive at the point in which the
runtime is passed as a parameter in all places where it's needed: pyscript.ts
is now free of global state, but I couldn't kill it yet because it's used
heavily by base.ts and pyrepl.ts. I will do it in another PR.
Other misc changes:
I added sanity checks (and corresponding tests!) which complain if you specify
0 or multiple runtimes. Currently we support having one and only one, so there
is no point to pretend otherwise
I modified the messages displayed by the loader, to be more informative from
the user point of view.
This PR is the first step to improve and rationalize the life-cycle of a pyscript app along the lines of what I described in #763 .
It is not a complete solution, more PRs will follow.
Highlights:
- py-config is no longer a web component: the old code relied on PyConfig.connectedCallback to do some logic, but then if no <py-config> tag was present, we had to introduce a dummy one with the sole goal of activating the callback. Now the logic is much more linear.
- the new pyconfig.ts only contains the code which is needed to parse the config; I also moved some relevant code from utils.ts because it didn't really belong to it
- the old PyConfig class did much more than dealing with the config: in particular, it contained the code to initialize the env and the runtime. Now this logic has been moved directly into main.ts, inside the new PyScriptApp class. I plan to refactor the initialization code in further PRs
- the current code relies too much on global state and global variables, they are everywhere. This PR is a first step to solve the problem by introducing a PyScriptApp class, which will hold all the mutable state of the page. Currently only config is stored there, but eventually I will migrate more state to it, until we will have only one global singleton, globalApp
- thanks to what I described above, I could kill the appConfig svelte store: one less store to kill :).
* TOML Error Cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove traceback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* any is a ts feature
* full eslint
* format and disable eslint for toml.js
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Madhur Tandon <madhurtandon23@gmail.com>
* Minimium viable py-config default schema
* Add packages and paths to default
* Add in default for plugins
* Remove tests for values no longer in the default config
* Loading and Script evaluate with Promise.all
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move to 'void script.evaluate();'
* chmod files to 644
* Add a test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add unit tests for pyloader and pytitle
* Add more unit tests for pyrepl, pybox and pyinputbox
* Add more tests for pyscript and pyconfig
* White space
* Fix d3 tests and improve more examples test
* Update matplotlib test
* Add numpy to dependencies
* Address Madhur comments
* Update test name
* add toml support for configs with fast-toml
* fix package-lock.json and pin pyodide to 0.21.2
* use browser version of fast-toml
* disable eslint and add credits
* fix jest issues
* use type attribute for py-config
* refactor py-config to use toml
* switch from toml to json and add unit tests
* fix test for py-config
* fix integration test
* use flat structure for JSON
* allow merging configs
* replace arrays instead of concatenating them
* remove extra keys from inline config of integration test
* simplify array replacement logic
* allow config from src to be partial as well
* add comments to unit tests
* add unit test for setting config from both inline and src
* handle parse errors + validate config supplied
* separate functions for src and inline
* suggested improvements
* show error message in red on parser error
* fix eslint
* use resolveJsonModule as true
* use default config defined as a variable without import
* remove disable eslint comment
* remove import for covfefe.json as well
* metadata injection
* add support for schema + extra keys
* use schema_version
* Execute pys-on* events when triggered, not at load
Mimicing the behavior of Javascripts 'onLoad' event, we should
not be executing the use code at page-load time, only when
the event is triggered.
* Update examples to new syntax
* Fix merge issue
* Await running event handler code
* Restore pys-on* events with original behavior, deprecation warning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* xfail toga example
* Add missing { (typo)
* Adjust callback chandling to make linter happy
* Change alpha to latest (#760)
* Don't create custom elements in main and fix various small issues on tests (#747)
* Create custom elements when the runtime finishes loading
* Remove xfails and fix repl integration test
* Fix commented ignore
* Address Antonio's comments
* Fix bad rebase
* Make ure to wait for repl to be in attached state before asserting content
* Move createCustomeElement up so it runs before we close the loader, xfail flaky d3 test
* Fix xfail
* [pre-commit.ci] pre-commit autoupdate (#762)
updates:
- [github.com/pre-commit/mirrors-eslint: v8.23.0 → v8.23.1](https://github.com/pre-commit/mirrors-eslint/compare/v8.23.0...v8.23.1)
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* change documentation to point to latest instead of frozen alpha (#764)
* Toga example is xpass
* Correct 'xpass' to 'xfail' mark
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Peter W <34256109+pww217@users.noreply.github.com>
Co-authored-by: Fábio Rosado <fabiorosado@outlook.com>
* Create custom elements when the runtime finishes loading
* Remove xfails and fix repl integration test
* Fix commented ignore
* Address Antonio's comments
* Fix bad rebase
* Make ure to wait for repl to be in attached state before asserting content
* Move createCustomeElement up so it runs before we close the loader, xfail flaky d3 test
* Fix xfail
