* Make core app compatible with Python 3
No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.
* Use Python 3.7 as base docker image
Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.
* Upgrade some requirements to newest versions
Some of the older versions were not compatible with Python 3.
* Migrate tests to Python 3
* Build frontend on Python 3
* Make the HMAC sign function compatible with Python 3
In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.
* Use assertCountEqual instead of assertItemsEqual
The latter is not available in Python 3.
See https://bugs.python.org/issue17866
* Remove redundant encoding header for Python 3 modules
* Remove redundant string encoding in CLI
* Rename list() functions in CLI
These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().
Only the Python function is renamed, from the perspective of the
CLI nothing changes.
* Replace usage of Exception.message in CLI
`message` is not available anymore, instead use the string
representation of the exception.
* Adapt test handlers to Python 3
* Fix test that relied on dict ordering
* Make sure test results are always uploaded (#4215)
* Support encoding memoryview to JSON
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156
* Fix test relying on object address ordering
* Decode bytes returned from Redis
* Stop using e.message for most exceptions
Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.
* Fix writing XLSX files in Python 3
The buffer for the file should be made of bytes and the actual
content written to it strings.
Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.
* Fix test by comparing strings to strings
* Fix another exception message unavailable in Python 3
* Fix export to CSV in Python 3
The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.
* (Python 3) Use Redis' decode_responses=True option (#4232)
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)
* Make sure Redis connection uses decoded_responses option
* Remove unused imports.
* Use Redis' decode_responses option
* Remove cases of explicit Redis decoding
* Rename helper function and make sure it doesn't apply twice.
* Don't add decode_responses to Celery Redis connection URL
* Fix displaying error while connecting to SQLite
The exception message is always a string in Python 3, so no
need to try to decode things.
* Fix another missing exception message
* Handle JSON encoding for datasources returning bytes
SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.
This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.
* Fix Python 3 compatibility with RQ
* Revert some changes 2to3 tends to do (#4261)
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.
- Also fixed some naming issues with one character variables in list comprehensions.
- Fix Flask warning.
* Upgrade dependencies
* Remove useless `iter` added by 2to3
* Fix get_next_path tests (#4280)
* Removed setting SERVER_NAME in tests setup to avoid a warning.
* Change get_next_path to not return empty string in case of a domain only value.
* Fix redirect tests:
Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.
* Remove explicit dependency for Werkzeug
* Switched pytz and certifi to unbinded versions.
* Switch to new library for getting country from IP
`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .
* Python 3 RQ modifications (#4281)
* show current worker job (alongside with minor cosmetic column tweaks)
* avoid loading entire job data for queued jobs
* track general RQ queues (default, periodic and schemas)
* get all active RQ queues
* call get_celery_queues in another place
* merge dicts the Python 3 way
* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
* Remove legacy Python flake8 tests
* check that e-mail server is configured before marking the email address
as not verified and sending out a verification e-mail
* use helper method in `invite_user`
* move email_server_configured helper to settings
* add test to verify that email addresses arent marked as unverified if
there's no e-mail server to verify them
* simplify a couple of tests with patch
* combine conditions into single variable
* Booleans, gotta love 'em
* Create React version for the EmailSettingsWarning
* Migrate the Create User Page
* Migrate UserProfile to React
* Add /users/me to the routes (Percy ftw)
* Fix UserShow test spec
* Remove Error Messages component
* Show invitation link if email server not setup (#3519)
* return invite link to client if e-mail server is not set up
* add a couple of tests to make sure invite links are only returned when neccessary
* show invite link when e-mail is not configured
* remove "an e-mail has been sent" when there's no e-mail configured
* return invite_url in re-invites as well. Also refactor to reuse the code.
* Use CreateUserDialog instead of Page
* Render invite link on Resend Invitation click
* Add email validation to DynamicForm
* Fix EmailWarning position + update user list with user creation success
* Fix console error on UserProfile
* Redirect from /users/new + rename createUser -> showCreateUserDialog
* Use alert instead of toastr for user creation errors
* Remove logic from CreateUserDialog
* CR
* Use Promise.reject instead of throw to avoid console error
* default `is_invitation_pending` to false and actively set it to true
when inviting users, so that existing users won't show "Invitation
Pending"
* fix tests that broke due to default is_invitation_pending value
* update Flask-OAuthLib
* determine invitation_pending according to empty password. This commit will be reverted, I'm just deferring the implementation
* show '(Invitation Pending)' to users who haven't accepted their invitation yet
* allow resending invitations
* allow deletion of pending users from user list
* set invitation as not pending when following invite link
* prevent deleting activated users
* test that users who follow invitation links are set as non-pending invitations
* prevent re-using invitations
* invitees who use SSO will now also be marked as "non-pending"
* lint
* add last_active_at to users page
* Use our JSON encoder as the SQLAlchemy JSON serializer.
* Fixed some inconsistencies in the user query class methods.
* Minor cosmetic fixes.
* Add some make tasks for easier development.
* Add user detail sync system based on Redis backend.
There is a periodic Celery task that updates a new “details” JSONB column in the “user” table with the data from Redis.
Currently this is only used for tracking the date of last activity of a user but can be extended with other user information later.
Updates a few dependencies.
* Normalize a few Flask extension API names.
* Reduce implementation complexity of JSONEncoder.
* Use request_started signal to make sure we have a request context.
Otherwise loading the user based on the request won’t work.
* Fix test that checks if disabled users can login.
This correctly uses a URL path that includes the current organization and checks for the error message.
The previous test seems to have been a red herring.
* Minor cosmetic fixes.
* Remove needs_sync in favor of just deleting things.
* Misc review fixes.
* Ignore line length.
* Split redash.models import several modules.
* Move walrus UTC DateTimeField into redash.models.types.
* Restore distinctly loading dashboards.
* Simplify default values for user details.
* Define __repr__ methods generically.
* Consistently have underscore methods at the top of model methods.
* Fix tests.
* Split redash.models import several modules.
* Update to latest walrus and redis-py.
* Update kombu to 4.2.2 for redis-py 3.x compatibility.
* Remove redis-cli container after running Make task.
* Move buffer condition after datetime/time conditions.
* Update walrus to 0.7.1.
* Refactor some query APIs.
This uses the flask-sqlalchemy helpers consistently and makes more use of mixins.
* Post rebase fixes.
* Use correct kombu version
* Fix migration down revision
* display correct error message when attempting to disable yourself
* 403 (Forbidden) feels like a better status code than 400 (Bad Request)
* fix broken test
* remove redundant error title
While RFC 5321 alows for the username/mailbox art of the address to be
case-sensitive, it is hightly discouraged.
The local-part of a mailbox MUST BE treated as case
sensitive. Therefore, SMTP implementations MUST take care to
preserve the case of mailbox local-parts. In particular, for
some hosts, the user "smith" is different from the user
"Smith". However, exploiting the case sensitivity of mailbox
local-parts impedes interoperability and is discouraged.
Mailbox domains follow normal DNS rules and are hence not
case sensitive.
-- https://tools.ietf.org/rfc/rfc5321.txt
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
This is one huge change for the permissions system and related:
* (Backward incompatible:) Remove the table based permissions in favour of the new model.
* Manage permission to view or query datasources based on groups.
* Add the concept of Organization. It's irrelevant for most deployments, but allows for
multi-tenant support in re:dash.
* Replace ActivityLog with Event based rows (old data in activity_log table is retained).
* Enforce permissions on the server-side. There were some permissions that were only enforced
on the client side. This is no more. All permissions are enforced by the server.
* Added new permission: 'super-admin' to access the status and Flask-Admin interface.
* Make sure that html is never cached by the browser - this is to make sure that the browser
will always ask for the new Javascript/CSS resources (if such are available).
