I've experience this on my Redash in production. I'm not sure what can cause the value to exist, but be None. I guess it depends on the SQL query.
I followed the same idea of returning a self.UNKNOWN_STATE for cases that we can't know what's happening.
* convert unexpected validation errors to InvalidParameterError
* get rid of local except blocks - any failure to validate is caught in _valid()
* avoid bare exception
* don't assign exception if we're not going to use it
* preferentially raise querydetachedfromdatasource error if it is present
---------
Co-authored-by: Omer Lachish <omer@rauchy.net>
Co-authored-by: Guido Petri <18634426+guidopetri@users.noreply.github.com>
* Use multi_byte_search_enabled option for My Queries search and Favorite List search
* make format
* add search by user tests
---------
Co-authored-by: Justin Clift <justin@postgresql.org>
Co-authored-by: Guido Petri <18634426+guidopetri@users.noreply.github.com>
- Escape all variables by default since Mustache only has a syntax for raw values
- Generate a generic HTML result table to allow an alert template to display any query
- Optionally allow alert template to be defined to be set using REDASH_ALERTS_DEFAULT_MAIL_BODY_TEMPLATE
- Formatting updated by black
* Fix: pagination is broken on the dashboard list page (#5516)
* Add test that reproduces issue #5466
* Fix: Duplicate dashboard rows were returned by Dashboard.all() (#5466)
* Update changelog for V10
* Update changelog for #5516
* Add My Dashboards filter option to the Dashboards list. Added API endpoint to get the list of a user's dashboards, similar to the My Queries feature.
* Update empty dashboard list state to show an invite to create a new dashboard, like My Queries
* Update to Levko's suggested approach. Clean up some of the formatting for consistency. Put the 'My Queries/Dashboards' item before the Favorites since that organization seems cleaner to me.
* Address Levko's comments
* Adds logic to sort column names returned by the query runner. If `sorted`
raises an Exception it returns the column names unaltered from the query
runner.
* Moves table name sorting from model code into schema handler.
* Moves token sorting into the model code.
* Replaces single-quotes with double-quotes for consistency.
* Applies black formatting to changes.
* Moves schema sort into separate method. Adds test.
* Fixes output schema variable name. Without this the sorted cache is never returned!
____ ____ ____ _____
/ __ \/ __ \/ __ \/ ___/
/ /_/ / /_/ / /_/ (__ )
\____/\____/ .___/____/
/_/
* Adds test case guaranteeing that the model actually _uses_ the schema sorter.
Related to a31f90178c
* Make core app compatible with Python 3
No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.
* Use Python 3.7 as base docker image
Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.
* Upgrade some requirements to newest versions
Some of the older versions were not compatible with Python 3.
* Migrate tests to Python 3
* Build frontend on Python 3
* Make the HMAC sign function compatible with Python 3
In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.
* Use assertCountEqual instead of assertItemsEqual
The latter is not available in Python 3.
See https://bugs.python.org/issue17866
* Remove redundant encoding header for Python 3 modules
* Remove redundant string encoding in CLI
* Rename list() functions in CLI
These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().
Only the Python function is renamed, from the perspective of the
CLI nothing changes.
* Replace usage of Exception.message in CLI
`message` is not available anymore, instead use the string
representation of the exception.
* Adapt test handlers to Python 3
* Fix test that relied on dict ordering
* Make sure test results are always uploaded (#4215)
* Support encoding memoryview to JSON
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156
* Fix test relying on object address ordering
* Decode bytes returned from Redis
* Stop using e.message for most exceptions
Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.
* Fix writing XLSX files in Python 3
The buffer for the file should be made of bytes and the actual
content written to it strings.
Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.
* Fix test by comparing strings to strings
* Fix another exception message unavailable in Python 3
* Fix export to CSV in Python 3
The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.
* (Python 3) Use Redis' decode_responses=True option (#4232)
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)
* Make sure Redis connection uses decoded_responses option
* Remove unused imports.
* Use Redis' decode_responses option
* Remove cases of explicit Redis decoding
* Rename helper function and make sure it doesn't apply twice.
* Don't add decode_responses to Celery Redis connection URL
* Fix displaying error while connecting to SQLite
The exception message is always a string in Python 3, so no
need to try to decode things.
* Fix another missing exception message
* Handle JSON encoding for datasources returning bytes
SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.
This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.
* Fix Python 3 compatibility with RQ
* Revert some changes 2to3 tends to do (#4261)
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.
- Also fixed some naming issues with one character variables in list comprehensions.
- Fix Flask warning.
* Upgrade dependencies
* Remove useless `iter` added by 2to3
* Fix get_next_path tests (#4280)
* Removed setting SERVER_NAME in tests setup to avoid a warning.
* Change get_next_path to not return empty string in case of a domain only value.
* Fix redirect tests:
Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.
* Remove explicit dependency for Werkzeug
* Switched pytz and certifi to unbinded versions.
* Switch to new library for getting country from IP
`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .
* Python 3 RQ modifications (#4281)
* show current worker job (alongside with minor cosmetic column tweaks)
* avoid loading entire job data for queued jobs
* track general RQ queues (default, periodic and schemas)
* get all active RQ queues
* call get_celery_queues in another place
* merge dicts the Python 3 way
* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
* Remove legacy Python flake8 tests
* Add interface to implement custom persistence for QueryResult data
Co-authored-by: Omer Lachish <omer@rauchy.net>
* Deserialize query results data in the model
* Change order of mixins.
* Make DBPersistence.data setter in sycn with getter + tests
* avoid using 'abort' in parameterized query - raise an exception instead
* when facing invalid parameters or detached dropdown queries - continue to refresh the rest of the outdated queries
* test that dropdown queries detached from data source raise an exception when fetch values is attempted
* test that queries with invalid parameters arent refreshed
* test that queries with dropdown query parameters which are detached from the data source are skipped
* fix stale test double name
* newlines. newlines everywhere.
* pass org into dropdown_values
* pass in org in every ParameterizedQuery usage
* Update redash/tasks/queries.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* reduce refresh_queries log noise
* track failure count for queries that failed to apply parameters, and also notify the failures
* Update redash/tasks/queries.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* newlines. newlines everywhere.
* Allow multiple values for enum parameter
* Allow multi-select for Query dropdown parameters
* CR + make sure list values are allowed
* Add prefix, suffix and separator
* Rename multipleValues and cast options as strings
* Replicate serialization logic on frontend
* Add Quote Option Select
* Make sure it's enum or query before join
* Add a couple of tests
* Add help to quote option
* Add min-width and normalize empty array
* Improve behavior when changing parameter settings
- Set parameter value again to pass through checks
- Add setValue check for multi values
* Validate enum values on setValue + CodeClimate
* Ran wording suggestions
* Updates after Apply Changes
* Fix failing Cypress tests
* Make sure enumOptions exists before split
* Improve propTypes for QueyBasedParameterInput
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* CR
* Cypress: Test for multi-select Enum
* Fix multi-selection Cypress spec
* Update Refresh Schedule
* Support multi-byte search for query names and descriptions
* add multi_byte_support_enabled option to organization settings
* add `ilike %...%` to query search conditions when the option is enabled
* Improve description for multi_byte_search_enabled option
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* Remove tsvector from search when multi_byte_search_enabled
* Add a multi-byte search test case
* Celery task to clear schedule was added
* fix formating
* empty_schedules task was put in separate task
* worker interval changed, new tests added
* past artifact deleted
* test queries moved to right class, lambda was used to filter data
* unnecessary changes eliminated
* more unnecessary files deleted
* line shortened
* Line shortened more
* codeclimate changes
* Unused test deleted, logs added
* Add rate limits for user resources.
* Disable rate limiting in tests (except for tests that need it).
* Update strings to unicode to avoid SQLA warnings
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* use the textless endpoint (/api/queries/:id/results) for pristine
queriest
* Revert "use the textless endpoint (/api/queries/:id/results) for pristine"
This reverts commit cd2cee7738.
* go to textless /api/queries/:id/results by default
* change `run_query`'s signature to accept a ParameterizedQuery instead of
constructing it inside
* raise HTTP 400 when receiving invalid parameter values. Fixes#3394
* support querystring params
* extract coercing of numbers to function, along with a friendlier
implementation
* wire embeds to textless endpoint
* allow users with view_only permissions to execute queries on the
textless endpoint, as it only allows safe queries to run
* enqueue jobs for ApiUsers
* add parameters component for embeds
* include existing parameters in embed code
* fetch correct values for json requests
* remove previous embed parameter code
* rename `id` to `user_id`
* support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query
* bring back ALLOW_PARAMETERS_IN_EMBEDS (with link on deprecation coming up)
* show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move
other message (email not verified) to use the same mechanism
* add link to forum message on setting deprecation
* rephrase deprecation message
* add link to forum message regarding embed deprecation
* change API to /api/queries/:id/dropdowns/:dropdown_id
* split to 2 different dropdown endpoints and implement the second
* add test cases for /api/queries/:id/dropdowns/:id
* use new /dropdowns endpoint in frontend
* first e2e test for sharing embeds
* Pleasing the CodeClimate overlords
* All glory to CodeClimate
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* split has_access between normal users and ApiKey users
* remove residues from bad rebase
* allow access to safe queries via api keys
* rename `object` to `obj`
* support both objects and group dicts in `has_access` and `require_access`
* simplify permission tests once `has_access` accepts groups
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* rename `object` to `obj`
* support both objects and group dicts in `has_access` and `require_access`
* simplify permission tests once `has_access` accepts groups
* fix bad rebase
* send embed parameters through POST data
* no need to log `is_api_key`
* move query fetching by api_key to within the Query model
* fetch user by adding a get_by_id function on the User model
* pass parameters as POST data (fixes test failure introduced by switching
from query string parameters to POST data)
* test the right thing - queries with safe parameters should be embeddable
* introduce cy.clickThrough
* add another Cypress test to make sure unsafe queries cannot be embedded
* serialize Parameters into query string
* set is_api_key as the last parameter to (hopefully) avoid
backward-dependency problems
* Update redash/models/parameterized_query.py
Co-Authored-By: rauchy <omer@rauchy.net>
* attempt to fix empty percy snapshots
* snap percies after DOM is fully loaded
* change API to /api/queries/:id/dropdowns/:dropdown_id
* extract property
* split to 2 different dropdown endpoints and implement the second
* make access control optional for dropdowns (assuming it is verified at a
different level)
* add test cases for /api/queries/:id/dropdowns/:id
* use new /dropdowns endpoint in frontend
* require access to dropdown queries when creating or updating parent
queries
* rename Query resource dropdown endpoints
* check access to dropdown query associations in one fugly query
* move ParameterizedQuery to models folder
* add dropdown association tests to query creation
* move group by query ids query into models.Query
* use bound parameters for groups query
* format groups query
* use new associatedDropdowns endpoint in dashboards
* pass down parameter and let it return dropdown options. Go Levko!
* change API to /api/queries/:id/dropdowns/:dropdown_id
* split to 2 different dropdown endpoints and implement the second
* use new /dropdowns endpoint in frontend
* pass down parameter and let it return dropdown options. Go Levko!
* fix bad rebase
* add comment to clarify the purpose of checking the queryId
* Process extra column metadata for a few sql-based data sources.
* Add Table and Column metadata tables.
* Periodically update table and column schema tables in a celery task.
* Fetching schema returns data from table and column metadata tables.
* Add tests for backend changes.
* Front-end shows extra table metadata and uses new schema response.
* Delete datasource schema data when deleting a data source.
* Process and store data source schema when a data source is first created or after a migration.
* Tables should have a unique name per datasource.
* Addressing review comments.
* Update migration file for mixins.
* Appease PEP8
* Upgrade migration file for rebase.
* Cascade delete.
* Adding org_id
* Remove redundant column and table prefixes.
* Non-existing tables and columns should be filtered out on the server side not client side.
* Fetching table samples should be optional and should happen in a separate task per table.
* Allow users to force a schema refresh.
* Use updated_at to help prune old schema metadata periodically.
* Using settings.SCHEMAS_REFRESH_QUEUE
* Use null as the default scheduled value.
* Don't serialize None to json, so we can use SQL is not null predicate.
* Fix warning about unicode in tests
* Handling empty query.schedule in UI (#3283)
* Add migration to convert empty schedules to null and drop the not null contraint.
* add last_active_at to users page
* Use our JSON encoder as the SQLAlchemy JSON serializer.
* Fixed some inconsistencies in the user query class methods.
* Minor cosmetic fixes.
* Add some make tasks for easier development.
* Add user detail sync system based on Redis backend.
There is a periodic Celery task that updates a new “details” JSONB column in the “user” table with the data from Redis.
Currently this is only used for tracking the date of last activity of a user but can be extended with other user information later.
Updates a few dependencies.
* Normalize a few Flask extension API names.
* Reduce implementation complexity of JSONEncoder.
* Use request_started signal to make sure we have a request context.
Otherwise loading the user based on the request won’t work.
* Fix test that checks if disabled users can login.
This correctly uses a URL path that includes the current organization and checks for the error message.
The previous test seems to have been a red herring.
* Minor cosmetic fixes.
* Remove needs_sync in favor of just deleting things.
* Misc review fixes.
* Ignore line length.
* Split redash.models import several modules.
* Move walrus UTC DateTimeField into redash.models.types.
* Restore distinctly loading dashboards.
* Simplify default values for user details.
* Define __repr__ methods generically.
* Consistently have underscore methods at the top of model methods.
* Fix tests.
* Split redash.models import several modules.
* Update to latest walrus and redis-py.
* Update kombu to 4.2.2 for redis-py 3.x compatibility.
* Remove redis-cli container after running Make task.
* Move buffer condition after datetime/time conditions.
* Update walrus to 0.7.1.
* Refactor some query APIs.
This uses the flask-sqlalchemy helpers consistently and makes more use of mixins.
* Post rebase fixes.
* Use correct kombu version
* Fix migration down revision
* Fix tag counts for dashboards to be distinct.
This also makes use of the Dashboard.all base query.
Fix#3108.
* Use Query.all_queries as the base query for Query.all_tags.
* Add test case for Dashboard.all_tags.
* avoid Query's updated_at from changing when it is linked to new query results
* move comment to previous line
* move QueryResult tests to their own module
* add test which verifies that updated_at is not changed on query data
updates
* tests were false positives - they compared HH:MM:SS, but that never
changed because the original time was 1 week behind.
* remove redundant constructor
* remove hack and use a proper event to prevent updated_at from changing
* use self.assertEqual instead of assert
