* add default limit 1000
* Add frontend changes and connect to backend
* Fix query hash because of default limit
* fix CircleCI test
* adjust for comment
* Support multiple queries in a single query box
* Implement statement splitting function and add tests for it
* Add a test for databricks-specific syntax
* Split statements before running query
* add flask-wtf
* add CSRF tokens to all static forms
* add CSRF tokens to all axios requests
* disable CSRF validation in unit tests
* support CSRF-protected requests in *most* cypress tests
* don't enfroce CSRF checks by default
* avoid CSRF enforcement in unit tests
* remove redundant spread
* some camel casing hiccups
* always yield the CSRF cookie, but avoid enforcing it if CSRF toggle is off
* Restyled by prettier (#5056)
Co-authored-by: Restyled.io <commits@restyled.io>
* set a CSRF header only if cookie is present
* enforce CSRF in CI
* install lodash directly for Cypress
* install request-cookies directly for Cypress. We should probably start loading package.json deps
* enable CSRF support when logout and login happen within the same spec
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <commits@restyled.io>
* on dashboard api calls - take the id from the beginning of the slug, unless there is no number in it - in that case, take the entire slug as id
* add dashboard id when showing links to dashboards
* change path to include new name when renaming dashboards
* move slug generation to backend
* redirect to new name after changing (this time with a proper promise)
* oh right, we already have a slug function
* add spec that makes sure that renamed dashboards are redirected to the
url which contains their new name
* use id-slug in all Cypress specs
* move dashboards from /dashboard/:slug to /dashboards/:id-:name_as_slug
* Update dashboard url as its name changes
* Update separator to be "/"
* Update missing dashboard urls
* Update api not to depend on int id
* Use '-' instead of '/' as separator and update Dashboard.get calls
* slug -> name_as_slug
* Keep slug urls on cypress
* Update route path
* Use legacy attr for GET
* Use getter for urlForDashboard
* Update dashboard url when loaded by slug
* Update Dashboard routes to use id instead of slug
* Update Dashboard handler tests
* Update Cypress tests
* Fix create new dashboard spec
* Use axios { params }
* Drop Ternary operator
* Send updated slug directly in 'slug' attr
* Update multiple urls Dashboard test name
* Update route names
Co-authored-by: Levko Kravets <levko.ne@gmail.com>
Co-authored-by: Gabriel Dutra <nesk.frz@gmail.com>
Co-authored-by: Levko Kravets <levko.ne@gmail.com>
* feat: provide ssl options for Cassandra data source
* remove Log and prints
* Refactor to create module methods and unit tests
* Switch to using Enumerator and temp file
* Fix temporary file lifecycle for cert
* Align with changes on master
* Fix non certificate but ssl enabled usecase
* move filtering of invalid schedules to the query
* simplify retrieved_at assignment and wrap in a try/except block to avoid one query blowing up the rest
* refactor refresh_queries to use simpler functions with a single responsibility and add try/except blocks to avoid one query blowing up the rest
* avoid blowing up when job locks point to expired Job objects. Enqueue them again instead
* there's no need to check for the existence of interval - all schedules have intervals
* disable faulty schedules
* reduce FP style in refresh_queries
* report refresh_queries errors to Sentry (if it is configured)
* avoid using exists+fetch and use exceptions instead
* Adds logic to sort column names returned by the query runner. If `sorted`
raises an Exception it returns the column names unaltered from the query
runner.
* Moves table name sorting from model code into schema handler.
* Moves token sorting into the model code.
* Replaces single-quotes with double-quotes for consistency.
* Applies black formatting to changes.
* Moves schema sort into separate method. Adds test.
* Fixes output schema variable name. Without this the sorted cache is never returned!
____ ____ ____ _____
/ __ \/ __ \/ __ \/ ___/
/ /_/ / /_/ / /_/ (__ )
\____/\____/ .___/____/
/_/
* Adds test case guaranteeing that the model actually _uses_ the schema sorter.
Related to a31f90178c
* enforce hard limits on non-responsive work horses by workers
* move differences from Worker to helper methods to help make the specialization clearer
* move HardLimitingWorker to redash/tasks
* move schedule.py to /tasks
* explain the motivation for HardLimitingWorker
* pleasing CodeClimate
* pleasing CodeClimate
* port query execution to RQ
* get rid of argsrepr
* avoid star imports
* allow queries to be cancelled in RQ
* return QueryExecutionErrors as job results
* fix TestTaskEnqueue and QueryExecutorTests
* remove Celery monitoring
* get rid of QueryTask and use RQ jobs directly (with a job serializer)
* Revert "remove Celery monitoring"
This reverts commit 37a74ea403.
* reduce occurences of the word 'task'
* use Worker, Queue and Job instead of spreading names that share behavior details
* remove locks for failed jobs as well
* did I not commit that colon? oh my
* push the redis connection to RQ's stack on every request to avoid verbose connection setting
* use a connection context for tests
* remove Celery monitoring
* 👋 Celery
* remove Celery from Cypress
* black it up
* some more black
* return all started/queued job ids (for future monitoring
* Restyled by prettier (#4522)
* remove celery.py
* remove some frontend residuals that reappeared after a merge
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
* enforce hard limits on non-responsive work horses by workers
* move differences from Worker to helper methods to help make the specialization clearer
* move HardLimitingWorker to redash/tasks
* move schedule.py to /tasks
* explain the motivation for HardLimitingWorker
* pleasing CodeClimate
* pleasing CodeClimate
* port query execution to RQ
* get rid of argsrepr
* avoid star imports
* allow queries to be cancelled in RQ
* return QueryExecutionErrors as job results
* fix TestTaskEnqueue and QueryExecutorTests
* remove Celery monitoring
* get rid of QueryTask and use RQ jobs directly (with a job serializer)
* Revert "remove Celery monitoring"
This reverts commit 37a74ea403.
* reduce occurences of the word 'task'
* use Worker, Queue and Job instead of spreading names that share behavior details
* remove locks for failed jobs as well
* did I not commit that colon? oh my
* push the redis connection to RQ's stack on every request to avoid verbose connection setting
* use a connection context for tests
* black it up
* run RQ on all queues when running in Cypress
* Allow executing query with either view_query or execute_query permissions.
* Render AuthHeader according to permissions.
* Don't return dashboards where you only have access to textbox widget.
Closes#4099.
* add some logging to scheduler
* schedule jobs only if they are not already scheduled
* jobs scheduled with an interval over 24 hours were not repeated
* schedule version_check using standard scheduling
* clean up old jobs that are not part of the definition anymore
* add some tests
* add one more test to verify that reschedules are not done when not neccesary
* no need to check for func existence - all jobs have a func to run
* Make core app compatible with Python 3
No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.
* Use Python 3.7 as base docker image
Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.
* Upgrade some requirements to newest versions
Some of the older versions were not compatible with Python 3.
* Migrate tests to Python 3
* Build frontend on Python 3
* Make the HMAC sign function compatible with Python 3
In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.
* Use assertCountEqual instead of assertItemsEqual
The latter is not available in Python 3.
See https://bugs.python.org/issue17866
* Remove redundant encoding header for Python 3 modules
* Remove redundant string encoding in CLI
* Rename list() functions in CLI
These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().
Only the Python function is renamed, from the perspective of the
CLI nothing changes.
* Replace usage of Exception.message in CLI
`message` is not available anymore, instead use the string
representation of the exception.
* Adapt test handlers to Python 3
* Fix test that relied on dict ordering
* Make sure test results are always uploaded (#4215)
* Support encoding memoryview to JSON
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156
* Fix test relying on object address ordering
* Decode bytes returned from Redis
* Stop using e.message for most exceptions
Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.
* Fix writing XLSX files in Python 3
The buffer for the file should be made of bytes and the actual
content written to it strings.
Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.
* Fix test by comparing strings to strings
* Fix another exception message unavailable in Python 3
* Fix export to CSV in Python 3
The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.
* (Python 3) Use Redis' decode_responses=True option (#4232)
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)
* Make sure Redis connection uses decoded_responses option
* Remove unused imports.
* Use Redis' decode_responses option
* Remove cases of explicit Redis decoding
* Rename helper function and make sure it doesn't apply twice.
* Don't add decode_responses to Celery Redis connection URL
* Fix displaying error while connecting to SQLite
The exception message is always a string in Python 3, so no
need to try to decode things.
* Fix another missing exception message
* Handle JSON encoding for datasources returning bytes
SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.
This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.
* Fix Python 3 compatibility with RQ
* Revert some changes 2to3 tends to do (#4261)
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.
- Also fixed some naming issues with one character variables in list comprehensions.
- Fix Flask warning.
* Upgrade dependencies
* Remove useless `iter` added by 2to3
* Fix get_next_path tests (#4280)
* Removed setting SERVER_NAME in tests setup to avoid a warning.
* Change get_next_path to not return empty string in case of a domain only value.
* Fix redirect tests:
Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.
* Remove explicit dependency for Werkzeug
* Switched pytz and certifi to unbinded versions.
* Switch to new library for getting country from IP
`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .
* Python 3 RQ modifications (#4281)
* show current worker job (alongside with minor cosmetic column tweaks)
* avoid loading entire job data for queued jobs
* track general RQ queues (default, periodic and schemas)
* get all active RQ queues
* call get_celery_queues in another place
* merge dicts the Python 3 way
* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
* Remove legacy Python flake8 tests
* add rq and an rq_worker service
* add rq_scheduler and an rq_scheduler service
* move beat schedule to periodic_jobs queue
* move version checks to RQ
* move query result cleanup to RQ
* use timedelta and DRY up a bit
* move custom tasks to RQ
* do actual schema refreshes in rq
* rename 'period_jobs' to 'periodic', as it obviously holds jobs
* move send_email to rq
* DRY up enqueues
* ditch and use a partially applied decorator
* move subscribe to rq
* move check_alerts_for_query to rq
* move record_event to rq
* make tests play nicely with rq
* 👋 beat
* rename rq_scheduler to plain scheduler, now that there's no Celery scheduler entrypoint
* add some color to rq-worker's output
* add logging context to rq jobs (while keeping execute_query context via get_task_logger for now)
* move schedule to its own module
* cancel previously scheduled periodic jobs. not sure this is a good idea.
* rename redash.scheduler to redash.schedule
* allow custom dynamic jobs to be added decleratively
* add basic monitoring to rq queues
* add worker monitoring
* pleasing the CodeClimate overlords
* adjust cypress docker-compose.yml to include rq changes
* DRY up Cypress docker-compose
* add rq dependencies to cypress docker-compose service
* an odd attempt at watching docker-compose logs when running with Cypress
* Revert "an odd attempt at watching docker-compose logs when running with Cypress"
This reverts commit 016bd1a93e.
* show docker-compose logs at Cypress shutdown
* Revert "DRY up Cypress docker-compose"
This reverts commit 43abac7084.
* minimal version for binding is 3.2
* remove unneccesary code reloads on cypress
* add a command which errors if any of the workers running inside the current machine haven't been active in the last minute
* SCHEMAS_REFRESH_QUEUE is no longer a required setting
* split tasks/queries.py to execution.py and maintenance.py
* fix tests after query execution split
* pleasing the CodeClimate overlords
* rename worker to celery_worker and rq_worker to worker
* use /rq_status instead of /jobs
* show started jobs' time ago according to UTC
* replace all spaces in column names
* fix query tests after execution split
* exit with an int
* general lint
* add an entrypoint for rq_healthcheck
* fix indentation
* delete all existing periodic jobs before scheduling them
* remove some unrequired requires
* move schedule example to redash.schedule
* add RQ integration to Sentry's setup
* pleasing the CodeClimate overlords
* remove replication settings from docker-compose - a proper way to scale using docker-compose would be the --scale CLI option, which will be described in the knowledge based
* revert to calling a function in dynamic settings to allow periodic jobs to be scheduled after app has been loaded
* don't need to depend on context when templating failure reports
* set the timeout_ttl to double the interval to avoid job results from expiring and having periodic jobs not reschedule
* whoops, bad merge
* describe custom jobs and don't actually schedule them
* fix merge
* don't need to depend on context when templating failure reports
* extract a render_template function with some docs
* CodeClimate has really outdone itself this time. Removed a whitespace character in order to fix 2 CodeClimate errors
* apparently whitespace doesn't count as a character
* Add interface to implement custom persistence for QueryResult data
Co-authored-by: Omer Lachish <omer@rauchy.net>
* Deserialize query results data in the model
* Change order of mixins.
* Make DBPersistence.data setter in sycn with getter + tests
* Fix loading of periodic tasks and clean up extension loading.
This does a few things:
- add tests for extension loading
- refactor the extension and periodic task loading
- better handle assertions raised by extensions (e.g. when an extension tries to override an already registered view)
- attach exception traceback to error log during loading for improved debugging
* Use site.addsitedir instead of calling pip.
* Use sys.path instead of site.addsitedir and also the setup.py egg_info command.
* avoid using 'abort' in parameterized query - raise an exception instead
* when facing invalid parameters or detached dropdown queries - continue to refresh the rest of the outdated queries
* test that dropdown queries detached from data source raise an exception when fetch values is attempted
* test that queries with invalid parameters arent refreshed
* test that queries with dropdown query parameters which are detached from the data source are skipped
* fix stale test double name
* newlines. newlines everywhere.
* pass org into dropdown_values
* pass in org in every ParameterizedQuery usage
* Update redash/tasks/queries.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* reduce refresh_queries log noise
* track failure count for queries that failed to apply parameters, and also notify the failures
* Update redash/tasks/queries.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* newlines. newlines everywhere.
* Allow multiple values for enum parameter
* Allow multi-select for Query dropdown parameters
* CR + make sure list values are allowed
* Add prefix, suffix and separator
* Rename multipleValues and cast options as strings
* Replicate serialization logic on frontend
* Add Quote Option Select
* Make sure it's enum or query before join
* Add a couple of tests
* Add help to quote option
* Add min-width and normalize empty array
* Improve behavior when changing parameter settings
- Set parameter value again to pass through checks
- Add setValue check for multi values
* Validate enum values on setValue + CodeClimate
* Ran wording suggestions
* Updates after Apply Changes
* Fix failing Cypress tests
* Make sure enumOptions exists before split
* Improve propTypes for QueyBasedParameterInput
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* CR
* Cypress: Test for multi-select Enum
* Fix multi-selection Cypress spec
* Update Refresh Schedule
* initial work on e-mail report for failed queries
* send failure report only for scheduled queries and not for adhoc queries
* add setting to determine if to send failure reports
* add setting to determine interval of aggregated e-mail report
* html templating of scheduled query failure report
* break line
* support timeouts for failure reports
* aggregate errors within message and warn if approaching threshold
* handle errors in QueryExecutor.run instead of on_failure
* move failure report to its own module
* indicate that failure count is since last report
* copy changes
* format with <code>
* styling, copy and add a link to the query instead of the query text
* separate reports with <hr>
* switch to UTC
* move <h2> to actual e-mail subject
* add explicit message for SoftTimeLimitExceeded
* fix test to use soft time limits
* default query failure threshold to 100
* use base_url from utils
* newlines. newlines everywhere.
* remove redundant import
* apply new design for failure report
* use jinja to format the failure report
* don't show comment block if no comment is provided
* don't send emails if, for some reason, there are no available errors
* subtract 1 from failure count, because the first one is represented by 'Last failed'
* don't show '+X more failures' if there's only one
* extract subject to variable
* format as text, while we're at it
* allow scrolling for long exception messages
* test that e-mails are scheduled only when beneath limit
* test for indicating when approaching report limits + refactor
* test that failures are aggregated
* test that report counts per query and reason
* test that the latest failure occurence is reported
* force sending reports for testing purposes
* Update redash/templates/emails/failures.html
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* Update redash/templates/emails/failures.html
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* Update redash/tasks/failure_report.py
* add org setting for email reports
* remove logo from failure report email
* correctly use the organization setting for sending failure reports
* use user id as key for failure reports data structure
* Update redash/tasks/failure_report.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* build comments while creating context for e-mail templates
* figure out the base url when creating the e-mail
* no need to expire pending failure report keys as they are deleted anyway when sent
* a couple of CodeClimate changes
* refactor key creationg to a single location
* refactor tests to send e-mail from a single function
* use beat to schedule a periodic send_aggregated_errors task instead of using countdown per email
* remove pending key as it is no longer required when a periodic task picks up the reports to send
* a really important blank line. REALLY important.
* Revert "a really important blank line. REALLY important."
This reverts commit c7d8ed8972.
* a really important blank line. REALLY important. It is the best blank line.
* don't send failure emails to disabled users
* add a deprecated flag to query runners and show only non-deprecated query runners when adding a new data source
* add a deprecated flag to alert destinations and show only non-deprecated alert destinations when adding a new alert destination
* add a deprecated() decorator for a more succint way to deprecate
* deprecate URL query runner and HipChat alert destination
* use class properties instead of class methods for deprecation
* I <3 newlines
* return message explaining unsafe sharing
* use backend-generated message for public dashboards
* use backend-generated message for embeds
* Update redash/handlers/query_results.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* refactor simple (non-interpolated) query result handler error messages to a single location
* use error_messages to test out unsafe error messages (along with a couple of others)
* Update redash/handlers/query_results.py
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* Update redash/handlers/query_results.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* avoid catching errors on text widgets' load(), as they don't have a visualization and therefore do not return any promise
* throw error when failing to load widgets on public dashboards - in case something needs to be done with it at a later time, and it's the right thing to do anyway
* use Promise.resolve instead of checking for undefined
* call serialize_query_result instead of directly calling to_dict
* filter unneeded query result fields for unauthenticated users
* test for serialization filtering
* lint
* use project instead of list comprehension
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* use the textless endpoint (/api/queries/:id/results) for pristine
queriest
* Revert "use the textless endpoint (/api/queries/:id/results) for pristine"
This reverts commit cd2cee7738.
* go to textless /api/queries/:id/results by default
* change `run_query`'s signature to accept a ParameterizedQuery instead of
constructing it inside
* raise HTTP 400 when receiving invalid parameter values. Fixes#3394
* enqueue jobs for ApiUsers
* rename `id` to `user_id`
* support executing queries using Query api_keys by instantiating an ApiUser that would be able to execute the specific query
* show deprecation messages for ALLOW_PARAMETERS_IN_EMBEDS. Also, move
other message (email not verified) to use the same mechanism
* add link to forum message regarding embed deprecation
* change API to /api/queries/:id/dropdowns/:dropdown_id
* split to 2 different dropdown endpoints and implement the second
* add test cases for /api/queries/:id/dropdowns/:id
* use new /dropdowns endpoint in frontend
* first e2e test for sharing embeds
* Pleasing the CodeClimate overlords
* All glory to CodeClimate
* remove residues from bad rebase
* add query id and data source id to serialized public dashboards
* add global parameters directive to public dashboards page
* allow access to a query by the api_key of the dashboard which includes
it
* rename `object` to `obj`
* simplify permission tests once `has_access` accepts groups
* support global parameters for public dashboards
* change has_access and require_access signatures to work with the objects that require access, instead of their groups
* rename `object` to `obj`
* simplify permission tests once `has_access` accepts groups
* no need to log `is_api_key`
* send parameters to public dashboard page
* allow access to a query by the api_key of the dashboard which includes it
* disable sharing if dashboard is associated with unsafe queries
* remove cypress test added in the wrong place due to a faulty rebase
* add support for clicking buttons in cy.clickThrough
* Cypress test which verifies that dashboards with safe queries can be shared
* Cypress test which verifies that dashboards with unsafe queries can't be shared
* remove duplicate tests
* use this.enabled and negate when needed
* remove stale comment
* add another Cypress test to verify that unauthenticated users have access to public dashboards with parameters
* obviously, I commit 'only' the first time I use it
* search for query access by query id and not api_key
* no need to fetch latest query data as it is loaded by frontend from the textless endpoint
* test that queries associated with dashboards are accessible when supplying the dashboard api_key
* propagate `isDirty` down to `QueryBasedParameterInput`
* go to /api/:id/dropdown while editing a query, since dropdown queries might still not be associated with the parent. see #3711
* show helpful error message if dropdown values cannot be fetched
* use backticks instead of line concatenation
* remove requirement to have direct access to dropdown query in order validate it. parent query association checks are sufficient
* remove isDirty-based implementation and allow dropdown queries through nested ACL even if they aren't associated yet (given that the user has _direct_ access to the dropdown query)
* fix tests to cover all cases for /api/queries/:id/dropdowns/:id
* fix indentation
* require access to the query, not the data source
* resolve dashboard user by query id
* apply new copy to Cypress tests
* if only something would have prevented me from commiting an 'only' call 🤔
* very important handling of whitespace
* respond to parameter's Apply button
* text widgets are safe for sharing
* remove redundant event
* add a safety check that object has dashboard_api_keys before calling it
* supply a parameter value for text parameters to have it show up
* add parameter values for date and datetime
* use the current year and month to avoid pagination
* use Cypress.moment() instead of preinstalled moment()
* explicitly create parameters
* refresh query data if a querystring parameter is provided
* avoid sending a data_source_id - it's only relevant to unsaved queries, since a saved query's data_source is available in the backend
* remove empty query text workaround
* provide default value to parameter
* add a few more dashboard sharing specs
* lint
* wait for DynamicTable to appear to reveal that actual results are displaying
* override error message for unsafely shared widgets
