* add rq and an rq_worker service
* add rq_scheduler and an rq_scheduler service
* move beat schedule to periodic_jobs queue
* move version checks to RQ
* move query result cleanup to RQ
* use timedelta and DRY up a bit
* move custom tasks to RQ
* do actual schema refreshes in rq
* rename 'period_jobs' to 'periodic', as it obviously holds jobs
* move send_email to rq
* DRY up enqueues
* ditch and use a partially applied decorator
* move subscribe to rq
* move check_alerts_for_query to rq
* move record_event to rq
* make tests play nicely with rq
* 👋 beat
* rename rq_scheduler to plain scheduler, now that there's no Celery scheduler entrypoint
* add some color to rq-worker's output
* add logging context to rq jobs (while keeping execute_query context via get_task_logger for now)
* move schedule to its own module
* cancel previously scheduled periodic jobs. not sure this is a good idea.
* rename redash.scheduler to redash.schedule
* allow custom dynamic jobs to be added decleratively
* add basic monitoring to rq queues
* add worker monitoring
* pleasing the CodeClimate overlords
* adjust cypress docker-compose.yml to include rq changes
* DRY up Cypress docker-compose
* add rq dependencies to cypress docker-compose service
* an odd attempt at watching docker-compose logs when running with Cypress
* Revert "an odd attempt at watching docker-compose logs when running with Cypress"
This reverts commit 016bd1a93e.
* show docker-compose logs at Cypress shutdown
* Revert "DRY up Cypress docker-compose"
This reverts commit 43abac7084.
* minimal version for binding is 3.2
* remove unneccesary code reloads on cypress
* add a command which errors if any of the workers running inside the current machine haven't been active in the last minute
* SCHEMAS_REFRESH_QUEUE is no longer a required setting
* split tasks/queries.py to execution.py and maintenance.py
* fix tests after query execution split
* pleasing the CodeClimate overlords
* rename worker to celery_worker and rq_worker to worker
* use /rq_status instead of /jobs
* show started jobs' time ago according to UTC
* replace all spaces in column names
* fix query tests after execution split
* exit with an int
* general lint
* add an entrypoint for rq_healthcheck
* fix indentation
* delete all existing periodic jobs before scheduling them
* remove some unrequired requires
* move schedule example to redash.schedule
* add RQ integration to Sentry's setup
* pleasing the CodeClimate overlords
* remove replication settings from docker-compose - a proper way to scale using docker-compose would be the --scale CLI option, which will be described in the knowledge based
* revert to calling a function in dynamic settings to allow periodic jobs to be scheduled after app has been loaded
* don't need to depend on context when templating failure reports
* set the timeout_ttl to double the interval to avoid job results from expiring and having periodic jobs not reschedule
* whoops, bad merge
* describe custom jobs and don't actually schedule them
* fix merge
* don't need to depend on context when templating failure reports
* extract a render_template function with some docs
* CodeClimate has really outdone itself this time. Removed a whitespace character in order to fix 2 CodeClimate errors
* apparently whitespace doesn't count as a character
* initial work on e-mail report for failed queries
* send failure report only for scheduled queries and not for adhoc queries
* add setting to determine if to send failure reports
* add setting to determine interval of aggregated e-mail report
* html templating of scheduled query failure report
* break line
* support timeouts for failure reports
* aggregate errors within message and warn if approaching threshold
* handle errors in QueryExecutor.run instead of on_failure
* move failure report to its own module
* indicate that failure count is since last report
* copy changes
* format with <code>
* styling, copy and add a link to the query instead of the query text
* separate reports with <hr>
* switch to UTC
* move <h2> to actual e-mail subject
* add explicit message for SoftTimeLimitExceeded
* fix test to use soft time limits
* default query failure threshold to 100
* use base_url from utils
* newlines. newlines everywhere.
* remove redundant import
* apply new design for failure report
* use jinja to format the failure report
* don't show comment block if no comment is provided
* don't send emails if, for some reason, there are no available errors
* subtract 1 from failure count, because the first one is represented by 'Last failed'
* don't show '+X more failures' if there's only one
* extract subject to variable
* format as text, while we're at it
* allow scrolling for long exception messages
* test that e-mails are scheduled only when beneath limit
* test for indicating when approaching report limits + refactor
* test that failures are aggregated
* test that report counts per query and reason
* test that the latest failure occurence is reported
* force sending reports for testing purposes
* Update redash/templates/emails/failures.html
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* Update redash/templates/emails/failures.html
Co-Authored-By: Ran Byron <ranbena@gmail.com>
* Update redash/tasks/failure_report.py
* add org setting for email reports
* remove logo from failure report email
* correctly use the organization setting for sending failure reports
* use user id as key for failure reports data structure
* Update redash/tasks/failure_report.py
Co-Authored-By: Arik Fraimovich <arik@arikfr.com>
* build comments while creating context for e-mail templates
* figure out the base url when creating the e-mail
* no need to expire pending failure report keys as they are deleted anyway when sent
* a couple of CodeClimate changes
* refactor key creationg to a single location
* refactor tests to send e-mail from a single function
* use beat to schedule a periodic send_aggregated_errors task instead of using countdown per email
* remove pending key as it is no longer required when a periodic task picks up the reports to send
* a really important blank line. REALLY important.
* Revert "a really important blank line. REALLY important."
This reverts commit c7d8ed8972.
* a really important blank line. REALLY important. It is the best blank line.
* don't send failure emails to disabled users
