Also adds the POSTGRES_HOST_AUTH_METHOD environment variable to the CI docker compose files. Modern PostgreSQL docker images need it, at least as they're used by our CI.
* Update ngines definition to allow for newer versions of Node.
With Node version 19 I stumbled into some issues so for now bumped it to v16, until we get to updating the libraries we use.
* docker-compose.yml updates:
1. Switch to newer maildev docker image.
2. Update local port to 5001 as 5000 seems to be used by a system process now.
* Update pymssql and pyarrow. Also commented out ibm-db until we have a way to not install it only on ARM.
* Microsoft Teams Webhook alert destination
* Text formatting and new image for Microsoft Teams Webhook
* Comment on how to build frontend
* Add title to clarify webhook URL
* Make the message into a configurable template.
* add flask-wtf
* add CSRF tokens to all static forms
* add CSRF tokens to all axios requests
* disable CSRF validation in unit tests
* support CSRF-protected requests in *most* cypress tests
* don't enfroce CSRF checks by default
* avoid CSRF enforcement in unit tests
* remove redundant spread
* some camel casing hiccups
* always yield the CSRF cookie, but avoid enforcing it if CSRF toggle is off
* Restyled by prettier (#5056)
Co-authored-by: Restyled.io <commits@restyled.io>
* set a CSRF header only if cookie is present
* enforce CSRF in CI
* install lodash directly for Cypress
* install request-cookies directly for Cypress. We should probably start loading package.json deps
* enable CSRF support when logout and login happen within the same spec
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
Co-authored-by: Restyled.io <commits@restyled.io>
* Add build arg to Dockerfile to control if we should build frontend assets
* Move more env settings into the shared one.
* Use build arg in docker-compose to skip frontend build.
* CirlceCI: Skip building frontend assets in backend tests
* Create dummy template files
* Expand file names manually.
* Add build arg to skip dev dependencies.
* Update Dockerfile
* Reverse logic of skip_dev_deps to what it should be.
Redash's docker-compose file will no longer bring up an environment from
a cold start due to recent upstream changes to the postgres image that
force the user to either set a password for the default superuser or
opt-in to allowing all connections without a password via environment
variable.
Upstream PR: https://github.com/docker-library/postgres/pull/658
Related Discussion: https://github.com/docker-library/postgres/issues/681
* enforce hard limits on non-responsive work horses by workers
* move differences from Worker to helper methods to help make the specialization clearer
* move HardLimitingWorker to redash/tasks
* move schedule.py to /tasks
* explain the motivation for HardLimitingWorker
* pleasing CodeClimate
* pleasing CodeClimate
* port query execution to RQ
* get rid of argsrepr
* avoid star imports
* allow queries to be cancelled in RQ
* return QueryExecutionErrors as job results
* fix TestTaskEnqueue and QueryExecutorTests
* remove Celery monitoring
* get rid of QueryTask and use RQ jobs directly (with a job serializer)
* Revert "remove Celery monitoring"
This reverts commit 37a74ea403.
* reduce occurences of the word 'task'
* use Worker, Queue and Job instead of spreading names that share behavior details
* remove locks for failed jobs as well
* did I not commit that colon? oh my
* push the redis connection to RQ's stack on every request to avoid verbose connection setting
* use a connection context for tests
* remove Celery monitoring
* 👋 Celery
* remove Celery from Cypress
* black it up
* some more black
* return all started/queued job ids (for future monitoring
* Restyled by prettier (#4522)
* remove celery.py
* remove some frontend residuals that reappeared after a merge
Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com>
* launch and monitor multiple workers using supervisor
* run supervisord in non-daemon mode
* redirect all output to stdout/stderr
* no need to log supervisord's output because it is redirected to stdout anyway
* updated and less brittle healthcheck
* add supervisor healthchecks
* remove redundant supervisor installation as it is installed by pip
* add a 5 minute check gate
* add rq and an rq_worker service
* add rq_scheduler and an rq_scheduler service
* move beat schedule to periodic_jobs queue
* move version checks to RQ
* move query result cleanup to RQ
* use timedelta and DRY up a bit
* move custom tasks to RQ
* do actual schema refreshes in rq
* rename 'period_jobs' to 'periodic', as it obviously holds jobs
* move send_email to rq
* DRY up enqueues
* ditch and use a partially applied decorator
* move subscribe to rq
* move check_alerts_for_query to rq
* move record_event to rq
* make tests play nicely with rq
* 👋 beat
* rename rq_scheduler to plain scheduler, now that there's no Celery scheduler entrypoint
* add some color to rq-worker's output
* add logging context to rq jobs (while keeping execute_query context via get_task_logger for now)
* move schedule to its own module
* cancel previously scheduled periodic jobs. not sure this is a good idea.
* rename redash.scheduler to redash.schedule
* allow custom dynamic jobs to be added decleratively
* add basic monitoring to rq queues
* add worker monitoring
* pleasing the CodeClimate overlords
* adjust cypress docker-compose.yml to include rq changes
* DRY up Cypress docker-compose
* add rq dependencies to cypress docker-compose service
* an odd attempt at watching docker-compose logs when running with Cypress
* Revert "an odd attempt at watching docker-compose logs when running with Cypress"
This reverts commit 016bd1a93e.
* show docker-compose logs at Cypress shutdown
* Revert "DRY up Cypress docker-compose"
This reverts commit 43abac7084.
* minimal version for binding is 3.2
* remove unneccesary code reloads on cypress
* add a command which errors if any of the workers running inside the current machine haven't been active in the last minute
* SCHEMAS_REFRESH_QUEUE is no longer a required setting
* split tasks/queries.py to execution.py and maintenance.py
* fix tests after query execution split
* pleasing the CodeClimate overlords
* rename worker to celery_worker and rq_worker to worker
* use /rq_status instead of /jobs
* show started jobs' time ago according to UTC
* replace all spaces in column names
* fix query tests after execution split
* exit with an int
* general lint
* add an entrypoint for rq_healthcheck
* fix indentation
* delete all existing periodic jobs before scheduling them
* remove some unrequired requires
* move schedule example to redash.schedule
* add RQ integration to Sentry's setup
* pleasing the CodeClimate overlords
* remove replication settings from docker-compose - a proper way to scale using docker-compose would be the --scale CLI option, which will be described in the knowledge based
* revert to calling a function in dynamic settings to allow periodic jobs to be scheduled after app has been loaded
* don't need to depend on context when templating failure reports
* set the timeout_ttl to double the interval to avoid job results from expiring and having periodic jobs not reschedule
* whoops, bad merge
* describe custom jobs and don't actually schedule them
* fix merge
* The sender email address has to be None for the test of "is email server
configured" to be correct. Moved the dev setting into docker-compose.yml.
* Move the REDASH_MAIL_SERVER setting into docker-compose.yml to revert the default value to its original value in case anyone was using it.
* Make worker dependant on email as it's the one that actually using it.
With the default Docker installed from sources on Ubuntu 19.04 it failed starting the project when asking for Compose version 3.7, but everything worked fine with 3.2.
* pick up *.py file changes and restart scheduler
* only watch /redash in order to avoid reloading on other file changes (e.g. tests)
* add dev_scheduler entrypoint
* use exec
* Update bin/docker-entrypoint
* rename dev_scheduler to dev_worker
* use same defaults as worker
* monitor "schemas" queue to run refresh_schema
`refresh_schema` tasks won't run because "schemas" queue isn't consumed
with default settings.
and it cause leaking redis storage, a "schemas" list is growing with time.
this PR fix it, adds "schemas" queue to monitor by celery.
* use scheduled_worker for "schemas" queue
instead of "adhoc_worker"
* open port 3000 for remote debugging
* add ptvsd
* use port 5678 to avoid changes in VSCode's default config
* attach to ptvsd
* no need to wait for attach
* actually, --debugger seems to be working
* create a new docker entry point for remote debugging
* alternative method to switch to debugging
* Fix tag counts for dashboards to be distinct.
This also makes use of the Dashboard.all base query.
Fix#3108.
* Use Query.all_queries as the base query for Query.all_tags.
* Add test case for Dashboard.all_tags.
* Update docker-compose.yml
jut updating docker-compose dev to version 3, any special reason to keep redis on 3? and pg on 9.5? I could also add a volume to pg, any reason not to do so?
* rollback to redis 3 and pg 9.5 due to consistency in project defaults
* Configure volume directly in worker service.
`restart: always` will cause containers to restart when the machine is rebooted or the docker process restarts. Setting this to `unless-stopped` will mean the container will restart only if the docker process/computer was rebooted without stopping the container first. Otherwise, running `docker-compose stop` and rebooting will cause the docker containers all to start again (and goodbye laptop battery!)
We might also consider `restart: on-failures` which would mean that rebooting would _not_ cause containers to start again unless requested via `docker-compose up`.
