* Check all Trino catalogs if no catalog configured
- If catalog is not set, query 'SHOW CATALOGS' to get catalogs
- For each catalog, discover schema, table, columns
- If catalog has a `.`, then quote the catalog name
- New table name returned includes catalog name
- If catalog is set, same result as before
* Use self._handle_run_query_error(error)
* Fix lint problem & format w/ black
* Fix condition so catalog_prefixes are [""] when catalog is set
* Fix for loop / Remove default values for catalog & schema
* Add untested tests for Trino's get_schema
* Black formatting
* Add test_get_schema_catalog_set test & fix tests
* Parameterize the side_effect fn thoroughly
* Get catalogs (not catalog_prefixes) bc confusing
* Trino._get_catalogs added for testability
* Support to find a worksheet by the title for google_spreadsheets
* Add tests for parse_query
* Fix types
* Add tests when finding a worksheet by a title is failed
* Proxy by a wrapper instead of using spreadsheet directly. Do not use gspread in tests
* Add tests. Fix format of quoted strings for titles
* Fix an error format
* Add a newline
* Fix formatting
---------
Co-authored-by: Justin Clift <justin@postgresql.org>
* increase duplicate column names based on the original column name and not the number of duplicates detected
* add fetch columns test for base query runner
---------
Co-authored-by: Omer Lachish <omer@rauchy.net>
Co-authored-by: Guido Petri <18634426+guidopetri@users.noreply.github.com>
* Fix python query runner crashing the GUI due to missing 'rows' and/or 'columns' in the JSON returned data.
* Fix typo of previous commit.
* Throw exception when python query runner has invalid result.
* Update test_python.py
---------
Co-authored-by: YuhengChen <yuheng.chen@imaygou.com>
- A runner supporting the newest versions of ES,
aggregation, nested aggregations and nested fields.
- A runner for the SQL OpenDistro flavor
- A runner for the SQL X-Pack flavor
Co-authored-by: Nicolas Le Manchet <nicolas@lemanchet.fr>
Co-authored-by: wwl717195673 <717195673@qq.com>
ClickHouse query runner splits query into several and execute each query in turn. The result of the last execution is returned. Implementation uses ClickHouse sessions in the HTTP protocol. `session_id` is generated for the first query and then it is used with the subsequent queries (together with the `session_check` parameter).
If query runner gets a success response with empty body from ClickHouse (for example, in case of temporary table creation request) query runner returns empty response.
authored-by: Liubov Ulitina <ulitinalm@vl.ru>
* add default limit 1000
* Add frontend changes and connect to backend
* Fix query hash because of default limit
* fix CircleCI test
* adjust for comment
* Support multiple queries in a single query box
* Implement statement splitting function and add tests for it
* Add a test for databricks-specific syntax
* Split statements before running query
* feat: provide ssl options for Cassandra data source
* remove Log and prints
* Refactor to create module methods and unit tests
* Switch to using Enumerator and temp file
* Fix temporary file lifecycle for cert
* Align with changes on master
* Fix non certificate but ssl enabled usecase
* Make core app compatible with Python 3
No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.
* Use Python 3.7 as base docker image
Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.
* Upgrade some requirements to newest versions
Some of the older versions were not compatible with Python 3.
* Migrate tests to Python 3
* Build frontend on Python 3
* Make the HMAC sign function compatible with Python 3
In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.
* Use assertCountEqual instead of assertItemsEqual
The latter is not available in Python 3.
See https://bugs.python.org/issue17866
* Remove redundant encoding header for Python 3 modules
* Remove redundant string encoding in CLI
* Rename list() functions in CLI
These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().
Only the Python function is renamed, from the perspective of the
CLI nothing changes.
* Replace usage of Exception.message in CLI
`message` is not available anymore, instead use the string
representation of the exception.
* Adapt test handlers to Python 3
* Fix test that relied on dict ordering
* Make sure test results are always uploaded (#4215)
* Support encoding memoryview to JSON
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156
* Fix test relying on object address ordering
* Decode bytes returned from Redis
* Stop using e.message for most exceptions
Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.
* Fix writing XLSX files in Python 3
The buffer for the file should be made of bytes and the actual
content written to it strings.
Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.
* Fix test by comparing strings to strings
* Fix another exception message unavailable in Python 3
* Fix export to CSV in Python 3
The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.
* (Python 3) Use Redis' decode_responses=True option (#4232)
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)
* Make sure Redis connection uses decoded_responses option
* Remove unused imports.
* Use Redis' decode_responses option
* Remove cases of explicit Redis decoding
* Rename helper function and make sure it doesn't apply twice.
* Don't add decode_responses to Celery Redis connection URL
* Fix displaying error while connecting to SQLite
The exception message is always a string in Python 3, so no
need to try to decode things.
* Fix another missing exception message
* Handle JSON encoding for datasources returning bytes
SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.
This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.
* Fix Python 3 compatibility with RQ
* Revert some changes 2to3 tends to do (#4261)
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.
- Also fixed some naming issues with one character variables in list comprehensions.
- Fix Flask warning.
* Upgrade dependencies
* Remove useless `iter` added by 2to3
* Fix get_next_path tests (#4280)
* Removed setting SERVER_NAME in tests setup to avoid a warning.
* Change get_next_path to not return empty string in case of a domain only value.
* Fix redirect tests:
Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.
* Remove explicit dependency for Werkzeug
* Switched pytz and certifi to unbinded versions.
* Switch to new library for getting country from IP
`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .
* Python 3 RQ modifications (#4281)
* show current worker job (alongside with minor cosmetic column tweaks)
* avoid loading entire job data for queued jobs
* track general RQ queues (default, periodic and schemas)
* get all active RQ queues
* call get_celery_queues in another place
* merge dicts the Python 3 way
* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
* Remove legacy Python flake8 tests
* flatten lists and dicts to json to be used with SQLite's json_extract functions
* add test that verifies that lists and dicts are saved
* add test that verifies that lists and dicts are saved
* Process extra column metadata for a few sql-based data sources.
* Add Table and Column metadata tables.
* Periodically update table and column schema tables in a celery task.
* Fetching schema returns data from table and column metadata tables.
* Add tests for backend changes.
* Front-end shows extra table metadata and uses new schema response.
* Delete datasource schema data when deleting a data source.
* Process and store data source schema when a data source is first created or after a migration.
* Tables should have a unique name per datasource.
* Addressing review comments.
* Update migration file for mixins.
* Appease PEP8
* Upgrade migration file for rebase.
* Cascade delete.
* Adding org_id
* Remove redundant column and table prefixes.
* Non-existing tables and columns should be filtered out on the server side not client side.
* Fetching table samples should be optional and should happen in a separate task per table.
* Allow users to force a schema refresh.
* Use updated_at to help prune old schema metadata periodically.
* Using settings.SCHEMAS_REFRESH_QUEUE
* Added support for Apache Drill datasource
* Improvements in `Drill` query runner and minor refactoring
1. Drill query runner now inherits from `BaseHTTPQueryRunner`, because they both have a lot of common code.
2. `BaseHTTPQueryRunner.get_response` method now accepts `http_method` argument (original implementation was only capable of sending `GET` HTTP requests).
3. Added `order` to `BaseHTTPRequestRunner` configuration schema to fix order of UI elements based on the schema.
4. Eliminated duplicate method `_guess_type` in `GoogleSpreadsheet`, `Results` and `Drill` query runners, moved `guess_type` to `redash.query_runner`.
5. Removed tests for `_guess_type` in `GoogleSpreadsheet`, `Results` and `Drill` query runners, merged them into single test case and moved to `tests.query_runner.test_utils`.
6. Various minor changes (code style, imports, etc).
* Consistently use simplejson to loading and dumping JSON.
This introduces the new functions redash.utils.json_dumps and redash.utils.json_loads and simplifies the custom encoder setup.
UUIDs are now handled by the default encoder, too.
Fixes#2807.
Use string comparison in parse_boolean instead of the (simple)json module.
