mirror of
https://github.com/apache/impala.git
synced 2026-01-05 21:00:54 -05:00
54665120cb
- Rework the discrepancy searcher to run DML statements. We do this by
using the query profile to choose a table, copy that table, and
generate a statement that will INSERT into that copy. We chose a slow
copy over other methods because INSERTing into a copy is a more
reliable test that prevents table sizes from getting out of hand or
time-consuming replay to reproduce a particular statement.
- Introduce a statement generator stub. The real generator work is
tracked in IMPALA-4351 and IMPALA-4353. Here we simply generate a
basic INSERT INTO ... VALUES statement to make sure our general query
execution flow is working.
- Add query profile stub for DML statements (INSERT-only at this time).
Since we'll want INSERT INTO ... SELECT very soon, this inherits from
DefaultProfile. Also add building blocks for choosing random
statements in the DefaultProfile.
- Improve the concept of an "execution mode" and add new modes. Before,
we had "RAW", "CREATE_TABLE_AS", and "CREATE_VIEW_AS". The idea here
is that some random SELECT queries could be generated as "CREATE
TABLE|VIEW AS" at execution time, based on weights in the query
profile. First, we remove the use of raw string literals for this,
since raw string literals can be error-prone, and introduce a
StatementExecutionMode class to contain a namespace for the enumerated
statement execution modes. Second, we introduce a couple new execution
modes. The first is DML_SETUP: this is a DML statement that needs to
be run in both the test and reference databases concurrently. For our
purposes, it's the INSERT ... SELECT that copies data from the chosen
random table into the table copy. The second is DML_TEST: this is a
randomly-generated DML statement.
- Switch to using absolute imports in many places. There was a mix of
absolute and relative imports happening here, and they were causing
problems, especially when comparing data types. In Python,
<class 'db_types.Int'> != <class 'tests.comparison.db_types.Int'>.
Using
from __future__ import absolute_import
didn't seem to catch the relative import usage anyway, so I haven't
employed that.
- Rename some, but not nearly all, names from "query" to "statement".
Doing this is a rather large undertaking leading to much larger diffs
and testing (IMPALA-4602).
- Fix a handful of flake8 warnings. There are a bunch that went unfixed
for over- and under-indentation.
- Testing
o ./discrepancy_searcher.py runs with and without --explain-only, and
with --profile default and --profile dmlonly. For tpch_kudu data, it
seems sufficient to use a --timeout of about 300.
o Leopard run to make sure standard SELECT-only generation still works
o Generated random stress queries locally
o Generated random data locally
Change-Id: Ia4c63a2223185d0e056cc5713796772e5d1b8414
Reviewed-on: http://gerrit.cloudera.org:8080/5387
Reviewed-by: Jim Apple <jbapple-impala@apache.org>
Tested-by: Impala Public Jenkins
Summary
-------
This package runs the query generator continuously. Compares Impala and Postgres results
for a randomly generated query and produces several reports per day. Reports are
displayed on a web page which allows the user to conveniently examine the discovered
issues. The user can also start a custom run against a private Impala branch using the
web interface.
Requirements
------------
Docker -- A docker image with Impala and PostgresQL installed and at
least one reference database loaded into PostgresQL. data_generator.py is a useful
tool to migrate data from Impala into PostgresQL.
To get started, run ./controller.py and ./front_end.py. You should be able to view the
web page at http://localhost:5000. Results and logs are saved to /tmp/query_gen
Basic Configuration
-------------------
The following are useful environment variables for running the
controller and Docker images within it.
DOCKER_USER - user *within* the Impala Docker container who owns the
Impala source tree and test data.
DOCKER_PASSWORD - password for the user *within* the Impala Docker
container.
TARGET_HOST - host system on which Docker Engine is running. This is the
host that the controller will use to issue Docker commands like "docker
run".
TARGET_HOST_USERNAME - username for controller process to use to SSH
into TARGET_HOST. Via Fabric, one can either type a password or use SSH
keys.
DOCKER_IMAGE_NAME - image to pull via "docker pull"
External Volume Configuration
-----------------------------
To run Leopard against Impala with Kudu, we need to work around
KUDU-1419. KUDU-1419 is likely to occur if your Docker Storage Engine is
AUFS, or maybe others. The easiest way to overcome this is to mount an
external Docker volume that contain the necessary test data. To try to
handle this automatically, you can export any or all of the environment
variables, depending on your host and container setups:
DOCKER_IMPALA_USER_UID, DOCKER_IMPALA_USER_GID - numeric UID and GID for
the owner of the Impala test data (testdata/cluster from an Impala
source checkout) within your Docker container. Numeric IDs are needed,
because there is no guarantee the symbolic owner and group on the
container match the IDs on the target host.
HOST_TESTDATA_EXTERNAL_VOLUME_PATH - path on TARGET_HOST where the
external volume will reside. This is the destination for rsync to warm
the volume and the left-hand side of "docker run -v".
DOCKER_TESTDATA_VOLUME_PATH - path on your Docker container to the
testdata/cluster Impala directory. This is source for rsync to warm the
volume and the right-hand side of "docker run -v".
HOST_TO_DOCKER_SSH_KEY - name of private key on TARGET_HOST for use with
rsync so as to "warm" the external volume automatically.
You are encouraged to configure your container in such a way that rsync
with passwordless SSH is possible so as to create the external volume
using the environment variables above.
To do that, this is a handy guide on how to use rsync with SSH keys:
https://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/