Python tests and infra scripts will now use "python" from the virtualenv
via $IMPALA_HOME/bin/impala-python. Some scripts could be simplified now
that python 2.6 and a dependable set of third-party libraries are
available but that is not done as part of this commit.
Change-Id: If1cf96898d6350e78ea107b9026b12ba63a4162f
Reviewed-on: http://gerrit.cloudera.org:8080/603
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: Internal Jenkins
cgroups.py was using unsupported "except <Exception> as <var>" syntax.
generate_metrics.py was using the json module which is not available
in Python 2.4, but contains simplejson which provides the same
functionality.
Change-Id: If2c176c15a9573dd2a2acf5ee459ff24ce891ce3
Reviewed-on: http://gerrit.cloudera.org:8080/396
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Allows the base cgroup hierarchy path used by the impala
test cluster to be specified with the environment variable
IMPALA_CGROUP_BASE_PATH. This is needed to support older
kernels that do not use the proper default cgroup path
and do not even support finding the hierarchy via mount.
This will be used in jenkins test runs with RM enabled
which run on Centos6 images.
Change-Id: I30984a58fbcf990410f75f7feb5c1d549afa6ddd
Reviewed-on: http://gerrit.cloudera.org:8080/397
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Internal Jenkins
Context managers are not supported before Python 2.7. Removes the
use of the 'with' clause in cgroups.py because this code is
executed on Centos 6 packaging boxes with an older version of
Python.
Change-Id: Ic6bcf161086f671ec2010df16f9bb23534c57697
Reviewed-on: http://gerrit.cloudera.org:8080/385
Reviewed-by: Martin Grund <mgrund@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Adds a flag to start-impala-cluster.py (--enable_rm) to set up the
mini Impala cluster using Yarn and Llama. This hides a number of
flags that must be set on the impalads:
-enable_rm
-llama_addressess: set to the local llama service
-fair_scheduler_allocation_path: set to the path of the fair-scheduler.xml
in each node's hadoop conf directory
-cgroup_hierarchy_path: set to a path in the CPU cgroup hierarchy which
has the correct permissions for Impala to manage a child cgroup. The
path comes from cgroups.py.
The new module cgroups.py was added to contain cgroups-related
utilities. Right now it provides paths to the CPU controller
hierarchy root and a path within the hierarchy that can be used
for impalads (i.e. have the proper permissions, one for each
cluster node).
Change-Id: Ic2181ec5613c180f240958c84f885c6b136a64d4
Reviewed-on: http://gerrit.cloudera.org:8080/369
Tested-by: Internal Jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
I did a local benchmark and there's minimal performance impact(<1%)
Change-Id: I8d84a145acad886c52587258b27d33cff96ea399
(cherry picked from commit 7e750ad5d90007cc85ebe493af4dce7a537ad7c0)
Reviewed-on: http://gerrit.cloudera.org:8080/189
Reviewed-by: Juan Yu <jyu@cloudera.com>
Tested-by: Internal Jenkins
For wide Avro tables, ReadZLong() would get inlined many times into a
single function body, causing LLVM to crash. Not inlining doesn't seem
to have a performance impact on narrow tables, and helps with wide
tables.
This change also adds tests over wide (i.e. many-column) tables. The
test tables are produced by specifying shell commands to generate test
tables in functional_schema_template.sql, which are executed in
generate-schema-statements.py. In the SQL templates, sections starting
with a ` are treated as shell commands. The output of the shell
command is then used as the section text. This is only a starting
point; it isn't currently implemented for all sections, and may have
to be tweaked if we use this mechanism for all tables.
Change-Id: Ife0d857d19b21534167a34c8bc06bc70bef34910
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2206
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
(cherry picked from commit 1c5951e3cce25a048208ab9bb3a3aed95e41cf67)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2353
Tested-by: jenkins
