This should allow individual service components, such as a single nodemanager,
to be shutdown for failure testing. The mini-cluster bundled with hadoop is a
single process that does not expose the ability to control individual roles.
Now each role can be controlled and configured independently of the others.
Change-Id: Ic1d42e024226c6867e79916464d184fce886d783
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1432
Tested-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2297
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
* One last NotifyThreadUsageChange() mismatched pair
* Don't set resource in plan fragment params if there isn't a resource
available. This fixes the problem where if no fragment with resources
was assigned to the same node as the coordinator, the coordinator
would have a dummy resource allocation which didn't work with
expansion.
* Substitute #ID in all impalad arguments to start-impala-cluster.py
with the 0-indexed ID of the impalad being started. This is required
to have different Impala processes use different cgroups.
Change-Id: If8c8fd8bef0809bdaf16115a45a9695fc2bf3e1b
(cherry picked from commit c71ce45e97570b8c09900eb5ae2e26984d3306a4)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2060
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
The problem was that were were deleting the version.info file because the default
of gen_build_version.py recently changed from --noclean to --clean.
Also fixed a bug in the shell version generation and made debugging a bit easier
by dumping the contents of version.info whenever it is generated.
Change-Id: I764d01c9e46eed1bd39de79bf076c15afa599486
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1901
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
(cherry picked from commit fa673b4d3342fc825ee7fa942bd254234d222906)
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1910
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
The HS2 metadata operations do not go through analysis() so the prioritized
loading will not happen for them. Most of the HS2 metadata ops work purely
on table/db names, but GetColumns() requires loading the table metadata. This
patch updates MetadataOp to collect a set of missing tables and request these
tables be loaded from the catalog server. The operation will wait until the tables
are loaded in the local catalog before proceeding.
Change-Id: I070f2a0d9194d3317f09431971be9a8dffbc7386
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1542
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1557
Before this patch the -noclean option had almost no effect on the BE build time because
some source files were re-generated with .py scripts regardless.
This change allows ./buildall -skiptests -noclean to do a true incremental rebuild.
Change-Id: Ib3af85db05bdc96a2279a22c1d49d735f2cabd4e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1394
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1415
Changes include:
- version changes in impala-config
- version changes in various loading scripts
- hbase jars are no longer in hive/lib
- mini-llama script changes
- updates due to sentry api changes
- JDBC tests disabled
- unsupported types tests disabled.
Change-Id: If8cf1b7ad8e22aa4d23094b9a4b1047f7e9d93ee
Fixed codepath with rm disabled. Set enable_rm to false by default.
Change-Id: I3bf2d0525d91243ec3c0ea048b0c03680befcda2
Conflicts:
be/src/runtime/runtime-state.cc
Impala reserves resources from YARN via Llama and handles resources
preemptions by cancelling affected queries. Adds the Impala Resource
Broker for interacting with Llama. Refactors scheduler and coordinator
to move fragment-to-host assignment logic into scheduler. Local test
setup uses MiniLLama.
Change-Id: Ic7b0fe43de52d30f4207b4e65cce7e6a294e54e1
While loading parquet, there are a few table creation queries that use the 'like'
keyword; this ends up opening a small race window when all the table formats are created
concurrently. With this change, we create the text tables first before attempting to
parallelize the rest of the data loading.
Change-Id: Ib84cf0e5120b3588d3f0503d7119ca055e08e53f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1241
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
* upload_codereview.py is no longer used since Rietveld is long gone
* runplanservice is deprecated as there is no longer a separate
PlanService
* README only mentions a single internal wiki page.
Change-Id: Iba61a3d62381deb882c4168f142574f2492e0969
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1249
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Reviewed-by: Nong Li <nong@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Enables JVM debugging by default for the catalogd and impalads
created via bin/start-impala-cluster.py.
Adds a -jvm_args command line option for passing additional JVM args to
the catalogd and impalads.
Change-Id: I68e901661bd1fd7eefa05ba84dbacf29dd124685
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1213
Tested-by: jenkins
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
During a full data load, we load all the data (except parquet) via hive, and then load the
parquet data via Impala. The catalog service does not update the metadata of tables
changed outside Impala, so we need to explicitly invalidate the metadata before loading
parquet data.
Change-Id: Iec39db9ea46e4a11b17589881732629a56444120
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1207
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Instead of calling 'invalidate metadata' before loading each workload
we should call it once, after loading all test data. This will allow
us to pickup data inserted by Hive. The only reason this worked before
is because we restart Impala before running the tests. This will also
be a bit faster if loading multiple workloads.
Change-Id: I28d42bbf5d7a24b5fde687d67a4b41472ec4b897
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1153
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Goodnight, sweet non-blocking prince. We didn't support, or test, this
configuration, and it doesn't work with security or sessions and brings
in some annoying dependencies that are a pain to build.
We have other RPC-stack options to investigate; we may wind up re-adding
the non-blocking server but only in a way that supports all required
features more regularly.
Change-Id: Ifbcabc5014441f6d31c342c4e288dd7fc6201443
This patch makes the workload runner's logging concise and more informative. Specifically,
it
- logs the time taken for each iteration of a query.
- changes the default log level to INFO.
- The output is less verbose.
Change-Id: I5f964cf76269fd64ce127b9e4c51fe1deafd1d1b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1076
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
This change cleans up our FE pom.xml file by removing unneeded
dependencies and system dependencies (system dependencies are now pulled in
from the Maven release repository).
The upside is that our pom is cleaner and it will also help reduce the likelihood of
broken dependencies since Maven will pull in the right versions. The downside
is that we now pull in quite a few more JARs.
Note: I was unable to find release artifacts for Sentry and Parquet so I leaving
those as "system" for now.
Change-Id: I0b917b09a02243d78d89747591ab6bccacf7cf38
Saving changes
Change-Id: I3697a7b44884c40e077b3e354fef76625e1b881d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1011
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
The following changes have been made:
-- Update hbase
-- Update hive
-- Update hadoop
-- Update the parquet version to 1.2.5
Change-Id: Id6ceaef0e9eebab27ffd408160116fa84ed300fb
The audit logs currently have the "impersonator" field set to what we call the doAsUser
and the "user" field set as the connected user. They should be reversed.
Added basic tests to validate the correct event gets audited.
Change-Id: Idfa0aaa6c88debedc4993bd0489dbd3f696fcf17
Reviewed-on: http://gerrit.ent.cloudera.com:8080/958
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
This helps speed up the restart time becuase we don't need to restart
the catalog server and reload the table metadata. This is useful if you
want to restart the impalad with a different command line parameter
or if you are making changes to only the impalad binary.
Change-Id: I0b714afaf7e508c450a353a53d67d95165de3486
Reviewed-on: http://gerrit.ent.cloudera.com:8080/897
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
