mirror of
https://github.com/apache/impala.git
synced 2026-01-10 00:00:16 -05:00
d46de9bba1
This commit will be backported to 5.4.x to improve plans when using Isilon and S3. The planner currently estimates the number of backends that an hdfs scan node will execute on as the number of datanodes holding block replica for the corresponding table. This can be a bad estimate for various reasons: 1) It's completely wrong when the scan is remote (e.g. S3 or Isilon). 2) It doesn't account for partition pruning. 3) The size of the set of hosts holding block replica may larger than the number of scan ranges. Improve the estimate by examing the scan ranges and taking locality into account. While this new estimate will eventually be used in all cases, this change uses the new estimate only when there is a remote scan range as to not change plans produced for local ranges (since this commit will be backported to 5.4.x). So, this commit purposely addresses only case 1. A follow on commit will enable the new logic for all cases. Also set up the S3PlannerTest so that we can enable it in the nightly jenkins S3 run. It was inadvertantly never enabled there. Change-Id: I3fd3f7c5431a535fb044c98c326338c21b8a1898 Reviewed-on: http://gerrit.cloudera.org:8080/425 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins
Welcome to Impala
Lightning-fast, distributed SQL queries for petabytes of data stored in Apache Hadoop clusters.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
- Best of breed performance and scalability.
- Support for data stored in HDFS, Apache HBase and Amazon S3.
- Wide analytic SQL support, including window functions and subqueries.
- On-the-fly code generation using LLVM to generate CPU-efficient code tailored specifically to each individual query.
- Support for the most commonly-used Hadoop file formats, including the Apache Parquet (incubating) project.
- Apache-licensed, 100% open source.
More about Impala
To learn more about Impala as a business user, or to try Impala live or in a VM, please visit the Impala homepage.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.