mirror of
https://github.com/apache/impala.git
synced 2026-01-09 06:05:09 -05:00
d46de9bba1
This commit will be backported to 5.4.x to improve plans when using Isilon and S3. The planner currently estimates the number of backends that an hdfs scan node will execute on as the number of datanodes holding block replica for the corresponding table. This can be a bad estimate for various reasons: 1) It's completely wrong when the scan is remote (e.g. S3 or Isilon). 2) It doesn't account for partition pruning. 3) The size of the set of hosts holding block replica may larger than the number of scan ranges. Improve the estimate by examing the scan ranges and taking locality into account. While this new estimate will eventually be used in all cases, this change uses the new estimate only when there is a remote scan range as to not change plans produced for local ranges (since this commit will be backported to 5.4.x). So, this commit purposely addresses only case 1. A follow on commit will enable the new logic for all cases. Also set up the S3PlannerTest so that we can enable it in the nightly jenkins S3 run. It was inadvertantly never enabled there. Change-Id: I3fd3f7c5431a535fb044c98c326338c21b8a1898 Reviewed-on: http://gerrit.cloudera.org:8080/425 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Internal Jenkins