mirror of
https://github.com/apache/impala.git
synced 2026-01-07 00:02:28 -05:00
d58aedff42
The change of the PARTITION_FANOUT from 32 to 16 exposed a pathological case due to the lack of coordination across concurrently executing spilling nodes of the same query. In particular, when we repartition a partition we try to initialize hash tables for the new partitions. But each hash table needs a block (for the nodes). In case there were not any IO-sized blocks available, because they had been consumed by other nodes, we would get into a loop trying to repartition those smaller partitions that couldn't initialize their hash table. Additional repartitions that, among others, would need additional blocks for the new streams. These partitions would end up being very small, still we would fail the query when we were reaching the MAX_PARTITION_DEPTH limit, which was fixed to 4. This patch fixes the problem by initializing the hash tables during repartitions with small pages. That is, the hash tables always first use a 64KB and a 512KB block for their nodes before switching to IO-sized blocks. This helps the partitioning algorithm to finish when we end up with partitions that can fit in those small pages. The performance may not be optimal, still the memory consumption is lower and the algorithm finishes. For example, without this patch and with PARTITION_FANOUT == 16 in order to run TPC-H Q18 and Q20 we needed 3.4GB and 3.1GB respectively. With this patch TPC-H Q18 needs ~1GB and Q20 975MB. This patch also removes the restriction of stopping repartitioning when we are reaching 4 levels of repartitioning. Instead, whenever we repartition we compare the size of the input partition to the size of the largest new partition. If there is no reduction on the size we stop the algorithm. Otherwise, we keep on repartitioning. That should help in cases of skew (e.g. due to bad hashing). There is a new MAX_PARTITION_DEPTH limit of 16. It is very unlikely we will ever hit this limit. Change-Id: Ib33fece10585448bc2d07bb39d0535d78b168ccc Reviewed-on: http://gerrit.cloudera.org:8080/119 Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com> Tested-by: Internal Jenkins