mirror of
https://github.com/apache/impala.git
synced 2026-01-31 18:00:17 -05:00
f38da0df8e
Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can be done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * Ran exhaustive tests. * Ran core tests with ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +----------+-----------------------+---------+------------+------------+----------------+ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +----------+-----------------------+---------+------------+------------+----------------+ | TPCH(30) | parquet / none / none | 10.14 | -7.29% | 5.05 | -11.68% | +----------+-----------------------+---------+------------+------------+----------------+ +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ | Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 38.87 | 38.44 | +1.13% | 7.17% | * 10.92% * | 20 | +0.72% | 0.72 | 0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.28 | 4.26 | +0.50% | 1.92% | 1.09% | 20 | +0.03% | 0.31 | 1.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32 | +0.05% | 2.01% | 1.89% | 20 | -0.03% | -0.36 | 0.08 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.73 | 3.75 | -0.42% | 0.84% | 1.05% | 20 | -0.25% | -0.77 | -1.40 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.80 | 9.83 | -0.38% | 0.51% | 0.80% | 20 | -0.32% | -1.30 | -1.81 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.98 | 2.00 | -1.32% | 1.74% | 2.81% | 20 | -0.64% | -1.71 | -1.79 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.22 | 1.25 | -2.14% | 2.66% | 4.15% | 20 | -0.96% | -2.00 | -1.95 | | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.13 | 5.22 | -1.65% | 1.20% | 1.40% | 20 | -1.76% | -3.34 | -4.02 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.46 | 2.56 | -4.13% | 2.49% | 1.99% | 20 | -4.31% | -4.04 | -5.94 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 81.63 | 85.07 | -4.05% | 4.94% | 3.06% | 20 | -5.46% | -3.28 | -3.21 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.07 | 5.50 | I -7.92% | 0.96% | 1.33% | 20 | I -8.51% | -5.27 | -22.14 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.00 | 26.24 | I -8.57% | 0.46% | 0.38% | 20 | I -9.34% | -5.27 | -67.47 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.66 | 9.50 | I -8.86% | 0.62% | 0.44% | 20 | I -9.75% | -5.27 | -55.17 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.01 | 6.70 | I -10.19% | 1.01% | 0.90% | 20 | I -11.25% | -5.27 | -35.76 | | TPCH(30) | TPCH-Q12 | parquet / none / none | 2.98 | 3.39 | I -12.23% | 1.48% | 1.48% | 20 | I -13.56% | -5.27 | -27.75 | | TPCH(30) | TPCH-Q11 | parquet / none / none | 1.69 | 2.00 | I -15.55% | 1.63% | 1.47% | 20 | I -18.09% | -5.27 | -34.60 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.42 | 2.87 | I -15.69% | 1.48% | 1.26% | 20 | I -18.61% | -5.27 | -39.50 | | TPCH(30) | TPCH-Q14 | parquet / none / none | 4.64 | 6.27 | I -26.02% | 1.35% | 0.73% | 20 | I -35.37% | -5.27 | -94.07 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.19 | 4.37 | I -27.01% | 1.54% | 0.99% | 20 | I -36.85% | -5.27 | -80.74 | | TPCH(30) | TPCH-Q5 | parquet / none / none | 4.57 | 6.39 | I -28.36% | 1.04% | 0.75% | 20 | I -39.56% | -5.27 | -120.02 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 3.15 | 4.71 | I -33.06% | 1.59% | 1.31% | 20 | I -49.43% | -5.27 | -87.64 | | TPCH(30) | TPCH-Q8 | parquet / none / none | 5.25 | 7.95 | I -33.95% | 0.95% | 0.53% | 20 | I -51.11% | -5.27 | -185.02 | +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Reviewed-on: http://gerrit.cloudera.org:8080/14538 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>