mirror of
https://github.com/apache/impala.git
synced 2026-01-05 21:00:54 -05:00
IMPALA-5167: Reduce the number of Kudu clients created (FE)
Creating Kudu clients is very expensive as each will fetch metadata from the Kudu master, so we should minimize the number of Kudu clients that get created. This patch stores a map from Kudu master addressed to Kudu clients in KuduUtil to be used across the FE and catalog. Another patch has already addressed the BE. Future work will consider providing a way to invalidate the stored Kudu clients in case something goes wrong (IMPALA-5685) This relies on two changes on the Kudu side: one that clears non-covered range entries from the client's cache on table open (d07ecd6ded01201c912d2e336611a6a941f48d98), and one that automatically refreshes auth tokens when they expire (603c1578c78c0377ffafdd9c427ebfd8a206bda3). This patch disables some tests that no longer work as they relied on Kudu metadata loading operations timing out, but since we're reusing clients the metadata is already loaded when the test is run. Testing: - Ran a stress test on a 10 node cluster: scan of a small Kudu table, 1000 concurrent queries, load on the Kudu master was reduced signficantly, from ~50% cpu to ~5%. (with the BE changes included) - Ran the Kudu e2e tests. - Manually ran a test with concurrent INSERTs and 'ALTER TABLE ADD PARTITION' (which is affected by the Kudu side change mentiond above) and verified correctness. Change-Id: I9b0b346f37ee43f7f0eefe34a093eddbbdcf2a5e Reviewed-on: http://gerrit.cloudera.org:8080/6898 Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com> Tested-by: Impala Public Jenkins
This commit is contained in:
committed by
Impala Public Jenkins
parent
322ccb0e49
commit
399b184bbc
@@ -1,3 +1,10 @@
|
||||
# TODO: enable this once we have a way to invalidate kudu clients (IMPALA-5685)
|
||||
#====
|
||||
#---- QUERY
|
||||
#show create table functional_kudu.alltypestiny
|
||||
#---- CATCH
|
||||
#Error opening Kudu table 'impala::functional_kudu.alltypestiny'
|
||||
#====
|
||||
====
|
||||
---- QUERY
|
||||
# TODO: improve error messages (here and below) when KUDU-1734 is resolved
|
||||
@@ -6,11 +13,6 @@ describe functional_kudu.alltypes
|
||||
Error opening Kudu table 'impala::functional_kudu.alltypes'
|
||||
====
|
||||
---- QUERY
|
||||
show create table functional_kudu.alltypes
|
||||
---- CATCH
|
||||
Error opening Kudu table 'impala::functional_kudu.alltypes'
|
||||
====
|
||||
---- QUERY
|
||||
create table test_kudu (x int primary key)
|
||||
partition by hash(x) partitions 3 stored as kudu
|
||||
---- CATCH
|
||||
|
||||
@@ -1,3 +1,10 @@
|
||||
# TODO: enable this once we have a way to invalidate kudu clients (IMPALA-5685)
|
||||
#====
|
||||
#---- QUERY
|
||||
#show table stats functional_kudu.alltypestiny
|
||||
#---- CATCH
|
||||
#Error accessing Kudu for table stats
|
||||
#====
|
||||
====
|
||||
---- QUERY
|
||||
# Expected timeout while planning the scan node.
|
||||
@@ -6,8 +13,3 @@ select * from functional_kudu.alltypes
|
||||
---- CATCH
|
||||
Unable to initialize the Kudu scan node
|
||||
====
|
||||
---- QUERY
|
||||
show table stats functional_kudu.alltypes
|
||||
---- CATCH
|
||||
Error accessing Kudu for table stats
|
||||
====
|
||||
|
||||
Reference in New Issue
Block a user