jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events

Browse Source 
from metastore

HIVE-27746 introduced ALTER_PARTITIONS event type which is an
optimization of reducing the bulk ALTER_PARTITION events into a single
event. The components version is updated to pick up this change. It
would be a good optimization to include this in Impala so that the
number of events consumed by event processor would be significantly
reduced and help event processor to catch up with events quickly.

This patch enables the ability to consume ALTER_PARTITIONS event. The
downside of this patch is that, there is no before_partitions object in
the event message. This can cause partitions to be refreshed even on
trivial changes to them. HIVE-29141 will address this concern.

Testing:
- Added an end-to-end test to verify consuming the ALTER_PARTITIONS
event. Also, bigger time outs were added in this test as there was
flakiness observed while looping this test several times.

Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737
Reviewed-on: http://gerrit.cloudera.org:8080/22554
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
This commit is contained in:
Sai Hemanth Gantasala
2024-12-09 14:41:00 -08:00
committed by Csaba Ringhofer
parent 669d74f467
commit b67a9cecb3
9 changed files with 275 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
bin/create-test-configuration.sh
View File

@@ -169,6 +169,13 @@ rm -f hive-site-housekeeping-on/hive-site.xml
ln -s "${CONFIG_DIR}/hive-site_housekeeping_on.xml" \
hive-site-housekeeping-on/hive-site.xml
export HIVE_VARIANT=events_config_change
$IMPALA_HOME/bin/generate_xml_config.py hive-site.xml.py hive-site_events_config.xml
mkdir -p hive-site-events-config
rm -f hive-site-events-config/hive-site.xml
ln -s "${CONFIG_DIR}/hive-site_events_config.xml" \
hive-site-events-config/hive-site.xml
export HIVE_VARIANT=ranger_auth
HIVE_RANGER_CONF_DIR=hive-site-ranger-auth
$IMPALA_HOME/bin/generate_xml_config.py hive-site.xml.py hive-site_ranger_auth.xml

22
bin/impala-config.sh
View File

@@ -240,19 +240,19 @@ fi
: ${IMPALA_TOOLCHAIN_HOST:=native-toolchain.s3.amazonaws.com}
export IMPALA_TOOLCHAIN_HOST
export CDP_BUILD_NUMBER=58457853
export CDP_BUILD_NUMBER=66846208
export CDP_MAVEN_REPOSITORY=\
"https://${IMPALA_TOOLCHAIN_HOST}/build/cdp_components/${CDP_BUILD_NUMBER}/maven"
export CDP_AVRO_JAVA_VERSION=1.11.1.7.3.1.0-160
export CDP_HADOOP_VERSION=3.1.1.7.3.1.0-160
export CDP_HBASE_VERSION=2.4.17.7.3.1.0-160
export CDP_HIVE_VERSION=3.1.3000.7.3.1.0-160
export CDP_ICEBERG_VERSION=1.3.1.7.3.1.0-160
export CDP_KNOX_VERSION=2.0.0.7.3.1.0-160
export CDP_OZONE_VERSION=1.3.0.7.3.1.0-160
export CDP_PARQUET_VERSION=1.12.3.7.3.1.0-160
export CDP_RANGER_VERSION=2.4.0.7.3.1.0-160
export CDP_TEZ_VERSION=0.9.1.7.3.1.0-160
export CDP_AVRO_JAVA_VERSION=1.11.1.7.3.1.500-30
export CDP_HADOOP_VERSION=3.1.1.7.3.1.500-30
export CDP_HBASE_VERSION=2.4.17.7.3.1.500-30
export CDP_HIVE_VERSION=3.1.3000.7.3.1.500-30
export CDP_ICEBERG_VERSION=1.3.1.7.3.1.500-30
export CDP_KNOX_VERSION=2.0.0.7.3.1.500-30
export CDP_OZONE_VERSION=1.4.0.7.3.1.500-30
export CDP_PARQUET_VERSION=1.12.3.7.3.1.500-30
export CDP_RANGER_VERSION=2.4.0.7.3.1.500-30
export CDP_TEZ_VERSION=0.9.1.7.3.1.500-30
# Ref: https://infra.apache.org/release-download-pages.html#closer
: ${APACHE_MIRROR:="https://www.apache.org/dyn/closer.cgi"}