mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
51e8175c62
Makefile was updated to include zstd in the ${IMPALA_HOME}/toolchain
directory. Other changes were made to make zstd headers and libs
accessible.
Class ZstandardCompressor/ZstandardDecompressor was added to provide
interfaces for calling ZSTD_compress/ZSTD_decompress functions. Zstd
supports different compression levels (clevel) from 1 to
ZSTD_maxCLevel(). Zstd also supports -ive clevels, but since the -ive
values represents uncompressed data they won't be supported. The default
clevel is ZSTD_CLEVEL_DEFAULT.
HdfsParquetTableWriter was updated to support ZSTD codec. The
new codecs can be set using existing query option as follows:
set COMPRESSION_CODEC=ZSTD:<clevel>;
set COMPRESSION_CODEC=ZSTD; // uses ZSTD_CLEVEL_DEFAULT
Testing:
- Added unit test in DecompressorTest class with ZSTD_CLEVEL_DEFAULT
clevel and a random clevel. The test unit decompresses an input
compressed data and validates the result. It also tests for
expected behavior when passing an over/under sized buffer for
decompressing.
- Added unit tests for valid/invalid values for COMPRESSION_CODEC.
- Added e2e test in test_insert_parquet.py which tests writing/read-
ing (null/non-null) data into/from a table (w different data type
columns) using multiple codecs. Other existing e2e tests were
updated to also use parquet/zstd table format.
- Manual interoperability tests were run between Impala and Hive.
Change-Id: Id2c0e26e6f7fb2dc4024309d733983ba5197beb7
Reviewed-on: http://gerrit.cloudera.org:8080/13507
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
67 lines
2.0 KiB
CMake
67 lines
2.0 KiB
CMake
##############################################################################
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
##############################################################################
|
|
|
|
# - Find ZSTD (zstd.h, libzstd.a, libzstd.so, and libzstd.so.1)
|
|
# ZSTD_ROOT hints the location
|
|
#
|
|
# This module defines
|
|
# ZSTD_INCLUDE_DIR, directory containing headers
|
|
# ZSTD_LIBS, directory containing zstd libraries
|
|
# ZSTD_STATIC_LIB, path to libzstd.a
|
|
|
|
set(ZSTD_SEARCH_HEADER_PATHS ${ZSTD_ROOT}/include)
|
|
|
|
set(ZSTD_SEARCH_LIB_PATH ${ZSTD_ROOT}/lib)
|
|
|
|
find_path(ZSTD_INCLUDE_DIR
|
|
NAMES zstd.h
|
|
PATHS ${ZSTD_SEARCH_HEADER_PATHS}
|
|
NO_DEFAULT_PATH
|
|
DOC "Path to ZSTD headers"
|
|
)
|
|
|
|
find_library(ZSTD_LIBS NAMES zstd
|
|
PATHS ${ZSTD_SEARCH_LIB_PATH}
|
|
NO_DEFAULT_PATH
|
|
DOC "Path to ZSTD library"
|
|
)
|
|
|
|
find_library(ZSTD_STATIC_LIB NAMES libzstd.a
|
|
PATHS ${ZSTD_SEARCH_LIB_PATH}
|
|
NO_DEFAULT_PATH
|
|
DOC "Path to ZSTD static library"
|
|
)
|
|
|
|
if (NOT ZSTD_LIBS OR NOT ZSTD_STATIC_LIB)
|
|
message(FATAL_ERROR "Zstd includes and libraries NOT found. "
|
|
"Looked for headers in ${ZSTD_SEARCH_HEADER_PATHS}, "
|
|
"and for libs in ${ZSTD_SEARCH_LIB_PATH}")
|
|
set(ZSTD_FOUND FALSE)
|
|
else()
|
|
set(ZSTD_FOUND TRUE)
|
|
endif ()
|
|
|
|
mark_as_advanced(
|
|
ZSTD_INCLUDE_DIR
|
|
ZSTD_LIBS
|
|
ZSTD_STATIC_LIB
|
|
)
|
|
|
|
|