3e6326538c
This patch adds support for serving all the webUI content with gzip
content encoding.
For large JSONs and text profiles, Impala's webUI renderings maybe
hindered by the user's network bandwidth.
As the browser's native gzip decompression is very fast e.g. 300-400MB/s,
combining it with a faster compression level(i.e. gzip Z_BEST_SPEED) in
backend results in significant increases in speed i.e. faster load times.
During compression, instead of multiple reallocations, existing string
data is reinterpreted to reduce memory usage.
In case of failure during compression, the content is served in plain
format as before.
As currently, none of the memory allocation's are being tracked for the
rapidjson's generated documents(or any daemon webserver's served string),
it would be helpful to display the peak memory usage of a single buffer
used to serve all webUI content.
In the future, it is recommended to implement and use custom allocators
for all large served strings and rapidjson generated documents.
(See IMPALA-14178, IMPALA-14179)
Memory trackers within ExecEnv are now initialized before enabling
the webserver, allowing their use as parent memory trackers.
For now, the memory used by the compressed buffer, for each compressed
response is being tracked.
(i.e. through the "WebserverCompressedBuffer" MemTracker)
Example:
For Impala daemon, it is included in the execution environment's
process memory tracker and displayed on the /memz page as follows.
# After serving a general webpage like /memz
WebserverCompressedBuffer: Total=0 Peak=227.56 KB
# After serving a query profile text / JSON
WebserverCompressedBuffer: Total=0 Peak=4.09 MB
Tests:
* Added new tests to validate plain and gzipped content encoding headers
in test_web_pages.py - TestWebPage:test_content_encoding
in util/webserver-test.cc - Webserver::ContentEncodingHeadersTest
* The pre-existing tests validate the content
in test_web_pages.py, all tests request and validate gzipped content
in util/webserver-test.cc, all tests request and validate plain text
* Performance:
Approximate improvements for a TPC-DS 14 query ran locally with 3 nodes
with defaults
-> JSON profile : 4.53MB to 428.94KB
Without throttling / Raw local: 421ms to 421ms
Based on firefox's throttling(8 mbps): 8s to 2s
-> Text profile : 1.24MB to 219KB
Without throttling / Raw local: 281ms to 281ms
Based on firefox's throttling(8 mbps): 1.3s to 281ms
Change-Id: I431088a30337bbef2c8d6e16dd15fb6572db0f15
Reviewed-on: http://gerrit.cloudera.org:8080/22599
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Welcome to Impala
Lightning-fast, distributed SQL queries for petabytes of data stored in open data and table formats.
Impala is a modern, massively-distributed, massively-parallel, C++ query engine that lets you analyze, transform and combine data from a variety of data sources:
- Best of breed performance and scalability.
- Support for data stored in Apache Iceberg, HDFS, Apache HBase, Apache Kudu, Amazon S3, Azure Data Lake Storage, Apache Hadoop Ozone and more!
- Wide analytic SQL support, including window functions and subqueries.
- On-the-fly code generation using LLVM to generate lightning-fast code tailored specifically to each individual query.
- Support for the most commonly-used Hadoop file formats, including Apache Parquet and Apache ORC.
- Support for industry-standard security protocols, including Kerberos, LDAP and TLS.
- Apache-licensed, 100% open source.
More about Impala
The fastest way to try out Impala is a quickstart Docker container. You can try out running queries and processing data sets in Impala on a single machine without installing dependencies. It can automatically load test data sets into Apache Kudu and Apache Parquet formats and you can start playing around with Apache Impala SQL within minutes.
To learn more about Impala as a user or administrator, or to try Impala, please visit the Impala homepage. Detailed documentation for administrators and users is available at Apache Impala documentation.
If you are interested in contributing to Impala as a developer, or learning more about Impala's internals and architecture, visit the Impala wiki.
Supported Platforms
Impala only supports Linux at the moment. Impala supports x86_64 and has experimental support for arm64 (as of Impala 4.0). Impala Requirements contains more detailed information on the minimum CPU requirements.
Supported OS Distributions
Impala runs on Linux systems only. The supported distros are
- Ubuntu 16.04/18.04
- CentOS/RHEL 7/8
Other systems, e.g. SLES12, may also be supported but are not tested by the community.
Export Control Notice
This distribution uses cryptographic software and may be subject to export controls. Please refer to EXPORT_CONTROL.md for more information.
Build Instructions
See Impala's developer documentation to get started.
Detailed build notes has some detailed information on the project layout and build.