35375b3287
There are 3 builtin case conversion string functions: upper(), lower(),
and initcap(). Previously they only convert English alphabetic
characters. This patch adds support to deal with Unicode characters.
There are many corner cases in case conversion depending on the locale
and context. E.g.
1) Case conversion is locale-sensitive.
Turkish has 4 letter "I"s. English has only two, a lowercase dotted i
and an uppercase dotless I. Turkish has lowercase and uppercase forms of
both dotted and dotless I. So simply converting "i" to "I" for upper
case is wrong in Turkish:
+-------+--------+---------+
| | Dotted | Dotless |
+-------+--------+---------+
| Upper | İ | I |
+-------+--------+---------+
| Lower | i | ı |
+-------+--------+---------+
2) Case conversion may change a string's length.
The German word "grüßen" should be converted to "GRÜSSEN" in upper case:
the letter "ß" should be converted to "SS".
3) Case conversion is context-sensitive.
The Greek word "ὈΔΥΣΣΕΎΣ" should be converted to "ὀδυσσεύς", where the
Greek letter "Σ" is converted to "σ" or to "ς", depending on its
position in the word.
The above cases will be focus in follow-up JIRAs. This patch addes the
initial implementation of UTF-8 aware case conversion functions.
--------
Implementation:
In UTF-8 mode (turned on by set UTF8_MODE=true) of these functions, the
bytes in strings are converted to wide characters using std::mbrtowc().
Each wide character (wchar_t) will then be converted using std::towupper
or std::towlower correspondingly. We then convert them back to multi
bytes using std::wcrtomb().
Note that these builtins are locale aware. If impalad is launched
without a UTF-8 aware locale, e.g. LC_ALL="C", these builtins can't
recognize non-ascii characters, which will return unexpected results.
Thus we modify our docker images to set LC_ALL="C.UTF-8" instead of "C".
This patch also logs the current locale when launching impala daemons
for better debugging. We will support customized locale in IMPALA-11080.
Test:
- Add BE unit tests and e2e tests.
Change-Id: I443e89d46f4638ce85664b021666bc4f03ee8abd
Reviewed-on: http://gerrit.cloudera.org:8080/17785
Reviewed-by: Csaba Ringhofer <csringhofer@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Docker-related scripts for Impala
test-with-docker.py runs the Impala build and tests inside of Docker
containers, parallelizing the test execution across test suites. See that file
for more details.
This also contains infrastructure to build impala_base, catalogd,
statestored, impalad_coordinator, impalad_executor and
impalad_coord_exec container images from the output of an Impala build.
The containers can be built via the CMake target docker_images. See
CMakeLists.txt for the build targets.
Docker Quickstart with docker-compose
Various docker-compose files in this directory provide a convenient way to run
a basic Impala service with a single Impala Daemon and minimal set of supporting
services. A Hive MetaStore service is used to manage metadata. All filesystem data
is stored in Docker volumes. The default storage location for tables is in the
impala-quickstart-warehouse volume, i.e. if you create a table in Impala, it will
be stored in that volume by default.
Prerequisites:
A docker network called quickstart-network must be created, and the QUICKSTART_IP and
QUICKSTART_LISTEN_ADDR environment variables must be set.
docker network create -d bridge quickstart-network
export QUICKSTART_IP=$(docker network inspect quickstart-network -f '{{(index .IPAM.Config 0).Gateway}}')
export QUICKSTART_LISTEN_ADDR=$QUICKSTART_IP
If you want the cluster to be open to connections from other hosts, you can set
QUICKSTART_LISTEN_ADDR:
export QUICKSTART_LISTEN_ADDR=0.0.0.0
You can optionally set IMPALA_QUICKSTART_IMAGE_PREFIX to pull prebuilt images from a DockerHub repo.
For example, the following will use images like apache/impala:81d5377c2-impalad_coordinator:
export IMPALA_QUICKSTART_IMAGE_PREFIX="apache/impala:81d5377c2-"
Leave IMPALA_QUICKSTART_IMAGE_PREFIX unset to use images built from a local Impala dev environment.
Starting the cluster:
To start the base quickstart cluster without Kudu:
docker-compose -f docker/quickstart.yml up -d
To load data in background into Parquet and Kudu formats:
docker-compose -f docker/quickstart.yml -f docker/quickstart-kudu-minimal.yml \
-f docker/quickstart-load-data.yml up -d
To follow the data loading process, you can use the docker logs command, e.g.:
docker logs -f docker_data-loader_1
Connecting to the cluster:
The impala service can be connected to $QUICKSTART_IP, or if you set
QUICKSTART_LISTEN_ADDR=0.0.0.0, you can connect to it on localhost or your
machine's host name.
Connecting with containerized impala-shell:
docker run --network=quickstart-network -it \
${IMPALA_QUICKSTART_IMAGE_PREFIX}impala_quickstart_client impala-shell
Or with a pre-installed impala-shell:
impala-shell -i ${QUICKSTART_IP}
Accessing the Warehouse volume
If you want to directly interact with the contents of the warehouse in the
impala-quickstart-warehouse Docker volume or copy data from the host into the
quickstart warehouse, you can mount the volume in another container. E.g. to run
an Ubuntu 18.04 container with the warehouse directory mounted at
/user/hive/warehouse and your home directory mounted at /host_dir, you
can run the following command:
docker run -v ~:/host_dir -v docker_impala-quickstart-warehouse:/user/hive/warehouse \
-it ubuntu:18.04 /bin/bash
In the container, you can find the external and managed tablespaces stored in
the impala-quickstart-warehouse volume, for example:
root@377747c68bfa:/# ls /user/hive/warehouse/external/tpcds_raw/
call_center customer_demographics inventory store_returns web_sales
catalog_page date_dim item store_sales web_site
catalog_returns dbgen_version promotion time_dim
catalog_sales generated reason warehouse
customer household_demographics ship_mode web_page
customer_address income_band store web_returns
t@377747c68bfa:/# head -n2 /user/hive/warehouse/external/tpcds_raw/time_dim/time_dim.dat
0|AAAAAAAABAAAAAAA|0|0|0|0|AM|third|night||
1|AAAAAAAACAAAAAAA|1|0|0|1|AM|third|night||
It is then possible to copy data files from the host into an external table. In impala-shell, create an external table:
create external table quickstart_example(s string)
stored as textfile
location '/user/hive/warehouse/external/quickstart_example';
Then in the host and container shells, create a text file and copy it into the external table directory.
# On host:
echo 'hello world' > ~/hw.txt
# In container:
cp /host_dir/hw.txt /user/hive/warehouse/external/quickstart_example
You can then refresh the table to pick up the data file and query the table:
refresh quickstart_example;
select * from quickstart_example;
Environment Variable Overrides:
The following environment variables influence the behaviour of the various quickstart docker compose files.
KUDU_QUICKSTART_VERSION- defaults to latest, can be overridden to a different tag to use different Kudu images.IMPALA_QUICKSTART_IMAGE_PREFIX- defaults to using local images, change to to a different prefix to pick up prebuilt images.QUICKSTART_LISTEN_ADDR- can be set to either$QUICKSTART_IPto listen on only the docker network interface, or0.0.0.0to listen on all interfaces.
Publishing Quickstart Docker Images (for developers)
To publish the images you need to build locally then run publish_images_to_apache.sh
to tag and push them to a docker repository. For example, to tag the images with the
current commit hash and upload them to the default apache/impala Docker repository,
you can run the following commands:
cd $IMPALA_HOME
IMAGE_VERSION=$(git rev-parse --short HEAD)
./buildall.sh -release -noclean -ninja -skiptests -notests
ninja docker_images quickstart_docker_images
./docker/publish_images_to_apache.sh -v ${IMAGE_VERSION} -
For official Impala releases you will want to use the release version instead.