mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
b39cd79ae8
This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py This iteration should support all types with the exception of complex types. Calcite does not have a STRING type, so the string type is represented as VARCHAR(MAXINT) similar to how Hive represents their STRING type. The ImpalaTypeConverter file is used to convert the Impala Type object to corresponding Calcite objects. Authorization is not yet working with this current commit. A Jira has been filed (IMPALA-13011) to deal with this. Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Reviewed-on: http://gerrit.cloudera.org:8080/21109 Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
76 lines
2.7 KiB
Bash
76 lines
2.7 KiB
Bash
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
# This script explicitly sets the CLASSPATH for embedded JVMs (e.g. in
|
|
# Impalad or in runquery) Because embedded JVMs do not honour
|
|
# CLASSPATH wildcard expansion, we have to add every dependency jar
|
|
# explicitly to the CLASSPATH.
|
|
|
|
if [ "$0" = "$BASH_SOURCE" ]; then
|
|
>&2 echo "error: $BASH_SOURCE should be sourced, not executed. e.g:"
|
|
>&2 echo " source $BASH_SOURCE"
|
|
exit 1
|
|
fi
|
|
|
|
# IMPALA-10057: The fe/target/classes and fe/target/test-classes
|
|
# directory can get cleaned out / rebuilt as part of running the
|
|
# frontend tests. This uses jars to avoid issues related to that
|
|
# inconsistency.
|
|
CLASSPATH=\
|
|
"$IMPALA_HOME"/fe/src/test/resources:\
|
|
"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}.jar:\
|
|
"$IMPALA_HOME"/fe/target/dependency:\
|
|
"$IMPALA_HOME"/fe/target/impala-frontend-${IMPALA_VERSION}-tests.jar:
|
|
|
|
FE_CP_FILE="$IMPALA_HOME/fe/target/build-classpath.txt"
|
|
|
|
if [ ! -s "$FE_CP_FILE" ]; then
|
|
>&2 echo FE classpath file $FE_CP_FILE missing.
|
|
>&2 echo Build the front-end first.
|
|
return 1
|
|
fi
|
|
|
|
CLASSPATH=$(cat $FE_CP_FILE):"$CLASSPATH"
|
|
|
|
# Currently the Calcite planner is experimental and not included by default.
|
|
# If the USE_CALCITE_PLANNER is explicitly set, then the jar dependencies
|
|
# are added to the CLASSPATH
|
|
USE_CALCITE_PLANNER=${USE_CALCITE_PLANNER:-false}
|
|
if [ "true" = "$USE_CALCITE_PLANNER" ]; then
|
|
|
|
CALCITE_CP_FILE="$IMPALA_HOME/java/calcite-planner/target/calcite-build-classpath.txt"
|
|
if [ ! -s "$CALCITE_CP_FILE" ]; then
|
|
>&2 echo Calcite front-end classpath file $CALCITE_CP_FILE missing.
|
|
>&2 echo Build the Calcite front-end first.
|
|
return 1
|
|
fi
|
|
CLASSPATH="$CLASSPATH":$(cat $CALCITE_CP_FILE):\
|
|
"$IMPALA_HOME"/java/calcite-planner/target/calcite-planner-${IMPALA_VERSION}.jar:
|
|
fi
|
|
|
|
if [[ "${1:-notest}" = "test" ]]; then
|
|
FE_TEST_CP_FILE="$IMPALA_HOME/fe/target/test-classpath.txt"
|
|
CLASSPATH=$(cat $FE_TEST_CP_FILE):"$CLASSPATH"
|
|
fi
|
|
|
|
: ${CUSTOM_CLASSPATH=}
|
|
CLASSPATH="$CUSTOM_CLASSPATH:$CLASSPATH"
|
|
|
|
export CLASSPATH
|