IMPALA-14092 Part2: Support querying of paimon data table via JNI

This patch mainly implement the querying of paimon data table
through JNI based scanner.

Features implemented:
- support column pruning.
The partition pruning and predicate push down will be submitted
as the third part of the patch.

We implemented this by treating the paimon table as normal
unpartitioned table. When querying paimon table:
- PaimonScanNode will decide paimon splits need to be scanned,
  and then transfer splits to BE do the jni-based scan operation.

- We also collect the required columns that need to be scanned,
  and pass the columns to Scanner for column pruning. This is
  implemented by passing the field ids of the columns to BE,
  instead of column position to support schema evolution.

- In the original implementation, PaimonJniScanner will directly
  pass paimon row object to BE, and call corresponding paimon row
  field accessor, which is a java method to convert row fields to
  impala row batch tuples. We find it is slow due to overhead of
  JVM method calling.
  To minimize the overhead, we refashioned the implementation,
  the PaimonJniScanner will convert the paimon row batches to
  arrow recordbatch, which stores data in offheap region of
  impala JVM. And PaimonJniScanner will pass the arrow offheap
  record batch memory pointer to the BE backend.
  BE PaimonJniScanNode will directly read data from JVM offheap
  region, and convert the arrow record batch to impala row batch.

  The benchmark shows the later implementation is 2.x better
  than the original implementation.

  The lifecycle of arrow row batch is mainly like this:
  the arrow row batch is generated in FE,and passed to BE.
  After the record batch is imported to BE successfully,
  BE will be in charge of freeing the row batch.
  There are two free paths: the normal path, and the
  exception path. For the normal path, when the arrow batch
  is totally consumed by BE, BE will call jni to fetch the next arrow
  batch. For this case, the arrow batch is freed automatically.
  For the exceptional path, it happends when query  is cancelled, or memory
  failed to allocate. For these corner cases, arrow batch is freed in the
  method close if it is not totally consumed by BE.

Current supported impala data types for query includes:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE

TODO:
    - Patches pending submission:
        - Support tpcds/tpch data-loading
          for paimon data table.
        - Virtual Column query support for querying
          paimon data table.
        - Query support with time travel.
        - Query support for paimon meta tables.
    - WIP:
        - Snapshot incremental read.
        - Complex type query support.
        - Native paimon table scanner, instead of
          jni based.

Testing:
    - Create tests table in functional_schema_template.sql
    - Add TestPaimonScannerWithLimit in test_scanners.py
    - Add test_paimon_query in test_paimon.py.
    - Already passed the tpcds/tpch test for paimon table, due to the
      testing table data is currently generated by spark, and it is
      not supported by impala now, we have to do this since hive
      doesn't support generating paimon table for dynamic-partitioned
      tables. we plan to submit a separate patch for tpcds/tpch data
      loading and associated tpcds/tpch query tests.
    - JVM Offheap memory leak tests, have run looped tpch tests for
      1 day, no obvious offheap memory increase is observed,
      offheap memory usage is within 10M.

Change-Id: Ie679a89a8cc21d52b583422336b9f747bdf37384
Reviewed-on: http://gerrit.cloudera.org:8080/23613
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <boroknagyz@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>

tests/query_test/test_paimon.py

@@ -52,3 +52,7 @@ class TestCreatingPaimonTable(ImpalaTestSuite):
   def test_paimon_negative(self, vector, unique_database):
     self.run_test_case('QueryTest/paimon-negative',
                        vector, unique_database)
+
+  def test_paimon_query(self, vector, unique_database):
+      self.run_test_case('QueryTest/paimon-query',
+                       vector, unique_database)

tests/query_test/test_scanners.py

@@ -2014,3 +2014,38 @@ class TestSingleFileTable(ImpalaTestSuite):
     select_stmt = "select count(*) from {db}.{tbl}".format(**params)
     res = self.execute_query_expect_success(self.client, select_stmt, options)
     assert res.data[0].split("\t")[0] == '1'
+
+
+class TestPaimonScannerWithLimit(ImpalaTestSuite):
+  """Test paimon scanners with a simple limit clause. The limit clause triggers
+   cancellation in the scanner code paths."""
+
+  @classmethod
+  def add_test_dimensions(cls):
+    super(TestPaimonScannerWithLimit, cls).add_test_dimensions()
+    # Use a small batch size so changing the limit affects the timing of cancellation
+    cls.ImpalaTestMatrix.add_dimension(
+      create_exec_option_dimension(batch_sizes=[100]))
+    cls.ImpalaTestMatrix.add_constraint(
+      lambda v: v.get_value('table_format').file_format == 'parquet')
+
+  def test_limit(self, vector):
+    vector.get_value('exec_option')['abort_on_error'] = 1
+    self._test_limit(vector)
+    # IMPALA-3337: when continuing on error, the error log should not show errors
+    # (e.g. "Cancelled").
+    vector.get_value('exec_option')['abort_on_error'] = 0
+    self._test_limit(vector)
+
+  def _test_limit(self, vector):
+    iterations = 50
+    query_template = "select * from functional_parquet.alltypes_paimon limit %s"
+    for i in range(1, iterations):
+      # Vary the limit to vary the timing of cancellation
+      limit = (i * 100) % 1001 + 1
+      query = query_template % limit
+      result = self.execute_query(query, vector.get_value('exec_option'),
+                                  table_format=vector.get_value('table_format'))
+      assert len(result.data) == limit
+      # IMPALA-3337: The error log should be empty.
+      assert not result.log