mirror of
https://github.com/apache/impala.git
synced 2025-12-19 18:12:08 -05:00
ce35e81bca
The file descriptors in HdfsPartition are cached as byte arrays to keep the memory footprint low. They are transformed into actual FileDescriptor objects once queried. This patch changes IcebergContentFileStore to similarly use byte arrays as an internal representation for file descriptors. Note, file descriptors for Iceberg tables have 2 components: one is the same as in HdfsPartition and the other stores Iceberg specific file metadata in an additional byte array. Measurements and observations: - I have a test table that has 110k data files. For this table the JVM memory usage in the catalogd got reduced from 80MB to 65MB. - Both HdfsPartition.FileDescriptor and IcebergContentFileStore use flatbuffers and in turn byte arrays to represent file descriptors and these byte arrays are shared between these 2 places. As a result there is no redundancy in storing the file descriptors both for the Iceberg and the Hdfs table. - There is no measurable difference in planning times with this patch. Change-Id: I9d7794df999bdaf118158eace26cea610f911c0a Reviewed-on: http://gerrit.cloudera.org:8080/21869 Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-by: Gabor Kaszab <gaborkaszab@cloudera.com>