mirror of
https://github.com/apache/impala.git
synced 2026-01-28 09:03:52 -05:00
5ced9160bd
This adds support to FileMetadataLoader to recursively list a directory and create file descriptors. The changes are as follows: * FileMetadataLoader can now take a 'recursive' argument to trigger the new behavior. All the non-test code paths still use non-recursive (i.e. this new feature isn't exposed for real tables as of yet). * FileSystemUtil has some functionality for recursive directory listing. There are a few notes there around unexpected optimizations for S3 vs HDFS. * Renamed the 'file_name' field to 'relative_path' for FileDescriptor and HDFS splits, since now the file descriptors may be more than a single path component. The new functionality is just unit tested at the moment. Later, this functionality will be tied into the actual table code paths to solve issues with Hive interop, along with end-to-end tests. Change-Id: I9b151d7abb8443c0d9de0a0d82a9f13e07ad5109 Reviewed-on: http://gerrit.cloudera.org:8080/12991 Tested-by: Todd Lipcon <todd@apache.org> Reviewed-by: Todd Lipcon <todd@apache.org>