mirror of
https://github.com/apache/impala.git
synced 2025-12-26 14:02:53 -05:00
d9986fd3a2
Lately we have seen several failures during S3 builds that relied on AWS EC2 IAM instances roles for S3 bucket access credentials. The failure mode was a spurious failure for the s3a IAM Instance Credential Provider to actually provide the credentials. This patch is an attempt to extract more debugging information from such a failure: according to Hadoop-AWS developers, "Unable to load credentials from system settings..." is a generic error message from the Hadoop credential providers when they operate in a chained fashion. This happens when there is no explicit credetial provider specified in core-site.xml, and the credential providers are tried in sequence. The patch specifies the Hadoop s3a IAM Instance Credential Provider when all of the following conditions are true: - the default file system is S3, - the minicluster is running on an AWS EC2 VM, - AWS credentials are provided by an IAM instance role attached to the VM. The conditions are detected by following the rule matrix set up in bin/impala-config.sh; the same rule set is evaluated in core-site.xml.py, the Python script that generates the working copy of core-site.xml. The patch also XFAILs test_keys_do_not_work() in test_s3_access.py in custom cluster tests, because the test assumes the default chain of Hadoop AWS credential providers being available. This patch restricts the credential provider set to the IAM Instance Credential Provider only, which breaks the test mechanism. The test will be reinstated when a suitable workaround mechanism is found, but at this stage the extended diagnostic provided by this patch for investigating the credential provider flakiness is considered more valuable. Tests: - private build on S3 for the positive case - regression tested on a HDFS private build - in both cases the generated core-site.xml file was inspected during the test run. Change-Id: Ia8c09f8d042a69c5d3227398c720ea38e1c7e12f Reviewed-on: http://gerrit.cloudera.org:8080/21510 Reviewed-by: Michael Smith <michael.smith@cloudera.com> Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>