jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-13222: Clean up .Trash and temp files at the end of S3 test runs

Browse Source 
Remove the .Trash directory for HDFS, and temporary files left in
/tmp and in /other from the S3 bucket used for an S3 test run.
Deletion happens using AWSCLI after the minicluster is shut down.

Files are deleted only from selected refixes (subdirectories) so that
the cleanup logic is safe to use for private buckets, or the regular
bucket for private-s3-parameterized runs, impala-test-uswest2-3 too,
where other files may exist besides the ones generated for a test run.

Tested by running an S3 build then checking the contents of the test
bucket.

Change-Id: I60a23394de8a67768a0b5b4c9c9576ee6a24348e
Reviewed-on: http://gerrit.cloudera.org:8080/21585
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Laszlo Gaal
2024-06-26 00:28:26 +02:00
committed by Impala Public Jenkins
parent f49110c26d
commit 874e4fa117
1 changed files with 8 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
bin/jenkins/release_cloud_resources.sh
View File

@@ -43,8 +43,12 @@ if [[ "${TARGET_FILESYSTEM}" == "s3" ]]; then
-region "${S3GUARD_DYNAMODB_REGION}"
echo "Done cleaning up s3guard"
fi
# Remove the test warehouse
echo "Removing test warehouse from s3://${S3_BUCKET}${TEST_WAREHOUSE_DIR} ..."
aws s3 rm --recursive --quiet s3://${S3_BUCKET}${TEST_WAREHOUSE_DIR}
echo "Done removing test warehouse"
echo "Removing temporary data files and test warehouse..."
# Add temp files and HDFS .Trash to the delete list
for del_dir in ${TEST_WAREHOUSE_DIR} /other /tmp /user/jenkins/.Trash ; do
echo "Removing s3://${S3_BUCKET}${del_dir}"
aws s3 rm --recursive --quiet s3://${S3_BUCKET}${del_dir}
done
echo "Done cleaning the test bucket s3://${S3_BUCKET}"
fi