Henry Robinson 89a0beb56a IMPALA-449: Better cleanup after an INSERT fails ...

This patch goes some way to improving recovery after an INSERT
fails. Inserts now write intermediate results to
<table_dir>/.impala_insert_staging. After execution completes, either
successfully or not, the query-specific directory under that directory
is deleted.

This doesn't complete the job for better cleanup (although this goes as
far as IMPALA-449 suggests). Two things to do in the future:

* Have each backend delete its own staging files on error. The
  difficulty getting there now is that backends don't know if they are
  cancelled in error or because a LIMIT was reached.
* If the operation to move files to their final destinations should
  fail during FinalizeQuery(), the coordinator should perform
  compensation actions and delete the files that made it.

Note: We also considered a query-wide and impalad-wide option to change
the staging dir. There are advantages to this (all intermediate results
go to a known location which is easy to clean up on failure), but also
security and other operational concerns. Worth revisiting in the future.

Change-Id: Ia54cf36db6a382e359877f87d7d40aad7fdb77be
Reviewed-on: http://gerrit.ent.cloudera.com:8080/670
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins

2014-01-08 10:53:37 -08:00

2.4 KiB

Executable File

Raw Blame History

View Raw