diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml index 6f0d61dd0..732bc38a6 100644 --- a/docs/topics/impala_scalability.xml +++ b/docs/topics/impala_scalability.xml @@ -202,6 +202,23 @@ Memory Usage: Additional Notes + + Limiting file metadata fetched in Catalog requests (<keyword keyref="impala50_full"/> and + higher) + +

To prevent Catalog service (Catalogd) Out-of-Memory (OOM) errors when coordinator fetching metadata for + tables with millions of files, the new configuration flag + catalog_partial_fetch_max_files has been introduced.

+

This flag limits the maximum number of file descriptors returned in a single Catalog fetch + response. This response is for the GetPartialCatalogObject RPC, which is + used in local catalog mode. See

+

Default: 1,000,000 files

+

If a request exceeds this limit, Catalogd truncates the response at the partition level. + The Impala coordinator then automatically sends subsequent requests to fetch the remaining + metadata, and it detects any version changes to force a query replan, ensuring metadata + consistency.

+
+