From a098965d85e61cc3a4991201060ab4e9814f7b9a Mon Sep 17 00:00:00 2001 From: spelluru <6722422+spelluru@users.noreply.github.com> Date: Tue, 15 Sep 2026 12:00:41 -0400 Subject: [PATCH] Document archiveBestEffort ingestion property for ZIP archives Add the archiveBestEffort ingestion property to the ingestion properties reference table, describe ZIP archive error-handling behavior in the supported formats article, and note the HasErrors caveat on .ingest into when the property is set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- data-explorer/kusto/includes/ingestion-properties.md | 6 +++++- data-explorer/kusto/ingestion-properties.md | 2 +- data-explorer/kusto/ingestion-supported-formats.md | 8 +++++++- .../management/data-ingestion/ingest-into-command.md | 5 ++++- 4 files changed, 17 insertions(+), 4 deletions(-) diff --git a/data-explorer/kusto/includes/ingestion-properties.md b/data-explorer/kusto/includes/ingestion-properties.md index 18af950c5c..ee91f9eb78 100644 --- a/data-explorer/kusto/includes/ingestion-properties.md +++ b/data-explorer/kusto/includes/ingestion-properties.md @@ -1,6 +1,6 @@ --- ms.topic: include -ms.date: 08/11/2024 +ms.date: 09/15/2026 --- ## Ingestion properties @@ -14,6 +14,7 @@ The following table lists and describes the supported properties, and provides e |--|--|--| | `ingestionMapping` | A string value that indicates how to map data from the source file to the actual columns in the table. Define the `format` value with the relevant mapping type. See [data mappings](../management/mappings.md). | `with (format="json", ingestionMapping = "[{\"column\":\"rownumber\", \"Properties\":{\"Path\":\"$.RowNumber\"}}, {\"column\":\"rowguid\", \"Properties\":{\"Path\":\"$.RowGuid\"}}]")`
(deprecated: `avroMapping`, `csvMapping`, `jsonMapping`) | | `ingestionMappingReference` | A string value that indicates how to map data from the source file to the actual columns in the table using a named mapping policy object. Define the `format` value with the relevant mapping type. See [data mappings](../management/mappings.md). | `with (format="csv", ingestionMappingReference = "Mapping1")`
(deprecated: `avroMappingReference`, `csvMappingReference`, `jsonMappingReference`) | +| `archiveBestEffort` | A Boolean value that, if set to `true`, indicates that ingestion from a ZIP archive should continue processing even if some entries in the archive are empty or faulty, instead of failing the whole ingestion. This property applies only to ZIP archives. By default, `false` is assumed. Ingestion succeeds if at least one entry is successfully processed and produces records. Ingestion fails if all entries in the archive are empty or faulty. | `with (archiveBestEffort=true)` | | `creationTime` | The datetime value (formatted as an ISO8601 string) to use at the creation time of the ingested data extents. If unspecified, the current value (`now()`) is used. Overriding the default is useful when ingesting older data, so that the retention policy is applied correctly. When specified, make sure the `Lookback` property in the target table's effective [Extents merge policy](../management/merge-policy.md) is aligned with the specified value. | `with (creationTime="2017-02-13")` | | `extend_schema` | A Boolean value that, if specified, instructs the command to extend the schema of the table (defaults to `false`). This option applies only to `.append` and `.set-or-append` commands. The only allowed schema extensions have more columns added to the table at the end. | If the original table schema is `(a:string, b:int)`, a valid schema extension would be `(a:string, b:int, c:datetime, d:string)`, but `(a:string, c:datetime)` wouldn't be valid | | `folder` | For [ingest-from-query](../management/data-ingestion/ingest-from-query.md) commands, the folder to assign to the table. If the table already exists, this property overrides the table's folder. | `with (folder="Tables/Temporary")` | @@ -26,3 +27,6 @@ The following table lists and describes the supported properties, and provides e | `TreatGzAsUncompressed` | A Boolean value that, if set to `true`, indicates that files with the extension `.gz` are not compressed. This flag is sometimes needed when ingesting from Amazon AWS S3. | `with (treatGzAsUncompressed=true)` | | `validationPolicy` | A JSON string that indicates which validations to run during ingestion of data represented using CSV format. See [Data ingestion](/azure/data-explorer/ingest-data-overview) for an explanation of the different options. | `with (validationPolicy='{"ValidationOptions":1, "ValidationImplications":1}')` (this is the default policy) | | `zipPattern` | Use this property when ingesting data from storage that has a ZIP archive. This is a string value indicating the regular expression to use when selecting which files in the ZIP archive to ingest. All other files in the archive are ignored. | `with (zipPattern="*.csv")` | + +> [!IMPORTANT] +> Success reported for a ZIP archive ingested with `archiveBestEffort=true` doesn't guarantee that every file in the archive was ingested. Because per-entry failures are suppressed, use this property only when incomplete ingestion of the archive's contents is acceptable. diff --git a/data-explorer/kusto/ingestion-properties.md b/data-explorer/kusto/ingestion-properties.md index b2d9fdcca5..1474755bc9 100644 --- a/data-explorer/kusto/ingestion-properties.md +++ b/data-explorer/kusto/ingestion-properties.md @@ -3,7 +3,7 @@ title: Data ingestion properties description: Optimize data ingestion by configuring properties that align with your data formats. ms.reviewer: tzgitlin ms.topic: reference -ms.date: 09/25/2025 +ms.date: 09/15/2026 monikerRange: "azure-data-explorer || microsoft-fabric" --- # Data ingestion properties diff --git a/data-explorer/kusto/ingestion-supported-formats.md b/data-explorer/kusto/ingestion-supported-formats.md index a3da1c2195..7525280ea1 100644 --- a/data-explorer/kusto/ingestion-supported-formats.md +++ b/data-explorer/kusto/ingestion-supported-formats.md @@ -3,7 +3,7 @@ title: Data Ingestion - Supported Formats and Compression description: Explore the various data formats like CSV, JSON, Parquet, and more, supported for ingestion. Understand compression options and best practices for data preparation. ms.reviewer: tzgitlin ms.topic: reference -ms.date: 10/29/2025 +ms.date: 09/15/2026 monikerRange: "azure-data-explorer || microsoft-fabric" --- # Data formats supported for ingestion @@ -75,6 +75,12 @@ as an ingestion property because it can't be inferred. > * Don't confuse these with internal chunk-level compression codecs used by `Parquet`, `AVRO`, and `ORC` formats. The internal compression name is usually added before the file format extension (for example, `file1.gz.parquet`, `file1.snappy.avro`). > * The [Deflate64/Enhanced Deflate](https://en.wikipedia.org/wiki/Deflate#Deflate64/Enhanced_Deflate) zip compression method isn't supported. Windows built-in zip compressor can use this method on files larger than 2 GB. +### ZIP archive error handling + +By default, ingestion of a ZIP archive fails if any entry in the archive is empty or can't be parsed. To have ingestion continue past such entries instead of failing the whole archive, set the [`archiveBestEffort`](ingestion-properties.md#ingestion-properties) ingestion property to `true`. With this property set, the command skips empty or faulty entries and continues processing the remaining entries. Ingestion succeeds if at least one entry is successfully processed and produces records, and fails only if every entry in the archive is empty or faulty. + +This behavior applies only to entries within a ZIP archive that can be opened and read. It doesn't apply to files that aren't valid ZIP containers, or to compression formats other than ZIP. + ## Related content * [Supported data formats](ingestion-supported-formats.md) diff --git a/data-explorer/kusto/management/data-ingestion/ingest-into-command.md b/data-explorer/kusto/management/data-ingestion/ingest-into-command.md index 6b616e48e3..5cbc7dedd1 100644 --- a/data-explorer/kusto/management/data-ingestion/ingest-into-command.md +++ b/data-explorer/kusto/management/data-ingestion/ingest-into-command.md @@ -3,7 +3,7 @@ title: .ingest into command (pull data from storage) description: This article describes The .ingest into command (pull data from storage). ms.reviewer: alexans ms.topic: reference -ms.date: 11/25/2024 +ms.date: 09/15/2026 --- # .ingest into @@ -72,6 +72,9 @@ If no data shards were generated, a single record is returned with an empty (zer >[!NOTE] > This command doesn't modify the schema of the table being ingested into. If necessary, the data is "coerced" into this schema during ingestion, not the other way around (extra columns are ignored, and missing columns are treated as null values). +> [!NOTE] +> If the [`archiveBestEffort`](../../ingestion-properties.md#ingestion-properties) ingestion property is set to `true`, the command can return `HasErrors=false` even when some empty or faulty entries in the ZIP archive were skipped. A successful result doesn't guarantee that every file in the archive was ingested. + ## Examples ### Azure Blob Storage with shared access signature