From b1759afef5b695511f657761b4a8a2d6ba73270d Mon Sep 17 00:00:00 2001 From: "Diego Nieto (lesandie)" Date: Mon, 7 Sep 2026 16:43:10 +0200 Subject: [PATCH] async inserts refreshed --- .../async-inserts.md | 221 +++++++++--------- 1 file changed, 113 insertions(+), 108 deletions(-) diff --git a/content/en/altinity-kb-queries-and-syntax/async-inserts.md b/content/en/altinity-kb-queries-and-syntax/async-inserts.md index 2bb0486557..8f32b5aa05 100644 --- a/content/en/altinity-kb-queries-and-syntax/async-inserts.md +++ b/content/en/altinity-kb-queries-and-syntax/async-inserts.md @@ -2,156 +2,161 @@ title: "Async INSERTs" linkTitle: "Async INSERTs" description: > - Comprehensive guide to ClickHouse Async INSERTs - configuration, best practices, and monitoring + Configure, scope, and monitor asynchronous inserts in ClickHouse. --- ## Overview -Async INSERTs is a ClickHouse® feature that enables automatic server-side batching of data. While we generally recommend batching at the application/ingestor level for better control and decoupling, async inserts are valuable when you have hundreds or thousands of clients performing small inserts and client-side batching is not feasible. +Async INSERTs let ClickHouse® batch many small inserts in server memory before writing them to storage. Use them when clients cannot form sufficiently large batches themselves. Client-side batching remains preferable when you control the writers because it keeps buffering outside ClickHouse. -**Key Documentation:** [Official Async Inserts Documentation](https://clickhouse.com/docs/en/optimize/asynchronous-inserts) +Starting with the 26.2.4 release line, ClickHouse [enables async inserts by default](https://github.com/ClickHouse/ClickHouse/pull/97590). Earlier releases default to synchronous inserts. Set it explicitly in an ingestion user's profile when you need stable behavior across versions: + +```sql +ALTER USER ingest SETTINGS async_insert = 1, wait_for_async_insert = 1; +``` + +Keep `wait_for_async_insert = 1` for production workloads. ClickHouse then acknowledges an insert only after the batch reaches storage and returns flush errors to the client. With `wait_for_async_insert = 0` (fire-and-forget), the client receives an acknowledgment while data is still buffered and cannot observe later failures. ## How Async Inserts Work -When `async_insert=1` is enabled, ClickHouse buffers incoming inserts and flushes them to disk when one of these conditions is met: -1. Buffer reaches specified size (`async_insert_max_data_size`) -2. Time threshold elapses (`async_insert_busy_timeout_ms`) -3. Maximum number of queries accumulate (`async_insert_max_query_number`) +ClickHouse keeps separate buffers for different target tables, insert shapes, formats, and settings. Each server node has its own buffers. A buffer flushes when the first applicable condition is met: -## Critical Configuration Settings +- buffered data reaches `async_insert_max_data_size` +- the busy timeout expires `async_insert_busy_timeout_ms` +- buffered queries reach `async_insert_max_query_number` while insert deduplication is enabled. + +Async inserts apply to `INSERT ... VALUES` and inserts with inline formats over HTTP or the native protocol. `INSERT ... SELECT` always runs synchronously. + +### Adaptive timeout + +Since ClickHouse 24.2, [adaptive busy timeouts are enabled by default](https://github.com/ClickHouse/ClickHouse/pull/58486). Frequent inserts increase the timeout from `async_insert_busy_timeout_min_ms` toward `async_insert_busy_timeout_max_ms`, allowing larger batches. Sparse inserts decrease it toward the minimum to reduce latency. + +Keep the adaptive mechanism enabled unless you need a fixed timeout for a tested workload. Tune the minimum for latency and the maximum for batching. Change `async_insert_max_data_size` when batch size, rather than time, should drive flushing. Leave the polling interval and timeout increase/decrease rates at their defaults unless measurements show a specific problem. -### Core Settings +### Fixed timeout mode + +Set `async_insert_use_adaptive_busy_timeout = 0` to use a fixed busy timeout. In this mode, ClickHouse uses `async_insert_busy_timeout_max_ms`. + +`async_insert_busy_timeout_ms` is an alias for `async_insert_busy_timeout_max_ms`, not a separate setting. You can use either name, but use `async_insert_busy_timeout_max_ms` in new +configurations. ```sql --- Enable async inserts (0=disabled, 1=enabled) -SET async_insert = 1; - --- Wait behavior (STRONGLY RECOMMENDED: use 1) --- 0 = fire-and-forget mode (risky - no error feedback) --- 1 = wait for data to be written to storage -SET wait_for_async_insert = 1; - --- Buffer flush conditions -SET async_insert_max_data_size = 1000000; -- 1MB default -SET async_insert_busy_timeout_ms = 1000; -- 1 second -SET async_insert_max_query_number = 100; -- max queries before flush +SET async_insert_use_adaptive_busy_timeout = 0; +SET async_insert_busy_timeout_max_ms = 1000; ``` -### Adaptive Timeout (Since 24.3) +When adaptive timeouts are disabled, `async_insert_busy_timeout_min_ms`, `async_insert_busy_timeout_increase_rate`, and `async_insert_busy_timeout_decrease_rate` do not affect the flush timeout. + +## Critical Configuration Settings + +| Setting | Scope | Guidance | +|---|---|---| +| `async_insert` | Query/profile | Enables buffering. The default changed from `0` to `1` in the 26.2.4 release line. | +| `wait_for_async_insert` | Query/profile | Keep at `1` so clients receive flush errors and acknowledgment means the data reached storage. | +| `async_insert_max_data_size` | Query/profile | Flushes when a buffer reaches this size. Defaults vary by release and ClickHouse Cloud. | +| `async_insert_busy_timeout_min_ms` / `max_ms` | Query/profile | Bound the adaptive wait. Use the minimum for the latency target and the maximum for the batching window. | +| `async_insert_max_query_number` | Query/profile | Flushes after this many queries when deduplication is enabled. | +| `async_insert_threads` | Server | Limits background parsing and insert threads. It is a [server setting since 23.7](https://github.com/ClickHouse/ClickHouse/pull/49160) and requires a restart. The current default is `16`; `0` disables the async queue. | + +Check effective values in `system.settings`, `system.server_settings`, and `system.merge_tree_settings` instead of assuming defaults match another deployment. + +## Scoped Async Inserts + +Using user profiles or per-query `SETTINGS` for ordinary writers. A [MergeTree-level async insert setting](https://github.com/ClickHouse/ClickHouse/pull/49122), is available since 23.5, can force-enable async inserts for one table: ```sql --- Adaptive timeout automatically adjusts flush timing based on server load --- Default: 1 (enabled) - OVERRIDES manual timeout settings --- Set to 0 for deterministic behavior with manual settings -SET async_insert_use_adaptive_busy_timeout = 0; +ALTER TABLE db.events MODIFY SETTING async_insert = 1; ``` -## Important Behavioral Notes +The table value is combined with the query/profile value using logical OR. Therefore, table-level `async_insert = 0` does **not** override an inherited `async_insert = 1`. -### What Works and What Doesn't +### Disable async inserts for one pipeline -✅ **Works with Async Inserts:** -- Direct INSERT with VALUES -- INSERT with FORMAT (JSONEachRow, CSV, etc.) -- Native protocol inserts (since 22.x) +A materialized view that writes through a `Distributed` table with `distributed_foreground_insert = 1` enabled can create a secondary regular INSERT on a remote shard. If that query inherits `async_insert = 1` (enabled by default) and batches little data, each pipeline execution can wait for the busy timeout. Prefer writing the view to the local table when that preserves the required sharding. Otherwise, use a dedicated definer profile to force disable asynchronous inserts for the pipeline. -❌ **Does NOT Work:** -- `INSERT .. SELECT` statements - Other strategies are needed for managing performance and load. Do not use `async_insert`. +{{% alert title="Security" color="warning" %}} +`SQL SECURITY DEFINER` runs the materialized view with the definer's identity. Use a dedicated account and grant only the required source `SELECT` and target `INSERT` privileges. Apply the user, profile, and view metadata on every relevant node unless your access storage and database replicate them. +{{% /alert %}} -### Data Safety Considerations +```sql +CREATE SETTINGS PROFILE mv_sync_insert +SETTINGS async_insert = 0 CONST; -**ALWAYS use `wait_for_async_insert = 1` in production!** +CREATE USER mv_sync_definer +IDENTIFIED WITH no_password +SETTINGS PROFILE mv_sync_insert; -Risks with `wait_for_async_insert = 0`: -- **Silent data loss** on errors (read-only table, disk full, too many parts) -- Data loss on sudden restart (no fsync by default) -- Data not immediately queryable after acknowledgment -- No error feedback to client +GRANT SELECT ON db.source TO mv_sync_definer; +GRANT INSERT ON db.destination TO mv_sync_definer; -### Deduplication Behavior +ALTER TABLE db.events_mv +MODIFY SQL SECURITY DEFINER DEFINER = mv_sync_definer; +``` -- **Sync inserts:** Automatic deduplication enabled by default -- **Async inserts:** Deduplication disabled by default -- Enable with `async_insert_deduplicate = 1` (since 22.x) -- **Warning:** Don't use with `deduplicate_blocks_in_dependent_materialized_views = 1` +The `CONST` constraint prevents an inherited value from replacing the profile value. This method does not work if the destination MergeTree table has `async_insert = 1`, because the table setting force-enables async mode. -# features / improvements +## Observability -* Async insert dedup: Support block deduplication for asynchronous inserts. Before this change, async inserts did not support deduplication, because multiple small inserts coexisted in one inserted batch: - - [#38075](https://github.com/ClickHouse/ClickHouse/issues/38075) - - [#43304](https://github.com/ClickHouse/ClickHouse/pull/43304) -* Added system table `asynchronous_insert_log`. It contains information about asynchronous inserts (including results of queries in fire-and-forget mode. (with wait_for_async_insert=0)) for better introspection [#42040](https://github.com/ClickHouse/ClickHouse/pull/42040) -* Support async inserts in **clickhouse-client** for queries with inlined data **(Native protocol)**: - - [#34267](https://github.com/ClickHouse/ClickHouse/pull/34267) - - [#54098](https://github.com/ClickHouse/ClickHouse/issues/54098) - - [#54381](https://github.com/ClickHouse/ClickHouse/issues/54381) -* Async insert backpressure [#4762](https://github.com/ClickHouse/ClickHouse/issues/47623) -* Limit the deduplication overhead when using `async_insert_deduplicate` [#46549](https://github.com/ClickHouse/ClickHouse/pull/46549) -* `SYSTEM FLUSH ASYNC INSERTS` [#49160](https://github.com/ClickHouse/ClickHouse/pull/49160) -* Adjustable asynchronous insert timeouts [#58486](https://github.com/ClickHouse/ClickHouse/pull/58486) +Use `system.asynchronous_inserts` to inspect buffers that have not flushed. Use `system.asynchronous_insert_log` to inspect completed flushes, failures, and time spent waiting. +To identify available async-insert metrics in `system.metric_log`, run: -## bugfixes +```bash +clickhouse-client --connection localhost --query " +SELECT name +FROM system.columns +WHERE table = 'metric_log' + AND (name ILIKE '%asyncinsert%' OR name ILIKE '%asynchronousinsert%') +ORDER BY name +" +``` -- Fixed bug which could lead to deadlock while using asynchronous inserts [#43233](https://github.com/ClickHouse/ClickHouse/pull/43233). -- Fix crash when async inserts with deduplication are used for ReplicatedMergeTree tables using a nondefault merging algorithm [#51676](https://github.com/ClickHouse/ClickHouse/pull/51676) -- Async inserts not working with log_comment setting [48430](https://github.com/ClickHouse/ClickHouse/issues/48430) -- Fix misbehaviour with async inserts with deduplication [#50663](https://github.com/ClickHouse/ClickHouse/pull/50663) -- Reject Insert if `async_insert=1` and `deduplicate_blocks_in_dependent_materialized_views=1`[#60888](https://github.com/ClickHouse/ClickHouse/pull/60888) -- Disable `async_insert_use_adaptive_busy_timeout` correctly with compatibility settings [#61486](https://github.com/ClickHouse/ClickHouse/pull/61468) +Check the current pool, queue, and pending-buffer metrics with: +```bash +clickhouse-client --connection localhost --query " +SELECT metric, value, description +FROM system.metrics +WHERE metric ILIKE '%asyncinsert%' + OR metric ILIKE '%asynchronousinsert%' +ORDER BY metric; +" +``` -## observability / introspection +`AsynchronousInsertThreads` is the pool's current size; `AsynchronousInsertThreadsActive` is the number running work; and `AsynchronousInsertThreadsScheduled` counts queued or active jobs. `AsynchronousInsertQueueSize`, `AsynchronousInsertQueueBytes`, and `PendingAsyncInsert` show pending work. Increase `async_insert_threads` only when the pool reaches its configured maximum, active threads stay near it, and scheduled work remains above it during normal load. Confirm CPU and storage headroom first: more insert threads can increase contention. -In 22.x versions, it is not possible to relate `part_log/query_id` column with `asynchronous_insert_log/query_id` column. We need to use `query_log/query_id`: +## Version and Reliability Notes -`asynchronous_insert_log` shows up the `query_id` and `flush_query_id` of each async insert. The `query_id` from `asynchronous_insert_log` shows up in the `system.query_log` as `type = 'QueryStart'` but the same `query_id` does not show up in the `query_id` column of the `system.part_log`. Because the `query_id` column in the `part_log` is the identifier of the INSERT query that created a data part, and it seems it is for sync INSERTS but not for async inserts. +ClickHouse 24.2 introduced adaptive busy timeouts and [SQL security definers for materialized views](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelogs/archive/v24.2.1.2248-stable.md). -So in `asynchronous_inserts` table you can check the current batch that still has not been flushed. In the `asynchronous_insert_log` you can find a log of all the flushed async inserts. +### Deduplication changes in 26.2 -This has been improved in **ClickHouse 23.7** Flush queries for async inserts (the queries that do the final push of data) are now logged in the `system.query_log` where they appear as `query_kind = 'AsyncInsertFlush'` [#51160](https://github.com/ClickHouse/ClickHouse/pull/51160) +ClickHouse 26.2 [unified synchronous and asynchronous insert deduplication](https://clickhouse.com/docs/concepts/features/operations/insert/asyncinserts#deduplication-and-reliability) under `deduplicate_insert`. It defaults to `enable` for Replicated* tables that retain a deduplication log, and supersedes `insert_deduplicate` and `async_insert_deduplicate`. To preserve the earlier per-insert behavior, set `deduplicate_insert = 'backward_compatible_choice'`. The legacy settings then select deduplication for their respective insert type. +Deduplication for dependent materialized views also defaults to enabled in 26.2. Async inserts cannot deduplicate a view that produces more than one output block per input block. Use synchronous inserts for that view, or set `deduplicate_blocks_in_dependent_materialized_views = 0`; the latter allows retries to duplicate rows in the view target. -## Versions +### Deduplication-hash migration -- **23.8** is a good version to start using async inserts because of the improvements and bugfixes. -- **24.3** the new adaptive timeout mechanism has been added so ClickHouse will throttle the inserts based on the server load.[#58486](https://github.com/ClickHouse/ClickHouse/pull/58486) This new feature is enabled by default and will OVERRRIDE current async insert settings, so better to disable it if your async insert settings are working. Here's how to do it in a clickhouse-client session: `SET async_insert_use_adaptive_busy_timeout = 0;` You can also add it as a setting on the INSERT or as a profile setting. +[`insert_deduplication_version`](https://github.com/ClickHouse/ClickHouse/pull/95409) is a server setting that migrates old, separate synchronous and asynchronous deduplication hashes to one unified hash. It affects async inserts whenever insert deduplication is enabled. +- `old_separate_hashes` uses the historical, different hashes. +- `compatible_double_hashes` writes both the old and unified hashes for every deduplicated block. This is the default in 26.2 through 26.5 and can increase Keeper work and deduplication-log entries on Replicated* tables. +- `new_unified_hash` writes only the unified hash. It becomes the default in [26.6](https://github.com/ClickHouse/ClickHouse/commit/4ebffc68f7e5468796ee5d323789144b6fb5d1ea). From [26.7](https://github.com/ClickHouse/ClickHouse/commit/500c083e4c4), ClickHouse accepts only this value. -## Metrics +Check the effective value before and after an upgrade: ```sql -SELECT name -FROM system.columns -WHERE (`table` = 'metric_log') AND ((name ILIKE '%asyncinsert%') OR (name ILIKE '%asynchronousinsert%')) - -┌─name─────────────────────────────────────────────┐ -│ ProfileEvent_AsyncInsertQuery │ -│ ProfileEvent_AsyncInsertBytes │ -│ ProfileEvent_AsyncInsertRows │ -│ ProfileEvent_AsyncInsertCacheHits │ -│ ProfileEvent_FailedAsyncInsertQuery │ -│ ProfileEvent_DistributedAsyncInsertionFailures │ -│ CurrentMetric_AsynchronousInsertThreads │ -│ CurrentMetric_AsynchronousInsertThreadsActive │ -│ CurrentMetric_AsynchronousInsertThreadsScheduled │ -│ CurrentMetric_AsynchronousInsertQueueSize │ -│ CurrentMetric_AsynchronousInsertQueueBytes │ -│ CurrentMetric_PendingAsyncInsert │ -│ CurrentMetric_AsyncInsertCacheSize │ -└──────────────────────────────────────────────────┘ - -SELECT * -FROM system.metrics -WHERE (metric ILIKE '%asyncinsert%') OR (metric ILIKE '%asynchronousinsert%') - -┌─metric─────────────────────────────┬─value─┬─description─────────────────────────────────────────────────────────────┐ -│ AsynchronousInsertThreads │ 1 │ Number of threads in the AsynchronousInsert thread pool. │ -│ AsynchronousInsertThreadsActive │ 0 │ Number of threads in the AsynchronousInsert thread pool running a task. │ -│ AsynchronousInsertThreadsScheduled │ 0 │ Number of queued or active jobs in the AsynchronousInsert thread pool. │ -│ AsynchronousInsertQueueSize │ 1 │ Number of pending tasks in the AsynchronousInsert queue. │ -│ AsynchronousInsertQueueBytes │ 680 │ Number of pending bytes in the AsynchronousInsert queue. │ -│ PendingAsyncInsert │ 7 │ Number of asynchronous inserts that are waiting for flush. │ -│ AsyncInsertCacheSize │ 0 │ Number of async insert hash id in cache │ -└────────────────────────────────────┴───────┴─────────────────────────────────────────────────────────────────────────┘ +SELECT name, value, changed, description +FROM system.server_settings +WHERE name = 'insert_deduplication_version'; ``` + +Do not move directly from `old_separate_hashes` to `new_unified_hash`. First run `compatible_double_hashes` on every replica. For Replicated* tables, keep it for at least the largest effective `replicated_deduplication_window_seconds` (one hour by default). For non-replicated tables, run enough inserts to cover the count-based `non_replicated_deduplication_window`. Then change to `new_unified_hash`. This period lets ClickHouse record unified IDs for data that a client might still retry. + +Run `SYSTEM FLUSH ASYNC INSERT QUEUE` before planned maintenance to flush pending buffers. Graceful shutdown also flushes them by default. + +## Related resources + +- [Asynchronous inserts](https://clickhouse.com/docs/concepts/features/operations/insert/asyncinserts) +- [Async insert server settings](https://clickhouse.com/docs/reference/settings/server-settings/settings/async-insert) +- [SQL security for views](https://clickhouse.com/docs/reference/statements/create/view#sql_security)