diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx index 84140c9f..fe0b903e 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/dotnet.mdx @@ -73,7 +73,7 @@ On Amazon ECS, the agent injects the .NET instrumentation into the services you ## On Docker -When the agent runs in Docker mode, eBPF traces your .NET containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. +When the agent monitors Docker containers, whether in Docker mode or as a Linux service on a Docker host, eBPF traces your .NET containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On Windows, systemd, Kubernetes, or Amazon ECS, the agent injects the profiler for you instead. diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx index 14106f37..b570e612 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/java.mdx @@ -52,7 +52,7 @@ On Amazon ECS, the agent injects the OpenTelemetry Java agent into the services ## On Docker -When the agent runs in Docker mode, eBPF traces your Java containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. +When the agent monitors Docker containers, whether in Docker mode or as a Linux service on a Docker host, eBPF traces your Java containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add the OpenTelemetry Java agent to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the Java agent for you instead. diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx index e0140204..5ff74bc3 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/nodejs.mdx @@ -43,7 +43,7 @@ On Amazon ECS, the agent injects the Node.js instrumentation into the services y ## On Docker -When the agent runs in Docker mode, eBPF traces your Node.js containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. +When the agent monitors Docker containers, whether in Docker mode or as a Linux service on a Docker host, eBPF traces your Node.js containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the instrumentation for you instead. diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx index 93c1a2d1..e3bbb090 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/php.mdx @@ -19,7 +19,9 @@ When you instrument a PHP service, the agent adds a config file for the target's ## On Docker -In Docker mode, the agent instruments PHP containers automatically, in place, and matches the container's PHP build. It reloads the SAPI (Apache graceful or php-fpm) with no redeploy, so this is separate from the **Off / eBPF** toggle. Narrow or turn it off with the `KM_CONTAINER_INSTRUMENT_*` environment variables. The instrumented application sends traces to the agent through the container's host gateway. +A PHP container offers an **SDK** option in Discovered Services. Turn it on and the agent injects the tracer, matches it to the container's PHP build, and reloads the web server (Apache graceful or php-fpm) with no redeploy and no container restart. The instrumented application sends traces to the agent through the container's host gateway. + +This SDK option is on by default in Docker mode. On a Linux host agent it's opt-in: set `KM_CONTAINER_INSTRUMENT_ENABLED=true`. To instrument PHP containers automatically without picking each one, use `KM_CONTAINER_INSTRUMENT_ALL`, `KM_CONTAINER_INSTRUMENT_NAMES`, or `KM_CONTAINER_INSTRUMENT_IMAGES`. The built-in PHP server (`php -S`) and Alpine (musl) images aren't supported: the built-in server has no graceful reload, and the bundled tracer assets target glibc. For the details, including the PHP 7 build-time recipe, see [Docker platform notes](../../platform-notes/docker/). diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx index 7187cd1e..1f7c928f 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/python.mdx @@ -43,7 +43,7 @@ On Amazon ECS, the agent injects the Python instrumentation into the services yo ## On Docker -When the agent runs in Docker mode, eBPF traces your Python containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. +When the agent monitors Docker containers, whether in Docker mode or as a Linux service on a Docker host, eBPF traces your Python containers per service through the **Off / eBPF** toggle in Discovered Services, with no code change and no restart. The agent can't inject the in-process SDK into a running container, so SDK mode isn't offered for containers. For full SDK depth, add OpenTelemetry to your image yourself and point it at the KloudMate agent's OTLP endpoint. See [Docker platform notes](../../platform-notes/docker/) for the endpoint addresses. On systemd, Kubernetes, or Amazon ECS, the agent injects the instrumentation for you instead. diff --git a/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx b/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx index f10a2b23..5736728f 100644 --- a/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx +++ b/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx @@ -11,13 +11,14 @@ Host metrics and logs are what the agent collects automatically, with no setup. ## What flows automatically -What the agent collects depends on the deployment mode, because each platform exposes host telemetry differently. +What the agent collects depends on the deployment mode, because each platform exposes host telemetry differently. Container telemetry is the exception: on any host with a Docker socket, container metrics and logs are collected automatically, whether the agent runs as a Linux service or in Docker mode. - **Host metrics:** CPU, memory, disk, network, and process metrics. - **Logs:** none are collected from the host automatically. Add application log files with [log monitoring](../../log-monitoring/), or the systemd journal with a [custom config override](../../custom-config-override/). +- **Docker containers:** if the host runs Docker, per-container metrics and logs are collected automatically, the same as in Docker mode. See [Monitor Docker containers on the host](../../installation/linux-agent/#monitor-docker-containers-on-the-host). diff --git a/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx b/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx index b75589ed..9683d05f 100644 --- a/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx +++ b/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx @@ -24,6 +24,8 @@ When eBPF monitoring is active, the agent collects: Because eBPF observes traffic in the kernel, it can trace a service with no code change and no restart — including Go and other services no language agent can inject. eBPF RED metrics and trace spans are a per-service choice you turn on (the **eBPF** APM mode); the network topology map and database activity monitoring run automatically. When a service also runs application APM, its eBPF spans and the application spans share the same trace, so hops you haven't put on the SDK still appear in end-to-end traces. See the [Application APM overview](../auto-instrumentation/overview/). +On a host running Docker, the agent reads each container's name and listen ports from the Docker socket, so a container can be traced with eBPF the same way a host process can, whether the agent runs as a Linux service or in Docker mode. In **Discovered Services** a container offers **Off** or **eBPF**, because the agent can't attach an in-process SDK to a running container. PHP containers can also offer an **SDK** option that injects a tracer with no redeploy. See [Docker platform notes](../platform-notes/docker/). + ## Deployment modes and requirements eBPF applies on **Linux, Kubernetes, and Docker**. It does **not** apply on **Windows**, which has no eBPF equivalent; on Windows the agent provides a comparable monitoring through Event Tracing for Windows (ETW) instead. See [Windows platform notes](../platform-notes/windows/). diff --git a/src/content/docs/docs/kloudmate-agent/index.mdx b/src/content/docs/docs/kloudmate-agent/index.mdx index 53a62bbe..eb939d72 100644 --- a/src/content/docs/docs/kloudmate-agent/index.mdx +++ b/src/content/docs/docs/kloudmate-agent/index.mdx @@ -51,7 +51,7 @@ The agent runs across four deployment modes. Not every capability applies to eve | Agent health and self-logs | Yes | Yes | Yes | Yes | :::note -On Linux, eBPF monitoring needs kernel 4.14 or newer, and does more on 5.8 and above. On Docker, host and container metrics and logs are automatic; application tracing is opt-in per service (Off / eBPF). PHP containers are traced in place without a redeploy, and Java, Node.js, Python, and .NET use eBPF (no SDK injection on Docker). See [Docker platform notes](platform-notes/docker/). +On Linux, eBPF monitoring needs kernel 4.14 or newer, and does more on 5.8 and above. On any host with a Docker socket, container metrics and logs are collected automatically, whether the agent runs as a Linux service or in Docker mode. Container tracing is opt-in per service: **Off** or **eBPF**, plus an **SDK** option for PHP containers that traces without a redeploy. See [Docker platform notes](platform-notes/docker/). ::: ## Install the agent diff --git a/src/content/docs/docs/kloudmate-agent/installation/docker-agent.mdx b/src/content/docs/docs/kloudmate-agent/installation/docker-agent.mdx index 366104e0..0d0cde13 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/docker-agent.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/docker-agent.mdx @@ -83,7 +83,7 @@ For a hand-written `filelog` receiver in manual mode, see [Collect File Logs wit ## Coverage and configuration -In Docker mode, the agent collects host and container metrics and container and host logs, and, where the host kernel supports it, places your containers on the service map from [eBPF](../../ebpf-observability/) network flows — with no code changes. Application tracing is opt-in per service: set a container to **eBPF** for RED metrics and spans (no restart), or leave it **Off**. PHP containers are the exception — the agent instruments them in place, no redeploy. Java, Node.js, Python, and .NET use eBPF; SDK injection isn't available on Docker. See [Docker platform notes](../../platform-notes/docker/) for the per-runtime detail. +In Docker mode, the agent collects host and container metrics and container and host logs, and, where the host kernel supports it, places your containers on the service map from [eBPF](../../ebpf-observability/) network flows, with no code changes. Application tracing is opt-in per service: set a container to **eBPF** for RED metrics and spans (no restart), or leave it **Off**. Java, Node.js, Python, .NET, and Go containers can't take the in-process SDK, so they use eBPF. PHP containers are the exception: they also offer an **SDK** option that injects the tracer and reloads the web server, with no redeploy. See [Docker platform notes](../../platform-notes/docker/) for the per-runtime detail. New agents start in managed mode, so KloudMate generates the collector configuration for you. See the [configuration model](../../concepts/config-model/). diff --git a/src/content/docs/docs/kloudmate-agent/installation/linux-agent.mdx b/src/content/docs/docs/kloudmate-agent/installation/linux-agent.mdx index ea9e24dd..305e08b7 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/linux-agent.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/linux-agent.mdx @@ -25,6 +25,7 @@ Key capabilities: - Tracking performance trends over time - Detecting resource bottlenecks - Collecting and analyzing system and application logs +- Monitoring Docker containers on the host, when Docker is present - Visualizing data with dashboards - Creating alerts for critical conditions @@ -81,6 +82,16 @@ To collect your own application log **files**, use log monitoring. In the agent' For a hand-written `filelog` receiver in manual mode, see [Collect File Logs with KloudMate Agent](../../../logs/collect-file-logs/). +## Monitor Docker containers on the host + +If the host runs Docker, the agent finds the Docker socket and starts collecting **container metrics and container logs** on its own, the same telemetry a Docker-mode agent collects. This happens as soon as the agent starts, before and independent of application tracing. To turn it off, set `KM_DOCKER_STATS=false` for metrics or `KM_DOCKER_LOGS=false` for logs. + +Each container also shows up in **Discovered Services**, where you can turn on per-container tracing. Most containers offer **Off** or **eBPF**: eBPF captures RED metrics and trace spans in the kernel, with no code change and no restart. There's no in-process **SDK** option for them, because the agent can't change a running container's start-time environment. Host processes that aren't containerized still offer **Off**, **eBPF**, or **SDK**. + +PHP containers are the exception: they can take an **SDK** option that injects the tracer and reloads the web server (Apache or php-fpm), with no redeploy and no container restart. On a Linux host this is opt-in. Set `KM_CONTAINER_INSTRUMENT_ENABLED=true`, and PHP containers start offering **SDK** in Discovered Services. In Docker mode that option is on by default. + +For monitoring containers, a Linux service and a Docker-mode agent are equivalent. Choose between them by how you want to run the agent, not by what they can see. See [Docker mode vs the Linux agent](../../platform-notes/docker/#docker-mode-vs-the-linux-agent). + ## Configuration and next checks New agents start in **managed mode**, with a default of host metrics and eBPF monitoring. You review and change what is collected from the web interface, without editing files on the host. See the [configuration model](../../concepts/config-model/). diff --git a/src/content/docs/docs/kloudmate-agent/platform-notes/docker.mdx b/src/content/docs/docs/kloudmate-agent/platform-notes/docker.mdx index 952a678d..2b8c399b 100644 --- a/src/content/docs/docs/kloudmate-agent/platform-notes/docker.mdx +++ b/src/content/docs/docs/kloudmate-agent/platform-notes/docker.mdx @@ -15,35 +15,49 @@ Running in Docker mode, the agent covers your host and containers without touchi - **Infrastructure, metrics, and logs:** host and container metrics and container logs, from the containerized collector. See [Host metrics and logs](../../baseline/host-metrics-and-logs/). +Container monitoring isn't unique to Docker mode. A [Linux agent](../../installation/linux-agent/) on a Docker host collects the same per-container metrics and logs automatically, because it reads the host's Docker socket. The section below covers when to pick each. + +## Docker mode vs the Linux agent + +For monitoring containers, the two are equivalent. A Linux (systemd) agent on a Docker host and a containerized Docker-mode agent both collect container metrics and logs automatically, and both trace containers per service with eBPF (**Off** or **eBPF**). Pick between them by how you want to run the agent, not by what they can see: + +- **Host processes.** The Linux agent also instruments the host's own non-containerized processes with the OpenTelemetry SDK, through systemd. A Docker-mode agent is itself a container, so it can't do that, and the host's processes are traced with eBPF only. +- **PHP containers.** A PHP container can offer an **SDK** option that injects the tracer with no redeploy. That option is on by default in Docker mode. On the Linux agent it's opt-in: set `KM_CONTAINER_INSTRUMENT_ENABLED=true`. +- **How the agent runs.** Docker mode runs the agent as a container and needs the Docker socket mounted, plus a privileged container with host access for eBPF. The Linux agent runs as a systemd service on the host and already has that access. + ## Application traces -Application tracing — Rate, Errors, and Duration (RED) metrics and trace spans — is **not** automatic. The agent lists every host process and container it finds under **Discovered Services**, and for each one you choose how it is traced: +Application tracing (Rate, Errors, and Duration metrics and trace spans) is **not** automatic. The agent lists every host process and container it finds under **Discovered Services**, and for each one you choose how it's traced: -- **Off** — not traced. This is the default, so nothing is traced (and nothing costs) until you turn it on — which matters on a host running many containers. -- **eBPF** — RED metrics and trace spans, captured in the host kernel with no code change and no restart. +- **Off:** not traced. This is the default, so nothing is traced (and nothing costs) until you turn it on, which matters on a host running many containers. +- **eBPF:** RED metrics and trace spans, captured in the host kernel with no code change and no restart. +- **SDK:** offered for PHP containers. It injects the tracer and reloads the web server, so tracing starts with no redeploy and no container restart. See [Instrument PHP containers](#instrument-php-containers). -**SDK injection is not offered on Docker.** The in-process OpenTelemetry SDK — the deep-span option on a Linux host — reads its configuration from the process's start-time environment, which can't change without recreating the container, too invasive for the agent to do automatically. So a container's choice is **Off or eBPF**, the same as Go on a Linux host. For full SDK depth, add OpenTelemetry to the app yourself (below), or run it on a [Linux host](../../installation/linux-agent/) or [Amazon ECS](../../installation/ecs-agent/), where the agent injects it for you. +The in-process OpenTelemetry SDK can't be injected into a running container, because it reads its configuration from the process's start-time environment, and that can't change without recreating the container. So Java, Node.js, Python, .NET, and Go containers offer **Off** or **eBPF** only. For full SDK depth on those runtimes, add OpenTelemetry to the app yourself (below), or run it on a [Linux host](../../installation/linux-agent/) or [Amazon ECS](../../installation/ecs-agent/), where the agent injects it for you. | Runtime | In Discovered Services | Full SDK depth | |---|---|---| -| Java, Node.js, Python, .NET | **Off / eBPF** — RED + spans, opt-in, no restart | Not injectable on Docker; [add OpenTelemetry yourself](#send-full-opentelemetry-traces-from-your-containers), or run on Linux / ECS | -| Go | **Off / eBPF** | eBPF only — Go is a static binary with no runtime to attach | -| PHP 7 and 8 | Auto-injected in place — see below | Automatic, no redeploy | - -PHP is the exception: the agent can reload the SAPI without a restart, so it instruments PHP containers in place. That is controlled by environment variables, not the per-service toggle. +| Java, Node.js, Python, .NET | **Off / eBPF**: RED and spans, opt-in, no restart | Not injectable on Docker. [Add OpenTelemetry yourself](#send-full-opentelemetry-traces-from-your-containers), or run on Linux or ECS. | +| Go | **Off / eBPF** | eBPF only. Go is a static binary with no runtime to attach. | +| PHP 7 and 8 | **Off / eBPF / SDK**: SDK injects the tracer, no redeploy | Covered by the SDK option, no redeploy | ## Instrument PHP containers -In Docker mode, the agent instruments your PHP containers automatically and reloads the web server. +A PHP container offers an **SDK** option in **Discovered Services**. Turn it on for a container and the agent injects the tracer, matches it to the container's PHP build, and reloads the web server (an Apache graceful reload or a php-fpm reload). Tracing starts with no redeploy and no container restart. + +The SDK option appears whenever the PHP-container instrumentation path is enabled. It's on by default in Docker mode. On a [Linux host agent](../../installation/linux-agent/) it's opt-in: set `KM_CONTAINER_INSTRUMENT_ENABLED=true`. -**Narrow or turn it off** with environment variables on the agent: +For automated or immutable deployments, instrument PHP containers without picking each one: ```bash -# Limit instrumentation to specific containers (default: every PHP container) +# Instrument every PHP container, with no per-service selection +KM_CONTAINER_INSTRUMENT_ALL=true + +# Or restrict the automatic set to specific containers or images KM_CONTAINER_INSTRUMENT_NAMES=app1,app2 KM_CONTAINER_INSTRUMENT_IMAGES=wordpress,my-php-app -# Turn PHP container instrumentation off entirely +# Turn the PHP-container SDK path off entirely KM_CONTAINER_INSTRUMENT_ENABLED=false ``` @@ -54,9 +68,9 @@ KM_CONTAINER_INSTRUMENT_ENABLED=false The agent cannot instrument a container that serves requests from the **PHP built-in server** (`php -S`), because that server does not reload configuration without a full restart. The official Adminer image is one example. Alpine (musl) images are also a problem, because the bundled tracer assets target glibc. -### PHP containers are configured by environment, not the per-service toggle +### How PHP-container tracing is selected -PHP container instrumentation is controlled by the environment variables above, not the **Off / eBPF** toggle in **Discovered Services**. A PHP container can still appear in that list, but its in-place tracer — not the toggle — is what gives it full traces. Narrow or disable it with the variables above. +By default, a PHP container is traced when you set it to **SDK** in **Discovered Services**, the same per-service choice as any other service. The environment variables above override that for automated rollouts: `KM_CONTAINER_INSTRUMENT_ALL`, `KM_CONTAINER_INSTRUMENT_NAMES`, or `KM_CONTAINER_INSTRUMENT_IMAGES` instrument a fixed set with no UI step, and [autonomous mode](../../advanced-configuration/) instruments every PHP container. Setting `KM_CONTAINER_INSTRUMENT_ENABLED=false` turns the SDK path off, so PHP containers fall back to **Off** or **eBPF**. ## Send full OpenTelemetry traces from your containers diff --git a/src/content/docs/docs/kloudmate-agent/reference/agent-config.mdx b/src/content/docs/docs/kloudmate-agent/reference/agent-config.mdx index 9c8519ea..7c9b10d0 100644 --- a/src/content/docs/docs/kloudmate-agent/reference/agent-config.mdx +++ b/src/content/docs/docs/kloudmate-agent/reference/agent-config.mdx @@ -61,6 +61,17 @@ Add any of the settings below on their own line to change the agent's behavior, | `docker-endpoint` | `KM_DOCKER_ENDPOINT` | Docker default socket | The Docker API endpoint the agent uses in Docker mode. | | `fargate-mode` | `KM_FARGATE_MODE` | `false` | Runs the agent as a slim ECS Fargate collector sidecar, with no host receivers, eBPF, or discovery. The [ECS tooling](../../installation/ecs-agent/) sets this; do not set it by hand. | +## Container monitoring + +On any host with a Docker socket, the agent collects per-container metrics and logs automatically, whether it runs as a Linux service or in Docker mode. This is independent of application tracing (APM) and turns on when the agent starts. Container metrics and logs are both on by default; set `KM_DOCKER_STATS=false` or `KM_DOCKER_LOGS=false` to opt out. + +| Environment variable | Default | Description | +|---|---|---| +| `KM_DOCKER_STATS` | `true` | Per-container CPU, memory, network, and block-I/O metrics, read from the Docker socket. Collected when a Docker socket is present. Set `false` to turn container metrics off. | +| `KM_DOCKER_LOGS` | `true` | Per-container stdout and stderr, read from the container log files. Collected when a Docker socket is present. Set `false` to turn container logs off. | +| `KM_DOCKER_LOGS_PATH` | `/var/lib/docker/containers/*/*-json.log` | The container-log glob. Set it when Docker uses a non-default data-root. | +| `KM_CONTAINER_INSTRUMENT_ENABLED` | On in Docker mode; off on a host agent | Enable the PHP-container SDK path, so PHP containers offer an **SDK** option that injects the tracer and reloads the web server, with no redeploy. On by default in Docker mode. On a host (systemd) agent, set `true` to turn it on. To instrument PHP containers without the per-service UI selection, use `KM_CONTAINER_INSTRUMENT_ALL`, `KM_CONTAINER_INSTRUMENT_NAMES`, or `KM_CONTAINER_INSTRUMENT_IMAGES`. See [Docker platform notes](../../platform-notes/docker/). | + ## eBPF opt-out toggles eBPF monitoring has two independent parts, and each has its own opt-out toggle. Both are on by default. They are separate switches: turning one off leaves the other running, and there is no single eBPF on/off setting.