Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
7659574
Add Experiments project access control docs
imran-at-datadog Sep 3, 2026
d6c3875
Clarify the SDK span limitation
imran-at-datadog Sep 4, 2026
a316793
Narrow the span coverage claim in the overview
imran-at-datadog Sep 4, 2026
80c1227
Correct the user_access_manage claim
imran-at-datadog Sep 4, 2026
94dd76b
Merge master into experiments-dac-docs
imran-at-datadog Sep 9, 2026
cb90d80
Reduce jargon on the Experiments access control page
imran-at-datadog Sep 9, 2026
add1934
Tighten the Experiments access control page
imran-at-datadog Sep 9, 2026
2b000ad
Shorten the SDK span limitation
imran-at-datadog Sep 9, 2026
e6cde5b
Merge branch 'master' into imran.hendley/experiments-dac-docs
imran-at-datadog Sep 9, 2026
04e5e20
Update Agent Observability telemetry details
CFLJacquet Sep 9, 2026
2f79cf1
Revise title and content for access control documentation
CFLJacquet Sep 9, 2026
db895da
Address review: de-emphasize ml_app, move page under Improve
imran-at-datadog Sep 10, 2026
7d2b5ea
Apply docs review: Restricted Dataset terminology, branch UI steps
imran-at-datadog Sep 10, 2026
580688e
Apply second docs review pass
imran-at-datadog Sep 10, 2026
ecdf767
Apply suggestion from @domalessi
domalessi Sep 10, 2026
27ae0cf
Apply suggestion from @domalessi
domalessi Sep 10, 2026
a754b7b
Apply batched suggestions from code review
domalessi Sep 10, 2026
dbcd103
Apply batched suggestions from code review
domalessi Sep 10, 2026
64e6829
Apply suggestion from @domalessi
domalessi Sep 10, 2026
dab816c
Apply suggestion from @domalessi
domalessi Sep 10, 2026
87c326a
Apply suggestion from @domalessi
domalessi Sep 11, 2026
ef95cd3
Apply suggestion from @domalessi
domalessi Sep 11, 2026
7f633cb
Apply suggestion from @domalessi
domalessi Sep 11, 2026
c61b8a3
Apply suggestion from @domalessi
domalessi Sep 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions hugo/config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5774,6 +5774,11 @@ menu:
parent: llm_obs_experiments
identifier: llm_obs_experiments_api
weight: 50305
- name: Data Access Control
url: llm_observability/improve/access_control
parent: llm_obs_improve
identifier: llm_obs_improve_access_control
weight: 504
- name: Configure
url: llm_observability/configure/
parent: llm_obs
Expand Down
8 changes: 6 additions & 2 deletions hugo/content/en/account_management/rbac/data_access.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ You may create a maximum of 100 Restricted Datasets under the Enterprise plan, a

### Supported telemetry types {#supported-telemetry}

- Agent Observability traces
- Agent Observability traces, Experiments, and Datasets
- APM traces
- Cloud costs
- Error Tracking issues
Expand Down Expand Up @@ -142,7 +142,10 @@ If you are concerned about unauthorized data access through monitors, Datadog re

### Agent Observability

* **Supported telemetry**: Agent Observability traces are supported. Experiment event data (spans and evaluation metrics) for experiments in a project is also restricted by `ml_app`-keyed Restricted Datasets. Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported.
* **Supported telemetry**:
* Traces, using any tag.
* Experiments and Datasets, restricted by project. An `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14].
* **Not supported**: Annotation queues and managed prompts.
* **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries.


Expand Down Expand Up @@ -270,3 +273,4 @@ When querying data through Datadog APIs with restrictions enabled, users without
[11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs
[12]: /dashboards/sharing/shared_dashboards/
[13]: /llm_observability/instrument/otel_instrumentation/
[14]: /llm_observability/improve/access_control/
3 changes: 3 additions & 0 deletions hugo/content/en/llm_observability/data_governance.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ Agent Observability allows you to restrict access to potentially sensitive data

Access controls in Agent Observability are built on Datadog's [Data Access Control][11] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific AI applications within your organization.

You can also restrict individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14].

## Redacting data with span processors

You can redact or modify sensitive data at the application level before it is sent to Datadog. Use span processors in the Agent Observability SDK to conditionally modify input and output data on spans, or prevent spans from being emitted entirely.
Expand Down Expand Up @@ -134,3 +136,4 @@ For the full list of available metrics, see [Agent Observability metrics][8].
[11]: /account_management/rbac/data_access
[12]: /llm_observability/instrument/sdk/#span-processing
[13]: /security/sensitive_data_scanner/
[14]: /llm_observability/improve/access_control/
4 changes: 4 additions & 0 deletions hugo/content/en/llm_observability/improve/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,7 @@ Iterate on your LLM application with datasets, experiments, and the playground.
{{< nextlink href="/llm_observability/improve/datasets" >}}Datasets{{< /nextlink >}}
{{< nextlink href="/llm_observability/improve/experiments" >}}Experiments{{< /nextlink >}}
{{< /whatsnext >}}

{{< whatsnext desc="Control who can see your data:" >}}
{{< nextlink href="/llm_observability/improve/access_control" >}}Data Access Control{{< /nextlink >}}
{{< /whatsnext >}}
109 changes: 109 additions & 0 deletions hugo/content/en/llm_observability/improve/access_control.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
title: Data Access Control in Agent Observability
description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles.
further_reading:
- link: "/account_management/rbac/data_access/"
tag: "Documentation"
text: "Data Access Control"
- link: "/llm_observability/data_governance/"
tag: "Documentation"
text: "Data Governance"
- link: "/account_management/rbac/permissions/#access-management"
tag: "Documentation"
text: "Access management permissions"
---

## Overview

Experiments projects can hold sensitive material, including dataset prompts and expected outputs, traces from experiment runs, and evaluation results. [Data Access Control][1] lets you restrict a project so that only the teams or roles you specify can see it.

When a project is restricted, users outside the teams or roles you granted access to cannot:

- See the project, or its experiments and datasets, in any list view or search result
- Read the project's dataset records, including inputs and expected outputs
- Read the evaluation metrics produced by the project's experiment runs
- Read the spans produced by those runs, except for experiments run through the SDK (see [Limitations](#limitations))
- Create, modify, or delete anything inside the project, even with an ID they obtained earlier

Users outside those teams or roles receive a *not found* response when they open a direct link to the project or to anything inside it.

Restrictions apply in the Datadog UI and in the API. Application keys are subject to the same restrictions as the user who owns them.

## Prerequisites

- Data Access Control is configured for your organization. See [Data Access Control][1].
- You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2].
- The project you want to restrict already exists in Experiments.

## Restrict a project in the UI

<div class="alert alert-info">Datadog is rolling out a redesigned access control page. Your organization has either the <strong>Data Access Controls</strong> page or the redesigned <strong>Access Control</strong> page. The link in step 1 takes you to whichever one you have, and both configure the same restriction.</div>

1. Navigate to [Organization Settings > Data Access Controls][3].
2. Create a restriction that covers a subset of data:
- On the Data Access Controls page, click **New Restricted Dataset**.
- On the Access Control page, click **New Policy > Sensitive Data Partition**.
3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`.
4. Add a filter on the **Agent Observability** product, then specify the project:
- On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group.

<div class="alert alert-warning"><ul><li>Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.</li><li>Projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time.</li></ul></div>

- On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id).
Comment thread
domalessi marked this conversation as resolved.

5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset.
6. Save the Restricted Dataset.

**Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key.

The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations).

## Restrict a project through the API

You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value:

```json
{
"data": {
"type": "dataset",
"attributes": {
"name": "Experiments - Fraud Detection",
"product_filters": [
{
"product": "ml_obs",
"filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"]
}
],
"principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"]
}
}
}
```

Comment thread
domalessi marked this conversation as resolved.
## Find a project's ID

The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects.

## Grant and revoke access

Grant access by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access.

Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would.

## Limitations

- **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset.
- **Spans and evaluation metrics that were not tagged with the project at ingestion are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged. Events that carry the tag are hidden as soon as the restriction is saved. List views, metadata, and dataset records are hidden regardless of when they were created.
- **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry.
- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles.

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /account_management/rbac/data_access/
[2]: /account_management/rbac/permissions/#access-management
[3]: https://app.datadoghq.com/organization-settings/data-access-controls/
[4]: /llm_observability/improve/experiments/api/
[5]: /api/latest/datasets/
[6]: /account_management/rbac/data_access/#strict-mode
Loading