From 7659574754bd04a9c9c4bd882a835f24a0e2cc31 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Thu, 3 Sep 2026 11:21:17 -0400 Subject: [PATCH 01/22] Add Experiments project access control docs Documents restricting an Agent Observability Experiments project with Data Access Control: what a restriction hides, how to author the policy on either the current or the redesigned access control page, the API equivalent, and the limitations (SDK-run spans keyed on the app's own ml_app, and event data ingested before the feature was available). Also corrects the Agent Observability entry on the Data Access Control page, which said only event data is restricted and that datasets are unsupported. Co-Authored-By: Claude Opus 5 (1M context) --- hugo/config/_default/menus/main.en.yaml | 5 + .../en/account_management/rbac/data_access.md | 3 +- .../data_privacy_security_and_rbac.md | 4 +- .../improve/experiments/access_control.md | 110 ++++++++++++++++++ 4 files changed, 120 insertions(+), 2 deletions(-) create mode 100644 hugo/content/en/llm_observability/improve/experiments/access_control.md diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 4f7486a03f5..01b99f3675f 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5764,6 +5764,11 @@ menu: parent: llm_obs_experiments identifier: llm_obs_experiments_api weight: 50305 + - name: Restrict Access to Projects + url: llm_observability/improve/experiments/access_control + parent: llm_obs_experiments + identifier: llm_obs_experiments_access_control + weight: 50306 - name: Configure url: llm_observability/configure/ parent: llm_obs diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index f2102448e4e..1287d89e7eb 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -142,7 +142,7 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability -* **Supported telemetry**: Agent Observability traces are supported. Experiment event data (spans and evaluation metrics) for experiments in a project is also restricted by `ml_app`-keyed Restricted Datasets. Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported. +* **Supported telemetry**: Agent Observability traces are supported. Experiments are also supported: a Restricted Dataset keyed on `ml_app` whose value is an Experiments project ID restricts that project's spans, evaluation metrics, datasets, and dataset records, and hides the project and its experiments from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. Annotation queues and managed prompts are not supported. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. @@ -270,3 +270,4 @@ When querying data through Datadog APIs with restrictions enabled, users without [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs [12]: /dashboards/sharing/shared_dashboards/ [13]: /llm_observability/instrument/otel_instrumentation/ +[14]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md index b7062d174c4..82a38d9089f 100644 --- a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md +++ b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md @@ -18,6 +18,8 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][1] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific LLM applications within your organization. +The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Restrict Access to Experiments Projects][4]. + ## Redacting data with span processors You can redact or modify sensitive data at the application level before it is sent to Datadog. Use span processors in the Agent Observability SDK to conditionally modify input and output data on spans, or prevent spans from being emitted entirely. @@ -42,4 +44,4 @@ By proactively scanning for sensitive data, Agent Observability ensures that con [1]: /account_management/rbac/data_access [2]: /llm_observability/instrument/sdk/#span-processing [3]: /security/sensitive_data_scanner/ - +[4]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md new file mode 100644 index 00000000000..b362ffd46fe --- /dev/null +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -0,0 +1,110 @@ +--- +title: Restrict Access to Experiments Projects +description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. +further_reading: +- link: "/account_management/rbac/data_access" + tag: "Documentation" + text: "Data Access Control" +- link: "/llm_observability/data_privacy_security_and_rbac" + tag: "Documentation" + text: "Data Privacy, Security, and RBAC" +- link: "/account_management/rbac/permissions/#access-management" + tag: "Documentation" + text: "Access management permissions" +--- + +## Overview + +Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. + +When a project is restricted, users outside the granted principals: + +- Do not see the project, or its experiments and datasets, in any list view or search result. +- Receive a *not found* response when they open a direct link to the project or to anything inside it. +- Cannot read the spans and evaluation metrics produced by the project's experiment runs. +- Cannot read the project's dataset records, including inputs and expected outputs. +- Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. + +Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. + +## Prerequisites + +- Data Access Control is configured for your organization. See [Data Access Control][1]. +- You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. +- The project you want to restrict already exists in Experiments. + +## Restrict a project + +Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: + +- An **ML app** value restricts the traces your instrumented application sends under that `ml_app` name. +- An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. + +A project is matched by its ID, never by its name. + +1. Navigate to [Organization Settings > Data Access Controls][3]. +2. Create a new policy that restricts a subset of data. +3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. +4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. Selecting a project stores its ID, which is what the restriction matches on. +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 principals can be attached to one policy. +6. Save the policy. + +The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are matched by a tag applied when they are ingested, so experiment runs from before your organization had this feature available are not covered; see [Limitations](#limitations). + +
Select the project from the list rather than typing its name. The value field also accepts free text, and a policy holding a project's name matches no Experiments data: the project stays visible to everyone while the policy looks like it is working. The same applies to a partially typed or misspelled name.
+ +Two things to expect while filling in the filter: + +- **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. +- **The key may be locked to `ml_app`.** Data Access Control allows one tag key per telemetry type, so if your organization already has an Agent Observability policy, new ones reuse the same key. + +
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value.
+ +### Find a project's ID + +If your organization's access control page shows raw values rather than project names, take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. + +## Restrict a project through the API + +You can also create the policy with the [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: + +```json +{ + "data": { + "type": "dataset", + "attributes": { + "name": "Experiments - Fraud Detection", + "product_filters": [ + { + "product": "ml_obs", + "filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"] + } + ], + "principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"] + } + } +} +``` + +## Grant and revoke access + +Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. + +Users with the `user_access_manage` permission are not subject to restrictions, so verify a restriction with an account that does not hold that permission. + +## Limitations + +- **Spans from an experiment run by the SDK are not restricted by project.** Datadog only fills in the project ID as the `ml_app` value when your code has not set one, and never overrides a value you set. Running an experiment through the SDK stamps your application's `ml_app` on every task span it produces, so those spans are matched by a policy on your application's `ml_app` value, not by one on the project. The project, its experiments, its datasets, and its dataset records are still hidden from the Experiments UI either way. To restrict the span contents as well, add a filter for your application's own `ml_app` value alongside the project. +- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. +- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /account_management/rbac/data_access/ +[2]: /account_management/rbac/permissions/#access-management +[3]: https://app.datadoghq.com/organization-settings/data-access-controls +[4]: /llm_observability/improve/experiments/api/ +[5]: /api/latest/datasets/ From d6c3875b541e2f2243ee59b0aa0e4a78cfdbd272 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 13:36:11 -0400 Subject: [PATCH 02/22] Clarify the SDK span limitation The previous wording explained the backfill mechanism (project ID is used as ml_app only when none is set), which reads as a lever the reader can pull. It is not: the SDK always sends an ml_app value, so the backfill never applies to SDK traffic. State the outcome and the remedy instead. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/llm_observability/improve/experiments/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index b362ffd46fe..6e2c9b73479 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -94,7 +94,7 @@ Users with the `user_access_manage` permission are not subject to restrictions, ## Limitations -- **Spans from an experiment run by the SDK are not restricted by project.** Datadog only fills in the project ID as the `ml_app` value when your code has not set one, and never overrides a value you set. Running an experiment through the SDK stamps your application's `ml_app` on every task span it produces, so those spans are matched by a policy on your application's `ml_app` value, not by one on the project. The project, its experiments, its datasets, and its dataset records are still hidden from the Experiments UI either way. To restrict the span contents as well, add a filter for your application's own `ml_app` value alongside the project. +- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. From a3167934db95127612d8aa7e58786bf8cf6b5a39 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 13:42:29 -0400 Subject: [PATCH 03/22] Narrow the span coverage claim in the overview The overview promised that a restricted user cannot read the spans or evaluation metrics of a restricted project's runs, and the limitations section then took half of that back. For an SDK-run experiment the two differ: eval metrics carry no ml_app and are backfilled to the project id, while task spans carry the application's ml_app and are not matched. Say so once, up front. Co-Authored-By: Claude Opus 5 (1M context) --- .../llm_observability/improve/experiments/access_control.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index 6e2c9b73479..e6f76e18596 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -21,8 +21,9 @@ When a project is restricted, users outside the granted principals: - Do not see the project, or its experiments and datasets, in any list view or search result. - Receive a *not found* response when they open a direct link to the project or to anything inside it. -- Cannot read the spans and evaluation metrics produced by the project's experiment runs. - Cannot read the project's dataset records, including inputs and expected outputs. +- Cannot read the evaluation metrics produced by the project's experiment runs. +- Cannot read the spans produced by those runs, unless the spans carry an `ml_app` value of their own. Experiments run through the SDK do produce such spans; see [Limitations](#limitations). - Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. @@ -94,7 +95,7 @@ Users with the `user_access_manage` permission are not subject to restrictions, ## Limitations -- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. +- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. From 80c1227abf13a65eb845e7b4c870b3c9eb92af90 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 14:24:35 -0400 Subject: [PATCH 04/22] Correct the user_access_manage claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The page said holders of user_access_manage are not subject to restrictions. They are: the permission gates authoring a policy, while the access decision is taken over the user's teams and roles with no permissions input. Verified on staging as a non-granted admin — the project is hidden and its dataset records 404. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/llm_observability/improve/experiments/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index e6f76e18596..568dcf60bd0 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -91,7 +91,7 @@ You can also create the policy with the [Datasets API][5]. The `ml_obs` product Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. -Users with the `user_access_manage` permission are not subject to restrictions, so verify a restriction with an account that does not hold that permission. +Access follows grant membership, not permission level. The `user_access_manage` permission lets you author and edit a policy; it does not exempt you from one. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. ## Limitations From cb90d803313353f399b1b2ab05d8230af9864533 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Wed, 9 Sep 2026 15:59:22 -0400 Subject: [PATCH 05/22] Reduce jargon on the Experiments access control page Replace "principals" with "teams and roles" in prose, keeping it only as the API field name. Replace the coined phrase "grant membership" with a plain statement that being an admin does not exempt you. State the span rule before its SDK exception so the overview list stays scannable, shorten the application-key sentence, say restrictions take effect immediately rather than on the next request, link Strict Mode to its section, and stop defining ml_app in terms of ml_app. Co-Authored-By: Claude Opus 5 (1M context) --- .../improve/experiments/access_control.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index 9e779fd23f9..e08a0cb109b 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -17,16 +17,16 @@ further_reading: Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. -When a project is restricted, users outside the granted principals: +When a project is restricted, users outside the teams and roles you granted access to: - Do not see the project, or its experiments and datasets, in any list view or search result. - Receive a *not found* response when they open a direct link to the project or to anything inside it. - Cannot read the project's dataset records, including inputs and expected outputs. - Cannot read the evaluation metrics produced by the project's experiment runs. -- Cannot read the spans produced by those runs, unless the spans carry an `ml_app` value of their own. Experiments run through the SDK do produce such spans; see [Limitations](#limitations). +- Cannot read the spans produced by those runs, with one exception for experiments run through the SDK. See [Limitations](#limitations). - Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. -Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. +Restrictions apply in the Datadog UI and in the API. Application keys are subject to the same restrictions as the user who owns them. ## Prerequisites @@ -38,7 +38,7 @@ Restrictions are enforced consistently in the Datadog UI and in the API, includi Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: -- An **ML app** value restricts the traces your instrumented application sends under that `ml_app` name. +- An **ML app** name restricts the traces that application sends to Agent Observability. - An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. A project is matched by its ID, never by its name. @@ -47,7 +47,7 @@ A project is matched by its ID, never by its name. 2. Create a new policy that restricts a subset of data. 3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. 4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. Selecting a project stores its ID, which is what the restriction matches on. -5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 principals can be attached to one policy. +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams and roles can be attached to one policy. 6. Save the policy. The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are matched by a tag applied when they are ingested, so experiment runs from before your organization had this feature available are not covered; see [Limitations](#limitations). @@ -59,7 +59,7 @@ Two things to expect while filling in the filter: - **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. - **The key may be locked to `ml_app`.** Data Access Control allows one tag key per telemetry type, so if your organization already has an Agent Observability policy, new ones reuse the same key. -
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value.
+
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value until then.
### Find a project's ID @@ -89,16 +89,16 @@ You can also create the policy with the [Datasets API][5]. The `ml_obs` product ## Grant and revoke access -Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. +Access is granted by editing the teams and roles on the policy. Removing a team or role takes effect immediately. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. -Access follows grant membership, not permission level. The `user_access_manage` permission lets you author and edit a policy; it does not exempt you from one. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. +Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit policies, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would. ## Limitations - **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. -- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. +- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams and roles. ## Further reading @@ -109,3 +109,4 @@ Access follows grant membership, not permission level. The `user_access_manage` [3]: https://app.datadoghq.com/organization-settings/data-access-controls [4]: /llm_observability/improve/experiments/api/ [5]: /api/latest/datasets/ +[6]: /account_management/rbac/data_access/#strict-mode From add19347d74d40e0b2fd18e4c82f62b196bca887 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Wed, 9 Sep 2026 16:17:11 -0400 Subject: [PATCH 06/22] Tighten the Experiments access control page Move the post-save effects paragraph below the form guidance so everything about filling in the filter sits together, and let the Limitations link carry the retroactive-coverage caveat rather than stating it twice. Drop two restatements of "a project is matched by its ID, not its name"; the warning already explains the failure mode. Standardize on "teams or roles", rename the UI section for parallelism with the API one, and even up trailing slashes on links. Co-Authored-By: Claude Opus 5 (1M context) --- .../improve/experiments/access_control.md | 24 +++++++++---------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index e08a0cb109b..95cb31799a2 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -2,10 +2,10 @@ title: Restrict Access to Experiments Projects description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. further_reading: -- link: "/account_management/rbac/data_access" +- link: "/account_management/rbac/data_access/" tag: "Documentation" text: "Data Access Control" -- link: "/llm_observability/data_governance" +- link: "/llm_observability/data_governance/" tag: "Documentation" text: "Data Governance" - link: "/account_management/rbac/permissions/#access-management" @@ -17,7 +17,7 @@ further_reading: Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. -When a project is restricted, users outside the teams and roles you granted access to: +When a project is restricted, users outside the teams or roles you granted access to: - Do not see the project, or its experiments and datasets, in any list view or search result. - Receive a *not found* response when they open a direct link to the project or to anything inside it. @@ -34,24 +34,20 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec - You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. - The project you want to restrict already exists in Experiments. -## Restrict a project +## Restrict a project in the UI Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: - An **ML app** name restricts the traces that application sends to Agent Observability. - An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. -A project is matched by its ID, never by its name. - 1. Navigate to [Organization Settings > Data Access Controls][3]. 2. Create a new policy that restricts a subset of data. 3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. -4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. Selecting a project stores its ID, which is what the restriction matches on. -5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams and roles can be attached to one policy. +4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one policy. 6. Save the policy. -The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are matched by a tag applied when they are ingested, so experiment runs from before your organization had this feature available are not covered; see [Limitations](#limitations). -
Select the project from the list rather than typing its name. The value field also accepts free text, and a policy holding a project's name matches no Experiments data: the project stays visible to everyone while the policy looks like it is working. The same applies to a partially typed or misspelled name.
Two things to expect while filling in the filter: @@ -61,6 +57,8 @@ Two things to expect while filling in the filter:
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value until then.
+The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). + ### Find a project's ID If your organization's access control page shows raw values rather than project names, take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. @@ -89,7 +87,7 @@ You can also create the policy with the [Datasets API][5]. The `ml_obs` product ## Grant and revoke access -Access is granted by editing the teams and roles on the policy. Removing a team or role takes effect immediately. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. +Access is granted by editing the teams or roles on the policy. Removing a team or role takes effect immediately. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit policies, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would. @@ -98,7 +96,7 @@ Being an admin does not exempt you from a restriction. The `user_access_manage` - **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. -- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams and roles. +- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. ## Further reading @@ -106,7 +104,7 @@ Being an admin does not exempt you from a restriction. The `user_access_manage` [1]: /account_management/rbac/data_access/ [2]: /account_management/rbac/permissions/#access-management -[3]: https://app.datadoghq.com/organization-settings/data-access-controls +[3]: https://app.datadoghq.com/organization-settings/data-access-controls/ [4]: /llm_observability/improve/experiments/api/ [5]: /api/latest/datasets/ [6]: /account_management/rbac/data_access/#strict-mode From 2b000ad919e4a6ccc19b696ae5a74487957ab54b Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Wed, 9 Sep 2026 16:21:13 -0400 Subject: [PATCH 07/22] Shorten the SDK span limitation Lead with what the restriction does hide before naming the exception, and drop the provenance of the SDK's ml_app value: it is implementation detail, and describing it as coming from configuration or the service name risks implying that different configuration avoids the gap. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/llm_observability/improve/experiments/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index 95cb31799a2..7f636497326 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -93,7 +93,7 @@ Being an admin does not exempt you from a restriction. The `user_access_manage` ## Limitations -- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. +- **Spans from an experiment run through the SDK carry your application's `ml_app`, not the project's ID.** A policy on the project does not match those spans, so it hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs recorded on its spans. To cover those as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. From 04e5e20b35ecf9db6600ee1d502d4d58f3aadae0 Mon Sep 17 00:00:00 2001 From: Charles Jacquet Date: Wed, 9 Sep 2026 23:38:38 +0200 Subject: [PATCH 08/22] Update Agent Observability telemetry details Clarified supported telemetry details for Agent Observability, including restrictions on Experiments and Datasets based on project ID. --- hugo/content/en/account_management/rbac/data_access.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index 1287d89e7eb..fc1000ee263 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -142,7 +142,10 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability -* **Supported telemetry**: Agent Observability traces are supported. Experiments are also supported: a Restricted Dataset keyed on `ml_app` whose value is an Experiments project ID restricts that project's spans, evaluation metrics, datasets, and dataset records, and hides the project and its experiments from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. Annotation queues and managed prompts are not supported. +* **Supported telemetry**: + * Traces, using any tag + * Experiments and Datasets, using the `ml_app` tag. Restrictions on Experiments and Datasets are based on their project. Set the `ml_app` tag to match the project ID, and the project and its content (experiments, spans, evaluation metrics, datasets, and dataset records) will be hidden from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. +* Annotation queues and managed prompts are not supported. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. From 2f79cf1e0873f0238b70e0edaa658a303aeccdfb Mon Sep 17 00:00:00 2001 From: Charles Jacquet Date: Wed, 9 Sep 2026 23:42:51 +0200 Subject: [PATCH 09/22] Revise title and content for access control documentation Updated the title and adjusted wording for clarity regarding access control in experiments projects. --- .../improve/experiments/access_control.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index 7f636497326..804341a670b 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -1,5 +1,5 @@ --- -title: Restrict Access to Experiments Projects +title: Data Access Control on Projects description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. further_reading: - link: "/account_management/rbac/data_access/" @@ -15,7 +15,7 @@ further_reading: ## Overview -Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. +Projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. When a project is restricted, users outside the teams or roles you granted access to: @@ -39,12 +39,12 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: - An **ML app** name restricts the traces that application sends to Agent Observability. -- An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. +- A **Project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. 1. Navigate to [Organization Settings > Data Access Controls][3]. 2. Create a new policy that restricts a subset of data. 3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. -4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. +4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your projects. 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one policy. 6. Save the policy. From db895da82d128f63ccc094259b9bfb9d2d07bc9c Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Thu, 10 Sep 2026 12:19:00 -0400 Subject: [PATCH 10/22] Address review: de-emphasize ml_app, move page under Improve Remove the ml_app explainer from the UI procedure; the tag is an implementation detail and is hidden in the Experiments UI, so the procedure now just says to select the project. Keep ml_app only where it is load-bearing: the API payload, the locked-filter-key note, and the older access control page where the value is typed by hand. Reword the data_access.md entry, which read as though you tag your own telemetry, and reword the SDK limitation to lead with the consequence rather than the tag. Move the page from Improve > Experiments to Improve so it can also cover Datasets, and retitle it accordingly. Co-Authored-By: Claude Opus 5 (1M context) --- hugo/config/_default/menus/main.en.yaml | 10 +++++----- .../en/account_management/rbac/data_access.md | 4 ++-- .../en/llm_observability/data_governance.md | 4 ++-- .../en/llm_observability/improve/_index.md | 4 ++++ .../improve/{experiments => }/access_control.md | 17 ++++++----------- 5 files changed, 19 insertions(+), 20 deletions(-) rename hugo/content/en/llm_observability/improve/{experiments => }/access_control.md (77%) diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 325701301e2..620c91c03d7 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5774,11 +5774,11 @@ menu: parent: llm_obs_experiments identifier: llm_obs_experiments_api weight: 50305 - - name: Restrict Access to Projects - url: llm_observability/improve/experiments/access_control - parent: llm_obs_experiments - identifier: llm_obs_experiments_access_control - weight: 50306 + - name: Data Access Control + url: llm_observability/improve/access_control + parent: llm_obs_improve + identifier: llm_obs_improve_access_control + weight: 504 - name: Configure url: llm_observability/configure/ parent: llm_obs diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index fc1000ee263..220f545c97d 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -144,7 +144,7 @@ If you are concerned about unauthorized data access through monitors, Datadog re * **Supported telemetry**: * Traces, using any tag - * Experiments and Datasets, using the `ml_app` tag. Restrictions on Experiments and Datasets are based on their project. Set the `ml_app` tag to match the project ID, and the project and its content (experiments, spans, evaluation metrics, datasets, and dataset records) will be hidden from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. + * Experiments and Datasets, restricted by project. Both use the `ml_app` tag key, with a project ID as the value. A restricted project and its content—its experiments, datasets, dataset records, spans, and evaluation metrics—are hidden from list views, search, and direct links. For instructions, see [Data Access Control in Agent Observability][14]. * Annotation queues and managed prompts are not supported. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. @@ -273,4 +273,4 @@ When querying data through Datadog APIs with restrictions enabled, users without [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs [12]: /dashboards/sharing/shared_dashboards/ [13]: /llm_observability/instrument/otel_instrumentation/ -[14]: /llm_observability/improve/experiments/access_control/ +[14]: /llm_observability/improve/access_control/ diff --git a/hugo/content/en/llm_observability/data_governance.md b/hugo/content/en/llm_observability/data_governance.md index 528f487ef7a..0ea2ec0df21 100644 --- a/hugo/content/en/llm_observability/data_governance.md +++ b/hugo/content/en/llm_observability/data_governance.md @@ -28,7 +28,7 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][11] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific AI applications within your organization. -The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Restrict Access to Experiments Projects][14]. +The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14]. ## Redacting data with span processors @@ -136,4 +136,4 @@ For the full list of available metrics, see [Agent Observability metrics][8]. [11]: /account_management/rbac/data_access [12]: /llm_observability/instrument/sdk/#span-processing [13]: /security/sensitive_data_scanner/ -[14]: /llm_observability/improve/experiments/access_control/ +[14]: /llm_observability/improve/access_control/ diff --git a/hugo/content/en/llm_observability/improve/_index.md b/hugo/content/en/llm_observability/improve/_index.md index f5c5f3f0dbb..bde1ec6c226 100644 --- a/hugo/content/en/llm_observability/improve/_index.md +++ b/hugo/content/en/llm_observability/improve/_index.md @@ -10,3 +10,7 @@ Iterate on your LLM application with datasets, experiments, and the playground. {{< nextlink href="/llm_observability/improve/datasets" >}}Datasets{{< /nextlink >}} {{< nextlink href="/llm_observability/improve/experiments" >}}Experiments{{< /nextlink >}} {{< /whatsnext >}} + +{{< whatsnext desc="Control who can see your data:" >}} + {{< nextlink href="/llm_observability/improve/access_control" >}}Data Access Control{{< /nextlink >}} +{{< /whatsnext >}} diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md similarity index 77% rename from hugo/content/en/llm_observability/improve/experiments/access_control.md rename to hugo/content/en/llm_observability/improve/access_control.md index 804341a670b..b5c1a6c0c3a 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -1,5 +1,5 @@ --- -title: Data Access Control on Projects +title: Data Access Control in Agent Observability description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. further_reading: - link: "/account_management/rbac/data_access/" @@ -36,15 +36,10 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec ## Restrict a project in the UI -Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: - -- An **ML app** name restricts the traces that application sends to Agent Observability. -- A **Project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. - 1. Navigate to [Organization Settings > Data Access Controls][3]. 2. Create a new policy that restricts a subset of data. 3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. -4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your projects. +4. Add a filter on the **Agent Observability** product, then **select the project from the list of values**. The list holds two groups: your projects, and the ML apps sending traces to Agent Observability. Make sure you select from the projects group. 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one policy. 6. Save the policy. @@ -53,9 +48,9 @@ Access to Agent Observability data is keyed on the **`ml_app`** tag, and that on Two things to expect while filling in the filter: - **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. -- **The key may be locked to `ml_app`.** Data Access Control allows one tag key per telemetry type, so if your organization already has an Agent Observability policy, new ones reuse the same key. +- **The filter key may be locked.** Agent Observability policies restrict both applications and projects through a single tag key, `ml_app`, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability policy, new ones reuse the same key. -
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value until then.
+
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting a project by name is only available on the redesigned page; on the older page, enter the project ID as the ml_app value until then.
The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). @@ -93,8 +88,8 @@ Being an admin does not exempt you from a restriction. The `user_access_manage` ## Limitations -- **Spans from an experiment run through the SDK carry your application's `ml_app`, not the project's ID.** A policy on the project does not match those spans, so it hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs recorded on its spans. To cover those as well, add a second filter for your application's own `ml_app` value to the same policy. -- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Spans from an experiment run through the SDK are not restricted by a policy on the project.** Those spans are attributed to the application that ran the experiment rather than to the project, so a policy on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs recorded on its spans. To cover those as well, add a second filter for your application's own `ml_app` value to the same policy. +- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project is attached to these events as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. From 7d2b5ea9ff6988ddf6165bb1c091a98b119c5b59 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Thu, 10 Sep 2026 16:46:26 -0400 Subject: [PATCH 11/22] Apply docs review: Restricted Dataset terminology, branch UI steps Use Restricted Dataset as the primary term throughout, matching the label most orgs see and the Data Access Control page this one links to. The redesigned UI's "New Policy > Sensitive Data Partition" label is kept where it names a button. Move the dual-UI callout above the procedure and branch steps 2 and 4, so someone on the older page no longer reads steps written for the redesigned page and finds the caveat afterward. Scope the free-text warning to the page it applies to. Name the API as the Data Access Control Datasets API, to distinguish it from the Experiments datasets this page also discusses. Tighten the overview, make the restriction list parallel, and pull the direct-link behavior out of it, since that is a response rather than something a user cannot do. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/account_management/rbac/data_access.md | 4 +- .../en/llm_observability/data_governance.md | 2 +- .../improve/access_control.md | 59 ++++++++++--------- 3 files changed, 35 insertions(+), 30 deletions(-) diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index 220f545c97d..111c6e2d115 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -144,8 +144,8 @@ If you are concerned about unauthorized data access through monitors, Datadog re * **Supported telemetry**: * Traces, using any tag - * Experiments and Datasets, restricted by project. Both use the `ml_app` tag key, with a project ID as the value. A restricted project and its content—its experiments, datasets, dataset records, spans, and evaluation metrics—are hidden from list views, search, and direct links. For instructions, see [Data Access Control in Agent Observability][14]. -* Annotation queues and managed prompts are not supported. + * Experiments and Datasets, restricted by project: an `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. +* **Not supported**: Annotation queues and managed prompts. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. diff --git a/hugo/content/en/llm_observability/data_governance.md b/hugo/content/en/llm_observability/data_governance.md index 0ea2ec0df21..24083049fa4 100644 --- a/hugo/content/en/llm_observability/data_governance.md +++ b/hugo/content/en/llm_observability/data_governance.md @@ -28,7 +28,7 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][11] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific AI applications within your organization. -The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14]. +You can also restrict individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14]. ## Redacting data with span processors diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index b5c1a6c0c3a..d64356c9a9c 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -15,16 +15,17 @@ further_reading: ## Overview -Projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. +Experiments projects can hold sensitive material, including dataset prompts and expected outputs, traces from experiment runs, and evaluation results. [Data Access Control][1] lets you restrict a project so that only the teams or roles you specify can see it. -When a project is restricted, users outside the teams or roles you granted access to: +When a project is restricted, users outside the teams or roles you granted access to cannot: -- Do not see the project, or its experiments and datasets, in any list view or search result. -- Receive a *not found* response when they open a direct link to the project or to anything inside it. -- Cannot read the project's dataset records, including inputs and expected outputs. -- Cannot read the evaluation metrics produced by the project's experiment runs. -- Cannot read the spans produced by those runs, with one exception for experiments run through the SDK. See [Limitations](#limitations). -- Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. +- See the project, or its experiments and datasets, in any list view or search result +- Read the project's dataset records, including inputs and expected outputs +- Read the evaluation metrics produced by the project's experiment runs +- Read the spans produced by those runs, except for experiments run through the SDK (see [Limitations](#limitations)) +- Create, modify, or delete anything inside the project, even with an ID they obtained earlier + +Users outside those teams or roles receive a *not found* response when they open a direct link to the project or to anything inside it. Restrictions apply in the Datadog UI and in the API. Application keys are subject to the same restrictions as the user who owns them. @@ -36,31 +37,35 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec ## Restrict a project in the UI -1. Navigate to [Organization Settings > Data Access Controls][3]. -2. Create a new policy that restricts a subset of data. -3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. -4. Add a filter on the **Agent Observability** product, then **select the project from the list of values**. The list holds two groups: your projects, and the ML apps sending traces to Agent Observability. Make sure you select from the projects group. -5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one policy. -6. Save the policy. +
Datadog is rolling out a redesigned access control page, so some steps below depend on which page your organization has. The link in step 1 takes you to either the Data Access Controls page or the redesigned Access Control page. Both configure the same restriction.
-
Select the project from the list rather than typing its name. The value field also accepts free text, and a policy holding a project's name matches no Experiments data: the project stays visible to everyone while the policy looks like it is working. The same applies to a partially typed or misspelled name.
+1. Navigate to [Organization Settings > Data Access Controls][3]. +2. Create a restriction that covers a subset of data: + - On the Data Access Controls page, click **New Restricted Dataset**. + - On the Access Control page, click **New Policy > Sensitive Data Partition**. +3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`. +4. Add a filter on the **Agent Observability** product, then specify the project: + - On the Access Control page, select the project from the list of values. The list has two groups—your projects, and the applications that send traces to Agent Observability—so select from the projects group. + - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. +6. Save the Restricted Dataset. + +
On the Access Control page, select the project from the list rather than typing its name. The value field also accepts free text, and a Restricted Dataset holding a project's name matches no Experiments data: the project stays visible to everyone while the restriction looks like it is working. The same applies to a partially typed or misspelled name.
Two things to expect while filling in the filter: -- **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. -- **The filter key may be locked.** Agent Observability policies restrict both applications and projects through a single tag key, `ml_app`, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability policy, new ones reuse the same key. - -
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting a project by name is only available on the redesigned page; on the older page, enter the project ID as the ml_app value until then.
+- **Projects already covered by another Restricted Dataset do not appear in the list.** A project can belong to only one Restricted Dataset at a time. +- **The filter key may be locked.** Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. -The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). +The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). ### Find a project's ID -If your organization's access control page shows raw values rather than project names, take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. +The Data Access Controls page takes a raw `ml_app` value rather than a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. ## Restrict a project through the API -You can also create the policy with the [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: +You can also create the restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: ```json { @@ -82,16 +87,16 @@ You can also create the policy with the [Datasets API][5]. The `ml_obs` product ## Grant and revoke access -Access is granted by editing the teams or roles on the policy. Removing a team or role takes effect immediately. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. +Access is granted by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. -Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit policies, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would. +Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would. ## Limitations -- **Spans from an experiment run through the SDK are not restricted by a policy on the project.** Those spans are attributed to the application that ran the experiment rather than to the project, so a policy on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs recorded on its spans. To cover those as well, add a second filter for your application's own `ml_app` value to the same policy. -- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project is attached to these events as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Spans from an experiment run through the SDK are not restricted with the project.** Those spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. +- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project is attached to these events as a tag at ingestion time and past events are not re-tagged, so a restriction on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. -- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. +- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. ## Further reading From 580688e89de1507ef6774628d085f76090deaf26 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Thu, 10 Sep 2026 17:51:50 -0400 Subject: [PATCH 12/22] Apply second docs review pass Tighten the dual-UI callout, move the project-picker caveat onto the step it applies to, and drop the shared lead-in so only the locked-key caveat remains. Promote "Find a project's ID" to a sibling section, since the API needs the ID as well as the older UI. Correct the retention limitation. It conflated the date project tagging shipped with the date an admin saves a Restricted Dataset; only the first matters, and tagged events are hidden as soon as a restriction exists. Name that mechanism and keep the boundary a reader can apply. Split the confirmation advice into its own sentence so it no longer reads as applying only when a value matches nothing. Co-Authored-By: Claude Opus 5 (1M context) --- .../improve/access_control.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index d64356c9a9c..0e3a3587bc0 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -37,7 +37,7 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec ## Restrict a project in the UI -
Datadog is rolling out a redesigned access control page, so some steps below depend on which page your organization has. The link in step 1 takes you to either the Data Access Controls page or the redesigned Access Control page. Both configure the same restriction.
+
Datadog is rolling out a redesigned access control page. Your organization has either the Data Access Controls page or the redesigned Access Control page. The link in step 1 takes you to whichever one you have, and both configure the same restriction.
1. Navigate to [Organization Settings > Data Access Controls][3]. 2. Create a restriction that covers a subset of data: @@ -45,27 +45,26 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec - On the Access Control page, click **New Policy > Sensitive Data Partition**. 3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`. 4. Add a filter on the **Agent Observability** product, then specify the project: - - On the Access Control page, select the project from the list of values. The list has two groups—your projects, and the applications that send traces to Agent Observability—so select from the projects group. + - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). + + **Note**: On the Access Control page, projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time. 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. 6. Save the Restricted Dataset. -
On the Access Control page, select the project from the list rather than typing its name. The value field also accepts free text, and a Restricted Dataset holding a project's name matches no Experiments data: the project stays visible to everyone while the restriction looks like it is working. The same applies to a partially typed or misspelled name.
- -Two things to expect while filling in the filter: +
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
-- **Projects already covered by another Restricted Dataset do not appear in the list.** A project can belong to only one Restricted Dataset at a time. -- **The filter key may be locked.** Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. +**Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). -### Find a project's ID +## Find a project's ID -The Data Access Controls page takes a raw `ml_app` value rather than a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. +The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. ## Restrict a project through the API -You can also create the restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: +You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: ```json { @@ -89,14 +88,14 @@ You can also create the restriction with the Data Access Control [Datasets API][ Access is granted by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. -Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would. +Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. ## Limitations -- **Spans from an experiment run through the SDK are not restricted with the project.** Those spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. -- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project is attached to these events as a tag at ingestion time and past events are not re-tagged, so a restriction on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. +- **Spans and evaluation metrics ingested before project tagging became available are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged, so a restriction does not hide event data from experiment runs that predate the rollout. Events that carry the tag are hidden as soon as the restriction is saved, whatever their age. List views, metadata, and dataset records are hidden regardless of when they were created. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. -- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles. +- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles. ## Further reading From ecdf767c31fd62b0ce408d506549aef2a86f5fc3 Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:01:30 -0400 Subject: [PATCH 13/22] Apply suggestion from @domalessi --- hugo/content/en/account_management/rbac/data_access.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index 111c6e2d115..c736012713f 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -61,7 +61,7 @@ You may create a maximum of 100 Restricted Datasets under the Enterprise plan, a ### Supported telemetry types {#supported-telemetry} -- Agent Observability traces +- Agent Observability traces, Experiments, and Datasets - APM traces - Cloud costs - Error Tracking issues From 27ae0cf6627e6a611eb6fe47287b06f6c9dedb87 Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:04:34 -0400 Subject: [PATCH 14/22] Apply suggestion from @domalessi --- hugo/content/en/llm_observability/improve/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index 0e3a3587bc0..7855da087df 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -93,7 +93,7 @@ Being an admin does not exempt you from a restriction. The `user_access_manage` ## Limitations - **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. -- **Spans and evaluation metrics ingested before project tagging became available are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged, so a restriction does not hide event data from experiment runs that predate the rollout. Events that carry the tag are hidden as soon as the restriction is saved, whatever their age. List views, metadata, and dataset records are hidden regardless of when they were created. +- **Spans and evaluation metrics that were not tagged with the project at ingestion are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged. Events that carry the tag are hidden as soon as the restriction is saved. List views, metadata, and dataset records are hidden regardless of when they were created. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles. From a754b7b869d93e25ff0e0f3e0baa52f3d2904205 Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:07:03 -0400 Subject: [PATCH 15/22] Apply batched suggestions from code review Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- .../en/llm_observability/improve/access_control.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index 7855da087df..d3f2d3ab368 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -58,10 +58,6 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). -## Find a project's ID - -The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. - ## Restrict a project through the API You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: @@ -84,6 +80,10 @@ You can also create a restriction with the Data Access Control [Datasets API][5] } ``` +## Find a project's ID + +The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. + ## Grant and revoke access Access is granted by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. From dbcd103d7b47506d7202dbbf33e59885e74f147e Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:09:56 -0400 Subject: [PATCH 16/22] Apply batched suggestions from code review Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com> --- hugo/content/en/llm_observability/improve/access_control.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index d3f2d3ab368..2e4b95ac90a 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -49,11 +49,12 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). **Note**: On the Access Control page, projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time. + +
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
+ 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. 6. Save the Restricted Dataset. -
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
- **Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). From 64e68291dbb0ae6fa11b70d4c97a29631523e10d Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:17:04 -0400 Subject: [PATCH 17/22] Apply suggestion from @domalessi --- hugo/content/en/llm_observability/improve/access_control.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index 2e4b95ac90a..89ffb7ee8eb 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -48,9 +48,7 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). - **Note**: On the Access Control page, projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time. - -
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
+
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data: the project stays visible to everyone, and the restriction looks like it is working. Projects already covered by another Restricted Dataset do not appear in the list; a project can belong to only one Restricted Dataset at a time.
5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. 6. Save the Restricted Dataset. From dab816c0c9c0ef89ccec1175b2fa146d96bde098 Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Thu, 10 Sep 2026 19:21:00 -0400 Subject: [PATCH 18/22] Apply suggestion from @domalessi --- hugo/content/en/llm_observability/improve/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index 89ffb7ee8eb..b29d51c521f 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -48,7 +48,7 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). -
On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data: the project stays visible to everyone, and the restriction looks like it is working. Projects already covered by another Restricted Dataset do not appear in the list; a project can belong to only one Restricted Dataset at a time.
+
On the Access Control page:
  • Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
  • Projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time.
5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. 6. Save the Restricted Dataset. From 87c326a23ab66b214fac307d4ac94c3862b61fbb Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:34:00 -0400 Subject: [PATCH 19/22] Apply suggestion from @domalessi --- hugo/content/en/account_management/rbac/data_access.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index c736012713f..4533c637d9f 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -144,7 +144,9 @@ If you are concerned about unauthorized data access through monitors, Datadog re * **Supported telemetry**: * Traces, using any tag - * Experiments and Datasets, restricted by project: an `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. + * Experiments and Datasets, restricted by project + + **Note:** An `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. * **Not supported**: Annotation queues and managed prompts. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. From ef95cd3b32a56f96b4952bfc49e8acbd5f1adc7b Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:36:08 -0400 Subject: [PATCH 20/22] Apply suggestion from @domalessi --- hugo/content/en/account_management/rbac/data_access.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index 4533c637d9f..9eabedb1598 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -143,10 +143,8 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability * **Supported telemetry**: - * Traces, using any tag - * Experiments and Datasets, restricted by project - - **Note:** An `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. + * Traces, using any tag. + * Experiments and Datasets, restricted by project. An `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. * **Not supported**: Annotation queues and managed prompts. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. From 7f633cbec2567e8832705acee60ef762c8633131 Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:45:44 -0400 Subject: [PATCH 21/22] Apply suggestion from @domalessi --- hugo/content/en/llm_observability/improve/access_control.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index b29d51c521f..6933d663266 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -46,10 +46,11 @@ Restrictions apply in the Datadog UI and in the API. Application keys are subjec 3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`. 4. Add a filter on the **Agent Observability** product, then specify the project: - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. + +
  • Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
  • Projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time.
+ - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). -
On the Access Control page:
  • Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.
  • Projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time.
- 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. 6. Save the Restricted Dataset. From c61b8a3aab7adf48bd4b6210217add5e5a3e572a Mon Sep 17 00:00:00 2001 From: domalessi <111786334+domalessi@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:47:44 -0400 Subject: [PATCH 22/22] Apply suggestion from @domalessi --- hugo/content/en/llm_observability/improve/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md index 6933d663266..9f6ebf57d92 100644 --- a/hugo/content/en/llm_observability/improve/access_control.md +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -86,7 +86,7 @@ The Data Access Controls page and the Datasets API take a project ID as the `ml_ ## Grant and revoke access -Access is granted by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. +Grant access by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would.