diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 79cb84d3f2f..620c91c03d7 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5774,6 +5774,11 @@ menu: parent: llm_obs_experiments identifier: llm_obs_experiments_api weight: 50305 + - name: Data Access Control + url: llm_observability/improve/access_control + parent: llm_obs_improve + identifier: llm_obs_improve_access_control + weight: 504 - name: Configure url: llm_observability/configure/ parent: llm_obs diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index f2102448e4e..9eabedb1598 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -61,7 +61,7 @@ You may create a maximum of 100 Restricted Datasets under the Enterprise plan, a ### Supported telemetry types {#supported-telemetry} -- Agent Observability traces +- Agent Observability traces, Experiments, and Datasets - APM traces - Cloud costs - Error Tracking issues @@ -142,7 +142,10 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability -* **Supported telemetry**: Agent Observability traces are supported. Experiment event data (spans and evaluation metrics) for experiments in a project is also restricted by `ml_app`-keyed Restricted Datasets. Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported. +* **Supported telemetry**: + * Traces, using any tag. + * Experiments and Datasets, restricted by project. An `ml_app` filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14]. +* **Not supported**: Annotation queues and managed prompts. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. @@ -270,3 +273,4 @@ When querying data through Datadog APIs with restrictions enabled, users without [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs [12]: /dashboards/sharing/shared_dashboards/ [13]: /llm_observability/instrument/otel_instrumentation/ +[14]: /llm_observability/improve/access_control/ diff --git a/hugo/content/en/llm_observability/data_governance.md b/hugo/content/en/llm_observability/data_governance.md index 6f1b8055513..24083049fa4 100644 --- a/hugo/content/en/llm_observability/data_governance.md +++ b/hugo/content/en/llm_observability/data_governance.md @@ -28,6 +28,8 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][11] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific AI applications within your organization. +You can also restrict individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14]. + ## Redacting data with span processors You can redact or modify sensitive data at the application level before it is sent to Datadog. Use span processors in the Agent Observability SDK to conditionally modify input and output data on spans, or prevent spans from being emitted entirely. @@ -134,3 +136,4 @@ For the full list of available metrics, see [Agent Observability metrics][8]. [11]: /account_management/rbac/data_access [12]: /llm_observability/instrument/sdk/#span-processing [13]: /security/sensitive_data_scanner/ +[14]: /llm_observability/improve/access_control/ diff --git a/hugo/content/en/llm_observability/improve/_index.md b/hugo/content/en/llm_observability/improve/_index.md index f5c5f3f0dbb..bde1ec6c226 100644 --- a/hugo/content/en/llm_observability/improve/_index.md +++ b/hugo/content/en/llm_observability/improve/_index.md @@ -10,3 +10,7 @@ Iterate on your LLM application with datasets, experiments, and the playground. {{< nextlink href="/llm_observability/improve/datasets" >}}Datasets{{< /nextlink >}} {{< nextlink href="/llm_observability/improve/experiments" >}}Experiments{{< /nextlink >}} {{< /whatsnext >}} + +{{< whatsnext desc="Control who can see your data:" >}} + {{< nextlink href="/llm_observability/improve/access_control" >}}Data Access Control{{< /nextlink >}} +{{< /whatsnext >}} diff --git a/hugo/content/en/llm_observability/improve/access_control.md b/hugo/content/en/llm_observability/improve/access_control.md new file mode 100644 index 00000000000..9f6ebf57d92 --- /dev/null +++ b/hugo/content/en/llm_observability/improve/access_control.md @@ -0,0 +1,109 @@ +--- +title: Data Access Control in Agent Observability +description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. +further_reading: +- link: "/account_management/rbac/data_access/" + tag: "Documentation" + text: "Data Access Control" +- link: "/llm_observability/data_governance/" + tag: "Documentation" + text: "Data Governance" +- link: "/account_management/rbac/permissions/#access-management" + tag: "Documentation" + text: "Access management permissions" +--- + +## Overview + +Experiments projects can hold sensitive material, including dataset prompts and expected outputs, traces from experiment runs, and evaluation results. [Data Access Control][1] lets you restrict a project so that only the teams or roles you specify can see it. + +When a project is restricted, users outside the teams or roles you granted access to cannot: + +- See the project, or its experiments and datasets, in any list view or search result +- Read the project's dataset records, including inputs and expected outputs +- Read the evaluation metrics produced by the project's experiment runs +- Read the spans produced by those runs, except for experiments run through the SDK (see [Limitations](#limitations)) +- Create, modify, or delete anything inside the project, even with an ID they obtained earlier + +Users outside those teams or roles receive a *not found* response when they open a direct link to the project or to anything inside it. + +Restrictions apply in the Datadog UI and in the API. Application keys are subject to the same restrictions as the user who owns them. + +## Prerequisites + +- Data Access Control is configured for your organization. See [Data Access Control][1]. +- You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. +- The project you want to restrict already exists in Experiments. + +## Restrict a project in the UI + +
Datadog is rolling out a redesigned access control page. Your organization has either the Data Access Controls page or the redesigned Access Control page. The link in step 1 takes you to whichever one you have, and both configure the same restriction.
+ +1. Navigate to [Organization Settings > Data Access Controls][3]. +2. Create a restriction that covers a subset of data: + - On the Data Access Controls page, click **New Restricted Dataset**. + - On the Access Control page, click **New Policy > Sensitive Data Partition**. +3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`. +4. Add a filter on the **Agent Observability** product, then specify the project: + - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. + +
+ + - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). + +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. +6. Save the Restricted Dataset. + +**Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. + +The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). + +## Restrict a project through the API + +You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: + +```json +{ + "data": { + "type": "dataset", + "attributes": { + "name": "Experiments - Fraud Detection", + "product_filters": [ + { + "product": "ml_obs", + "filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"] + } + ], + "principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"] + } + } +} +``` + +## Find a project's ID + +The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. + +## Grant and revoke access + +Grant access by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. + +Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. + +## Limitations + +- **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. +- **Spans and evaluation metrics that were not tagged with the project at ingestion are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged. Events that carry the tag are hidden as soon as the restriction is saved. List views, metadata, and dataset records are hidden regardless of when they were created. +- **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. +- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /account_management/rbac/data_access/ +[2]: /account_management/rbac/permissions/#access-management +[3]: https://app.datadoghq.com/organization-settings/data-access-controls/ +[4]: /llm_observability/improve/experiments/api/ +[5]: /api/latest/datasets/ +[6]: /account_management/rbac/data_access/#strict-mode