-
Notifications
You must be signed in to change notification settings - Fork 1.3k
[DOCS-15652] Add Experiments project access control docs #39707
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
imran-at-datadog
wants to merge
24
commits into
master
Choose a base branch
from
imran.hendley/experiments-dac-docs
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+127
−2
Open
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
7659574
Add Experiments project access control docs
imran-at-datadog d6c3875
Clarify the SDK span limitation
imran-at-datadog a316793
Narrow the span coverage claim in the overview
imran-at-datadog 80c1227
Correct the user_access_manage claim
imran-at-datadog 94dd76b
Merge master into experiments-dac-docs
imran-at-datadog cb90d80
Reduce jargon on the Experiments access control page
imran-at-datadog add1934
Tighten the Experiments access control page
imran-at-datadog 2b000ad
Shorten the SDK span limitation
imran-at-datadog e6cde5b
Merge branch 'master' into imran.hendley/experiments-dac-docs
imran-at-datadog 04e5e20
Update Agent Observability telemetry details
CFLJacquet 2f79cf1
Revise title and content for access control documentation
CFLJacquet db895da
Address review: de-emphasize ml_app, move page under Improve
imran-at-datadog 7d2b5ea
Apply docs review: Restricted Dataset terminology, branch UI steps
imran-at-datadog 580688e
Apply second docs review pass
imran-at-datadog ecdf767
Apply suggestion from @domalessi
domalessi 27ae0cf
Apply suggestion from @domalessi
domalessi a754b7b
Apply batched suggestions from code review
domalessi dbcd103
Apply batched suggestions from code review
domalessi 64e6829
Apply suggestion from @domalessi
domalessi dab816c
Apply suggestion from @domalessi
domalessi 87c326a
Apply suggestion from @domalessi
domalessi ef95cd3
Apply suggestion from @domalessi
domalessi 7f633cb
Apply suggestion from @domalessi
domalessi c61b8a3
Apply suggestion from @domalessi
domalessi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
109 changes: 109 additions & 0 deletions
109
hugo/content/en/llm_observability/improve/access_control.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| --- | ||
| title: Data Access Control in Agent Observability | ||
| description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. | ||
| further_reading: | ||
| - link: "/account_management/rbac/data_access/" | ||
| tag: "Documentation" | ||
| text: "Data Access Control" | ||
| - link: "/llm_observability/data_governance/" | ||
| tag: "Documentation" | ||
| text: "Data Governance" | ||
| - link: "/account_management/rbac/permissions/#access-management" | ||
| tag: "Documentation" | ||
| text: "Access management permissions" | ||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| Experiments projects can hold sensitive material, including dataset prompts and expected outputs, traces from experiment runs, and evaluation results. [Data Access Control][1] lets you restrict a project so that only the teams or roles you specify can see it. | ||
|
|
||
| When a project is restricted, users outside the teams or roles you granted access to cannot: | ||
|
|
||
| - See the project, or its experiments and datasets, in any list view or search result | ||
| - Read the project's dataset records, including inputs and expected outputs | ||
| - Read the evaluation metrics produced by the project's experiment runs | ||
| - Read the spans produced by those runs, except for experiments run through the SDK (see [Limitations](#limitations)) | ||
| - Create, modify, or delete anything inside the project, even with an ID they obtained earlier | ||
|
|
||
| Users outside those teams or roles receive a *not found* response when they open a direct link to the project or to anything inside it. | ||
|
|
||
| Restrictions apply in the Datadog UI and in the API. Application keys are subject to the same restrictions as the user who owns them. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Data Access Control is configured for your organization. See [Data Access Control][1]. | ||
| - You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. | ||
| - The project you want to restrict already exists in Experiments. | ||
|
|
||
| ## Restrict a project in the UI | ||
|
|
||
| <div class="alert alert-info">Datadog is rolling out a redesigned access control page. Your organization has either the <strong>Data Access Controls</strong> page or the redesigned <strong>Access Control</strong> page. The link in step 1 takes you to whichever one you have, and both configure the same restriction.</div> | ||
|
|
||
| 1. Navigate to [Organization Settings > Data Access Controls][3]. | ||
| 2. Create a restriction that covers a subset of data: | ||
| - On the Data Access Controls page, click **New Restricted Dataset**. | ||
| - On the Access Control page, click **New Policy > Sensitive Data Partition**. | ||
| 3. Name it something that identifies the project it protects, for example `Experiments - Fraud Detection`. | ||
| 4. Add a filter on the **Agent Observability** product, then specify the project: | ||
| - On the Access Control page, select the project from the list of values. The list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group. | ||
|
|
||
| <div class="alert alert-warning"><ul><li>Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.</li><li>Projects already covered by another Restricted Dataset do not appear in the list. A project can belong to only one Restricted Dataset at a time.</li></ul></div> | ||
|
|
||
| - On the Data Access Controls page, enter the project ID as the `ml_app` value. See [Find a project's ID](#find-a-projects-id). | ||
|
|
||
| 5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 teams or roles can be attached to one Restricted Dataset. | ||
| 6. Save the Restricted Dataset. | ||
|
|
||
| **Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key. | ||
|
|
||
| The restriction takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations). | ||
|
|
||
| ## Restrict a project through the API | ||
|
|
||
| You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: | ||
|
|
||
| ```json | ||
| { | ||
| "data": { | ||
| "type": "dataset", | ||
| "attributes": { | ||
| "name": "Experiments - Fraud Detection", | ||
| "product_filters": [ | ||
| { | ||
| "product": "ml_obs", | ||
| "filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"] | ||
| } | ||
| ], | ||
| "principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
|
domalessi marked this conversation as resolved.
|
||
| ## Find a project's ID | ||
|
|
||
| The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. | ||
|
|
||
| ## Grant and revoke access | ||
|
|
||
| Grant access by editing the teams or roles on the Restricted Dataset. Removing a team or role takes effect immediately. Deleting the Restricted Dataset removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. | ||
|
|
||
| Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset. | ||
| - **Spans and evaluation metrics that were not tagged with the project at ingestion are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged. Events that carry the tag are hidden as soon as the restriction is saved. List views, metadata, and dataset records are hidden regardless of when they were created. | ||
| - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. | ||
| - **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles. | ||
|
|
||
| ## Further reading | ||
|
|
||
| {{< partial name="whats-next/whats-next.html" >}} | ||
|
|
||
| [1]: /account_management/rbac/data_access/ | ||
| [2]: /account_management/rbac/permissions/#access-management | ||
| [3]: https://app.datadoghq.com/organization-settings/data-access-controls/ | ||
| [4]: /llm_observability/improve/experiments/api/ | ||
| [5]: /api/latest/datasets/ | ||
| [6]: /account_management/rbac/data_access/#strict-mode | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.