[DOCS-15652] Add Experiments project access control docs

[imran-at-datadog]

What does this PR do? What is the motivation?

Fixes DOCS-15652

Documents Data Access Control for Agent Observability Experiments projects, which lets an admin restrict a single project so only chosen teams or roles can see it.

Adds llm_observability/improve/access_control covering what a restriction hides (list views, direct links, spans, evaluation metrics, dataset records, and writes), how to author the policy, the API equivalent, and the limitations.

Also corrects the Agent Observability entry on the Data Access Control page. Before this PR it read:

Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported.

Both statements describe an earlier state of the feature. List views, metadata, and Experiments datasets are restricted as of this rollout; annotation queues and managed prompts remain unsupported.

File	Change
llm_observability/improve/access_control.md	New page
llm_observability/improve/_index.md	Section link to the new page
account_management/rbac/data_access.md	Corrected the Agent Observability entry, added a link
llm_observability/data_governance.md	One-line pointer to the new page
config/_default/menus/main.en.yaml	Nav entry under Improve

Merge readiness

• Ready for merge

The Do Not Merge label has been removed, but please read this before merging. The feature is behind a rollout gate that is not yet enabled in production, so the correction to data_access.md describes behavior that is not live. Merging now makes that page accurate for the rollout and inaccurate until it happens. Reviews are welcome in the meantime; I will comment here when the rollout completes.

For Datadog employees:

Review history and one open question:

Resolved in review. The first draft opened the UI procedure by explaining that access is keyed on the ml_app tag and that the same key covers both applications and projects. That framing is gone — ml_app is an implementation detail that the Experiments UI does not surface, so the procedure now just says to select the project. The tag survives only where it is load-bearing: the API payload, a note that the filter key may be locked to a value the org already uses, and the older access control page where the value is typed by hand. The data_access.md entry was reworded for the same reason; its previous phrasing ("Set the ml_app tag to match the project ID") read as though you tag your own telemetry, which is not what happens.

The page also moved from Improve > Experiments to Improve, so it can cover Datasets as well — which it already does, since datasets are restricted by their project — and was retitled Data Access Control in Agent Observability.

Still open — vocabulary between two pages. This page calls the object a policy, matching the redesigned access control page. account_management/rbac/data_access.md calls it a Restricted Dataset throughout, matching the older page that most organizations still see. So step 1 links to a page written in the older vocabulary, and an info callout naming both labels is the only bridge. This is a deliberate choice to track where the product is going, but if Docs would rather the two pages agree today, the fix here is a find-and-replace.

One thing to watch before merge: the procedure tells the admin to select the project from the list of values, which depends on a project-name picker in the access control page. If that is not in place when the feature turns on, admins enter the project ID by hand in every organization rather than only on the older page. A mistyped ID silently restricts nothing while the policy looks correct, which the warning callout covers — but the emphasis would need inverting.

AI assistance

Drafted with Claude Code from the feature's design docs and from the QA notes on the implementation PRs, then reviewed and corrected by hand. AI-assisted research into which app URLs and UI states each customer cohort sees. A later editing pass, also with Claude Code, reduced jargon, removed duplicated statements, and reordered the procedure section. Review feedback was applied with Claude Code and checked by hand.

Additional notes

Preview links should show the new page under Agent Observability > Use Agent Observability to Improve Applications > Data Access Control.

🤖 Generated with Claude Code

[github-actions]

Preview links (active after the build_preview check completes)

New or renamed files

• https://docs-staging.datadoghq.com/imran.hendley/experiments-dac-docs/llm_observability/improve/access_control

Modified Files

• https://docs-staging.datadoghq.com/imran.hendley/experiments-dac-docs/account_management/rbac/data_access
• https://docs-staging.datadoghq.com/imran.hendley/experiments-dac-docs/llm_observability/data_governance
• https://docs-staging.datadoghq.com/imran.hendley/experiments-dac-docs/llm_observability/improve/

[gsvigruha]

thx - overall looks good except for the ml_app comments (in general)

[gsvigruha]

from the user's perspective, is DAC on the ml_app tag or the project_id/project_name tag for Experiments and Datasets?

[imran-at-datadog]

It's ml_app — there is no separate project_id or project_name key. Experiments and Datasets reuse the ml_app key with a project ID as the value.

The reason is the constraint further down this page: Data Access Control allows a single tag key per telemetry type. A dedicated project key would make "restrict a project" and "restrict an application's traces by ml_app" mutually exclusive within an org, so both share the one key.

I've reworded the bullet — the old phrasing ("Set the ml_app tag to match the project ID") read as though you tag your own telemetry, which isn't what happens:

Experiments and Datasets, restricted by project. Both use the ml_app tag key, with a project ID as the value. A restricted project and its content—its experiments, datasets, dataset records, spans, and evaluation metrics—are hidden from list views, search, and direct links.

[gsvigruha]

ml_app is a hacky implementation detail - not sure if this helps users

[imran-at-datadog]

Agreed — removed. This section opened with a two-bullet explainer of the ml_app key; the procedure now just says to select the project, with no mention of the tag. The tag isn't surfaced in the Experiments UI, so a reader has no reason to have met it.

I kept it in three places where it's load-bearing rather than explanatory:

• The API example, where the payload is literally ml_app:<project_id>.
• A note that the filter key may be locked to a value your org already uses, since Data Access Control allows one tag key per telemetry type. That's a surprise an admin hits mid-task.
• The older access control page, where the value is typed by hand rather than picked from a list.

I also reworded the SDK limitation to lead with the consequence instead of the tag — "spans from an experiment run through the SDK are not restricted by a policy on the project," with the tag mentioned only in the remediation sentence.

Two related notes:

The page has moved from improve/experiments/ to improve/ (per Charles), so it can also cover Datasets — which it already does, since datasets restrict by project. It's now titled Data Access Control in Agent Observability. That's why this thread will show as outdated.

Separately, worth a look before this merges: the procedure says to select the project from the list of values, which depends on the project-name picker. If that isn't in place when the feature turns on, admins will need to enter the project ID by hand in every org, not just the ones on the older page — and a mistyped ID silently restricts nothing while the policy looks correct. The warning callout covers the hazard, but the emphasis may need inverting if the picker lands later than the rest.

[domalessi]

Left some initial feedback! Give that a review and ping me when ready for another look.

[domalessi]

Suggested change

	Projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it.
	Experiments projects can hold sensitive material, including dataset prompts and expected outputs, traces from experiment runs, and evaluation results. [Data Access Control][1] lets you restrict a project so that only the teams or roles you specify can see it.

[imran-at-datadog]

Applied, thanks.

[domalessi]

Suggested change

	When a project is restricted, users outside the teams or roles you granted access to:
	- Do not see the project, or its experiments and datasets, in any list view or search result.
	- Receive a *not found* response when they open a direct link to the project or to anything inside it.
	- Cannot read the project's dataset records, including inputs and expected outputs.
	- Cannot read the evaluation metrics produced by the project's experiment runs.
	- Cannot read the spans produced by those runs, with one exception for experiments run through the SDK. See [Limitations](#limitations).
	- Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier.
	When a project is restricted, users outside the teams or roles you granted access to cannot:
	- See the project, or its experiments and datasets, in any list view or search result
	- Open a direct link to the project or to anything inside it (the response is *not found*)
	- Read the project's dataset records, including inputs and expected outputs
	- Read the evaluation metrics produced by the project's experiment runs
	- Read the spans produced by those runs, except for experiments run through the SDK (see [Limitations](#limitations))
	- Create, modify, or delete anything inside the project, even with an ID they obtained earlier

[imran-at-datadog]

Applied. I moved the direct-link bullet out into its own sentence below the list — under a "cannot" stem it read as though the link fails to open, when what actually happens is they open it and get a not found.

[domalessi]

Keep Restricted Dataset as the primary term until the redesigned UI is the default. Most orgs still see that label, and step 1 links to a page that only uses it.

Suggested change

	2. Create a new policy that restricts a subset of data.
	2. Create a Restricted Dataset that restricts a subset of data.

[imran-at-datadog]

Done, and taken page-wide — "policy" appeared 15 times, so changing step 2 alone would have left the page mixed. Everything is now Restricted Dataset except the literal New Policy > Sensitive Data Partition button label in step 2.

Heads up that this changed your suggestions in the two threads below, which both used "policy" — I applied them with the new term.

[domalessi]

Suggested change

	4. Add a filter on the **Agent Observability** product, then **select the project from the list of values**. The list holds two groups: your projects, and the ML apps sending traces to Agent Observability. Make sure you select from the projects group.
	4. Add a filter on **Agent Observability**, then select the project from the projects group.
	**Note**: The value list has two groups: your projects, and the applications that send traces to Agent Observability. Select from the projects group.

[imran-at-datadog]

Folded into the branched step 4 — see the thread below on the callout placement.

[domalessi]

This dual-UI note comes after steps written for the redesigned page, so someone on the older page can finish the procedure incorrectly first.

Move this callout above the numbered list, and make step 4 branch: redesigned page → select the project from the projects group; older page → enter the project ID as the ml_app value.

[imran-at-datadog]

Good catch — this was the most useful comment in the batch. Restructured:

• The callout now sits above the numbered list.
• Steps 2 and 4 both branch, since the button label differs too (New Restricted Dataset vs New Policy > Sensitive Data Partition).
• The free-text warning is now scoped to the Access Control page, since the older page requires typing the value.

I folded your step 4 suggestion from the other thread into the branch rather than adding it as a separate note, so the two-groups detail sits with the step it applies to.

If you'd rather see this as {{< tabs >}}, say so and I'll switch — about half the procedure branches now, so it's a reasonable fit. I kept it inline because only two of six steps actually differ.

[domalessi]

Suggested change

	The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations).
	The policy takes effect as soon as it is saved. The project and its experiments, datasets, and dataset records are hidden immediately, regardless of when they were created. Spans and evaluation metrics are subject to the exceptions in [Limitations](#limitations).

[imran-at-datadog]

Applied.

[domalessi]

On this page, "Datasets API" reads as the Experiments Datasets API. This link goes to the Data Access Control Restricted Datasets API.

Suggested change

	You can also create the policy with the [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value:
	You can also create the restriction with the [Restricted Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value:

[imran-at-datadog]

Agreed on the problem — "dataset" means something else on this page, so the link text was misleading.

Slightly different fix, though: "Restricted Datasets API" isn't the API's name. It's registered as Datasets in full_spec.yaml, and that's the heading the link lands on, so renaming it here would send people looking for a page that doesn't exist under that name.

Went with the qualifier instead:

You can also create the restriction with the Data Access Control [Datasets API][5].

That disambiguates without inventing a name. Happy to reword if you have a preferred pattern for this.

[domalessi]

Suggested change

- **Spans from an experiment run through the SDK are not restricted by a policy on the project.** Those spans are attributed to the application that ran the experiment rather than to the project, so a policy on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs recorded on its spans. To cover those as well, add a second filter for your application's own `ml_app` value to the same policy.

- **Spans from an experiment run through the SDK are not restricted by a policy on the project.** Those spans are attributed to the application that ran the experiment, not to the project. A project policy hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same policy.

[imran-at-datadog]

Applied with "Restricted Dataset" per the terminology thread:

• Spans from an experiment run through the SDK are not restricted with the project. Those spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's ml_app value to the same Restricted Dataset.

[domalessi]

Suggested change

	* **Supported telemetry**:
	* Traces, using any tag
	* Experiments and Datasets, restricted by project. Both use the `ml_app` tag key, with a project ID as the value. A restricted project and its content—its experiments, datasets, dataset records, spans, and evaluation metrics—are hidden from list views, search, and direct links. For instructions, see [Data Access Control in Agent Observability][14].
	* Annotation queues and managed prompts are not supported.
	* **Supported telemetry**:
	* Traces, using any tag
	* Experiments and Datasets, restricted by project (`ml_app` + project ID). See [Data Access Control in Agent Observability][14].
	* **Not supported**: Annotation queues and managed prompts.

[imran-at-datadog]

Took the Not supported reformat — my plain bullet looked out of place next to the bolded ones.

I kept a clause on what a restriction actually does, though:

• Experiments and Datasets, restricted by project: an ml_app filter set to a project ID hides that project and its experiments, datasets, dataset records, spans, and evaluation metrics from list views, search, and direct links. See [Data Access Control in Agent Observability][14].

Two reasons for not going all the way down to (ml_app + project ID). The list-views-and-metadata part is the correction this PR is making — the old text said only event data was restricted — so dropping it loses the fix. And @gsvigruha asked on this exact line whether the key was ml_app or project_id, so the parenthetical alone seemed likely to raise the same question again.

Still a lot shorter than what was there. If you want it tighter I can cut the enumeration and just say "the project and its contents".

[domalessi]

Suggested change

	The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14].
	You can also restrict individual Experiments projects, including their datasets and dataset records. See [Data Access Control in Agent Observability][14].

[imran-at-datadog]

Applied.

[imran-at-datadog]

Thanks @domalessi — all ten addressed in 7d2b5ea. Seven applied as suggested, three with changes:

• Terminology: taken page-wide, not just step 2. "policy" appeared 15 times, so a one-line change would have left the page mixed. This also changed two of your other suggestions, which used "policy" — applied those with the new term.
• Procedure structure: callout moved above the list, steps 2 and 4 branch by UI. Ping me if you'd prefer tabs.
• API link: qualified as the Data Access Control Datasets API rather than renamed — "Restricted Datasets API" isn't the registered name.
• data_access.md: took the Not supported reformat, kept a shortened clause on what a restriction hides, since that's the correction this PR exists to make.

Ready for another look.

[domalessi]

A few more things! Should be good to go after this pass.

[domalessi]

Suggested change

	<div class="alert alert-info">Datadog is rolling out a redesigned access control page, so some steps below depend on which page your organization has. The link in step 1 takes you to either the <strong>Data Access Controls</strong> page or the redesigned <strong>Access Control</strong> page. Both configure the same restriction.</div>
	<div class="alert alert-info">Datadog is rolling out a redesigned access control page. Your organization has either the <strong>Data Access Controls</strong> page or the redesigned <strong>Access Control</strong> page. Both configure the same restriction.</div>

[imran-at-datadog]

Applied, with one clause kept:

The link in step 1 takes you to whichever one you have, and both configure the same restriction.

Step 1 is labelled Organization Settings > Data Access Controls, so anyone whose org has the redesigned Access Control page lands somewhere that doesn't match the link text. Without that clause it looks like a broken link rather than a redirect. Dropped the "some steps below depend on" part as you had it — the branches speak for themselves.

[domalessi]

Suggested change

<div class="alert alert-warning">On the Access Control page, select the project from the list rather than typing its name. The value field also accepts free text, and a Restricted Dataset holding a project's <em>name</em> matches no Experiments data: the project stays visible to everyone while the restriction looks like it is working. The same applies to a partially typed or misspelled name.</div>

<div class="alert alert-warning">On the Access Control page, select the project from the list. Typing the project name, even in part or with a typo, matches no Experiments data. The project stays visible to everyone, and the restriction looks like it is working.</div>

[imran-at-datadog]

Applied.

[domalessi]

Drop the shared lead-in. Keep only the locked-key caveat here — it applies on both pages.

Suggested change

	Two things to expect while filling in the filter:
	- **Projects already covered by another Restricted Dataset do not appear in the list.** A project can belong to only one Restricted Dataset at a time.
	- **The filter key may be locked.** Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key.
	**Note**: The filter key may be locked. Agent Observability uses one tag key, `ml_app`, for both applications and projects, and Data Access Control allows one tag key per telemetry type. If your organization already has an Agent Observability Restricted Dataset, new ones reuse the same key.

[imran-at-datadog]

Applied, and dropped the "Two things to expect" lead-in with it.

[domalessi]

The UI and the API both need the project ID, so this should be a sibling section rather than a subsection of the UI procedure.

Suggested change

	### Find a project's ID
	The Data Access Controls page takes a raw `ml_app` value rather than a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects.
	## Find a project's ID
	The Data Access Controls page and the Datasets API take a project ID as the `ml_app` value, not a project name. Take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects.

[imran-at-datadog]

Applied. Good catch on the API needing the ID too — that's why it read oddly as a subsection.

[domalessi]

Suggested change

	You can also create the restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value:
	You can also create a restriction with the Data Access Control [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value:

[imran-at-datadog]

Applied.

[domalessi]

Suggested change

	Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only: an admin who is not in a granted team or role sees the project as not found, exactly as any other user would.
	Being an admin does not exempt you from a restriction. The `user_access_manage` permission lets you author and edit Restricted Datasets, but access to a restricted project follows team and role membership only. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would.

[imran-at-datadog]

Applied.

[domalessi]

Suggested change

- **Spans from an experiment run through the SDK are not restricted with the project.** Those spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project, its datasets, its dataset records, and its evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset.

- **Spans from an experiment run through the SDK are not restricted by a Restricted Dataset on the project.** These spans are attributed to the application that ran the experiment, not to the project. A Restricted Dataset on the project hides the project and its datasets, dataset records, and evaluation metrics, but not the inputs and outputs on those spans. To restrict those spans too, add a second filter for the application's `ml_app` value to the same Restricted Dataset.

[imran-at-datadog]

Applied.

[domalessi]

The rest of this bullet says the project is attached as a tag at ingestion and past events are not re-tagged. That is about whether the event was tagged, not about when an admin saved the Restricted Dataset. Already-tagged spans should be hidden when the restriction is saved. Suggested wording names that mechanism and drops "became available":

Suggested change

	- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project is attached to these events as a tag at ingestion time and past events are not re-tagged, so a restriction on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age.
	- **Spans and evaluation metrics that were not tagged with the project at ingestion are not restricted.** The project is attached to these events as a tag at ingestion time, and past events are not re-tagged. A restriction on a project does not hide untagged experiment-run data. List views, metadata, and dataset records are hidden regardless of when they were created.

[imran-at-datadog]

You're right, and my version was sloppy — it conflated the date project tagging shipped with the date an admin saves the Restricted Dataset. Only the first one matters, and tagged events are hidden as soon as a restriction exists, whatever their age. I've taken that mechanism and the "hidden when saved" fact from your comment.

One change: I kept a boundary the reader can apply.

• Spans and evaluation metrics ingested before project tagging became available are not restricted. The project is attached to these events as a tag at ingestion time, and past events are not re-tagged, so a restriction does not hide event data from experiment runs that predate the rollout. Events that carry the tag are hidden as soon as the restriction is saved, whatever their age. List views, metadata, and dataset records are hidden regardless of when they were created.

"Events that were not tagged at ingestion are not restricted" is accurate, but a reader can't tell which of their events those are — and the body then says a restriction "does not hide untagged experiment-run data", which restates the heading. The question someone actually arrives with is whether their older experiment data will be hidden, so naming the rollout as the cutoff answers it. Happy to swap in your wording if you'd rather not tie it to the rollout.

[domalessi]

The "so" ties confirmation only to the mismatch case. Confirming is worth doing for every new restriction.

Suggested change

	- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted teams or roles.
	- **A project with no Restricted Dataset is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled [Strict Mode][6] for Agent Observability. A Restricted Dataset whose value matches no project silently restricts nothing. Confirm every new restriction with a user outside the granted teams or roles.

[imran-at-datadog]

Applied.

[imran-at-datadog]

All ten applied in 580688e. Two with small adjustments:

• Dual-UI callout: kept a clause saying the step 1 link redirects to whichever page you have — step 1 is labelled "Data Access Controls", so orgs on the redesigned page would otherwise think the link is wrong.
• Retention limitation: your diagnosis was right, mine conflated the tagging rollout with when the Restricted Dataset is saved. Took your mechanism and the "tagged events are hidden on save" fact, but kept the rollout as the cutoff so a reader can tell whether their older data is affected.

Details in both threads — say the word if you'd rather have either verbatim.

[domalessi]

Combine the step 4 note and warning into one warning. Both are about the Access Control list; two admonitions on one step is heavy. Leave the locked-key note after the steps.

[domalessi]

Split the step 4 warning into two bullets so the silent-failure case and the already-covered list behavior don't sit in one paragraph.

[domalessi]

Thanks for all the changes!

I committed the leftover suggestions, and will now approve. Take a quick look-through to make sure you're cool with these final changes:

• Supported telemetry types now includes Experiments and Datasets.
• Combined the step 4 picker note and the typing warning into one warning, with two bullets.
• Moved Find a project's ID to after the API example.
• Tightened the untagged-spans limitation so it keys off tagging at ingest, not “when the feature shipped.”

[imran-at-datadog]

• Moved Find a project's ID to after the API example.

@domalessi Thanks a lot for the clean up. Everything looks great.