docs: explain datasets vs index patterns for log analytics

[TackAdam]

Description

Expands the Datasets documentation to explain the benefits of datasets over index patterns for log analytics, and how the two coexist. Motivated by repeated customer questions about when to use a dataset vs. an index pattern, and how datasets work for teams still on the classic Discover experience.

All changes land in the existing investigate/datasets.md page (rather than a new page) so there is one canonical Datasets doc. Wording stays in neutral "OpenSearch Dashboards" voice, consistent with the rest of the docs.

Content added to docs/starlight-docs/src/content/docs/investigate/datasets.md:

• Why datasets over index patterns — a benefits section covering:

  • Signal typing (logs/traces) that routes a dataset to the right Discover page
  • OpenTelemetry schema mappings that power log↔trace correlation
  • Query-language awareness (hides languages the engine cannot run when engine/version are known; fail-open otherwise)
  • Cross-cluster reach (remote clusters via cross-cluster search)
  • User-friendly names and descriptions shared across a workspace

  A :::note scopes the newer capabilities (engine/version-aware language support, remote-cluster datasets, persisted schema mappings) to the newer Discover/Explore experience.

• Datasets vs. index patterns at a glance — a comparison table.

• Relationship to index patterns — explains that a saved dataset is persisted as an index-pattern saved object (ad-hoc datasets are temporary in-memory index patterns), so saved datasets and index patterns share storage (no migration), classic Discover keeps working unchanged, and the management view is workspace-scoped. This is the "how does this work for teams still on the classic experience?" answer.

• Enabling datasets — the data_source / workspace / explore configuration needed to surface datasets. Clarifies that explore.enabled force-enables only its query-side settings; workspace and data_source are independent flags.

Developer-experience fix (unrelated to content):

• Documented that the Starlight docs site requires npm install in both docs/ and docs/starlight-docs/. The parent docs/tsconfig.json extends astro/tsconfigs/strict, so astro must be installed at the docs/ level too — otherwise astro dev fails to load its config with Tsconfig not found astro/tsconfigs/strict before rendering any page. Note added to docs/CLAUDE.md and docs/starlight-docs/README.md.

Verification: built/served locally via the Starlight dev server; the datasets page renders with all new sections (Why datasets over index patterns, Relationship to index patterns, Enabling datasets).

Issues Resolved

N/A — documentation improvement. Raised as a draft for review/iteration.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.62%. Comparing base (99ef51b) to head (a6b84b9).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #450   +/-   ##
=======================================
  Coverage   55.62%   55.62%           
=======================================
  Files           4        4           
  Lines         169      169           
  Branches       48       47    -1     
=======================================
  Hits           94       94           
  Misses         74       74           
  Partials        1        1           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ps48]

Nice work, this reads well and most of the technical claims check out against the OSD code (signal typing and Discover routing, OpenTelemetry schema mappings, per-dataset language support, cross-cluster/remote-cluster sources via the index type, the index-pattern saved-object storage model, and the workspace-scoped Datasets page). A few corrections before merge, one of them substantive:

1. S3 is not a selectable dataset type in this experience. The doc lists Amazon S3 (and data-lake tables) as a dataset source in the benefits section, the comparison table, and the "Enabling datasets" note. In the current datasets / Explore / observability-workspace flow, the dataset pickers restrict the selectable types to index patterns (and Prometheus for metrics); no path surfaces an S3 dataset. S3 connections exist in the older query-enhancements data-connections path, but that is not the datasets experience this page documents. Suggest removing S3 from the benefits list, the table, and the enabling note, and keeping the "beyond OpenSearch indexes" point scoped to remote clusters via cross-cluster search (which is supported through the index type). If S3-as-dataset is on the roadmap, phrase it as future rather than current.

2. Explore does not enable the workspace setting. In "Enabling datasets":

Enabling the Explore experience also turns on the related query and workspace settings it depends on.

explore.enabled force-enables the query-side settings it depends on (saved queries new UI, query enhancements, new home page, default theme), but it does not toggle workspace.enabled. That flag is independent and is already listed separately in the yaml, which is correct. Suggest dropping "and workspace" here.

3. Version and branch scope. Some described behaviors (per-dataset engine and version aware language support, remote-cluster datasets, persisting schema mappings on the saved object) are recent. Older lines expose a static SQL/PPL list with no version gating. Worth a one-line note scoping these to the newer Discover/Explore experience.

Optional precision nits:

• "A dataset is persisted as an index-pattern saved object" is exactly true for saved datasets; ad-hoc types are held as temporary in-memory index patterns with no backing saved object.
• The per-dataset language support is effectively fail-open (unknown engine, missing or unparseable version, or a local-cluster dataset keeps all languages enabled), so it is closer to "hide unsupported languages when known" than a strict allowlist.

[anasalkouz]

Seems the link is broken

[TackAdam]

It only appears broken in GitHub's file preview because GitHub resolves /docs/... against the repo root. It matches the convention already used in apm/service-map.md, apm/configuring-apm.md, and investigate/index.md

/docs/investigate/correlations/ is a site-root-relative link using the /docs base Astro applies at build, it resolves on the deployed site (and passes starlightLinksValidator on build).

[anasalkouz]

We might need to highlight that dataset is more focused into log analytics and time series data, I think time field is required for Dataset but not for index pattern

[TackAdam]

Time field is optional for Dataset but the enhanced functionality is worth calling out

Adding
Enhanced functionality on logs and traces pages: The Logs and Traces Discover pages consume datasets, and datasets unlock their time-based features there -- the histogram, time-range controls, and signal-specific views.

[anasalkouz]

Do customer know what is Classic Discover vs Observability workspace? maybe ui snippet make it more easier for understand