Microsoft Graph -> Elasticsearch (ELK) Ingestion Starter

This starter ingests Microsoft Entra ID users from Microsoft Graph into Elasticsearch using:

• msgraph-sdk (Microsoft Graph SDK for Python)
• elasticsearch (official Elasticsearch Python client)

It uses Graph delta queries so subsequent runs only ingest changes.

1) Prerequisites

• Python 3.10+
• Elasticsearch reachable from this machine
• A Microsoft Entra app registration with application permissions

2) Required Graph permissions

For the sample users/delta ingestion:

• User.Read.All (Application)

After adding permissions, click Grant admin consent.

3) Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Fill .env values.

4) Run

python ingest_graph_users.py

First run performs an initial sync. Later runs use the saved delta token in .state/users_delta.json.

5) Verify in Elasticsearch

curl -s "$ELASTICSEARCH_URL/$ELASTICSEARCH_INDEX/_count" | jq

Notes

• Uses stable document IDs (id) to upsert users.
• Handles deleted users from delta stream by issuing Elasticsearch delete operations.
• For production use, run this script on a schedule (cron, Airflow, etc.) and add structured logging/alerting.