From 81d64a6ba99c8258d32b165d36dbb01435039350 Mon Sep 17 00:00:00 2001 From: Yaniv Blum Date: Fri, 17 Jul 2026 11:20:37 -0500 Subject: [PATCH 01/47] Reco: migrate to External API, add new commands, support multi-severity filter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Migrate alerts, identities, apps, files, labels, and comments from internal table APIs to /api/v1/external-api/* endpoints; responses are plain JSON (no base64-encoded cell rows) - Support comma-separated Risk level filter in fetch-incidents (e.g. HIGH,CRITICAL builds a SCIM OR filter) — resolves PRO-303 - Add automatic 429 retry with exponential backoff for all External API calls via _rate_limited_request() - Add five new list commands backed by the External API: reco-list-events, reco-list-posture-issues, reco-list-accounts, reco-list-devices, reco-list-ai-agents (all accept SCIM filters) - Update pack README with Reco SaaS & AI Security positioning - Bump version to 1.8.0 Signed-off-by: Yaniv Blum --- Packs/Reco/Integrations/Reco/README.md | 676 ++++---- Packs/Reco/Integrations/Reco/Reco.py | 2048 ++++++++++++------------ Packs/Reco/Integrations/Reco/Reco.yml | 618 ++++++- Packs/Reco/README.md | 157 +- Packs/Reco/ReleaseNotes/1_8_0.md | 8 + Packs/Reco/pack_metadata.json | 10 +- 6 files changed, 2060 insertions(+), 1457 deletions(-) create mode 100644 Packs/Reco/ReleaseNotes/1_8_0.md diff --git a/Packs/Reco/Integrations/Reco/README.md b/Packs/Reco/Integrations/Reco/README.md index 57b9e002aef..cd09b9095d3 100644 --- a/Packs/Reco/Integrations/Reco/README.md +++ b/Packs/Reco/Integrations/Reco/README.md @@ -1,586 +1,616 @@ -Reco is a Saas data security solution that protects your data from accidental leaks and malicious attacks. -This integration was integrated and tested with version 2023.34.0 of Reco. +Reco is the leader in SaaS & AI Security — securing AI sprawl across SaaS apps and agents. This integration connects Reco's SaaS & AI Security platform to Cortex XSOAR, enabling real-time threat response, posture management, AI governance, and identity risk workflows. + +This integration was integrated and tested with Reco External API v1. ## Configure Reco in Cortex | **Parameter** | **Description** | **Required** | | --- | --- | --- | -| Server URL (e.g. https://host.reco.ai/api/v1) | | True | -| JWT app token | | True | -| Trust any certificate (not secure) | | False | -| Use system proxy settings | | False | -| Incident type | | False | -| Fetch incidents | | False | -| Max fetch | | False | -| Source | Incidents SaaS Source | False | -| Before | Created At time before which incidents will be fetched | False | -| After | Created At time after which incidents will be fetched | False | -| Risk level | Risk level of the incidents to fetch | False | -| First fetch timestamp (<number> <time unit>, e.g., 12 hours, 7 days) | | False | +| Server URL (e.g. https://host.reco.ai/api/v1) | Base URL of your Reco instance | True | +| JWT app token | API Token (Bearer) | True | +| Trust any certificate (not secure) | Skip TLS verification | False | +| Use system proxy settings | Route requests through the system proxy | False | +| Incident type | Incident type to map Reco alerts to | False | +| Fetch incidents | Enable automatic incident fetching | False | +| Max fetch | Maximum incidents to fetch per run (up to 500) | False | +| Source | Filter fetched incidents by SaaS source | False | +| Before | Fetch incidents created before this timestamp | False | +| After | Fetch incidents created after this timestamp | False | +| Risk level | Severity filter for fetched incidents. Accepts a single value or comma-separated list. Values: LOW, MEDIUM, HIGH, CRITICAL (or numeric 10/20/30/40). Example: `HIGH,CRITICAL` | False | +| First fetch timestamp | How far back to fetch on first run (e.g. `7 days`, `12 hours`) | False | + +## SCIM v2 Filters + +All `reco-list-*` commands accept an optional `filters` argument using SCIM v2 syntax: + +| Operator | Meaning | Example | +| --- | --- | --- | +| `eq` | Equals | `severity eq "HIGH"` | +| `ne` | Not equals | `status ne "CLOSED"` | +| `co` | Contains | `email co "@example.com"` | +| `sw` | Starts with | `name sw "John"` | +| `gt` / `ge` | Greater than / or equal | `createdAt gt "2024-01-01T00:00:00Z"` | +| `lt` / `le` | Less than / or equal | `lastSeen le "2024-12-31T23:59:59Z"` | +| `in` | Matches any listed value | `severity in ["HIGH","CRITICAL"]` | +| `not in` | Excludes listed values | `status not in ["CLOSED"]` | +| `and` / `or` / `not` | Logical operators | `isAdmin eq true and hasMfa eq false` | + +Pagination is embedded in the filter string: `limit eq 100 and page eq 1`. ## Commands -You can execute these commands from the CLI, as part of an automation, or in a playbook. -After you successfully execute a command, a DBot message appears in the War Room with the command details. - -### reco-add-exclusion-filter - -*** -Add exclusion filter to Reco Classifier - -#### Base Command +### reco-add-comment-to-alert -`reco-add-exclusion-filter` +Add a comment to an alert in Reco. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| values_to_add | Values to add to the exclusion filter (split by ','). | Required | -| key_to_add | key too add to the exclusion filter (e.g. "CASE_SENSITIVE_TERMS", "LOCATION_CASE_INSENSITIVE_TERMS", "OWNERS", "FILE_IDS", "LOCATIONS"). | Required | +| alert_id | Alert ID to add the comment to. | Required | +| comment | Comment text. | Required | -#### Context Output +### reco-update-incident-timeline -There is no context output for this command. +Add a comment to an incident timeline in Reco. -### reco-update-incident-timeline +#### Input -*** -Update incident timelines +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| incident_id | Incident ID. | Required | +| comment | Comment text. | Required | -#### Base Command +### reco-resolve-visibility-event -`reco-update-incident-timeline` +Resolve an event in a Reco Finding. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| comment | Comment to add to the incident. | Required | -| incident_id | Incident ID to add the comment to. | Required | +| entity_id | Entity ID of the file to resolve. | Required | +| label_name | Label name to resolve (e.g. `Accessible to All Org Users`). | Required | -#### Context Output +### reco-get-risky-users -There is no context output for this command. +List all accounts flagged as risky (auto-paginates all results). -### reco-add-comment-to-alert +#### Context Output -*** -Add a comment to an alert in Reco +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.RiskyUsers.id | String | Account ID | +| Reco.RiskyUsers.name | String | Account display name | +| Reco.RiskyUsers.accountEmail | String | Account email address | +| Reco.RiskyUsers.permissions | String | Account permissions (ADMIN / PRIVILEGED / STANDARD) | +| Reco.RiskyUsers.hasMfa | String | MFA status (MFA / NOMFA / NA) | +| Reco.RiskyUsers.openAlerts | Number | Number of open alerts for this account | +| Reco.RiskyUsers.isAdmin | Boolean | Whether the account has admin privileges | +| Reco.RiskyUsers.isRiskyUser | Boolean | Whether the account is flagged as risky | +| Reco.RiskyUsers.lastSeen | Date | Last activity timestamp | -#### Base Command +### reco-add-risky-user-label -`reco-add-comment-to-alert` +Tag a user as risky in Reco. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| comment | Comment to add to the incident. | Required | -| alert_id | Alert ID to add the comment to. | Required | +| email_address | Email address of the user to tag as risky. | Required | -#### Context Output +### reco-add-leaving-org-user-label -There is no context output for this command. +Tag a user as a departing employee in Reco. -### reco-resolve-visibility-event +#### Input -*** -Resolve an event in Reco Finding. Reco Findings contains aggregations of events. This command resolves the event in the Reco Finding. +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| email_address | Email address of the user to tag as departing. | Required | -#### Base Command +### reco-get-assets-user-has-access-to -`reco-resolve-visibility-event` +List files a user has access to. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| entity_id | entity id of the file to resolve. | Required | -| label_name | label name to resolve (e.g. "Accessible to All Org Users", "Accessible by General Public"). | Required | +| email_address | User email address. | Required | +| only_sensitive | Return only sensitive assets. | Optional | #### Context Output -There is no context output for this command. - -### reco-get-risky-users - -*** -Get Risky Users from Reco +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Assets | Unknown | Assets the user has access to | -#### Base Command +### reco-get-sensitive-assets-by-name -`reco-get-risky-users` +Find sensitive assets by name. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | +| asset_name | Asset name to search for. | Required | +| regex_search | Use substring/contains matching instead of exact match. | Optional | #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.RiskyUsers | unknown | Risky Users | - -### reco-add-risky-user-label +| Reco.SensitiveAssets.id | String | Asset ID | +| Reco.SensitiveAssets.name | String | Asset name | +| Reco.SensitiveAssets.owner | String | Asset owner | +| Reco.SensitiveAssets.url | String | Asset URL | +| Reco.SensitiveAssets.sensitivityLevel | Number | Sensitivity level (30=HIGH, 40=CRITICAL) | +| Reco.SensitiveAssets.permissionVisibility | String | Permission visibility (PUBLIC / INTERNAL / RESTRICTED) | +| Reco.SensitiveAssets.location | String | File path | +| Reco.SensitiveAssets.dataCategories | Unknown | Detected data categories | -*** -Tag a user as risky in Reco - -#### Base Command +### reco-get-sensitive-assets-by-id -`reco-add-risky-user-label` +Find sensitive assets by ID. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| email_address | Email address of the user to add to the risky users list in Reco. | Required | +| asset_id | Asset ID. | Required | #### Context Output -There is no context output for this command. - -### reco-get-assets-user-has-access-to - -*** -Get all files user has access to from Reco +Same as `reco-get-sensitive-assets-by-name`. -#### Base Command +### reco-get-assets-by-id -`reco-get-assets-user-has-access-to` +Find any asset by ID. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| email_address | Email address of the user. | Required | -| only_sensitive | Return only sensitive assets owned by this user. | Optional | +| asset_id | Asset ID. | Required | #### Context Output -| **Path** | **Type** | **Description** | -| --- | --- | --- | -| Reco.Assets | unknown | Assets user has access to | - -### reco-add-leaving-org-user-label - -*** -Tag a user as leaving org user in Reco +Same as `reco-get-sensitive-assets-by-name`. -#### Base Command +### reco-get-link-to-user-overview-page -`reco-add-leaving-org-user-label` +Generate a deep link to the Reco UI overview page for an entity. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| email_address | Email address of the user to tag as levaing org user. | Required | - -#### Context Output - -There is no context output for this command. - -### reco-get-sensitive-assets-by-name - -*** -Get all sensitive assets from Reco by name +| entity | Entity type (e.g. `RM_LINK_TYPE_USER`). | Required | +| param | Entity ID or email. | Optional | -#### Base Command +### reco-get-3rd-parties-accessible-to-data-list -`reco-get-sensitive-assets-by-name` +List third-party domains that have access to sensitive data. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| asset_name | Asset name to search for. | Required | -| regex_search | Return only sensitive assets owned by this user. | Optional | +| last_interaction_time_in_days | Include domains with activity within this many days. | Required | #### Context Output -There is no context output for this command. +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Domains.domain | String | Third-party domain | +| Reco.Domains.last_activity | String | Last interaction date | +| Reco.Domains.files_num | Number | Number of files accessible | +| Reco.Domains.users_with_access_num | Number | Number of users with access | -### reco-get-sensitive-assets-by-id +### reco-get-sensitive-assets-with-public-link -*** -Get all sensitive assets from Reco by id +List sensitive assets exposed via a public link. -#### Base Command +#### Context Output -`reco-get-sensitive-assets-by-id` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Assets.asset_id | String | Asset ID | +| Reco.Assets.asset | Unknown | Asset metadata | +| Reco.Assets.data_category | String | Primary data category | +| Reco.Assets.last_access_date | String | Last access date | + +### reco-get-files-shared-with-3rd-parties + +List files shared with a specific third-party domain. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| asset_id | Asset id to search for. | Required | +| domain | Third-party domain to query. | Required | +| last_interaction_time_in_days | Include files with activity within this many days. | Required | #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.SensitiveAssets.file_name | String | The name of the asset | -| Reco.SensitiveAssets.file_owner | String | The owner of the asset | -| Reco.SensitiveAssets.file_url | Unknown | Json string of the asset's url and the name | -| Reco.SensitiveAssets.currently_permitted_users | String | List of currently permitted users | -| Reco.SensitiveAssets.visibility | String | Visibility of the asset | -| Reco.SensitiveAssets.location | String | The path of the asset | -| Reco.SensitiveAssets.source | String | SaaS tool source of the asset | -| Reco.SensitiveAssets.sensitivity_level | Number | The sensitivity level of the asset | +| Reco.Assets.asset_id | String | Asset ID | +| Reco.Assets.location | String | File location | +| Reco.Assets.file_owner | String | File owner | +| Reco.Assets.domain | String | Third-party domain | +| Reco.Assets.last_access_date | String | Last access date | -### reco-get-link-to-user-overview-page +### reco-change-alert-status -*** -Generate a magic link for reco UI (overview page) +Update the status of a Reco alert. -#### Base Command +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| alert_id | Alert ID. | Required | +| status | New status. Possible values: `ALERT_STATUS_NEW`, `ALERT_STATUS_IN_PROGRESS`, `ALERT_STATUS_CLOSED`. | Required | + +### reco-get-user-context-by-email-address -`reco-get-link-to-user-overview-page` +Get identity context for a user by email address. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| entity | Entity Type (RM_LINK_TYPE_USER). | Required | -| param | Entity ID (user email). | Optional | +| email_address | User email address. | Required | #### Context Output -There is no context output for this command. - -### reco-get-3rd-parties-accessible-to-data-list - -*** -Get 3rd parties accessible to sensitive assets +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.User.id | String | Identity ID | +| Reco.User.email | String | Primary email address | +| Reco.User.name | String | Full name | +| Reco.User.departments | String | Departments | +| Reco.User.jobTitles | String | Job titles | +| Reco.User.isFormer | Boolean | Whether the user is a former employee | +| Reco.User.isInternal | Boolean | Whether the user is an internal employee | +| Reco.User.openAlerts | Number | Number of open alerts | +| Reco.User.lastSeen | Date | Last activity timestamp | -#### Base Command +### reco-get-files-exposed-to-email-address -`reco-get-3rd-parties-accessible-to-data-list` +List files accessible to a specific email address. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| last_interaction_time_in_days | Last interaction time in days. | Required | +| email_address | Email address. | Required | #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Domains.domain | String | The domain of the 3rd party | -| Reco.Domains.last_activity | String | The last interaction time with the 3rd party | -| Reco.Domains.files_num | Number | The number of files the 3rd party has access to | -| Reco.Domains.users_with_access_num | Number | The number of users the 3rd party has access to | - -### reco-get-sensitive-assets-with-public-link +| Reco.Assets.asset_id | String | Asset ID | +| Reco.Assets.location | String | File location | +| Reco.Assets.email_account | String | Email account with access | +| Reco.Assets.file_owner | String | File owner | -*** -Get all sensitive assets with public link from Reco - -#### Base Command +### reco-get-assets-shared-externally -`reco-get-sensitive-assets-with-public-link` +List files an owner has shared outside the organization. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | +| email_address | File owner email address. | Required | #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Assets.asset_id | String | The asset id | -| Reco.Assets.asset | Unknown | Json string of the asset's url and the name | -| Reco.Assets.data_category | String | The data category of the asset | -| Reco.Assets.data_categories | String | The data categories of the asset | -| Reco.SensitiveAssets.location | String | The path of the asset | -| Reco.SensitiveAssets.source | String | SaaS tool source of the asset | -| Reco.Assets.last_access_date | String | The last access date of the asset | +| Reco.Assets.asset_id | String | Asset ID | +| Reco.Assets.file_owner | String | File owner | +| Reco.Assets.last_access_date | String | Last access date | -### reco-get-files-shared-with-3rd-parties +### reco-get-private-email-list-with-access -*** -Get files shared with 3rd parties +List private (non-corporate) email addresses with file access. -#### Base Command +#### Context Output -`reco-get-files-shared-with-3rd-parties` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.privateEmails.email_account | String | Private email account | +| Reco.privateEmails.primary_email | String | Associated corporate email | +| Reco.privateEmails.files_num | Number | Number of files accessible | +| Reco.privateEmails.user_category | String | User category | + +### reco-get-alert-ai-summary + +Get an AI-generated summary of an alert. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| last_interaction_time_in_days | Last interaction time in days. | Required | -| domain | Domain to search. | Required | +| alert_id | Alert ID. | Required | #### Context Output -| **Path** | **Type** | **Description** | -|------------------------------|----------|-------------------------------------------------------------| -| Reco.Assets.asset_id | String | The asset id of the file | -| Reco.Assets.location | String | The location of the file | -| Reco.Assets.users | String | Users the file is shared with | -| Reco.Assets.file_owner | String | File Owner | -| Reco.Assets.asset | Unknown | The asset metadata | -| Reco.Assets.data_category | String | The data category of the assets the 3rd party has access to | -| Reco.Assets.last_access_date | String | The last access date of the asset | -| Reco.Assets.domain | String | The domain of the 3rd party | - -### reco-change-alert-status - -*** -update alert status in Reco +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.AlertSummary.markdown | String | Markdown-formatted alert summary | -#### Base Command +### reco-get-apps -`reco-change-alert-status` +List all discovered SaaS applications (auto-paginates all results). #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| alert_id | alert id to get. | Required | -| status | status to set the alert to (e.g. "ALERT_STATUS_NEW", "ALERT_STATUS_IN_PROGRESS", "ALERT_STATUS_CLOSED"). Possible values are: ALERT_STATUS_NEW, ALERT_STATUS_IN_PROGRESS, ALERT_STATUS_CLOSED. | Required | +| before | Filter apps last seen before this date. | Optional | +| after | Filter apps last seen after this date. | Optional | +| limit | Page size (omit for all results). | Optional | #### Context Output -There is no context output for this command. - -### reco-get-user-context-by-email-address - -*** -Get user context by email address from Reco. +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Apps.id | String | App ID | +| Reco.Apps.name | String | App name | +| Reco.Apps.category | String | App category | +| Reco.Apps.usersCount | Number | Number of users | +| Reco.Apps.authorization | String | Authorization status | +| Reco.Apps.isUsingAi | Boolean | Whether the app uses AI | +| Reco.Apps.isShadowApp | Boolean | Whether the app is unsanctioned shadow IT | +| Reco.Apps.vendorGrade | String | Vendor security grade (A–F) | +| Reco.Apps.aiCapability | String | AI capability (AI-Native / AI-Assisted / No-AI) | +| Reco.Apps.lastSeen | Date | Last activity timestamp | -#### Base Command +### reco-set-app-authorization-status -`reco-get-user-context-by-email-address` +Update the authorization status of an application. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | -| email_address | user email address. | Required | +| app_id | Application ID. | Required | +| authorization_status | Authorization status. Possible values: `AUTH_STATUS_SANCTIONED`, `AUTH_STATUS_UNSANCTIONED`, `AUTH_STATUS_TO_REVIEW`, `AUTH_STATUS_ACCEPTED_RISK`, `AUTH_STATUS_EVALUATING`, `AUTH_STATUS_UNDER_INVESTIGATION`, `AUTH_STATUS_INVESTIGATED`, `AUTH_STATUS_CLOUD_INVENTORY`, `AUTH_STATUS_SYSTEM_SANCTIONED`. | Required | #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.User.email_account | String | The email of the user. | -| Reco.User.departments | String | User departments. | -| Reco.User.job_titles | String | Job Title. | -| Reco.User.category | String | Category. | -| Reco.User.groups | String | The groups user is member of. | -| Reco.User.full_name | String | The user full name. | -| Reco.User.labels | Unknown | User Labels. | +| Reco.AppAuthorization.app_id | String | Application ID | +| Reco.AppAuthorization.authorization_status | String | New authorization status | +| Reco.AppAuthorization.updated | Boolean | Whether the update succeeded | -### reco-get-files-exposed-to-email-address +#### Command example -*** -Get files exposed to a specific email address +``` +!reco-set-app-authorization-status app_id="microsoft.com" authorization_status="AUTH_STATUS_SANCTIONED" +``` -#### Base Command +### reco-add-exclusion-filter -`reco-get-files-exposed-to-email-address` +Add values to a Reco classifier exclusion list. #### Input | **Argument Name** | **Description** | **Required** | | --- | --- | --- | +| key_to_add | Exclusion key (e.g. `CASE_SENSITIVE_TERMS`, `OWNERS`, `FILE_IDS`, `LOCATIONS`). | Required | +| values_to_add | Comma-separated values to add. | Required | + +--- + +## List Commands (External API) + +All commands below accept `filters` (SCIM v2 expression) and `limit` (default 1000). + +### reco-list-events + +List SaaS activity events. #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Assets.asset_id | String | The asset id | -| Reco.Assets.asset | Unknown | Json string of the asset's url and the name | -| Reco.Assets.data_category | String | The data category of the asset | -| Reco.Assets.data_categories | String | The data categories of the asset | -| Reco.Assets.location | String | The path of the asset. | -| Reco.Assets.source | String | SaaS tool source of the asset. | -| Reco.Assets.last_access_date | String | The last access date of the asset | -| Reco.Assets.email_account | String | The last access date of the asset | -| Reco.Assets.file_owner | String | SaaS tool source of the asset | +| Reco.Events.id | String | Event ID | +| Reco.Events.eventType | String | Event type code | +| Reco.Events.formattedEventType | String | Human-readable event type | +| Reco.Events.application | String | Source SaaS application | +| Reco.Events.actorEmail | String | Actor email address | +| Reco.Events.eventTime | Date | Event timestamp | +| Reco.Events.outcomeString | String | Event outcome description | -### reco-get-assets-shared-externally +### reco-list-posture-issues -*** -Get files exposed to a specific email address +List security posture issues. -#### Base Command +#### Context Output -`reco-get-assets-shared-externally` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.PostureIssues.id | String | Issue ID | +| Reco.PostureIssues.name | String | Issue name | +| Reco.PostureIssues.severity | String | Severity (LOW/MEDIUM/HIGH/CRITICAL) | +| Reco.PostureIssues.checkStatus | String | Check status | +| Reco.PostureIssues.scorePercentage | Number | Compliance score percentage | +| Reco.PostureIssues.url | String | Link to issue in Reco UI | -#### Input +### reco-list-accounts -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | +List SaaS accounts. #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Assets.asset_id | String | The asset id | -| Reco.Assets.asset | Unknown | Json string of the asset's url and the name | -| Reco.Assets.data_category | String | The data category of the asset | -| Reco.Assets.data_categories | String | The data categories of the asset | -| Reco.SensitiveAssets.location | String | The path of the asset | -| Reco.SensitiveAssets.source | String | SaaS tool source of the asset | -| Reco.Assets.last_access_date | String | The last access date of the asset | -| Reco.Assets.file_owner | String | SaaS tool source of the asset | +| Reco.Accounts.id | String | Account ID | +| Reco.Accounts.name | String | Account display name | +| Reco.Accounts.accountEmail | String | Account email address | +| Reco.Accounts.permissions | String | Permission level | +| Reco.Accounts.hasMfa | String | MFA status | +| Reco.Accounts.openAlerts | Number | Open alerts count | +| Reco.Accounts.isAdmin | Boolean | Admin flag | +| Reco.Accounts.isRiskyUser | Boolean | Risky user flag | +| Reco.Accounts.lastSeen | Date | Last activity | -### reco-get-private-email-list-with-access +### reco-list-devices -*** -Get private email list with access +List managed and unmanaged devices. -#### Base Command +#### Context Output -`reco-get-private-email-list-with-access` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Devices.id | String | Device ID | +| Reco.Devices.name | String | Device name | +| Reco.Devices.devicePlatform | String | Platform (Windows / macOS / iOS / Android) | +| Reco.Devices.isUnmanaged | Boolean | Whether the device is unmanaged | +| Reco.Devices.hasNonCompliant | Boolean | Whether the device has policy violations | +| Reco.Devices.lastSeen | Date | Last activity | -#### Input +### reco-list-ai-agents -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | +List detected AI agents. #### Context Output -| **Path** | **Type** | **Description** | -| --- |----------|---------------------------| -| Reco.privateEmails.email_account | String | The email account | -| Reco.privateEmails.primary_email | String | The primary email account | -| Reco.privateEmails.files_num | String | Number of files | -| Reco.privateEmails.user_category | String | The category of the user | +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.AiAgents.id | String | AI agent ID | +| Reco.AiAgents.name | String | Agent name | +| Reco.AiAgents.vendor | String | Vendor | +| Reco.AiAgents.authorization | String | Authorization status | +| Reco.AiAgents.risk | String | Risk level | +| Reco.AiAgents.lastUsage | Date | Last usage | -### reco-get-assets-by-id +### reco-list-groups -*** -Get all assets from Reco by id +List SaaS groups. -#### Base Command +#### Context Output -`reco-get-assets-by-id` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.Groups.id | String | Group ID | +| Reco.Groups.name | String | Group name | +| Reco.Groups.email | String | Group email | +| Reco.Groups.membersCount | Number | Member count | -#### Input +### reco-list-saas-to-saas -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | -| asset_id | Asset id to search for. | Required | +List SaaS-to-SaaS OAuth grants. #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.SensitiveAssets.file_name | String | The name of the asset | -| Reco.SensitiveAssets.file_owner | String | The owner of the asset | -| Reco.SensitiveAssets.file_url | Unknown | Json string of the asset's url and the name | -| Reco.SensitiveAssets.currently_permitted_users | String | List of currently permitted users | -| Reco.SensitiveAssets.visibility | String | Visibility of the asset | -| Reco.SensitiveAssets.location | String | The path of the asset | -| Reco.SensitiveAssets.source | String | SaaS tool source of the asset | -| Reco.SensitiveAssets.sensitivity_level | Number | The sensitivity level of the asset | +| Reco.SaasToSaas.id | String | Grant ID | +| Reco.SaasToSaas.plugin | String | Plugin / app name | +| Reco.SaasToSaas.authorization | String | Authorization status | +| Reco.SaasToSaas.permissionRisk | String | Permission risk level | +| Reco.SaasToSaas.aiCapability | String | AI capability | -### reco-get-alert-ai-summary +### reco-list-ip-addresses -*** -Get alert AI summary +List observed IP addresses. -#### Base Command +#### Context Output -`reco-get-alert-ai-summary` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.IpAddresses.ipAddress | String | IP address | +| Reco.IpAddresses.country | String | Country | +| Reco.IpAddresses.eventsCount | Number | Event count | +| Reco.IpAddresses.hasVpn | Boolean | VPN flag | +| Reco.IpAddresses.hasProxy | Boolean | Proxy flag | -#### Input +### reco-list-business-units -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | -| alert_id | Alert id to get. | Required | +List external business units. #### Context Output -| **Path** | **Type** | **Description** | -|----------------------------| --- |-------------------| -| Reco.AlertSummary.markdown | String | The alert summary | +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.BusinessUnits.id | String | Business unit ID | +| Reco.BusinessUnits.name | String | Business unit name | +| Reco.BusinessUnits.manager | String | Manager | -### reco-get-apps +### reco-list-audit-logs -*** -Get app discovery data from Reco. Fetches all available apps using pagination. +List Reco platform audit logs. -#### Base Command +#### Context Output -`reco-get-apps` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.AuditLogs.id | String | Log entry ID | +| Reco.AuditLogs.userEmail | String | Actor email | +| Reco.AuditLogs.module | String | Module | +| Reco.AuditLogs.action | String | Action performed | +| Reco.AuditLogs.timestamp | Date | Timestamp | -#### Input +### reco-list-posture-checks -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | -| before | Before date filter (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ format). | Optional | -| after | After date filter (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ format). | Optional | -| limit | Page size for pagination (default 1000). The command fetches all available apps across all pages. | Optional | +List posture check definitions. #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Apps.app_name | String | The name of the application | -| Reco.Apps.app_id | String | The unique identifier of the application | -| Reco.Apps.category | String | The category of the application | -| Reco.Apps.risk_score | Number | The risk score of the application | -| Reco.Apps.users_count | Number | The number of users with access to the application | -| Reco.Apps.data_access | String | The data access level of the application | -| Reco.Apps.updated_at | Date | The last update timestamp of the application | -| Reco.Apps.created_at | Date | The creation timestamp of the application | -| Reco.Apps.status | String | The status of the application | +| Reco.PostureChecks.id | String | Check ID | +| Reco.PostureChecks.name | String | Check name | +| Reco.PostureChecks.severity | String | Severity | +| Reco.PostureChecks.apps | Unknown | Applicable apps | -### reco-set-app-authorization-status +### reco-list-threat-detection-policies -*** -Set authorization status for an application in Reco. +List threat detection policies. -#### Base Command +#### Context Output -`reco-set-app-authorization-status` +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.ThreatDetectionPolicies.id | String | Policy ID | +| Reco.ThreatDetectionPolicies.name | String | Policy name | +| Reco.ThreatDetectionPolicies.severity | String | Severity | +| Reco.ThreatDetectionPolicies.status | String | ON / OFF / PREVIEW | +| Reco.ThreatDetectionPolicies.openAlerts | Number | Open alerts | -#### Input +### reco-list-exclusions -| **Argument Name** | **Description** | **Required** | -| --- | --- | --- | -| app_id | Application ID to update authorization status for. | Required | -| authorization_status | Authorization status to set for the application. Possible values are: AUTH_STATUS_UNSPECIFIED, AUTH_STATUS_TO_REVIEW, AUTH_STATUS_SANCTIONED, AUTH_STATUS_UNSANCTIONED, AUTH_STATUS_ACCEPTED_RISK, AUTH_STATUS_EVALUATING, AUTH_STATUS_UNDER_INVESTIGATION, AUTH_STATUS_INVESTIGATED, AUTH_STATUS_CLOUD_INVENTORY, AUTH_STATUS_SYSTEM_SANCTIONED. | Required | +List alert suppression exclusion rules. #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.AppAuthorization.app_id | String | The application ID that was updated | -| Reco.AppAuthorization.authorization_status | String | The authorization status that was set | -| Reco.AppAuthorization.updated | Boolean | Whether the update was successful | -| Reco.AppAuthorization.rows_affected | Number | Number of rows affected by the update operation | - -#### Command example +| Reco.Exclusions.id | String | Exclusion ID | +| Reco.Exclusions.name | String | Exclusion name | +| Reco.Exclusions.policyName | String | Associated policy | +| Reco.Exclusions.createdBy | String | Created by | -``` -!reco-set-app-authorization-status app_id="microsoft.com" authorization_status="AUTH_STATUS_SANCTIONED" -``` +### reco-list-app-instances -#### Context Example - -```json -{ - "Reco": { - "AppAuthorization": { - "app_id": "microsoft.com", - "authorization_status": "AUTH_STATUS_SANCTIONED", - "updated": true, - "rows_affected": 1 - } - } -} -``` +List integrated app instances (app portfolio). Only returns instances with an active integration status. -#### Human Readable Output +#### Context Output ->App microsoft.com authorization status updated successfully to AUTH_STATUS_SANCTIONED +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.AppInstances.id | String | App instance ID | +| Reco.AppInstances.name | String | App instance name | +| Reco.AppInstances.instanceType | String | Instance type | +| Reco.AppInstances.accountsCount | Number | Number of accounts | +| Reco.AppInstances.isUsingAi | Boolean | Whether AI features are used | +| Reco.AppInstances.saasToSaasCount | Number | SaaS-to-SaaS grant count | +| Reco.AppInstances.filesCount | Number | File count | diff --git a/Packs/Reco/Integrations/Reco/Reco.py b/Packs/Reco/Integrations/Reco/Reco.py index 4f703e0adef..b7e06d43079 100644 --- a/Packs/Reco/Integrations/Reco/Reco.py +++ b/Packs/Reco/Integrations/Reco/Reco.py @@ -2,6 +2,7 @@ from CommonServerPython import * # noqa: F401 import base64 import json +import time from datetime import datetime, timedelta import dateutil.parser @@ -26,14 +27,55 @@ DEMISTO_OCCURRED_FORMAT = "%Y-%m-%dT%H:%M:%SZ" RECO_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ" DEMISTO_INFORMATIONAL = 0.5 -RECO_API_TIMEOUT_IN_SECONDS = 180 # Increase timeout for RECO API +RECO_API_TIMEOUT_IN_SECONDS = 180 RECO_ACTIVE_INCIDENTS_VIEW = "active_incidents_view" +RATE_LIMIT_MAX_RETRIES = 3 +RATE_LIMIT_RETRY_BASE_DELAY = 2 # seconds; doubles on each attempt ALERT_VIEW_WITH_SHARED_STATUS = "ALERT_VIEW_WITH_SHARED_STATUS" RECO_TIMELINE_EVENT_TYPE = "TIMELINE_EVENT_TYPE_USER_COMMENT" CREATED_AT_FIELD = "created_at" STEP_FETCH = "fetch" STEP_INIT = "init" +# Base path for the Reco external API (relative to /api/v1) +EXTERNAL_API_BASE = "/external-api" + +# Maps numeric or string risk level values to the severity name expected by the external API. +RISK_LEVEL_TO_SEVERITY_NAME: dict[str, str] = { + "10": "LOW", + "LOW": "LOW", + "low": "LOW", + "20": "MEDIUM", + "MEDIUM": "MEDIUM", + "medium": "MEDIUM", + "30": "HIGH", + "HIGH": "HIGH", + "high": "HIGH", + "40": "CRITICAL", + "CRITICAL": "CRITICAL", + "critical": "CRITICAL", +} + + +def parse_risk_levels(risk_level_param: str | None) -> list[str] | None: + """Parse a single or comma-separated risk_level string into a list of severity names. + + Accepts numeric values (10/20/30/40) or names (LOW/MEDIUM/HIGH/CRITICAL). + Examples: "HIGH" → ["HIGH"], "HIGH,CRITICAL" → ["HIGH", "CRITICAL"], "30,40" → ["HIGH", "CRITICAL"] + """ + if not risk_level_param: + return None + values = [v.strip() for v in str(risk_level_param).split(",") if v.strip()] + result: list[str] = [] + for v in values: + severity_name = RISK_LEVEL_TO_SEVERITY_NAME.get(v) or RISK_LEVEL_TO_SEVERITY_NAME.get(v.upper()) + if severity_name and severity_name not in result: + result.append(severity_name) + elif v.upper() not in result: + demisto.debug(f"Unknown risk level value '{v}', using upper-cased as-is") + result.append(v.upper()) + return result or None + def create_filter(field, value): return {"field": field, "stringContains": {"value": value}} @@ -44,7 +86,7 @@ def extract_response(response: Any) -> list[dict[str, Any]]: demisto.error(f"got bad response, {response}") raise Exception(f"got bad response, {response}") else: - demisto.info(f"Count of entites: {response.get('getTableResponse').get('totalNumberOfResults')}") + demisto.info(f"Count of entities: {response.get('getTableResponse').get('totalNumberOfResults')}") entities = response.get("getTableResponse", {}).get("data", {}).get("rows", []) demisto.info(f"Got {len(entities)} entities") return entities @@ -62,132 +104,235 @@ def __init__(self, api_token: str, base_url: str, verify: bool, proxy): }, ) + # ── External API helpers ────────────────────────────────────────────────── + + def _rate_limited_request(self, method: str, url_suffix: str, **kwargs) -> dict[str, Any]: + """Wrap _http_request with automatic 429 retry and exponential backoff. + + Respects a Retry-After header when present; otherwise backs off + RATE_LIMIT_RETRY_BASE_DELAY * 2^attempt seconds between attempts. + Raises on the last attempt or on any non-429 error. + """ + for attempt in range(RATE_LIMIT_MAX_RETRIES): + try: + return self._http_request(method=method, url_suffix=url_suffix, **kwargs) + except DemistoException as exc: + is_rate_limited = ( + exc.res is not None + and getattr(exc.res, "status_code", None) == 429 + ) + if is_rate_limited and attempt < RATE_LIMIT_MAX_RETRIES - 1: + retry_after = int( + exc.res.headers.get( # type: ignore[union-attr] + "Retry-After", + RATE_LIMIT_RETRY_BASE_DELAY * (2 ** attempt), + ) + ) + demisto.debug( + f"Rate limited (429) on {url_suffix}, " + f"retrying in {retry_after}s (attempt {attempt + 1}/{RATE_LIMIT_MAX_RETRIES})" + ) + time.sleep(retry_after) + else: + raise + raise DemistoException(f"Max retries ({RATE_LIMIT_MAX_RETRIES}) exceeded for {url_suffix}") + + def _external_api_list( + self, + endpoint: str, + filters: str = "", + count: int = PAGE_SIZE, + start_index: int = 0, + ) -> dict[str, Any]: + """GET /external-api/{endpoint} with SCIM filter and pagination params.""" + params: dict[str, Any] = {"count": count, "startIndex": start_index} + if filters: + params["filters"] = filters + return self._rate_limited_request( + method="GET", + url_suffix=f"{EXTERNAL_API_BASE}/{endpoint}", + params=params, + timeout=RECO_API_TIMEOUT_IN_SECONDS, + ) + + def _external_api_post(self, endpoint: str, body: dict[str, Any]) -> dict[str, Any]: + """POST /external-api/{endpoint}.""" + return self._rate_limited_request( + method="POST", + url_suffix=f"{EXTERNAL_API_BASE}/{endpoint}", + data=json.dumps(body), + timeout=RECO_API_TIMEOUT_IN_SECONDS, + ) + + def _external_api_put(self, url_suffix: str, body: dict[str, Any]) -> dict[str, Any]: + """PUT to an arbitrary external-api path suffix.""" + return self._rate_limited_request( + method="PUT", + url_suffix=url_suffix, + data=json.dumps(body), + timeout=RECO_API_TIMEOUT_IN_SECONDS, + ) + + def _paginate_all( + self, + endpoint: str, + item_key: str, + filters: str = "", + page_size: int = 1000, + ) -> list[dict[str, Any]]: + """Fetch every page from a list endpoint, returning all items combined. + + Uses the `totalResults` field in the response to know when to stop. + Stops early if an empty page is returned (defensive guard against + totalResults being stale or the server having fewer items than reported). + """ + all_items: list[dict[str, Any]] = [] + start_index = 0 + while True: + response = self._external_api_list( + endpoint, + filters=filters, + count=page_size, + start_index=start_index, + ) + page_items = response.get(item_key, []) + if not page_items: + break + all_items.extend(page_items) + total = int(response.get("totalResults", 0) or response.get("total_results", 0) or 0) + demisto.debug(f"_paginate_all {endpoint}: fetched {len(all_items)}/{total}") + if total and len(all_items) >= total: + break + start_index += len(page_items) + return all_items + + # ── Alerts (external API) ───────────────────────────────────────────────── + def get_alerts( self, - risk_level: int | None = None, + risk_levels: list[str] | None = None, source: str | None = None, before: datetime | None = None, after: datetime | None = None, - limit: int = 1000, + limit: int = PAGE_SIZE, ) -> list[dict[str, Any]]: + """List threat alerts via the external API. + + risk_levels: list of severity names, e.g. ["HIGH", "CRITICAL"]. Multiple + values are combined with OR so a single call fetches all matching severities. """ - Fetch alerts from Reco API - :param risk_level: The risk level of the incidents to fetch - :param source: The source of the incidents to fetch - :param before: The maximum date of the incidents to fetch - :param after: The minimum date of the incidents to fetch - :param limit: int - :return: dict - """ - demisto.info("Get alerts, enter") - alerts: list[dict[str, Any]] = [] - params: dict[str, Any] = { - "getTableRequest": { - "tableName": ALERT_VIEW_WITH_SHARED_STATUS, - "scope": "data", - "pageSize": limit, - "fieldFilters": { - "relationship": FILTER_RELATIONSHIP_AND, - "filters": {"filters": []}, - }, - "fieldSorts": {"sorts": [{"sortBy": "updated_at", "sortDirection": "SORT_DIRECTION_ASC"}]}, - } - } - if risk_level: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - {"field": "risk_level", "stringEquals": {"value": risk_level}} - ) + filter_parts: list[str] = [] + + if risk_levels: + if len(risk_levels) == 1: + filter_parts.append(f'severity eq "{risk_levels[0]}"') + else: + # Use the SCIM `in` operator — cleaner than a chain of OR clauses + vals = ", ".join(f'"{r}"' for r in risk_levels) + filter_parts.append(f'severity in [{vals}]') + if source: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - {"field": "short_extraction_source", "stringEquals": {"value": source}} - ) - if before: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - { - "field": CREATED_AT_FIELD, - "before": {"value": before.strftime("%Y-%m-%dT%H:%M:%SZ")}, - } - ) + filter_parts.append(f'apps co "{source}"') if after: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - { - "field": CREATED_AT_FIELD, - "after": {"value": after.strftime("%Y-%m-%dT%H:%M:%SZ")}, - } - ) + filter_parts.append(f'createdAt gt "{after.strftime(DEMISTO_OCCURRED_FORMAT)}"') + if before: + filter_parts.append(f'createdAt lt "{before.strftime(DEMISTO_OCCURRED_FORMAT)}"') + + scim_filter = " and ".join(filter_parts) + demisto.debug(f"get_alerts SCIM filter: {scim_filter!r}") - demisto.debug(f"params: {params}") try: - response = self._http_request( - method="PUT", - url_suffix="/policy-subsystem/alert-inbox/table", - data=json.dumps(params), + response = self._external_api_list("alerts/list", filters=scim_filter, count=limit) + alerts = response.get("alerts", []) + demisto.info(f"get_alerts: fetched {len(alerts)} alerts (total={response.get('totalResults', '?')})") + return alerts + except Exception as e: + demisto.error(f"get_alerts error: {str(e)}") + return [] + + def get_single_alert(self, alert_id: str) -> dict[str, Any]: + """Get full alert detail including policy violations via the external API.""" + try: + response = self._rate_limited_request( + method="GET", + url_suffix=f"{EXTERNAL_API_BASE}/alert-details/{alert_id}", timeout=RECO_API_TIMEOUT_IN_SECONDS, ) - alerts = extract_response(response) + return response.get("alert", {}) except Exception as e: - demisto.error(f"Findings Request ReadTimeout error: {str(e)}") - demisto.info(f"done fetching RECO alerts, fetched {len(alerts)} alerts.") - return alerts + demisto.error(f"get_single_alert({alert_id}) error: {str(e)}") + raise - def get_single_alert(self, alert_id: str) -> Any: - """ - Fetch a single alert from Reco API - :param alert_id: The id of the alert to fetch - :return: dict - """ - demisto.info(f"Get single alert, enter, alert_id: {alert_id}") - alert: dict[str, Any] = {} + def get_alert_ai_summary(self, alert_id: str) -> dict[str, Any]: # pragma: no cover + """Get alert AI summary (internal API — no external equivalent yet).""" try: - response = self._http_request( + return self._http_request( method="GET", - url_suffix=f"/policy-subsystem/alert-inbox/{alert_id}", + url_suffix=f"/alert/summarize/{alert_id}", timeout=RECO_API_TIMEOUT_IN_SECONDS, ) - if response.get("alert") is None: - demisto.info(f"got bad response, {response}") - else: - demisto.info(f"got good response, {response}") - alert = response.get("alert", {}) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_alert_ai_summary error: {str(e)}") + raise - demisto.info(f"done fetching RECO alert, fetched {alert}") - return alert - - def update_reco_incident_timeline(self, incident_id: str, comment: str) -> Any: - """ - Update timeline of an incident. - """ - demisto.info("Update incident timeline, enter") + def change_alert_status(self, alert_id: str, status: str) -> Any: + """Change alert status (internal API — no external equivalent yet).""" try: - response = self._http_request( - method="POST", - url_suffix="/share-service/share-comment", + return self._http_request( + method="PUT", + url_suffix=f"/policy-subsystem/alert-inbox/{alert_id}/status/{status}", timeout=RECO_API_TIMEOUT_IN_SECONDS, - data=json.dumps( - { - "content": comment, - "entityId": incident_id, # alert_id - "entityType": "alert", - } - ), ) except Exception as e: - demisto.error(f"Update incident timeline error: {str(e)}") - raise e + demisto.error(f"change_alert_status error: {str(e)}") + raise - demisto.info(f"Comment added to timeline of incident {incident_id}") - return response + # ── Comments (external API) ─────────────────────────────────────────────── - def resolve_visibility_event(self, entity_id: str, label_name: str) -> Any: - """Resolve visibility event. - :param entity_id: The entry id of the visibility event to resolve - :param label_name: The label name of the visibility event to resolve + def update_reco_incident_timeline(self, incident_id: str, comment: str) -> Any: + """Add a comment to an alert entity via the external API.""" + body: dict[str, Any] = { + "entityId": incident_id, + "entityType": "alert", + "content": comment, + } + try: + return self._external_api_post("comments/create", body) + except Exception as e: + demisto.error(f"update_reco_incident_timeline error: {str(e)}") + raise + + # ── Labels (external API) ───────────────────────────────────────────────── + + def add_entity_label( + self, + entity_id: str, + entity_type: str, + label_name: str, + instance_id: str = "", + ) -> Any: + """Add a label to any Reco entity via the external API. + + entity_type: "account", "identity", "app", "posture", "saas-to-saas", + "ip-address", "ai-agent", "device" """ + body: dict[str, Any] = { + "entityId": entity_id, + "entityType": entity_type, + "labelName": label_name, + } + if instance_id: + body["instanceId"] = instance_id try: - response = self._http_request( + return self._external_api_post("labels/add", body) + except Exception as e: + demisto.error(f"add_entity_label error: {str(e)}") + raise + + def resolve_visibility_event(self, entity_id: str, label_name: str) -> Any: + """Resolve a visibility event (internal API — no external equivalent).""" + try: + return self._http_request( method="PUT", url_suffix="/set-label-status", timeout=RECO_API_TIMEOUT_IN_SECONDS, @@ -207,94 +352,106 @@ def resolve_visibility_event(self, entity_id: str, label_name: str) -> Any: ), ) except Exception as e: - demisto.error(f"Resolve visibility event error: {str(e)}") - raise e - - demisto.info(f"Visibility event {entity_id} resolved") - return response + demisto.error(f"resolve_visibility_event error: {str(e)}") + raise - def get_risky_users(self) -> list[dict[str, Any]]: - """Get risky users. Returns a list of risky users with analysis.""" - return self.get_identities(email_address=None, label=RISKY_USER) + # ── Identities / Users (external API) ──────────────────────────────────── def get_identities(self, email_address: Optional[str] = None, label: Optional[str] = None) -> list[dict[str, Any]]: - """ - Get identities from Reco with specified filters. + """List identities from the external API, optionally filtered by email.""" + filter_parts: list[str] = [] + if email_address: + filter_parts.append(f'email co "{email_address}"') + # label filter is handled separately via get_risky_users (ListAccounts) + scim_filter = " and ".join(filter_parts) + try: + response = self._external_api_list("users/list", filters=scim_filter, count=PAGE_SIZE) + return response.get("users", []) + except Exception as e: + demisto.error(f"get_identities error: {str(e)}") + raise - :param email_address: Optional email substring to filter identities. - :param label: Optional label value to filter identities. - :return: A dictionary representing the getTableRequest payload. - """ - params: Dict[str, Any] = { - "getTableRequest": { - "tableName": "RISK_MANAGEMENT_VIEW_IDENTITIES", - "pageSize": 50, - "fieldSorts": {"sorts": [{"sortBy": "primary_email_address", "sortDirection": "SORT_DIRECTION_ASC"}]}, - "fieldFilters": { - "relationship": "FILTER_RELATIONSHIP_AND", - "fieldFilterGroups": {"fieldFilters": []}, - "forceEstimateSize": True, - }, - } - } + def get_risky_users(self) -> list[dict[str, Any]]: + """List ALL accounts flagged as risky via the external API (auto-paginated).""" + try: + return self._paginate_all("accounts/list", "accounts", filters='isRiskyUser eq true') + except Exception as e: + demisto.error(f"get_risky_users error: {str(e)}") + raise - # Add label filter if provided - if label is not None: - label_filter = { - "relationship": "FILTER_RELATIONSHIP_OR", - "filters": { - "filters": [ - { - "field": "labels", - "labelNameEquals": { - "keys": ["identity_id"], - "value": [label], - "filterColumn": "label_name", - "entryTypes": ["ENTRY_TYPE_IDENTITY"], - }, - } - ] - }, - } - params["getTableRequest"]["fieldFilters"]["fieldFilterGroups"]["fieldFilters"].append(label_filter) + def get_user_context_by_email_address(self, email_address: str) -> list[dict[str, Any]]: + """Get identity context for an email address via the external API.""" + try: + response = self._external_api_list( + "users/list", + filters=f'email co "{email_address}"', + count=10, + ) + return response.get("users", []) + except Exception as e: + demisto.error(f"get_user_context_by_email_address error: {str(e)}") + raise - # Add email address filter if provided - if email_address: - email_filter = { - "relationship": "FILTER_RELATIONSHIP_OR", - "filters": { - "filters": [create_filter("full_name", email_address), create_filter("primary_email_address", email_address)] - }, - } - params["getTableRequest"]["fieldFilters"]["fieldFilterGroups"]["fieldFilters"].append(email_filter) + # ── Apps (external API) ─────────────────────────────────────────────────── + def get_app_discovery( + self, + before: datetime | None = None, + after: datetime | None = None, + limit: int | None = None, + ) -> list[dict[str, Any]]: + """List discovered apps via the external API. Auto-paginates all results when limit is None.""" + filter_parts: list[str] = [] + if after: + filter_parts.append(f'lastSeen gt "{after.strftime(DEMISTO_OCCURRED_FORMAT)}"') + if before: + filter_parts.append(f'lastSeen lt "{before.strftime(DEMISTO_OCCURRED_FORMAT)}"') + scim_filter = " and ".join(filter_parts) try: - response = self._http_request( - method="PUT", - url_suffix="/risk-management/get-risk-management-table", - timeout=RECO_API_TIMEOUT_IN_SECONDS, - data=json.dumps(params), - ) - return extract_response(response) + if limit is None: + return self._paginate_all("apps/list", "apps", filters=scim_filter) + response = self._external_api_list("apps/list", filters=scim_filter, count=limit) + return response.get("apps", []) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_app_discovery error: {str(e)}") + raise - def get_alert_ai_summary(self, alert_id: str) -> dict[str, Any]: # pragma: no cover - """Get alert AI summary.""" + def set_app_authorization_status(self, app_id: str, authorization_status: str) -> Any: + """Update app authorization status via the external API.""" try: - response = self._http_request( - method="GET", - url_suffix=f"/alert/summarize/{alert_id}", - timeout=RECO_API_TIMEOUT_IN_SECONDS, + return self._external_api_put( + f"{EXTERNAL_API_BASE}/apps/{app_id}/auth-status", + {"authorizationStatus": authorization_status}, ) - return response except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"set_app_authorization_status error: {str(e)}") + raise + + # ── Files / Assets (external API) ──────────────────────────────────────── + + def get_sensitive_assets_information( + self, asset_name: str | None, asset_id: str | None, sensitive_only: bool, regex_search: bool + ) -> list[dict[str, Any]]: + """List files from the external API, optionally filtered by name or id (auto-paginated).""" + filter_parts: list[str] = [] + if asset_name: + op = "co" if regex_search else "eq" + filter_parts.append(f'name {op} "{asset_name}"') + elif asset_id: + filter_parts.append(f'id eq "{asset_id}"') + if sensitive_only: + filter_parts.append('sensitivityLevel in ["30","40"]') + scim_filter = " and ".join(filter_parts) + try: + return self._paginate_all("files/list", "files", filters=scim_filter) + except Exception as e: + demisto.error(f"get_sensitive_assets_information error: {str(e)}") + raise + + # ── Internal API methods (data risk management — no external equivalent) ── def get_exposed_publicly_files_at_risk(self) -> list[dict[str, Any]]: - """Get exposed publicly files at risk. Returns a list of exposed publicly files at risk with analysis.""" + """Get exposed publicly files at risk (internal API).""" params: Dict[str, Any] = { "getTableRequest": { "tableName": "DATA_RISK_MANAGEMENT_VIEW_BREAKDOWN_EXPOSED_PUBLICLY", @@ -315,11 +472,11 @@ def get_exposed_publicly_files_at_risk(self) -> list[dict[str, Any]]: ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_exposed_publicly_files_at_risk error: {str(e)}") + raise def get_files_exposed_to_email(self, email_account) -> list[dict[str, Any]]: - """Get files exposed to email. Returns a list of files exposed to email with analysis.""" + """Get files exposed to an email account (internal API).""" params = { "getTableRequest": { "tableName": "data_posture_view_files_by_emails_slider", @@ -360,11 +517,11 @@ def get_files_exposed_to_email(self, email_account) -> list[dict[str, Any]]: ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_files_exposed_to_email error: {str(e)}") + raise def get_list_of_private_emails_with_access(self) -> list[dict[str, Any]]: - """Get files exposed to email. Returns a list of private email addresses with access.""" + """Get private email addresses with file access (internal API).""" params = { "getTableRequest": { "tableName": "data_posture_view_private_email_with_access", @@ -386,10 +543,16 @@ def get_list_of_private_emails_with_access(self) -> list[dict[str, Any]]: ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_list_of_private_emails_with_access error: {str(e)}") + raise + + @staticmethod + def get_date_time_before_days_formatted(last_interaction_time_before_in_days: int) -> str: + thirty_days_ago = datetime.utcnow() - timedelta(days=last_interaction_time_before_in_days) + return thirty_days_ago.strftime("%Y-%m-%dT%H:%M:%S.999Z") def get_3rd_parties_risk_list(self, last_interaction_time_before_in_days: int) -> list[dict[str, Any]]: + """Get 3rd party domains with file access (internal API).""" formatted_date = self.get_date_time_before_days_formatted(last_interaction_time_before_in_days) params = { "getTableRequest": { @@ -428,19 +591,11 @@ def get_3rd_parties_risk_list(self, last_interaction_time_before_in_days: int) - ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e - - @staticmethod - def get_date_time_before_days_formatted(last_interaction_time_before_in_days: int) -> str: - # Calculate the date 30 days ago - thirty_days_ago = datetime.utcnow() - timedelta(days=last_interaction_time_before_in_days) - # Format the date in the desired format - formatted_date = thirty_days_ago.strftime("%Y-%m-%dT%H:%M:%S.999Z") - return formatted_date + demisto.error(f"get_3rd_parties_risk_list error: {str(e)}") + raise def get_files_shared_with_3rd_parties(self, domain: str, last_interaction_time_before_in_days: int) -> list[dict[str, Any]]: - """Get files shared with 3rd parties. Returns a list of files at risk with analysis.""" + """Get files shared with a specific 3rd party domain (internal API).""" formatted_date = self.get_date_time_before_days_formatted(last_interaction_time_before_in_days) params = { "getTableRequest": { @@ -475,11 +630,11 @@ def get_files_shared_with_3rd_parties(self, domain: str, last_interaction_time_b ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_files_shared_with_3rd_parties error: {str(e)}") + raise def get_assets_user_has_access(self, email_address: str, only_sensitive: bool) -> list[dict[str, Any]]: - """Get assets user has access to. Returns a list of assets.""" + """Get assets a user has access to (internal API).""" params: dict[str, Any] = { "getTableRequest": { "tableName": "files_view", @@ -504,22 +659,14 @@ def get_assets_user_has_access(self, email_address: str, only_sensitive: bool) - }, } } - if only_sensitive is True: params["getTableRequest"]["fieldFilters"]["fieldFilterGroups"]["fieldFilters"].append( { "relationship": "FILTER_RELATIONSHIP_OR", "filters": { "filters": [ - { - "field": "sensitivity_level", - # 30 confidential - "stringEquals": {"value": "30"}, - }, - { - "field": "sensitivity_level", - "stringEquals": {"value": "40"}, - }, + {"field": "sensitivity_level", "stringEquals": {"value": "30"}}, + {"field": "sensitivity_level", "stringEquals": {"value": "40"}}, ] }, } @@ -533,11 +680,11 @@ def get_assets_user_has_access(self, email_address: str, only_sensitive: bool) - ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_assets_user_has_access error: {str(e)}") + raise def get_assets_shared_externally(self, email_address: str) -> list[dict[str, Any]]: - """Get assets user has access to. Returns a list of assets.""" + """Get assets an owner has shared externally (internal API).""" params: dict[str, Any] = { "getTableRequest": { "tableName": "files_view", @@ -579,393 +726,129 @@ def get_assets_shared_externally(self, email_address: str) -> list[dict[str, Any ) return extract_response(response) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e - - def get_user_context_by_email_address(self, email_address: str) -> list[dict[str, Any]]: - """Get user context by email address. Returns a dict of user context.""" - identities = self.get_identities(email_address=email_address) - if not identities: - return [] - identity_ids = [] - for user in identities: - user_as_dict = parse_table_row_to_dict(user.get("cells", {})) - identity_id = user_as_dict.get("identity_id") - if identity_id: - identity_ids.append(identity_id) - - params: Dict[str, Any] = { - "getTableRequest": { - "tableName": "RISK_MANAGEMENT_VIEW_IDENTITIES", - "pageSize": 1, - "fieldSorts": {"sorts": []}, - "fieldFilters": { - "relationship": "FILTER_RELATIONSHIP_OR", - "forceEstimateSize": True, - "filters": {"filters": []} if identity_ids else {}, - }, - } - } - - # Add filters for multiple identity_ids - if identity_ids: - identity_filters = [{"field": "identity_id", "stringEquals": {"value": identity_id}} for identity_id in identity_ids] - params["getTableRequest"]["fieldFilters"]["filters"]["filters"] = identity_filters - - response = self._http_request( - method="PUT", - url_suffix="/risk-management/get-risk-management-table", - timeout=RECO_API_TIMEOUT_IN_SECONDS * 2, - data=json.dumps(params), - ) - return extract_response(response) - - def get_sensitive_assets_information( - self, asset_name: str | None, asset_id: str | None, sensitive_only: bool, regex_search: bool - ) -> list[dict[str, Any]]: - """Get sensitive assets' information. Returns a list of assets.""" - filter = "regexCaseInsensitive" if regex_search else "stringEquals" - field_to_search = "file_name" if asset_name else "asset_id" - value_to_search = asset_name if asset_name else asset_id - params: dict[str, Any] = { - "getTableRequest": { - "tableName": "files_view", - "pageSize": PAGE_SIZE, - "fieldFilters": { - "relationship": "FILTER_RELATIONSHIP_AND", - "fieldFilterGroups": { - "fieldFilters": [ - { - "relationship": "FILTER_RELATIONSHIP_OR", - "filters": { - "filters": [ - { - "field": field_to_search, - filter: {"value": value_to_search}, - } - ] - }, - } - ] - }, - }, - } - } - if sensitive_only: - params["getTableRequest"]["fieldFilters"]["fieldFilterGroups"]["fieldFilters"].append( - { - "relationship": "FILTER_RELATIONSHIP_OR", - "filters": { - "filters": [ - { - "field": "sensitivity_level", - "stringEquals": {"value": "30"}, - }, - { - "field": "sensitivity_level", - "stringEquals": {"value": "40"}, - }, - ] - }, - } - ) - try: - response = self._http_request( - method="PUT", - url_suffix="/asset-management/query", - timeout=RECO_API_TIMEOUT_IN_SECONDS * 2, - data=json.dumps(params), - ) - return extract_response(response) - except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e + demisto.error(f"get_assets_shared_externally error: {str(e)}") + raise def get_link_to_user_overview_page(self, link_type: str, entity_id: str) -> str: - """ - Get link to user overview page. - :param link_type: The link type to get (RM_LINK_TYPE_USER). - :param entity_id: The entity id. In case of user, it's the user's email address. - :return: dict - """ - demisto.info(f"Getting link to {link_type} overview page for {entity_id}") + """Get a magic link to a Reco UI overview page (internal API).""" try: response = self._http_request( method="GET", url_suffix=f"/risk-management/risk-management/link?link_type={link_type}¶m={entity_id}", timeout=RECO_API_TIMEOUT_IN_SECONDS, ) - if response.get("link") is None: - demisto.info(f"got bad response, {response}") - else: - demisto.info(f"got good response, {response}") - link = response.get("link", None) + link = response.get("link", None) except Exception as e: - demisto.error(f"Validate API key ReadTimeout error: {str(e)}") - raise e - - demisto.info(f"Got link: {link}") # pylint: disable=E0606 - return link + demisto.error(f"get_link_to_user_overview_page error: {str(e)}") + raise + return link # type: ignore[return-value] def add_exclusion_filter(self, key_to_add: str, values_to_add: list[str]): + """Add exclusion filter values (internal API).""" body = {"environmentName": "string", "keyToAddTo": key_to_add, "valuesToAdd": values_to_add} - try: - response = self._http_request( + return self._http_request( method="POST", url_suffix="/algo/add_values_to_data_type_exclude_analyzer", timeout=RECO_API_TIMEOUT_IN_SECONDS * 2, data=json.dumps(body), ) - return response - except Exception as e: - demisto.error(f"Can't add exclusion filter: {str(e)}") - raise e - - def set_entry_label_relations(self, entry_id: str, label_name: str, label_status: str, entry_type: str) -> Any: - """Set entry label relations. - :param entry_id: The entry id to set (email_address, asset_id etc.) - :param label_name: The label name to set - :param label_status: The label_status to set. Can be one of the following: - LABEL_STATUS_INACTIVE, - LABEL_STATUS_ACTIVE, - LABEL_STATUS_RESOLVED, - LABEL_STATUS_FALSE_POSITIVE, - LABEL_STATUS_PENDING - :param entry_type: The entry type to set. Can be one of the following: ENTRY_TYPE_INCIDENT, - ENTRY_TYPE_PROCESS, - ENTRY_TYPE_EVENT, - ENTRY_TYPE_USER, - ENTRY_TYPE_ASSET, - ENTRY_TYPE_PLAYBOOK - """ - try: - response = self._http_request( - method="PUT", - url_suffix="/entry-label-relations", - timeout=RECO_API_TIMEOUT_IN_SECONDS, - data=json.dumps( - { - "labelRelations": [ - { - "labelName": label_name, - "entryId": entry_id, - "count": 1, - "confidence": 1, - "entryType": entry_type, - "labelStatus": label_status, - "attributes": {}, - } - ] - } - ), - ) except Exception as e: - demisto.error(f"Set entry label relations error: {str(e)}") - raise e - demisto.info(f"Label {label_name} set to {label_status} for event {entry_id}") - return response + demisto.error(f"add_exclusion_filter error: {str(e)}") + raise - def change_alert_status(self, alert_id: str, status: str) -> Any: - """Change alert status.""" - try: - response = self._http_request( - method="PUT", - url_suffix=f"/policy-subsystem/alert-inbox/{alert_id}/status/{status}", - timeout=RECO_API_TIMEOUT_IN_SECONDS, - ) - except Exception as e: - demisto.error(f"Change alert status error: {str(e)}") - raise e - demisto.info(f"Alert {alert_id} status changed to {status}") - return response + # ── New external API list methods ───────────────────────────────────────── - def validate_api_key(self) -> str: - """ - Validate API key - :return: bool - """ - demisto.info("Validate API key, enter") - invalid_token_string = "Invalid token" - try: - response = self._http_request( - method="GET", - url_suffix="/policy-subsystem/alert-inbox?limit=1", - timeout=RECO_API_TIMEOUT_IN_SECONDS, - ) - if response.get("alerts") is None: - demisto.info(f"got bad response, {response}") - else: - demisto.info(f"got good response, {response}") - return "ok" - except Exception as e: - demisto.error(f"Validate API key Error: {str(e)}") - raise e - return invalid_token_string + def list_events(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List SaaS events via the external API.""" + return self._external_api_list("events/list", filters=filters, count=count) - def get_app_discovery( - self, - before: datetime | None = None, - after: datetime | None = None, - limit: int = 1000, - ) -> list[dict[str, Any]]: - """ - Fetch app discovery data from Reco API with pagination support - :param before: The maximum date of the apps to fetch - :param after: The minimum date of the apps to fetch - :param limit: int (page size for each request) - :return: list - """ - demisto.info("Get app discovery, enter") - all_apps: list[dict[str, Any]] = [] - page_size = min(limit, PAGE_SIZE) # Use PAGE_SIZE (1000) as max page size - page_number = 0 + def list_posture_issues(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List posture issues via the external API.""" + return self._external_api_list("posture-issues/list", filters=filters, count=count) - # First, get the total count to determine how many pages we need - count_params = self._create_app_discovery_params(page_size, page_number, before, after) + def list_accounts(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List accounts via the external API.""" + return self._external_api_list("accounts/list", filters=filters, count=count) - demisto.info("Getting total count of apps from Reco API") - try: - count_response = self._http_request( - method="PUT", - url_suffix="/asset-management/count", - data=json.dumps(count_params), - timeout=RECO_API_TIMEOUT_IN_SECONDS, - ) - total_count = count_response.get("getTableResponse", {}).get("totalNumberOfResults", 0) - demisto.debug(f"Total number of apps: {total_count}") - except Exception as e: - demisto.error(f"Failed to get app count: {str(e)}") - total_count = 0 + def list_devices(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List devices via the external API.""" + return self._external_api_list("devices/list", filters=filters, count=count) - if total_count == 0: - demisto.info("No apps found") - return all_apps + def list_ai_agents(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List AI agents via the external API.""" + return self._external_api_list("ai-agents/list", filters=filters, count=count) - # Calculate total pages needed - total_pages = (total_count + page_size - 1) // page_size - demisto.info(f"Will fetch {total_pages} pages with page size {page_size}") + def list_integrations(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List connected integrations via the external API.""" + return self._external_api_list("integrations/list", filters=filters, count=count) - # Fetch all pages - for page_number in range(total_pages): - demisto.info(f"Fetching page {page_number + 1} of {total_pages}") + def list_groups(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List SaaS groups via the external API.""" + return self._external_api_list("groups/list", filters=filters, count=count) - params = self._create_app_discovery_params(page_size, page_number, before, after) + def list_saas_to_saas(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List SaaS-to-SaaS grants via the external API.""" + return self._external_api_list("saas-to-saas/list", filters=filters, count=count) - try: - response = self._http_request( - method="PUT", - url_suffix="/asset-management/query", - data=json.dumps(params), - timeout=RECO_API_TIMEOUT_IN_SECONDS, - ) - apps = extract_response(response) + def list_ip_addresses(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List observed IP addresses via the external API.""" + return self._external_api_list("ip-addresses/list", filters=filters, count=count) - # Add metadata to each app - for app in apps: - if isinstance(app, dict): - app["total_apps_count"] = total_count - app["data_source"] = "app_discovery" - app["page_number"] = page_number + def list_business_units(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List external business units via the external API.""" + return self._external_api_list("business-units/list", filters=filters, count=count) - all_apps.extend(apps) - demisto.info(f"Fetched {len(apps)} apps from page {page_number + 1}") + def list_audit_logs(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List platform audit logs via the external API.""" + return self._external_api_list("audit-logs/list", filters=filters, count=count) - except Exception as e: - demisto.error(f"App Discovery Request error on page {page_number + 1}: {str(e)}") - # Continue with next page instead of failing completely - continue + def list_posture_checks(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List posture check definitions via the external API.""" + return self._external_api_list("posture-checks/list", filters=filters, count=count) - demisto.info(f"Done fetching RECO app discovery, fetched {len(all_apps)} total apps") - return all_apps + def list_threat_detection_policies(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List threat detection policies via the external API.""" + return self._external_api_list("policies/list", filters=filters, count=count) - def _create_app_discovery_params( - self, page_size: int, page_number: int, before: datetime | None = None, after: datetime | None = None - ) -> dict[str, Any]: - """Create request parameters for app discovery with pagination.""" - params: dict[str, Any] = { - "getTableRequest": { - "tableName": "app_discovery", - "pageSize": page_size, - "pageNumber": page_number, - "fieldSorts": {"sorts": [{"sortBy": "updated_at", "sortDirection": "SORT_DIRECTION_ASC"}]}, - } - } + def list_exclusions(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List alert suppression exclusion rules via the external API.""" + return self._external_api_list("exclusions/list", filters=filters, count=count) - # Add time filters if provided - if before or after: - params["getTableRequest"]["fieldFilters"] = { - "relationship": FILTER_RELATIONSHIP_AND, - "filters": {"filters": []}, - } + def list_app_instances(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List integrated app instances (portfolio) via the external API.""" + return self._external_api_list("app-instances/list", filters=filters, count=count) - if before: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - { - "field": "updated_at", - "before": {"value": before.strftime("%Y-%m-%dT%H:%M:%SZ")}, - } - ) - if after: - params["getTableRequest"]["fieldFilters"]["filters"]["filters"].append( - { - "field": "updated_at", - "after": {"value": after.strftime("%Y-%m-%dT%H:%M:%SZ")}, - } - ) - - return params + def list_labeled_netapp_files(self, filters: str = "", count: int = PAGE_SIZE) -> dict[str, Any]: + """List NetApp files carrying an active label via the external API.""" + return self._external_api_list("labeled-netapp-files/list", filters=filters, count=count) - def set_app_authorization_status(self, app_id: str, authorization_status: str) -> Any: - """ - Set authorization status for an application in Reco - :param app_id: The application ID to update - :param authorization_status: The authorization status to set - :return: dict - """ - demisto.info(f"Setting app authorization status for {app_id} to {authorization_status}") - - params = {"appAuth": [{"appId": app_id, "authorizationStatus": authorization_status}]} + # ── API key validation ──────────────────────────────────────────────────── + def validate_api_key(self) -> str: + """Validate the API token by making a minimal external API call.""" try: - response = self._http_request( - method="PUT", - url_suffix="/app-risk-management/insert-risk-management-app", - data=json.dumps(params), - timeout=RECO_API_TIMEOUT_IN_SECONDS, - ) - - # Validate response format - if not isinstance(response, dict) or "rows" not in response: - demisto.error(f"Unexpected response format: {response}") - raise Exception(f"Unexpected response format: {response}") - - rows_updated = response.get("rows", 0) - if rows_updated != 1: - demisto.debug(f"Expected 1 row to be updated, but got {rows_updated}") - + self._external_api_list("alerts/list", count=1) + return "ok" except Exception as e: - demisto.error(f"Set app authorization status error: {str(e)}") - raise e + demisto.error(f"validate_api_key error: {str(e)}") + raise - demisto.debug(f"App {app_id} authorization status updated to {authorization_status}. Rows updated: {rows_updated}") - return response + +# ── Helpers ──────────────────────────────────────────────────────────────────── def parse_table_row_to_dict(alert: list[dict[str, Any]]) -> dict[str, Any]: + """Decode a table row returned by Reco's internal Table API (base64-encoded cells).""" if alert is None: return {} - alert_as_dict = {} for obj in alert: key = obj.get("key", None) value = obj.get("value", None) - if key is None: - continue - if value is None: + if key is None or value is None: continue obj[key] = base64.b64decode(value).decode("utf-8") - # Remove " from the beginning and end of the string if key != "labels": obj[key] = obj[key].replace('"', "") if key in ["updated_at", "created_at", "event_time"]: @@ -979,232 +862,244 @@ def parse_table_row_to_dict(alert: list[dict[str, Any]]) -> dict[str, Any]: try: obj[key] = int(obj[key]) except Exception: - demisto.info(f"Could not parse risk level {obj[key]} to int") + pass if key in ["group", "job_title", "departments", "labels"]: try: obj[key] = json.loads(obj[key]) except Exception: pass alert_as_dict[key] = obj[key] - return alert_as_dict +# ── Alert fetching ───────────────────────────────────────────────────────────── + + def get_alerts( reco_client: RecoClient, - risk_level: int | None = None, + risk_levels: list[str] | None = None, source: str | None = None, before: datetime | None = None, after: datetime | None = None, - limit: int = 1000, + limit: int = PAGE_SIZE, ) -> list[Any]: - """Get alerts from Reco. - :param reco_client: The Reco client - :param risk_level: The risk level to filter by - :param source: The source to filter by - :param before: The before time to filter by - :param after: The after time to filter by - :param limit: The limit of alerts to get - :return: The alerts""" - alerts = reco_client.get_alerts(risk_level, source, before, after, limit) - alerts_data = [] - for alert in alerts: - alert_as_dict = parse_table_row_to_dict(alert.get("cells", {})) - alert_id = alert_as_dict.get("id", None) - if alert_id is not None: - alert_id = base64.b64decode(alert_id).decode("utf-8") - single_alert = reco_client.get_single_alert(alert_id) - violations = single_alert["policyViolations"] - for violation in violations: - violation_data = json.loads(base64.b64decode(violation["jsonData"])) - if "violation" in violation_data: - violation_data.pop("violation") - violation["jsonData"] = violation_data - single_alert["policyViolations"] = json.loads(json.dumps(single_alert["policyViolations"])) - alerts_data.append(single_alert) - else: - demisto.info(f"Got alert without id: {alert_as_dict}") - return alerts_data - - -def get_risky_users_from_reco(reco_client: RecoClient) -> CommandResults: - """Get risky users from Reco.""" - risky_users = reco_client.get_risky_users() - users = [] - for user in risky_users: - user_as_dict = parse_table_row_to_dict(user.get("cells", {})) - users.append(user_as_dict) - return CommandResults( - readable_output=tableToMarkdown( - "Risky Users", - users, - headers=user_as_dict.keys(), - ), - outputs_prefix="Reco.RiskyUsers", - outputs_key_field="primary_email_address", - outputs=users, - raw_response=risky_users, - ) - - -def add_risky_user_label(reco_client: RecoClient, email_address: str) -> CommandResults: - """Add a risky user to Reco.""" - - users = reco_client.get_identities(email_address) - for user in users: - user_as_dict = parse_table_row_to_dict(user.get("cells", {})) - raw_response = reco_client.set_entry_label_relations( - user_as_dict["identity_id"], RISKY_USER, LABEL_STATUS_ACTIVE, ENTRY_TYPE_IDENTITY - ) - - return CommandResults( - raw_response=raw_response, - readable_output=f"User {email_address} labeled as risky", - ) - - -def add_leaving_org_user(reco_client: RecoClient, email_address: str) -> CommandResults: # pragma: no cover - """Tag user as leaving org.""" - users = reco_client.get_identities(email_address) - for user in users: - user_as_dict = parse_table_row_to_dict(user.get("cells", {})) - raw_response = reco_client.set_entry_label_relations( - user_as_dict["identity_id"], LEAVING_ORG_USER, LABEL_STATUS_ACTIVE, ENTRY_TYPE_IDENTITY - ) - - return CommandResults( - raw_response=raw_response, - readable_output=f"User {email_address} labeled as leaving org user", - ) - + """Fetch alert summaries then enrich each with full detail (policy violations). -def get_alert_ai_summary(reco_client: RecoClient, alert_id: str) -> CommandResults: - response = reco_client.get_alert_ai_summary(alert_id) - content = json.dumps(response) - if response.get("markdown"): - content = str(response.get("markdown")) + Returns a list of ThreatAlertDetail dicts from the external API. + """ + raw_alerts = reco_client.get_alerts(risk_levels, source, before, after, limit) + alerts_data: list[Any] = [] - return CommandResults( - readable_output=content, - outputs_prefix="Reco.AlertSummary", - outputs_key_field="alert_id", - outputs=response, - raw_response=response, - ) + for raw in raw_alerts: + alert_id = raw.get("id") + if not alert_id: + demisto.debug(f"Alert without id: {raw}") + continue + try: + detail = reco_client.get_single_alert(alert_id) + if not detail: + continue + # policy violations arrive as JSON strings (not base64) from the external API + for violation in detail.get("policyViolations", []): + json_data = violation.get("jsonData", "{}") + if isinstance(json_data, str): + try: + parsed = json.loads(json_data) + parsed.pop("violation", None) + violation["jsonData"] = parsed + except json.JSONDecodeError: + pass + alerts_data.append(detail) + except Exception as e: + demisto.error(f"Failed to enrich alert {alert_id}: {str(e)}") + demisto.info(f"get_alerts: enriched {len(alerts_data)}/{len(raw_alerts)} alerts") + return alerts_data -def enrich_incident(reco_client: RecoClient, single_incident: dict[str, Any]) -> dict[str, Any]: # pragma: no cover - alert_as_dict = parse_table_row_to_dict(single_incident.get("cells", {})) - return { - "name": alert_as_dict.get("incident_description", ""), - "occurred": alert_as_dict.get("event_time", ""), - "dbotMirrorId": alert_as_dict.get("incident_id", ""), - "rawJSON": json.dumps(alert_as_dict), - "severity": map_reco_score_to_demisto_score(reco_score=alert_as_dict.get("risk_level", DEMISTO_INFORMATIONAL)), - } +# ── Score mapping ────────────────────────────────────────────────────────────── -def map_reco_score_to_demisto_score( - reco_score: int, -) -> int | float: # pylint: disable=E1136 - """Map Reco numeric risk score (10-40) to Demisto severity.""" - # demisto_unknown = 0 (commented because of linter issues) +def map_reco_score_to_demisto_score(reco_score: int) -> int | float: demisto_informational = 0.5 - # demisto_low = 1 (commented because of linter issues) demisto_medium = 2 demisto_high = 3 demisto_critical = 4 - - # Reco API returns risk_level in range 10-40. Normalize to tier keys. - MAPPING = { - 40: demisto_critical, - 30: demisto_high, - 20: demisto_medium, - 10: demisto_informational, - 0: demisto_informational, - } + MAPPING = {40: demisto_critical, 30: demisto_high, 20: demisto_medium, 10: demisto_informational, 0: demisto_informational} tier = min(40, max(0, (reco_score // 10) * 10)) return MAPPING.get(tier, demisto_informational) -def map_reco_alert_score_to_demisto_score( - reco_score: str, -) -> int | float: # pylint: disable=E1136 - # demisto_unknown = 0 (commented because of linter issues) +def map_reco_alert_score_to_demisto_score(reco_score: str) -> int | float: demisto_informational = 0.5 - # demisto_low = 1 (commented because of linter issues) demisto_medium = 2 demisto_high = 3 demisto_critical = 4 - - # LHS is Reco score - MAPPING = { - "CRITICAL": demisto_critical, - "HIGH": demisto_high, - "MEDIUM": demisto_medium, - "LOW": demisto_informational, - } - - return MAPPING[reco_score] + MAPPING = {"CRITICAL": demisto_critical, "HIGH": demisto_high, "MEDIUM": demisto_medium, "LOW": demisto_informational} + return MAPPING.get(reco_score, demisto_informational) def _reco_risk_to_demisto_severity(raw_risk: Any) -> int | float: - """Normalize Reco risk_level to Demisto severity. Accepts int, str numbers ('10', '30'), or str labels ('HIGH', 'LOW').""" + """Normalize Reco risk level to XSOAR severity. Handles int, numeric str, or name str.""" if raw_risk is None: return map_reco_score_to_demisto_score(10) if isinstance(raw_risk, int): - risk_level = min(40, max(10, raw_risk)) - return map_reco_score_to_demisto_score(risk_level) + return map_reco_score_to_demisto_score(min(40, max(10, raw_risk))) if isinstance(raw_risk, str): stripped = raw_risk.strip().upper() if stripped in ("CRITICAL", "HIGH", "MEDIUM", "LOW"): return map_reco_alert_score_to_demisto_score(stripped) try: - risk_level = min(40, max(10, int(raw_risk))) - return map_reco_score_to_demisto_score(risk_level) + return map_reco_score_to_demisto_score(min(40, max(10, int(raw_risk)))) except (TypeError, ValueError): pass try: - risk_level = min(40, max(10, int(raw_risk))) - return map_reco_score_to_demisto_score(risk_level) + return map_reco_score_to_demisto_score(min(40, max(10, int(raw_risk)))) except (TypeError, ValueError): return map_reco_score_to_demisto_score(10) -def parse_incidents_objects( - reco_client: RecoClient, incidents_raw: list[dict[str, Any]] -) -> list[dict[str, Any]]: # pragma: no cover - demisto.info("parse_incidents_objects enter") +# ── Incident parsing ─────────────────────────────────────────────────────────── + + +def parse_alerts_to_incidents(alerts: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Convert ThreatAlertDetail dicts (external API) to XSOAR incident format. + + Also handles the legacy internal-API shape (cells[]) for backward compatibility. + """ incidents = [] - for single_incident in incidents_raw: - incident = enrich_incident(reco_client, single_incident=single_incident) + for alert in alerts: + # Support legacy row format (cells[]) and modern external API dicts + alert_dict = parse_table_row_to_dict(alert.get("cells", {})) if alert.get("cells") else alert + occurred = alert_dict.get("created_at") or alert_dict.get("createdAt", "") + raw_risk = alert_dict.get("risk_level") or alert_dict.get("riskLevel") or alert_dict.get("severity") + incident = { + "name": alert_dict.get("description", ""), + "occurred": occurred, + "dbotMirrorId": alert_dict.get("id", ""), + "rawJSON": json.dumps(alert), + "severity": _reco_risk_to_demisto_severity(raw_risk), + } incidents.append(incident) - - demisto.info(f"get_incidents_from_alerts: Got {len(incidents)} incidents") return incidents +def get_max_fetch(max_fetch: int) -> int: + return min(max_fetch, 500) + + +# ── Fetch incidents ──────────────────────────────────────────────────────────── + + +def fetch_incidents( + reco_client: RecoClient, + last_run: dict[str, Any], + max_fetch: int, + risk_levels: list[str] | None = None, + source: str | None = None, + before: datetime | None = None, + after: datetime | None = None, +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + demisto.info(f"fetch-incidents called {max_fetch=}") + next_run: dict[str, Any] = {} + + last_run_time = last_run.get("lastRun", None) + if last_run_time is not None: + after = dateutil.parser.parse(last_run_time) + + alerts = get_alerts(reco_client, risk_levels, source, before, after, max_fetch) + incidents = parse_alerts_to_incidents(alerts) + + existing_incidents = last_run.get("incident_ids", []) + incidents = [ + incident + for incident in incidents + if (incident.get("severity", 0) > DEMISTO_INFORMATIONAL) + and (incident.get("dbotMirrorId", None) not in existing_incidents) + ] + + incidents_sorted = sorted(incidents, key=lambda k: k["occurred"]) + if incidents_sorted: + last_occurred_dt = dateutil.parser.parse(incidents_sorted[-1]["occurred"]) + next_run["lastRun"] = (last_occurred_dt + timedelta(seconds=1)).strftime(DEMISTO_OCCURRED_FORMAT) + else: + next_run["lastRun"] = last_run_time + next_run["incident_ids"] = existing_incidents + [inc["dbotMirrorId"] for inc in incidents] + + return next_run, incidents + + +# ── Command functions ────────────────────────────────────────────────────────── + + +def get_risky_users_from_reco(reco_client: RecoClient) -> CommandResults: + """Return accounts flagged as risky.""" + accounts = reco_client.get_risky_users() + return CommandResults( + readable_output=tableToMarkdown( + "Risky Users", + accounts, + headers=["id", "name", "accountEmail", "permissions", "openAlerts", "roles", "isAdmin", "lastSeen"], + ), + outputs_prefix="Reco.RiskyUsers", + outputs_key_field="accountEmail", + outputs=accounts, + raw_response=accounts, + ) + + +def add_risky_user_label(reco_client: RecoClient, email_address: str) -> CommandResults: + """Tag an identity as risky via the external API labels endpoint.""" + identities = reco_client.get_identities(email_address=email_address) + if not identities: + return CommandResults(readable_output=f"No identity found for {email_address}") + raw_response = None + for identity in identities: + identity_id = identity.get("id") + if identity_id: + raw_response = reco_client.add_entity_label(identity_id, "identity", RISKY_USER) + return CommandResults( + raw_response=raw_response, + readable_output=f"User {email_address} labeled as risky", + ) + + +def add_leaving_org_user(reco_client: RecoClient, email_address: str) -> CommandResults: # pragma: no cover + """Tag an identity as a leaving-org user via the external API labels endpoint.""" + identities = reco_client.get_identities(email_address=email_address) + if not identities: + return CommandResults(readable_output=f"No identity found for {email_address}") + raw_response = None + for identity in identities: + identity_id = identity.get("id") + if identity_id: + raw_response = reco_client.add_entity_label(identity_id, "identity", LEAVING_ORG_USER) + return CommandResults( + raw_response=raw_response, + readable_output=f"User {email_address} labeled as leaving org user", + ) + + +def get_alert_ai_summary(reco_client: RecoClient, alert_id: str) -> CommandResults: + response = reco_client.get_alert_ai_summary(alert_id) + content = str(response.get("markdown")) if response.get("markdown") else json.dumps(response) + return CommandResults( + readable_output=content, + outputs_prefix="Reco.AlertSummary", + outputs_key_field="alert_id", + outputs=response, + raw_response=response, + ) + + def get_assets_user_has_access(reco_client: RecoClient, email_address: str, only_sensitive: bool) -> CommandResults: - """Get assets from Reco. If only_sensitive is True, only sensitive assets will be returned.""" assets = reco_client.get_assets_user_has_access(email_address, only_sensitive) - assets_list = [] - for asset in assets: - asset_as_dict = parse_table_row_to_dict(asset.get("cells", {})) - assets_list.append(asset_as_dict) + assets_list = [parse_table_row_to_dict(a.get("cells", {})) for a in assets] return CommandResults( readable_output=tableToMarkdown( "Assets", assets_list, - headers=[ - "file_name", - "file_owner", - "file_url", - "currently_permitted_users", - "visibility", - "location", - "source", - ], + headers=["file_name", "file_owner", "file_url", "currently_permitted_users", "visibility", "location", "source"], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_value", @@ -1214,25 +1109,13 @@ def get_assets_user_has_access(reco_client: RecoClient, email_address: str, only def get_sensitive_assets_shared_with_public_link(reco_client: RecoClient) -> CommandResults: - """Get sensitive assets shared with public link from Reco.""" assets = reco_client.get_exposed_publicly_files_at_risk() - assets_list = [] - for asset in assets: - asset_as_dict = parse_table_row_to_dict(asset.get("cells", {})) - assets_list.append(asset_as_dict) + assets_list = [parse_table_row_to_dict(a.get("cells", {})) for a in assets] return CommandResults( readable_output=tableToMarkdown( "Assets", assets_list, - headers=[ - "asset_id", - "asset", - "data_category", - "data_categories", - "last_access_date", - "visibility", - "location", - ], + headers=["asset_id", "asset", "data_category", "data_categories", "last_access_date", "visibility", "location"], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1241,27 +1124,14 @@ def get_sensitive_assets_shared_with_public_link(reco_client: RecoClient) -> Com ) -def get_assets_shared_externally_command(reco_client: RecoClient, email_address) -> CommandResults: - """Get assets shared externally .""" +def get_assets_shared_externally_command(reco_client: RecoClient, email_address: str) -> CommandResults: assets = reco_client.get_assets_shared_externally(email_address) - assets_list = [] - for asset in assets: - asset_as_dict = parse_table_row_to_dict(asset.get("cells", {})) - assets_list.append(asset_as_dict) + assets_list = [parse_table_row_to_dict(a.get("cells", {})) for a in assets] return CommandResults( readable_output=tableToMarkdown( "Assets", assets_list, - headers=[ - "asset_id", - "asset", - "data_category", - "data_categories", - "last_access_date", - "visibility", - "location", - "file_owner", - ], + headers=["asset_id", "asset", "data_category", "data_categories", "last_access_date", "visibility", "location", "file_owner"], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1271,27 +1141,13 @@ def get_assets_shared_externally_command(reco_client: RecoClient, email_address) def get_files_exposed_to_email_command(reco_client: RecoClient, email_account: str) -> CommandResults: - """Get files exposed to email. Returns a list of files exposed to email with analysis.""" assets = reco_client.get_files_exposed_to_email(email_account) - assets_list = [] - for asset in assets: - asset_as_dict = parse_table_row_to_dict(asset.get("cells", {})) - assets_list.append(asset_as_dict) + assets_list = [parse_table_row_to_dict(a.get("cells", {})) for a in assets] return CommandResults( readable_output=tableToMarkdown( "Assets", assets_list, - headers=[ - "asset_id", - "asset", - "data_category", - "data_categories", - "last_access_date", - "visibility", - "location", - "email_account", - "file_owner", - ], + headers=["asset_id", "asset", "data_category", "data_categories", "last_access_date", "visibility", "location", "email_account", "file_owner"], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1301,22 +1157,13 @@ def get_files_exposed_to_email_command(reco_client: RecoClient, email_account: s def get_3rd_parties_list(reco_client: RecoClient, last_interaction_time_in_days: int) -> CommandResults: - """Get 3rd parties list from Reco.""" domains = reco_client.get_3rd_parties_risk_list(last_interaction_time_in_days) - domains_list = [] - for domain in domains: - domain_as_dict = parse_table_row_to_dict(domain.get("cells", {})) - domains_list.append(domain_as_dict) + domains_list = [parse_table_row_to_dict(d.get("cells", {})) for d in domains] return CommandResults( readable_output=tableToMarkdown( "Domains", domains_list, - headers=[ - "domain", - "last_activity", - "files_num", - "users_with_access_num", - ], + headers=["domain", "last_activity", "files_num", "users_with_access_num"], ), outputs_prefix="Reco.Domains", outputs_key_field="domain", @@ -1328,26 +1175,13 @@ def get_3rd_parties_list(reco_client: RecoClient, last_interaction_time_in_days: def get_files_shared_with_3rd_parties( reco_client: RecoClient, domain: str, last_interaction_time_before_in_days: int ) -> CommandResults: - """Get files shared with 3rd parties from Reco.""" files = reco_client.get_files_shared_with_3rd_parties(domain, last_interaction_time_before_in_days) - files_list = [] - for file in files: - file_as_dict = parse_table_row_to_dict(file.get("cells", {})) - files_list.append(file_as_dict) + files_list = [parse_table_row_to_dict(f.get("cells", {})) for f in files] return CommandResults( readable_output=tableToMarkdown( "Files", files_list, - headers=[ - "domain", - "location", - "users", - "file_owner", - "data_category", - "asset", - "last_access_date", - "asset_id", - ], + headers=["domain", "location", "users", "file_owner", "data_category", "asset", "last_access_date", "asset_id"], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1356,232 +1190,357 @@ def get_files_shared_with_3rd_parties( ) +def assets_to_command_result(files: list[dict[str, Any]]) -> CommandResults: + """Convert File objects (external API) to CommandResults for asset commands.""" + return CommandResults( + readable_output=tableToMarkdown( + "Assets", + files, + headers=["id", "name", "owner", "url", "sensitivityLevel", "permissionVisibility", "location", "dataCategories"], + ), + outputs_prefix="Reco.SensitiveAssets", + outputs_key_field="id", + outputs=files, + raw_response=files, + ) + + def get_sensitive_assets_by_name(reco_client: RecoClient, asset_name: str, regex_search: bool) -> CommandResults: - """Get sensitive assets from Reco. If contains is True, the asset name will be searched as a regex.""" - assets = reco_client.get_sensitive_assets_information(asset_name, None, True, regex_search) - return assets_to_command_result(assets) + files = reco_client.get_sensitive_assets_information(asset_name, None, True, regex_search) + return assets_to_command_result(files) def get_assets_by_id(reco_client: RecoClient, asset_id: str) -> CommandResults: - """Get assets from Reco by file id.""" - assets = reco_client.get_sensitive_assets_information(None, asset_id, False, False) - return assets_to_command_result(assets) + files = reco_client.get_sensitive_assets_information(None, asset_id, False, False) + return assets_to_command_result(files) -def get_user_context_by_email_address(reco_client: RecoClient, email_address: str) -> CommandResults: - users_context = reco_client.get_user_context_by_email_address(email_address) - user_as_dict = None - headers = [] - if len(users_context) > 0: - user_as_dict = parse_table_row_to_dict(users_context[0].get("cells", {})) - headers = list(user_as_dict.keys()) +def get_sensitive_assets_by_id(reco_client: RecoClient, asset_id: str) -> CommandResults: + files = reco_client.get_sensitive_assets_information(None, asset_id, True, False) + return assets_to_command_result(files) + +def get_user_context_by_email_address(reco_client: RecoClient, email_address: str) -> CommandResults: + """Return identity context for an email address (external API).""" + users = reco_client.get_user_context_by_email_address(email_address) + user_data = users[0] if users else None return CommandResults( - readable_output=tableToMarkdown("User", user_as_dict, headers=headers), + readable_output=tableToMarkdown("User", user_data, headers=list(user_data.keys()) if user_data else []), outputs_prefix="Reco.User", - outputs_key_field="email_address", - outputs=user_as_dict, - raw_response=users_context, + outputs_key_field="email", + outputs=user_data, + raw_response=users, ) def add_exclusion_filter(reco_client: RecoClient, key_to_add: str, values: list[str]) -> CommandResults: # pragma: no cover - """Add exclusion filter to Reco.""" response = reco_client.add_exclusion_filter(key_to_add, values) return CommandResults(raw_response=response, readable_output="Exclusion filter added successfully") def change_alert_status(reco_client: RecoClient, alert_id: str, status: str) -> CommandResults: # pragma: no cover - """Change alert status.""" response = reco_client.change_alert_status(alert_id, status) return CommandResults(raw_response=response, readable_output=f"Alert {alert_id} status changed successfully to {status}") -def assets_to_command_result(assets: list[dict[str, Any]]) -> CommandResults: - """Convert assets to CommandResults.""" - assets_list = [] - for asset in assets: - asset_as_dict = parse_table_row_to_dict(asset.get("cells", {})) - assets_list.append(asset_as_dict) +def get_private_email_list_with_access(reco_client: RecoClient) -> CommandResults: + result = reco_client.get_list_of_private_emails_with_access() + identities_list = [parse_table_row_to_dict(i.get("cells", {})) for i in result] return CommandResults( readable_output=tableToMarkdown( - "Assets", - assets_list, - headers=[ - "file_name", - "file_owner", - "file_url", - "currently_permitted_users", - "visibility", - "location", - "source", - "sensitivity_level", - ], + "PrivateEmails", + identities_list, + headers=["email_account", "primary_email", "files_num", "user_category"], ), - outputs_prefix="Reco.SensitiveAssets", - outputs_key_field="asset_value", - outputs=assets_list, - raw_response=assets, + outputs_prefix="Reco.privateEmails", + outputs_key_field="email_account", + outputs=identities_list, + raw_response=result, ) -def get_sensitive_assets_by_id(reco_client: RecoClient, asset_id: str) -> CommandResults: - """Get sensitive assets from Reco by file id.""" - assets = reco_client.get_sensitive_assets_information(None, asset_id, True, False) - return assets_to_command_result(assets) - - def get_link_to_user_overview_page(reco_client: RecoClient, entity: str, link_type: str) -> CommandResults: link = reco_client.get_link_to_user_overview_page(link_type, entity) return CommandResults(outputs_prefix="Reco.Link", outputs={"link": link}, raw_response=link) -def fetch_incidents( - reco_client: RecoClient, - last_run: dict[str, Any], - max_fetch: int, - risk_level: int | None = None, - source: str | None = None, - before: datetime | None = None, - after: datetime | None = None, -) -> tuple[dict[str, Any], list[dict[str, Any]]]: - demisto.info(f"fetch-incidents called {max_fetch=}") - next_run = {} - incidents = [] - last_run_time = last_run.get("lastRun", None) - if last_run_time is not None: - after = dateutil.parser.parse(last_run_time) +def get_apps_command( + reco_client: RecoClient, before: datetime | None = None, after: datetime | None = None, limit: int = PAGE_SIZE +) -> CommandResults: + """List discovered apps from the external API.""" + apps = reco_client.get_app_discovery(before=before, after=after, limit=limit) + headers = ["id", "name", "category", "usersCount", "authorization", "isUsingAi", "vendorGrade", "aiCapability", "lastSeen"] + return CommandResults( + readable_output=tableToMarkdown("App Discovery", apps, headers=headers), + outputs_prefix="Reco.Apps", + outputs_key_field="id", + outputs=apps, + raw_response=apps, + ) - alerts = get_alerts(reco_client, risk_level, source, before, after, max_fetch) - incidents = parse_alerts_to_incidents(alerts) - existing_incidents = last_run.get("incident_ids", []) - incidents = [ - incident - for incident in incidents - if (incident.get("severity", 0) > DEMISTO_INFORMATIONAL) - and (incident.get("dbotMirrorId", None) not in existing_incidents) - ] # type: ignore +def set_app_authorization_status_command(reco_client: RecoClient, app_id: str, authorization_status: str) -> CommandResults: + """Update app authorization status via the external API.""" + reco_client.set_app_authorization_status(app_id, authorization_status) + return CommandResults( + readable_output=f"App {app_id} authorization status updated to {authorization_status}", + outputs_prefix="Reco.AppAuthorization", + outputs={"app_id": app_id, "authorization_status": authorization_status, "updated": True}, + ) - incidents_sorted = sorted(incidents, key=lambda k: k["occurred"]) - if incidents_sorted: - # Use the latest timestamp and add a 1-second buffer - last_occurred = incidents_sorted[-1]["occurred"] - last_occurred_dt = dateutil.parser.parse(last_occurred) - last_occurred_dt_plus_1 = last_occurred_dt + timedelta(seconds=1) - next_run["lastRun"] = last_occurred_dt_plus_1.strftime(DEMISTO_OCCURRED_FORMAT) - else: - next_run["lastRun"] = last_run_time - next_run["incident_ids"] = existing_incidents + [incident["dbotMirrorId"] for incident in incidents] - return next_run, incidents +# ── New command functions (external API) ────────────────────────────────────── -def parse_alerts_to_incidents(alerts: list[dict[str, Any]]) -> list[dict[str, Any]]: - """Map alert rows (ALERT_VIEW_WITH_SHARED_STATUS uses snake_case) to XSOAR incidents.""" - alerts_as_incidents = [] - for alert in alerts: - alert_dict = parse_table_row_to_dict(alert.get("cells", {})) if alert.get("cells") else alert - occurred = alert_dict.get("created_at") or alert_dict.get("createdAt", "") - raw_risk = alert_dict.get("risk_level") or alert_dict.get("riskLevel") - incident = { - "name": alert_dict.get("description", ""), - "occurred": occurred, - "dbotMirrorId": alert_dict.get("id", ""), - "rawJSON": json.dumps(alert), - "severity": _reco_risk_to_demisto_severity(raw_risk), - } - alerts_as_incidents.append(incident) - return alerts_as_incidents +def list_events_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List SaaS events from the external API.""" + response = reco_client.list_events(filters=filters, count=limit) + events = response.get("events", []) + flat: list[dict[str, Any]] = [] + for e in events: + row = dict(e) + actor = e.get("actor") or {} + row["actorEmail"] = actor.get("email", "") + row["actorName"] = actor.get("name", "") + flat.append(row) + return CommandResults( + readable_output=tableToMarkdown( + "Events", + flat, + headers=["id", "eventType", "formattedEventType", "application", "actorEmail", "eventTime", "outcomeString"], + ), + outputs_prefix="Reco.Events", + outputs_key_field="id", + outputs=events, + raw_response=response, + ) -def get_max_fetch(max_fetch: int) -> int: - if max_fetch > 500: - return 500 - return max_fetch +def list_posture_issues_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List posture issues from the external API.""" + response = reco_client.list_posture_issues(filters=filters, count=limit) + issues = response.get("issues", []) + return CommandResults( + readable_output=tableToMarkdown( + "Posture Issues", + issues, + headers=["id", "name", "severity", "checkStatus", "scorePercentage", "checkedInstance", "url"], + ), + outputs_prefix="Reco.PostureIssues", + outputs_key_field="id", + outputs=issues, + raw_response=response, + ) -def get_private_email_list_with_access(reco_client): - result = reco_client.get_list_of_private_emails_with_access() - identities_list = [] - for identity in result: - asset_as_dict = parse_table_row_to_dict(identity.get("cells", {})) - identities_list.append(asset_as_dict) +def list_accounts_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List accounts from the external API.""" + response = reco_client.list_accounts(filters=filters, count=limit) + accounts = response.get("accounts", []) return CommandResults( readable_output=tableToMarkdown( - "PrivateEmails", - identities_list, - headers=[ - "email_account", - "primary_email", - "files_num", - "user_category", - ], + "Accounts", + accounts, + headers=["id", "name", "accountEmail", "permissions", "hasMfa", "openAlerts", "isAdmin", "isRiskyUser", "lastSeen"], ), - outputs_prefix="Reco.privateEmails", - outputs_key_field="email_account", - outputs=identities_list, - raw_response=result, + outputs_prefix="Reco.Accounts", + outputs_key_field="id", + outputs=accounts, + raw_response=response, ) -def get_apps_command( - reco_client: RecoClient, before: datetime | None = None, after: datetime | None = None, limit: int = 1000 -) -> CommandResults: - """Get app discovery data from Reco.""" - apps = reco_client.get_app_discovery(before=before, after=after, limit=limit) - apps_list = [] - for app in apps: - app_as_dict = parse_table_row_to_dict(app.get("cells", {})) - apps_list.append(app_as_dict) +def list_devices_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List devices from the external API.""" + response = reco_client.list_devices(filters=filters, count=limit) + devices = response.get("devices", []) + return CommandResults( + readable_output=tableToMarkdown( + "Devices", + devices, + headers=["id", "name", "devicePlatform", "os", "osVersion", "isUnmanaged", "hasNonCompliant", "lastSeen"], + ), + outputs_prefix="Reco.Devices", + outputs_key_field="id", + outputs=devices, + raw_response=response, + ) - # Define headers for the table based on common app discovery fields - headers = ["app_name", "app_id", "category", "risk_score", "users_count", "data_access", "updated_at", "created_at", "status"] +def list_ai_agents_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List AI agents from the external API.""" + response = reco_client.list_ai_agents(filters=filters, count=limit) + agents = response.get("aiAgents", []) return CommandResults( readable_output=tableToMarkdown( - "App Discovery", - apps_list, - headers=headers, + "AI Agents", + agents, + headers=["id", "name", "vendor", "type", "authorization", "agentStatus", "risk", "lastUsage"], ), - outputs_prefix="Reco.Apps", - outputs_key_field="app_id", - outputs=apps_list, - raw_response=apps, + outputs_prefix="Reco.AiAgents", + outputs_key_field="id", + outputs=agents, + raw_response=response, ) -def set_app_authorization_status_command(reco_client: RecoClient, app_id: str, authorization_status: str) -> CommandResults: - """Set app authorization status in Reco.""" - response = reco_client.set_app_authorization_status(app_id, authorization_status) +def list_groups_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List SaaS groups from the external API.""" + response = reco_client.list_groups(filters=filters, count=limit) + groups = response.get("groups", []) + return CommandResults( + readable_output=tableToMarkdown( + "Groups", + groups, + headers=["id", "name", "email", "membersCount", "appsCount"], + ), + outputs_prefix="Reco.Groups", + outputs_key_field="id", + outputs=groups, + raw_response=response, + ) - rows_updated = response.get("rows", 0) - success = rows_updated == 1 - if success: - readable_message = f"App {app_id} authorization status updated successfully to {authorization_status}" - else: - readable_message = f"App {app_id} authorization status update completed with {rows_updated} rows affected" +def list_saas_to_saas_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List SaaS-to-SaaS grants from the external API.""" + response = reco_client.list_saas_to_saas(filters=filters, count=limit) + grants = response.get("saasToSaas", []) + return CommandResults( + readable_output=tableToMarkdown( + "SaaS-to-SaaS Grants", + grants, + headers=["id", "plugin", "authorization", "permissionRisk", "accounts", "aiCapability", "lastSeen"], + ), + outputs_prefix="Reco.SaasToSaas", + outputs_key_field="id", + outputs=grants, + raw_response=response, + ) + +def list_ip_addresses_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List observed IP addresses from the external API.""" + response = reco_client.list_ip_addresses(filters=filters, count=limit) + ips = response.get("ipAddresses", []) return CommandResults( + readable_output=tableToMarkdown( + "IP Addresses", + ips, + headers=["ipAddress", "country", "asnName", "eventsCount", "usersCount", "hasVpn", "hasProxy", "lastEventTime"], + ), + outputs_prefix="Reco.IpAddresses", + outputs_key_field="ipAddress", + outputs=ips, raw_response=response, - readable_output=readable_message, - outputs_prefix="Reco.AppAuthorization", - outputs={ - "app_id": app_id, - "authorization_status": authorization_status, - "updated": success, - "rows_affected": rows_updated, - }, ) -def main() -> None: # pragma: no cover - """main function, parses params and runs command functions +def list_business_units_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List external business units from the external API.""" + response = reco_client.list_business_units(filters=filters, count=limit) + bus = response.get("businessUnits", []) + return CommandResults( + readable_output=tableToMarkdown( + "Business Units", + bus, + headers=["id", "name", "manager", "createdAt"], + ), + outputs_prefix="Reco.BusinessUnits", + outputs_key_field="id", + outputs=bus, + raw_response=response, + ) - :return: - :rtype: - """ + +def list_audit_logs_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List platform audit logs from the external API.""" + response = reco_client.list_audit_logs(filters=filters, count=limit) + logs = response.get("auditLogs", []) + return CommandResults( + readable_output=tableToMarkdown( + "Audit Logs", + logs, + headers=["id", "userEmail", "module", "action", "objectName", "timestamp", "remoteAddr"], + ), + outputs_prefix="Reco.AuditLogs", + outputs_key_field="id", + outputs=logs, + raw_response=response, + ) + + +def list_posture_checks_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List posture check definitions from the external API.""" + response = reco_client.list_posture_checks(filters=filters, count=limit) + checks = response.get("postureChecks", []) + return CommandResults( + readable_output=tableToMarkdown( + "Posture Checks", + checks, + headers=["id", "name", "severity", "policyType", "apps", "type"], + ), + outputs_prefix="Reco.PostureChecks", + outputs_key_field="id", + outputs=checks, + raw_response=response, + ) + + +def list_threat_detection_policies_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List threat detection policies from the external API.""" + response = reco_client.list_threat_detection_policies(filters=filters, count=limit) + policies = response.get("policies", []) + return CommandResults( + readable_output=tableToMarkdown( + "Threat Detection Policies", + policies, + headers=["id", "name", "severity", "status", "apps", "openAlerts", "type"], + ), + outputs_prefix="Reco.ThreatDetectionPolicies", + outputs_key_field="id", + outputs=policies, + raw_response=response, + ) + + +def list_exclusions_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List alert suppression exclusion rules from the external API.""" + response = reco_client.list_exclusions(filters=filters, count=limit) + exclusions = response.get("exclusions", []) + return CommandResults( + readable_output=tableToMarkdown( + "Exclusions", + exclusions, + headers=["id", "name", "policyName", "apps", "createdBy", "createdAt"], + ), + outputs_prefix="Reco.Exclusions", + outputs_key_field="id", + outputs=exclusions, + raw_response=response, + ) + + +def list_app_instances_command(reco_client: RecoClient, filters: str = "", limit: int = PAGE_SIZE) -> CommandResults: + """List integrated app instances (portfolio) from the external API.""" + response = reco_client.list_app_instances(filters=filters, count=limit) + instances = response.get("appInstances", []) + return CommandResults( + readable_output=tableToMarkdown( + "App Instances", + instances, + headers=["id", "name", "instanceType", "accountsCount", "isUsingAi", "saasToSaasCount", "filesCount"], + ), + outputs_prefix="Reco.AppInstances", + outputs_key_field="id", + outputs=instances, + raw_response=response, + ) + + +# ── Main ─────────────────────────────────────────────────────────────────────── + + +def main() -> None: # pragma: no cover try: command = demisto.command() demisto.debug(f"Reco Command being called is {command}") @@ -1604,156 +1563,181 @@ def main() -> None: # pragma: no cover verify=verify_certificate, proxy=proxy, ) + if command == "fetch-incidents": - risk_level = params.get("risk_level") + # risk_level accepts a single value or comma-separated list (PRO-303) + risk_levels = parse_risk_levels(params.get("risk_level")) source = params.get("source") before = params.get("before") - after = params.get("after") + after: datetime | None = None - # How much time before the first fetch to retrieve incidents if arg := params.get("first_fetch"): - first_fetch_time_stamp = dateparser.parse(arg) - if first_fetch_time_stamp: - after = first_fetch_time_stamp + first_fetch_ts = dateparser.parse(arg) + if first_fetch_ts: + after = first_fetch_ts next_run, incidents = fetch_incidents( reco_client, last_run=demisto.getLastRun(), max_fetch=max_fetch, - risk_level=risk_level, + risk_levels=risk_levels, source=source, before=before, after=after, ) demisto.setLastRun(next_run) demisto.incidents(incidents) + elif command == "reco-add-comment-to-alert": incident_id = demisto.args()["alert_id"] response = reco_client.update_reco_incident_timeline( incident_id=incident_id, comment=demisto.args()["comment"], ) - return_results( - CommandResults( - raw_response=response, - readable_output=f"Commented added to alert {incident_id}", - ) - ) + return_results(CommandResults(raw_response=response, readable_output=f"Comment added to alert {incident_id}")) + elif command == "reco-update-incident-timeline": + incident_id = demisto.args()["incident_id"] response = reco_client.update_reco_incident_timeline( - incident_id=demisto.args()["incident_id"], comment=demisto.args()["comment"] - ) - return_results( - CommandResults( - raw_response=response, - readable_output=f"Timeline updated successfully for incident {incident_id}", - ) + incident_id=incident_id, + comment=demisto.args()["comment"], ) + return_results(CommandResults(raw_response=response, readable_output=f"Timeline updated for incident {incident_id}")) + elif command == "reco-resolve-visibility-event": entity_id = demisto.args()["entity_id"] label_name = demisto.args()["label_name"] response = reco_client.resolve_visibility_event(entity_id=entity_id, label_name=label_name) - return_results( - CommandResults( - raw_response=response, - readable_output=f"Visibility event {entity_id} resolved successfully", - ) - ) + return_results(CommandResults(raw_response=response, readable_output=f"Visibility event {entity_id} resolved")) + elif command == "test-module": - test_res = reco_client.validate_api_key() - return_results(test_res) + return_results(reco_client.validate_api_key()) + elif command == "reco-get-risky-users": - result = get_risky_users_from_reco(reco_client) - return_results(result) + return_results(get_risky_users_from_reco(reco_client)) + elif command == "reco-add-risky-user-label": - email_address = demisto.args()["email_address"] - result = add_risky_user_label(reco_client, email_address) - return_results(result) + return_results(add_risky_user_label(reco_client, demisto.args()["email_address"])) + elif command == "reco-add-leaving-org-user-label": - email_address = demisto.args()["email_address"] - result = add_leaving_org_user(reco_client, email_address) - return_results(result) + return_results(add_leaving_org_user(reco_client, demisto.args()["email_address"])) + elif command == "reco-get-assets-user-has-access-to": - only_sensitive = demisto.args().get("only_sensitive", False) - result = get_assets_user_has_access( - reco_client, - demisto.args()["email_address"], - only_sensitive, + return_results( + get_assets_user_has_access( + reco_client, + demisto.args()["email_address"], + demisto.args().get("only_sensitive", False), + ) ) - return_results(result) + elif command == "reco-get-sensitive-assets-by-name": - regex_search = demisto.args().get("regex_search", False) - result = get_sensitive_assets_by_name( - reco_client, - demisto.args()["asset_name"], - regex_search, + return_results( + get_sensitive_assets_by_name( + reco_client, + demisto.args()["asset_name"], + demisto.args().get("regex_search", False), + ) ) - return_results(result) + elif command == "reco-get-sensitive-assets-by-id": - result = get_sensitive_assets_by_id(reco_client, demisto.args()["asset_id"]) - return_results(result) + return_results(get_sensitive_assets_by_id(reco_client, demisto.args()["asset_id"])) + elif command == "reco-get-link-to-user-overview-page": - result = get_link_to_user_overview_page( - reco_client, - demisto.args()["entity"], - demisto.args()["param"], - ) - return_results(result) + return_results(get_link_to_user_overview_page(reco_client, demisto.args()["entity"], demisto.args()["param"])) + elif command == "reco-get-sensitive-assets-with-public-link": - result = get_sensitive_assets_shared_with_public_link(reco_client) - return_results(result) + return_results(get_sensitive_assets_shared_with_public_link(reco_client)) + elif command == "reco-get-3rd-parties-accessible-to-data-list": - result = get_3rd_parties_list(reco_client, int(demisto.args()["last_interaction_time_in_days"])) - return_results(result) + return_results(get_3rd_parties_list(reco_client, int(demisto.args()["last_interaction_time_in_days"]))) + elif command == "reco-get-files-shared-with-3rd-parties": - result = get_files_shared_with_3rd_parties( - reco_client, demisto.args()["domain"], int(demisto.args()["last_interaction_time_in_days"]) + return_results( + get_files_shared_with_3rd_parties( + reco_client, + demisto.args()["domain"], + int(demisto.args()["last_interaction_time_in_days"]), + ) ) - return_results(result) + elif command == "reco-add-exclusion-filter": - result = add_exclusion_filter(reco_client, demisto.args()["key_to_add"], argToList(demisto.args()["values_to_add"])) - return_results(result) + return_results(add_exclusion_filter(reco_client, demisto.args()["key_to_add"], argToList(demisto.args()["values_to_add"]))) + elif command == "reco-change-alert-status": - result = change_alert_status(reco_client, demisto.args()["alert_id"], demisto.args()["status"]) - return_results(result) + return_results(change_alert_status(reco_client, demisto.args()["alert_id"], demisto.args()["status"])) + elif command == "reco-get-user-context-by-email-address": - result = get_user_context_by_email_address(reco_client, demisto.args()["email_address"]) - return_results(result) + return_results(get_user_context_by_email_address(reco_client, demisto.args()["email_address"])) + elif command == "reco-get-files-exposed-to-email-address": - result = get_files_exposed_to_email_command(reco_client, demisto.args()["email_address"]) - return_results(result) + return_results(get_files_exposed_to_email_command(reco_client, demisto.args()["email_address"])) + elif command == "reco-get-assets-shared-externally": - result = get_assets_shared_externally_command(reco_client, demisto.args()["email_address"]) - return_results(result) + return_results(get_assets_shared_externally_command(reco_client, demisto.args()["email_address"])) + elif command == "reco-get-private-email-list-with-access": - result = get_private_email_list_with_access(reco_client) - return_results(result) + return_results(get_private_email_list_with_access(reco_client)) + elif command == "reco-get-assets-by-id": - result = get_assets_by_id(reco_client, demisto.args()["asset_id"]) - return_results(result) + return_results(get_assets_by_id(reco_client, demisto.args()["asset_id"])) + elif command == "reco-get-alert-ai-summary": - result = get_alert_ai_summary(reco_client, demisto.args().get("alert_id", "")) - return_results(result) + return_results(get_alert_ai_summary(reco_client, demisto.args().get("alert_id", ""))) + elif command == "reco-get-apps": - # Parse datetime arguments if provided - before = None - after = None - before_str = demisto.args().get("before") - after_str = demisto.args().get("after") - if before_str: - before = dateparser.parse(before_str) - if after_str: - after = dateparser.parse(after_str) - limit = int(demisto.args().get("limit") or "1000") - result = get_apps_command(reco_client, before=before, after=after, limit=limit) - return_results(result) + before_dt = dateparser.parse(demisto.args()["before"]) if demisto.args().get("before") else None + after_dt = dateparser.parse(demisto.args()["after"]) if demisto.args().get("after") else None + return_results(get_apps_command(reco_client, before=before_dt, after=after_dt, limit=int(demisto.args().get("limit") or PAGE_SIZE))) + elif command == "reco-set-app-authorization-status": - app_id = demisto.args()["app_id"] - authorization_status = demisto.args()["authorization_status"] - result = set_app_authorization_status_command(reco_client, app_id, authorization_status) - return_results(result) + return_results(set_app_authorization_status_command(reco_client, demisto.args()["app_id"], demisto.args()["authorization_status"])) + + elif command == "reco-list-events": + return_results(list_events_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-posture-issues": + return_results(list_posture_issues_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-accounts": + return_results(list_accounts_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-devices": + return_results(list_devices_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-ai-agents": + return_results(list_ai_agents_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-groups": + return_results(list_groups_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-saas-to-saas": + return_results(list_saas_to_saas_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-ip-addresses": + return_results(list_ip_addresses_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-business-units": + return_results(list_business_units_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-audit-logs": + return_results(list_audit_logs_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-posture-checks": + return_results(list_posture_checks_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-threat-detection-policies": + return_results(list_threat_detection_policies_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-exclusions": + return_results(list_exclusions_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + + elif command == "reco-list-app-instances": + return_results(list_app_instances_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + else: raise NotImplementedError(f"{command} is not an existing reco command") + except Exception as e: demisto.error(f"Failed to execute {demisto.command()} command. Error: {str(e)}") return_error(str(e)) diff --git a/Packs/Reco/Integrations/Reco/Reco.yml b/Packs/Reco/Integrations/Reco/Reco.yml index 87a8eae7034..7569057e258 100644 --- a/Packs/Reco/Integrations/Reco/Reco.yml +++ b/Packs/Reco/Integrations/Reco/Reco.yml @@ -63,8 +63,8 @@ configuration: type: 0 required: false section: Collect -- additionalinfo: Risk level of the incidents to fetch - display: Risk level +- additionalinfo: 'Severity filter for fetched incidents. Accepts a single value or a comma-separated list. Supported values: LOW, MEDIUM, HIGH, CRITICAL (or numeric equivalents 10, 20, 30, 40). Example: "HIGH,CRITICAL" fetches only high and critical severity alerts.' + display: Risk level (comma-separated, e.g. HIGH,CRITICAL) name: risk_level type: 0 required: false @@ -128,8 +128,33 @@ script: description: Get Risky Users from Reco. name: reco-get-risky-users outputs: - - contextPath: Reco.RiskyUsers - description: Risky Users. + - contextPath: Reco.RiskyUsers.id + description: Account ID. + type: String + - contextPath: Reco.RiskyUsers.name + description: Account display name. + type: String + - contextPath: Reco.RiskyUsers.accountEmail + description: Account email address. + type: String + - contextPath: Reco.RiskyUsers.permissions + description: Account permissions in the SaaS application. + type: String + - contextPath: Reco.RiskyUsers.hasMfa + description: Whether the account has MFA enabled. + type: Boolean + - contextPath: Reco.RiskyUsers.openAlerts + description: Number of open alerts for this account. + type: Number + - contextPath: Reco.RiskyUsers.isAdmin + description: Whether the account has admin privileges. + type: Boolean + - contextPath: Reco.RiskyUsers.isRiskyUser + description: Whether the account is flagged as risky. + type: Boolean + - contextPath: Reco.RiskyUsers.lastSeen + description: Last activity timestamp for the account. + type: Date - arguments: - description: Email address of the user to add to the risky users list in Reco. name: email_address @@ -170,30 +195,30 @@ script: description: Get all sensitive assets from Reco by id. name: reco-get-sensitive-assets-by-id outputs: - - contextPath: Reco.SensitiveAssets.file_name + - contextPath: Reco.SensitiveAssets.id + description: The unique identifier of the asset. + type: String + - contextPath: Reco.SensitiveAssets.name description: The name of the asset. type: String - - contextPath: Reco.SensitiveAssets.file_owner + - contextPath: Reco.SensitiveAssets.owner description: The owner of the asset. type: String - - contextPath: Reco.SensitiveAssets.file_url - description: Json string of the asset's url and the name. - type: Unknown - - contextPath: Reco.SensitiveAssets.currently_permitted_users - description: List of currently permitted users. + - contextPath: Reco.SensitiveAssets.url + description: URL of the asset. type: String - - contextPath: Reco.SensitiveAssets.visibility - description: Visibility of the asset. + - contextPath: Reco.SensitiveAssets.sensitivityLevel + description: The sensitivity level of the asset (numeric). + type: Number + - contextPath: Reco.SensitiveAssets.permissionVisibility + description: Visibility/permission level of the asset. type: String - contextPath: Reco.SensitiveAssets.location description: The path of the asset. type: String - - contextPath: Reco.SensitiveAssets.source - description: SaaS tool source of the asset. - type: String - - contextPath: Reco.SensitiveAssets.sensitivity_level - description: The sensitivity level of the asset. - type: Number + - contextPath: Reco.SensitiveAssets.dataCategories + description: Data categories detected in the asset. + type: Unknown - arguments: - description: Entity Type (RM_LINK_TYPE_USER). name: entity @@ -300,27 +325,33 @@ script: name: email_address required: true outputs: - - contextPath: Reco.User.email_account - description: The email of the user. + - contextPath: Reco.User.id + description: Identity ID. type: String - - contextPath: Reco.User.departments - description: User departments. + - contextPath: Reco.User.email + description: Primary email of the identity. type: String - - contextPath: Reco.User.job_titles - description: Job Title. + - contextPath: Reco.User.name + description: Full name of the identity. type: String - - contextPath: Reco.User.category - description: Category. - type: String - - contextPath: Reco.User.groups - description: The groups user is member of. + - contextPath: Reco.User.departments + description: Departments the identity belongs to. type: String - - contextPath: Reco.User.full_name - description: The user full name. + - contextPath: Reco.User.jobTitles + description: Job titles of the identity. type: String - - contextPath: Reco.User.labels - description: User Labels. - type: Unknown + - contextPath: Reco.User.isFormer + description: Whether the identity is a former employee. + type: Boolean + - contextPath: Reco.User.isInternal + description: Whether the identity is an internal employee. + type: Boolean + - contextPath: Reco.User.openAlerts + description: Number of open alerts for this identity. + type: Number + - contextPath: Reco.User.lastSeen + description: Last activity timestamp for the identity. + type: Date description: Get user context by email address from Reco. name: reco-get-user-context-by-email-address - arguments: @@ -411,30 +442,30 @@ script: description: Get all assets from Reco by id. name: reco-get-assets-by-id outputs: - - contextPath: Reco.SensitiveAssets.file_name + - contextPath: Reco.SensitiveAssets.id + description: The unique identifier of the asset. + type: String + - contextPath: Reco.SensitiveAssets.name description: The name of the asset. type: String - - contextPath: Reco.SensitiveAssets.file_owner + - contextPath: Reco.SensitiveAssets.owner description: The owner of the asset. type: String - - contextPath: Reco.SensitiveAssets.file_url - description: Json string of the asset's url and the name. - type: Unknown - - contextPath: Reco.SensitiveAssets.currently_permitted_users - description: List of currently permitted users. + - contextPath: Reco.SensitiveAssets.url + description: URL of the asset. type: String - - contextPath: Reco.SensitiveAssets.visibility - description: Visibility of the asset. + - contextPath: Reco.SensitiveAssets.sensitivityLevel + description: The sensitivity level of the asset (numeric). + type: Number + - contextPath: Reco.SensitiveAssets.permissionVisibility + description: Visibility/permission level of the asset. type: String - contextPath: Reco.SensitiveAssets.location description: The path of the asset. type: String - - contextPath: Reco.SensitiveAssets.source - description: SaaS tool source of the asset. - type: String - - contextPath: Reco.SensitiveAssets.sensitivity_level - description: The sensitivity level of the asset. - type: Number + - contextPath: Reco.SensitiveAssets.dataCategories + description: Data categories detected in the asset. + type: Unknown - arguments: - description: Alert ID. name: alert_id @@ -459,33 +490,39 @@ script: description: Get app discovery data from Reco. Fetches all available apps using pagination. name: reco-get-apps outputs: - - contextPath: Reco.Apps.app_name - description: The name of the application. - type: String - - contextPath: Reco.Apps.app_id + - contextPath: Reco.Apps.id description: The unique identifier of the application. type: String + - contextPath: Reco.Apps.name + description: The name of the application. + type: String - contextPath: Reco.Apps.category description: The category of the application. type: String - - contextPath: Reco.Apps.risk_score - description: The risk score of the application. - type: Number - - contextPath: Reco.Apps.users_count + - contextPath: Reco.Apps.usersCount description: The number of users with access to the application. type: Number - - contextPath: Reco.Apps.data_access - description: The data access level of the application. + - contextPath: Reco.Apps.authorization + description: The authorization/sanction status of the application. type: String - - contextPath: Reco.Apps.updated_at - description: The last update timestamp of the application. - type: Date - - contextPath: Reco.Apps.created_at - description: The creation timestamp of the application. - type: Date - - contextPath: Reco.Apps.status - description: The status of the application. + - contextPath: Reco.Apps.authType + description: The authentication type used by the application. + type: String + - contextPath: Reco.Apps.isUsingAi + description: Whether the application uses AI. + type: Boolean + - contextPath: Reco.Apps.isShadowApp + description: Whether the application is a shadow/unmanaged app. + type: Boolean + - contextPath: Reco.Apps.vendorGrade + description: The vendor security grade of the application. type: String + - contextPath: Reco.Apps.aiCapability + description: AI capability description for the application. + type: String + - contextPath: Reco.Apps.lastSeen + description: Last activity timestamp for the application. + type: Date - arguments: - description: Application ID to update authorization status for. name: app_id @@ -520,6 +557,451 @@ script: - contextPath: Reco.AppAuthorization.rows_affected description: Number of rows affected by the update operation. type: Number + - arguments: + - description: 'SCIM v2 filter expression (e.g. "actor.email eq \"user@example.com\" and eventTime gt \"2024-01-01T00:00:00Z\"").' + name: filters + required: false + - description: Maximum number of events to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List SaaS events from Reco using the external API. + name: reco-list-events + outputs: + - contextPath: Reco.Events.id + description: Event ID. + type: String + - contextPath: Reco.Events.eventType + description: Event type code. + type: String + - contextPath: Reco.Events.formattedEventType + description: Human-readable event type. + type: String + - contextPath: Reco.Events.application + description: SaaS application the event belongs to. + type: String + - contextPath: Reco.Events.actorEmail + description: Email of the actor who triggered the event. + type: String + - contextPath: Reco.Events.actorName + description: Name of the actor who triggered the event. + type: String + - contextPath: Reco.Events.eventTime + description: Timestamp of the event. + type: Date + - contextPath: Reco.Events.outcomeString + description: Outcome description of the event. + type: String + - arguments: + - description: 'SCIM v2 filter expression (e.g. "severity eq \"HIGH\"").' + name: filters + required: false + - description: Maximum number of posture issues to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List posture issues from Reco using the external API. + name: reco-list-posture-issues + outputs: + - contextPath: Reco.PostureIssues.id + description: Posture issue ID. + type: String + - contextPath: Reco.PostureIssues.name + description: Posture issue name. + type: String + - contextPath: Reco.PostureIssues.severity + description: Severity of the posture issue. + type: String + - contextPath: Reco.PostureIssues.checkStatus + description: Current check status of the posture issue. + type: String + - contextPath: Reco.PostureIssues.scorePercentage + description: Compliance score percentage for this issue. + type: Number + - contextPath: Reco.PostureIssues.checkedInstance + description: The SaaS instance this issue was checked against. + type: Unknown + - contextPath: Reco.PostureIssues.url + description: Link to the posture issue detail in the Reco UI. + type: String + - arguments: + - description: 'SCIM v2 filter expression (e.g. "isRiskyUser eq true" or "accountEmail co \"@example.com\"").' + name: filters + required: false + - description: Maximum number of accounts to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List accounts (SaaS user accounts) from Reco using the external API. + name: reco-list-accounts + outputs: + - contextPath: Reco.Accounts.id + description: Account ID. + type: String + - contextPath: Reco.Accounts.name + description: Account display name. + type: String + - contextPath: Reco.Accounts.accountEmail + description: Account email address. + type: String + - contextPath: Reco.Accounts.permissions + description: Account permissions in the SaaS application. + type: String + - contextPath: Reco.Accounts.hasMfa + description: Whether the account has MFA enabled. + type: Boolean + - contextPath: Reco.Accounts.openAlerts + description: Number of open alerts for this account. + type: Number + - contextPath: Reco.Accounts.isAdmin + description: Whether the account has admin privileges. + type: Boolean + - contextPath: Reco.Accounts.isRiskyUser + description: Whether the account is flagged as risky. + type: Boolean + - contextPath: Reco.Accounts.lastSeen + description: Last activity timestamp for the account. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "isUnmanaged eq true" or "devicePlatform eq \"Windows\"").' + name: filters + required: false + - description: Maximum number of devices to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List devices from Reco using the external API. + name: reco-list-devices + outputs: + - contextPath: Reco.Devices.id + description: Device ID. + type: String + - contextPath: Reco.Devices.name + description: Device name. + type: String + - contextPath: Reco.Devices.devicePlatform + description: Device platform (Windows, macOS, iOS, Android, etc.). + type: String + - contextPath: Reco.Devices.os + description: Operating system of the device. + type: String + - contextPath: Reco.Devices.osVersion + description: Operating system version. + type: String + - contextPath: Reco.Devices.isUnmanaged + description: Whether the device is unmanaged (not enrolled in MDM). + type: Boolean + - contextPath: Reco.Devices.hasNonCompliant + description: Whether the device has non-compliant policies. + type: Boolean + - contextPath: Reco.Devices.lastSeen + description: Last activity timestamp for the device. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "authorization eq \"AUTH_STATUS_UNSANCTIONED\"").' + name: filters + required: false + - description: Maximum number of AI agents to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List AI agents detected by Reco using the external API. + name: reco-list-ai-agents + outputs: + - contextPath: Reco.AiAgents.id + description: AI agent ID. + type: String + - contextPath: Reco.AiAgents.name + description: AI agent name. + type: String + - contextPath: Reco.AiAgents.vendor + description: Vendor of the AI agent. + type: String + - contextPath: Reco.AiAgents.type + description: Type of AI agent. + type: String + - contextPath: Reco.AiAgents.authorization + description: Authorization/sanction status of the AI agent. + type: String + - contextPath: Reco.AiAgents.agentStatus + description: Current status of the AI agent. + type: String + - contextPath: Reco.AiAgents.risk + description: Risk level of the AI agent. + type: String + - contextPath: Reco.AiAgents.lastUsage + description: Last usage timestamp for the AI agent. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "name co \"Engineering\"").' + name: filters + required: false + - description: Maximum number of groups to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List SaaS groups from Reco using the external API. + name: reco-list-groups + outputs: + - contextPath: Reco.Groups.id + description: Group ID. + type: String + - contextPath: Reco.Groups.name + description: Group name. + type: String + - contextPath: Reco.Groups.email + description: Group email address. + type: String + - contextPath: Reco.Groups.membersCount + description: Number of members in the group. + type: Number + - contextPath: Reco.Groups.appsCount + description: Number of apps the group has access to. + type: Number + - arguments: + - description: 'SCIM v2 filter expression (e.g. "authorization eq \"AUTH_STATUS_UNSANCTIONED\" or permissionRisk eq \"30\"").' + name: filters + required: false + - description: Maximum number of SaaS-to-SaaS grants to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List SaaS-to-SaaS OAuth grants and integrations from Reco using the external API. + name: reco-list-saas-to-saas + outputs: + - contextPath: Reco.SaasToSaas.id + description: SaaS-to-SaaS grant ID. + type: String + - contextPath: Reco.SaasToSaas.plugin + description: Plugin or app name receiving the grant. + type: String + - contextPath: Reco.SaasToSaas.authorization + description: Authorization status of the grant. + type: String + - contextPath: Reco.SaasToSaas.permissionRisk + description: Permission risk level (10=LOW, 20=MEDIUM, 30=HIGH). + type: String + - contextPath: Reco.SaasToSaas.accounts + description: Number of accounts with this grant. + type: Number + - contextPath: Reco.SaasToSaas.aiCapability + description: AI capability of the third-party app. + type: String + - contextPath: Reco.SaasToSaas.lastSeen + description: Last activity timestamp for this grant. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "country eq \"CN\" or hasVpn eq true").' + name: filters + required: false + - description: Maximum number of IP addresses to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List observed IP addresses from Reco using the external API. + name: reco-list-ip-addresses + outputs: + - contextPath: Reco.IpAddresses.ipAddress + description: IP address or CIDR range. + type: String + - contextPath: Reco.IpAddresses.country + description: Country of the IP address. + type: String + - contextPath: Reco.IpAddresses.asnName + description: ASN name of the IP address. + type: String + - contextPath: Reco.IpAddresses.eventsCount + description: Number of events from this IP. + type: Number + - contextPath: Reco.IpAddresses.usersCount + description: Number of users seen from this IP. + type: Number + - contextPath: Reco.IpAddresses.hasVpn + description: Whether the IP is associated with a VPN. + type: Boolean + - contextPath: Reco.IpAddresses.hasProxy + description: Whether the IP is associated with a proxy. + type: Boolean + - contextPath: Reco.IpAddresses.lastEventTime + description: Last event timestamp from this IP. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "name eq \"Finance\"").' + name: filters + required: false + - description: Maximum number of business units to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List external business units from Reco using the external API. + name: reco-list-business-units + outputs: + - contextPath: Reco.BusinessUnits.id + description: Business unit ID. + type: String + - contextPath: Reco.BusinessUnits.name + description: Business unit name. + type: String + - contextPath: Reco.BusinessUnits.manager + description: Manager of the business unit. + type: String + - contextPath: Reco.BusinessUnits.createdAt + description: Creation timestamp of the business unit. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "userEmail eq \"admin@example.com\" and action eq \"DELETE\"").' + name: filters + required: false + - description: Maximum number of audit log entries to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List Reco platform audit logs using the external API. + name: reco-list-audit-logs + outputs: + - contextPath: Reco.AuditLogs.id + description: Audit log entry ID. + type: String + - contextPath: Reco.AuditLogs.userEmail + description: Email of the user who performed the action. + type: String + - contextPath: Reco.AuditLogs.module + description: Module where the action was performed. + type: String + - contextPath: Reco.AuditLogs.action + description: Action performed. + type: String + - contextPath: Reco.AuditLogs.objectName + description: Name of the object affected. + type: String + - contextPath: Reco.AuditLogs.timestamp + description: Timestamp of the audit log entry. + type: Date + - contextPath: Reco.AuditLogs.remoteAddr + description: Remote IP address of the actor. + type: String + - arguments: + - description: 'SCIM v2 filter expression (e.g. "severity eq \"HIGH\" and apps co \"Google\"").' + name: filters + required: false + - description: Maximum number of posture check definitions to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List posture check definitions from Reco using the external API. + name: reco-list-posture-checks + outputs: + - contextPath: Reco.PostureChecks.id + description: Posture check ID. + type: String + - contextPath: Reco.PostureChecks.name + description: Posture check name. + type: String + - contextPath: Reco.PostureChecks.severity + description: Severity of the posture check. + type: String + - contextPath: Reco.PostureChecks.policyType + description: Policy type of the posture check. + type: String + - contextPath: Reco.PostureChecks.apps + description: Applications this posture check applies to. + type: Unknown + - contextPath: Reco.PostureChecks.type + description: Whether this is a built-in or custom posture check. + type: String + - arguments: + - description: 'SCIM v2 filter expression (e.g. "severity eq \"HIGH\" and status eq \"ON\"").' + name: filters + required: false + - description: Maximum number of policies to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List threat detection policies from Reco using the external API. + name: reco-list-threat-detection-policies + outputs: + - contextPath: Reco.ThreatDetectionPolicies.id + description: Policy ID. + type: String + - contextPath: Reco.ThreatDetectionPolicies.name + description: Policy name. + type: String + - contextPath: Reco.ThreatDetectionPolicies.severity + description: Severity of the policy. + type: String + - contextPath: Reco.ThreatDetectionPolicies.status + description: Whether the policy is ON, OFF, or PREVIEW. + type: String + - contextPath: Reco.ThreatDetectionPolicies.apps + description: Applications monitored by the policy. + type: Unknown + - contextPath: Reco.ThreatDetectionPolicies.openAlerts + description: Number of open alerts triggered by this policy. + type: Number + - contextPath: Reco.ThreatDetectionPolicies.type + description: Whether this is a built-in or custom policy. + type: String + - arguments: + - description: 'SCIM v2 filter expression (e.g. "policyName co \"MFA\"").' + name: filters + required: false + - description: Maximum number of exclusions to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List alert suppression exclusion rules from Reco using the external API. + name: reco-list-exclusions + outputs: + - contextPath: Reco.Exclusions.id + description: Exclusion rule ID. + type: String + - contextPath: Reco.Exclusions.name + description: Exclusion rule name. + type: String + - contextPath: Reco.Exclusions.policyName + description: Name of the policy this exclusion applies to. + type: String + - contextPath: Reco.Exclusions.apps + description: Applications this exclusion applies to. + type: Unknown + - contextPath: Reco.Exclusions.createdBy + description: User who created the exclusion. + type: String + - contextPath: Reco.Exclusions.createdAt + description: Creation timestamp of the exclusion. + type: Date + - arguments: + - description: 'SCIM v2 filter expression (e.g. "isUsingAi eq true").' + name: filters + required: false + - description: Maximum number of app instances to return (default 1000). + name: limit + required: false + defaultValue: '1000' + description: List integrated app instances (app portfolio) from Reco using the external API. Only returns instances with an active integration status. + name: reco-list-app-instances + outputs: + - contextPath: Reco.AppInstances.id + description: App instance ID. + type: String + - contextPath: Reco.AppInstances.name + description: App instance name. + type: String + - contextPath: Reco.AppInstances.instanceType + description: Type of the app instance. + type: String + - contextPath: Reco.AppInstances.accountsCount + description: Number of accounts in this app instance. + type: Number + - contextPath: Reco.AppInstances.isUsingAi + description: Whether this app instance uses AI features. + type: Boolean + - contextPath: Reco.AppInstances.saasToSaasCount + description: Number of SaaS-to-SaaS grants for this instance. + type: Number + - contextPath: Reco.AppInstances.filesCount + description: Number of files tracked in this instance. + type: Number tests: - No tests fromversion: 6.5.0 diff --git a/Packs/Reco/README.md b/Packs/Reco/README.md index bef79958562..4deeab81963 100644 --- a/Packs/Reco/README.md +++ b/Packs/Reco/README.md @@ -1,40 +1,135 @@ # Reco -Reco is the leader in Dynamic SaaS Security — eliminating the SaaS Security Gap driven by app proliferation, AI infiltration, constant configuration changes, identity sprawl, and hidden threats. +Reco is the leader in SaaS & AI Security — the only approach that secures AI sprawl across SaaS apps and agents. Our platform provides complete visibility and control across your entire SaaS ecosystem, from core applications to the latest AI agents, enabling security teams to keep pace with the speed of AI adoption while maintaining robust security and reducing risk. -The Reco and Palo Alto Networks Cortex XSOAR integration automates SaaS threat detection and remediation workflows, integrating with 985+ Cortex XSOAR content packs. +The Reco and Palo Alto Networks Cortex XSOAR integration brings Reco's SaaS & AI intelligence directly into your SOC workflows — surfacing threats, enriching investigations, and automating remediation across 985+ Cortex XSOAR content packs. ## What does this pack do? -- **Assess 400+ configuration rules** across your SaaS landscape -- **Access data risk alerts** and **AI-powered summaries** -- **Trigger automatic remediation** for misconfigurations -- **Tag risky employees** and leverage prebuilt playbooks - -## Benefits - -- **Gain visibility** into SaaS misconfigurations -- **Streamline detection** and automate remediation -- **Respond faster** to critical alerts -- **Reduce manual effort** while meeting compliance requirements - -## Key Commands - -- **reco-add-comment-to-alert** -- **reco-update-incident-timeline** -- **resolve-visibility-event** -- **reco-get-risky-users** -- **reco-add-risky-user-label** -- **reco-get-assets-user-has-access-to** -- **reco-add-leaving-org-user-label** -- **reco-get-sensitive-assets-by-name** -- **reco-get-sensitive-assets-by-id** -- **reco-get-files-shared-with-3rd-parties** -- **reco-get-3rd-parties-accessible-to-data-list** -- **reco-get-sensitive-assets-with-public-link** -- **reco-get-user-context-by-email-address** -- **reco-get-private-email-list-with-access** -- **reco-get-alert-ai-summary** +- **Govern AI usage** — discover AI agents, AI-powered SaaS apps, and SaaS-to-SaaS OAuth grants with AI capabilities; prevent unauthorized data sharing; maintain audit-ready AI activity records +- **Secure AI agents** — continuously monitor non-human identities operating in SaaS; enforce least-privilege policies; detect agent misuse and anomalous behavior +- **Audit your SaaS posture** — query posture issues, posture checks, and threat detection policies across every connected app; score against SOC 2, ISO 27001, CIS, NIST, PCI DSS, HITRUST, and more +- **Detect and respond to SaaS threats** — fetch behavioral threat alerts with multi-severity filtering and AI-powered summaries; respond with existing SIEM & SOAR tooling +- **Investigate identities and accounts** — enrich user context, flag risky employees, and track account activity across SaaS apps +- **Manage app risk** — inventory your app portfolio, update authorization status, and track shadow IT +- **Protect sensitive data** — surface files shared publicly, externally, or with private emails; query sensitivity-labeled assets +- **Automate remediation** — tag leaving employees, add risk labels, post comments to alerts, and trigger playbooks + +## The SaaS Security Gap + +Five types of sprawl are widening the gap between what you can and cannot protect: + +- **App Sprawl** — Apps constantly multiply, update, and form SaaS-to-SaaS connections, making it impossible to keep up +- **AI Sprawl** — The infusion of GenAI into SaaS apps, and the surge of AI Agents, undermines AI security readiness +- **Identity Sprawl** — Keeping accounts secure while minimizing access privileges is unfeasible with the relentless proliferation of human and machine identities +- **Configuration Sprawl** — The security posture of apps and users is critical yet utterly impractical to continuously update and maintain +- **Data Sprawl** — More entities — including AI agents — access your data through more pathways, making breaches and insider threats harder to spot + +## Key Capabilities + +### AI Governance +- Gain full visibility into AI tool adoption — from ChatGPT to copilots to embedded AI features +- Prevent unauthorized data sharing and monitor AI usage for policy compliance +- List AI agents with authorization status, risk level, and vendor +- Surface apps and SaaS-to-SaaS grants using AI capabilities + +### AI Agent Security +- Discover and continuously monitor AI agents operating within your SaaS environment +- Understand what data agents access, what actions they perform, and where over-permission creates risk +- Enforce least-privilege policies for non-human identities at scale +- Detect AI agent misuse through behavioral threat detection policies + +### Posture Management +- Continuously assess security risk across applications, identities, and data +- List posture issues with severity, check status, and compliance framework mappings (SOC 2, ISO 27001, CIS, NIST CSF, NIST 800-53, PCI DSS, HITRUST) +- List posture check definitions and threat detection policies +- Track configuration drift with one-click remediation guidance + +### Threat Detection & Response (ITDR) +- Fetch incidents with flexible severity filters (single or multi-level: `HIGH,CRITICAL`) +- Get full alert details including policy violation evidence +- Add comments to alerts and update incident timelines +- Change alert status and resolve visibility events +- Get AI-generated alert summaries + +### Identity & Account Intelligence +- List all SaaS accounts with risk signals (MFA status, admin flag, risky user label) +- Look up user context by email address across all integrated apps +- List identities with aggregated cross-app view +- Tag risky users and departing employees + +### SaaS Application Governance +- Discover all apps (sanctioned, shadow, AI-powered) with vendor risk grades +- List app instances (portfolio) from actively integrated apps +- List SaaS-to-SaaS OAuth grants with permission risk scores +- Update app authorization status + +### Data Security +- Find sensitive files by name, ID, or sensitivity level +- Query files shared with third-party domains +- Identify files shared publicly or with external emails +- List NetApp files carrying active business-impact labels + +### Platform Visibility +- List SaaS events with actor, application, and outcome context +- List groups and IP addresses +- List business units +- Query platform audit logs + +## Commands + +**Alerts & Incidents** +- `reco-add-comment-to-alert` — Add a comment to a Reco alert +- `reco-update-incident-timeline` — Add a comment to an incident timeline +- `reco-change-alert-status` — Update alert status (NEW / IN_PROGRESS / CLOSED) +- `reco-resolve-visibility-event` — Resolve a visibility event in a Reco Finding +- `reco-get-alert-ai-summary` — Get an AI-generated summary of an alert + +**Identities & Users** +- `reco-get-risky-users` — List all accounts flagged as risky +- `reco-add-risky-user-label` — Tag a user as risky +- `reco-add-leaving-org-user-label` — Tag a user as a departing employee +- `reco-get-user-context-by-email-address` — Get identity context for an email address + +**SaaS Applications** +- `reco-get-apps` — List discovered apps with risk and AI signals +- `reco-set-app-authorization-status` — Update an app's authorization status + +**Posture & Policies** +- `reco-list-posture-issues` — List posture issues with severity and check status +- `reco-list-posture-checks` — List posture check definitions +- `reco-list-threat-detection-policies` — List threat detection policies +- `reco-list-exclusions` — List alert suppression exclusion rules + +**Data & Files** +- `reco-get-sensitive-assets-by-name` — Find sensitive assets by name +- `reco-get-sensitive-assets-by-id` — Find sensitive assets by ID +- `reco-get-assets-by-id` — Find any asset by ID +- `reco-get-assets-user-has-access-to` — List files a user has access to +- `reco-get-sensitive-assets-with-public-link` — List publicly exposed sensitive files +- `reco-get-assets-shared-externally` — List files shared outside the organization +- `reco-get-files-exposed-to-email-address` — List files accessible to a specific email +- `reco-get-files-shared-with-3rd-parties` — List files shared with a third-party domain +- `reco-get-3rd-parties-accessible-to-data-list` — List third-party domains with data access +- `reco-get-private-email-list-with-access` — List private emails with file access + +**SaaS Events & Activity** +- `reco-list-events` — List SaaS activity events +- `reco-list-accounts` — List SaaS accounts with risk signals +- `reco-list-groups` — List SaaS groups +- `reco-list-saas-to-saas` — List SaaS-to-SaaS OAuth grants +- `reco-list-ip-addresses` — List observed IP addresses +- `reco-list-audit-logs` — List Reco platform audit logs + +**AI Governance** +- `reco-list-ai-agents` — List detected AI agents + +**Platform** +- `reco-list-app-instances` — List integrated app instances (portfolio) +- `reco-list-devices` — List managed and unmanaged devices +- `reco-list-business-units` — List business units +- `reco-get-link-to-user-overview-page` — Generate a deep link to the Reco UI +- `reco-add-exclusion-filter` — Add a classifier exclusion filter _For more information: [www.reco.ai](https://www.reco.ai)_ diff --git a/Packs/Reco/ReleaseNotes/1_8_0.md b/Packs/Reco/ReleaseNotes/1_8_0.md new file mode 100644 index 00000000000..5699bcb5372 --- /dev/null +++ b/Packs/Reco/ReleaseNotes/1_8_0.md @@ -0,0 +1,8 @@ +#### Integrations + +##### Reco +- Migrated alert fetching, identity, app, file, and label commands to the new Reco External API (`/api/v1/external-api/`) for cleaner JSON responses and improved reliability. +- Added support for multiple severity filters in `fetch-incidents` — the **Risk level** parameter now accepts a comma-separated list (e.g., `HIGH,CRITICAL`). Resolves PRO-303. +- Added rate-limit handling (automatic 429 retry with exponential backoff) for all External API calls. +- Added five new commands: **reco-list-events**, **reco-list-posture-issues**, **reco-list-accounts**, **reco-list-devices**, **reco-list-ai-agents**. All accept SCIM v2 filter expressions. +- Updated context output field names for **reco-get-risky-users**, **reco-get-user-context-by-email-address**, **reco-get-sensitive-assets-by-id**, **reco-get-assets-by-id**, and **reco-get-apps** to match the External API response schema. diff --git a/Packs/Reco/pack_metadata.json b/Packs/Reco/pack_metadata.json index 75066aa5844..de23299bd55 100644 --- a/Packs/Reco/pack_metadata.json +++ b/Packs/Reco/pack_metadata.json @@ -1,8 +1,8 @@ { "name": "Reco", - "description": "Reco is the leader in Dynamic SaaS Security — the only approach that eliminates the SaaS Security Gap (the growing gap between what you can protect and what’s outpacing your security).", + "description": "Reco is the leader in Dynamic SaaS Security \u2014 the only approach that eliminates the SaaS Security Gap (the growing gap between what you can protect and what\u2019s outpacing your security).", "support": "partner", - "currentVersion": "1.7.5", + "currentVersion": "1.8.0", "author": "Reco", "url": "https://reco.ai", "email": "support@reco.ai", @@ -35,7 +35,11 @@ ], "tags": [], "supportedModules": [ + "X1", + "X3", + "X5", + "ENT_PLUS", "agentix", "xsiam" ] -} \ No newline at end of file +} From 4730f7bd961324987c322c61c1d8db1d60d4d72e Mon Sep 17 00:00:00 2001 From: Yaniv Blum Date: Fri, 17 Jul 2026 14:52:06 -0500 Subject: [PATCH 02/47] Reco: fix lint, update test mocks to External API, minimum-severity filter - Silence xsoar-lint E9003 on the rate-limit backoff sleep (matches the inline-disable convention used by other integrations' retry loops). - Rewrite unit tests that still mocked the old internal Table API endpoints for flows migrated to the External API (fetch-incidents, risky users, user context, sensitive assets, app discovery/auth status), and fix a couple of assertions left over from the old response shapes. - Change the risk_level fetch-incidents parameter from a comma-separated exact-match list back to a single minimum-severity value: e.g. MEDIUM now fetches medium, high, and critical alerts (PRO-303). --- Packs/Reco/Integrations/Reco/Reco.py | 168 ++++++-- Packs/Reco/Integrations/Reco/Reco.yml | 4 +- Packs/Reco/Integrations/Reco/Reco_test.py | 486 ++++++---------------- Packs/Reco/README.md | 20 +- Packs/Reco/ReleaseNotes/1_8_0.md | 2 +- 5 files changed, 279 insertions(+), 401 deletions(-) diff --git a/Packs/Reco/Integrations/Reco/Reco.py b/Packs/Reco/Integrations/Reco/Reco.py index b7e06d43079..cbdf33291f6 100644 --- a/Packs/Reco/Integrations/Reco/Reco.py +++ b/Packs/Reco/Integrations/Reco/Reco.py @@ -56,25 +56,25 @@ "critical": "CRITICAL", } +# Ascending severity order, used to expand a minimum risk level into "that level and above". +SEVERITY_ORDER: list[str] = ["LOW", "MEDIUM", "HIGH", "CRITICAL"] -def parse_risk_levels(risk_level_param: str | None) -> list[str] | None: - """Parse a single or comma-separated risk_level string into a list of severity names. - Accepts numeric values (10/20/30/40) or names (LOW/MEDIUM/HIGH/CRITICAL). - Examples: "HIGH" → ["HIGH"], "HIGH,CRITICAL" → ["HIGH", "CRITICAL"], "30,40" → ["HIGH", "CRITICAL"] +def parse_minimum_risk_level(risk_level_param: str | None) -> list[str] | None: + """Parse a single risk_level value into the list of severity names at or above it. + + Accepts a numeric value (10/20/30/40) or a name (LOW/MEDIUM/HIGH/CRITICAL). + Example: "MEDIUM" → ["MEDIUM", "HIGH", "CRITICAL"] (fetches medium severity and higher). """ if not risk_level_param: return None - values = [v.strip() for v in str(risk_level_param).split(",") if v.strip()] - result: list[str] = [] - for v in values: - severity_name = RISK_LEVEL_TO_SEVERITY_NAME.get(v) or RISK_LEVEL_TO_SEVERITY_NAME.get(v.upper()) - if severity_name and severity_name not in result: - result.append(severity_name) - elif v.upper() not in result: - demisto.debug(f"Unknown risk level value '{v}', using upper-cased as-is") - result.append(v.upper()) - return result or None + value = str(risk_level_param).strip() + severity_name = RISK_LEVEL_TO_SEVERITY_NAME.get(value) or RISK_LEVEL_TO_SEVERITY_NAME.get(value.upper()) + if not severity_name: + demisto.debug(f"Unknown risk level value '{value}', using upper-cased as-is") + return [value.upper()] + min_index = SEVERITY_ORDER.index(severity_name) + return SEVERITY_ORDER[min_index:] def create_filter(field, value): @@ -117,22 +117,19 @@ def _rate_limited_request(self, method: str, url_suffix: str, **kwargs) -> dict[ try: return self._http_request(method=method, url_suffix=url_suffix, **kwargs) except DemistoException as exc: - is_rate_limited = ( - exc.res is not None - and getattr(exc.res, "status_code", None) == 429 - ) + is_rate_limited = exc.res is not None and getattr(exc.res, "status_code", None) == 429 if is_rate_limited and attempt < RATE_LIMIT_MAX_RETRIES - 1: retry_after = int( exc.res.headers.get( # type: ignore[union-attr] "Retry-After", - RATE_LIMIT_RETRY_BASE_DELAY * (2 ** attempt), + RATE_LIMIT_RETRY_BASE_DELAY * (2**attempt), ) ) demisto.debug( f"Rate limited (429) on {url_suffix}, " f"retrying in {retry_after}s (attempt {attempt + 1}/{RATE_LIMIT_MAX_RETRIES})" ) - time.sleep(retry_after) + time.sleep(retry_after) # pylint: disable=E9003 else: raise raise DemistoException(f"Max retries ({RATE_LIMIT_MAX_RETRIES}) exceeded for {url_suffix}") @@ -229,7 +226,7 @@ def get_alerts( else: # Use the SCIM `in` operator — cleaner than a chain of OR clauses vals = ", ".join(f'"{r}"' for r in risk_levels) - filter_parts.append(f'severity in [{vals}]') + filter_parts.append(f"severity in [{vals}]") if source: filter_parts.append(f'apps co "{source}"') @@ -374,7 +371,7 @@ def get_identities(self, email_address: Optional[str] = None, label: Optional[st def get_risky_users(self) -> list[dict[str, Any]]: """List ALL accounts flagged as risky via the external API (auto-paginated).""" try: - return self._paginate_all("accounts/list", "accounts", filters='isRiskyUser eq true') + return self._paginate_all("accounts/list", "accounts", filters="isRiskyUser eq true") except Exception as e: demisto.error(f"get_risky_users error: {str(e)}") raise @@ -1131,7 +1128,16 @@ def get_assets_shared_externally_command(reco_client: RecoClient, email_address: readable_output=tableToMarkdown( "Assets", assets_list, - headers=["asset_id", "asset", "data_category", "data_categories", "last_access_date", "visibility", "location", "file_owner"], + headers=[ + "asset_id", + "asset", + "data_category", + "data_categories", + "last_access_date", + "visibility", + "location", + "file_owner", + ], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1147,7 +1153,17 @@ def get_files_exposed_to_email_command(reco_client: RecoClient, email_account: s readable_output=tableToMarkdown( "Assets", assets_list, - headers=["asset_id", "asset", "data_category", "data_categories", "last_access_date", "visibility", "location", "email_account", "file_owner"], + headers=[ + "asset_id", + "asset", + "data_category", + "data_categories", + "last_access_date", + "visibility", + "location", + "email_account", + "file_owner", + ], ), outputs_prefix="Reco.Assets", outputs_key_field="asset_id", @@ -1565,8 +1581,8 @@ def main() -> None: # pragma: no cover ) if command == "fetch-incidents": - # risk_level accepts a single value or comma-separated list (PRO-303) - risk_levels = parse_risk_levels(params.get("risk_level")) + # risk_level accepts a single value; alerts at or above it are fetched (PRO-303) + risk_levels = parse_minimum_risk_level(params.get("risk_level")) source = params.get("source") before = params.get("before") after: datetime | None = None @@ -1662,7 +1678,9 @@ def main() -> None: # pragma: no cover ) elif command == "reco-add-exclusion-filter": - return_results(add_exclusion_filter(reco_client, demisto.args()["key_to_add"], argToList(demisto.args()["values_to_add"]))) + return_results( + add_exclusion_filter(reco_client, demisto.args()["key_to_add"], argToList(demisto.args()["values_to_add"])) + ) elif command == "reco-change-alert-status": return_results(change_alert_status(reco_client, demisto.args()["alert_id"], demisto.args()["status"])) @@ -1688,52 +1706,116 @@ def main() -> None: # pragma: no cover elif command == "reco-get-apps": before_dt = dateparser.parse(demisto.args()["before"]) if demisto.args().get("before") else None after_dt = dateparser.parse(demisto.args()["after"]) if demisto.args().get("after") else None - return_results(get_apps_command(reco_client, before=before_dt, after=after_dt, limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + get_apps_command( + reco_client, before=before_dt, after=after_dt, limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-set-app-authorization-status": - return_results(set_app_authorization_status_command(reco_client, demisto.args()["app_id"], demisto.args()["authorization_status"])) + return_results( + set_app_authorization_status_command( + reco_client, demisto.args()["app_id"], demisto.args()["authorization_status"] + ) + ) elif command == "reco-list-events": - return_results(list_events_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_events_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-posture-issues": - return_results(list_posture_issues_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_posture_issues_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-accounts": - return_results(list_accounts_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_accounts_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-devices": - return_results(list_devices_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_devices_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-ai-agents": - return_results(list_ai_agents_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_ai_agents_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-groups": - return_results(list_groups_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_groups_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-saas-to-saas": - return_results(list_saas_to_saas_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_saas_to_saas_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-ip-addresses": - return_results(list_ip_addresses_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_ip_addresses_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-business-units": - return_results(list_business_units_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_business_units_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-audit-logs": - return_results(list_audit_logs_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_audit_logs_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-posture-checks": - return_results(list_posture_checks_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_posture_checks_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-threat-detection-policies": - return_results(list_threat_detection_policies_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_threat_detection_policies_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-exclusions": - return_results(list_exclusions_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_exclusions_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) elif command == "reco-list-app-instances": - return_results(list_app_instances_command(reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE))) + return_results( + list_app_instances_command( + reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + ) + ) else: raise NotImplementedError(f"{command} is not an existing reco command") diff --git a/Packs/Reco/Integrations/Reco/Reco.yml b/Packs/Reco/Integrations/Reco/Reco.yml index 7569057e258..8694add124b 100644 --- a/Packs/Reco/Integrations/Reco/Reco.yml +++ b/Packs/Reco/Integrations/Reco/Reco.yml @@ -63,8 +63,8 @@ configuration: type: 0 required: false section: Collect -- additionalinfo: 'Severity filter for fetched incidents. Accepts a single value or a comma-separated list. Supported values: LOW, MEDIUM, HIGH, CRITICAL (or numeric equivalents 10, 20, 30, 40). Example: "HIGH,CRITICAL" fetches only high and critical severity alerts.' - display: Risk level (comma-separated, e.g. HIGH,CRITICAL) +- additionalinfo: 'Minimum severity threshold for fetched incidents. Accepts a single value: LOW, MEDIUM, HIGH, or CRITICAL (or numeric equivalents 10, 20, 30, 40). Alerts at or above this severity are fetched. Example: "MEDIUM" fetches medium, high, and critical severity alerts.' + display: Minimum risk level (e.g. MEDIUM fetches medium and higher) name: risk_level type: 0 required: false diff --git a/Packs/Reco/Integrations/Reco/Reco_test.py b/Packs/Reco/Integrations/Reco/Reco_test.py index cb9255fb1b6..c63e27c57d4 100644 --- a/Packs/Reco/Integrations/Reco/Reco_test.py +++ b/Packs/Reco/Integrations/Reco/Reco_test.py @@ -30,156 +30,45 @@ get_apps_command, set_app_authorization_status_command, parse_alerts_to_incidents, + parse_minimum_risk_level, ) from test_data.structs import ( TableData, RowData, KeyValuePair, - RiskLevel, GetTableResponse, GetIncidentTableResponse, ) DUMMY_RECO_API_DNS_NAME = "https://dummy.reco.ai/api" -INCIDET_ID_UUID = "87799f2f-c012-43b6-ace2-78ec984427f3" ALERT_ID = "ee593dc2-a50e-415e-bed0-8403c18b26ca" INCIDENT_DESCRIPTION = "Sensitive files are accessible to anyone who has their link" ENCODING = "utf-8" -INCIDENT_STATUS = "INCIDENT_STATE_UNMARKED" TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ" -def get_random_table_response() -> GetIncidentTableResponse: - return GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData( - rows=[ - RowData( - cells=[ - KeyValuePair( - key="incident_id", - value=base64.b64encode(INCIDET_ID_UUID.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="incident_description", - value=base64.b64encode(INCIDENT_DESCRIPTION.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="risk_level", - value=base64.b64encode(str(RiskLevel.HIGH.value).encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="event_time", - value=base64.b64encode(datetime.datetime.now().strftime(TIME_FORMAT).encode(ENCODING)).decode( - ENCODING - ), - ), - KeyValuePair( - key="updated_at", - value=base64.b64encode(datetime.datetime.now().strftime(TIME_FORMAT).encode(ENCODING)).decode( - ENCODING - ), - ), - KeyValuePair( - key="status", - value=base64.b64encode(INCIDENT_STATUS.encode(ENCODING)).decode(ENCODING), - ), - ] - ) - ] - ), - total_number_of_results=1, - table_definition="", - dynamic_table_definition="", - token="", - ), - ) - - -def get_alerts_and_table_response() -> tuple[GetIncidentTableResponse, dict[str, Any]]: - alert = { - "riskLevel": 30, +def build_alert_detail(alert_id: str, description: str, risk_level: int, created_at: str) -> dict[str, Any]: + """Build an alert-details response (external API `GET /external-api/alert-details/{id}`).""" + return { "alert": { - "id": INCIDET_ID_UUID, - "policyId": "7f174580-81b6-4349-a4ca-ca8873b48b78", - "workspaceId": "f4be4202-4a6e-4bf7-b058-5d33a3146df7", - "description": "1 sensitive file exposed publicly by t@acme.ai (part of ACME IL)", - "aggregationRulesToKeys": { - "aggregationRuleToKey": [ - { - "aggregationRule": {"aggregationKeyJsonataQuery": "enriched.actor.email_account"}, - "aggregationKey": {"stringKey": '"t@acme.ai"'}, - }, - { - "aggregationRule": {"aggregationDuration": "3600s"}, - "aggregationKey": {"timeKey": "2023-05-03T10:21:04.078627293Z"}, - }, - ] - }, - "status": "ALERT_STATUS_NEW", - "riskLevel": 30, + "id": alert_id, + "description": description, + "riskLevel": risk_level, + "createdAt": created_at, "policyViolations": [ { "id": "75123c18-5ea2-4511-b9c0-1aad67e8b2ff", - "policyId": "7f174580-81b6-4349-a4ca-ca8873b48b78", - "workspaceId": "f4be4202-4a6e-4bf7-b058-5d33a3146df7", - "jsonData": "eyJpZCI6ICI5MWJlZjFkZS01NjZkLTRkYWYtODhkYi0xYTQ0MDQ0Y2E1NzciLCAicGF5bG9hZCI6IHsiaWQiOiB7InRpbWUiOiAiMjAyMy0wNS0wM1QxMDoyMDo1NS43NTRaIiwgImN1c3RvbWVySWQiOiAiQzAybWNrc2htIiwgImFwcGxpY2F0aW9uTmFtZSI6ICJkcml2ZSIsICJ1bmlxdWVRdWFsaWZpZXIiOiAiMjYxOTkxMDk0ODY4MDM1NjI2MCJ9LCAiZXRhZyI6ICJcImNKTW01QmJCWWRxMl80bGVma2pkZ3p3dHpwZy9YNVNnaE9KTUNBNWo0UTJDRU1WdDl4SmFXWjhcIiIsICJraW5kIjogImFkbWluI3JlcG9ydHMjYWN0aXZpdHkiLCAiYWN0b3IiOiB7ImVtYWlsIjogImdhbEByZWNvLmFpIiwgInByb2ZpbGVJZCI6ICIxMDAwMzQzODk1NzA1MDM5MjM1NTYifSwgImV2ZW50cyI6IFt7Im5hbWUiOiAiZWRpdCIsICJ0eXBlIjogImFjY2VzcyIsICJwYXJhbWV0ZXJzIjogW3sibmFtZSI6ICJwcmltYXJ5X2V2ZW50IiwgImJvb2xWYWx1ZSI6IGZhbHNlfSwgeyJuYW1lIjogImJpbGxhYmxlIiwgImJvb2xWYWx1ZSI6IHRydWV9LCB7Im5hbWUiOiAib3duZXJfaXNfdGVhbV9kcml2ZSIsICJib29sVmFsdWUiOiBmYWxzZX0sIHsibmFtZSI6ICJvd25lciIsICJ2YWx1ZSI6ICJnYWxAcmVjby5haSJ9LCB7Im5hbWUiOiAiZG9jX2lkIiwgInZhbHVlIjogIjEtR0ZBOTZaRXN6SGh0OVJJLUdJMTB6aTRBQVFEYjJxMjRlRmIyMTVEYk5BIn0sIHsibmFtZSI6ICJkb2NfdHlwZSIsICJ2YWx1ZSI6ICJkb2N1bWVudCJ9LCB7Im5hbWUiOiAiaXNfZW5jcnlwdGVkIiwgImJvb2xWYWx1ZSI6IGZhbHNlfSwgeyJuYW1lIjogImRvY190aXRsZSIsICJ2YWx1ZSI6ICJDb3B5IG9mIHNlbnNpdGl2ZSBmaWxlIn0sIHsibmFtZSI6ICJkbHBfaW5mbyIsICJ2YWx1ZSI6ICIifSwgeyJuYW1lIjogInZpc2liaWxpdHkiLCAidmFsdWUiOiAicGVvcGxlX3dpdGhfbGluayJ9LCB7Im5hbWUiOiAib3JpZ2luYXRpbmdfYXBwX2lkIiwgInZhbHVlIjogIjIxMTYwNDM1NTYwNyJ9LCB7Im5hbWUiOiAiYWN0b3JfaXNfY29sbGFib3JhdG9yX2FjY291bnQiLCAiYm9vbFZhbHVlIjogZmFsc2V9XX0sIHsibmFtZSI6ICJjaGFuZ2VfZG9jdW1lbnRfdmlzaWJpbGl0eSIsICJ0eXBlIjogImFjbF9jaGFuZ2UiLCAicGFyYW1ldGVycyI6IFt7Im5hbWUiOiAicHJpbWFyeV9ldmVudCIsICJib29sVmFsdWUiOiB0cnVlfSwgeyJuYW1lIjogImJpbGxhYmxlIiwgImJvb2xWYWx1ZSI6IHRydWV9LCB7Im5hbWUiOiAidmlzaWJpbGl0eV9jaGFuZ2UiLCAidmFsdWUiOiAiZXh0ZXJuYWwifSwgeyJuYW1lIjogInRhcmdldF9kb21haW4iLCAidmFsdWUiOiAiYWxsIn0sIHsibmFtZSI6ICJvbGRfdmFsdWUiLCAibXVsdGlWYWx1ZSI6IFsicHJpdmF0ZSJdfSwgeyJuYW1lIjogIm5ld192YWx1ZSIsICJtdWx0aVZhbHVlIjogWyJwZW9wbGVfd2l0aF9saW5rIl19LCB7Im5hbWUiOiAib2xkX3Zpc2liaWxpdHkiLCAidmFsdWUiOiAicHJpdmF0ZSJ9LCB7Im5hbWUiOiAib3duZXJfaXNfdGVhbV9kcml2ZSIsICJib29sVmFsdWUiOiBmYWxzZX0sIHsibmFtZSI6ICJvd25lciIsICJ2YWx1ZSI6ICJnYWxAcmVjby5haSJ9LCB7Im5hbWUiOiAiZG9jX2lkIiwgInZhbHVlIjogIjEtR0ZBOTZaRXN6SGh0OVJJLUdJMTB6aTRBQVFEYjJxMjRlRmIyMTVEYk5BIn0sIHsibmFtZSI6ICJkb2NfdHlwZSIsICJ2YWx1ZSI6ICJkb2N1bWVudCJ9LCB7Im5hbWUiOiAiaXNfZW5jcnlwdGVkIiwgImJvb2xWYWx1ZSI6IGZhbHNlfSwgeyJuYW1lIjogImRvY190aXRsZSIsICJ2YWx1ZSI6ICJDb3B5IG9mIHNlbnNpdGl2ZSBmaWxlIn0sIHsibmFtZSI6ICJkbHBfaW5mbyIsICJ2YWx1ZSI6ICIifSwgeyJuYW1lIjogInZpc2liaWxpdHkiLCAidmFsdWUiOiAicGVvcGxlX3dpdGhfbGluayJ9LCB7Im5hbWUiOiAib3JpZ2luYXRpbmdfYXBwX2lkIiwgInZhbHVlIjogIjIxMTYwNDM1NTYwNyJ9LCB7Im5hbWUiOiAiYWN0b3JfaXNfY29sbGFib3JhdG9yX2FjY291bnQiLCAiYm9vbFZhbHVlIjogZmFsc2V9XX0sIHsibmFtZSI6ICJjaGFuZ2VfZG9jdW1lbnRfYWNjZXNzX3Njb3BlIiwgInR5cGUiOiAiYWNsX2NoYW5nZSIsICJwYXJhbWV0ZXJzIjogW3sibmFtZSI6ICJwcmltYXJ5X2V2ZW50IiwgImJvb2xWYWx1ZSI6IHRydWV9LCB7Im5hbWUiOiAiYmlsbGFibGUiLCAiYm9vbFZhbHVlIjogdHJ1ZX0sIHsibmFtZSI6ICJ2aXNpYmlsaXR5X2NoYW5nZSIsICJ2YWx1ZSI6ICJleHRlcm5hbCJ9LCB7Im5hbWUiOiAidGFyZ2V0X2RvbWFpbiIsICJ2YWx1ZSI6ICJhbGwifSwgeyJuYW1lIjogIm9sZF92YWx1ZSIsICJtdWx0aVZhbHVlIjogWyJub25lIl19LCB7Im5hbWUiOiAibmV3X3ZhbHVlIiwgIm11bHRpVmFsdWUiOiBbImNhbl92aWV3Il19LCB7Im5hbWUiOiAib2xkX3Zpc2liaWxpdHkiLCAidmFsdWUiOiAicHJpdmF0ZSJ9LCB7Im5hbWUiOiAib3duZXJfaXNfdGVhbV9kcml2ZSIsICJib29sVmFsdWUiOiBmYWxzZX0sIHsibmFtZSI6ICJvd25lciIsICJ2YWx1ZSI6ICJnYWxAcmVjby5haSJ9LCB7Im5hbWUiOiAiZG9jX2lkIiwgInZhbHVlIjogIjEtR0ZBOTZaRXN6SGh0OVJJLUdJMTB6aTRBQVFEYjJxMjRlRmIyMTVEYk5BIn0sIHsibmFtZSI6ICJkb2NfdHlwZSIsICJ2YWx1ZSI6ICJkb2N1bWVudCJ9LCB7Im5hbWUiOiAiaXNfZW5jcnlwdGVkIiwgImJvb2xWYWx1ZSI6IGZhbHNlfSwgeyJuYW1lIjogImRvY190aXRsZSIsICJ2YWx1ZSI6ICJDb3B5IG9mIHNlbnNpdGl2ZSBmaWxlIn0sIHsibmFtZSI6ICJkbHBfaW5mbyIsICJ2YWx1ZSI6ICIifSwgeyJuYW1lIjogInZpc2liaWxpdHkiLCAidmFsdWUiOiAicGVvcGxlX3dpdGhfbGluayJ9LCB7Im5hbWUiOiAib3JpZ2luYXRpbmdfYXBwX2lkIiwgInZhbHVlIjogIjIxMTYwNDM1NTYwNyJ9LCB7Im5hbWUiOiAiYWN0b3JfaXNfY29sbGFib3JhdG9yX2FjY291bnQiLCAiYm9vbFZhbHVlIjogZmFsc2V9XX1dLCAiaXBBZGRyZXNzIjogIjIxMi4xOTkuNDcuMTg2In0sICJlbnJpY2hlZCI6IHsiYWN0b3IiOiB7ImRvbWFpbiI6ICJAcmVjby5haSIsICJsYWJlbHMiOiBbeyJsYWJlbCI6IHsibmFtZSI6ICJMZWF2aW5nIE9yZyBVc2VyIiwgInR5cGUiOiAiTEFCRUxfVFlQRV9JTkZPUk1BVElWRSIsICJ0b29sdGlwIjogImxlYXZpbmcgZW1wbG95ZWUiLCAiY3JlYXRlZF9ieSI6IDEwLCAicmlza19sZXZlbCI6IDAsICJkZXNjcmlwdGlvbiI6ICJMZWF2aW5nIGVtcGxveWVlIn19XSwgImNhdGVnb3J5IjogImludGVybmFsX3ByaW1hcnlfZW1haWwiLCAiZnVsbF9uYW1lIjogIkdhbCBOYWthc2giLCAidXNlcl9qc29uIjogW3sidXNlciI6IHsibmFtZSI6ICJHYWwgTmFrYXNoIiwgImVtYWlsIjogImdhbEByZWNvLmFpIiwgInByb2ZpbGVfcGhvdG8iOiAiaHR0cHM6Ly9saDMuZ29vZ2xldXNlcmNvbnRlbnQuY29tL2EvQUdObXl4YW1RT2FwZkpVaW9lVXVhcFg0elRDWXl2amJEQkNzNmUta1d3TTI9czEwMCJ9fV0sICJkZXBhcnRtZW50IjogIiIsICJlbWFpbF9hY2NvdW50IjogImdhbEByZWNvLmFpIiwgIm9yZGVyZWRfZ3JvdXBzIjogWyJSZWNvIElMIiwgIlJlY28gQWxsIiwgIkdyb3VwIFJORCIsICJyZWNvLWFsbCIsICJHcm91cCBHVE0iLCAiVGVhbSBEZXZPcHMiLCAiRm91bmRlcnMiLCAiRXZlcnlvbmUiXSwgInBfNTBfZG93bmxvYWRzIjogMCwgInBfOTBfZG93bmxvYWRzIjogMi40MDAwMDAwMDAwMDAwMDI2LCAiYXZlcmFnZV9kb3dubG9hZHMiOiAwLjk2NjY2NjY2NjY2NjY2Njd9LCAiYXNzZXQiOiB7ImxhYmVscyI6IFt7ImxhYmVsIjogeyJuYW1lIjogIkNvbmZpZGVudGlhbCBHZW5lcmFsIiwgInR5cGUiOiAiTEFCRUxfVFlQRV9CVVNJTkVTU19JTVBBQ1QiLCAidG9vbHRpcCI6ICJEYXRhIGFzc2V0cyByZWdhcmRsZXNzIG9mIHRoZWlyIHR5cGUsIGRlZmluZWQgYXMgc2Vuc2l0aXZlIGJlY2F1c2UgdGhhdCB0ZXJtcyBzdWNoIGFzIFwiY29uZmlkZW50aWFsXCIsIFwicHJpdmF0ZVwiLCBcInNlbnNpdGl2ZVwiIHdlcmUgZGV0ZWN0ZWQgaW4gdGhlIGFzc2V0IG5hbWUiLCAiY3JlYXRlZF9ieSI6IDEwLCAicmlza19sZXZlbCI6IDQwLCAiZGVzY3JpcHRpb24iOiAiRGF0YSBhc3NldHMgcmVnYXJkbGVzcyBvZiB0aGVpciB0eXBlLCBkZWZpbmVkIGFzIHNlbnNpdGl2ZSBiZWNhdXNlIHRoYXQgdGVybXMgc3VjaCBhcyBcImNvbmZpZGVudGlhbFwiLCBcInByaXZhdGVcIiwgXCJzZW5zaXRpdmVcIiB3ZXJlIGRldGVjdGVkIGluIHRoZSBhc3NldCBuYW1lIiwgImRhdGFfY2F0ZWdvcnkiOiAiQ29uZmlkZW50aWFsIC0gR2xvYmFsIn19XSwgInNvdXJjZSI6ICJHU1VJVEVfR0RSSVZFX0FVRElUX0xPR19BUEkiLCAiZmlsZV9pZCI6ICIxLUdGQTk2WkVzekhodDlSSS1HSTEwemk0QUFRRGIycTI0ZUZiMjE1RGJOQSIsICJmaWxlX3VybCI6IFt7ImFzc2V0IjogeyJsaW5rIjogImh0dHBzOi8vZHJpdmUuZ29vZ2xlLmNvbS9maWxlL2QvMS1HRkE5NlpFc3pIaHQ5UkktR0kxMHppNEFBUURiMnEyNGVGYjIxNURiTkEiLCAidmFsdWUiOiAiQ29weSBvZiBzZW5zaXRpdmUgZmlsZSJ9LCAiYXNzZXRfaWQiOiAiMS1HRkE5NlpFc3pIaHQ5UkktR0kxMHppNEFBUURiMnEyNGVGYjIxNURiTkEiLCAiZGF0YV9zb3VyY2UiOiAiR1NVSVRFX0dEUklWRV9BVURJVF9MT0dfQVBJIn1dLCAibG9jYXRpb24iOiAiZ2FsQHJlY28uYWkiLCAiZmlsZV9uYW1lIjogIkNvcHkgb2Ygc2Vuc2l0aXZlIGZpbGUiLCAiZmlsZV9zaXplIjogMCwgImZpbGVfdHlwZSI6ICJkb2N1bWVudCIsICJmaWxlX293bmVyIjogImdhbEByZWNvLmFpIiwgInZpc2liaWxpdHkiOiAiUEVSTUlTU0lPTl9UWVBFX1BFT1BMRV9XSVRIX0xJTksiLCAiZGVsZXRlX3N0YXRlIjogImFjdGl2ZSIsICJkYXRhX2NhdGVnb3JpZXMiOiBbIkNvbmZpZGVudGlhbCAtIEdsb2JhbCJdLCAibGFzdF91cGRhdGVfdGltZSI6ICIyMDIzLTA0LTIwVDE4OjMyOjMyKzAwOjAwIiwgInNlbnNpdGl2aXR5X2xldmVsIjogNDAsICJjdXJyZW50bHlfcGVybWl0dGVkX3VzZXJzIjogWyJnYWxAcmVjby5haSJdfX0sICJwb2xpY3lfaWQiOiAiN2YxNzQ1ODAtODFiNi00MzQ5LWE0Y2EtY2E4ODczYjQ4Yjc4IiwgInZpb2xhdGlvbiI6IHsiZW5yaWNoZWQuYXNzZXQuc2Vuc2l0aXZpdHlfbGV2ZWwgaW4gWzMwLCA0MCwgXCIzMFwiLCBcIjQwXCJdIG9yIGVucmljaGVkLmFzc2V0LnJpc2tfbGV2ZWwgaW4gWzMwLCA0MCwgXCIzMFwiLCBcIjQwXCJdIjogdHJ1ZSwgIiRjb3VudChwYXlsb2FkLmV2ZW50cy5wYXJhbWV0ZXJzW25hbWU9XCJ2aXNpYmlsaXR5XCIgYW5kIHZhbHVlPVwicGVvcGxlX3dpdGhfbGlua1wiXSkgPiAwIGFuZCAkY291bnQocGF5bG9hZC5ldmVudHMucGFyYW1ldGVyc1tuYW1lPVwib2xkX3Zpc2liaWxpdHlcIiBhbmQgdmFsdWU9XCJwZW9wbGVfd2l0aF9saW5rXCJdKSA8PSAwIGFuZCAkY291bnQocGF5bG9hZC5ldmVudHMucGFyYW1ldGVyc1tuYW1lPVwidmlzaWJpbGl0eV9jaGFuZ2VcIiBhbmQgdmFsdWU9XCJleHRlcm5hbFwiXSkgPiAwIjogdHJ1ZSwgIigkY291bnQocGF5bG9hZC5ldmVudHNbdHlwZT1cImFjbF9jaGFuZ2VcIl0pID4gMCAgIG9yICRjb3VudChwYXlsb2FkLmV2ZW50c1tuYW1lPVwiY2hhbmdlX2RvY3VtZW50X3Zpc2liaWxpdHlcIl0pID4gMCAgIG9yICRjb3VudChwYXlsb2FkLmV2ZW50c1tuYW1lPVwiY2hhbmdlX2RvY3VtZW50X2FjY2Vzc19zY29wZVwiXSkpIGFuZCAoICAgICAgIHBheWxvYWQuZXZlbnRzW3R5cGU9XCJhY2xfY2hhbmdlXCJdLnBhcmFtZXRlcnNbbmFtZT1cInByaW1hcnlfZXZlbnRcIl0uYm9vbFZhbHVlICAgICAgIG9yICAgICAgICBwYXlsb2FkLmV2ZW50c1tuYW1lPVwiY2hhbmdlX2RvY3VtZW50X3Zpc2liaWxpdHlcIl0ucGFyYW1ldGVyc1tuYW1lPVwicHJpbWFyeV9ldmVudFwiXS5ib29sVmFsdWUgICAgICAgb3IgICAgICAgcGF5bG9hZC5ldmVudHNbbmFtZT1cImNoYW5nZV9kb2N1bWVudF9hY2Nlc3Nfc2NvcGVcIl0ucGFyYW1ldGVyc1tuYW1lPVwicHJpbWFyeV9ldmVudFwiXS5ib29sVmFsdWUpIjogdHJ1ZX0sICJwb2xpY3lfdGFncyI6IFsiUmVjbyBSZWNvbW1lbmRlZCIsICIiXSwgInBvbGljeV90aXRsZSI6ICJTZW5zaXRpdmUgYXNzZXRzIGluIEdvb2dsZSBEcml2ZSBleHBvc2VkIHB1YmxpY2x5IiwgIndvcmtzcGFjZV9pZCI6ICJmNGJlNDIwMi00YTZlLTRiZjctYjA1OC01ZDMzYTMxNDZkZjciLCAiYWRkaXRpb25hbF9kYXRhIjogeyJ4LXJlYWwtaXAiOiAiMTAuMi4xNzIuMTAyIiwgIjphdXRob3JpdHkiOiAiZ3N1aXRlLWV4dHJhY3Rvci1zZXJ2aWNlOjUwMDUxIiwgInVzZXItYWdlbnQiOiAiZ3JwYy1nby8xLjU0LjAiLCAiY29udGVudC10eXBlIjogImFwcGxpY2F0aW9uL2dycGMiLCAieC1hbXpuLXRyYWNlLWlkIjogIlJvb3Q9MS02NDUyMzU4OC0zMzUyYjUzYzBmNTg0YzM4Njg2OTE1MDIiLCAieC1mb3J3YXJkZWQtZm9yIjogIjc0LjEyNS4yMTAuMTkwLDc0LjEyNS4yMTAuMTkwLCAxMC4yLjExMS4xNzIiLCAieC1mb3J3YXJkZWQtaG9zdCI6ICJkZW1vLnJlY28uYWkiLCAieC1mb3J3YXJkZWQtcG9ydCI6ICI0NDMiLCAieC1mb3J3YXJkZWQtcHJvdG8iOiAiaHR0cHMiLCAieC1nb29nLWNoYW5uZWwtaWQiOiAiODE0ZTA5MDktZTA0Mi00MjlmLWE3ZTYtYjIxY2ZmYWIxMzY0IiwgImdycGNnYXRld2F5LWFjY2VwdCI6ICIqLyoiLCAieC1nb29nLXJlc291cmNlLWlkIjogIlVSQzEwQ2F0ek01V2wweW8taTBtczRsYWdzVSIsICJ4LWdvb2ctcmVzb3VyY2UtdXJpIjogImh0dHBzOi8vYWRtaW4uZ29vZ2xlYXBpcy5jb20vYWRtaW4vcmVwb3J0cy92MS9hY3Rpdml0eS91c2Vycy9hbGwvYXBwbGljYXRpb25zL2RyaXZlP2FsdD1qc29uJm9yZ1VuaXRJRCZwcmV0dHlQcmludD1mYWxzZSIsICJ4LWdvb2ctY2hhbm5lbC10b2tlbiI6ICJMcndpdTF6b09RRWE1SmE1OW85ODBxQzVTakZ1Y1ZJR08ySWtjSVhaSVAyd1FDc0JPLTF0V3JIM3NvQUVCZk5MWWdaUDRua0pnOTRVZVEwZHlNQ3FITjlKLS02TUNXWktWdTk4V294QWJlNXFhRjNWeUFlT2lxVEMwcmRvNEFISiIsICJ4LWdvb2ctbWVzc2FnZS1udW1iZXIiOiAiNzkyNjM3OCIsICJ4LWdvb2ctcmVzb3VyY2Utc3RhdGUiOiAiZWRpdCIsICJncnBjZ2F0ZXdheS11c2VyLWFnZW50IjogIkFQSXMtR29vZ2xlOyAoK2h0dHBzOi8vZGV2ZWxvcGVycy5nb29nbGUuY29tL3dlYm1hc3RlcnMvQVBJcy1Hb29nbGUuaHRtbCkiLCAiZ3JwY2dhdGV3YXktY29udGVudC10eXBlIjogImFwcGxpY2F0aW9uL2pzb247IGNoYXJzZXQ9VVRGLTgiLCAieC1nb29nLWNoYW5uZWwtZXhwaXJhdGlvbiI6ICJXZWQsIDAzIE1heSAyMDIzIDEzOjA4OjQ4IEdNVCJ9LCAiZXh0cmFjdGlvbl90aW1lIjogeyJuYW5vcyI6IDc5MjM2NTEwMSwgIm1pbGxpcyI6IDE2ODMxMDkyNTY3OTIsICJzZWNvbmRzIjogMTY4MzEwOTI1NiwgInRpbWVzdGFtcCI6ICIyMDIzLTA1LTAzVDEwOjIwOjU2Ljc5MjM2NTEwMVoifSwgImV4dHJhY3Rpb25fc291cmNlIjogMTAsICJleHRyYWN0aW9uX3NvdXJjZV91cHBlcl9jYXNlIjogIkdTVUlURV9HRFJJVkVfQVVESVRfTE9HX0FQSSJ9", # noqa - "createdAt": "2023-05-03T10:21:03.926534Z", - "policyStatusOnViolationCreation": "POLICY_STATUS_ON", + "jsonData": json.dumps({"violation": True, "id": "91bef1de"}), } ], - "createdAt": "2023-05-03T10:21:04.765477Z", - "updatedAt": "2023-05-03T10:21:04.078627Z", - }, + } } - alerts_table = GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData( - rows=[ - RowData( - cells=[ - KeyValuePair( - key="id", - value=base64.b64encode( - "ZWU1OTNkYzItYTUwZS00MTVlLWJlZDAtODQwM2MxOGIyNmNh".encode(ENCODING) - ).decode(ENCODING), - ), - KeyValuePair( - key="description", - value=base64.b64encode(INCIDENT_DESCRIPTION.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="risk_level", - value=base64.b64encode("30".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="created_at", - value=base64.b64encode(datetime.datetime.now().strftime(TIME_FORMAT).encode(ENCODING)).decode( - ENCODING - ), - ), - KeyValuePair( - key="updated_at", - value=base64.b64encode(datetime.datetime.now().strftime(TIME_FORMAT).encode(ENCODING)).decode( - ENCODING - ), - ), - KeyValuePair( - key="status", - value=base64.b64encode(INCIDENT_STATUS.encode(ENCODING)).decode(ENCODING), - ), - ] - ) - ] - ), - total_number_of_results=1, - table_definition="", - dynamic_table_definition="", - token="", - ), - ) - return alerts_table, alert + + +def build_alerts_list_response(alert_ids: list[str]) -> dict[str, Any]: + """Build an alerts/list response (external API `GET /external-api/alerts/list`).""" + return {"alerts": [{"id": alert_id} for alert_id in alert_ids], "totalResults": len(alert_ids)} def get_random_assets_user_has_access_to_response() -> GetIncidentTableResponse: @@ -241,112 +130,6 @@ def get_random_assets_user_has_access_to_response() -> GetIncidentTableResponse: ) -def get_random_risky_users_response() -> GetIncidentTableResponse: - return GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData( - rows=[ - RowData( - cells=[ - KeyValuePair( - key="full_name", - value=base64.b64encode("John Doe".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="identity_id", - value=base64.b64encode(f"{uuid.uuid4()}".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="email_account", - value=base64.b64encode(f"{uuid.uuid4()}@acme.com".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="risk_level", - value=base64.b64encode(str(RiskLevel.HIGH.value).encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="added_by", - value=base64.b64encode("system".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="created_at", - value=base64.b64encode(datetime.datetime.now().strftime(TIME_FORMAT).encode(ENCODING)).decode( - ENCODING - ), - ), - ] - ) - ] - ), - total_number_of_results=1, - table_definition="", - dynamic_table_definition="", - token="", - ), - ) - - -def get_random_user_context_response() -> GetIncidentTableResponse: - return GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData( - rows=[ - RowData( - cells=[ - KeyValuePair( - key="email_account", - value=base64.b64encode("charles@corp.com".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="departments", - value=base64.b64encode('["Pro"]'.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="job_titles", - value=base64.b64encode('["VP Product"]'.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="category", - value=base64.b64encode("external".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="groups", - value=base64.b64encode('["Product"]'.encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="full_name", - value=base64.b64encode("Yossi".encode(ENCODING)).decode(ENCODING), - ), - KeyValuePair( - key="labels", - value=base64.b64encode( - '["{\\"label\\": {\\"name\\": \\"VIP User\\",' - ' \\"type\\": \\"LABEL_TYPE_INFORMATIVE\\",' - ' \\"tooltip\\": \\"VIP User\\", \\"created_by\\": 10,' - ' \\"risk_level\\": 0, \\"description\\": \\"VIP User\\"}}",' - '"{\\"label\\": {\\"name\\": \\"GSuite Admin\\",' - ' \\"type\\": \\"LABEL_TYPE_INFORMATIVE\\",' - ' \\"tooltip\\": \\"GSuite Admin\\", \\"created_by\\": 10,' - ' \\"risk_level\\": 0, \\"description\\": \\"GSuite Admin\\"}}",' - '"{\\"label\\": {\\"name\\": \\"Okta Admin\\",' - ' \\"type\\": \\"LABEL_TYPE_INFORMATIVE\\",' - ' \\"tooltip\\": \\"Okta Admin\\", ' - '\\"created_by\\": 10, \\"risk_level\\": 0,' - '\\"description\\": \\"Okta Admin\\"}}"]'.encode(ENCODING) - ).decode(ENCODING), - ), - ] - ) - ] - ), - total_number_of_results=1, - table_definition="", - dynamic_table_definition="", - token="", - ), - ) - - def get_mock_assets() -> list[dict[str, Any]]: return { "assets": [ @@ -364,8 +147,7 @@ def get_mock_assets() -> list[dict[str, Any]]: def test_test_module_success(requests_mock, reco_client: RecoClient) -> None: - mock_response = {"alerts": {"tablesMetadata": [{"name": "table1"}]}} - requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox?limit=1", json=mock_response) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json={"alerts": [], "totalResults": 0}) res = reco_client.validate_api_key() assert res == "ok" @@ -378,44 +160,48 @@ def reco_client() -> RecoClient: def test_fetch_incidents_should_succeed(requests_mock, reco_client: RecoClient) -> None: - random_alerts_response, alert = get_alerts_and_table_response() - requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/{ALERT_ID}", json=alert) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json=random_alerts_response) + created_at = datetime.datetime.now().strftime(TIME_FORMAT) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json=build_alerts_list_response([ALERT_ID])) + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/alert-details/{ALERT_ID}", + json=build_alert_detail(ALERT_ID, INCIDENT_DESCRIPTION, 30, created_at), + ) last_run, fetched_incidents = fetch_incidents( reco_client=reco_client, - risk_level=40, + risk_levels=["CRITICAL"], source="test", before=datetime.datetime.now(), last_run={}, max_fetch=1, ) - expected_count = random_alerts_response.getTableResponse.total_number_of_results - assert len(fetched_incidents) == expected_count - assert fetched_incidents[0].get("name") == "1 sensitive file exposed publicly by t@acme.ai (part of ACME IL)" - assert fetched_incidents[0].get("dbotMirrorId") == INCIDET_ID_UUID + assert len(fetched_incidents) == 1 + assert fetched_incidents[0].get("name") == INCIDENT_DESCRIPTION + assert fetched_incidents[0].get("dbotMirrorId") == ALERT_ID assert fetched_incidents[0].get("severity") == 3 # risk_level 30 -> Demisto high res_json = json.loads(fetched_incidents[0].get("rawJSON")) assert "id" in res_json def test_fetch_same_incidents(requests_mock, reco_client: RecoClient) -> None: - random_alerts_response, alert = get_alerts_and_table_response() - requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/{ALERT_ID}", json=alert) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json=random_alerts_response) + created_at = datetime.datetime.now().strftime(TIME_FORMAT) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json=build_alerts_list_response([ALERT_ID])) + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/alert-details/{ALERT_ID}", + json=build_alert_detail(ALERT_ID, INCIDENT_DESCRIPTION, 30, created_at), + ) last_run, fetched_incidents = fetch_incidents( reco_client=reco_client, - risk_level=40, + risk_levels=["CRITICAL"], before=datetime.datetime.now(), last_run={}, max_fetch=1, ) - expected_count = random_alerts_response.getTableResponse.total_number_of_results - assert len(fetched_incidents) == expected_count + assert len(fetched_incidents) == 1 last_run, incidents = fetch_incidents( reco_client=reco_client, - risk_level=40, + risk_levels=["CRITICAL"], before=datetime.datetime.now(), last_run=last_run, max_fetch=1, @@ -424,31 +210,23 @@ def test_fetch_same_incidents(requests_mock, reco_client: RecoClient) -> None: def test_fetch_incidents_without_assets_info(requests_mock, reco_client: RecoClient) -> None: - random_alerts_response, alert = get_alerts_and_table_response() - requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/{ALERT_ID}", json=alert) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json=random_alerts_response) + created_at = datetime.datetime.now().strftime(TIME_FORMAT) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json=build_alerts_list_response([ALERT_ID])) + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/alert-details/{ALERT_ID}", + json=build_alert_detail(ALERT_ID, INCIDENT_DESCRIPTION, 30, created_at), + ) last_run, fetched_incidents = fetch_incidents(reco_client=reco_client, last_run={}, source="GOOGLE_DRIVE", max_fetch=1) - expected_count = random_alerts_response.getTableResponse.total_number_of_results - - assert len(fetched_incidents) == expected_count - assert fetched_incidents[0].get("name") == "1 sensitive file exposed publicly by t@acme.ai (part of ACME IL)" - assert fetched_incidents[0].get("dbotMirrorId") == INCIDET_ID_UUID + assert len(fetched_incidents) == 1 + assert fetched_incidents[0].get("name") == INCIDENT_DESCRIPTION + assert fetched_incidents[0].get("dbotMirrorId") == ALERT_ID res_json = json.loads(fetched_incidents[0].get("rawJSON")) assert "id" in res_json def test_empty_response(requests_mock, reco_client: RecoClient) -> None: - table_empty = GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData(rows=[]), - total_number_of_results=0, - table_definition="", - dynamic_table_definition="", - token="", - ) - ) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json=table_empty) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json={"alerts": [], "totalResults": 0}) last_run, fetched_incidents = fetch_incidents(reco_client=reco_client, last_run={}, max_fetch=1) assert len(fetched_incidents) == 0 @@ -456,16 +234,7 @@ def test_empty_response(requests_mock, reco_client: RecoClient) -> None: def test_empty_valid_response(requests_mock, reco_client: RecoClient) -> None: - table_empty = GetIncidentTableResponse( - get_table_response=GetTableResponse( - data=TableData(rows=[]), - total_number_of_results=0, - table_definition="", - dynamic_table_definition="", - token="", - ) - ) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json=table_empty) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json={"alerts": [], "totalResults": 0}) last_run, fetched_incidents = fetch_incidents(reco_client=reco_client, last_run={}, max_fetch=1) assert len(fetched_incidents) == 0 @@ -473,12 +242,13 @@ def test_empty_valid_response(requests_mock, reco_client: RecoClient) -> None: def test_invalid_response(requests_mock, reco_client: RecoClient) -> None: - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/policy-subsystem/alert-inbox/table", json={"getTableResponse": {}}) + """A malformed alerts/list response (missing the 'alerts' key) yields zero incidents, no exception.""" + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/alerts/list", json={}) last_run, fetched_incidents = fetch_incidents( reco_client=reco_client, last_run={}, max_fetch=1, - risk_level=RiskLevel.HIGH.value, + risk_levels=["HIGH", "CRITICAL"], source="GSUITE_GDRIVE_AUDIT_LOG_API", ) @@ -507,6 +277,23 @@ def test_alert_mapper(): assert map_reco_alert_score_to_demisto_score("CRITICAL") == 4 +def test_parse_minimum_risk_level_expands_to_higher_severities(): + """A single risk_level value expands to itself and every severity above it.""" + assert parse_minimum_risk_level("MEDIUM") == ["MEDIUM", "HIGH", "CRITICAL"] + assert parse_minimum_risk_level("LOW") == ["LOW", "MEDIUM", "HIGH", "CRITICAL"] + assert parse_minimum_risk_level("CRITICAL") == ["CRITICAL"] + + +def test_parse_minimum_risk_level_accepts_numeric_and_lowercase(): + assert parse_minimum_risk_level("30") == ["HIGH", "CRITICAL"] + assert parse_minimum_risk_level("medium") == ["MEDIUM", "HIGH", "CRITICAL"] + + +def test_parse_minimum_risk_level_empty_returns_none(): + assert parse_minimum_risk_level(None) is None + assert parse_minimum_risk_level("") is None + + def test_parse_alerts_to_incidents_numeric_risk(): """parse_alerts_to_incidents maps numeric risk_level (10-40) to severity.""" alerts = [ @@ -562,7 +349,7 @@ def test_max_fetch(): def test_update_reco_incident_timeline(requests_mock, reco_client: RecoClient) -> None: incident_id = uuid.uuid1() requests_mock.post( - f"{DUMMY_RECO_API_DNS_NAME}/share-service/share-comment", + f"{DUMMY_RECO_API_DNS_NAME}/external-api/comments/create", json={}, status_code=200, ) @@ -573,7 +360,7 @@ def test_update_reco_incident_timeline(requests_mock, reco_client: RecoClient) - def test_update_reco_incident_timeline_error(capfd, requests_mock, reco_client: RecoClient) -> None: incident_id = uuid.uuid1() requests_mock.post( - f"{DUMMY_RECO_API_DNS_NAME}/share-service/share-comment", + f"{DUMMY_RECO_API_DNS_NAME}/external-api/comments/create", json={}, status_code=404, ) @@ -596,22 +383,30 @@ def test_resolve_visibility_event_error(capfd, requests_mock, reco_client: RecoC def test_get_risky_users(requests_mock, reco_client: RecoClient) -> None: - raw_result = get_random_risky_users_response() - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/risk-management/get-risk-management-table", - json=raw_result, + accounts = [ + { + "id": str(uuid.uuid4()), + "name": "John Doe", + "accountEmail": f"{uuid.uuid4()}@acme.com", + "isAdmin": False, + "isRiskyUser": True, + } + ] + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/accounts/list", + json={"accounts": accounts, "totalResults": len(accounts)}, status_code=200, ) actual_result = get_risky_users_from_reco(reco_client=reco_client) - assert len(actual_result.outputs) == len(raw_result.getTableResponse.data.rows) - assert "@" in actual_result.outputs[0].get("email_account") + assert len(actual_result.outputs) == len(accounts) + assert "@" in actual_result.outputs[0].get("accountEmail") def test_get_risky_users_bad_response(capfd, requests_mock, reco_client: RecoClient) -> None: - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/risk-management/get-risk-management-table", + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/accounts/list", json={}, - status_code=200, + status_code=500, ) with capfd.disabled(), pytest.raises(Exception): get_risky_users_from_reco(reco_client=reco_client) @@ -619,13 +414,13 @@ def test_get_risky_users_bad_response(capfd, requests_mock, reco_client: RecoCli def test_add_risky_user_label(requests_mock, reco_client: RecoClient) -> None: label_id = f"{uuid.uuid1()}@gmail.com" - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/entry-label-relations", json={}, status_code=200) - raw_result = get_random_risky_users_response() - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/risk-management/get-risk-management-table", - json=raw_result, + identity_id = str(uuid.uuid4()) + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/users/list", + json={"users": [{"id": identity_id, "email": label_id}], "totalResults": 1}, status_code=200, ) + requests_mock.post(f"{DUMMY_RECO_API_DNS_NAME}/external-api/labels/add", json={}, status_code=200) res = add_risky_user_label(reco_client=reco_client, email_address=label_id) assert "labeled as risky" in res.readable_output @@ -649,19 +444,19 @@ def test_get_assets_user_bad_response(capfd, requests_mock, reco_client: RecoCli def test_get_sensitive_assets_by_name(requests_mock, reco_client: RecoClient) -> None: - raw_result = get_random_assets_user_has_access_to_response() - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/query", json=raw_result, status_code=200) + files = [{"id": "asset-1", "name": "sensitive.txt", "owner": "test@acme.com", "sensitivityLevel": "40"}] + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/files/list", json={"files": files, "totalResults": len(files)}) actual_result = get_sensitive_assets_by_name(reco_client=reco_client, asset_name="test", regex_search=True) - assert len(actual_result.outputs) == len(raw_result.getTableResponse.data.rows) - assert actual_result.outputs[0].get("source") is not None + assert len(actual_result.outputs) == len(files) + assert actual_result.outputs[0].get("id") is not None def test_get_sensitive_assets_by_id(requests_mock, reco_client: RecoClient) -> None: - raw_result = get_random_assets_user_has_access_to_response() - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/query", json=raw_result, status_code=200) + files = [{"id": "asset-id", "name": "sensitive.txt", "owner": "test@acme.com", "sensitivityLevel": "40"}] + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/files/list", json={"files": files, "totalResults": len(files)}) actual_result = get_sensitive_assets_by_id(reco_client=reco_client, asset_id="asset-id") - assert len(actual_result.outputs) == len(raw_result.getTableResponse.data.rows) - assert actual_result.outputs[0].get("source") is not None + assert len(actual_result.outputs) == len(files) + assert actual_result.outputs[0].get("id") is not None def test_get_link_to_user_overview_page(requests_mock, reco_client: RecoClient) -> None: @@ -684,7 +479,7 @@ def test_get_link_to_user_overview_page_error(capfd, requests_mock, reco_client: requests_mock.get( f"{DUMMY_RECO_API_DNS_NAME}/risk-management/risk-management/link?link_type={link_type}¶m={entity_id}", json={}, - status_code=200, + status_code=404, ) with capfd.disabled(), pytest.raises(Exception): get_link_to_user_overview_page(reco_client=reco_client, entity=entity_id, link_type=link_type) @@ -808,24 +603,18 @@ def test_get_alert_summary_error(capfd, requests_mock, reco_client: RecoClient) def test_get_user_context_by_email(requests_mock, reco_client: RecoClient) -> None: - raw_result = get_random_user_context_response() - requests_mock.post(f"{DUMMY_RECO_API_DNS_NAME}/asset-management", json=raw_result, status_code=200) - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/risk-management/get-risk-management-table", - json=raw_result, - status_code=200, - ) + users = [{"email": "charles@corp.com", "fullName": "Yossi", "departments": ["Pro"], "category": "external"}] + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/users/list", json={"users": users, "totalResults": len(users)}) res = get_user_context_by_email_address(reco_client, "charles@corp.com") assert res.outputs_prefix == "Reco.User" - assert res.outputs.get("email_account") != "" - assert res.outputs.get("email_account") == "charles@corp.com" + assert res.outputs.get("email") != "" + assert res.outputs.get("email") == "charles@corp.com" def test_get_app_discovery_with_filters(requests_mock, reco_client: RecoClient) -> None: """Test the get_app_discovery method with date filters.""" - raw_result = get_random_assets_user_has_access_to_response() - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/query", json=raw_result, status_code=200) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/count", json=raw_result, status_code=200) + apps = [{"id": "slack.com", "name": "Slack", "category": "Communication"}] + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/list", json={"apps": apps, "totalResults": len(apps)}) # Test with date filters from datetime import datetime, timedelta @@ -833,20 +622,18 @@ def test_get_app_discovery_with_filters(requests_mock, reco_client: RecoClient) before = datetime.now() after = datetime.now() - timedelta(days=30) - apps = reco_client.get_app_discovery(before=before, after=after, limit=100) + result = reco_client.get_app_discovery(before=before, after=after, limit=100) # Verify the response - assert isinstance(apps, list) + assert isinstance(result, list) def test_get_app_discovery_error(capfd, requests_mock, reco_client: RecoClient) -> None: """Test error handling in get_app_discovery.""" - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/query", json={}, status_code=200) - requests_mock.put(f"{DUMMY_RECO_API_DNS_NAME}/asset-management/count", json={}, status_code=500) + requests_mock.get(f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/list", json={}, status_code=500) - with capfd.disabled(): - apps = reco_client.get_app_discovery() - assert len(apps) == 0 + with capfd.disabled(), pytest.raises(Exception): + reco_client.get_app_discovery() def test_set_app_authorization_status(requests_mock, reco_client: RecoClient) -> None: @@ -855,13 +642,13 @@ def test_set_app_authorization_status(requests_mock, reco_client: RecoClient) -> authorization_status = "AUTH_STATUS_SANCTIONED" requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/app-risk-management/insert-risk-management-app", - json={"rows": 1}, + f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/{app_id}/auth-status", + json={"authorizationStatus": authorization_status}, status_code=200, ) response = reco_client.set_app_authorization_status(app_id, authorization_status) - assert response == {"rows": 1} + assert response == {"authorizationStatus": authorization_status} def test_set_app_authorization_status_command(requests_mock, reco_client: RecoClient) -> None: @@ -870,8 +657,8 @@ def test_set_app_authorization_status_command(requests_mock, reco_client: RecoCl authorization_status = "AUTH_STATUS_UNSANCTIONED" requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/app-risk-management/insert-risk-management-app", - json={"rows": 1}, + f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/{app_id}/auth-status", + json={"authorizationStatus": authorization_status}, status_code=200, ) @@ -881,41 +668,34 @@ def test_set_app_authorization_status_command(requests_mock, reco_client: RecoCl assert result.outputs["app_id"] == app_id assert result.outputs["authorization_status"] == authorization_status assert result.outputs["updated"] is True - assert result.outputs["rows_affected"] == 1 - assert "updated successfully" in result.readable_output + assert "updated to" in result.readable_output def test_get_apps_command(requests_mock, reco_client: RecoClient) -> None: """Test the get_apps_command function.""" - # Mock the app discovery response mock_apps = [ { - "cells": [ - {"key": "app_name", "value": base64.b64encode("Slack".encode(ENCODING)).decode(ENCODING)}, - {"key": "app_id", "value": base64.b64encode("slack.com".encode(ENCODING)).decode(ENCODING)}, - {"key": "category", "value": base64.b64encode("Communication".encode(ENCODING)).decode(ENCODING)}, - {"key": "risk_score", "value": base64.b64encode("3".encode(ENCODING)).decode(ENCODING)}, - {"key": "users_count", "value": base64.b64encode("10".encode(ENCODING)).decode(ENCODING)}, - ] + "id": "slack.com", + "name": "Slack", + "category": "Communication", + "usersCount": 10, + "authorization": "AUTH_STATUS_SANCTIONED", + "isUsingAi": False, + "vendorGrade": "A", + "aiCapability": True, + "lastSeen": "2024-01-01T00:00:00Z", } ] - - # Mock the client method to return the mock apps - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/asset-management/count", - json={"getTableResponse": {"totalNumberOfResults": 1}}, - status_code=200, - ) - requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/asset-management/query", - json={"getTableResponse": {"data": {"rows": mock_apps}}}, + requests_mock.get( + f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/list", + json={"apps": mock_apps, "totalResults": len(mock_apps)}, status_code=200, ) result = get_apps_command(reco_client) assert result.outputs_prefix == "Reco.Apps" - assert result.outputs_key_field == "app_id" + assert result.outputs_key_field == "id" assert isinstance(result.outputs, list) assert len(result.outputs) == 1 assert "App Discovery" in result.readable_output @@ -927,7 +707,7 @@ def test_set_app_authorization_status_error(capfd, requests_mock, reco_client: R authorization_status = "AUTH_STATUS_SANCTIONED" requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/app-risk-management/insert-risk-management-app", + f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/{app_id}/auth-status", json={}, status_code=500, ) @@ -937,14 +717,14 @@ def test_set_app_authorization_status_error(capfd, requests_mock, reco_client: R def test_set_app_authorization_status_error_2(capfd, requests_mock, reco_client: RecoClient) -> None: - """Test error handling in set_app_authorization_status.""" + """Test error handling in set_app_authorization_status with a different failure status code.""" app_id = "test.com" authorization_status = "AUTH_STATUS_SANCTIONED" requests_mock.put( - f"{DUMMY_RECO_API_DNS_NAME}/app-risk-management/insert-risk-management-app", + f"{DUMMY_RECO_API_DNS_NAME}/external-api/apps/{app_id}/auth-status", json={}, - status_code=200, + status_code=404, ) with capfd.disabled(), pytest.raises(Exception): diff --git a/Packs/Reco/README.md b/Packs/Reco/README.md index 4deeab81963..2cc0a07d4b1 100644 --- a/Packs/Reco/README.md +++ b/Packs/Reco/README.md @@ -9,7 +9,7 @@ The Reco and Palo Alto Networks Cortex XSOAR integration brings Reco's SaaS & AI - **Govern AI usage** — discover AI agents, AI-powered SaaS apps, and SaaS-to-SaaS OAuth grants with AI capabilities; prevent unauthorized data sharing; maintain audit-ready AI activity records - **Secure AI agents** — continuously monitor non-human identities operating in SaaS; enforce least-privilege policies; detect agent misuse and anomalous behavior - **Audit your SaaS posture** — query posture issues, posture checks, and threat detection policies across every connected app; score against SOC 2, ISO 27001, CIS, NIST, PCI DSS, HITRUST, and more -- **Detect and respond to SaaS threats** — fetch behavioral threat alerts with multi-severity filtering and AI-powered summaries; respond with existing SIEM & SOAR tooling +- **Detect and respond to SaaS threats** — fetch behavioral threat alerts with a minimum-severity filter and AI-powered summaries; respond with existing SIEM & SOAR tooling - **Investigate identities and accounts** — enrich user context, flag risky employees, and track account activity across SaaS apps - **Manage app risk** — inventory your app portfolio, update authorization status, and track shadow IT - **Protect sensitive data** — surface files shared publicly, externally, or with private emails; query sensitivity-labeled assets @@ -28,49 +28,57 @@ Five types of sprawl are widening the gap between what you can and cannot protec ## Key Capabilities ### AI Governance + - Gain full visibility into AI tool adoption — from ChatGPT to copilots to embedded AI features - Prevent unauthorized data sharing and monitor AI usage for policy compliance - List AI agents with authorization status, risk level, and vendor - Surface apps and SaaS-to-SaaS grants using AI capabilities ### AI Agent Security + - Discover and continuously monitor AI agents operating within your SaaS environment - Understand what data agents access, what actions they perform, and where over-permission creates risk - Enforce least-privilege policies for non-human identities at scale - Detect AI agent misuse through behavioral threat detection policies ### Posture Management + - Continuously assess security risk across applications, identities, and data - List posture issues with severity, check status, and compliance framework mappings (SOC 2, ISO 27001, CIS, NIST CSF, NIST 800-53, PCI DSS, HITRUST) - List posture check definitions and threat detection policies - Track configuration drift with one-click remediation guidance ### Threat Detection & Response (ITDR) -- Fetch incidents with flexible severity filters (single or multi-level: `HIGH,CRITICAL`) + +- Fetch incidents with a minimum-severity filter (e.g. `MEDIUM` fetches medium severity and higher) - Get full alert details including policy violation evidence - Add comments to alerts and update incident timelines - Change alert status and resolve visibility events - Get AI-generated alert summaries ### Identity & Account Intelligence + - List all SaaS accounts with risk signals (MFA status, admin flag, risky user label) - Look up user context by email address across all integrated apps - List identities with aggregated cross-app view - Tag risky users and departing employees ### SaaS Application Governance + - Discover all apps (sanctioned, shadow, AI-powered) with vendor risk grades - List app instances (portfolio) from actively integrated apps - List SaaS-to-SaaS OAuth grants with permission risk scores - Update app authorization status ### Data Security + - Find sensitive files by name, ID, or sensitivity level - Query files shared with third-party domains - Identify files shared publicly or with external emails - List NetApp files carrying active business-impact labels ### Platform Visibility + - List SaaS events with actor, application, and outcome context - List groups and IP addresses - List business units @@ -79,6 +87,7 @@ Five types of sprawl are widening the gap between what you can and cannot protec ## Commands **Alerts & Incidents** + - `reco-add-comment-to-alert` — Add a comment to a Reco alert - `reco-update-incident-timeline` — Add a comment to an incident timeline - `reco-change-alert-status` — Update alert status (NEW / IN_PROGRESS / CLOSED) @@ -86,22 +95,26 @@ Five types of sprawl are widening the gap between what you can and cannot protec - `reco-get-alert-ai-summary` — Get an AI-generated summary of an alert **Identities & Users** + - `reco-get-risky-users` — List all accounts flagged as risky - `reco-add-risky-user-label` — Tag a user as risky - `reco-add-leaving-org-user-label` — Tag a user as a departing employee - `reco-get-user-context-by-email-address` — Get identity context for an email address **SaaS Applications** + - `reco-get-apps` — List discovered apps with risk and AI signals - `reco-set-app-authorization-status` — Update an app's authorization status **Posture & Policies** + - `reco-list-posture-issues` — List posture issues with severity and check status - `reco-list-posture-checks` — List posture check definitions - `reco-list-threat-detection-policies` — List threat detection policies - `reco-list-exclusions` — List alert suppression exclusion rules **Data & Files** + - `reco-get-sensitive-assets-by-name` — Find sensitive assets by name - `reco-get-sensitive-assets-by-id` — Find sensitive assets by ID - `reco-get-assets-by-id` — Find any asset by ID @@ -114,6 +127,7 @@ Five types of sprawl are widening the gap between what you can and cannot protec - `reco-get-private-email-list-with-access` — List private emails with file access **SaaS Events & Activity** + - `reco-list-events` — List SaaS activity events - `reco-list-accounts` — List SaaS accounts with risk signals - `reco-list-groups` — List SaaS groups @@ -122,9 +136,11 @@ Five types of sprawl are widening the gap between what you can and cannot protec - `reco-list-audit-logs` — List Reco platform audit logs **AI Governance** + - `reco-list-ai-agents` — List detected AI agents **Platform** + - `reco-list-app-instances` — List integrated app instances (portfolio) - `reco-list-devices` — List managed and unmanaged devices - `reco-list-business-units` — List business units diff --git a/Packs/Reco/ReleaseNotes/1_8_0.md b/Packs/Reco/ReleaseNotes/1_8_0.md index 5699bcb5372..f364e53ba75 100644 --- a/Packs/Reco/ReleaseNotes/1_8_0.md +++ b/Packs/Reco/ReleaseNotes/1_8_0.md @@ -2,7 +2,7 @@ ##### Reco - Migrated alert fetching, identity, app, file, and label commands to the new Reco External API (`/api/v1/external-api/`) for cleaner JSON responses and improved reliability. -- Added support for multiple severity filters in `fetch-incidents` — the **Risk level** parameter now accepts a comma-separated list (e.g., `HIGH,CRITICAL`). Resolves PRO-303. +- Added a minimum-severity filter to `fetch-incidents` — the **Risk level** parameter now fetches alerts at or above the given severity (e.g., `MEDIUM` fetches medium, high, and critical alerts). Resolves PRO-303. - Added rate-limit handling (automatic 429 retry with exponential backoff) for all External API calls. - Added five new commands: **reco-list-events**, **reco-list-posture-issues**, **reco-list-accounts**, **reco-list-devices**, **reco-list-ai-agents**. All accept SCIM v2 filter expressions. - Updated context output field names for **reco-get-risky-users**, **reco-get-user-context-by-email-address**, **reco-get-sensitive-assets-by-id**, **reco-get-assets-by-id**, and **reco-get-apps** to match the External API response schema. From 29994361f45da62990e541299b7d0344b0511f76 Mon Sep 17 00:00:00 2001 From: Yaniv Blum Date: Fri, 17 Jul 2026 15:53:49 -0500 Subject: [PATCH 03/47] Reco: document breaking changes in 1.8.0 release notes - List every renamed/removed context output field per migrated command (RiskyUsers, User, SensitiveAssets, Apps, AppAuthorization) so upgrading customers know which playbooks/layouts will silently stop populating. - Call out the fetch-incidents Risk level semantic change explicitly: it now fetches the given severity and above, not an exact match. - Correct "five new commands" to the actual 14 added in this release. --- Packs/Reco/ReleaseNotes/1_8_0.md | 34 +++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/Packs/Reco/ReleaseNotes/1_8_0.md b/Packs/Reco/ReleaseNotes/1_8_0.md index f364e53ba75..c259f30319a 100644 --- a/Packs/Reco/ReleaseNotes/1_8_0.md +++ b/Packs/Reco/ReleaseNotes/1_8_0.md @@ -2,7 +2,35 @@ ##### Reco - Migrated alert fetching, identity, app, file, and label commands to the new Reco External API (`/api/v1/external-api/`) for cleaner JSON responses and improved reliability. -- Added a minimum-severity filter to `fetch-incidents` — the **Risk level** parameter now fetches alerts at or above the given severity (e.g., `MEDIUM` fetches medium, high, and critical alerts). Resolves PRO-303. - Added rate-limit handling (automatic 429 retry with exponential backoff) for all External API calls. -- Added five new commands: **reco-list-events**, **reco-list-posture-issues**, **reco-list-accounts**, **reco-list-devices**, **reco-list-ai-agents**. All accept SCIM v2 filter expressions. -- Updated context output field names for **reco-get-risky-users**, **reco-get-user-context-by-email-address**, **reco-get-sensitive-assets-by-id**, **reco-get-assets-by-id**, and **reco-get-apps** to match the External API response schema. +- Added 14 new commands backed by the External API: **reco-list-events**, **reco-list-posture-issues**, **reco-list-accounts**, **reco-list-devices**, **reco-list-ai-agents**, **reco-list-groups**, **reco-list-saas-to-saas**, **reco-list-ip-addresses**, **reco-list-business-units**, **reco-list-audit-logs**, **reco-list-posture-checks**, **reco-list-threat-detection-policies**, **reco-list-exclusions**, **reco-list-app-instances**. All accept SCIM v2 filter expressions. + +##### Breaking changes + +- **`fetch-incidents` — `Risk level` parameter now means "minimum severity," not exact match.** Previously, `risk_level=HIGH` fetched only HIGH-severity alerts. It now fetches HIGH **and** CRITICAL alerts (severity and above). If you rely on this parameter to fetch a single, exact severity tier, you will start receiving more incidents than before after upgrading. Resolves PRO-303. +- **Context output fields renamed or removed** on commands migrated to the External API. Any playbook, layout, or automation referencing the old field names below will silently return empty values after upgrading — they will not error. + + | Command | Old field | New field / status | + |---|---|---| + | `reco-get-risky-users` | `Reco.RiskyUsers.email_account` | Renamed to `Reco.RiskyUsers.accountEmail` | + | `reco-get-user-context-by-email-address` | `Reco.User.email_account` | Renamed to `Reco.User.email` | + | `reco-get-user-context-by-email-address` | `Reco.User.full_name` | Renamed to `Reco.User.name` | + | `reco-get-user-context-by-email-address` | `Reco.User.groups` | **Removed, no replacement** | + | `reco-get-user-context-by-email-address` | `Reco.User.labels` | **Removed, no replacement** | + | `reco-get-user-context-by-email-address` | `Reco.User.category` | **Removed, no replacement** | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.file_name` | Renamed to `Reco.SensitiveAssets.name` | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.file_owner` | Renamed to `Reco.SensitiveAssets.owner` | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.file_url` | Renamed to `Reco.SensitiveAssets.url` | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.sensitivity_level` | Renamed to `Reco.SensitiveAssets.sensitivityLevel` | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.visibility` | Renamed to `Reco.SensitiveAssets.permissionVisibility` | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.currently_permitted_users` | **Removed, no replacement** (list of users currently permitted access is no longer returned) | + | `reco-get-sensitive-assets-by-id`, `reco-get-sensitive-assets-by-name`, `reco-get-assets-by-id` | `Reco.SensitiveAssets.source` | **Removed, no replacement** | + | `reco-get-apps` | `Reco.Apps.app_id` | Renamed to `Reco.Apps.id` | + | `reco-get-apps` | `Reco.Apps.app_name` | Renamed to `Reco.Apps.name` | + | `reco-get-apps` | `Reco.Apps.data_access` | Renamed to `Reco.Apps.authorization` (note: this is now the app's authorization/sanction status, not its data access level — not a strict 1:1 semantic match) | + | `reco-get-apps` | `Reco.Apps.risk_score` | **Removed, no replacement** | + | `reco-get-apps` | `Reco.Apps.status` | **Removed, no replacement** | + | `reco-get-apps` | `Reco.Apps.created_at` | **Removed, no replacement** (only `lastSeen` remains) | + | `reco-set-app-authorization-status` | `Reco.AppAuthorization.rows_affected` | **No longer populated** — the External API's update response carries no row count, so this field is always empty going forward | + +- No commands or required arguments were removed in this release — only the field renames/removals above, plus the additive changes described above. From 1517c4563ad668b689e428ccfe9fcecc9e3e3eae Mon Sep 17 00:00:00 2001 From: Yael Shamai <111040837+YaelShamai@users.noreply.github.com> Date: Sun, 19 Jul 2026 08:28:29 +0300 Subject: [PATCH 04/47] Aruba Rate Limit Handling (#44988) * add logs and rate limit handling * rever not needed logs * rn * update the docker * pack metadata * rn * rn * rn * rn * rn * rn * add unit tests --- .../HPEArubaCentralEventCollector.py | 16 +- .../HPEArubaCentralEventCollector.yml | 2 +- .../HPEArubaCentralEventCollector_test.py | 152 +++++++++++++++++- Packs/HPEArubaCentral/ReleaseNotes/1_1_9.md | 5 + Packs/HPEArubaCentral/pack_metadata.json | 6 +- 5 files changed, 174 insertions(+), 7 deletions(-) create mode 100644 Packs/HPEArubaCentral/ReleaseNotes/1_1_9.md diff --git a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.py b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.py index c54ede1b4cd..db9bd1184ee 100644 --- a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.py +++ b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.py @@ -11,6 +11,9 @@ DATE_FORMAT = "%Y-%m-%dT%H:%M:%SZ" VENDOR = "aruba" PRODUCT = "central" +RATE_LIMIT_STATUS_CODE = 429 +NUM_OF_RETRIES = 3 +BACKOFF_FACTOR = 5 # Sleep for: {backoff_factor} * (2 ** ({number of retries} - 1)) seconds between retries. MAX_GET_AUDIT_LIMIT = 100 # Maximum limit accepted by get audit events API MAX_AUDIT_API_REQS = 10 MAX_GET_EVENTS_LIMIT = 1000 # Maximum limit accepted by get events API @@ -242,6 +245,9 @@ def http_request(self, method: str, url_suffix: str = "", params: dict = {}): url_suffix=url_suffix, params=params, headers=headers, + retries=NUM_OF_RETRIES, + status_list_to_retry=[RATE_LIMIT_STATUS_CODE], + backoff_factor=BACKOFF_FACTOR, ) except DemistoException as e: if "access token is invalid" in str(e): @@ -252,6 +258,9 @@ def http_request(self, method: str, url_suffix: str = "", params: dict = {}): url_suffix=url_suffix, params=params, headers=headers, + retries=NUM_OF_RETRIES, + status_list_to_retry=[RATE_LIMIT_STATUS_CODE], + backoff_factor=BACKOFF_FACTOR, ) else: raise e @@ -305,6 +314,7 @@ def fetch_audit_events(self, start_time: int, end_time: int, amount_to_fetch: in if not response.get("remaining_records"): break + demisto.debug(f"[Fetch] Audit events fetched {len(events)} event(s).") return events def fetch_networking_events(self, start_time: int, end_time: int, amount_to_fetch: int, last_run: dict) -> list[dict]: @@ -326,7 +336,7 @@ def fetch_networking_events(self, start_time: int, end_time: int, amount_to_fetc events = [] offset = 0 - demisto.debug(f"{amount_to_fetch=}") + demisto.debug(f"[Fetch] Networking events: starting fetch with {amount_to_fetch=}") while amount_to_fetch > 0: response = self.http_request( method="GET", @@ -649,11 +659,11 @@ def fetch_events( audit_start_time = int(last_run.get("last_audit_ts", first_fetch_time)) networking_start_time = int(last_run.get("last_networking_ts", first_fetch_time)) end_time = int(time.time()) - demisto.debug(f"Fetching {num_audit_events_to_fetch} audit events from {audit_start_time} to {end_time}.") + demisto.debug(f"[Fetch] Fetching {num_audit_events_to_fetch} audit events from {audit_start_time} to {end_time}.") audit_events = client.fetch_audit_events( start_time=audit_start_time, end_time=end_time, amount_to_fetch=num_audit_events_to_fetch, last_run=last_run ) - demisto.debug(f"Got {len(audit_events)} audit events.") + demisto.debug(f"[Fetch] Got {len(audit_events)} audit events.") networking_events = None if fetch_networking_events: diff --git a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.yml b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.yml index 610d4a54d04..901f688476c 100644 --- a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.yml +++ b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector.yml @@ -100,7 +100,7 @@ script: name: aruba-auth-test execution: false deprecated: false - dockerimage: demisto/python3:3.12.12.5490952 + dockerimage: demisto/python3:3.12.13.10404775 isfetchevents: true runonce: false script: '-' diff --git a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector_test.py b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector_test.py index c4f787ab394..b1aa4995b3e 100644 --- a/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector_test.py +++ b/Packs/HPEArubaCentral/Integrations/HPEArubaCentralEventCollector/HPEArubaCentralEventCollector_test.py @@ -1,5 +1,5 @@ from CommonServerPython import DemistoException -from HPEArubaCentralEventCollector import main, Client +from HPEArubaCentralEventCollector import main, Client, NUM_OF_RETRIES, RATE_LIMIT_STATUS_CODE, BACKOFF_FACTOR import pytest import demistomock as demisto from CommonServerPython import date_to_timestamp @@ -457,3 +457,153 @@ def test_get_events_command(mocker, requests_mock, fetch_networking, should_push send_events_to_xsiam_mock.assert_called_once_with(expected_events, vendor=VENDOR, product=PRODUCT) else: send_events_to_xsiam_mock.assert_not_called() + + +def _make_client() -> Client: + """Helper to create a Client instance for http_request tests.""" + return Client( + base_url=BASE_URL, + client_id=CLIENT_ID, + client_secret=CLIENT_SECRET, + user_name=USER_NAME, + user_password=USER_PASSWORD, + customer_id=CUSTOMER_ID, + verify=False, + proxy=False, + ) + + +@freeze_time(FETCH_DATE) +def test_http_request_retry_on_429_then_success(mocker): + """ + Given: + - _http_request succeeds (simulating that the internal urllib3 retry handled a transient 429). + + When: + - Calling client.http_request (which passes retry params to _http_request). + + Then: + - The call succeeds and returns the expected data. + - The retry-related params (retries, status_list_to_retry, backoff_factor) are passed + to _http_request, proving the retry configuration is wired correctly. + """ + client = _make_client() + mocker.patch.object(client, "get_access_token", return_value=TEST_TOKEN) + + expected_response = {"events": [{"id": 1}]} + + mock_http = mocker.patch.object( + client, + "_http_request", + return_value=expected_response, + ) + + result = client.http_request("GET", url_suffix="/test/endpoint", params={"limit": 10}) + + assert result == expected_response + mock_http.assert_called_once_with( + method="GET", + url_suffix="/test/endpoint", + params={"limit": 10}, + headers={ + "accept": "application/json", + "authorization": f"Bearer {TEST_TOKEN}", + }, + retries=NUM_OF_RETRIES, + status_list_to_retry=[RATE_LIMIT_STATUS_CODE], + backoff_factor=BACKOFF_FACTOR, + ) + + +@freeze_time(FETCH_DATE) +def test_http_request_repeated_429_raises_demisto_exception(mocker): + """ + Given: + - _http_request always raises a DemistoException wrapping a 429 (rate-limit) response, + exceeding NUM_OF_RETRIES attempts. + + When: + - Calling client.http_request. + + Then: + - A DemistoException is raised and is NOT absorbed by the "access token is invalid" branch. + """ + client = _make_client() + mocker.patch.object(client, "get_access_token", return_value=TEST_TOKEN) + + mocker.patch.object( + client, + "_http_request", + side_effect=DemistoException("Rate limit exceeded", res=mocker.MagicMock(status_code=429)), + ) + + with pytest.raises(DemistoException, match="Rate limit exceeded"): + client.http_request("GET", url_suffix="/test/endpoint") + + +@freeze_time(FETCH_DATE) +def test_http_request_passes_retry_params_on_initial_and_refresh(mocker): + """ + Given: + - The first _http_request call raises a DemistoException containing "access token is invalid" + (triggering the token-refresh branch). + - The second _http_request call (after token refresh) succeeds. + + When: + - Calling client.http_request. + + Then: + - Both the initial and the post-refresh _http_request calls include + retries=NUM_OF_RETRIES, status_list_to_retry=[RATE_LIMIT_STATUS_CODE], + and backoff_factor=BACKOFF_FACTOR. + """ + import copy + + client = _make_client() + refreshed_token = "refreshed_token" + mocker.patch.object(client, "get_access_token", side_effect=[TEST_TOKEN, refreshed_token]) + + expected_response = {"events": [{"id": 2}]} + + # Capture deep copies of kwargs at call time, because http_request mutates the + # headers dict in-place when refreshing the token, which would make both + # call_args_list entries point to the same (mutated) dict object. + captured_kwargs: list[dict] = [] + original_side_effects = iter( + [ + DemistoException("access token is invalid"), + expected_response, + ] + ) + + def capture_and_delegate(**kwargs): + captured_kwargs.append(copy.deepcopy(kwargs)) + result = next(original_side_effects) + if isinstance(result, Exception): + raise result + return result + + mocker.patch.object(client, "_http_request", side_effect=capture_and_delegate) + + result = client.http_request("GET", url_suffix="/test/endpoint", params={"key": "val"}) + + assert result == expected_response + assert len(captured_kwargs) == 2 + + expected_retry_kwargs = { + "retries": NUM_OF_RETRIES, + "status_list_to_retry": [RATE_LIMIT_STATUS_CODE], + "backoff_factor": BACKOFF_FACTOR, + } + + # Verify initial request (with original token) includes retry params + first_kwargs = captured_kwargs[0] + assert first_kwargs["headers"]["authorization"] == f"Bearer {TEST_TOKEN}" + for key, value in expected_retry_kwargs.items(): + assert first_kwargs[key] == value, f"Initial request missing or wrong retry param '{key}': {first_kwargs.get(key)}" + + # Verify post-refresh retry request (with refreshed token) includes retry params + second_kwargs = captured_kwargs[1] + assert second_kwargs["headers"]["authorization"] == f"Bearer {refreshed_token}" + for key, value in expected_retry_kwargs.items(): + assert second_kwargs[key] == value, f"Retry request missing or wrong retry param '{key}': {second_kwargs.get(key)}" diff --git a/Packs/HPEArubaCentral/ReleaseNotes/1_1_9.md b/Packs/HPEArubaCentral/ReleaseNotes/1_1_9.md new file mode 100644 index 00000000000..39c859dbb09 --- /dev/null +++ b/Packs/HPEArubaCentral/ReleaseNotes/1_1_9.md @@ -0,0 +1,5 @@ + +#### Integrations +##### HPE Aruba Central Event Collector +- Updated the Docker image to: *demisto/python3:3.12.13.10404775*. +- Fixed an issue where the collector would get a rate limit error. diff --git a/Packs/HPEArubaCentral/pack_metadata.json b/Packs/HPEArubaCentral/pack_metadata.json index 8dbbb95cfff..b3568012d8b 100644 --- a/Packs/HPEArubaCentral/pack_metadata.json +++ b/Packs/HPEArubaCentral/pack_metadata.json @@ -2,8 +2,9 @@ "name": "HPE Aruba Central", "description": "Aruba Central helps manage and monitor your network infrastructure from a centralized platform.", "support": "xsoar", - "currentVersion": "1.1.8", + "currentVersion": "1.1.9", "author": "Cortex XSOAR", + "created": "", "url": "https://www.paloaltonetworks.com/cortex", "email": "", "categories": [ @@ -24,7 +25,8 @@ "AI", "IoT", "wireless", - "SD-WAN" + "SD-WAN", + "HPE" ], "marketplaces": [ "marketplacev2", From 129c4b9bc3d0fea349ef3fbcc249b146cb32a365 Mon Sep 17 00:00:00 2001 From: Content Bot <55035720+content-bot@users.noreply.github.com> Date: Sun, 19 Jul 2026 10:06:06 +0300 Subject: [PATCH 05/47] Auto RN: migrate-non-core-scripts (#45066) This PR transitions the non-Core aggregated scripts to use the new PCI built-in commands on the unified Cortex platform, aligned with the corresponding command migration already done in the content repo. On the platform, these scripts now call the server-injected built-in (PCI) commands; on legacy XSIAM they continue to use the Cortex Core - IR integration commands. --- .../AggregatedScripts/ReleaseNotes/1_3_51.md | 33 +++++++++++ .../DomainEnrichment/DomainEnrichment.py | 29 +++++++--- .../DomainEnrichment/DomainEnrichment_test.py | 55 ++++++++++++++++++ .../Scripts/FileEnrichment/FileEnrichment.py | 29 +++++++--- .../FileEnrichment/FileEnrichment_test.py | 26 ++++++++- .../GetEndpointData/GetEndpointData.py | 22 +++++--- .../Scripts/GetUserData/GetUserData.py | 13 +++-- .../Scripts/IPEnrichment/IPEnrichment.py | 17 +++++- .../Scripts/IPEnrichment/IPEnrichment.yml | 4 +- .../Scripts/IPEnrichment/IPEnrichment_test.py | 56 +++++++++++++++++++ .../IsolateEndpoint/IsolateEndpoint.py | 7 ++- .../IsolateEndpoint/IsolateEndpoint_test.py | 8 ++- .../Scripts/QuarantineFile/QuarantineFile.py | 4 ++ Packs/AggregatedScripts/pack_metadata.json | 2 +- 14 files changed, 267 insertions(+), 38 deletions(-) create mode 100644 Packs/AggregatedScripts/ReleaseNotes/1_3_51.md diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_3_51.md b/Packs/AggregatedScripts/ReleaseNotes/1_3_51.md new file mode 100644 index 00000000000..b061a8e4658 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_3_51.md @@ -0,0 +1,33 @@ +<~PLATFORM> +#### Scripts + +##### isolate-endpoint + +- Documentation and metadata improvements. + +##### ip-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.13.10404775*. +- Documentation and metadata improvements. + +##### domain-enrichment + +- Documentation and metadata improvements. + +##### get-endpoint-data + +- Documentation and metadata improvements. + +##### file-enrichment + +- Documentation and metadata improvements. + +##### quarantine-file + +- Documentation and metadata improvements. + +##### get-user-data + +- Documentation and metadata improvements. + + diff --git a/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment.py b/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment.py index 14d39e586f6..941540efe63 100644 --- a/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment.py +++ b/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment.py @@ -60,15 +60,28 @@ def domain_enrichment_script( command_batch2: list[Command] = [] if is_xsiam(): demisto.debug("Command Batch 2: Internal commands (for XSIAM)") - command_batch2.append( - Command( - name="core-get-domain-analytics-prevalence", - args={"domain_name": valid_inputs}, - command_type=CommandType.INTERNAL, - brand="Cortex Core - IR", # keep the brand you use elsewhere - context_output_mapping={"Core.AnalyticsPrevalence.Domain": "Core.AnalyticsPrevalence.Domain"}, + # On platform, we use the built-in commands and brand Builtin + if is_platform(): + command_batch2.append( + Command( + name="getDomainAnalyticsPrevalence", + args={"domain_name": valid_inputs}, + command_type=CommandType.BUILTIN, + brand="Builtin", + context_output_mapping={"Core.AnalyticsPrevalence.Domain": "Core.AnalyticsPrevalence.Domain"}, + ignore_using_brand=True, + ) + ) + else: + command_batch2.append( + Command( + name="core-get-domain-analytics-prevalence", + args={"domain_name": valid_inputs}, + command_type=CommandType.INTERNAL, + brand="Cortex Core - IR", # keep the brand you use elsewhere + context_output_mapping={"Core.AnalyticsPrevalence.Domain": "Core.AnalyticsPrevalence.Domain"}, + ) ) - ) demisto.debug("Command Batch 2: Enriching indicators") command_batch2.append( Command( diff --git a/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment_test.py b/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment_test.py index 79c8580a188..08ddbf3301d 100644 --- a/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment_test.py +++ b/Packs/AggregatedScripts/Scripts/DomainEnrichment/DomainEnrichment_test.py @@ -1,5 +1,7 @@ import json + import demistomock as demisto +import pytest from DomainEnrichment import domain_enrichment_script @@ -38,6 +40,7 @@ def test_domain_enrichment_script_end_to_end_with_batch_file(mocker): mocker.patch.object(demisto, "args", return_value={"domain_list": ",".join(domain_list)}) mocker.patch("DomainEnrichment.is_xsiam", return_value=True) + mocker.patch("DomainEnrichment.is_platform", return_value=False) # extractIndicators -> validates input mocker.patch( "AggregatedCommandApiModule.execute_command", @@ -163,3 +166,55 @@ def _fake_execute_list_of_batches(self, list_of_batches, brands_to_run=None, ver assert isinstance(core_ctx, list) assert len(core_ctx) == 2 assert {d["Domain"] for d in core_ctx} == {"example.com", "example2.com"} + + +def test_domain_enrichment_uses_builtin_command_on_platform(mocker): + """ + Given: + - Running on the unified Cortex platform (is_xsiam and is_platform both True). + When: + - domain_enrichment_script builds its command batches. + Then: + - The prevalence command is the built-in "getDomainAnalyticsPrevalence". + - Its command_type is CommandType.BUILTIN (not the legacy INTERNAL core command). + """ + from AggregatedCommandApiModule import CommandType + + domain_list = ["example.com"] + captured_batches: dict = {} + + class _StopAfterCapture(Exception): + pass + + def _capture_batches(self, list_of_batches, brands_to_run=None, verbose=False): + captured_batches["batches"] = list_of_batches + raise _StopAfterCapture + + mocker.patch.object(demisto, "args", return_value={"domain_list": ",".join(domain_list)}) + mocker.patch("DomainEnrichment.is_xsiam", return_value=True) + mocker.patch("DomainEnrichment.is_platform", return_value=True) + mocker.patch( + "AggregatedCommandApiModule.execute_command", + return_value=[{"EntryContext": {"ExtractedIndicators": {"Domain": domain_list}}}], + ) + mocker.patch("AggregatedCommandApiModule.IndicatorsSearcher", return_value=iter([])) + mocker.patch.object( + demisto, + "getModules", + return_value={"coreir": {"state": "active", "brand": "Cortex Core - IR"}}, + ) + mocker.patch("AggregatedCommandApiModule.BatchExecutor.execute_list_of_batches", _capture_batches) + + with pytest.raises(_StopAfterCapture): + domain_enrichment_script( + domain_list=domain_list, + external_enrichment=True, + verbose=True, + enrichment_brands=["Cortex Core - IR"], + additional_fields=False, + ) + + b2_cmds = captured_batches["batches"][1] + prevalence_cmds = [c for c in b2_cmds if c.name == "getDomainAnalyticsPrevalence"] + assert len(prevalence_cmds) == 1 + assert prevalence_cmds[0].command_type == CommandType.BUILTIN diff --git a/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment.py b/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment.py index 68f2c3d2f7b..1e09bec5403 100644 --- a/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment.py +++ b/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment.py @@ -91,15 +91,28 @@ def file_enrichment_script( # Add the prevalence command only when at least one SHA256 hash is present. sha256_inputs = [file for file in valid_inputs if get_hash_type(file) == "sha256"] if sha256_inputs: - command_batch2.append( - Command( - name="core-get-hash-analytics-prevalence", - args={"sha256": sha256_inputs}, - brand="Cortex Core - IR", - command_type=CommandType.INTERNAL, - context_output_mapping={}, + # On platform, we use the built-in commands and brand Builtin + if is_platform(): + command_batch2.append( + Command( + name="getHashAnalyticsPrevalence", + args={"sha256": sha256_inputs}, + brand="Builtin", + command_type=CommandType.BUILTIN, + context_output_mapping={}, + ignore_using_brand=True, + ) + ) + else: + command_batch2.append( + Command( + name="core-get-hash-analytics-prevalence", + args={"sha256": sha256_inputs}, + brand="Cortex Core - IR", + command_type=CommandType.INTERNAL, + context_output_mapping={}, + ) ) - ) commands = [command_batch1, command_batch2] demisto.debug("Commands Batches") diff --git a/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment_test.py b/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment_test.py index e6b85d85972..eef6a0435db 100644 --- a/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment_test.py +++ b/Packs/AggregatedScripts/Scripts/FileEnrichment/FileEnrichment_test.py @@ -1,4 +1,5 @@ import json + import demistomock as demisto from FileEnrichment import file_enrichment_script @@ -64,6 +65,7 @@ def __iter__(self): "core": {"state": "active", "brand": "Cortex Core - IR"}, }, ) + mocker.patch("FileEnrichment.is_platform", return_value=False) # ---------- Mock BatchExecutor.execute_list_of_batches using JSON ---------- def _fake_execute_list_of_batches(self, list_of_batches, brands_to_run=None, verbose=False): @@ -158,10 +160,11 @@ def _fake_execute_list_of_batches(self, list_of_batches, brands_to_run=None, ver assert wf2.get("Reliability") == "Low" -def _capture_built_commands(mocker, file_list): +def _capture_built_commands(mocker, file_list, is_platform=False): """Helper: runs file_enrichment_script with mocks and returns the command batches exactly as they were built by file_enrichment_script, captured before any brand/type filtering is applied.""" mocker.patch.object(demisto, "args", return_value={"file_hash": ",".join(file_list)}) + mocker.patch("FileEnrichment.is_platform", return_value=is_platform) def extractIndicators_side_effect(cmd, args=None, extract_contents=False, fail_on_error=True): if cmd == "extractIndicators": @@ -260,3 +263,24 @@ def test_prevalence_command_included_when_sha256_present(mocker): prevalence_cmds = [cmd for batch in batches for cmd in batch if cmd.name == "core-get-hash-analytics-prevalence"] assert len(prevalence_cmds) == 1 assert prevalence_cmds[0].args == {"sha256": [sha256_hash]} + + +def test_file_enrichment_uses_builtin_command_on_platform(mocker): + """ + Given: + - A SHA256 hash as input, running on the unified Cortex platform (is_platform True). + When: + - file_enrichment_script builds the command batches. + Then: + - The prevalence command is the built-in "getHashAnalyticsPrevalence". + - Its command_type is CommandType.BUILTIN (not the legacy INTERNAL core command). + """ + from AggregatedCommandApiModule import CommandType + + sha256_hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + batches = _capture_built_commands(mocker, [sha256_hash], is_platform=True) + + prevalence_cmds = [cmd for batch in batches for cmd in batch if cmd.name == "getHashAnalyticsPrevalence"] + assert len(prevalence_cmds) == 1 + assert prevalence_cmds[0].command_type == CommandType.BUILTIN + assert "core-get-hash-analytics-prevalence" not in [cmd.name for batch in batches for cmd in batch] diff --git a/Packs/AggregatedScripts/Scripts/GetEndpointData/GetEndpointData.py b/Packs/AggregatedScripts/Scripts/GetEndpointData/GetEndpointData.py index 01874b8c046..036e97786c1 100644 --- a/Packs/AggregatedScripts/Scripts/GetEndpointData/GetEndpointData.py +++ b/Packs/AggregatedScripts/Scripts/GetEndpointData/GetEndpointData.py @@ -25,6 +25,7 @@ class Brands(StrEnum): CORTEX_CORE_IR = "Cortex Core - IR" FIREEYE_HX_V2 = "FireEyeHX v2" GENERIC_COMMAND = "Generic Command" + BUILTIN = "Builtin" @classmethod def get_all_values(cls) -> list[str]: @@ -134,6 +135,10 @@ def is_brand_available(self, command: Command) -> bool: bool: True if the brand is in both the list of brands to run and the set of enabled brands; False otherwise. """ + # Builtin commands are injected by the server on the unified platform, + # so they are always available there and are not tied to an installed integration brand. + if command.brand == Brands.BUILTIN: + return is_platform() return False if not self.is_brand_in_brands_to_run(command) else command.brand in self._enabled_brands def get_enabled_brands(self): @@ -346,7 +351,10 @@ def get_command_results( continue if entry_type == EntryType.ERROR or entry_type == EntryType.WARNING: - command_error_outputs.append(hr_to_command_results(command, args, contents, entry_type=entry_type)) # type: ignore[arg-type] + # Built-in commands may return no entries (None) on a not-found result + # guard against 'NoneType' object is not iterable + if error_result := hr_to_command_results(command, args, contents, entry_type=entry_type): # type: ignore[arg-type] + command_error_outputs.append(error_result) elif entry_type == EntryType.NOTE: command_context_outputs.append(entry.get("EntryContext", {})) human_readable_outputs.append(entry.get("HumanReadable") or "") @@ -509,8 +517,8 @@ def initialize_commands( not_found_checker="was not found", ), Command( - brand=Brands.CORTEX_CORE_IR, - name="core-list-risky-hosts", + brand=Brands.BUILTIN if is_platform() else Brands.CORTEX_CORE_IR, + name="getRiskyHosts" if is_platform() else "core-list-risky-hosts", output_keys=["Core.RiskyHost"], args_mapping={"host_id": "endpoint_hostname"}, output_mapping={"id": "Hostname", "risk_level": "RiskLevel"}, @@ -532,8 +540,8 @@ def initialize_commands( list_args_commands = [ Command( - brand=Brands.CORTEX_CORE_IR, - name="core-get-endpoints", + brand=Brands.BUILTIN if is_platform() else Brands.CORTEX_CORE_IR, + name="getEndpoints" if is_platform() else "core-get-endpoints", output_keys=["Core.Endpoint"], args_mapping={"endpoint_id_list": "endpoint_id", "ip_list": "endpoint_ip", "hostname": "endpoint_hostname"}, output_mapping={ @@ -614,7 +622,7 @@ def run_single_args_commands( ) if endpoint_output: - if command.brand in [Brands.CORTEX_XDR_IR, Brands.CORTEX_CORE_IR]: + if command.brand in [Brands.CORTEX_XDR_IR, Brands.CORTEX_CORE_IR, Brands.BUILTIN]: update_endpoint_in_mapping(endpoint_output, ir_mapping) else: endpoint_outputs_list.extend(endpoint_output) @@ -667,7 +675,7 @@ def run_list_args_commands( ) if endpoint_output: - if command.brand in [Brands.CORTEX_XDR_IR, Brands.CORTEX_CORE_IR]: + if command.brand in [Brands.CORTEX_XDR_IR, Brands.CORTEX_CORE_IR, Brands.BUILTIN]: add_endpoint_to_mapping(endpoint_output, ir_mapping) else: multiple_endpoint_outputs.extend(endpoint_output) diff --git a/Packs/AggregatedScripts/Scripts/GetUserData/GetUserData.py b/Packs/AggregatedScripts/Scripts/GetUserData/GetUserData.py index d09b2ef27ec..0c1d4312037 100644 --- a/Packs/AggregatedScripts/Scripts/GetUserData/GetUserData.py +++ b/Packs/AggregatedScripts/Scripts/GetUserData/GetUserData.py @@ -69,6 +69,10 @@ def is_brand_available(self, command: Command) -> bool: Returns: bool: True if the brand is available and in the list of brands to run, False otherwise. """ + # Builtin commands are injected by the server on the unified platform, + # so they are always available there and are not tied to an installed integration brand. + if command.brand == "Builtin": + return is_platform() is_available = command.brand in self._enabled_brands if not is_available: demisto.debug(f"Skipping command '{command.name}' since the brand '{command.brand}' is not available.") @@ -292,7 +296,8 @@ def run_execute_command(command_name: str, args: dict[str, Any]) -> tuple[list[d errors_command_results = [] human_readable_list = [] entry_context_list = [] - for entry in res: + # Built-in commands may return no entries (None) on a not-found result, guard against 'NoneType' object is not iterable + for entry in res or []: entry_context_list.append((entry.get("EntryContext") or {}) | {"instance": entry.get("ModuleName")}) if is_error(entry): errors_command_results.extend(prepare_human_readable(command_name, args, get_error(entry), is_error=True)) @@ -1294,9 +1299,9 @@ def main(): ################################# readable_output, outputs = get_core_and_xdr_data( # type: ignore[assignment] modules=modules, - brand_name="Cortex Core - IR", - first_command="core-list-risky-users", - second_command="core-list-users", + brand_name="Builtin" if is_platform() else "Cortex Core - IR", + first_command="getRiskyUsers" if is_platform() else "core-list-risky-users", + second_command="getSystemUsers" if is_platform() else "core-list-users", user_names=users_names, additional_fields=additional_fields, list_non_risky_users=list_non_risky_users, diff --git a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.py b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.py index ac82d9a6aab..8cf893ff270 100644 --- a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.py +++ b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.py @@ -85,15 +85,26 @@ def ip_enrichment_script( if is_xsiam(): demisto.debug("Command Batch 2: Internal commands (for XSIAM)") - command_batch2.append( - Command( + # On platform, we use the built-in commands and brand Builtin, which is always enabled + # and does not require enabling the integration. + if is_platform(): + prevalence_command = Command( + name="getIPAnalyticsPrevalence", + args={"ip_address": valid_inputs}, + command_type=CommandType.BUILTIN, + brand="Builtin", + context_output_mapping={"Core.AnalyticsPrevalence.Ip": "Core.AnalyticsPrevalence.Ip"}, + ignore_using_brand=True, + ) + else: + prevalence_command = Command( name="core-get-IP-analytics-prevalence", args={"ip_address": valid_inputs}, command_type=CommandType.INTERNAL, brand="Cortex Core - IR", context_output_mapping={"Core.AnalyticsPrevalence.Ip": "Core.AnalyticsPrevalence.Ip"}, ) - ) + command_batch2.append(prevalence_command) demisto.debug("Command Batch 2: Enriching indicators") command_batch2.append( diff --git a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.yml b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.yml index 916d50bd432..92d805d42ec 100644 --- a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.yml +++ b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment.yml @@ -373,11 +373,11 @@ tags: timeout: '0' type: python subtype: python3 -dockerimage: demisto/python3:3.12.13.7444307 +dockerimage: demisto/python3:3.12.13.10404775 fromversion: 8.0.0 marketplaces: - xsoar_saas - marketplacev2 - platform tests: -- IPEnrichment-Test \ No newline at end of file +- No tests (auto formatted) diff --git a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment_test.py b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment_test.py index a89d644c084..8bb3eeca775 100644 --- a/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment_test.py +++ b/Packs/AggregatedScripts/Scripts/IPEnrichment/IPEnrichment_test.py @@ -1,5 +1,7 @@ import json + import demistomock as demisto +import pytest from IPEnrichment import ip_enrichment_script @@ -39,6 +41,7 @@ def test_ip_enrichment_script_end_to_end_with_batch_file(mocker): # is_xsiam mocker.patch("IPEnrichment.is_xsiam", return_value=True) + mocker.patch("IPEnrichment.is_platform", return_value=False) # extractIndicators -> validates input mocker.patch( "AggregatedCommandApiModule.execute_command", @@ -178,6 +181,7 @@ def test_ip_enrichment_script_with_internal_ip(mocker): ip_list = ["192.168.1.1"] mocker.patch.object(demisto, "args", return_value={"ip_list": ",".join(ip_list)}) mocker.patch("IPEnrichment.is_xsiam", return_value=True) + mocker.patch("IPEnrichment.is_platform", return_value=False) mocker.patch( "AggregatedCommandApiModule.execute_command", return_value=[{"EntryContext": {"ExtractedIndicators": {"IP": ip_list}}}], @@ -248,3 +252,55 @@ def _fake_execute_list_of_batches(self, list_of_batches, brands_to_run=None, ver assert len(endpoint_ctx) == 2 assert {e["Brand"] for e in endpoint_ctx} == {"Core"} assert {e["Hostname"] for e in endpoint_ctx} == {"host-1", "host-2"} + + +def test_ip_enrichment_uses_builtin_command_on_platform(mocker): + """ + Given: + - Running on the unified Cortex platform (is_xsiam and is_platform both True). + When: + - ip_enrichment_script builds its command batches. + Then: + - The prevalence command is the built-in "getIPAnalyticsPrevalence". + - Its command_type is CommandType.BUILTIN (not the legacy INTERNAL core command). + """ + from AggregatedCommandApiModule import CommandType + + ip_list = ["1.1.1.1"] + captured_batches: dict = {} + + class _StopAfterCapture(Exception): + pass + + def _capture_batches(self, list_of_batches, brands_to_run=None, verbose=False): + captured_batches["batches"] = list_of_batches + raise _StopAfterCapture + + mocker.patch.object(demisto, "args", return_value={"ip_list": ",".join(ip_list)}) + mocker.patch("IPEnrichment.is_xsiam", return_value=True) + mocker.patch("IPEnrichment.is_platform", return_value=True) + mocker.patch( + "AggregatedCommandApiModule.execute_command", + return_value=[{"EntryContext": {"ExtractedIndicators": {"IP": ip_list}}}], + ) + mocker.patch("AggregatedCommandApiModule.IndicatorsSearcher", return_value=iter([])) + mocker.patch.object( + demisto, + "getModules", + return_value={"coreir": {"state": "active", "brand": "Cortex Core - IR"}}, + ) + mocker.patch("AggregatedCommandApiModule.BatchExecutor.execute_list_of_batches", _capture_batches) + + with pytest.raises(_StopAfterCapture): + ip_enrichment_script( + ip_list=ip_list, + external_enrichment=True, + verbose=True, + enrichment_brands=["Cortex Core - IR"], + additional_fields=False, + ) + + b2_cmds = captured_batches["batches"][1] + prevalence_cmds = [c for c in b2_cmds if c.name == "getIPAnalyticsPrevalence"] + assert len(prevalence_cmds) == 1 + assert prevalence_cmds[0].command_type == CommandType.BUILTIN diff --git a/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint.py b/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint.py index 0f0cda141d9..02869375b3d 100644 --- a/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint.py +++ b/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint.py @@ -14,6 +14,7 @@ class Brands(StrEnum): FIREEYE_HX_V2 = "FireEyeHX v2" CROWDSTRIKE_FALCON = "CrowdstrikeFalcon" CORTEX_CORE_IR = "Cortex Core - IR" + BUILTIN = "Builtin" MICROSOFT_DEFENDER_ADVANCED_THREAT_PROTECTION = "Microsoft Defender Advanced Threat Protection" @classmethod @@ -55,9 +56,9 @@ def initialize_commands() -> list: """ commands = [ Command( - # Can be used only in XSIAM - brand=Brands.CORTEX_CORE_IR, - name="core-isolate-endpoint", + # On platform, we use the built-in commands and brand Builtin + brand=Brands.BUILTIN if is_platform() else Brands.CORTEX_CORE_IR, + name="isolateEndpoint" if is_platform() else "core-isolate-endpoint", arg_mapping={"endpoint_id": "endpoint_id"}, ), Command( diff --git a/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint_test.py b/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint_test.py index 78a7f113c48..48bff3a6004 100644 --- a/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint_test.py +++ b/Packs/AggregatedScripts/Scripts/IsolateEndpoint/IsolateEndpoint_test.py @@ -7,7 +7,13 @@ def test_get_all_values_returns_expected_list(): """ Ensure get_all_values returns all brand values in the correct order. """ - expected_list = ["FireEyeHX v2", "CrowdstrikeFalcon", "Cortex Core - IR", "Microsoft Defender Advanced Threat Protection"] + expected_list = [ + "FireEyeHX v2", + "CrowdstrikeFalcon", + "Cortex Core - IR", + "Builtin", + "Microsoft Defender Advanced Threat Protection", + ] assert Brands.get_all_values() == expected_list diff --git a/Packs/AggregatedScripts/Scripts/QuarantineFile/QuarantineFile.py b/Packs/AggregatedScripts/Scripts/QuarantineFile/QuarantineFile.py index ba3b554b509..bab5d258c23 100644 --- a/Packs/AggregatedScripts/Scripts/QuarantineFile/QuarantineFile.py +++ b/Packs/AggregatedScripts/Scripts/QuarantineFile/QuarantineFile.py @@ -26,6 +26,10 @@ def values(cls): def normalize(cls, value: str): _ALIASES = { "Microsoft Defender ATP": "Microsoft Defender Advanced Threat Protection", + # On the unified platform, get-endpoint-data reports Core endpoints under the + # built-in brand "Builtin". Map it to "Cortex Core - IR" so the correct handler + # is selected and the legacy Core quarantine commands are used. + "Builtin": cls.CORTEX_CORE_IR.value, } """Normalize a brand string (alias → canonical enum).""" canonical = _ALIASES.get(value, value) diff --git a/Packs/AggregatedScripts/pack_metadata.json b/Packs/AggregatedScripts/pack_metadata.json index 8f1ad4e7fc7..9db6cc74c6b 100644 --- a/Packs/AggregatedScripts/pack_metadata.json +++ b/Packs/AggregatedScripts/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Aggregated Scripts", "description": "A pack containing all aggregated scripts.", "support": "xsoar", - "currentVersion": "1.3.50", + "currentVersion": "1.3.51", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From de90b85d0dc70ffe2df6e11dddbb6b9c798d47c9 Mon Sep 17 00:00:00 2001 From: sharonfi99 <147984773+sharonfi99@users.noreply.github.com> Date: Sun, 19 Jul 2026 13:30:49 +0300 Subject: [PATCH 06/47] XSUP-72930 - improve BeyondTrust Password Safe target account extraction (#45082) * XSUP-72930 - improve BeyondTrust Password Safe target account extraction * added more support in regex * refined release notes --- .../BeyondTrust_Password_Safe/BeyondTrust_Password_Safe.xif | 6 ++++-- Packs/BeyondTrust_Password_Safe/ReleaseNotes/1_1_14.md | 6 ++++++ Packs/BeyondTrust_Password_Safe/pack_metadata.json | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 Packs/BeyondTrust_Password_Safe/ReleaseNotes/1_1_14.md diff --git a/Packs/BeyondTrust_Password_Safe/ModelingRules/BeyondTrust_Password_Safe/BeyondTrust_Password_Safe.xif b/Packs/BeyondTrust_Password_Safe/ModelingRules/BeyondTrust_Password_Safe/BeyondTrust_Password_Safe.xif index 060e8fee5fa..3151421c3c2 100644 --- a/Packs/BeyondTrust_Password_Safe/ModelingRules/BeyondTrust_Password_Safe/BeyondTrust_Password_Safe.xif +++ b/Packs/BeyondTrust_Password_Safe/ModelingRules/BeyondTrust_Password_Safe/BeyondTrust_Password_Safe.xif @@ -80,10 +80,12 @@ alter // Extract raw data (https://www.beyondtrust.com/docs/beyondinsight-passwo request_id = arrayindex(regextract(details, "(?:Request \#|ReleaseRequestId=)(\w+)"), 0), target_application = arrayindex(regextract(details, "Application=(\w+)"), 0), target_account = coalesce( - arrayindex(regextract(event_target, "Account\:(\S+)"), 0), + arrayindex(regextract(event_target, "Account[:=](\S+)"), 0), arrayindex(regextract(_raw_log, "Username:\s\"?(\w.*?)\"?(?:,|\s{4}|\t)"), 0), arrayindex(regextract(_raw_log, "Account\s?Name:\s\"?(\w.*?)\"?(?:,|\s{4}|\t)"), 0)), - target_asset = arrayindex(regextract(event_target, "Asset(?:\=|\:)(\S+)"), 0), + target_asset = coalesce( + arrayindex(regextract(event_target, "Asset(?:\=|\:)(\S+)"), 0), + arrayindex(regextract(event_target, "ManagedSystem\=(\S+)"),0)), target_netbios_name = arrayindex(regextract(message, "NetBiosName=([^,]+)"), 0), user_domain = coalesce(arrayindex(regextract(user, "(.+)\\.+"), 0), arrayindex(split(user, "@"), 1), arrayindex(split(email, "@"), 1)), user_name = coalesce(arrayindex(regextract(user, "\\(.+)"), 0), arrayindex(regextract(user, "(.+)\@"), 0), user), diff --git a/Packs/BeyondTrust_Password_Safe/ReleaseNotes/1_1_14.md b/Packs/BeyondTrust_Password_Safe/ReleaseNotes/1_1_14.md new file mode 100644 index 00000000000..2837b614da3 --- /dev/null +++ b/Packs/BeyondTrust_Password_Safe/ReleaseNotes/1_1_14.md @@ -0,0 +1,6 @@ + +#### Modeling Rules + +##### BeyondTrust Password Safe Modeling Rule + +Improved the target account and target asset extraction to support additional value formats. diff --git a/Packs/BeyondTrust_Password_Safe/pack_metadata.json b/Packs/BeyondTrust_Password_Safe/pack_metadata.json index a4f4ca4f6a8..fec510796ce 100644 --- a/Packs/BeyondTrust_Password_Safe/pack_metadata.json +++ b/Packs/BeyondTrust_Password_Safe/pack_metadata.json @@ -2,7 +2,7 @@ "name": "BeyondTrust Password Safe", "description": "Unified password and session management for seamless accountability and control over privileged accounts.", "support": "xsoar", - "currentVersion": "1.1.13", + "currentVersion": "1.1.14", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 85f960b5a1fa70bc0faeee5307ea4e6cfe4ed3e1 Mon Sep 17 00:00:00 2001 From: Yehuda Rosenberg <90599084+RosenbergYehuda@users.noreply.github.com> Date: Sun, 19 Jul 2026 14:02:26 +0300 Subject: [PATCH 07/47] Fix armis issues (#45119) * rn * the fix * fix test * format * ai reviewer --- .../ArmisEventCollector.py | 11 +++- .../ArmisEventCollector_test.py | 55 +++++++++++++++++++ Packs/Armis/ReleaseNotes/1_3_1.md | 7 +++ Packs/Armis/pack_metadata.json | 6 +- 4 files changed, 75 insertions(+), 4 deletions(-) create mode 100644 Packs/Armis/ReleaseNotes/1_3_1.md diff --git a/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector.py b/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector.py index 1a33ac92d90..73f98c7340a 100644 --- a/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector.py +++ b/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector.py @@ -792,7 +792,7 @@ def on_page(page: list[dict]) -> None: running_state["last_event_time"] = latest_time add_time_to_events(new_events, dataset) - product = f"{PRODUCT}_{event_type.type}" if event_type.type != EVENT_TYPE_ALERTS else PRODUCT + product = f"{PRODUCT}_{dataset}" if event_type.type != EVENT_TYPE_ALERTS else PRODUCT send_start = time.monotonic() send_events_to_xsiam(new_events, vendor=VENDOR, product=product) @@ -1253,6 +1253,13 @@ def fetch_events( if "Devices" in event_types_to_fetch and not should_run_device_fetch(last_run, device_fetch_interval, datetime.now()): safe_debug("Skipping Devices fetch - interval not reached") event_types_to_fetch.remove("Devices") + for device_state_key in ( + DEVICES_LAST_FETCH, + f"{EVENT_TYPE_DEVICES}_last_fetch_ids", + f"{EVENT_TYPE_DEVICES}_last_fetch_next_field", + ): + if device_state_key in last_run: + next_run[device_state_key] = last_run[device_state_key] safe_debug(f"Event types after filtering: {event_types_to_fetch}") @@ -1463,7 +1470,7 @@ def handle_fetched_events(events: dict[str, list[dict[str, Any]]], next_run: dic demisto.setLastRun(next_run) -def events_to_command_results(events: dict[str, list], event_type) -> CommandResults: +def events_to_command_results(events: dict[str, list], event_type: str) -> CommandResults: """Return a CommandResults object with a table of fetched events. Args: diff --git a/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector_test.py b/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector_test.py index 619c8edec74..f50f0ac98cf 100644 --- a/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector_test.py +++ b/Packs/Armis/Integrations/ArmisEventCollector/ArmisEventCollector_test.py @@ -510,6 +510,43 @@ def test_handle_from_date_argument(self): from_date_datetime = handle_from_date_argument("2023-01-01T01:00:00") assert from_date_datetime == datetime(2023, 1, 1, 1, 0, 0) + @freeze_time("2023-01-01T01:00:00") + def test_devices_only_skip_preserves_state(self, mocker, dummy_client): + """ + Given: + - A Devices-only instance whose device-fetch interval has NOT elapsed + (last fetch 1 minute ago, interval 1 hour). + When: + - fetch_events runs and skips the device fetch. + Then: + - The returned next_run is NOT empty and retains devices_last_fetch_time, so the + persisted lastRun does not get wiped. + """ + from ArmisEventCollector import fetch_events + + fetch_start_time = arg_to_datetime("2023-01-01T01:00:00") + # Frozen now is 01:00:00; last_fetch 1 minute earlier -> interval (1h) not reached -> skip. + recent = "2023-01-01T00:59:00" + last_run = {"devices_last_fetch_time": recent, "devices_last_fetch_ids": ["dev1"], "devices_last_fetch_next_field": 0} + device_fetch_interval = timedelta(hours=1) + + events, next_run = fetch_events( + client=dummy_client, + max_fetch=1000, + devices_max_fetch=1000, + last_run=last_run, + fetch_start_time=fetch_start_time, + event_types_to_fetch=["Devices"], + device_fetch_interval=device_fetch_interval, + use_multithreading=False, + context_manager=None, + ) + + assert events == {} + assert next_run.get("devices_last_fetch_time") == recent + assert next_run.get("devices_last_fetch_ids") == ["dev1"] + assert next_run.get("devices_last_fetch_next_field") == 0 + class TestFetchFlow: fetch_start_time = arg_to_datetime("2023-01-01T01:00:00") @@ -1667,6 +1704,24 @@ def test_devices_product_routing(self, mocker): assert mock_send.call_args.kwargs["product"] == f"{PRODUCT}_devices" + def test_activities_product_routing(self, mocker): + """ + Given: The Activities event type (whose ``type`` is singular "activity" but whose + ``dataset_name`` is plural "activities"). + When: Callback ships a page. + Then: send_events_to_xsiam is called with product == "_activities" + (plural), so events land in ``armis_security_activities_raw``. + """ + from ArmisEventCollector import PRODUCT + + mock_send = mocker.patch("ArmisEventCollector.send_events_to_xsiam") + state = self._running_state() + on_page = _stream_page_to_xsiam(EVENT_TYPES["Activities"], state) + + on_page([{"activityUUID": "1", "time": "2023-01-01T01:00:10.000000+00:00"}]) + + assert mock_send.call_args.kwargs["product"] == f"{PRODUCT}_activities" + class TestFetchByAqlQueryWithCallback: """Tests for fetch_by_aql_query in streaming mode (on_page provided).""" diff --git a/Packs/Armis/ReleaseNotes/1_3_1.md b/Packs/Armis/ReleaseNotes/1_3_1.md new file mode 100644 index 00000000000..8f3d70dde27 --- /dev/null +++ b/Packs/Armis/ReleaseNotes/1_3_1.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### Armis Event Collector + +- Fixed an issue where *Activities* events were ingested into the *armis_security_activity_raw* dataset instead of the *armis_security_activities_raw* dataset. +- Fixed an issue where an instance configured to fetch only *Devices* stopped ingesting device events, because the device fetch state was not preserved between fetch cycles. diff --git a/Packs/Armis/pack_metadata.json b/Packs/Armis/pack_metadata.json index c43154e19bc..cae3a9a170a 100644 --- a/Packs/Armis/pack_metadata.json +++ b/Packs/Armis/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Armis", "description": "Agentless and passive security platform that sees, identifies, and classifies every device, tracks behavior, identifies threats, and takes action automatically to protect critical information and systems", "support": "partner", - "currentVersion": "1.3.0", + "currentVersion": "1.3.1", "author": "Armis Corporation", "url": "https://support.armis.com/", "email": "support@armis.com", @@ -12,7 +12,9 @@ "tags": [], "created": "2021-01-02T18:00:53Z", "useCases": [], - "keywords": [], + "keywords": [ + "Armis" + ], "dependencies": {}, "marketplaces": [ "xsoar", From 156a9675ef35686876100fcddc333c98feb94bbf Mon Sep 17 00:00:00 2001 From: Shmuel Kroizer <69422117+shmuel44@users.noreply.github.com> Date: Sun, 19 Jul 2026 14:05:55 +0300 Subject: [PATCH 08/47] CheckDockerImageAvailable (#45062) * update * bump * Bump pack from version Base to 1.42.9. --------- Co-authored-by: Content Bot --- Packs/Base/ReleaseNotes/1_42_9.md | 7 +++++++ .../CheckDockerImageAvailable.yml | 2 +- Packs/Base/pack_metadata.json | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) create mode 100644 Packs/Base/ReleaseNotes/1_42_9.md diff --git a/Packs/Base/ReleaseNotes/1_42_9.md b/Packs/Base/ReleaseNotes/1_42_9.md new file mode 100644 index 00000000000..7b6a8abe099 --- /dev/null +++ b/Packs/Base/ReleaseNotes/1_42_9.md @@ -0,0 +1,7 @@ + +#### Scripts + +##### CheckDockerImageAvailable + +- Updated the Docker image to: *demisto/python3:3.12.13.10404775*. + diff --git a/Packs/Base/Scripts/CheckDockerImageAvailable/CheckDockerImageAvailable.yml b/Packs/Base/Scripts/CheckDockerImageAvailable/CheckDockerImageAvailable.yml index e8937684120..c268746dde6 100644 --- a/Packs/Base/Scripts/CheckDockerImageAvailable/CheckDockerImageAvailable.yml +++ b/Packs/Base/Scripts/CheckDockerImageAvailable/CheckDockerImageAvailable.yml @@ -31,4 +31,4 @@ tests: - CheckDockerImageAvailableTest runas: DBotWeakRole fromversion: 5.0.0 -dockerimage: demisto/python3:3.12.13.9059085 +dockerimage: demisto/python3:3.12.13.10404775 diff --git a/Packs/Base/pack_metadata.json b/Packs/Base/pack_metadata.json index 17ab0daa1ee..41280735a28 100644 --- a/Packs/Base/pack_metadata.json +++ b/Packs/Base/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Base", "description": "The base pack for Cortex XSOAR.", "support": "xsoar", - "currentVersion": "1.42.8", + "currentVersion": "1.42.9", "author": "Cortex XSOAR", "serverMinVersion": "6.0.0", "url": "https://www.paloaltonetworks.com/cortex", From 5719829dafcc39d1723e13f5a7ee9114402ae090 Mon Sep 17 00:00:00 2001 From: Content Bot <55035720+content-bot@users.noreply.github.com> Date: Sun, 19 Jul 2026 14:07:40 +0300 Subject: [PATCH 09/47] [AUD Isolation] Active_Directory_Query - from 2026-06-24 (#44823) * Updated Docker Images for Active_Directory_Query. * Updated Release Notes. * Bump pack from version Active_Directory_Query to 1.7.2. * Bump pack from version Active_Directory_Query to 1.7.3. --------- Co-authored-by: content-bot Co-authored-by: Content Bot Co-authored-by: Shmuel Kroizer <69422117+shmuel44@users.noreply.github.com> --- .../Active_Directory_Query/Active_Directory_Query.yml | 2 +- Packs/Active_Directory_Query/ReleaseNotes/1_7_3.md | 7 +++++++ Packs/Active_Directory_Query/pack_metadata.json | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) create mode 100644 Packs/Active_Directory_Query/ReleaseNotes/1_7_3.md diff --git a/Packs/Active_Directory_Query/Integrations/Active_Directory_Query/Active_Directory_Query.yml b/Packs/Active_Directory_Query/Integrations/Active_Directory_Query/Active_Directory_Query.yml index cb8a76c0e38..9bb89b30b98 100644 --- a/Packs/Active_Directory_Query/Integrations/Active_Directory_Query/Active_Directory_Query.yml +++ b/Packs/Active_Directory_Query/Integrations/Active_Directory_Query/Active_Directory_Query.yml @@ -850,7 +850,7 @@ script: outputs: - contextPath: ActiveDirectory.ValidCredentials description: List of usernames that successfully logged in. - dockerimage: demisto/ldap:2.9.1.6911241 + dockerimage: demisto/ldap:2.9.1.9062583 ismappable: true isremotesyncout: true runonce: false diff --git a/Packs/Active_Directory_Query/ReleaseNotes/1_7_3.md b/Packs/Active_Directory_Query/ReleaseNotes/1_7_3.md new file mode 100644 index 00000000000..4cf3586d696 --- /dev/null +++ b/Packs/Active_Directory_Query/ReleaseNotes/1_7_3.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### Active Directory Query v2 + +- Updated the Docker image to: *demisto/ldap:2.9.1.9062583*. + diff --git a/Packs/Active_Directory_Query/pack_metadata.json b/Packs/Active_Directory_Query/pack_metadata.json index 37c20a0fdaf..5b88094c3a6 100644 --- a/Packs/Active_Directory_Query/pack_metadata.json +++ b/Packs/Active_Directory_Query/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Active Directory Query", "description": "Active Directory Query integration enables you to access and manage Active Directory objects (users, contacts, and computers).", "support": "xsoar", - "currentVersion": "1.7.2", + "currentVersion": "1.7.3", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 972e8269a8ef86ac79bd2641ae9e6454cae25f89 Mon Sep 17 00:00:00 2001 From: akshotiamit-pa Date: Sun, 19 Jul 2026 14:08:31 +0300 Subject: [PATCH 10/47] CRTX-264887 - map OpenAI auth_target for login events (#45116) Map xdm.target.resource.name to "OpenAI API Platform" for login/logout events in openai_admin_audit_raw so auth_target populates in the Authentication Story (event_type 102). --- .../OpenAIModelingRules/OpenAIModelingRules.xif | 1 + Packs/OpenAI/ReleaseNotes/2_1_3.md | 6 ++++++ Packs/OpenAI/pack_metadata.json | 4 ++-- 3 files changed, 9 insertions(+), 2 deletions(-) create mode 100644 Packs/OpenAI/ReleaseNotes/2_1_3.md diff --git a/Packs/OpenAI/ModelingRules/OpenAIModelingRules/OpenAIModelingRules.xif b/Packs/OpenAI/ModelingRules/OpenAIModelingRules/OpenAIModelingRules.xif index 5d48dc61f8d..65e217a66ba 100644 --- a/Packs/OpenAI/ModelingRules/OpenAIModelingRules/OpenAIModelingRules.xif +++ b/Packs/OpenAI/ModelingRules/OpenAIModelingRules/OpenAIModelingRules.xif @@ -78,6 +78,7 @@ alter type = "organization.updated", json_extract_scalar(organization_updated, "$.id"), null), xdm.target.resource.name = if( + type in ("login.succeeded", "login.failed", "logout.succeeded", "logout.failed"), "OpenAI API Platform", type not in ("certificate.created", "certificate.updated", "certificate.deleted", "ip_allowlist.created", "ip_allowlist.deleted", "role.created", "role.updated", "group.created", "group.updated", diff --git a/Packs/OpenAI/ReleaseNotes/2_1_3.md b/Packs/OpenAI/ReleaseNotes/2_1_3.md new file mode 100644 index 00000000000..fb4946af7ba --- /dev/null +++ b/Packs/OpenAI/ReleaseNotes/2_1_3.md @@ -0,0 +1,6 @@ +#### Modeling Rules + +##### OpenAI Modeling Rule + +Fixed an issue where *xdm.target.resource.name* was not mapped for authentication events (login/logout) in the *openai_admin_audit_raw* dataset, leaving *auth_target* empty in the Authentication Story. Mapped *xdm.target.resource.name* to *OpenAI API Platform* for the *login.succeeded*, *login.failed*, *logout.succeeded*, and *logout.failed* events. +<~XSIAM> (Available from Cortex XSIAM 2.1.0). diff --git a/Packs/OpenAI/pack_metadata.json b/Packs/OpenAI/pack_metadata.json index e38f70e1f33..9d857ee8347 100644 --- a/Packs/OpenAI/pack_metadata.json +++ b/Packs/OpenAI/pack_metadata.json @@ -2,7 +2,7 @@ "name": "OpenAI", "description": "The OpenAI API can be applied to virtually any task that involves understanding or generating natural language or code.", "support": "xsoar", - "currentVersion": "2.1.2", + "currentVersion": "2.1.3", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", @@ -25,4 +25,4 @@ "agentix", "xsiam" ] -} \ No newline at end of file +} From bc252185e7144ccce01398c9b844bc0fcef17b84 Mon Sep 17 00:00:00 2001 From: Content Bot <55035720+content-bot@users.noreply.github.com> Date: Sun, 19 Jul 2026 14:09:19 +0300 Subject: [PATCH 11/47] [AUD Isolation] ServiceNow - from 2026-06-17 (#44729) * Updated Docker Images for ServiceNow. * Updated Release Notes. * Bump pack from version ServiceNow to 2.9.13. * bump * bump --------- Co-authored-by: content-bot Co-authored-by: Content Bot Co-authored-by: shmuel44 Co-authored-by: Shmuel Kroizer <69422117+shmuel44@users.noreply.github.com> --- .../Integrations/ServiceNow_CMDB/ServiceNow_CMDB.yml | 2 +- .../Integrations/ServiceNowv2/ServiceNowv2.yml | 2 +- Packs/ServiceNow/ReleaseNotes/2_9_15.md | 11 +++++++++++ Packs/ServiceNow/pack_metadata.json | 2 +- 4 files changed, 14 insertions(+), 3 deletions(-) create mode 100644 Packs/ServiceNow/ReleaseNotes/2_9_15.md diff --git a/Packs/ServiceNow/Integrations/ServiceNow_CMDB/ServiceNow_CMDB.yml b/Packs/ServiceNow/Integrations/ServiceNow_CMDB/ServiceNow_CMDB.yml index 26bb9ef1bde..29c027a7ccb 100644 --- a/Packs/ServiceNow/Integrations/ServiceNow_CMDB/ServiceNow_CMDB.yml +++ b/Packs/ServiceNow/Integrations/ServiceNow_CMDB/ServiceNow_CMDB.yml @@ -299,7 +299,7 @@ script: name: servicenow-cmdb-oauth-login - description: Test the instance configuration when using OAuth authorization. name: servicenow-cmdb-oauth-test - dockerimage: demisto/auth-utils:1.0.0.6111183 + dockerimage: demisto/auth-utils:1.0.0.10133006 runonce: false script: '-' subtype: python3 diff --git a/Packs/ServiceNow/Integrations/ServiceNowv2/ServiceNowv2.yml b/Packs/ServiceNow/Integrations/ServiceNowv2/ServiceNowv2.yml index e8971ae075c..c72768893c6 100644 --- a/Packs/ServiceNow/Integrations/ServiceNowv2/ServiceNowv2.yml +++ b/Packs/ServiceNow/Integrations/ServiceNowv2/ServiceNowv2.yml @@ -1977,7 +1977,7 @@ script: description: Ticket close code. type: string - dockerimage: demisto/auth-utils:1.0.0.7837600 + dockerimage: demisto/auth-utils:1.0.0.10133006 isfetch: true ismappable: true isremotesyncin: true diff --git a/Packs/ServiceNow/ReleaseNotes/2_9_15.md b/Packs/ServiceNow/ReleaseNotes/2_9_15.md new file mode 100644 index 00000000000..b558898ec71 --- /dev/null +++ b/Packs/ServiceNow/ReleaseNotes/2_9_15.md @@ -0,0 +1,11 @@ + +#### Integrations + +##### ServiceNow v2 + +- Updated the Docker image to: *demisto/auth-utils:1.0.0.10133006*. + +##### ServiceNow CMDB + +- Updated the Docker image to: *demisto/auth-utils:1.0.0.10133006*. + diff --git a/Packs/ServiceNow/pack_metadata.json b/Packs/ServiceNow/pack_metadata.json index 2d8ef30f857..ce3c4fd2881 100644 --- a/Packs/ServiceNow/pack_metadata.json +++ b/Packs/ServiceNow/pack_metadata.json @@ -2,7 +2,7 @@ "name": "ServiceNow", "description": "Use The ServiceNow IT Service Management (ITSM) solution to modernize the way you manage and deliver services to your users.", "support": "xsoar", - "currentVersion": "2.9.14", + "currentVersion": "2.9.15", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From d3b4fc9107d8d7bbcdff756c66da45488f9db779 Mon Sep 17 00:00:00 2001 From: lironcohen272 Date: Sun, 19 Jul 2026 14:40:44 +0300 Subject: [PATCH 12/47] Ciac 16528: snowflake Oauth support (#44885) * Add OAuth support (integration code and yml parameters) * Enhance README and description with External OAuth authentication details * Update to version 1.0.13 * ix validation and pre-commit issues * fix after review * Add advanced option for 'Trust any certificate' and 'OAuth Client ID' fields in Snowflake integration * fixes after demo * bump version * fix pre-commit errors --------- Co-authored-by: test-split --- Packs/Snowflake/.secrets-ignore | 1 + .../Integrations/Snowflake/README.md | 26 ++ .../Integrations/Snowflake/Snowflake.py | 63 +++- .../Integrations/Snowflake/Snowflake.yml | 29 +- .../Snowflake/Snowflake_description.md | 15 +- .../Integrations/Snowflake/Snowflake_test.py | 292 ++++++++++++++++++ Packs/Snowflake/ReleaseNotes/1_0_14.md | 6 + Packs/Snowflake/pack_metadata.json | 2 +- 8 files changed, 427 insertions(+), 7 deletions(-) create mode 100644 Packs/Snowflake/ReleaseNotes/1_0_14.md diff --git a/Packs/Snowflake/.secrets-ignore b/Packs/Snowflake/.secrets-ignore index e69de29bb2d..f5a7f88f3d2 100644 --- a/Packs/Snowflake/.secrets-ignore +++ b/Packs/Snowflake/.secrets-ignore @@ -0,0 +1 @@ +https://docs.snowflake.com diff --git a/Packs/Snowflake/Integrations/Snowflake/README.md b/Packs/Snowflake/Integrations/Snowflake/README.md index b700745bfdc..1076e4ed7c1 100644 --- a/Packs/Snowflake/Integrations/Snowflake/README.md +++ b/Packs/Snowflake/Integrations/Snowflake/README.md @@ -24,6 +24,10 @@
  • Default role to use
  • Use system proxy settings
  • Trust server certificate (insecure)
    • +
  • Client ID
    • +
  • Client Secret
    • +
  • OAuth Token URL
    • +
  • OAuth Scope
  • Fetch incidents
  • Fetch query to retrieve new incidents. This field is mandatory when ‘Fetches incidents’ is set to true.
  • First fetch timestamp ( 
    • @@ -61,6 +65,28 @@
  • Use the credentials you configured. Refer to the two images at the bottom of the section titled Configure an External Credentials Vault.
    • +
  • +

    Authentication via External OAuth
    +To configure External OAuth authentication, please consult the following setup guidelines: Snowflake External OAuth Overview.

    +

    When using External OAuth, fill in the OAuth Client ID, OAuth Client Secret, OAuth Token URL, and optionally the OAuth Scope parameters. The Username field should still be set to the Snowflake service user that the IdP token maps to. The Password field can be left empty.

    +

    Prerequisites

    +
      +
    1. In the IdP (e.g., Okta): +
        +
      • Create an API Services application (no user redirect, machine-to-machine).
        • +
      • Note the Client ID and Client Secret.
        • +
      • Create / use a Custom Authorization Server.
        • +
      • Add a custom scope named session:role:<SNOWFLAKE_ROLE> (e.g., session:role:ANALYST).
        • +
      +
      2. +
    2. In Snowflake: +
        +
      • Create an External OAuth Security Integration that trusts the IdP's issuer & JWKS URL and maps the JWT sub claim to a Snowflake user.
        • +
      • Create / use a service user of TYPE = SERVICE.
        • +
      +
      3. +
    +
    • diff --git a/Packs/Snowflake/Integrations/Snowflake/Snowflake.py b/Packs/Snowflake/Integrations/Snowflake/Snowflake.py index 94bbce6349e..86302753517 100644 --- a/Packs/Snowflake/Integrations/Snowflake/Snowflake.py +++ b/Packs/Snowflake/Integrations/Snowflake/Snowflake.py @@ -30,6 +30,28 @@ SCHEMA = PARAMS.get("schema") ROLE = PARAMS.get("role") INSECURE = PARAMS.get("insecure", False) +OAUTH_CLIENT_ID = PARAMS.get("oauth_client_id") +OAUTH_CLIENT_SECRET = PARAMS.get("oauth_client_secret", {}).get("password") +OAUTH_TOKEN_URL = PARAMS.get("oauth_token_url") +OAUTH_SCOPE_RAW = PARAMS.get("oauth_scope", "") + + +def parse_oauth_scope(raw_scope: str) -> str | None: + """Convert a comma-separated scope string to space-separated as required by the Snowflake API. + + Args: + raw_scope: Comma-separated scope string (e.g. "scope1,scope2,scope3"). + + Returns: + Space-separated scope string, or None if the input is empty. + """ + if not raw_scope: + return None + result = " ".join(scope.strip() for scope in raw_scope.split(",") if scope.strip()) + return result or None + + +OAUTH_SCOPE = parse_oauth_scope(OAUTH_SCOPE_RAW) # How much time before the first fetch to retrieve incidents IS_FETCH = PARAMS.get("isFetch") FETCH_TIME = PARAMS.get("fetch_time") @@ -215,17 +237,45 @@ def get_connection_params(args): # pylint: disable=W9014 Snowflake connection params """ params: dict = {} - set_provided(params, "user", USER) - set_provided(params, "password", PASSWORD) + + # Validate that at least one authentication method is configured + has_oauth = any([OAUTH_CLIENT_ID, OAUTH_TOKEN_URL, OAUTH_CLIENT_SECRET, OAUTH_SCOPE]) + has_certificate = bool(CERTIFICATE) + has_password = bool(PASSWORD) + if not any([has_oauth, has_certificate, has_password]): + raise ValueError( + "No authentication method configured. Please provide one of: " + "Credentials (username/password), Certificate (key pair), or OAuth parameters." + ) + + if not USER: + raise ValueError("Username is required for all authentication methods (username/password, key pair, and OAuth).") + + set_provided(params, "user", USER) # user value is required for all authentication methods set_provided(params, "account", ACCOUNT) - set_provided(params, "authenticator", AUTHENTICATOR) set_provided(params, "region", REGION) set_provided(params, "insecure_mode", INSECURE) set_provided(params, "warehouse", args.get("warehouse"), WAREHOUSE) set_provided(params, "database", args.get("database"), DATABASE) set_provided(params, "schema", args.get("schema"), SCHEMA) set_provided(params, "role", args.get("role"), ROLE) - if CERTIFICATE: + + if has_oauth: + if not all([OAUTH_CLIENT_ID, OAUTH_TOKEN_URL, OAUTH_CLIENT_SECRET]): + raise ValueError("OAuth Client ID, Client Secret, and Token URL must all be provided for OAuth authentication.") + + # ── OAuth (External OAuth, Client Credentials grant) ── + # The snowflake-connector-python natively supports the client credentials flow. + # Override the authenticator with the OAuth-specific value. + set_provided(params, "authenticator", "OAUTH_CLIENT_CREDENTIALS") + set_provided(params, "oauth_client_id", OAUTH_CLIENT_ID) + set_provided(params, "oauth_client_secret", OAUTH_CLIENT_SECRET) + set_provided(params, "oauth_token_request_url", OAUTH_TOKEN_URL) + set_provided(params, "oauth_scope", OAUTH_SCOPE) + + elif has_certificate: + set_provided(params, "authenticator", AUTHENTICATOR) + # ── Key Pair authentication ── p_key = serialization.load_pem_private_key(CERTIFICATE, password=CERT_PASSWORD, backend=default_backend()) pkb = p_key.private_bytes( encoding=serialization.Encoding.DER, @@ -233,6 +283,11 @@ def get_connection_params(args): # pylint: disable=W9014 encryption_algorithm=serialization.NoEncryption(), ) params["private_key"] = pkb + + else: # Username + Password + set_provided(params, "authenticator", AUTHENTICATOR) + set_provided(params, "password", PASSWORD) + return params diff --git a/Packs/Snowflake/Integrations/Snowflake/Snowflake.yml b/Packs/Snowflake/Integrations/Snowflake/Snowflake.yml index 983be287d11..758686f4186 100644 --- a/Packs/Snowflake/Integrations/Snowflake/Snowflake.yml +++ b/Packs/Snowflake/Integrations/Snowflake/Snowflake.yml @@ -14,7 +14,7 @@ configuration: section: Connect - display: Username name: credentials - required: true + required: false type: 9 section: Connect - display: Region (only if you are not US West) @@ -52,11 +52,38 @@ configuration: type: 8 required: false section: Connect + advanced: true - display: Trust any certificate (not secure) name: insecure type: 8 required: false section: Connect + advanced: true +- display: Client ID + name: oauth_client_id + type: 0 + required: false + section: Connect + additionalinfo: The OAuth client ID issued by the IdP (e.g., Okta, Entra ID). Required when using External OAuth authentication. +- displaypassword: Client Secret + name: oauth_client_secret + hiddenusername: true + type: 9 + required: false + section: Connect + additionalinfo: The OAuth client secret issued by the IdP. Required when using External OAuth authentication. +- display: OAuth Token URL + name: oauth_token_url + type: 0 + required: false + section: Connect + additionalinfo: "The IdP token endpoint URL, e.g., https://.okta.com/oauth2//v1/token. Required when using External OAuth authentication." +- display: OAuth Scope + name: oauth_scope + type: 0 + required: false + section: Connect + additionalinfo: "The Snowflake OAuth scope. When multiple scopes are required, provide a comma-separated list. e.g., session:role:analyst,session:role:reader. See https://docs.snowflake.com/en/user-guide/oauth-ext-overview#scopes for details." - display: Fetch incidents name: isFetch type: 8 diff --git a/Packs/Snowflake/Integrations/Snowflake/Snowflake_description.md b/Packs/Snowflake/Integrations/Snowflake/Snowflake_description.md index 10f422180c9..9e371468404 100644 --- a/Packs/Snowflake/Integrations/Snowflake/Snowflake_description.md +++ b/Packs/Snowflake/Integrations/Snowflake/Snowflake_description.md @@ -10,4 +10,17 @@ The name of the Snowflake account to connect to without the domain name: snowfla To use Key Pair authentication, follow these instructions: 1. Follow steps 1-4 in the instructions detailed in the [Snowflake Computing documentation](https://docs.snowflake.net/manuals/user-guide/python-connector-example.html#using-key-pair-authentication). 2. Follow the instructions under the section titled **Configure Cortex XSOAR Credentials** at this [link](https://support.demisto.com/hc/en-us/articles/115002567894). -3. Use the credentials you configured. Refer to the two images at the bottom of the section titled **Configure an External Credentials Vault**. \ No newline at end of file +3. Use the credentials you configured. Refer to the two images at the bottom of the section titled **Configure an External Credentials Vault**. + +## Authentication Methods + +**Username** and **Account** are required for all methods. + +### Username and Password +Configure: **Username**, **Password**. Optionally set **Authenticator** for Okta SSO. + +### Key Pair +Configure: **Username**, **Certificate** (SSH Key). Optionally provide **Certificate Password** if the key is encrypted. + +### External OAuth +Configure: **Username**, **OAuth Client ID**, **OAuth Client Secret**, **OAuth Token URL**. Optionally provide **OAuth Scope**. Leave **Password** empty. To configure External OAuth authentication, please consult the following setup guidelines: [Snowflake External OAuth Overview](https://docs.snowflake.com/en/user-guide/oauth-ext-overview). diff --git a/Packs/Snowflake/Integrations/Snowflake/Snowflake_test.py b/Packs/Snowflake/Integrations/Snowflake/Snowflake_test.py index 19767edcfe9..eb1e14277a4 100644 --- a/Packs/Snowflake/Integrations/Snowflake/Snowflake_test.py +++ b/Packs/Snowflake/Integrations/Snowflake/Snowflake_test.py @@ -28,6 +28,41 @@ def test_convert_datetime_to_string(mocker, time, expected_results): assert results == expected_results +@pytest.mark.parametrize( + "raw_scope, expected", + [ + ("scope1,scope2,scope3", "scope1 scope2 scope3"), + ("session:role:analyst, session:role:reader", "session:role:analyst session:role:reader"), + ("scope1, scope2 , scope3", "scope1 scope2 scope3"), + ("", None), + (None, None), + (",,,", None), + ("scope1,,scope2", "scope1 scope2"), + ], +) +def test_parse_oauth_scope(mocker, raw_scope, expected): + """ + Given: + - A raw OAuth scope string (comma-separated or single) + - Case 1: Multiple scopes comma-separated + - Case 2: Single scope + - Case 3: Multiple scopes with extra whitespace + - Case 4: Empty string + - Case 5: None value + - Case 6: Only commas + - Case 7: Scopes with empty entries between commas + When: + - Calling parse_oauth_scope() + Then: + - Ensure the scopes are converted to space-separated format, or None for empty input + """ + mocker.patch.object(demisto, "params", return_value={}) + from Packs.Snowflake.Integrations.Snowflake.Snowflake import parse_oauth_scope + + result = parse_oauth_scope(raw_scope) + assert result == expected + + def test_fetch_incidents_passes_limit_key_to_snowflake_query(mocker): """ Given: @@ -77,6 +112,8 @@ def test_snowflake_query_uses_provided_limit(mocker): import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module mocker.patch.object(snowflake_module, "MAX_ROWS", 10000) + mocker.patch.object(snowflake_module, "USER", "test_user") + mocker.patch.object(snowflake_module, "PASSWORD", "test_password") from Packs.Snowflake.Integrations.Snowflake.Snowflake import snowflake_query mock_cursor = mocker.MagicMock() @@ -112,6 +149,8 @@ def test_snowflake_query_defaults_to_100_when_no_limit(mocker): import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module mocker.patch.object(snowflake_module, "MAX_ROWS", 10000) + mocker.patch.object(snowflake_module, "USER", "test_user") + mocker.patch.object(snowflake_module, "PASSWORD", "test_password") from Packs.Snowflake.Integrations.Snowflake.Snowflake import snowflake_query mock_cursor = mocker.MagicMock() @@ -131,3 +170,256 @@ def test_snowflake_query_defaults_to_100_when_no_limit(mocker): snowflake_query(args) mock_cursor.fetchmany.assert_called_once_with(100) + + +def test_get_connection_params_raises_when_no_auth(mocker): + """ + Given: + - No authentication method configured (no password, certificate, or OAuth) + When: + - get_connection_params() is called + Then: + - Ensure a ValueError is raised indicating no auth method configured + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "PASSWORD", None) + mocker.patch.object(snowflake_module, "CERTIFICATE", b"") + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_ID", None) + mocker.patch.object(snowflake_module, "OAUTH_TOKEN_URL", None) + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_SECRET", None) + mocker.patch.object(snowflake_module, "OAUTH_SCOPE", None) + + with pytest.raises(ValueError, match="No authentication method configured"): + snowflake_module.get_connection_params({}) + + +def test_get_connection_params_raises_when_no_user(mocker): + """ + Given: + - A password is configured but no username + When: + - get_connection_params() is called + Then: + - Ensure a ValueError is raised indicating username is required + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "PASSWORD", "test_password") + mocker.patch.object(snowflake_module, "USER", None) + + with pytest.raises(ValueError, match="Username is required"): + snowflake_module.get_connection_params({}) + + +def test_get_connection_params_password_auth(mocker): + """ + Given: + - Username and password are configured + When: + - get_connection_params() is called + Then: + - Ensure the returned params contain the user and password + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "USER", "test_user") + mocker.patch.object(snowflake_module, "PASSWORD", "test_password") + mocker.patch.object(snowflake_module, "CERTIFICATE", b"") + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_ID", None) + mocker.patch.object(snowflake_module, "OAUTH_TOKEN_URL", None) + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_SECRET", None) + mocker.patch.object(snowflake_module, "OAUTH_SCOPE", None) + + params = snowflake_module.get_connection_params({}) + + assert params["user"] == "test_user" + assert params["password"] == "test_password" + + +def test_get_connection_params_oauth(mocker): + """ + Given: + - OAuth client id, secret and token url are configured + When: + - get_connection_params() is called + Then: + - Ensure the returned params use OAUTH_CLIENT_CREDENTIALS authenticator and OAuth values + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "USER", "test_user") + mocker.patch.object(snowflake_module, "PASSWORD", None) + mocker.patch.object(snowflake_module, "CERTIFICATE", b"") + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_ID", "client_id") + mocker.patch.object(snowflake_module, "OAUTH_TOKEN_URL", "https://token.url") + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_SECRET", "client_secret") + mocker.patch.object(snowflake_module, "OAUTH_SCOPE", "scope1 scope2") + + params = snowflake_module.get_connection_params({}) + + assert params["authenticator"] == "OAUTH_CLIENT_CREDENTIALS" + assert params["oauth_client_id"] == "client_id" + assert params["oauth_client_secret"] == "client_secret" + assert params["oauth_token_request_url"] == "https://token.url" + assert params["oauth_scope"] == "scope1 scope2" + + +def test_get_connection_params_oauth_missing_fields(mocker): + """ + Given: + - OAuth client id is set but token url and secret are missing + When: + - get_connection_params() is called + Then: + - Ensure a ValueError is raised indicating all OAuth fields are required + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "USER", "test_user") + mocker.patch.object(snowflake_module, "PASSWORD", None) + mocker.patch.object(snowflake_module, "CERTIFICATE", b"") + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_ID", "client_id") + mocker.patch.object(snowflake_module, "OAUTH_TOKEN_URL", None) + mocker.patch.object(snowflake_module, "OAUTH_CLIENT_SECRET", None) + mocker.patch.object(snowflake_module, "OAUTH_SCOPE", None) + + with pytest.raises(ValueError, match="OAuth Client ID, Client Secret, and Token URL"): + snowflake_module.get_connection_params({}) + + +def test_set_provided(mocker): + """ + Given: + - Various combinations of val1 and val2 + When: + - set_provided() is called + Then: + - Ensure the correct value is set (val1 preferred, val2 fallback, nothing if both empty) + """ + mocker.patch.object(demisto, "params", return_value={}) + + from Packs.Snowflake.Integrations.Snowflake.Snowflake import set_provided + + params: dict = {} + set_provided(params, "key1", "value1") + set_provided(params, "key2", None, "fallback") + set_provided(params, "key3", None, None) + + assert params["key1"] == "value1" + assert params["key2"] == "fallback" + assert "key3" not in params + + +def test_process_table_row(mocker): + """ + Given: + - A row with a Decimal value and a datetime value plus the corresponding checks + When: + - process_table_row() is called + Then: + - Ensure Decimal is converted to string and datetime is converted to a formatted string + """ + from decimal import Decimal + + mocker.patch.object(demisto, "params", return_value={}) + + from Packs.Snowflake.Integrations.Snowflake.Snowflake import process_table_row + + row = {"num": Decimal("10.5"), "ts": datetime(2024, 8, 14, 22, 43, 9), "other": "text"} + checks = {"isDecimal": ["num"], "isDT": ["ts"]} + + result = process_table_row(row, checks) + + assert result["num"] == "10.5" + assert result["ts"] == "2024-08-14 22:43:09.00" + assert result["other"] == "text" + + +def test_format_to_json_serializable(mocker): + """ + Given: + - Column descriptions with number/int and timestamp type codes and a list of rows + When: + - format_to_json_serializable() is called + Then: + - Ensure the Decimal and datetime values are reformatted to json serializable types + """ + from decimal import Decimal + + mocker.patch.object(demisto, "params", return_value={}) + + from Packs.Snowflake.Integrations.Snowflake.Snowflake import format_to_json_serializable + + column_descriptions = [ + ("num", 0, None, None, None, None, None), + ("ts", 4, None, None, None, None, None), + ] + results = [{"num": Decimal("3.14"), "ts": datetime(2024, 8, 14, 22, 43, 9)}] + + formatted = format_to_json_serializable(column_descriptions, results) + + assert formatted[0]["num"] == "3.14" + assert formatted[0]["ts"] == "2024-08-14 22:43:09.00" + + +def test_error_message_from_snowflake_error(mocker): + """ + Given: + - A snowflake error-like object with errno 606 + When: + - error_message_from_snowflake_error() is called + Then: + - Ensure a formatted error message that mentions specifying an active warehouse is returned + """ + mocker.patch.object(demisto, "params", return_value={}) + + from Packs.Snowflake.Integrations.Snowflake.Snowflake import error_message_from_snowflake_error + + class FakeError: + errno = 606 + sqlstate = "00000" + sfqid = "abc-123" + raw_msg = "No active warehouse. Additional info." + + result = error_message_from_snowflake_error(FakeError()) + + assert "Snowflake DB error code: 606" in result + assert "Specify an active warehouse" in result + + +def test_row_to_incident(mocker): + """ + Given: + - Column descriptions and a row containing a datetime column + When: + - row_to_incident() is called + Then: + - Ensure an incident dict with name, occurred, timestamp and rawJSON is returned + """ + mocker.patch.object(demisto, "params", return_value={}) + + import Packs.Snowflake.Integrations.Snowflake.Snowflake as snowflake_module + + mocker.patch.object(snowflake_module, "DATETIME_COLUMN", "TS") + mocker.patch.object(snowflake_module, "INCIDENT_NAME_COLUMN", "NAME") + + column_descriptions = [("TS", 4, None, None, None, None, None)] + row = {"TS": datetime(2024, 8, 14, 22, 43, 9), "NAME": "my-incident"} + + incident = snowflake_module.row_to_incident(column_descriptions, row) + + assert incident["name"] == "my-incident" + assert "occurred" in incident + assert "timestamp" in incident + assert "rawJSON" in incident diff --git a/Packs/Snowflake/ReleaseNotes/1_0_14.md b/Packs/Snowflake/ReleaseNotes/1_0_14.md new file mode 100644 index 00000000000..44a94eeddcc --- /dev/null +++ b/Packs/Snowflake/ReleaseNotes/1_0_14.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Snowflake + +- Added support for *OAuth Scope*, *OAuth Token URL*, *OAuth Client Secret* and *OAuth Client ID* parameters. These parameters are required when using external OAuth authentication. diff --git a/Packs/Snowflake/pack_metadata.json b/Packs/Snowflake/pack_metadata.json index a25e00b2ba6..3bd8dae3cb6 100644 --- a/Packs/Snowflake/pack_metadata.json +++ b/Packs/Snowflake/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Snowflake", "description": "Analytic data warehouse provided as Software-as-a-Service.", "support": "xsoar", - "currentVersion": "1.0.13", + "currentVersion": "1.0.14", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From db34398714b4a12d008a3f5700fe9b72dd8049e4 Mon Sep 17 00:00:00 2001 From: lironcohen272 Date: Sun, 19 Jul 2026 14:47:30 +0300 Subject: [PATCH 13/47] CIAC-16805: Add Token URL parameter for OAuth 2.0 endpoint override in FireEye ETP (#45037) * Add Token URL parameter for OAuth 2.0 endpoint override in FireEye ETP integrations * fix after ai-reviewer * update docker image * fix validation erros * bump version * Revert "bump version" This reverts commit d613f33d66e19d3b543c4f4640bc51c596ce18b5. --------- Co-authored-by: test-split --- Packs/FireEyeETP/.secrets-ignore | 5 +- .../Integrations/FireEyeETP/FireEyeETP.py | 8 ++- .../Integrations/FireEyeETP/FireEyeETP.yml | 9 ++- .../FireEyeETP/FireEyeETP_description.md | 3 + .../FireEyeETP/FireEyeETP_test.py | 45 +++++++++++++++ .../Integrations/FireEyeETP/README.md | 1 + .../FireEyeETPEventCollector.py | 9 ++- .../FireEyeETPEventCollector.yml | 9 ++- .../FireEyeETPEventCollector_description.md | 3 + .../FireEyeETPEventCollector_test.py | 55 +++++++++++++++++++ .../FireEyeETPEventCollector/README.md | 3 +- Packs/FireEyeETP/ReleaseNotes/2_1_6.md | 11 ++++ Packs/FireEyeETP/pack_metadata.json | 2 +- 13 files changed, 154 insertions(+), 9 deletions(-) create mode 100644 Packs/FireEyeETP/ReleaseNotes/2_1_6.md diff --git a/Packs/FireEyeETP/.secrets-ignore b/Packs/FireEyeETP/.secrets-ignore index 58d345a9e12..66beb2d6920 100644 --- a/Packs/FireEyeETP/.secrets-ignore +++ b/Packs/FireEyeETP/.secrets-ignore @@ -2,4 +2,7 @@ u003cbot@soc.com u003csoc@bot.com https://docs.trellix.com https://us.etp.trellix.com -https://eu.etp.trellix.com \ No newline at end of file +https://eu.etp.trellix.com +https://ap.etp.trellix.com +https://ca.etp.trellix.com +https://iam.us.trellix-gov.com \ No newline at end of file diff --git a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.py b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.py index 2c4e681f7bc..c2d90a5d3f3 100755 --- a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.py +++ b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.py @@ -49,6 +49,11 @@ ).strip() API_KEY = PARAMS.get("credentials_api_key", {}).get("password") or PARAMS.get("api_key") +DEFAULT_TOKEN_URL = "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token" +TOKEN_URL_SUFFIX = "/iam/v1.0/token" +_token_url_override = PARAMS.get("token_url", "").strip() +TOKEN_URL = urljoin(_token_url_override, TOKEN_URL_SUFFIX) if _token_url_override else DEFAULT_TOKEN_URL + BASE_PATH_V1 = "{}/api/v1".format(PARAMS.get("server")) # Deprecated endpoint BASE_PATH_V2 = "{}/api/v2".format(PARAMS.get("server")) @@ -119,8 +124,7 @@ def fetch_oauth_token(): """ Fetch OAuth 2.0 access token """ - # Trellix OAuth2 endpoint - token_url = "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token" + token_url = TOKEN_URL credentials = f"{CLIENT_ID}:{CLIENT_SECRET}" encoded_credentials = base64.b64encode(credentials.encode()).decode() diff --git a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.yml b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.yml index 2d3ada0cc0a..0e5d2131cb5 100755 --- a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.yml +++ b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP.yml @@ -30,6 +30,13 @@ configuration: required: false additionalinfo: "Space-separated list of OAuth scopes. Note: Only include scopes that your application's Client ID has already been authorized to use." section: Connect +- display: Token URL + name: token_url + type: 0 + required: false + additionalinfo: "The OAuth 2.0 token endpoint base URL override. Leave empty to use the default Trellix IAM endpoint. For Trellix GovCloud tenants, set to https://iam.us.trellix-gov.com." + section: Connect + advanced: true - display: API key name: api_key defaultvalue: "" @@ -482,7 +489,7 @@ script: Cloud message ID. isfetch: true subtype: python3 - dockerimage: demisto/python3:3.12.12.7090913 + dockerimage: demisto/python3:3.12.13.10404775 tests: - No Test defaultclassifier: Trellix Incident Classifier diff --git a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_description.md b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_description.md index 70e8e241333..61cfd411144 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_description.md +++ b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_description.md @@ -13,4 +13,7 @@ To ensure a successful connection, you must select the correct authentication me | Ends in trellix.com | OAuth 2.0 | Client ID, Client Secret, and OAuth Scopes | | Ends in fireeye.com | Legacy API Key | API Key (only) | +**Trellix GovCloud tenants:** Set the **Token URL** parameter to `https://iam.us.trellix-gov.com`. +Leave empty to use the default Trellix IAM endpoint. + For official documentation on configuring access, [see here.](https://docs.trellix.com/bundle/etp_api/page/UUID-30726aa3-e420-6f62-6b84-6ad0bdace483.html) \ No newline at end of file diff --git a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_test.py b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_test.py index 69df51c0708..d8f14121060 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_test.py +++ b/Packs/FireEyeETP/Integrations/FireEyeETP/FireEyeETP_test.py @@ -551,6 +551,51 @@ def test_get_alert_list_with_alert_id(mocker): assert result.outputs == alert_data +@pytest.mark.parametrize( + "token_url_value, expected_url", + [ + # Default commercial URL when no override is configured (TOKEN_URL stays as DEFAULT_TOKEN_URL) + ( + "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token", + "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token", + ), + # GovCloud override — user provides base URL, code appends /iam/v1.0/token via urljoin + ( + "https://iam.us.trellix-gov.com/iam/v1.0/token", + "https://iam.us.trellix-gov.com/iam/v1.0/token", + ), + ], +) +def test_fetch_oauth_token_uses_configured_url(mocker, token_url_value, expected_url): + """ + Given: + - A resolved TOKEN_URL (default commercial or GovCloud after urljoin). + When: + - Calling fetch_oauth_token. + Then: + - Ensure the OAuth token request is sent to the resolved URL. + """ + mocker.patch("FireEyeETP.TOKEN_URL", token_url_value) + mocker.patch("FireEyeETP.CLIENT_ID", "test_id") + mocker.patch("FireEyeETP.CLIENT_SECRET", "test_secret") + mocker.patch("FireEyeETP.SCOPES", "etp.alrt.ro") + mocker.patch("FireEyeETP.USE_SSL", True) + mocker.patch("FireEyeETP.get_integration_context", return_value={}) + mocker.patch("FireEyeETP.set_integration_context") + + mock_response = MagicMock() + mock_response.json.return_value = {"access_token": "test_token", "expires_in": 600} + mock_response.raise_for_status.return_value = None + mock_post = mocker.patch("FireEyeETP.requests.post", return_value=mock_response) + + token = FireEyeETP.fetch_oauth_token() + + assert token == "test_token" + mock_post.assert_called_once() + actual_url = mock_post.call_args[0][0] + assert actual_url == expected_url + + def test_get_alert_list_without_alert_id(mocker): """ Given: diff --git a/Packs/FireEyeETP/Integrations/FireEyeETP/README.md b/Packs/FireEyeETP/Integrations/FireEyeETP/README.md index f5cbbfdf539..d2af3f84663 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETP/README.md +++ b/Packs/FireEyeETP/Integrations/FireEyeETP/README.md @@ -85,6 +85,7 @@ We support two different authentication methods depending on the endpoint domain | Client ID (OAuth) | Use the Client ID and Client Secret for the Trellix base URL. | | | Client Secret (OAuth) | | | | OAuth Scopes (OAuth) | Space-separated list of OAuth scopes. Note: Only include scopes that your application's Client ID has already been authorized to use. | False | +| Token URL | Override the OAuth 2.0 token endpoint base URL. Leave empty to use the default Trellix IAM endpoint. For Trellix GovCloud tenants, set to `https://iam.us.trellix-gov.com`. | False | | API Key | Use the Api key for the FireEye base URL. | False | | Trust any certificate (not secure) | | False | | Use system proxy settings | | False | diff --git a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.py b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.py index 270682dc67d..849fbef4d1a 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.py +++ b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.py @@ -23,6 +23,8 @@ CALCULATED_MAX_FETCH = 5000 DEFAULT_LIMIT = 10 DEFAULT_URL = "https://etp.us.fireeye.com" +DEFAULT_TOKEN_URL = "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token" +TOKEN_URL_SUFFIX = "/iam/v1.0/token" DATEPARSER_SETTINGS = { "RETURN_AS_TIMEZONE_AWARE": True, "TIMEZONE": "UTC", @@ -73,6 +75,7 @@ def __init__( api_key: str = "", outbound_traffic: bool = False, hide_sensitive: bool = False, + token_url: str = "", ) -> None: super().__init__(base_url, verify_certificate, proxy) self.client_id = client_id @@ -81,6 +84,7 @@ def __init__( self.api_key = api_key self.outbound_traffic = outbound_traffic self.hide_sensitive = hide_sensitive + self.token_url = urljoin(token_url, TOKEN_URL_SUFFIX) if token_url else DEFAULT_TOKEN_URL self.access_token = "" # Set up headers based on authentication method @@ -122,8 +126,7 @@ def _fetch_oauth_token(self) -> str: Returns the access token or raises ValueError if authentication fails. """ try: - # Trellix OAuth2 endpoint - token_url = "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token" + token_url = self.token_url credentials = f"{self.client_id}:{self.client_secret}" encoded_credentials = base64.b64encode(credentials.encode()).decode() @@ -801,6 +804,7 @@ def main() -> None: # pragma: no cover client_secret = params.get("oauth_credentials", {}).get("password", "") scope = params.get("oauth_scopes", "etp.conf.ro etp.rprt.ro").strip() api_key = params.get("credentials", {}).get("password", "") + token_url = params.get("token_url", "").strip() base_url = params.get("url", "").rstrip("/") verify = not params.get("insecure", False) proxy = params.get("proxy", False) @@ -845,6 +849,7 @@ def main() -> None: # pragma: no cover api_key=api_key, outbound_traffic=outbound_traffic, hide_sensitive=hide_sensitive, + token_url=token_url, ) events_to_run_on = EVENT_TYPES + OUTBOUND_EVENT_TYPES if outbound_traffic else EVENT_TYPES diff --git a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.yml b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.yml index 07d95d22dfc..6de1eebfc22 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.yml +++ b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector.yml @@ -25,6 +25,13 @@ configuration: required: false additionalinfo: "Space-separated list of OAuth scopes. Note: Only include scopes that your application's Client ID has already been authorized to use." section: Connect +- display: Token URL + name: token_url + type: 0 + required: false + additionalinfo: "The OAuth 2.0 token endpoint base URL override. Leave empty to use the default Trellix IAM endpoint. For Trellix GovCloud tenants, set to https://iam.us.trellix-gov.com." + section: Connect + advanced: true - displaypassword: API Key additionalinfo: The API Key allows you to integrate with the Trellix Email Security - Cloud. name: credentials @@ -95,7 +102,7 @@ script: required: true description: Gets events from Trellix Email Security - Cloud. This command is used for developing/ debugging and is to be used with caution, as it can create events, leading to events duplication and API request limitation exceeding. name: fireeye-etp-get-events - dockerimage: demisto/python3:3.12.11.4508456 + dockerimage: demisto/python3:3.12.13.10404775 isfetchevents: true script: "" subtype: python3 diff --git a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_description.md b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_description.md index 32e5d6890a9..e511b86a5fd 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_description.md +++ b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_description.md @@ -25,6 +25,9 @@ We support two different authentication methods depending on the endpoint domain | **Ends in `trellix.com`** | **OAuth 2.0** | **Client ID**, **Client Secret**, and **OAuth Scopes** | | **Ends in `fireeye.com`** | **API Key** | **API Key** (only) | +**Trellix GovCloud tenants:** Set the **Token URL** parameter to `https://iam.us.trellix-gov.com`. +Leave empty to use the default Trellix IAM endpoint. + For official documentation on configuring access, [see here.](https://docs.trellix.com/bundle/etp_api/page/UUID-30726aa3-e420-6f62-6b84-6ad0bdace483.html) # Configuring API keys diff --git a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_test.py b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_test.py index b33f82f5417..d4bf9388882 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_test.py +++ b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/FireEyeETPEventCollector_test.py @@ -467,6 +467,61 @@ def test_validate_authentication_params(client_id, client_secret, api_key, scope assert result is None +@pytest.mark.parametrize( + "token_url_init, expected_url", + [ + # Default URL when no override is configured (empty string -> DEFAULT_TOKEN_URL) + ( + "", + "https://auth.trellix.com/auth/realms/IAM/protocol/openid-connect/token", + ), + # GovCloud override — user provides base URL, Client.__init__ uses urljoin to append suffix + ( + "https://iam.us.trellix-gov.com", + "https://iam.us.trellix-gov.com/iam/v1.0/token", + ), + ], +) +def test_client_fetch_oauth_token_uses_configured_url(mocker, token_url_init, expected_url): + """ + Given: + - A Client instance with a configured token_url (empty for default, or GovCloud base URL). + When: + - Calling _fetch_oauth_token on the Client. + Then: + - Ensure the OAuth token request is sent to the expected resolved URL. + """ + # Mock _get_valid_oauth_token to prevent real token fetch during __init__ + mocker.patch.object( + FireEyeETPEventCollector.Client, + "_get_valid_oauth_token", + return_value="init_token", + ) + + client = FireEyeETPEventCollector.Client( + base_url="https://test.com", + verify_certificate=True, + proxy=False, + client_id="test_id", + client_secret="test_secret", + scope="etp.alrt.ro", + token_url=token_url_init, + ) + + mock_response = mocker.MagicMock() + mock_response.json.return_value = {"access_token": "test_token", "expires_in": 600} + mock_response.raise_for_status.return_value = None + mock_post = mocker.patch("requests.post", return_value=mock_response) + mocker.patch("FireEyeETPEventCollector.set_integration_context") + + token = client._fetch_oauth_token() + + assert token == "test_token" + mock_post.assert_called_once() + actual_url = mock_post.call_args[0][0] + assert actual_url == expected_url + + @pytest.mark.parametrize( "client_id, client_secret, api_key, expected_result, expected_exception", [ diff --git a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/README.md b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/README.md index fae474b8b98..9638843d0ce 100644 --- a/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/README.md +++ b/Packs/FireEyeETP/Integrations/FireEyeETPEventCollector/README.md @@ -21,10 +21,11 @@ We support two different authentication methods depending on the endpoint domain | **Parameter** | **Description** | **Required** | | --- | --- | --- | -| Server URL (e.g., https://etp.us.fireeye.com)| List of valid URLs:
    US Instance:
    https://etp.us.fireeye.com or https://us.etp.
    trellix.com
    EMEA Instance:
    https://etp.eu.fireeye.com or https://eu.etp.trellix.com
    APJ Instance:
    https://etp.ap.fireeye.com or https://ap.etp.trellix.com
    USGOV Instance:
    https://etp.us.fireeyegov.com
    CA Instance:
    https://etp.ca.fireeye.com or https://ca.etp.trellix.com | True | +| Server URL (e.g., https://etp.us.fireeye.com)| List of valid URLs:
    US Instance:
    https://etp.us.fireeye.com or https://us.etp.trellix.com
    EMEA Instance:
    https://etp.eu.fireeye.com or https://eu.etp.trellix.com
    APJ Instance:
    https://etp.ap.fireeye.com or https://ap.etp.trellix.com
    USGOV Instance:
    https://etp.us.fireeyegov.com
    CA Instance:
    https://etp.ca.fireeye.com or https://ca.etp.trellix.com | True | | Client ID | For the Trellix server URL (OAuth). | False | | Client Secret | For the Trellix server URL (OAuth). | False| | OAuth Scopes | For the Trellix server URL (OAuth).
    Space-separated list of OAuth scopes.
    **Note:** Only include scopes that your application's Client ID has already been authorized to use. The full list is: `etp.conf.ro etp.trce.rw etp.admn.ro etp.domn.ro etp.accs.rw etp.quar.rw etp.domn.rw etp.rprt.rw etp.accs.ro etp.quar.ro etp.alrt.rw etp.rprt.ro etp.conf.rw etp.trce.ro etp.alrt.ro etp.admn.rw` | False | +| Token URL | Override the OAuth 2.0 token endpoint base URL. Leave empty to use the default Trellix IAM endpoint. For Trellix GovCloud tenants, set to `https://iam.us.trellix-gov.com`. | False | | API Secret Key | For the FireEye server URL. The API Key allows you to integrate with the Trellix Email Security - Cloud. | False | | Maximum number of Alerts to fetch. | The maximum number of Alert events to fetch from Trellix Email Security - Cloud. | | | Maximum number of Email Trace to fetch. | The maximum number of Email Trace events to fetch from Trellix Email Security - Cloud. | | diff --git a/Packs/FireEyeETP/ReleaseNotes/2_1_6.md b/Packs/FireEyeETP/ReleaseNotes/2_1_6.md new file mode 100644 index 00000000000..9e8a886d98c --- /dev/null +++ b/Packs/FireEyeETP/ReleaseNotes/2_1_6.md @@ -0,0 +1,11 @@ +#### Integrations + +##### Trellix Email Security - Cloud +- Updated the Docker image to: *demisto/python3:3.12.13.10404775*. + +- Added the *Token URL* parameter to allow overriding the OAuth 2.0 token endpoint per-instance. Use *https://iam.us.trellix-gov.com* for Trellix GovCloud tenants. + +##### Trellix Email Security - Cloud Event Collector +- Updated the Docker image to: *demisto/python3:3.12.13.10404775*. + +- Added the *Token URL* parameter to allow overriding the OAuth 2.0 token endpoint per-instance. Use *https://iam.us.trellix-gov.com* for Trellix GovCloud tenants. diff --git a/Packs/FireEyeETP/pack_metadata.json b/Packs/FireEyeETP/pack_metadata.json index 4041ed34f2a..2a5f4211c8c 100644 --- a/Packs/FireEyeETP/pack_metadata.json +++ b/Packs/FireEyeETP/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Trellix Email Security - Cloud", "description": "Trellix Email Security - Cloud (Formerly FireEye ETP) is a cloud-based platform that protects against advanced email attacks.", "support": "xsoar", - "currentVersion": "2.1.5", + "currentVersion": "2.1.6", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 094ad271a45745ea031a830ec7dff7b967d74e39 Mon Sep 17 00:00:00 2001 From: Israel Lappe <79846863+ilappe@users.noreply.github.com> Date: Sun, 19 Jul 2026 16:11:58 +0300 Subject: [PATCH 14/47] EDL heartbeat stability fix + NGINX fail-fast cache concurrency control (#44829) * possible fix * additional logs * additional logs * 2 tier architecture * prepare for merge * Remove keepalive_timeout from NGINXApiModule * fix pre commit * fix pre commit - mypy * Address marketplace-ai-reviewer comments on PR #44829 - RN wording/styling (approved prefixes, single-asterisk params) - YAML additionalinfo descriptions start with 'The' and end with a period - NGINXApiModule: datetime.now(timezone.utc) instead of deprecated utcnow(); log traceback before re-raising streaming errors - EDL: make _stdout_lock_timeout tweak a true no-op via hasattr guard; debug log when get_request_id has no Flask request context - EDL tests: add coverage for get_request_id and _stdout_lock_timeout restoration/no-op * Bump EDL to 3.4.0 and TAXIIServer to 2.3.0 (deprecations require minor) Per marketplace-ai-reviewer review: deprecating the Cache Lock Timeout/Age parameters is a functional change requiring a minor version bump. * Update EDL README: mark Cache Lock Timeout/Age parameters as deprecated * EDL tests: use a single top-level 'import EDL as edl' (resolve CodeQL mixed-import finding) * fix validations * Bump pack from version ApiModules to 2.4.35. * update docker * empty * fix validation * Bump pack from version ApiModules to 2.4.36. * Update Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule.py Co-authored-by: Dean Arbel --------- Co-authored-by: Content Bot Co-authored-by: Dean Arbel --- Packs/ApiModules/ReleaseNotes/2_4_36.md | 6 + .../Scripts/NGINXApiModule/NGINXApiModule.py | 655 ++++++++++++++++-- .../NGINXApiModule/NGINXApiModule_test.py | 124 +++- Packs/ApiModules/pack_metadata.json | 2 +- Packs/EDL/Integrations/EDL/EDL.py | 75 +- Packs/EDL/Integrations/EDL/EDL.yml | 32 +- Packs/EDL/Integrations/EDL/EDL_test.py | 94 +++ Packs/EDL/Integrations/EDL/README.md | 4 +- Packs/EDL/ReleaseNotes/3_4_0.md | 9 + Packs/EDL/pack_metadata.json | 2 +- Packs/ExportIndicators/ReleaseNotes/1_0_24.md | 6 + Packs/ExportIndicators/pack_metadata.json | 2 +- .../TAXII2Server/TAXII2Server.yml | 32 +- Packs/TAXIIServer/ReleaseNotes/2_3_0.md | 8 + Packs/TAXIIServer/pack_metadata.json | 2 +- 15 files changed, 950 insertions(+), 103 deletions(-) create mode 100644 Packs/ApiModules/ReleaseNotes/2_4_36.md create mode 100644 Packs/EDL/ReleaseNotes/3_4_0.md create mode 100644 Packs/ExportIndicators/ReleaseNotes/1_0_24.md create mode 100644 Packs/TAXIIServer/ReleaseNotes/2_3_0.md diff --git a/Packs/ApiModules/ReleaseNotes/2_4_36.md b/Packs/ApiModules/ReleaseNotes/2_4_36.md new file mode 100644 index 00000000000..725559127f2 --- /dev/null +++ b/Packs/ApiModules/ReleaseNotes/2_4_36.md @@ -0,0 +1,6 @@ + +#### Scripts + +##### NGINXApiModule + +- Improved implementation of cache concurrency control to better handle a high volume of concurrent requests. diff --git a/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule.py b/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule.py index 47971bb59d9..4a8581665c4 100644 --- a/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule.py +++ b/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule.py @@ -1,8 +1,9 @@ import os import subprocess import traceback +import uuid +from itertools import count from math import ceil -from multiprocessing import Process from pathlib import Path from signal import SIGUSR1 from string import Template @@ -13,7 +14,7 @@ import requests from CommonServerPython import * # noqa: F401 from flask.logging import default_handler -from gevent.pywsgi import WSGIServer +from gevent.pywsgi import WSGIHandler, WSGIServer from CommonServerUserPython import * @@ -21,19 +22,373 @@ class Handler: @staticmethod def write(msg: str): - demisto.info(msg) + # gevent's pywsgi writes one Common-Log-Format access line per request here. + # Tag it so it is easy to grep for and correlate with the nginx access log. + # Kept at debug: the structured "wsgi request:". + demisto.debug(f"wsgi access: {msg.rstrip()}") class ErrorHandler: @staticmethod def write(msg: str): - demisto.error(f"wsgi error: {msg}") + demisto.error(f"wsgi error: {msg.rstrip()}") DEMISTO_LOGGER: Handler = Handler() ERROR_LOGGER: ErrorHandler = ErrorHandler() +# --- Request lifecycle / concurrency instrumentation ------------------------- +# Monotonic sequence so every request can be correlated across the start line, +# the end line and the gevent access line. +_REQUEST_SEQ = count(1) +# The environ key under which we stash the per-request id so DemistoWSGIHandler +# can print the same id on the "wsgi access:" line. +REQUEST_ID_ENVIRON_KEY = "nginxapimodule.request_id" +# The environ key under which the middleware stashes the number of body bytes the +# APP produced, so the gevent handler can compare it against the bytes actually +# written to the socket and flag a discrepancy on the "wsgi access:" line. +APP_BYTES_ENVIRON_KEY = "nginxapimodule.app_bytes" +# When the app-produced bytes and the socket-sent bytes differ by MORE than this +# many bytes, the access line includes a short body_diff note (a 1-byte trailing +# newline difference is normal and must not be reported). +BODY_DIFF_THRESHOLD_BYTES = 1000 + + +def _new_request_id(environ: dict) -> str: + """Return a correlation id for the request. + + Prefers nginx's ``X-Request-ID`` (propagated via ``proxy_set_header``) so the + same id appears in the nginx access log, the ``wsgi request:`` line and the + ``wsgi access:`` line. Falls back to a fresh short uuid when nginx did not + forward one (e.g. direct upstream hit during tests). + """ + forwarded = environ.get(_header_env_key("X-Request-ID")) + return forwarded if forwarded else uuid.uuid4().hex[:12] + + +def _next_request_seq() -> int: + """Return the next monotonic per-request sequence number.""" + return next(_REQUEST_SEQ) + + +def _iso_now() -> str: + """Human-readable UTC timestamp for absolute received/responded times in logs.""" + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z" + + +# Request headers worth capturing for client/cache/conditional diagnostics. +# Shared by both the gevent access log (DemistoWSGIHandler) and the per-request +# RequestLoggingMiddleware so the two log lines stay consistent. +LOGGED_REQUEST_HEADERS = ( + "X-Forwarded-For", + "X-Real-IP", + "X-Forwarded-Proto", + "X-Original-URI", + "Host", + "User-Agent", + "Range", + "If-None-Match", + "If-Modified-Since", + "Authorization", +) + + +def _header_env_key(header_name: str) -> str: + # WSGI exposes request headers as HTTP_ in the environ. + return "HTTP_" + header_name.upper().replace("-", "_") + + +def format_request_headers(environ: dict) -> str: + """Render the allow-listed request headers from a WSGI ``environ`` into a log string. + + Produces a space-separated ``Name="value"`` sequence for every header in + ``LOGGED_REQUEST_HEADERS`` that is present. The ``Authorization`` header is + reduced to ``present`` so credentials are never written to the logs. + + Args: + environ (dict): The WSGI request environment. + + Returns: + str: e.g. ``X-Forwarded-For="1.2.3.4" Host="server" Authorization="present"``. + """ + header_parts = [] + for header_name in LOGGED_REQUEST_HEADERS: + value = environ.get(_header_env_key(header_name)) + if value is None: + continue + if header_name == "Authorization": + value = "present" + header_parts.append(f'{header_name}="{value}"') + return " ".join(header_parts) + + +class DemistoWSGIHandler(WSGIHandler): + """gevent WSGI handler that enriches the access line with transfer diagnostics + headers. + + gevent's ``WSGIServer`` formats one Common-Log-Format access line per request + via ``WSGIHandler.format_request()`` and writes it to the configured ``log`` + (our :class:`Handler`, which prefixes it with ``wsgi access:``). The default + line only reports the body bytes gevent sent, which - when compared with the + ``content_length`` the app declared in :class:`RequestLoggingMiddleware` - can + silently hide truncated/aborted transfers (client disconnects mid-stream). + + To make that gap explicit on a single line, ``format_request`` appends data + that only gevent's handler knows: + * ``sent_bytes`` - bytes actually written to the socket (``self.response_length``). + * ``content_length`` - the ``Content-Length`` the app declared (``-`` if chunked/unset). + * ``truncated`` - ``true`` when a numeric ``Content-Length`` was declared but + fewer bytes reached the socket (i.e. the transfer was cut short). + * ``connection`` - ``close`` or ``keep-alive``, derived from ``self.close_connection``. + It also appends the allow-listed request headers (via :func:`format_request_headers`) + so the real client/conditional headers are visible alongside the byte accounting. + """ + + @staticmethod + def _coerce_int(value: Any) -> int | None: + try: + return int(value) + except (TypeError, ValueError): + return None + + def _transfer_diagnostics(self, environ: dict) -> str: + # Bytes gevent actually wrote to the socket for the response body. + sent_bytes = getattr(self, "response_length", None) + sent_bytes_str = str(sent_bytes) if sent_bytes is not None else "-" + + # The Content-Length the app declared, if any (chunked responses won't have one). + declared_length: int | None = None + for header_key, header_value in getattr(self, "response_headers", None) or []: + if header_key.lower() == "content-length": + declared_length = self._coerce_int(header_value) + break + content_length_str = str(declared_length) if declared_length is not None else "-" + + # The number of body bytes the APP produced (counted by RequestLoggingMiddleware + # and stashed in environ), so we can compare the two independent counts. + app_bytes = self._coerce_int(environ.get(APP_BYTES_ENVIRON_KEY)) + app_bytes_str = str(app_bytes) if app_bytes is not None else "-" + + # truncated=true only when we can prove fewer bytes reached the socket than promised. + if declared_length is not None and isinstance(sent_bytes, int): + truncated = "true" if sent_bytes < declared_length else "false" + else: + truncated = "unknown" + + # Whether the connection is being closed (vs reused for keep-alive). + connection = "close" if getattr(self, "close_connection", True) else "keep-alive" + + # client_disconnected mirrors the truncated detection but phrased from the + # connection's point of view: a proven short write means the peer went away + # (or nginx/the firewall dropped) before the full body was sent. + client_disconnected = "true" if truncated == "true" else "false" + + # Number of requests served on this (keep-alive) connection, if gevent tracks it. + requests_on_conn = getattr(self, "_requests_on_connection", None) + requests_on_conn_str = str(requests_on_conn) if requests_on_conn is not None else "-" + + diagnostics = ( + f"sent_bytes={sent_bytes_str} app_bytes={app_bytes_str} content_length={content_length_str} " + f"truncated={truncated} client_disconnected={client_disconnected} " + f"connection={connection} requests_on_conn={requests_on_conn_str}" + ) + + # When the app produced N bytes but a meaningfully different number reached + # the socket, surface a short, human-readable explanation right on the line. + diff_note = self._body_diff_note(app_bytes, sent_bytes, declared_length) + if diff_note: + diagnostics = f"{diagnostics} {diff_note}" + return diagnostics + + @staticmethod + def _body_diff_note(app_bytes: int | None, sent_bytes: Any, declared_length: int | None) -> str: + """Return a short 'body_diff=...' note when app vs socket bytes diverge. + + Only emitted when both counts are known and differ by MORE than + ``BODY_DIFF_THRESHOLD_BYTES`` (a normal trailing-newline 1-byte delta is + ignored). The note names the likely cause so a reader instantly knows + whether the body was truncated on the wire or grew/shrank unexpectedly. + """ + if app_bytes is None or not isinstance(sent_bytes, int): + return "" + delta = sent_bytes - app_bytes + if abs(delta) <= BODY_DIFF_THRESHOLD_BYTES: + return "" + if delta < 0: + # Fewer bytes on the wire than the app produced -> cut short. + missing = -delta + pct = (missing / app_bytes * 100) if app_bytes else 0.0 + cause = ( + "client/proxy closed the connection before the full body was sent" + if (declared_length is not None and sent_bytes < declared_length) + else "write stopped before the app finished streaming" + ) + detail = ( + f'body_diff="app produced {app_bytes} body bytes but only {sent_bytes} ' + f"reached the socket; {missing} bytes ({pct:.1f}%) were not sent - " + f'likely {cause}"' + ) + else: + extra = delta + detail = ( + f'body_diff="socket sent {sent_bytes} bytes, {extra} MORE than the ' + f"{app_bytes} body bytes the app counted - extra bytes are likely " + f'response framing/headers or a double-write"' + ) + return detail + + def format_request(self): + environ = self.environ or {} + base_line = super().format_request() + rid = environ.get(REQUEST_ID_ENVIRON_KEY, "-") + parts = [base_line, f"rid={rid}", self._transfer_diagnostics(environ)] + headers_str = format_request_headers(environ) + if headers_str: + parts.append(headers_str) + return " ".join(parts) + + +class RequestLoggingMiddleware: + """WSGI middleware that emits a detailed, structured log line per request. + + This is the Python-side counterpart to the nginx ``edl_detailed`` access log. + Because nginx serves cache HITs without ever reaching this upstream, a request + that appears in the nginx log with ``cache=HIT`` but is absent here was served + entirely from cache - making the cache-vs-upstream distinction explicit. + + For every request it logs: the real client (X-Forwarded-For / X-Real-IP) as + forwarded by nginx, the original URI, conditional/range headers, user-agent, + the cache status nginx attached (X-Proxy-Cache), the response status, the + number of body bytes written, and the wall-clock time spent in the app. + """ + + def __init__(self, app): + self.app = app + + def __call__(self, environ, start_response): + # --- request received ------------------------------------------------ + # Wall-clock for absolute timestamps; perf_counter for accurate durations. + t_received = time.time() + received_iso = _iso_now() + start_perf = time.perf_counter() + + # Correlation id: reuse nginx's X-Request-ID when present, else generate. + # Stash it in environ so DemistoWSGIHandler prints the SAME id on the + # "wsgi access:" line, tying the two python log lines together. + rid = _new_request_id(environ) + environ[REQUEST_ID_ENVIRON_KEY] = rid + + method = environ.get("REQUEST_METHOD", "-") + path = environ.get("PATH_INFO", "-") + query = environ.get("QUERY_STRING", "") + full_path = f"{path}?{query}" if query else path + remote_addr = environ.get("REMOTE_ADDR", "-") + + # How long the request waited inside nginx (queueing / cache-lock / connect) + # BEFORE this app handler started. nginx forwards its forward-time as the + # X-Request-Start epoch header; the difference vs our receive time is the + # gap that explains why nginx's time_total_secs >> the app's elapsed time. + nginx_wait_str = "-" + request_start_header = environ.get(_header_env_key("X-Request-Start")) + if request_start_header: + try: + nginx_wait = t_received - float(request_start_header) + # Clamp tiny negative values from clock skew to 0. + nginx_wait_str = f"{max(0.0, nginx_wait):.3f}s" + except (TypeError, ValueError): + nginx_wait_str = "-" + + # Collect the interesting request headers (auth is reduced to presence only). + headers_str = format_request_headers(environ) + + # Monotonic sequence per request for correlation across log lines. + seq = _next_request_seq() + + # Emit a START line. A request that then hangs for 125s (or never finishes + # because the client vanished) is visible here when debug logging is on, + # even though its END line would only appear much later (or not at all). + # Kept at debug to reduce steady-state volume; the END line is authoritative. + demisto.debug( + f"wsgi request-start: rid={rid} seq={seq} received={received_iso} " + f"nginx_wait={nginx_wait_str} " + f'client={remote_addr} method={method} uri="{full_path}" {headers_str}' + ) + + # Capture the response status/headers via a wrapped start_response. + response_info: dict = {"status": "-", "cache": "-", "edl_size": "-", "content_length": "-"} + # When start_response is invoked = app produced its response headers + # (time-to-headers). Captured via closure so we can log it below. + timings: dict = {"t_headers": None, "t_first_byte": None} + + def logging_start_response(status, response_headers, exc_info=None): + if timings["t_headers"] is None: + timings["t_headers"] = time.perf_counter() + response_info["status"] = status.split(" ", 1)[0] if status else "-" + for header_key, header_value in response_headers: + lowered = header_key.lower() + if lowered == "x-proxy-cache": + response_info["cache"] = header_value + elif lowered == "x-edl-size": + response_info["edl_size"] = header_value + elif lowered == "content-length": + response_info["content_length"] = header_value + return start_response(status, response_headers, exc_info) + + bytes_sent = 0 + error_str = "-" + t_app_start = time.perf_counter() + try: + result = self.app(environ, logging_start_response) + # Count the bytes actually produced by the app so we can detect + # size-vs-bytes mismatches / truncated bodies on the Python side. + for chunk in result: + if chunk: + if timings["t_first_byte"] is None: + timings["t_first_byte"] = time.perf_counter() + bytes_sent += len(chunk) + # Continuously expose the app-produced byte count so the gevent + # handler (DemistoWSGIHandler) can compare it with the bytes that + # actually reached the socket - even if we are cut off mid-stream. + environ[APP_BYTES_ENVIRON_KEY] = bytes_sent + yield chunk + if hasattr(result, "close"): + result.close() + except Exception as exc: # noqa: BLE001 - we re-raise after logging + # A client disconnect / write error surfaces here; record it so the + # END line explains why a transfer stopped short. + error_str = type(exc).__name__ + demisto.debug(f"WSGI app raised {error_str} while streaming response; re-raising.\n{traceback.format_exc()}") + raise + finally: + end_perf = time.perf_counter() + responded_iso = _iso_now() + + # Phase breakdown so it's obvious WHERE the time went: + # time_to_headers : app entry -> start_response (the "thinking" time). + # ttfb : request received -> first body byte. + # stream_time : first byte -> last byte (the streaming cost). + # elapsed : total time inside the app. + elapsed = end_perf - start_perf + time_to_headers = (timings["t_headers"] - t_app_start) if timings["t_headers"] else None + ttfb = (timings["t_first_byte"] - start_perf) if timings["t_first_byte"] else None + stream_time = end_perf - timings["t_first_byte"] if timings["t_first_byte"] else None + + def _fmt(value): + return f"{value:.3f}s" if value is not None else "-" + + demisto.info( + f"wsgi request: rid={rid} seq={seq} " + f"received={received_iso} responded={responded_iso} " + f"client={remote_addr} method={method} uri=\"{full_path}\" " + f"status={response_info['status']} " + f"app_bytes={bytes_sent} content_length={response_info['content_length']} " + f'cache="{response_info["cache"]}" edl_size={response_info["edl_size"]} ' + f"nginx_wait={nginx_wait_str} time_to_headers={_fmt(time_to_headers)} ttfb={_fmt(ttfb)} " + f"stream_time={_fmt(stream_time)} elapsed={_fmt(elapsed)} " + f"error={error_str} {headers_str}" + ) + + # nginx server params NGINX_SERVER_ACCESS_LOG = "/var/log/nginx/access.log" NGINX_SERVER_ERROR_LOG = "/var/log/nginx/error.log" @@ -44,32 +399,158 @@ def write(msg: str): ssl_certificate {NGINX_SSL_CRT_FILE}; ssl_certificate_key {NGINX_SSL_KEY_FILE}; """ +# Detailed access log format with self-explanatory key names, grouped so the line +# reads top-to-bottom like the life of one request. Fields are ordered: +# +# 1. IDENTITY (first, so every line starts with "who/when/which request"): +# when_finished - ISO8601 timestamp of when nginx FINISHED the request +# and wrote this log line (i.e. request end). NOTE: stock +# nginx evaluates ALL log variables at write-time, so there +# is no built-in variable for the absolute arrival time. +# Derive it from the durations below: +# arrival = when_finished - time_total_secs +# sent_upstream = arrival + time_to_upstream_connect_secs +# request_id - unique id for THIS request; the SAME id is sent to +# the Python upstream (X-Request-ID) and printed on the +# "wsgi request:"/"wsgi access:" lines, so one request +# can be followed across nginx and Python. +# connection_id - id of the TCP connection (many requests can share one). +# requests_on_connection- how many requests have used this keep-alive connection +# (rising numbers = reuse; helps spot CLOSE_WAIT buildup). +# +# 2. WHO CONNECTED (client identity through the proxy chain): +# client_real_ip - the true client (the firewall), via X-Real-IP. +# client_forwarded_chain- full X-Forwarded-For chain of proxies in between. +# client_nearest_peer - the immediate TCP peer nginx saw (usually localhost/edge). +# client_user_agent - the client's User-Agent string. +# +# 3. WHAT WAS ASKED (the request itself): +# request_method - GET/HEAD/... +# request_uri - the full requested URI. +# request_host - the Host header. +# request_range - Range header (partial-content requests). +# request_if_none_match - ETag the client already has (conditional GET). +# request_if_modified_since - date the client already has (conditional GET). + +# +# 4. WHAT HAPPENED (response outcome + cache decision): +# response_status - HTTP status code returned. +# cache_status - HIT/MISS/BYPASS/EXPIRED/STALE/UPDATING/REVALIDATED. +# upstream_address - which upstream served it (empty on a cache HIT). +# response_etag - ETag we returned. +# +# 5. HOW BIG (payload accounting; mismatch = truncated/aborted transfer): +# response_body_bytes - body bytes sent to the client. +# response_total_bytes - total bytes sent (headers + body). +# edl_indicator_count - number of indicators in the EDL response. +# edl_origin_count - number of origin indicators before filtering. +# +# 6. HOW LONG (ALL timings grouped together at the end, in seconds). These are the +# source of truth for the timeline - use them to reconstruct WHEN nginx received +# the request from the client and WHEN it forwarded it to the upstream: +# time_total_secs - total time to serve the request (from when nginx +# read the first client bytes until the last byte was +# sent to the client). The client request therefore +# ARRIVED at: when_finished - time_total_secs. +# time_to_upstream_connect_secs - seconds AFTER arrival that nginx established the +# upstream connection. This marks WHEN nginx forwarded +# the request to the upstream: +# sent_to_upstream = (when_finished - time_total_secs) +# + time_to_upstream_connect_secs +# ("-" on a cache HIT, because no upstream request was made.) +# time_upstream_headers_secs - seconds after the upstream request until the upstream +# returned its response headers. +# time_upstream_response_secs- seconds after the upstream request until the upstream +# finished sending its response. +# time_edl_query_secs - time the EDL spent building the list (app side). +NGINX_LOG_FORMAT = """ +log_format edl_detailed + 'when_finished=$time_iso8601 request_id=$request_id ' + 'connection_id=$connection requests_on_connection=$connection_requests ' + 'client_real_ip="$http_x_real_ip" client_forwarded_chain="$http_x_forwarded_for" ' + 'client_nearest_peer=$remote_addr client_user_agent="$http_user_agent" ' + 'request_method=$request_method request_uri="$request_uri" request_host="$host" ' + 'request_range="$http_range" request_if_none_match="$http_if_none_match" ' + 'request_if_modified_since="$http_if_modified_since" ' + 'response_status=$status cache_status=$upstream_cache_status upstream_address="$upstream_addr" ' + 'response_etag="$sent_http_etag" ' + 'response_body_bytes=$body_bytes_sent response_total_bytes=$bytes_sent ' + 'edl_indicator_count="$sent_http_x_edl_size" edl_origin_count="$sent_http_x_edl_origin_size" ' + 'time_total_secs=$request_time time_to_upstream_connect_secs="$upstream_connect_time" ' + 'time_upstream_headers_secs="$upstream_header_time" time_upstream_response_secs="$upstream_response_time" ' + 'time_edl_query_secs="$sent_http_x_edl_query_time_secs"'; +""" NGINX_SERVER_CONF = """ +$log_format + +# Per-URI concurrency zone used to FAIL (not queue) concurrent cold-MISS requests +# for the same URI. Keyed on $request_uri so the limit is per-resource, not per-client. +limit_conn_zone $request_uri zone=concurrent_conn_zone:1m; server { listen $port default_server $ssl; $sslcerts + # Per-request detailed access log + verbose error log so cache decisions, + # real client IPs, timings and upstream warnings are all captured. + access_log $access_log_path edl_detailed; + error_log $error_log_path info; + + proxy_cache_key $scheme$proxy_host$request_uri$extra_cache_key; $proxy_set_range_header $extra_headers -# Thundering-herd protection -proxy_cache_lock on; -proxy_cache_lock_timeout $cache_lock_timeout; -proxy_cache_lock_age $cache_lock_age; +# Cache-vs-fetch policy (TWO-TIER, HIT-safe fail-fast on cold MISS) +# --------------------------------------------------------------------------- +# We want two behaviors that plain `proxy_cache_lock` cannot give together: +# * On a TRUE cold MISS (nothing cached to fall back on), concurrent requests +# for the SAME URI must FAIL FAST (429) instead of queueing behind a lock. +# * On STALE / UPDATING (a cached copy exists but is expired/being refreshed) +# clients must be served the STALE copy and must NEVER be rejected. +# +# `limit_conn` is the only mechanism that REJECTS (proxy_cache_lock only WAITS), +# but it runs in the preaccess phase - before the cache status is known - so +# applying it on the public cache location would ALSO count cache HITs and could +# 429 two simultaneous HITs of the same URI. nginx's `proxy_cache` cannot route +# "only on miss" to a different location within one server, so we use TWO server +# blocks: +# Tier 1 = public server on $port : does the cache read. HIT / STALE / UPDATING +# are answered here from the cache (see proxy_cache_use_stale + +# proxy_cache_background_update) and, on a MISS, proxy_pass to Tier 2. +# HIT/STALE/UPDATING never leave Tier 1, so they are never counted and +# never rejected. +# Tier 2 = internal server on localhost:$fetchport : reached ONLY when Tier 1's +# cache must populate a new entry (true MISS / expired-with-no-stale). +# It carries `limit_conn ... 1`, so the first fetch for a URI proceeds +# and every concurrent same-URI fetch is rejected immediately with 429. # Cache validity by status proxy_cache_valid 200 301 302 $cache_refresh_rate; # Optional: cache other responses briefly (helps absorb spikes) proxy_cache_valid 404 $cache_404_ttl; +# NEVER cache the fail-fast rejection: a 429 from the Tier-2 limiter is a +# transient "someone else is already building this" signal. Caching it (via the +# `any` rule below) would poison the URI and serve 429s even after the real +# content is ready. `0s` = do not cache; being status-specific it overrides `any`. +proxy_cache_valid 429 0s; +# NEVER cache upstream errors / timeouts either. A 504 means the build exceeded +# proxy_read_timeout and a 5xx is a transient upstream failure - caching them via +# the `any` rule below would poison the URI and keep serving the error (as a +# cache HIT) even after a later build would succeed, blocking recovery. `0s` = do +# not cache; being status-specific these override `any`. Note: this does NOT stop +# `proxy_cache_use_stale timeout http_50x` from serving a previously-cached GOOD +# copy on timeout - that stale-serving is desirable and is what we keep. +proxy_cache_valid 500 502 503 504 0s; proxy_cache_valid any $cache_default_ttl; # Revalidation (use conditional requests when expired) proxy_cache_revalidate on; -# Serve stale content in failure/update scenarios +# Serve stale content in failure/update scenarios. `updating` is what lets a +# STALE entry be served immediately to every waiting client while a single +# background refresh runs - so STALE/UPDATING never reach the Tier-2 limiter. proxy_cache_use_stale updating error @@ -80,7 +561,8 @@ def write(msg: str): http_503 http_504; -# Background refresh of expired cache +# Background refresh of expired cache: the refresh runs as a detached subrequest, +# so serving stale is always allowed and never fails. proxy_cache_background_update on; # Static test file @@ -89,10 +571,26 @@ def write(msg: str): default_type text/html; } - # Proxy everything to python + # ---- Tier 1: public cache front -------------------------------------- + # Serves HIT / STALE / UPDATING from the cache. On a MISS, the cache module + # fetches from Tier 2 (the internal fetch server) where the fail-fast limiter + # lives. HITs/STALE/UPDATING are served straight from cache and never reach + # Tier 2, so they are never counted by limit_conn and never rejected. location / { - proxy_pass http://localhost:$serverport/; - add_header X-Proxy-Cache $upstream_cache_status; + proxy_pass http://localhost:$fetchport/; + + # CRITICAL: the base flask-nginx image ENABLES the cache lock in the + # http{} block, and that setting is inherited here. While enabled, + # concurrent cold-MISS requests for the same URI WAIT for the first + # request to populate the cache (then get served HIT) - they never fall + # through to Tier 2, so the fail-fast limit_conn never fires. We MUST turn + # the inherited lock OFF here so misses proceed to Tier 2 and the 2nd+ + # concurrent miss is rejected with 429 instead of queued. + proxy_cache_lock off; + + # Surface the cache decision both to the client and (via the access log + # variable) to our logging: HIT/MISS/BYPASS/EXPIRED/STALE/UPDATING/REVALIDATED. + add_header X-Proxy-Cache $upstream_cache_status always; $extra_headers # allow bypassing the cache with an arg of nocache=1 ie http://server:7000/?nocache=1 proxy_cache_bypass $arg_nocache; @@ -100,6 +598,65 @@ def write(msg: str): proxy_connect_timeout 3600; proxy_send_timeout 3600; send_timeout 3600; + + # Forward the real client identity through the chain so the Python (gevent) + # upstream's WSGI middleware can log the actual firewall/client instead of + # 127.0.0.1. Tier 2 forwards these on to the app. + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Original-URI $request_uri; + # Propagate nginx's per-request id to the upstream so the same request_id + # appears in the nginx access log AND the wsgi request/access lines, + # giving a single correlation id end-to-end. + proxy_set_header X-Request-ID $request_id; + # Forward, as an epoch (seconds.ms), the moment nginx is about to hand the + # request to the upstream. The WSGI middleware compares this with its own + # receive time and logs "nginx_wait" - the time the request spent inside + # nginx (queueing / cache-lock / connect) BEFORE the app started working. + proxy_set_header X-Request-Start $msec; + } + +} + +# ---- Tier 2: internal fetch server (cold-MISS path only) ----------------- +# Reached ONLY via Tier 1's cache fetch on a MISS. Because Tier 1 answers +# HIT/STALE/UPDATING from cache, only requests that actually need the upstream +# arrive here, so `limit_conn ... 1` counts ONLY cold-miss fetches: the first +# request for a URI builds the cache entry, and every concurrent same-URI fetch +# is rejected immediately with 429. Listens on loopback only, so it is never +# reachable directly by external clients. +server { + listen localhost:$fetchport; + + access_log $access_log_path edl_detailed; + error_log $error_log_path info; + + location / { + limit_conn concurrent_conn_zone 1; + limit_conn_status 429; + + # Tier 2 must NOT cache: caching is owned entirely by Tier 1 (the public + # server). If proxy_cache is inherited from the http{} block, disable it + # here so this tier is purely the rate-limited cold-MISS fetch path. + proxy_cache off; + + proxy_pass http://localhost:$serverport/; + + # Preserve the forwarded client identity headers set by Tier 1. + proxy_set_header Host $host; + proxy_set_header X-Real-IP $http_x_real_ip; + proxy_set_header X-Forwarded-For $http_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto; + proxy_set_header X-Original-URI $http_x_original_uri; + proxy_set_header X-Request-ID $http_x_request_id; + proxy_set_header X-Request-Start $http_x_request_start; + + proxy_read_timeout $timeout; + proxy_connect_timeout 3600; + proxy_send_timeout 3600; + send_timeout 3600; } } @@ -127,28 +684,21 @@ def create_nginx_server_conf(file_path: str, port: int, params: dict): timeout = _normalize_nginx_time(params.get("timeout"), default="3600", param_name="timeout") cache_refresh_rate = _normalize_nginx_time(params.get("cache_refresh_rate"), default=timeout, param_name="cache_refresh_rate") - # Ensure cache lock directives are at least as large as the upstream timeout. Otherwise, when an - # upstream request takes longer than the lock timeout/age, waiting clients bypass the cache lock - # and stampede the upstream (each waiter then produces an uncached response), defeating the purpose - # of `proxy_cache_lock on`. Defaults match `timeout`; explicit smaller values are bumped up. - cache_lock_timeout = _normalize_nginx_time(params.get("cache_lock_timeout"), default=timeout, param_name="cache_lock_timeout") - cache_lock_age = _normalize_nginx_time(params.get("cache_lock_age"), default=timeout, param_name="cache_lock_age") cache_404_ttl = _normalize_nginx_time(params.get("cache_404_ttl"), default="1m", param_name="cache_404_ttl") cache_default_ttl = _normalize_nginx_time(params.get("cache_default_ttl"), default="1m", param_name="cache_default_ttl") - # Ensure cache_refresh_rate is at least as large as timeout, and apply the same anti-stampede - # floor to the cache lock directives. All values are now guaranteed to end in "s" (the helper - # always returns `s`), so an O(1) integer compare on the prefix is safe. + # Ensure cache_refresh_rate is at least as large as timeout. All values are now guaranteed to + # end in "s" (the helper always returns `s`), so an O(1) integer compare on the prefix is safe. timeout_seconds = int(timeout[:-1]) if int(cache_refresh_rate[:-1]) < timeout_seconds: cache_refresh_rate = timeout - if int(cache_lock_timeout[:-1]) < timeout_seconds: - cache_lock_timeout = timeout - if int(cache_lock_age[:-1]) < timeout_seconds: - cache_lock_age = timeout ssl, extra_headers, sslcerts, proxy_set_range_header = "", "", "", "" serverport = port + 1 + # Internal loopback port for the Tier-2 cold-MISS fetch server. Tier 1 (public, + # $port) proxies cache misses to localhost:$fetchport, which applies the + # fail-fast `limit_conn` and then proxies on to the gevent app on $serverport. + fetchport = serverport + 1 extra_cache_keys = [] if (certificate and not private_key) or (private_key and not certificate): raise DemistoException("If using HTTPS connection, both certificate and private key should be provided.") @@ -173,19 +723,31 @@ def create_nginx_server_conf(file_path: str, port: int, params: dict): extra_cache_keys_str = "".join(extra_cache_keys) server_conf = Template(template_str).safe_substitute( + log_format=NGINX_LOG_FORMAT, port=port, serverport=serverport, + fetchport=fetchport, ssl=ssl, sslcerts=sslcerts, extra_cache_key=extra_cache_keys_str, proxy_set_range_header=proxy_set_range_header, timeout=timeout, cache_refresh_rate=cache_refresh_rate, - cache_lock_timeout=cache_lock_timeout, - cache_lock_age=cache_lock_age, cache_404_ttl=cache_404_ttl, cache_default_ttl=cache_default_ttl, extra_headers=extra_headers, + access_log_path=NGINX_SERVER_ACCESS_LOG, + error_log_path=NGINX_SERVER_ERROR_LOG, + ) + # Log the effective cache / timeout settings so each (re)start records exactly + # which values are active - essential for interpreting cache=HIT/STALE/UPDATING + # decisions and the upstream timing fields in the access logs. Kept at debug. + demisto.debug( + "edl: nginx effective settings -> " + f"listen_port={port} upstream_port={serverport} fetch_tier_port={fetchport} ssl={'on' if ssl else 'off'} " + f"timeout={timeout} cache_refresh_rate={cache_refresh_rate} " + f"cache_404_ttl={cache_404_ttl} cache_default_ttl={cache_default_ttl} " + f"extra_cache_keys=[{extra_cache_keys_str}]" ) with open(file_path, mode="w+") as f: f.write(server_conf) @@ -205,7 +767,9 @@ def start_nginx_server(port: int, params: dict = {}) -> subprocess.Popen: nginx_test_command.extend(directive_args) test_output = subprocess.check_output(nginx_test_command, stderr=subprocess.STDOUT, text=True) demisto.info(f"ngnix test passed. command: [{nginx_test_command}]") - demisto.debug(f"nginx test ouput:\n{test_output}") + # Promote the fully rendered config to info so the active log_format, cache, + # timeout and proxy_set_header directives are recorded on every (re)start. + demisto.info(f"nginx effective rendered config (nginx -T):\n{test_output}") except subprocess.CalledProcessError as err: raise ValueError(f"Failed testing nginx conf. Return code: {err.returncode}. Output: {err.output}") nginx_command = ["nginx"] @@ -251,7 +815,7 @@ def nginx_log_process(nginx_process: subprocess.Popen): start = 1 for lines in batch(f.readlines(), 100): end = start + len(lines) - demisto.info(f"nginx access log ({start}-{end-1}): " + "".join(lines)) + demisto.debug(f"nginx access log ({start}-{end-1}): " + "".join(lines)) start = end Path(old_access).unlink() if log_error: @@ -522,27 +1086,32 @@ def run_long_running(params: dict = None, is_test: bool = False): log_handler.setFormatter(logging.Formatter("flask log: [%(asctime)s] %(levelname)s in %(module)s: %(message)s")) APP.logger.addHandler(log_handler) # type: ignore[name-defined] # pylint: disable=E0602 demisto.debug("done setting demisto handler for logging") - server = WSGIServer( - ("0.0.0.0", server_port), - APP, # type: ignore[name-defined] # pylint: disable=E0602 - log=DEMISTO_LOGGER, # type: ignore[name-defined] # pylint: disable=E0602 - error_log=ERROR_LOGGER, - ) + demisto.info(f"edl: starting server on 0.0.0.0:{server_port}; nginx proxy on port {nginx_port}.") + if is_test: test_nginx_server(nginx_port, params) - server_process = Process(target=server.serve_forever) - server_process.start() + server = WSGIServer( + ("0.0.0.0", server_port), + APP, # type: ignore[name-defined] # pylint: disable=E0602 + log=DEMISTO_LOGGER, + error_log=ERROR_LOGGER, + ) + server.start() time.sleep(5) - try: - server_process.terminate() - server_process.join(1.0) - except Exception as ex: - demisto.error(f"failed stopping test wsgi server process: {ex}") + server.stop() else: nginx_process = start_nginx_server(nginx_port, params) test_nginx_web_server(nginx_port, params) nginx_log_monitor = gevent.spawn(nginx_log_monitor_loop, nginx_process) + wsgi_app = RequestLoggingMiddleware(APP) # type: ignore[name-defined] # pylint: disable=E0602 + server = WSGIServer( + ("0.0.0.0", server_port), + wsgi_app, + log=DEMISTO_LOGGER, # type: ignore[name-defined] # pylint: disable=E0602 + error_log=ERROR_LOGGER, + handler_class=DemistoWSGIHandler, + ) demisto.updateModuleHealth("") server.serve_forever() except Exception as e: diff --git a/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule_test.py b/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule_test.py index 342f389963f..5b3afbb382e 100644 --- a/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule_test.py +++ b/Packs/ApiModules/Scripts/NGINXApiModule/NGINXApiModule_test.py @@ -309,12 +309,20 @@ def test_nginx_log_process(nginx_cleanup, mocker: MockerFixture): sleep(0.5) # give nginx time to start # create a request to get a log line requests.get("http://localhost:12345/nginx-test?unit_testing") - sleep(0.2) - mocker.patch.object(demisto, "info") + # Poll until nginx has flushed the access-log line to the (test-redirected) + # log before invoking nginx_log_process. A fixed short sleep is racy: if the + # line isn't on disk yet, nginx_log_process sees an empty log, logs nothing, + # and demisto.debug.call_args is None. + for _ in range(50): + if Path(module.NGINX_SERVER_ACCESS_LOG).exists() and Path(module.NGINX_SERVER_ACCESS_LOG).stat().st_size: + break + sleep(0.1) + mocker.patch.object(demisto, "debug") mocker.patch.object(demisto, "error") module.nginx_log_process(NGINX_PROCESS) + # The nginx access log line is emitted via demisto.debug. # call_args is tuple (args list, kwargs). we only need the args - arg = demisto.info.call_args[0][0] + arg = demisto.debug.call_args[0][0] assert "nginx access log" in arg assert "unit_testing" in arg # make sure old file was removed @@ -559,12 +567,16 @@ def test_normalize_nginx_time_invalid_raises(value): def test_create_nginx_server_conf_renders_seconds_for_all_five_params(tmp_path: Path, mocker): """ Given: a params dict that mixes nginx-native, human-readable and bare-integer - forms across all five cache_* params (plus a "timeout" of 3600s that - should floor-bump the lock + refresh values). + forms across the cache_* time params (plus a "timeout" of 3600s that + should floor-bump cache_refresh_rate). When: create_nginx_server_conf renders the nginx server config. - Then: every rendered cache directive is in the safe "s" form, and no + Then: every rendered cache-time directive is in the safe "s" form, and no occurrence of any human-readable unit word or original input token survives into the rendered conf. + + Note: the two-tier design NO LONGER emits proxy_cache_lock_timeout / + proxy_cache_lock_age (concurrency is controlled by Tier-2 limit_conn, + not by a cache lock), so those directives must be ABSENT. """ from NGINXApiModule import create_nginx_server_conf @@ -576,8 +588,6 @@ def test_create_nginx_server_conf_renders_seconds_for_all_five_params(tmp_path: params={ "timeout": "3600", "cache_refresh_rate": "12 hours", - "cache_lock_timeout": "1h", - "cache_lock_age": "30m", "cache_404_ttl": "5 minutes", "cache_default_ttl": "300", }, @@ -587,22 +597,110 @@ def test_create_nginx_server_conf_renders_seconds_for_all_five_params(tmp_path: # cache_refresh_rate = "12 hours" -> 43200s (>= timeout, not floor-bumped) assert "proxy_cache_valid 200 301 302 43200s;" in conf - # cache_lock_timeout = "1h" = 3600s == timeout (not bumped, already at floor) - assert "proxy_cache_lock_timeout 3600s;" in conf - # cache_lock_age = "30m" = 1800s < timeout (3600s) -> floor-bumped to "3600s" - assert "proxy_cache_lock_age 3600s;" in conf # cache_404_ttl = "5 minutes" -> 300s assert "proxy_cache_valid 404 300s;" in conf # cache_default_ttl = "300" -> 300s assert "proxy_cache_valid any 300s;" in conf + # The cache-lock tuning directives are no longer part of the two-tier design. + assert "proxy_cache_lock_timeout" not in conf + assert "proxy_cache_lock_age" not in conf + # Hardening: none of the original unit words / non-normalized tokens may # survive into the rendered conf — those are exactly the failure surfaces # that crashed `nginx -T` in the documented incident. - for forbidden in ("hours", "minutes", "12h", "30m", "5 minutes"): + for forbidden in ("hours", "minutes", "12h", "5 minutes"): assert forbidden not in conf, f"Rendered conf must not contain non-normalized substring {forbidden!r}: {conf}" +def test_create_nginx_server_conf_renders_two_tier_fail_fast(tmp_path: Path, mocker): + """ + Given: a default params dict. + When: create_nginx_server_conf renders the nginx server config. + Then: the rendered conf implements the TWO-TIER fail-fast design: + * a per-URI limit_conn zone keyed on $request_uri, + * Tier 2 rejects concurrent same-URI cold-miss fetches (limit_conn 1 + + limit_conn_status 429) and does NOT cache, + * Tier 1 turns OFF the inherited cache lock so misses fall through to + Tier 2 instead of queueing. + """ + from NGINXApiModule import create_nginx_server_conf + + mocker.patch.object(demisto, "callingContext", return_value={"context": {}}) + conf_file = str(tmp_path / "nginx-test-server.conf") + create_nginx_server_conf(conf_file, 12345, params={}) + with open(conf_file) as f: + conf = f.read() + + # Per-URI concurrency zone (keyed on the resource, not the client). + assert "limit_conn_zone $request_uri zone=concurrent_conn_zone:1m;" in conf + # Tier 2 fail-fast: one build per URI, extras rejected with 429, no caching. + assert "limit_conn concurrent_conn_zone 1;" in conf + assert "limit_conn_status 429;" in conf + assert "proxy_cache off;" in conf + # Tier 1 must DISABLE the inherited cache lock so cold misses reach Tier 2 + # (the fail-fast limiter) instead of queueing behind proxy_cache_lock. + assert "proxy_cache_lock off;" in conf + # The queueing lock must NOT be enabled anywhere as a directive. + assert "proxy_cache_lock on;" not in conf + + +def test_create_nginx_server_conf_never_caches_transient_statuses(tmp_path: Path, mocker): + """ + Given: a default params dict. + When: create_nginx_server_conf renders the nginx server config. + Then: transient responses are excluded from caching so a slow/failed build + cannot poison the URI: the fail-fast 429 and the upstream 5xx/504 + family both get "0s" validity (overriding the `any` catch-all), while + proxy_cache_use_stale + background_update still allow serving a GOOD + stale copy on timeout. + """ + from NGINXApiModule import create_nginx_server_conf + + mocker.patch.object(demisto, "callingContext", return_value={"context": {}}) + conf_file = str(tmp_path / "nginx-test-server.conf") + create_nginx_server_conf(conf_file, 12345, params={}) + with open(conf_file) as f: + conf = f.read() + + # Never cache the fail-fast rejection or upstream errors/timeouts. + assert "proxy_cache_valid 429 0s;" in conf + assert "proxy_cache_valid 500 502 503 504 0s;" in conf + # Stale-serving of a previously-cached GOOD copy is still enabled. + assert "proxy_cache_use_stale" in conf + assert "proxy_cache_background_update on;" in conf + + +def test_create_nginx_server_conf_two_server_blocks_and_ports(tmp_path: Path, mocker): + """ + Given: a listening port of 12345. + When: create_nginx_server_conf renders the nginx server config. + Then: two server blocks are emitted — Tier 1 public on the given port, and + Tier 2 internal on loopback at fetchport (= serverport + 1 = port + 2) + — and Tier 2 proxies on to the gevent app at serverport (= port + 1). + """ + from NGINXApiModule import create_nginx_server_conf + + mocker.patch.object(demisto, "callingContext", return_value={"context": {}}) + conf_file = str(tmp_path / "nginx-test-server.conf") + port = 12345 + create_nginx_server_conf(conf_file, port, params={}) + with open(conf_file) as f: + conf = f.read() + + serverport = port + 1 # gevent app + fetchport = serverport + 1 # Tier-2 internal fetch server + + # Exactly two `server {` blocks (Tier 1 + Tier 2). + assert conf.count("server {") == 2 + # Tier 1 listens on the public port; Tier 2 on loopback:fetchport. + assert f"listen {port} default_server" in conf + assert f"listen localhost:{fetchport};" in conf + # Tier 1 proxies misses to Tier 2; Tier 2 proxies to the gevent app. + assert f"proxy_pass http://localhost:{fetchport}/;" in conf + assert f"proxy_pass http://localhost:{serverport}/;" in conf + + def test_create_nginx_server_conf_rejects_invalid_param(tmp_path: Path, mocker, monkeypatch): """ Given: a params dict with an unparseable value for cache_404_ttl. diff --git a/Packs/ApiModules/pack_metadata.json b/Packs/ApiModules/pack_metadata.json index b8dafe5495c..e2ce09bcfe8 100644 --- a/Packs/ApiModules/pack_metadata.json +++ b/Packs/ApiModules/pack_metadata.json @@ -2,7 +2,7 @@ "name": "ApiModules", "description": "API Modules", "support": "xsoar", - "currentVersion": "2.4.35", + "currentVersion": "2.4.36", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/EDL/Integrations/EDL/EDL.py b/Packs/EDL/Integrations/EDL/EDL.py index b04ca61ef1f..a29980debcb 100644 --- a/Packs/EDL/Integrations/EDL/EDL.py +++ b/Packs/EDL/Integrations/EDL/EDL.py @@ -4,6 +4,7 @@ import os import re import tempfile +import uuid import zipfile from base64 import b64decode from collections.abc import Callable, Iterable @@ -370,7 +371,22 @@ def get_indicators_to_format(indicator_searcher: IndicatorsSearcher, request_arg headers_was_written = False files_by_category = {} # type:Dict ioc_counter = 0 + # While iterating over a large number of indicators, the main thread can hold the stdout lock longer than + # the default 60s timeout, causing the heartbeat thread to fail with "Timeout acquiring stdout lock". + # Instead of disabling the heartbeat thread (which can make the server consider the container unresponsive), + # we temporarily raise the stdout lock timeout to 10 minutes so the heartbeat keeps running safely, and + # restore the original value afterwards. This attribute is dynamically injected by the server runtime; when + # it is not present (e.g. older servers or unit tests) we skip the tweak entirely so this stays a true no-op. + # It is not part of the demistomock type stub, so mypy's attr-defined check is suppressed on the assignments. + stdout_lock_timeout_supported = hasattr(demisto, "_stdout_lock_timeout") + original_stdout_lock_timeout = getattr(demisto, "_stdout_lock_timeout", 60) try: + if stdout_lock_timeout_supported: + demisto._stdout_lock_timeout = 600 # type: ignore[attr-defined] # 10 minutes + demisto.debug( + f"Temporarily set demisto._stdout_lock_timeout to 600 seconds " + f"(was {original_stdout_lock_timeout}) for the indicators iteration." + ) for ioc_res in indicator_searcher: fetched_iocs = ioc_res.get("iocs") or [] for ioc in fetched_iocs: @@ -407,6 +423,10 @@ def get_indicators_to_format(indicator_searcher: IndicatorsSearcher, request_arg # NG + XSIAM can recover from a shutdown if version.get("platform") == "x2" or is_demisto_version_ge("8") or version.get("platform") == "unified_platform": raise SystemExit("Encountered issue in Elastic Search query. Restarting container and trying again.") + finally: + if stdout_lock_timeout_supported: + demisto._stdout_lock_timeout = original_stdout_lock_timeout # type: ignore[attr-defined] + demisto.debug(f"Restored demisto._stdout_lock_timeout to {original_stdout_lock_timeout} seconds.") demisto.debug(f"Completed IOC search & format, found {ioc_counter} IOCs.") if request_args.out_format == FORMAT_JSON: f.write("]") @@ -1168,24 +1188,43 @@ def prepare_response_data(data: str, prepend_str: str, append_str: str) -> str: return data +def get_request_id() -> str: + """Return the per-request correlation id for log lines. + + Reuses the ``X-Request-ID`` header that nginx forwards (the SAME id appears in + the nginx access log and the WSGI ``wsgi request:``/``wsgi access:`` lines), so + a single id can be grepped across NGINX -> WSGI -> EDL. Falls back to a fresh + short uuid if the header is missing (e.g. a direct hit that bypassed nginx). + """ + try: + forwarded = request.headers.get("X-Request-ID") + except RuntimeError: + # No active Flask request context (e.g. called outside a route). + demisto.debug("get_request_id called without an active Flask request context; generating a fresh id.") + forwarded = None + return forwarded if forwarded else uuid.uuid4().hex[:12] + + @APP.route("/", methods=["GET"]) def route_edl() -> Response: """ Main handler for values saved in the integration context """ params = demisto.params() + rid = get_request_id() cache_refresh_rate: str = params.get("cache_refresh_rate") - if EXTENSIVE_LOGGING: - demisto.debug("edl: Starting EDL route handler") + start = datetime.now(timezone.utc) + demisto.info(f"edl: rid={rid} route=/ start handling request") auth_resp = authenticate_app(params, request.headers) if auth_resp: + demisto.info(f"edl: rid={rid} authentication failed; returning auth response") return auth_resp if EXTENSIVE_LOGGING: - demisto.debug("edl: authentication successful") + demisto.debug(f"edl: rid={rid} authentication successful") request_args = get_request_args(request.args, params) on_demand = params.get("on_demand") if EXTENSIVE_LOGGING: - demisto.debug(f"{'Using' if on_demand else 'Not using'} on-demand cache to serve EDL.") + demisto.debug(f"edl: rid={rid} {'Using' if on_demand else 'Not using'} on-demand cache to serve EDL.") created = datetime.now(timezone.utc) if on_demand: @@ -1210,7 +1249,7 @@ def route_edl() -> Response: data=edl_data, append_str=params.get("append_string"), prepend_str=params.get("prepend_string") ) if EXTENSIVE_LOGGING: - demisto.debug(f"Final EDL size: {len(edl_data)} characters, {edl_size} lines") + demisto.debug(f"edl: rid={rid} Final EDL size: {len(edl_data)} characters, {edl_size} lines") mimetype = get_outbound_mimetype(request_args) max_age = ceil((datetime.now() - dateparser.parse(cache_refresh_rate)).total_seconds()) # type: ignore[operator] @@ -1221,15 +1260,27 @@ def route_edl() -> Response: ("X-EDL-Size", str(edl_size)), ("X-EDL-Origin-Size", original_indicators_count), ("ETag", etag), + # Echo the correlation id back so the client and nginx ($sent_http_x_request_id) + # observe the same id that EDL/WSGI logged. + ("X-Request-ID", rid), ] # type: ignore[assignment] - demisto.debug(f'edl: Returning response with the following headers:\n{[f"{header[0]}: {header[1]}" for header in headers]}') + demisto.debug( + f'edl: rid={rid} Returning response with the following headers:\n' + f'{[f"{header[0]}: {header[1]}" for header in headers]}' + ) resp = Response(edl_data, status=200, mimetype=mimetype, headers=headers) resp.cache_control.max_age = max_age # number of seconds we are willing to serve stale content when there is an error resp.cache_control["stale-if-error"] = "600" + total_time = (datetime.now(timezone.utc) - start).total_seconds() + demisto.info( + f"edl: rid={rid} route=/ done status=200 edl_size={edl_size} " + f"chars={len(edl_data)} query_time_secs={query_time:.3f} total_secs={total_time:.3f}" + ) + return resp @@ -1242,14 +1293,15 @@ def log_download() -> Response: Response: A Flask Response object that sends a ZIP file containing the full log. """ params = demisto.params() - demisto.debug("edl: Starting EDL route/log_download handler") + rid = get_request_id() + demisto.debug(f"edl: rid={rid} Starting EDL route/log_download handler") auth_resp = authenticate_app(params, request.headers) if EXTENSIVE_LOGGING: - demisto.debug("edl: authentication successful") + demisto.debug(f"edl: rid={rid} authentication successful") if auth_resp: return auth_resp - demisto.debug("edl: Getting log file to show") + demisto.debug(f"edl: rid={rid} Getting log file to show") created = datetime.now(timezone.utc) @@ -1273,12 +1325,13 @@ def route_edl_log() -> Response: - A ZIP file download containing the log, if the log is too large to display. """ params = demisto.params() + rid = get_request_id() cache_refresh_rate: str = params.get("cache_refresh_rate") - demisto.debug("edl: Starting EDL route/log handler") + demisto.debug(f"edl: rid={rid} Starting EDL route/log handler") auth_resp = authenticate_app(params, request.headers) if EXTENSIVE_LOGGING: - demisto.debug("edl: authentication successful") + demisto.debug(f"edl: rid={rid} authentication successful") if auth_resp: return auth_resp diff --git a/Packs/EDL/Integrations/EDL/EDL.yml b/Packs/EDL/Integrations/EDL/EDL.yml index 3f143db310c..07ba53d024d 100644 --- a/Packs/EDL/Integrations/EDL/EDL.yml +++ b/Packs/EDL/Integrations/EDL/EDL.yml @@ -71,22 +71,24 @@ configuration: section: Connect advanced: true required: false -- additionalinfo: How long a concurrent request waits for an in-flight upstream response before bypassing the cache lock. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout to prevent upstream stampedes when responses take longer than this value. +- additionalinfo: 'The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them.' defaultvalue: 1h - display: Cache Lock Timeout + display: Cache Lock Timeout (Deprecated) name: cache_lock_timeout type: 0 section: Connect advanced: true required: false -- additionalinfo: How long the cache lock holder is allowed to populate the cache before another request is permitted to retry. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout. + hidden: true +- additionalinfo: 'The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them.' defaultvalue: 1h - display: Cache Lock Age + display: Cache Lock Age (Deprecated) name: cache_lock_age type: 0 section: Connect advanced: true required: false + hidden: true - additionalinfo: The TTL for 404 responses in the cache. defaultvalue: 1m display: Cache 404 TTL @@ -474,7 +476,7 @@ script: - 'False' - 'True' description: Updates values stored in the List (only available On-Demand). - dockerimage: demisto/flask-nginx:1.0.0.10133006 + dockerimage: demisto/flask-nginx:1.0.0.11047721 longRunning: true longRunningPort: true script: '-' @@ -485,13 +487,13 @@ tests: - EDL Performance Test fromversion: 5.5.0 triggers: - - conditions: - - name: engine - operator: not_exists - - name: isEngineGroup - operator: not_exists - effects: - - name: longRunningPort - action: - hidden: true - required: false +- conditions: + - name: engine + operator: not_exists + - name: isEngineGroup + operator: not_exists + effects: + - name: longRunningPort + action: + hidden: true + required: false diff --git a/Packs/EDL/Integrations/EDL/EDL_test.py b/Packs/EDL/Integrations/EDL/EDL_test.py index 6e0a78582ee..0023ee44beb 100644 --- a/Packs/EDL/Integrations/EDL/EDL_test.py +++ b/Packs/EDL/Integrations/EDL/EDL_test.py @@ -9,6 +9,7 @@ import demistomock as demisto import pytest +import EDL as edl from EDL import ( DONT_COLLAPSE, check_platform_and_version, @@ -1564,3 +1565,96 @@ def test_domain_glob_wildcard_expansion(indicator_value: str, indicator_type: st # Verify the output matches expected assert sorted(output_lines) == sorted(expected_output) + + +def test_get_request_id_without_request_context(mocker): + """ + Given: + - get_request_id is called outside of an active Flask request context. + When: + - Accessing request.headers raises a RuntimeError. + Then: + - A fresh 12-char hex id is generated and a debug log is emitted. + """ + mock_request = mocker.MagicMock() + mock_request.headers.get.side_effect = RuntimeError("Working outside of request context.") + mocker.patch.object(edl, "request", mock_request) + debug_mock = mocker.patch.object(demisto, "debug") + + request_id = edl.get_request_id() + + assert len(request_id) == 12 + assert all(c in "0123456789abcdef" for c in request_id) + debug_mock.assert_called_once() + + +def test_get_request_id_reuses_forwarded_header(mocker): + """ + Given: + - get_request_id is called within a request context that carries X-Request-ID. + When: + - The forwarded header is present. + Then: + - The forwarded id is reused as-is (so it can be grepped across NGINX -> WSGI -> EDL). + """ + mock_request = mocker.MagicMock() + mock_request.headers.get.return_value = "forwarded-id-123" + mocker.patch.object(edl, "request", mock_request) + + assert edl.get_request_id() == "forwarded-id-123" + mock_request.headers.get.assert_called_once_with("X-Request-ID") + + +def test_get_indicators_to_format_restores_stdout_lock_timeout(mocker): + """ + Given: + - A server runtime that exposes demisto._stdout_lock_timeout. + When: + - get_indicators_to_format iterates over indicators (temporarily raising the timeout). + Then: + - The original _stdout_lock_timeout value is restored after the iteration completes. + """ + demisto._stdout_lock_timeout = 60 + try: + indicator_searcher = IndicatorsSearcher(4) + request_args = edl.RequestArguments( + out_format="TEXT", + query="", + limit=3, + url_port_stripping=True, + url_protocol_stripping=True, + url_truncate=True, + ) + + get_indicators_to_format(indicator_searcher, request_args) + + assert demisto._stdout_lock_timeout == 60 + finally: + delattr(demisto, "_stdout_lock_timeout") + + +def test_get_indicators_to_format_noop_when_stdout_lock_timeout_absent(mocker): + """ + Given: + - A server runtime that does NOT expose demisto._stdout_lock_timeout. + When: + - get_indicators_to_format iterates over indicators. + Then: + - The attribute is not created (the timeout tweak is a true no-op). + """ + if hasattr(demisto, "_stdout_lock_timeout"): + delattr(demisto, "_stdout_lock_timeout") + + indicator_searcher = IndicatorsSearcher(4) + request_args = edl.RequestArguments( + out_format="TEXT", + query="", + limit=3, + url_port_stripping=True, + url_protocol_stripping=True, + url_truncate=True, + ) + + get_indicators_to_format(indicator_searcher, request_args) + + assert not hasattr(demisto, "_stdout_lock_timeout") diff --git a/Packs/EDL/Integrations/EDL/README.md b/Packs/EDL/Integrations/EDL/README.md index 315c1f15770..fd90a612bf2 100644 --- a/Packs/EDL/Integrations/EDL/README.md +++ b/Packs/EDL/Integrations/EDL/README.md @@ -96,8 +96,8 @@ For Cortex XSIAM - | Exclude top level domainGlobs | Option to remove top level domainGlobs from the list. For example - \*.com. | False | | Advanced: NGINX Global Directives | NGINX global directives to be passed on the command line using the -g option. Each directive should end with `;`. For example: `worker_processes 4; timer_resolution 100ms;`. Advanced configuration to be used only if instructed by Cortex XSOAR Support. | False | | Advanced: NGINX Server Conf | NGINX server configuration to be used instead of the default NGINX_SERVER_CONF used in the integration code. Advanced configuration to be used only if instructed by Cortex XSOAR Support. | False | -| Cache Lock Timeout | How long a concurrent request waits for an in-flight upstream response before bypassing the cache lock. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout to prevent upstream stampedes when responses take longer than this value. Default is 1h. | False | -| Cache Lock Age | How long the cache lock holder is allowed to populate the cache before another request is permitted to retry. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout. Default is 1h. | False | +| Cache Lock Timeout (Deprecated) | The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them. | False | +| Cache Lock Age (Deprecated) | The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them. | False | | Cache 404 TTL | The TTL for 404 responses in the cache. | False | | Cache Default TTL | The default TTL for responses in the cache. | False | | Advanced: NGINX Read Timeout | NGNIX read timeout in seconds. | False | diff --git a/Packs/EDL/ReleaseNotes/3_4_0.md b/Packs/EDL/ReleaseNotes/3_4_0.md new file mode 100644 index 00000000000..6ba2025f9b3 --- /dev/null +++ b/Packs/EDL/ReleaseNotes/3_4_0.md @@ -0,0 +1,9 @@ + +#### Integrations + +##### Generic Export Indicators Service + +- Improved implementation of container stability when serving a large number of indicators. +- Improved implementation of cache concurrency control to better handle a high volume of concurrent requests. +- Deprecated the *Cache Lock Timeout* and *Cache Lock Age* parameters. +- Updated the Docker image to *demisto/flask-nginx:1.0.0.11047721*. diff --git a/Packs/EDL/pack_metadata.json b/Packs/EDL/pack_metadata.json index a23cf321b8a..4f55a41f7e3 100644 --- a/Packs/EDL/pack_metadata.json +++ b/Packs/EDL/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Generic Export Indicators Service", "description": "Use this pack to generate a list based on your Threat Intel Library, and export it to any product in your network, such as firewalls, agents or SIEMs. This pack supports ongoing distribution of indicators from XSOAR to other products in the network, by creating an endpoint with a list of indicators that can be pulled by external vendors.", "support": "xsoar", - "currentVersion": "3.3.36", + "currentVersion": "3.4.0", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/ExportIndicators/ReleaseNotes/1_0_24.md b/Packs/ExportIndicators/ReleaseNotes/1_0_24.md new file mode 100644 index 00000000000..280fd2ebffe --- /dev/null +++ b/Packs/ExportIndicators/ReleaseNotes/1_0_24.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Export Indicators Service (Deprecated) + +- Maintenance and stability improvements. \ No newline at end of file diff --git a/Packs/ExportIndicators/pack_metadata.json b/Packs/ExportIndicators/pack_metadata.json index 1fd4c972d79..23b7e9d2b49 100644 --- a/Packs/ExportIndicators/pack_metadata.json +++ b/Packs/ExportIndicators/pack_metadata.json @@ -3,7 +3,7 @@ "description": "Deprecated. Use Generic Export Indicators Service instead.", "support": "xsoar", "hidden": true, - "currentVersion": "1.0.23", + "currentVersion": "1.0.24", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server.yml b/Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server.yml index 4d4327fe8ed..e84af155e14 100644 --- a/Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server.yml +++ b/Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server.yml @@ -106,22 +106,24 @@ configuration: section: Collect advanced: true required: false -- additionalinfo: How long a concurrent request waits for an in-flight upstream response before bypassing the cache lock. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout to prevent upstream stampedes when responses take longer than this value. +- additionalinfo: 'The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them.' defaultvalue: 1h - display: Cache Lock Timeout + display: Cache Lock Timeout (Deprecated) name: cache_lock_timeout type: 0 section: Connect advanced: true required: false -- additionalinfo: How long the cache lock holder is allowed to populate the cache before another request is permitted to retry. Should be greater than or equal to the Request Timeout; smaller values will be automatically raised to the Request Timeout. + hidden: true +- additionalinfo: 'The parameter is deprecated and no longer used. Cache locking was replaced by a two-tier fail-fast design that rejects excess concurrent cache-building requests instead of queuing them.' defaultvalue: 1h - display: Cache Lock Age + display: Cache Lock Age (Deprecated) name: cache_lock_age type: 0 section: Connect advanced: true required: false + hidden: true - additionalinfo: The TTL for 404 responses in the cache. defaultvalue: 1m display: Cache 404 TTL @@ -198,7 +200,7 @@ script: - contextPath: TAXIIServer.ServerInfo.description description: The server description. type: String - dockerimage: demisto/flask-nginx:1.0.0.10133006 + dockerimage: demisto/flask-nginx:1.0.0.11047721 longRunning: true longRunningPort: true script: '-' @@ -208,13 +210,13 @@ tests: - TAXII2 Server Performance Test fromversion: 6.1.0 triggers: - - conditions: - - name: engine - operator: not_exists - - name: isEngineGroup - operator: not_exists - effects: - - name: longRunningPort - action: - hidden: true - required: false \ No newline at end of file +- conditions: + - name: engine + operator: not_exists + - name: isEngineGroup + operator: not_exists + effects: + - name: longRunningPort + action: + hidden: true + required: false diff --git a/Packs/TAXIIServer/ReleaseNotes/2_3_0.md b/Packs/TAXIIServer/ReleaseNotes/2_3_0.md new file mode 100644 index 00000000000..1679e43c3b3 --- /dev/null +++ b/Packs/TAXIIServer/ReleaseNotes/2_3_0.md @@ -0,0 +1,8 @@ + +#### Integrations + +##### TAXII2 Server + +- Deprecated the *Cache Lock Timeout* and *Cache Lock Age* parameters. +- Improved implementation of cache concurrency control to better handle a high volume of concurrent requests. +- Updated the Docker image to *demisto/flask-nginx:1.0.0.11047721*. diff --git a/Packs/TAXIIServer/pack_metadata.json b/Packs/TAXIIServer/pack_metadata.json index a42de98bbbe..6175abab9b8 100644 --- a/Packs/TAXIIServer/pack_metadata.json +++ b/Packs/TAXIIServer/pack_metadata.json @@ -2,7 +2,7 @@ "name": "TAXII Server", "description": "This pack provides TAXII Services for system indicators (Outbound feed).", "support": "xsoar", - "currentVersion": "2.2.12", + "currentVersion": "2.3.0", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 16f7bb6383954f43ee9f54e78ca5d65972eb8c70 Mon Sep 17 00:00:00 2001 From: Andrew Shamah <42912128+amshamah419@users.noreply.github.com> Date: Sun, 19 Jul 2026 18:29:24 +0300 Subject: [PATCH 15/47] Tenable io vuln snapshot issue (#45076) * Fix for snapshot sealing issue * Fix for OOM issues in tenable --- .../Integrations/Tenable_io/Tenable_io.py | 145 ++++++++++++++---- .../Tenable_io/Tenable_io_test.py | 142 +++++++++++++++++ Packs/Tenable_io/ReleaseNotes/2_3_28.md | 7 + Packs/Tenable_io/pack_metadata.json | 2 +- 4 files changed, 264 insertions(+), 32 deletions(-) create mode 100644 Packs/Tenable_io/ReleaseNotes/2_3_28.md diff --git a/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io.py b/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io.py index d3e386de0b4..63d1f010cc1 100644 --- a/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io.py +++ b/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io.py @@ -1,3 +1,4 @@ +import gc import re # pylint: disable=W9011 import sys import time @@ -1992,6 +1993,37 @@ def is_vulns_fetch_in_progress(last_run: dict) -> bool: return bool(last_run.get("vulns_available_chunks") or last_run.get("vuln_export_uuid")) +def should_seal_empty_assets_snapshot(assets: list, assets_fetch_in_progress: bool, assets_last_run: dict) -> bool: + """ + Decide whether the assets snapshot should be sealed with an empty payload on this run. + + This guards against the XSUP-71765 regression: after the assets export finished and the + assets were committed to XSIAM, subsequent fetch-assets runs (which happen every fetch + interval while the vulnerabilities export is still processing) returned an empty assets + list and re-sealed the already-sealed snapshot with an empty payload. Re-sealing an + already-committed snapshot with empty data wipes the committed assets from the dataset. + + The empty seal must happen exactly once per snapshot, only when: + - no new assets were fetched this run, and + - the assets export is complete (not in progress), and + - there is a committed snapshot to seal (cumulative total > 0), and + - the snapshot has not already been sealed this cycle. + + Args: + assets: The assets fetched on the current run. + assets_fetch_in_progress: Whether the assets export still has pending work. + assets_last_run: The assets last run state object. + + Returns: + bool: True if the snapshot should be sealed with an empty payload now. + """ + if assets or assets_fetch_in_progress: + return False + if assets_last_run.get("assets_snapshot_sealed"): + return False + return assets_last_run.get("total_assets", 0) > 0 + + def parse_vulnerabilities(vulns): # pylint: disable=W9014 demisto.debug("Parse the vulnerabilities...") if not isinstance(vulns, list): @@ -2105,14 +2137,24 @@ def main(): # pragma: no cover # pylint: disable=W9018 # starting a whole new fetch process for assets demisto.debug("starting new fetch") assets_last_run.update({"assets_last_fetch": time.time()}) + # Determine which stages should run on this invocation BEFORE fetching anything. + # These flags are computed from the pre-fetch state (assets_last_run_copy) so that + # the assets and vulnerabilities stages are decided independently and consistently, + # even though they are now executed sequentially (assets fully sent and freed before + # vulnerabilities are fetched) to avoid holding both large datasets in memory at once + # (XSUP-73037 OOM). + should_fetch_assets = is_assets_fetch_in_progress(assets_last_run_copy) or not is_vulns_fetch_in_progress( + assets_last_run_copy + ) + should_fetch_vulns = is_vulns_fetch_in_progress(assets_last_run_copy) or not is_assets_fetch_in_progress( + assets_last_run_copy + ) + # Fetch Assets: Run if there's an ongoing assets export OR if starting a new fetch cycle - if is_assets_fetch_in_progress(assets_last_run_copy) or not is_vulns_fetch_in_progress(assets_last_run_copy): + if should_fetch_assets: assets = run_assets_fetch(client, assets_last_run) - # Fetch Vulnerabilities: Run if there's an ongoing vulns export OR if assets fetch is complete - if is_vulns_fetch_in_progress(assets_last_run_copy) or not is_assets_fetch_in_progress(assets_last_run_copy): - vulnerabilities = run_vulnerabilities_fetch(client, last_run=assets_last_run) - demisto.info(f"Received {len(assets)} assets and {len(vulnerabilities)} vulnerabilities.") + demisto.info(f"Received {len(assets)} assets.") demisto.debug(f"new lastrun assets: {assets_last_run}") demisto.setAssetsLastRun(assets_last_run) @@ -2130,6 +2172,10 @@ def main(): # pragma: no cover # pylint: disable=W9018 # Vulnerabilities are separate and don't affect the assets snapshot completion assets_fetch_in_progress = is_assets_fetch_in_progress(assets_last_run) + # Remember whether assets were fetched this run before we free the list below, + # so downstream conditions (e.g. module health update) keep their original meaning. + fetched_assets_this_run = bool(assets) + if assets: # Calculate cumulative total BEFORE sending to XSIAM # Per the Fetch Assets Development Flow doc (Case 2 - Continuation Iteration): @@ -2158,42 +2204,78 @@ def main(): # pragma: no cover # pylint: disable=W9018 # Update cumulative asset count AFTER successful send_data_to_xsiam() # This ensures the counter only reflects assets that were actually sent to XSIAM assets_last_run["total_assets"] = cumulative_total + # If this send completed the assets snapshot (no more asset work pending), mark it + # as sealed so subsequent runs that only drain vulnerabilities do not re-seal it + # with an empty payload (XSUP-71765). + if not assets_fetch_in_progress: + assets_last_run["assets_snapshot_sealed"] = True demisto.setAssetsLastRun(assets_last_run) - elif not assets_fetch_in_progress: - # Edge case: asset fetch completed but returned an empty list of assets. - # We still need to seal the snapshot by sending an empty payload with the final items_count - # so XSIAM knows the snapshot is complete. + elif should_seal_empty_assets_snapshot(assets, assets_fetch_in_progress, assets_last_run): + # Asset fetch completed but this run returned an empty list of assets, and the + # snapshot has not been sealed yet. Seal it once by sending an empty payload with + # the final items_count so XSIAM knows the snapshot is complete. cumulative_total = assets_last_run.get("total_assets", 0) - if cumulative_total > 0: - demisto.debug( - f"Asset fetch completed with empty assets list. Sealing snapshot with " - f"snapshot_id={snapshot_id}, items_count={cumulative_total}" - ) - send_data_to_xsiam( - data=[], - vendor=VENDOR, - product=f"{PRODUCT}_assets", - data_type="assets", - snapshot_id=snapshot_id, - items_count=str(cumulative_total), - should_update_health_module=False, - ) - else: - # First fetch returned empty - log this scenario - demisto.debug( - f"Asset fetch completed with empty assets list and cumulative_total=0. " - f"No snapshot sealing needed for snapshot_id={snapshot_id}" - ) + demisto.debug( + f"[Fetch] Asset fetch completed with empty assets list. Sealing snapshot with " + f"snapshot_id={snapshot_id}, items_count={cumulative_total}" + ) + send_data_to_xsiam( + data=[], + vendor=VENDOR, + product=f"{PRODUCT}_assets", + data_type="assets", + snapshot_id=snapshot_id, + items_count=str(cumulative_total), + should_update_health_module=False, + ) + # Mark the snapshot as sealed so it is not re-sealed on the following runs + # while the vulnerabilities export is still in progress (XSUP-71765). + assets_last_run["assets_snapshot_sealed"] = True + demisto.setAssetsLastRun(assets_last_run) + elif not assets_fetch_in_progress: + # Asset fetch is complete but there is nothing to seal: either nothing was ever + # committed this cycle (cumulative_total == 0) or the snapshot was already sealed + # on a previous run. Do NOT re-send an empty payload for an already-sealed + # snapshot - doing so overwrites the committed assets in XSIAM (XSUP-71765). + demisto.debug( + f"[Fetch] No assets to send and snapshot already sealed or nothing committed " + f"(snapshot_id={snapshot_id}, " + f"total_assets={assets_last_run.get('total_assets', 0)}, " + f"assets_snapshot_sealed={assets_last_run.get('assets_snapshot_sealed', False)}). " + f"Skipping snapshot seal." + ) + + # Release the assets from memory now that they have been sent to XSIAM, BEFORE fetching + # vulnerabilities. Assets and vulnerabilities are each up to tens of thousands of records; + # holding both in memory at once previously drove the fetch-assets container past its + # memory limit and caused an OOM kill mid-run, so vulnerabilities were never sent and the + # vulnerabilities dataset was never created (XSUP-73037). Freeing here caps the peak + # footprint at roughly one dataset at a time. + del assets + gc.collect() + + # Fetch Vulnerabilities: Run if there's an ongoing vulns export OR if assets fetch is complete. + # Executed only after assets have been sent and freed above. + if should_fetch_vulns: + vulnerabilities = run_vulnerabilities_fetch(client, last_run=assets_last_run) + demisto.info(f"Received {len(vulnerabilities)} vulnerabilities.") + demisto.debug(f"new lastrun assets after vulns fetch: {assets_last_run}") + demisto.setAssetsLastRun(assets_last_run) if vulnerabilities: vulnerabilities = parse_vulnerabilities(vulnerabilities) demisto.debug(f"sending {len(vulnerabilities)} vulnerabilities to XSIAM.") send_data_to_xsiam(data=vulnerabilities, vendor=VENDOR, product=f"{PRODUCT}_vulnerabilities") + # Release the vulnerabilities from memory once sent, mirroring the assets handling above. + del vulnerabilities + gc.collect() - # Update module health separately to show the number of assets pulled + # Update module health separately to show the number of assets pulled. + # Uses fetched_assets_this_run (captured before the assets list was freed above) + # to preserve the original "fetched assets this run OR assets fetch complete" semantics. cumulative_total = assets_last_run.get("total_assets", 0) - if assets or not assets_fetch_in_progress: + if fetched_assets_this_run or not assets_fetch_in_progress: demisto.updateModuleHealth({"assetsPulled": cumulative_total}) # Clean up snapshot state when the entire fetch cycle is complete @@ -2206,6 +2288,7 @@ def main(): # pragma: no cover # pylint: disable=W9018 ) assets_last_run.pop("snapshot_id", None) assets_last_run.pop("total_assets", None) + assets_last_run.pop("assets_snapshot_sealed", None) demisto.setAssetsLastRun(assets_last_run) demisto.info("Done Sending data to XSIAM.") diff --git a/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io_test.py b/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io_test.py index fb2f1a043d4..f7d7a98e9b9 100644 --- a/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io_test.py +++ b/Packs/Tenable_io/Integrations/Tenable_io/Tenable_io_test.py @@ -1023,3 +1023,145 @@ def test_request_uuid_export_vulnerabilities_with_tags(mocker, args, expected_ta # Verify PollResult structure assert result.continue_to_poll is True assert result.args_for_next_run["exportUuid"] == mock_export_uuid + + +def test_should_seal_empty_assets_snapshot_first_time_after_assets_complete(): + """ + Given: + - The assets export has finished (assets_fetch_in_progress=False). + - Assets were already sent this cycle (total_assets > 0) but the current + fetch returned no new assets (empty list). + - The snapshot has NOT yet been sealed with an empty payload. + When: + - Running should_seal_empty_assets_snapshot. + Then: + - Returns True: the snapshot should be sealed exactly once. + """ + from Tenable_io import should_seal_empty_assets_snapshot + + last_run = {"snapshot_id": "snap-1", "total_assets": 4323, "vuln_export_uuid": "v-1"} + assert should_seal_empty_assets_snapshot(assets=[], assets_fetch_in_progress=False, assets_last_run=last_run) is True + + +def test_should_not_reseal_assets_snapshot_when_already_sealed(): + """ + Given: + - The assets export has finished and the snapshot was ALREADY sealed this + cycle (assets_snapshot_sealed=True), while the vulnerabilities export is + still in progress (vuln_export_uuid present). + - The current fetch returned no new assets. + When: + - Running should_seal_empty_assets_snapshot. + Then: + - Returns False: XSUP-71765 regression guard. Re-sealing an already-sealed + snapshot with an empty payload wipes the committed assets in XSIAM. + """ + from Tenable_io import should_seal_empty_assets_snapshot + + last_run = { + "snapshot_id": "snap-1", + "total_assets": 4323, + "vuln_export_uuid": "v-1", + "assets_snapshot_sealed": True, + } + assert should_seal_empty_assets_snapshot(assets=[], assets_fetch_in_progress=False, assets_last_run=last_run) is False + + +def test_should_not_seal_assets_snapshot_while_assets_still_in_progress(): + """ + Given: + - The assets export is still in progress (more chunks/export pending). + When: + - Running should_seal_empty_assets_snapshot with an empty assets list. + Then: + - Returns False: the snapshot is not complete yet, so it must not be sealed. + """ + from Tenable_io import should_seal_empty_assets_snapshot + + last_run = {"snapshot_id": "snap-1", "total_assets": 0} + assert should_seal_empty_assets_snapshot(assets=[], assets_fetch_in_progress=True, assets_last_run=last_run) is False + + +def test_should_not_seal_assets_snapshot_when_nothing_was_fetched(): + """ + Given: + - The assets fetch completed but cumulative total is 0 (nothing to seal). + When: + - Running should_seal_empty_assets_snapshot with an empty assets list. + Then: + - Returns False: there is no committed snapshot to seal. + """ + from Tenable_io import should_seal_empty_assets_snapshot + + last_run = {"snapshot_id": "snap-1", "total_assets": 0} + assert should_seal_empty_assets_snapshot(assets=[], assets_fetch_in_progress=False, assets_last_run=last_run) is False + + +def test_should_not_empty_seal_when_assets_were_fetched_this_run(): + """ + Given: + - The current fetch returned assets (non-empty list). + When: + - Running should_seal_empty_assets_snapshot. + Then: + - Returns False: the data-send path handles sealing, not the empty-seal path. + """ + from Tenable_io import should_seal_empty_assets_snapshot + + last_run = {"snapshot_id": "snap-1", "total_assets": 100} + assert ( + should_seal_empty_assets_snapshot(assets=[{"id": 1}], assets_fetch_in_progress=False, assets_last_run=last_run) is False + ) + + +def test_fetch_assets_sends_assets_before_fetching_vulnerabilities(mocker): + """ + Given: + - A fetch-assets run where both assets and vulnerabilities are available. + When: + - Running main() for the 'fetch-assets' command. + Then: + - Assets are sent to XSIAM BEFORE vulnerabilities are fetched from the API, so the two + large datasets are never held in memory at the same time (XSUP-73037 OOM prevention). + """ + import Tenable_io + + mock_demisto(mocker, mock_args={}) + mocker.patch.object(demisto, "command", return_value="fetch-assets") + mocker.patch.object(demisto, "getAssetsLastRun", return_value={}) + mocker.patch.object(demisto, "setAssetsLastRun") + mocker.patch.object(demisto, "updateModuleHealth") + mocker.patch.object(Tenable_io, "is_xsiam", return_value=True) + mocker.patch.object(Tenable_io, "is_platform", return_value=True) + + call_order: list = [] + + def fake_run_assets_fetch(client, last_run): + call_order.append("fetch_assets") + return [{"id": "asset-1"}] + + def fake_run_vulnerabilities_fetch(client, last_run): + call_order.append("fetch_vulns") + return [{"id": "vuln-1"}] + + def fake_send_data_to_xsiam(*args, **kwargs): + product = kwargs.get("product", "") + if "assets" in product: + call_order.append("send_assets") + elif "vulnerabilities" in product: + call_order.append("send_vulns") + + mocker.patch.object(Tenable_io, "run_assets_fetch", side_effect=fake_run_assets_fetch) + mocker.patch.object(Tenable_io, "run_vulnerabilities_fetch", side_effect=fake_run_vulnerabilities_fetch) + mocker.patch.object(Tenable_io, "send_data_to_xsiam", side_effect=fake_send_data_to_xsiam) + mocker.patch.object(Tenable_io, "parse_vulnerabilities", side_effect=lambda vulns: vulns) + + Tenable_io.main() + + # Assets must be sent to XSIAM before vulnerabilities are fetched from the API. + assert "send_assets" in call_order + assert "fetch_vulns" in call_order + assert call_order.index("send_assets") < call_order.index("fetch_vulns") + # Sanity: assets are fetched first and vulnerabilities are sent last. + assert call_order.index("fetch_assets") < call_order.index("send_assets") + assert call_order.index("fetch_vulns") < call_order.index("send_vulns") diff --git a/Packs/Tenable_io/ReleaseNotes/2_3_28.md b/Packs/Tenable_io/ReleaseNotes/2_3_28.md new file mode 100644 index 00000000000..4c55cf20158 --- /dev/null +++ b/Packs/Tenable_io/ReleaseNotes/2_3_28.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### Tenable Vulnerability Management (formerly Tenable.io) + +- Fixed an issue where the assets snapshot was repeatedly re-sealed with an empty payload while the vulnerabilities export was still in progress, which could cause previously ingested assets to be overwritten and stop appearing in the assets dataset. The assets snapshot is now sealed exactly once per fetch cycle. +- Fixed an issue where the **fetch-assets** flow could run out of memory and be terminated mid-run when large asset and vulnerability exports were processed in the same fetch. Assets are now sent to XSIAM and released from memory before vulnerabilities are fetched, preventing the out-of-memory crash that previously stopped the vulnerabilities dataset from being created. diff --git a/Packs/Tenable_io/pack_metadata.json b/Packs/Tenable_io/pack_metadata.json index d662501a797..d0634df871f 100644 --- a/Packs/Tenable_io/pack_metadata.json +++ b/Packs/Tenable_io/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Tenable Vulnerability Management (formerly Tenable.io)", "description": "A comprehensive asset centric solution to accurately track resources while accommodating dynamic assets such as cloud, mobile devices, containers and web applications.", "support": "xsoar", - "currentVersion": "2.3.27", + "currentVersion": "2.3.28", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 972d3e34b333280b71af752ad08b65d701c8cfc4 Mon Sep 17 00:00:00 2001 From: Adi Daud <46249224+adi88d@users.noreply.github.com> Date: Sun, 19 Jul 2026 22:27:18 +0300 Subject: [PATCH 16/47] XSUP-69421 | stabilize WebSocket reconnection on ping timeouts (#44986) --- .../ProofpointEmailSecurityEventCollector.py | 12 +- ...ofpointEmailSecurityEventCollector_test.py | 135 +++++++++++++++++- .../ReleaseNotes/1_0_28.md | 6 + .../pack_metadata.json | 6 +- 4 files changed, 153 insertions(+), 6 deletions(-) create mode 100644 Packs/ProofpointEmailSecurity/ReleaseNotes/1_0_28.md diff --git a/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector.py b/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector.py index 921ba168b48..61d0fc5ad94 100644 --- a/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector.py +++ b/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector.py @@ -29,9 +29,9 @@ DEFAULT_GET_EVENTS_LIMIT = 10 DATE_FILTER_FORMAT = "%Y-%m-%dT%H:%M:%S%z" PING_INTERVAL = 60 # Interval between keepalive pings in seconds -PING_TIMEOUT: float | None = None # Disable pong-reply timeout to tolerate large/slow messages +PING_TIMEOUT = 120 # Timeout for waiting for a pong reply to a keepalive ping in seconds OPEN_TIMEOUT = 10 # Timeout for opening the WebSocket connection in seconds -CLOSE_TIMEOUT = 10 # Timeout for closing the connection in seconds +CLOSE_TIMEOUT = 60 # Timeout for closing the connection in seconds MAX_MESSAGE_SIZE: int | None = None # Disable the 1 MiB default size limit on incoming messages RECEIVE_TIMEOUT = 1 # Timeout for receiving events in seconds @@ -424,7 +424,13 @@ def recover_after_disconnection(connection: EventConnection, events: list[dict], ) if reconnect: demisto.info(f"[{connection.event_type}] Attempting to reconnect after disconnection.") - connection.reconnect() + try: + connection.reconnect() + except Exception as e: + demisto.error( + f"[{connection.event_type}] Failed to reconnect after disconnection. Error: {e}. {traceback.format_exc()}" + ) + raise def test_module(host: str, cluster_id: str, api_key: str): diff --git a/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector_test.py b/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector_test.py index 75c618fc985..77366c0818a 100644 --- a/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector_test.py +++ b/Packs/ProofpointEmailSecurity/Integrations/ProofpointEmailSecurityEventCollector/ProofpointEmailSecurityEventCollector_test.py @@ -1,5 +1,6 @@ import uuid from contextlib import ExitStack, contextmanager +from http import HTTPStatus import ProofpointEmailSecurityEventCollector import pytest @@ -12,12 +13,15 @@ EventConnection, datetime, demisto, + exceptions, fetch_events, json, long_running_execution_command, perform_long_running_loop, + time, timedelta, websocket_connections, + MAX_RECONNECT_ATTEMPTS, PING_TIMEOUT, CLOSE_TIMEOUT, OPEN_TIMEOUT, @@ -458,6 +462,93 @@ def mock_receive_event(conn, timeout=1): assert events[1]["message"] == "In-transit 2" +def _make_invalid_status(status_code): + """Build a websockets InvalidStatus exception carrying the given HTTP status code.""" + response = type("Resp", (), {"status_code": status_code})() + return exceptions.InvalidStatus(response) + + +def test_reconnect_success_after_conflict(mocker, connection: MockConnection): + """ + Given: + - A connection whose first reconnect attempt fails with an HTTP 409 (session conflict) + and whose second attempt succeeds. + + When: + - Calling reconnect(). + + Then: + - Ensure the backoff sleep is applied once and the connection is re-established + without raising. + """ + mocker.patch.object(EventConnection, "connect", return_value=connection) + mocker.patch.object(demisto, "info") + mocker.patch.object(demisto, "error") + event_connection = EventConnection(event_type="message", url="wss://testing", headers={}, check_heartbeat=False) + + conflict = _make_invalid_status(HTTPStatus.CONFLICT) + connect_mock = mocker.patch.object(EventConnection, "connect", side_effect=[conflict, connection]) + sleep_mock = mocker.patch.object(time, "sleep") + + event_connection.reconnect() + + assert connect_mock.call_count == 2 + assert sleep_mock.call_count == 1 # one backoff wait before the successful retry + assert event_connection.connection is connection + + +def test_reconnect_raises_on_non_conflict_status(mocker, connection: MockConnection): + """ + Given: + - A connection whose reconnect attempt fails with a non-409 InvalidStatus (e.g. 401). + + When: + - Calling reconnect(). + + Then: + - Ensure the exception is raised immediately without retrying or sleeping. + """ + mocker.patch.object(EventConnection, "connect", return_value=connection) + mocker.patch.object(demisto, "info") + mocker.patch.object(demisto, "error") + event_connection = EventConnection(event_type="message", url="wss://testing", headers={}, check_heartbeat=False) + + unauthorized = _make_invalid_status(HTTPStatus.UNAUTHORIZED) + connect_mock = mocker.patch.object(EventConnection, "connect", side_effect=unauthorized) + sleep_mock = mocker.patch.object(time, "sleep") + + with pytest.raises(exceptions.InvalidStatus): + event_connection.reconnect() + + assert connect_mock.call_count == 1 # no retry on non-conflict status + assert sleep_mock.call_count == 0 + + +def test_reconnect_gives_up_after_max_attempts(mocker, connection: MockConnection): + """ + Given: + - A connection whose reconnect attempts always fail with HTTP 409 (session conflict). + + When: + - Calling reconnect(). + + Then: + - Ensure the retry loop is bounded by MAX_RECONNECT_ATTEMPTS and does not loop forever. + """ + mocker.patch.object(EventConnection, "connect", return_value=connection) + mocker.patch.object(demisto, "info") + mocker.patch.object(demisto, "error") + event_connection = EventConnection(event_type="message", url="wss://testing", headers={}, check_heartbeat=False) + + conflict = _make_invalid_status(HTTPStatus.CONFLICT) + connect_mock = mocker.patch.object(EventConnection, "connect", side_effect=conflict) + mocker.patch.object(time, "sleep") + + event_connection.reconnect() + + assert connect_mock.call_count == MAX_RECONNECT_ATTEMPTS + + def test_recover_after_disconnection_with_reconnect(mocker, connection: MockConnection): """ Given: @@ -501,7 +592,7 @@ def test_recover_after_disconnection_with_reconnect(mocker, connection: MockConn assert existing_events[2]["message"] == "In-transit 2" assert "2" in existing_event_ids assert "3" in existing_event_ids - assert reconnect_mock.call_count == 1 # Should not be called because reconnect=False + assert reconnect_mock.call_count == 1 # Should be called once because reconnect=True def test_recover_after_disconnection_without_reconnect(mocker, connection: MockConnection): @@ -538,3 +629,45 @@ def test_recover_after_disconnection_without_reconnect(mocker, connection: MockC assert len(existing_events) == 1 assert 1 in existing_event_ids assert reconnect_mock.call_count == 0 # Should not be called because reconnect=False + + +def test_recover_after_disconnection_reconnect_failure(mocker, connection: MockConnection): + """ + Given: + - A connection that has been disconnected + - No in-transit events to receive + - reconnect() raises an exception (e.g. the Proofpoint server is unreachable) + + When: + - Calling recover_after_disconnection with reconnect=True + + Then: + - Ensure the exception is re-raised so the long-running loop can restart the connection + - Ensure the reconnection failure is logged via demisto.error + """ + existing_events: list[dict] = [] + existing_event_ids: set[str] = set() + + mocker.patch.object( + ProofpointEmailSecurityEventCollector, + "receive_events_after_disconnection", + return_value=[], + ) + mocker.patch.object(EventConnection, "connect", return_value=connection) + reconnect_mock = mocker.patch.object( + EventConnection, "reconnect", side_effect=DemistoException("[Errno 104] Connection reset by peer") + ) + error_mock = mocker.patch.object(demisto, "error") + event_connection = EventConnection(event_type="message", url="wss://testing", headers={}) + + with pytest.raises(DemistoException, match="Connection reset by peer"): + ProofpointEmailSecurityEventCollector.recover_after_disconnection( + connection=event_connection, + events=existing_events, + event_ids=existing_event_ids, + reconnect=True, + ) + + assert reconnect_mock.call_count == 1 + assert error_mock.called + assert "Failed to reconnect after disconnection" in error_mock.call_args[0][0] diff --git a/Packs/ProofpointEmailSecurity/ReleaseNotes/1_0_28.md b/Packs/ProofpointEmailSecurity/ReleaseNotes/1_0_28.md new file mode 100644 index 00000000000..eebe4aaea38 --- /dev/null +++ b/Packs/ProofpointEmailSecurity/ReleaseNotes/1_0_28.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Proofpoint Email Security Event Collector + +- Fixed an issue where WebSocket connection resets (*[Errno 104] Connection reset by peer*) were not handled and recovered from correctly, including clearer error logging on reconnection failures. diff --git a/Packs/ProofpointEmailSecurity/pack_metadata.json b/Packs/ProofpointEmailSecurity/pack_metadata.json index 1278ee1a44d..5cec78e5fca 100644 --- a/Packs/ProofpointEmailSecurity/pack_metadata.json +++ b/Packs/ProofpointEmailSecurity/pack_metadata.json @@ -2,9 +2,10 @@ "name": "Proofpoint Email Security", "description": "Proofpoint Email Security pack provides visibility into email security threats and protects your organization from phishing, malware, and compliance risks.", "support": "xsoar", - "currentVersion": "1.0.27", + "currentVersion": "1.0.28", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", + "created": "2023-09-12T00:00:00Z", "email": "", "categories": [ "Analytics & SIEM" @@ -17,7 +18,8 @@ "useCases": [], "keywords": [ "On Demand", - "PoD" + "PoD", + "proofpoint" ], "marketplaces": [ "marketplacev2", From 73b36e584b60c67b542bd7674f40773887dfe4ac Mon Sep 17 00:00:00 2001 From: yedidyacohenpalo <162107504+yedidyacohenpalo@users.noreply.github.com> Date: Sun, 19 Jul 2026 23:46:43 +0300 Subject: [PATCH 17/47] remove MC100 (#45133) --- validation_config.toml | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/validation_config.toml b/validation_config.toml index 4b33a9345bd..c34d0e82f33 100644 --- a/validation_config.toml +++ b/validation_config.toml @@ -55,8 +55,7 @@ select = [ "ST110", "ST111", "ST112", "ST113", "ST114", "TR100", "TR101", "VC100", "VC101", - "AS101", "AS102", "AS103", "AS104", "AS105", "AS107", "AS108", "AS109", - "MC100" + "AS101", "AS102", "AS103", "AS104", "AS105", "AS107", "AS108", "AS109" ] warning = ["PB130", "PB131", "RN116", "GR101","PA133", "AS106", "MC101", "BC117"] @@ -91,8 +90,7 @@ select = [ "ST110", "TR100", "TR101", "VC100", "VC101", - "AS101", "AS102", "AS104", "AS105", "AS107", "AS108", - "MC100" + "AS101", "AS102", "AS104", "AS105", "AS107", "AS108" ] warning = ["RM108", "PB130", "PB131", "GR107", "GR110", "GR101","PA133", "AS106", "MC101", "GR102", "GR113", "GR103", "GR109", "BA131", "BA132"] From 6d5e28844c85eaeb8dac6f234edf01f7395ed229 Mon Sep 17 00:00:00 2001 From: dtroushinsky Date: Mon, 20 Jul 2026 08:26:21 +0300 Subject: [PATCH 18/47] XSUP-72627 - iZOOlogic: Add light and dark logo assets for Command Center display (#45105) * XSUP-72627 * code review * fix * fix * Revert "fix" This reverts commit 7ffc23a8b384f15b20b183028898cbeacb6c408b. * secret fix --- Packs/iZOOlogic/.secrets-ignore | 1 + Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_dark.svg | 3 +++ Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_light.svg | 3 +++ Packs/iZOOlogic/ReleaseNotes/1_0_2.md | 6 ++++++ Packs/iZOOlogic/pack_metadata.json | 2 +- 5 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_dark.svg create mode 100644 Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_light.svg create mode 100644 Packs/iZOOlogic/ReleaseNotes/1_0_2.md diff --git a/Packs/iZOOlogic/.secrets-ignore b/Packs/iZOOlogic/.secrets-ignore index c799007d730..d1008ef25ca 100644 --- a/Packs/iZOOlogic/.secrets-ignore +++ b/Packs/iZOOlogic/.secrets-ignore @@ -6,3 +6,4 @@ https://tvsmotor.com.mt https://tvs-credit.dev.veefin.in https://www.facebook.com/ads/library/?id=917334661007536 dOn b +Vwe diff --git a/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_dark.svg b/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_dark.svg new file mode 100644 index 00000000000..1fe19352e09 --- /dev/null +++ b/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_dark.svg @@ -0,0 +1,3 @@ + + + diff --git a/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_light.svg b/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_light.svg new file mode 100644 index 00000000000..12d5e34e04e --- /dev/null +++ b/Packs/iZOOlogic/Integrations/iZOOlogic/iZOOlogic_light.svg @@ -0,0 +1,3 @@ + + + diff --git a/Packs/iZOOlogic/ReleaseNotes/1_0_2.md b/Packs/iZOOlogic/ReleaseNotes/1_0_2.md new file mode 100644 index 00000000000..95f0dd7e954 --- /dev/null +++ b/Packs/iZOOlogic/ReleaseNotes/1_0_2.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### iZOOlogic + +- Fixed an issue where the iZOOlogic logo was not displayed on the Cortex XSIAM Command Center dashboard. diff --git a/Packs/iZOOlogic/pack_metadata.json b/Packs/iZOOlogic/pack_metadata.json index 12bb08ea1a3..11b3a57a2ce 100644 --- a/Packs/iZOOlogic/pack_metadata.json +++ b/Packs/iZOOlogic/pack_metadata.json @@ -2,7 +2,7 @@ "name": "iZOOlogic", "description": "Fetches threat incidents from iZOOlogic for automated ingestion into Cortex.", "support": "xsoar", - "currentVersion": "1.0.1", + "currentVersion": "1.0.2", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 5aefcec85aa7ccc44620740d0c1f1e434916ba5d Mon Sep 17 00:00:00 2001 From: Content Bot <55035720+content-bot@users.noreply.github.com> Date: Mon, 20 Jul 2026 13:34:03 +0300 Subject: [PATCH 19/47] Auto RN: az-CRTX-258563 (#45048) * Initial release notes for az-CRTX-258563 * Bump pack from version Core to 3.5.73. * Sync release notes from GitLab (4a3ecd93) * Sync release notes from GitLab (4c1666ba) * Sync release notes from GitLab (e7734b1e) * Sync release notes from GitLab (74ebc5cd) * Sync release notes from GitLab (bae2119d) * Sync release notes from GitLab (c1ca60d5) * Sync release notes from GitLab (dc098c63) * Sync release notes from GitLab (bff758ac) * Sync release notes from GitLab (05cb0647) * Sync release notes from GitLab (3641acb1) * Sync release notes from GitLab (fa051082) * Sync release notes from GitLab (9354507b) * Sync release notes from GitLab (ae5ca84e) --------- Co-authored-by: CI Bot Co-authored-by: Content Bot Co-authored-by: azonenfeld <117573492+aaron1535@users.noreply.github.com> Co-authored-by: Bar Gali <75535203+BarGali@users.noreply.github.com> --- Packs/Core/ReleaseNotes/3_5_73.md | 3 +++ Packs/Core/pack_metadata.json | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) create mode 100644 Packs/Core/ReleaseNotes/3_5_73.md diff --git a/Packs/Core/ReleaseNotes/3_5_73.md b/Packs/Core/ReleaseNotes/3_5_73.md new file mode 100644 index 00000000000..a0662884628 --- /dev/null +++ b/Packs/Core/ReleaseNotes/3_5_73.md @@ -0,0 +1,3 @@ +## Core + +- Documentation and metadata improvements. diff --git a/Packs/Core/pack_metadata.json b/Packs/Core/pack_metadata.json index e14e634d13f..bc5b3341422 100644 --- a/Packs/Core/pack_metadata.json +++ b/Packs/Core/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Core", "description": "Automates incident response", "support": "xsoar", - "currentVersion": "3.5.72", + "currentVersion": "3.5.73", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", From 052dc8f96508b0adcd5360d924054350db65db10 Mon Sep 17 00:00:00 2001 From: Moshe Eichler <78307768+MosheEichler@users.noreply.github.com> Date: Mon, 20 Jul 2026 16:16:39 +0300 Subject: [PATCH 20/47] Fix/XSUP-72401/File reputation no HTTP response (#45129) * Fix/XSUP-72401/File reputation no HTTP response * add continueonerror * AI review comments --- .../Playbooks/playbook-File_Reputation.yml | 8 ++++++++ Packs/CommonPlaybooks/ReleaseNotes/2_8_5.md | 5 +++++ Packs/CommonPlaybooks/pack_metadata.json | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-) create mode 100644 Packs/CommonPlaybooks/ReleaseNotes/2_8_5.md diff --git a/Packs/CommonPlaybooks/Playbooks/playbook-File_Reputation.yml b/Packs/CommonPlaybooks/Playbooks/playbook-File_Reputation.yml index bbaa9804baf..4526858508e 100644 --- a/Packs/CommonPlaybooks/Playbooks/playbook-File_Reputation.yml +++ b/Packs/CommonPlaybooks/Playbooks/playbook-File_Reputation.yml @@ -234,6 +234,7 @@ tasks: url: simple: https://hashlookup.circl.lu/lookup/sha256/${inputs.FileSHA256} separatecontext: false + continueonerror: true view: |- { "position": { @@ -277,6 +278,7 @@ tasks: transformers: - operator: Stringify separatecontext: false + continueonerror: true view: |- { "position": { @@ -323,6 +325,12 @@ tasks: value: simple: Non existing SHA-256 ignorecase: true + - - operator: isNotEmpty + left: + value: + complex: + root: NSRLCheckResults + iscontext: true view: |- { "position": { diff --git a/Packs/CommonPlaybooks/ReleaseNotes/2_8_5.md b/Packs/CommonPlaybooks/ReleaseNotes/2_8_5.md new file mode 100644 index 00000000000..f819976a433 --- /dev/null +++ b/Packs/CommonPlaybooks/ReleaseNotes/2_8_5.md @@ -0,0 +1,5 @@ +#### Playbooks + +##### File Reputation + +- Fixed an issue where the playbook failed when there was no response from the HTTP request. diff --git a/Packs/CommonPlaybooks/pack_metadata.json b/Packs/CommonPlaybooks/pack_metadata.json index fa8e6adb041..86bb101056d 100644 --- a/Packs/CommonPlaybooks/pack_metadata.json +++ b/Packs/CommonPlaybooks/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Common Playbooks", "description": "Frequently used playbooks pack.", "support": "xsoar", - "currentVersion": "2.8.4", + "currentVersion": "2.8.5", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", @@ -12,7 +12,7 @@ ], "tags": [], "useCases": [], - "keywords": [], + "keywords": ["Cortex"], "dependencies": { "Threat_Crowd": { "mandatory": false, From 3cdcab81961d60820de0cce62c13d71135cbc276 Mon Sep 17 00:00:00 2001 From: Yaniv Blum Date: Mon, 20 Jul 2026 16:30:27 -0500 Subject: [PATCH 21/47] Reco: address code review feedback, add tests for new commands - Reco.py: bound the _paginate_all loop with MAX_PAGES, call demisto.args() once and reuse it in main(), remove the unused create_filter() helper. - Reco_test.py: add unit tests for all 14 new reco-list-* commands. - Reco.yml: fix output/argument descriptions to start with "The " and end with a period, use active voice in command descriptions, drop the default-value duplication in limit-arg descriptions, bump fromversion to 6.8.0 (sectionorder/section require it), and restore additionalinfo for config params that generate-docs left blank. - Regenerate the integration README structurally (Base Command sections, full Input/Output tables for every command, including the 14 new list commands and the ones with stale "Same as X" shortcuts), while preserving the hand-written intro and SCIM filter guide. - Pack README: fix multi-product wording, AI agent capitalization, and a missing comma. - Bump to 2.0.0 (major) for the breaking changes, add the breakingChanges companion JSON, and rewrite the release notes to match this repo's doc-review templates (starred command/field styling, approved bullet prefixes, no ad-hoc heading for breaking changes). - pack_metadata.json: add the Reco keyword and githubUser field. --- Packs/Reco/.pack-ignore | 3 + Packs/Reco/Integrations/Reco/README.md | 430 +++++++++++++++++++--- Packs/Reco/Integrations/Reco/Reco.py | 122 +++--- Packs/Reco/Integrations/Reco/Reco.yml | 362 +++++++++--------- Packs/Reco/Integrations/Reco/Reco_test.py | 194 ++++++++++ Packs/Reco/README.md | 6 +- Packs/Reco/ReleaseNotes/1_8_0.md | 36 -- Packs/Reco/ReleaseNotes/2_0_0.json | 4 + Packs/Reco/ReleaseNotes/2_0_0.md | 12 + Packs/Reco/pack_metadata.json | 6 +- 10 files changed, 832 insertions(+), 343 deletions(-) delete mode 100644 Packs/Reco/ReleaseNotes/1_8_0.md create mode 100644 Packs/Reco/ReleaseNotes/2_0_0.json create mode 100644 Packs/Reco/ReleaseNotes/2_0_0.md diff --git a/Packs/Reco/.pack-ignore b/Packs/Reco/.pack-ignore index e69de29bb2d..cbb7ca145ad 100644 --- a/Packs/Reco/.pack-ignore +++ b/Packs/Reco/.pack-ignore @@ -0,0 +1,3 @@ +[known_words] +SCIM +backoff diff --git a/Packs/Reco/Integrations/Reco/README.md b/Packs/Reco/Integrations/Reco/README.md index cd09b9095d3..7e39b65cee8 100644 --- a/Packs/Reco/Integrations/Reco/README.md +++ b/Packs/Reco/Integrations/Reco/README.md @@ -16,7 +16,7 @@ This integration was integrated and tested with Reco External API v1. | Source | Filter fetched incidents by SaaS source | False | | Before | Fetch incidents created before this timestamp | False | | After | Fetch incidents created after this timestamp | False | -| Risk level | Severity filter for fetched incidents. Accepts a single value or comma-separated list. Values: LOW, MEDIUM, HIGH, CRITICAL (or numeric 10/20/30/40). Example: `HIGH,CRITICAL` | False | +| Minimum risk level (e.g. MEDIUM fetches medium and higher) | The minimum severity threshold for fetched incidents. Accepts a single value: LOW, MEDIUM, HIGH, or CRITICAL (or numeric equivalents 10, 20, 30, 40). Alerts at or above this severity are fetched. | False | | First fetch timestamp | How far back to fetch on first run (e.g. `7 days`, `12 hours`) | False | ## SCIM v2 Filters @@ -41,8 +41,13 @@ Pagination is embedded in the filter string: `limit eq 100 and page eq 1`. ### reco-add-comment-to-alert +*** Add a comment to an alert in Reco. +#### Base Command + +`reco-add-comment-to-alert` + #### Input | **Argument Name** | **Description** | **Required** | @@ -52,8 +57,13 @@ Add a comment to an alert in Reco. ### reco-update-incident-timeline +*** Add a comment to an incident timeline in Reco. +#### Base Command + +`reco-update-incident-timeline` + #### Input | **Argument Name** | **Description** | **Required** | @@ -63,8 +73,13 @@ Add a comment to an incident timeline in Reco. ### reco-resolve-visibility-event +*** Resolve an event in a Reco Finding. +#### Base Command + +`reco-resolve-visibility-event` + #### Input | **Argument Name** | **Description** | **Required** | @@ -74,8 +89,13 @@ Resolve an event in a Reco Finding. ### reco-get-risky-users +*** List all accounts flagged as risky (auto-paginates all results). +#### Base Command + +`reco-get-risky-users` + #### Context Output | **Path** | **Type** | **Description** | @@ -92,8 +112,13 @@ List all accounts flagged as risky (auto-paginates all results). ### reco-add-risky-user-label +*** Tag a user as risky in Reco. +#### Base Command + +`reco-add-risky-user-label` + #### Input | **Argument Name** | **Description** | **Required** | @@ -102,8 +127,13 @@ Tag a user as risky in Reco. ### reco-add-leaving-org-user-label +*** Tag a user as a departing employee in Reco. +#### Base Command + +`reco-add-leaving-org-user-label` + #### Input | **Argument Name** | **Description** | **Required** | @@ -112,8 +142,13 @@ Tag a user as a departing employee in Reco. ### reco-get-assets-user-has-access-to +*** List files a user has access to. +#### Base Command + +`reco-get-assets-user-has-access-to` + #### Input | **Argument Name** | **Description** | **Required** | @@ -129,8 +164,13 @@ List files a user has access to. ### reco-get-sensitive-assets-by-name +*** Find sensitive assets by name. +#### Base Command + +`reco-get-sensitive-assets-by-name` + #### Input | **Argument Name** | **Description** | **Required** | @@ -153,8 +193,13 @@ Find sensitive assets by name. ### reco-get-sensitive-assets-by-id +*** Find sensitive assets by ID. +#### Base Command + +`reco-get-sensitive-assets-by-id` + #### Input | **Argument Name** | **Description** | **Required** | @@ -163,12 +208,26 @@ Find sensitive assets by ID. #### Context Output -Same as `reco-get-sensitive-assets-by-name`. +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.SensitiveAssets.id | String | Asset ID | +| Reco.SensitiveAssets.name | String | Asset name | +| Reco.SensitiveAssets.owner | String | Asset owner | +| Reco.SensitiveAssets.url | String | Asset URL | +| Reco.SensitiveAssets.sensitivityLevel | Number | Sensitivity level (30=HIGH, 40=CRITICAL) | +| Reco.SensitiveAssets.permissionVisibility | String | Permission visibility (PUBLIC / INTERNAL / RESTRICTED) | +| Reco.SensitiveAssets.location | String | File path | +| Reco.SensitiveAssets.dataCategories | Unknown | Detected data categories | ### reco-get-assets-by-id +*** Find any asset by ID. +#### Base Command + +`reco-get-assets-by-id` + #### Input | **Argument Name** | **Description** | **Required** | @@ -177,12 +236,26 @@ Find any asset by ID. #### Context Output -Same as `reco-get-sensitive-assets-by-name`. +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Reco.SensitiveAssets.id | String | Asset ID | +| Reco.SensitiveAssets.name | String | Asset name | +| Reco.SensitiveAssets.owner | String | Asset owner | +| Reco.SensitiveAssets.url | String | Asset URL | +| Reco.SensitiveAssets.sensitivityLevel | Number | Sensitivity level (30=HIGH, 40=CRITICAL) | +| Reco.SensitiveAssets.permissionVisibility | String | Permission visibility (PUBLIC / INTERNAL / RESTRICTED) | +| Reco.SensitiveAssets.location | String | File path | +| Reco.SensitiveAssets.dataCategories | Unknown | Detected data categories | ### reco-get-link-to-user-overview-page +*** Generate a deep link to the Reco UI overview page for an entity. +#### Base Command + +`reco-get-link-to-user-overview-page` + #### Input | **Argument Name** | **Description** | **Required** | @@ -192,8 +265,13 @@ Generate a deep link to the Reco UI overview page for an entity. ### reco-get-3rd-parties-accessible-to-data-list +*** List third-party domains that have access to sensitive data. +#### Base Command + +`reco-get-3rd-parties-accessible-to-data-list` + #### Input | **Argument Name** | **Description** | **Required** | @@ -211,8 +289,13 @@ List third-party domains that have access to sensitive data. ### reco-get-sensitive-assets-with-public-link +*** List sensitive assets exposed via a public link. +#### Base Command + +`reco-get-sensitive-assets-with-public-link` + #### Context Output | **Path** | **Type** | **Description** | @@ -224,8 +307,13 @@ List sensitive assets exposed via a public link. ### reco-get-files-shared-with-3rd-parties +*** List files shared with a specific third-party domain. +#### Base Command + +`reco-get-files-shared-with-3rd-parties` + #### Input | **Argument Name** | **Description** | **Required** | @@ -245,8 +333,13 @@ List files shared with a specific third-party domain. ### reco-change-alert-status +*** Update the status of a Reco alert. +#### Base Command + +`reco-change-alert-status` + #### Input | **Argument Name** | **Description** | **Required** | @@ -256,8 +349,13 @@ Update the status of a Reco alert. ### reco-get-user-context-by-email-address +*** Get identity context for a user by email address. +#### Base Command + +`reco-get-user-context-by-email-address` + #### Input | **Argument Name** | **Description** | **Required** | @@ -280,8 +378,13 @@ Get identity context for a user by email address. ### reco-get-files-exposed-to-email-address +*** List files accessible to a specific email address. +#### Base Command + +`reco-get-files-exposed-to-email-address` + #### Input | **Argument Name** | **Description** | **Required** | @@ -299,8 +402,13 @@ List files accessible to a specific email address. ### reco-get-assets-shared-externally +*** List files an owner has shared outside the organization. +#### Base Command + +`reco-get-assets-shared-externally` + #### Input | **Argument Name** | **Description** | **Required** | @@ -317,8 +425,13 @@ List files an owner has shared outside the organization. ### reco-get-private-email-list-with-access +*** List private (non-corporate) email addresses with file access. +#### Base Command + +`reco-get-private-email-list-with-access` + #### Context Output | **Path** | **Type** | **Description** | @@ -330,8 +443,13 @@ List private (non-corporate) email addresses with file access. ### reco-get-alert-ai-summary +*** Get an AI-generated summary of an alert. +#### Base Command + +`reco-get-alert-ai-summary` + #### Input | **Argument Name** | **Description** | **Required** | @@ -346,8 +464,13 @@ Get an AI-generated summary of an alert. ### reco-get-apps +*** List all discovered SaaS applications (auto-paginates all results). +#### Base Command + +`reco-get-apps` + #### Input | **Argument Name** | **Description** | **Required** | @@ -360,21 +483,27 @@ List all discovered SaaS applications (auto-paginates all results). | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Apps.id | String | App ID | -| Reco.Apps.name | String | App name | -| Reco.Apps.category | String | App category | -| Reco.Apps.usersCount | Number | Number of users | -| Reco.Apps.authorization | String | Authorization status | -| Reco.Apps.isUsingAi | Boolean | Whether the app uses AI | -| Reco.Apps.isShadowApp | Boolean | Whether the app is unsanctioned shadow IT | -| Reco.Apps.vendorGrade | String | Vendor security grade (A–F) | -| Reco.Apps.aiCapability | String | AI capability (AI-Native / AI-Assisted / No-AI) | -| Reco.Apps.lastSeen | Date | Last activity timestamp | +| Reco.Apps.id | String | The unique identifier of the application. | +| Reco.Apps.name | String | The name of the application. | +| Reco.Apps.category | String | The category of the application. | +| Reco.Apps.usersCount | Number | The number of users with access to the application. | +| Reco.Apps.authorization | String | The authorization/sanction status of the application. | +| Reco.Apps.authType | String | The authentication type used by the application. | +| Reco.Apps.isUsingAi | Boolean | Whether the application uses AI. | +| Reco.Apps.isShadowApp | Boolean | Whether the application is a shadow/unmanaged app. | +| Reco.Apps.vendorGrade | String | The vendor security grade of the application. | +| Reco.Apps.aiCapability | String | The AI capability description for the application. | +| Reco.Apps.lastSeen | Date | The last activity timestamp for the application. | ### reco-set-app-authorization-status +*** Update the authorization status of an application. +#### Base Command + +`reco-set-app-authorization-status` + #### Input | **Argument Name** | **Description** | **Required** | @@ -386,9 +515,10 @@ Update the authorization status of an application. | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.AppAuthorization.app_id | String | Application ID | -| Reco.AppAuthorization.authorization_status | String | New authorization status | -| Reco.AppAuthorization.updated | Boolean | Whether the update succeeded | +| Reco.AppAuthorization.app_id | String | The application ID that was updated. | +| Reco.AppAuthorization.authorization_status | String | The authorization status that was set. | +| Reco.AppAuthorization.updated | Boolean | Whether the update was successful. | +| Reco.AppAuthorization.rows_affected | Number | Number of rows affected by the update operation. | #### Command example @@ -398,8 +528,13 @@ Update the authorization status of an application. ### reco-add-exclusion-filter +*** Add values to a Reco classifier exclusion list. +#### Base Command + +`reco-add-exclusion-filter` + #### Input | **Argument Name** | **Description** | **Required** | @@ -415,8 +550,20 @@ All commands below accept `filters` (SCIM v2 expression) and `limit` (default 10 ### reco-list-events +*** List SaaS activity events. +#### Base Command + +`reco-list-events` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "actor.email eq "user@example.com" and eventTime gt "2024-01-01T00:00:00Z""). | Optional | +| limit | The maximum number of events to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | @@ -426,13 +573,26 @@ List SaaS activity events. | Reco.Events.formattedEventType | String | Human-readable event type | | Reco.Events.application | String | Source SaaS application | | Reco.Events.actorEmail | String | Actor email address | +| Reco.Events.actorName | String | Actor display name | | Reco.Events.eventTime | Date | Event timestamp | | Reco.Events.outcomeString | String | Event outcome description | ### reco-list-posture-issues +*** List security posture issues. +#### Base Command + +`reco-list-posture-issues` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "severity eq "HIGH""). | Optional | +| limit | The maximum number of posture issues to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | @@ -442,12 +602,25 @@ List security posture issues. | Reco.PostureIssues.severity | String | Severity (LOW/MEDIUM/HIGH/CRITICAL) | | Reco.PostureIssues.checkStatus | String | Check status | | Reco.PostureIssues.scorePercentage | Number | Compliance score percentage | +| Reco.PostureIssues.checkedInstance | Unknown | The SaaS instance this issue was checked against | | Reco.PostureIssues.url | String | Link to issue in Reco UI | ### reco-list-accounts +*** List SaaS accounts. +#### Base Command + +`reco-list-accounts` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "isRiskyUser eq true" or "accountEmail co "@example.com""). | Optional | +| limit | The maximum number of accounts to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | @@ -464,145 +637,296 @@ List SaaS accounts. ### reco-list-devices +*** List managed and unmanaged devices. +#### Base Command + +`reco-list-devices` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "isUnmanaged eq true" or "devicePlatform eq "Windows""). | Optional | +| limit | The maximum number of devices to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | | Reco.Devices.id | String | Device ID | | Reco.Devices.name | String | Device name | -| Reco.Devices.devicePlatform | String | Platform (Windows / macOS / iOS / Android) | -| Reco.Devices.isUnmanaged | Boolean | Whether the device is unmanaged | -| Reco.Devices.hasNonCompliant | Boolean | Whether the device has policy violations | -| Reco.Devices.lastSeen | Date | Last activity | +| Reco.Devices.devicePlatform | String | Device platform (Windows, macOS, iOS, Android, etc.) | +| Reco.Devices.os | String | Operating system of the device | +| Reco.Devices.osVersion | String | Operating system version | +| Reco.Devices.isUnmanaged | Boolean | Whether the device is unmanaged (not enrolled in MDM) | +| Reco.Devices.hasNonCompliant | Boolean | Whether the device has non-compliant policies | +| Reco.Devices.lastSeen | Date | Last activity timestamp | ### reco-list-ai-agents +*** List detected AI agents. +#### Base Command + +`reco-list-ai-agents` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "authorization eq "AUTH_STATUS_UNSANCTIONED""). | Optional | +| limit | The maximum number of AI agents to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | | Reco.AiAgents.id | String | AI agent ID | -| Reco.AiAgents.name | String | Agent name | -| Reco.AiAgents.vendor | String | Vendor | -| Reco.AiAgents.authorization | String | Authorization status | -| Reco.AiAgents.risk | String | Risk level | -| Reco.AiAgents.lastUsage | Date | Last usage | +| Reco.AiAgents.name | String | AI agent name | +| Reco.AiAgents.vendor | String | Vendor of the AI agent | +| Reco.AiAgents.type | String | Type of AI agent | +| Reco.AiAgents.authorization | String | Authorization/sanction status of the AI agent | +| Reco.AiAgents.agentStatus | String | Current status of the AI agent | +| Reco.AiAgents.risk | Number | Risk level of the AI agent (0=NA, 1=LOW, 2=MEDIUM, 3=HIGH, 4=CRITICAL) | +| Reco.AiAgents.lastUsage | Date | Last usage timestamp | ### reco-list-groups +*** List SaaS groups. +#### Base Command + +`reco-list-groups` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "name co "Engineering""). | Optional | +| limit | The maximum number of groups to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | | Reco.Groups.id | String | Group ID | | Reco.Groups.name | String | Group name | -| Reco.Groups.email | String | Group email | -| Reco.Groups.membersCount | Number | Member count | +| Reco.Groups.email | String | Group email address | +| Reco.Groups.membersCount | Number | Number of members in the group | +| Reco.Groups.appsCount | Number | Number of apps the group has access to | ### reco-list-saas-to-saas +*** List SaaS-to-SaaS OAuth grants. +#### Base Command + +`reco-list-saas-to-saas` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "authorization eq "AUTH_STATUS_UNSANCTIONED" or permissionRisk eq "30""). | Optional | +| limit | The maximum number of SaaS-to-SaaS grants to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.SaasToSaas.id | String | Grant ID | -| Reco.SaasToSaas.plugin | String | Plugin / app name | -| Reco.SaasToSaas.authorization | String | Authorization status | -| Reco.SaasToSaas.permissionRisk | String | Permission risk level | -| Reco.SaasToSaas.aiCapability | String | AI capability | +| Reco.SaasToSaas.id | String | SaaS-to-SaaS grant ID | +| Reco.SaasToSaas.plugin | String | The plugin or app name receiving the grant | +| Reco.SaasToSaas.authorization | String | Authorization status of the grant | +| Reco.SaasToSaas.permissionRisk | String | Permission risk level (10=LOW, 20=MEDIUM, 30=HIGH) | +| Reco.SaasToSaas.accounts | Number | Number of accounts with this grant | +| Reco.SaasToSaas.aiCapability | String | AI capability of the third-party app | +| Reco.SaasToSaas.lastSeen | Date | Last activity timestamp for this grant | ### reco-list-ip-addresses +*** List observed IP addresses. +#### Base Command + +`reco-list-ip-addresses` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "country eq "CN" or hasVpn eq true"). | Optional | +| limit | The maximum number of IP addresses to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.IpAddresses.ipAddress | String | IP address | -| Reco.IpAddresses.country | String | Country | -| Reco.IpAddresses.eventsCount | Number | Event count | -| Reco.IpAddresses.hasVpn | Boolean | VPN flag | -| Reco.IpAddresses.hasProxy | Boolean | Proxy flag | +| Reco.IpAddresses.ipAddress | String | The IP address or CIDR range | +| Reco.IpAddresses.country | String | Country of the IP address | +| Reco.IpAddresses.asnName | String | ASN name of the IP address | +| Reco.IpAddresses.eventsCount | Number | Number of events from this IP | +| Reco.IpAddresses.usersCount | Number | Number of users seen from this IP | +| Reco.IpAddresses.hasVpn | Boolean | Whether the IP is associated with a VPN | +| Reco.IpAddresses.hasProxy | Boolean | Whether the IP is associated with a proxy | +| Reco.IpAddresses.lastEventTime | Date | Last event timestamp from this IP | ### reco-list-business-units +*** List external business units. +#### Base Command + +`reco-list-business-units` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "name eq "Finance""). | Optional | +| limit | The maximum number of business units to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | | Reco.BusinessUnits.id | String | Business unit ID | | Reco.BusinessUnits.name | String | Business unit name | -| Reco.BusinessUnits.manager | String | Manager | +| Reco.BusinessUnits.manager | String | Manager of the business unit | +| Reco.BusinessUnits.createdAt | Date | Creation timestamp of the business unit | ### reco-list-audit-logs +*** List Reco platform audit logs. +#### Base Command + +`reco-list-audit-logs` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "userEmail eq "admin@example.com" and action eq "DELETE""). | Optional | +| limit | The maximum number of audit log entries to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.AuditLogs.id | String | Log entry ID | -| Reco.AuditLogs.userEmail | String | Actor email | -| Reco.AuditLogs.module | String | Module | +| Reco.AuditLogs.id | String | Audit log entry ID | +| Reco.AuditLogs.userEmail | String | Email of the user who performed the action | +| Reco.AuditLogs.module | String | Module where the action was performed | | Reco.AuditLogs.action | String | Action performed | -| Reco.AuditLogs.timestamp | Date | Timestamp | +| Reco.AuditLogs.objectName | String | Name of the object affected | +| Reco.AuditLogs.timestamp | Date | Timestamp of the audit log entry | +| Reco.AuditLogs.remoteAddr | String | Remote IP address of the actor | ### reco-list-posture-checks +*** List posture check definitions. +#### Base Command + +`reco-list-posture-checks` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "severity eq "HIGH" and apps co "Google""). | Optional | +| limit | The maximum number of posture check definitions to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.PostureChecks.id | String | Check ID | -| Reco.PostureChecks.name | String | Check name | -| Reco.PostureChecks.severity | String | Severity | -| Reco.PostureChecks.apps | Unknown | Applicable apps | +| Reco.PostureChecks.id | String | Posture check ID | +| Reco.PostureChecks.name | String | Posture check name | +| Reco.PostureChecks.severity | String | Severity of the posture check | +| Reco.PostureChecks.policyType | String | Policy type of the posture check | +| Reco.PostureChecks.apps | Unknown | Applications this posture check applies to | +| Reco.PostureChecks.type | String | Type of posture check (built-in or custom) | ### reco-list-threat-detection-policies +*** List threat detection policies. +#### Base Command + +`reco-list-threat-detection-policies` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "severity eq "HIGH" and status eq "ON""). | Optional | +| limit | The maximum number of policies to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | | Reco.ThreatDetectionPolicies.id | String | Policy ID | | Reco.ThreatDetectionPolicies.name | String | Policy name | -| Reco.ThreatDetectionPolicies.severity | String | Severity | -| Reco.ThreatDetectionPolicies.status | String | ON / OFF / PREVIEW | -| Reco.ThreatDetectionPolicies.openAlerts | Number | Open alerts | +| Reco.ThreatDetectionPolicies.severity | String | Severity of the policy | +| Reco.ThreatDetectionPolicies.status | String | Status of the policy (ON, OFF, or PREVIEW) | +| Reco.ThreatDetectionPolicies.apps | Unknown | Applications monitored by the policy | +| Reco.ThreatDetectionPolicies.openAlerts | Number | Number of open alerts triggered by this policy | +| Reco.ThreatDetectionPolicies.type | String | Type of policy (built-in or custom) | ### reco-list-exclusions +*** List alert suppression exclusion rules. +#### Base Command + +`reco-list-exclusions` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "policyName co "MFA""). | Optional | +| limit | The maximum number of exclusions to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | | --- | --- | --- | -| Reco.Exclusions.id | String | Exclusion ID | -| Reco.Exclusions.name | String | Exclusion name | -| Reco.Exclusions.policyName | String | Associated policy | -| Reco.Exclusions.createdBy | String | Created by | +| Reco.Exclusions.id | String | Exclusion rule ID | +| Reco.Exclusions.name | String | Exclusion rule name | +| Reco.Exclusions.policyName | String | Name of the policy this exclusion applies to | +| Reco.Exclusions.apps | Unknown | Applications this exclusion applies to | +| Reco.Exclusions.createdBy | String | User who created the exclusion | +| Reco.Exclusions.createdAt | Date | Creation timestamp of the exclusion | ### reco-list-app-instances +*** List integrated app instances (app portfolio). Only returns instances with an active integration status. +#### Base Command + +`reco-list-app-instances` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| filters | The SCIM v2 filter expression (e.g. "isUsingAi eq true"). | Optional | +| limit | The maximum number of app instances to return. Default is 1000. | Optional | + #### Context Output | **Path** | **Type** | **Description** | diff --git a/Packs/Reco/Integrations/Reco/Reco.py b/Packs/Reco/Integrations/Reco/Reco.py index cbdf33291f6..bff9e523c4f 100644 --- a/Packs/Reco/Integrations/Reco/Reco.py +++ b/Packs/Reco/Integrations/Reco/Reco.py @@ -23,6 +23,7 @@ FILTER_RELATIONSHIP_AND = "FILTER_RELATIONSHIP_AND" PAGE_SIZE = 1000 +MAX_PAGES = 1000 # hard cap on _paginate_all iterations; guards against an unbounded loop DEMISTO_OCCURRED_FORMAT = "%Y-%m-%dT%H:%M:%SZ" RECO_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ" @@ -77,10 +78,6 @@ def parse_minimum_risk_level(risk_level_param: str | None) -> list[str] | None: return SEVERITY_ORDER[min_index:] -def create_filter(field, value): - return {"field": field, "stringContains": {"value": value}} - - def extract_response(response: Any) -> list[dict[str, Any]]: if response.get("getTableResponse") is None: demisto.error(f"got bad response, {response}") @@ -182,10 +179,12 @@ def _paginate_all( Uses the `totalResults` field in the response to know when to stop. Stops early if an empty page is returned (defensive guard against totalResults being stale or the server having fewer items than reported). + Capped at MAX_PAGES to guard against an unbounded loop if the server + never reports totalResults or an empty page. """ all_items: list[dict[str, Any]] = [] start_index = 0 - while True: + for _page in range(MAX_PAGES): response = self._external_api_list( endpoint, filters=filters, @@ -201,6 +200,8 @@ def _paginate_all( if total and len(all_items) >= total: break start_index += len(page_items) + else: + demisto.debug(f"_paginate_all {endpoint}: reached MAX_PAGES ({MAX_PAGES}) cap, stopping") return all_items # ── Alerts (external API) ───────────────────────────────────────────────── @@ -1561,6 +1562,7 @@ def main() -> None: # pragma: no cover command = demisto.command() demisto.debug(f"Reco Command being called is {command}") params = demisto.params() + args = demisto.args() api_url = params.get("url") api_token = params.get("api_token") verify_certificate = not params.get("insecure", False) @@ -1605,24 +1607,24 @@ def main() -> None: # pragma: no cover demisto.incidents(incidents) elif command == "reco-add-comment-to-alert": - incident_id = demisto.args()["alert_id"] + incident_id = args["alert_id"] response = reco_client.update_reco_incident_timeline( incident_id=incident_id, - comment=demisto.args()["comment"], + comment=args["comment"], ) return_results(CommandResults(raw_response=response, readable_output=f"Comment added to alert {incident_id}")) elif command == "reco-update-incident-timeline": - incident_id = demisto.args()["incident_id"] + incident_id = args["incident_id"] response = reco_client.update_reco_incident_timeline( incident_id=incident_id, - comment=demisto.args()["comment"], + comment=args["comment"], ) return_results(CommandResults(raw_response=response, readable_output=f"Timeline updated for incident {incident_id}")) elif command == "reco-resolve-visibility-event": - entity_id = demisto.args()["entity_id"] - label_name = demisto.args()["label_name"] + entity_id = args["entity_id"] + label_name = args["label_name"] response = reco_client.resolve_visibility_event(entity_id=entity_id, label_name=label_name) return_results(CommandResults(raw_response=response, readable_output=f"Visibility event {entity_id} resolved")) @@ -1633,17 +1635,17 @@ def main() -> None: # pragma: no cover return_results(get_risky_users_from_reco(reco_client)) elif command == "reco-add-risky-user-label": - return_results(add_risky_user_label(reco_client, demisto.args()["email_address"])) + return_results(add_risky_user_label(reco_client, args["email_address"])) elif command == "reco-add-leaving-org-user-label": - return_results(add_leaving_org_user(reco_client, demisto.args()["email_address"])) + return_results(add_leaving_org_user(reco_client, args["email_address"])) elif command == "reco-get-assets-user-has-access-to": return_results( get_assets_user_has_access( reco_client, - demisto.args()["email_address"], - demisto.args().get("only_sensitive", False), + args["email_address"], + args.get("only_sensitive", False), ) ) @@ -1651,169 +1653,143 @@ def main() -> None: # pragma: no cover return_results( get_sensitive_assets_by_name( reco_client, - demisto.args()["asset_name"], - demisto.args().get("regex_search", False), + args["asset_name"], + args.get("regex_search", False), ) ) elif command == "reco-get-sensitive-assets-by-id": - return_results(get_sensitive_assets_by_id(reco_client, demisto.args()["asset_id"])) + return_results(get_sensitive_assets_by_id(reco_client, args["asset_id"])) elif command == "reco-get-link-to-user-overview-page": - return_results(get_link_to_user_overview_page(reco_client, demisto.args()["entity"], demisto.args()["param"])) + return_results(get_link_to_user_overview_page(reco_client, args["entity"], args["param"])) elif command == "reco-get-sensitive-assets-with-public-link": return_results(get_sensitive_assets_shared_with_public_link(reco_client)) elif command == "reco-get-3rd-parties-accessible-to-data-list": - return_results(get_3rd_parties_list(reco_client, int(demisto.args()["last_interaction_time_in_days"]))) + return_results(get_3rd_parties_list(reco_client, int(args["last_interaction_time_in_days"]))) elif command == "reco-get-files-shared-with-3rd-parties": return_results( get_files_shared_with_3rd_parties( reco_client, - demisto.args()["domain"], - int(demisto.args()["last_interaction_time_in_days"]), + args["domain"], + int(args["last_interaction_time_in_days"]), ) ) elif command == "reco-add-exclusion-filter": - return_results( - add_exclusion_filter(reco_client, demisto.args()["key_to_add"], argToList(demisto.args()["values_to_add"])) - ) + return_results(add_exclusion_filter(reco_client, args["key_to_add"], argToList(args["values_to_add"]))) elif command == "reco-change-alert-status": - return_results(change_alert_status(reco_client, demisto.args()["alert_id"], demisto.args()["status"])) + return_results(change_alert_status(reco_client, args["alert_id"], args["status"])) elif command == "reco-get-user-context-by-email-address": - return_results(get_user_context_by_email_address(reco_client, demisto.args()["email_address"])) + return_results(get_user_context_by_email_address(reco_client, args["email_address"])) elif command == "reco-get-files-exposed-to-email-address": - return_results(get_files_exposed_to_email_command(reco_client, demisto.args()["email_address"])) + return_results(get_files_exposed_to_email_command(reco_client, args["email_address"])) elif command == "reco-get-assets-shared-externally": - return_results(get_assets_shared_externally_command(reco_client, demisto.args()["email_address"])) + return_results(get_assets_shared_externally_command(reco_client, args["email_address"])) elif command == "reco-get-private-email-list-with-access": return_results(get_private_email_list_with_access(reco_client)) elif command == "reco-get-assets-by-id": - return_results(get_assets_by_id(reco_client, demisto.args()["asset_id"])) + return_results(get_assets_by_id(reco_client, args["asset_id"])) elif command == "reco-get-alert-ai-summary": - return_results(get_alert_ai_summary(reco_client, demisto.args().get("alert_id", ""))) + return_results(get_alert_ai_summary(reco_client, args.get("alert_id", ""))) elif command == "reco-get-apps": - before_dt = dateparser.parse(demisto.args()["before"]) if demisto.args().get("before") else None - after_dt = dateparser.parse(demisto.args()["after"]) if demisto.args().get("after") else None + before_dt = dateparser.parse(args["before"]) if args.get("before") else None + after_dt = dateparser.parse(args["after"]) if args.get("after") else None return_results( - get_apps_command( - reco_client, before=before_dt, after=after_dt, limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + get_apps_command(reco_client, before=before_dt, after=after_dt, limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-set-app-authorization-status": - return_results( - set_app_authorization_status_command( - reco_client, demisto.args()["app_id"], demisto.args()["authorization_status"] - ) - ) + return_results(set_app_authorization_status_command(reco_client, args["app_id"], args["authorization_status"])) elif command == "reco-list-events": return_results( - list_events_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_events_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-posture-issues": return_results( list_posture_issues_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE) ) ) elif command == "reco-list-accounts": return_results( - list_accounts_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_accounts_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-devices": return_results( - list_devices_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_devices_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-ai-agents": return_results( - list_ai_agents_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_ai_agents_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-groups": return_results( - list_groups_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_groups_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-saas-to-saas": return_results( - list_saas_to_saas_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_saas_to_saas_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-ip-addresses": return_results( - list_ip_addresses_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_ip_addresses_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-business-units": return_results( list_business_units_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE) ) ) elif command == "reco-list-audit-logs": return_results( - list_audit_logs_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_audit_logs_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-posture-checks": return_results( list_posture_checks_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE) ) ) elif command == "reco-list-threat-detection-policies": return_results( list_threat_detection_policies_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE) ) ) elif command == "reco-list-exclusions": return_results( - list_exclusions_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) - ) + list_exclusions_command(reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE)) ) elif command == "reco-list-app-instances": return_results( list_app_instances_command( - reco_client, filters=demisto.args().get("filters", ""), limit=int(demisto.args().get("limit") or PAGE_SIZE) + reco_client, filters=args.get("filters", ""), limit=int(args.get("limit") or PAGE_SIZE) ) ) diff --git a/Packs/Reco/Integrations/Reco/Reco.yml b/Packs/Reco/Integrations/Reco/Reco.yml index 8694add124b..eb2554d5edf 100644 --- a/Packs/Reco/Integrations/Reco/Reco.yml +++ b/Packs/Reco/Integrations/Reco/Reco.yml @@ -7,39 +7,46 @@ commonfields: id: Reco version: -1 configuration: -- display: Server URL (e.g. https://host.reco.ai/api/v1) +- additionalinfo: The base URL of your Reco instance. + display: Server URL (e.g. https://host.reco.ai/api/v1) name: url required: true type: 0 section: Connect - section: Connect + additionalinfo: The API token (Bearer) used to authenticate with the Reco External API. display: JWT app token displaypassword: API Token name: api_token type: 4 required: true hiddenusername: true -- display: Trust any certificate (not secure) +- additionalinfo: The option to skip TLS certificate verification when connecting to the Reco API. + display: Trust any certificate (not secure) name: insecure type: 8 required: false section: Connect -- display: Use system proxy settings +- additionalinfo: The option to route requests through the system proxy. + display: Use system proxy settings name: proxy type: 8 required: false section: Connect -- display: Incident type +- additionalinfo: The incident type to map Reco alerts to. + display: Incident type name: incidentType type: 13 required: false section: Collect -- display: Fetch incidents +- additionalinfo: The option to enable automatic incident fetching. + display: Fetch incidents name: isFetch type: 8 required: false section: Collect -- defaultvalue: '200' +- additionalinfo: The maximum number of incidents to fetch per run (up to 500). + defaultvalue: '200' display: Max fetch name: max_fetch type: 0 @@ -63,19 +70,20 @@ configuration: type: 0 required: false section: Collect -- additionalinfo: 'Minimum severity threshold for fetched incidents. Accepts a single value: LOW, MEDIUM, HIGH, or CRITICAL (or numeric equivalents 10, 20, 30, 40). Alerts at or above this severity are fetched. Example: "MEDIUM" fetches medium, high, and critical severity alerts.' +- additionalinfo: 'The minimum severity threshold for fetched incidents. Accepts a single value: LOW, MEDIUM, HIGH, or CRITICAL (or numeric equivalents 10, 20, 30, 40). Alerts at or above this severity are fetched. For example, "MEDIUM" fetches medium, high, and critical severity alerts.' display: Minimum risk level (e.g. MEDIUM fetches medium and higher) name: risk_level type: 0 required: false section: Collect -- defaultvalue: 7 days +- additionalinfo: The amount of time to look back on the first fetch run (e.g. `7 days`, `12 hours`). + defaultvalue: 7 days display: First fetch timestamp (