From cc2a620517af5b9eadb1eb9d54baa94d6a05c6a0 Mon Sep 17 00:00:00 2001 From: Stan Rymkiewicz <149009265+stan-rym@users.noreply.github.com> Date: Sun, 18 Oct 2026 20:47:48 -0400 Subject: [PATCH] HubSpot audiences Add audience_from_hubspot and `adwarp audience from-hubspot`: a HubSpot list's members, or a CRM search by property filters, becomes a LinkedIn matched audience of contacts or companies, the way Salesforce audiences work. - Auth is a read-only HubSpot private app token (crm.objects.contacts.read, crm.objects.companies.read, crm.lists.read) from ADWARP_HUBSPOT_TOKEN or ~/.adwarp/hubspot.json. No OAuth. - Reads every page through HubSpot's 2026-09 date-versioned API (search, list memberships, batch read), paced under the search and private app limits, and refuses a search above HubSpot's 10,000 result cap. - Records go through the CSV cleaner and upload path (cleanAudienceRows, uploadCleanedAudience), so contacts leave only as SHA256 hashes. The result reports read, valid, uploaded and skipped with reasons. - dryRun reads and cleans, returns counts and a masked sample, and makes no LinkedIn call. - Local only: the hosted route registers tools with hubspot: false, so the tool is hosted: false in tools.json. - Company CSV uploads now also respect the 300,000 row cap. - Tests stub fetch for HubSpot and use a fake LinkedIn client. - New docs/hubspot.md; tools, cli, config, hosted, index, README, AGENTS and the adwarp-audiences skill updated; tools.json regenerated. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 40 +- CONTRIBUTING.md | 2 +- README.md | 36 +- apps/web/app/api/[transport]/route.ts | 9 +- docs/cli.md | 1 + docs/clients-claude-code.md | 2 +- docs/clients-claude-desktop.md | 2 +- docs/clients-cli.md | 2 +- docs/clients-codex.md | 2 +- docs/clients-cursor.md | 2 +- docs/config.md | 3 + docs/hosted.md | 3 +- docs/hubspot.md | 106 +++++ docs/index.md | 3 +- docs/platforms-linkedin-ads.md | 6 +- docs/safety.md | 2 +- docs/skills.md | 2 +- docs/tools.json | 88 ++++ docs/tools.md | 1 + packages/cli/src/index.ts | 47 +++ packages/core/src/audience.ts | 59 ++- packages/core/src/audienceCsv.ts | 9 + packages/core/src/hubspot.ts | 557 ++++++++++++++++++++++++++ packages/core/src/index.ts | 3 +- packages/core/src/schemas.ts | 49 +++ packages/core/test/hubspot.test.mjs | 326 +++++++++++++++ packages/mcp/scripts/gen-tools.mjs | 6 +- packages/mcp/src/tools.ts | 29 ++ packages/shared/src/env.ts | 8 +- skills/README.md | 2 +- skills/adwarp-audiences/SKILL.md | 19 +- 31 files changed, 1375 insertions(+), 51 deletions(-) create mode 100644 docs/hubspot.md create mode 100644 packages/core/src/hubspot.ts create mode 100644 packages/core/test/hubspot.test.mjs diff --git a/AGENTS.md b/AGENTS.md index 4f3a991..415cbcd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,8 @@ pnpm + TypeScript monorepo: math every platform shares. Anything here must be true of both platforms; anything true of only one belongs in that platform's package. - `packages/core` — the LinkedIn engine. No UI. Owns the LinkedIn REST client, every - resource module, audience hashing, targeting, conversions, and the Salesforce reader. + resource module, audience hashing, targeting, conversions, and the Salesforce and + HubSpot readers. - `packages/google` — the Google Ads engine. Same shape as core: config and auth, one HTTP chokepoint, thin typed resource modules over GAQL, zod schemas reused as MCP inputs. - `packages/mcp` — MCP server. `src/tools.ts` (LinkedIn) and `src/googleTools.ts` @@ -53,14 +54,16 @@ modules are thin typed wrappers over `LinkedInClient.request()`. `core/src/schemas.ts` and are reused as MCP tool input schemas (`Schema.shape`). Add or change a field there first, then thread it through the resource module. - **Secrets never enter the repo.** Local credentials live in `~/.adwarp/` (LinkedIn: - `config.json` + `credentials.json`; Google: `google.json` + `google-credentials.json`, - all mode 0600). Hosted credentials are `ADWARP_LINKEDIN_*` / `ADWARP_GOOGLE_*` env vars, or + `config.json` + `credentials.json`; Google: `google.json` + `google-credentials.json`; + HubSpot: `hubspot.json`, all mode 0600). Hosted credentials are `ADWARP_LINKEDIN_*` / `ADWARP_GOOGLE_*` env vars, or per-request `X-Adwarp-Linkedin-*` headers for bring-your-own-credentials callers. The config layer resolves request context first, then env, then files. Never log header credentials. - **The hosted MCP does not expose the Google tools**, and must not start doing so. That endpoint is multi-tenant over per-request LinkedIn credentials while Google credentials would come from shared server env vars, so exposing them would let any caller operate the - server's Google Ads account. `registerTools(server, { google: false })` in the web route. + server's Google Ads account. The same holds for `audience_from_hubspot`, whose token + would come from a server env var: `registerTools(server, { google: false, hubspot: false })` + in the web route. - **Every name says adwarp, and every pre-rename name keeps working.** Adwarp was Liam, and its internal names used `liads` and `gads`. Read env vars through `env()` in `@adwarp/shared`, whose `LEGACY_ENV_NAMES` table maps each `ADWARP_*` name to its @@ -206,3 +209,32 @@ change by initializing the MCP endpoint and listing tools (see README). `core/src/salesforce.ts` shells out to the authenticated `sf` CLI (`sf data query --json`). No new credentials; reuses the user's existing `sf` login. Read-only. `audienceFromSalesforce` turns a SOQL email query into a matched-audience DMP segment (reuses `uploadAudienceFromEmails`). + +## HubSpot + +`core/src/hubspot.ts` reads HubSpot with a private app access token the user creates in their +own account (`ADWARP_HUBSPOT_TOKEN`, else `{ "accessToken" }` in `hubspot.json` in the Adwarp +folder). Read-only scopes: `crm.objects.contacts.read`, `crm.objects.companies.read`, +`crm.lists.read`. No OAuth, local only (never registered on the hosted endpoint). +`audienceFromHubSpot` turns a list's members or a CRM search into rows, runs them through +`cleanAudienceRows` (the CSV cleaner) and uploads with `uploadCleanedAudience` (the CSV upload), +so contacts leave only as SHA256 hashes. `previewHubSpotAudience` is the dry run: no LinkedIn +call, emails masked in the sample. Verified against HubSpot's docs on 2026-10-18, not yet +against a live portal: + +- **Version is pinned** (`2026-09`, date-based, in `HUBSPOT_API_VERSION`). HubSpot moved from + v3 paths to `/crm/objects/2026-09/...` and `/crm/lists/2026-09/...` in March 2026; legacy v3 + paths stop being supported in September 2027, and each date version gets 18 months. +- **Search** is `POST /crm/objects/2026-09/{contacts|companies}/search`: `filterGroups[].filters[]` + of `{ propertyName, operator, value | values | highValue }`, 200 per page, cursor in + `paging.next.after`, at most 10,000 results per query (paging past it is a 400, so Adwarp + refuses up front from `total`), 5 requests per second, up to 6 filters per group. +- **Lists:** `GET /crm/lists/2026-09/{listId}` gives `list.objectTypeId` (`0-1` contacts, `0-2` + companies); `GET .../{listId}/memberships` pages `results[].recordId`. Records then come from + `POST /crm/objects/2026-09/{type}/batch/read`, 100 ids per call, which answers 207 when some + ids no longer exist. +- **Limits:** private apps get 100 requests per 10 seconds on Free and Starter (190 above). + The client paces under both and retries 429 / 5xx with the shared policy. +- **Change journal:** audience creation goes through `createLinkedInAds()` like Salesforce, so + it is captured exactly as Salesforce audiences are. The journal only tracks campaign groups, + campaigns and creatives, so DMP segment creation is not journaled for either source. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ca2a6ac..fdf0c9c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,7 +12,7 @@ pnpm -r typecheck ## Layout -- `packages/core`: LinkedIn REST client, OAuth, resource modules, audience hashing, Salesforce reader. +- `packages/core`: LinkedIn REST client, OAuth, resource modules, audience hashing, Salesforce and HubSpot readers. - `packages/mcp`: MCP server. `src/tools.ts` registers tools; `src/index.ts` is the stdio entry. - `packages/cli`: the `adwarp` CLI (also installed as `liam`, the legacy alias). - `packages/adwarp`: the npm package `adwarp`, which bundles the CLI and the MCP server. Test diff --git a/README.md b/README.md index 1be29c6..67f1267 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ campaigns can only be switched on in the Google Ads UI. > Unofficial and not affiliated with or endorsed by LinkedIn or Google. -> Covers creation, matched audiences (including building one straight from Salesforce), +> Covers creation, matched audiences (including building one straight from Salesforce or HubSpot), > conversion selection, performance reporting/insights, competitor ad intelligence, and a > change journal with before/after lift. See [ROADMAP.md](./ROADMAP.md) for what's shipped > and what's planned next. @@ -31,6 +31,8 @@ handles today, roughly in the order a campaign comes together: company domains to URLs) - "Build an audience from Salesforce: every contact on an account flagged as a target." (SOQL straight to a matched audience) +- "Turn our HubSpot target accounts list into a LinkedIn audience." (a HubSpot list or search + straight to a matched audience of contacts or companies) - "Launch a draft campaign for that audience, $100/day, tracking the demo-booked conversion." (campaign group + campaign + draft ads in one call) - "What ads is our biggest competitor running right now, and what offers are they pushing?" (reads the @@ -157,7 +159,7 @@ steps, plus the `ADWARP_GOOGLE_*` environment variables. ## Packages - `@adwarp/shared`: the platform-neutral layer: OAuth, credential stores, retry policy, the change journal, period and KPI math. -- `@adwarp/core`: LinkedIn REST client, resource modules, CSV + SHA256 hashing, Salesforce reader. +- `@adwarp/core`: LinkedIn REST client, resource modules, CSV + SHA256 hashing, Salesforce and HubSpot readers. - `@adwarp/google`: Google Ads REST client, GAQL reads, atomic mutate builder, Search campaign creation. - `@adwarp/mcp`: MCP server (stdio for local; reused by the hosted app). **Primary interface.** - `@adwarp/cli`: the `adwarp` CLI over the same engines, for scripted batch runs. @@ -318,6 +320,7 @@ so treat these credentials like an ad account password. Hosted limitations (all by design, they need local resources): `upload_audience_csv` reads a CSV path on the server, `audience_from_salesforce` needs your local `sf` CLI, +`audience_from_hubspot` is not registered there (its HubSpot token stays on your machine), competitor-ads scraping falls back to API-only metadata (no browser), and the change journal / lift tools need the local `~/.adwarp/changelog.jsonl`. Run those through the local MCP server or CLI. @@ -461,7 +464,7 @@ Operate: - **adwarp-launch**, "launch a campaign": brief to confirmed draft with house guardrails applied - **adwarp-experiments**, "did that change help": hypothesis logging plus before/after lift -- **adwarp-audiences**, "audit my audiences": health, dry-run uploads, Salesforce refresh loop +- **adwarp-audiences**, "audit my audiences": health, dry-run uploads, Salesforce and HubSpot refresh loops - **adwarp-account-audit**, "is my account set up right": hygiene scorecard and fix list ```bash @@ -485,7 +488,8 @@ whether it works on the hosted endpoint. estimates reach before building the campaign. - **Audiences:** `upload_audience_csv` (auto-cleans the CSV: normalizes column names, drops non-matcher columns, hashes emails, converts company domains to website URLs; supports both - contact and company lists), `audience_from_salesforce` (SOQL to matched audience), `get_audience_status` + contact and company lists), `audience_from_salesforce` (SOQL to matched audience), + `audience_from_hubspot` (HubSpot list or search to matched audience, local only), `get_audience_status` - **Conversions:** `list_conversions` (select an existing insight-tag conversion to track). `create_campaign` and `launch_from_brief` accept `conversionIds` or `conversionName`, and fall back to `defaultConversionName` from config. @@ -537,6 +541,8 @@ adwarp targeting estimate # audience size for one facet's URNs adwarp audience upload -n -f # clean + upload a CSV as a matched audience adwarp audience upload -n -f --dry-run # preview the cleaned CSV, no upload adwarp audience from-salesforce -n -q "" # Salesforce query -> matched audience +adwarp audience from-hubspot -n --list [--dry-run] # HubSpot list -> matched audience +adwarp audience from-hubspot -n --filter :: [-t company] # HubSpot search -> matched audience adwarp audience status # matching status + resolved size adwarp conversions list # account conversions (pick one to track) adwarp campaigns list [--group ] [--all] # campaign groups + ad groups, drafts included @@ -597,7 +603,9 @@ Local config lives in `~/.adwarp/config.json` (mode 0600, never in the repo): | `mcpAuthToken` | Bearer secret for the hosted MCP endpoint | Hosted equivalents are the `ADWARP_LINKEDIN_*` env vars plus `MCP_AUTH_TOKEN` (see [Configuration](https://adwarp.dev/docs/config)). -OAuth tokens are stored separately in `~/.adwarp/credentials.json`. +OAuth tokens are stored separately in `~/.adwarp/credentials.json`. A HubSpot private app +token, for [HubSpot audiences](https://adwarp.dev/docs/hubspot), goes in `~/.adwarp/hubspot.json` (mode 0600) +or `ADWARP_HUBSPOT_TOKEN`. Coming from Liam? Every name from before the rename still works: the `~/.liads` folder, the `LIADS_*` and `GADS_*` variables, and the `X-Liads-*` headers. See @@ -616,6 +624,24 @@ adwarp audience from-salesforce -n "Q3 target accounts" \ -q "SELECT Email FROM Contact WHERE Account.Target_List__c = true AND Email != null" ``` +## HubSpot integration + +Adwarp reads HubSpot with a private app access token you create in your own HubSpot account, +with 3 read-only scopes: `crm.objects.contacts.read`, `crm.objects.companies.read` and +`crm.lists.read`. Put the token in `~/.adwarp/hubspot.json` as `{ "accessToken": "..." }` +(mode 0600) or in `ADWARP_HUBSPOT_TOKEN`. `audience_from_hubspot` (and +`adwarp audience from-hubspot`) take a HubSpot list id or CRM search filters, read every +page, and turn the contacts or companies into a matched audience through the same cleaning +and hashing as a CSV upload, so only email hashes leave your machine. It runs locally only. +Example: + +``` +adwarp audience from-hubspot -n "Q3 target accounts" --list 123 --dry-run +adwarp audience from-hubspot -n "Marketing qualified leads" --filter lifecyclestage:EQ:marketingqualifiedlead +``` + +See [HubSpot audiences](https://adwarp.dev/docs/hubspot) for creating the private app and the limits. + ## Safety - Every campaign/creative is created **DRAFT/PAUSED**. Activation is a separate, explicit step. diff --git a/apps/web/app/api/[transport]/route.ts b/apps/web/app/api/[transport]/route.ts index ff8b18a..c60aa47 100644 --- a/apps/web/app/api/[transport]/route.ts +++ b/apps/web/app/api/[transport]/route.ts @@ -26,10 +26,11 @@ import { registerTools } from "@adwarp/mcp/tools"; const mcpHandler = createMcpHandler( // The adapter's server is the same MCP SDK type; cast across the version boundary. // eslint-disable-next-line @typescript-eslint/no-explicit-any - // Google tools are withheld here on purpose: this endpoint is multi-tenant - // over per-request LinkedIn credentials, and Google credentials would come - // from shared server env vars. See RegisterToolsOptions. - (server) => registerTools(server as any, { google: false }), + // Google tools and HubSpot audiences are withheld here on purpose: this + // endpoint is multi-tenant over per-request LinkedIn credentials, and Google + // and HubSpot credentials would come from shared server env vars. See + // RegisterToolsOptions. + (server) => registerTools(server as any, { google: false, hubspot: false }), { serverInfo: { name: "adwarp", version: VERSION } }, { basePath: "/api" }, ); diff --git a/docs/cli.md b/docs/cli.md index b622d37..0e44d7b 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -41,6 +41,7 @@ adwarp targeting estimate # audience size for 1 facet's URNs adwarp audience upload -n -f [-t contact|company] # clean and upload a CSV as a matched audience adwarp audience upload -n -f --dry-run # preview the cleaned CSV, no upload adwarp audience from-salesforce -n -q "" [--email-field ] # Salesforce query to matched audience +adwarp audience from-hubspot -n (--list | --filter :: ...) [-t contact|company] [--dry-run] # HubSpot list or search to matched audience adwarp audience status # matching status and resolved size adwarp conversions list # account conversions (pick 1 to track) adwarp campaigns list [--group ] [--all] # campaign groups and ad groups, drafts included diff --git a/docs/clients-claude-code.md b/docs/clients-claude-code.md index 564d549..ee4b6e1 100644 --- a/docs/clients-claude-code.md +++ b/docs/clients-claude-code.md @@ -1,7 +1,7 @@ --- title: Claude Code description: "Add Adwarp to Claude Code as a local MCP server or through the hosted endpoint, then check that it works." -order: 13 +order: 14 --- Claude Code runs Adwarp as a local MCP server with every tool, or connects to the hosted endpoint for the LinkedIn tools. Either way, [create a LinkedIn app](./linkedin-app.md) and [authenticate](./install.md#authenticate) first. diff --git a/docs/clients-claude-desktop.md b/docs/clients-claude-desktop.md index 066599d..23cdf87 100644 --- a/docs/clients-claude-desktop.md +++ b/docs/clients-claude-desktop.md @@ -1,7 +1,7 @@ --- title: Claude Desktop description: "Add Adwarp to Claude Desktop's MCP config, restart the app, and check that it works." -order: 14 +order: 15 --- Claude Desktop runs Adwarp as a local MCP server with every tool, including the Google Ads tools. [Create a LinkedIn app](./linkedin-app.md) and [authenticate](./install.md#authenticate) first. diff --git a/docs/clients-cli.md b/docs/clients-cli.md index 2788838..ee13e8b 100644 --- a/docs/clients-cli.md +++ b/docs/clients-cli.md @@ -1,7 +1,7 @@ --- title: CLI description: "Run Adwarp from your terminal with no AI client: install the adwarp command, check that it works, and put it on a schedule." -order: 17 +order: 18 --- Every capability of the MCP server is also an `adwarp` command, for your terminal, scripts and cron. It needs no AI client. [Create a LinkedIn app](./linkedin-app.md) first. diff --git a/docs/clients-codex.md b/docs/clients-codex.md index 5ab066b..f17e5c7 100644 --- a/docs/clients-codex.md +++ b/docs/clients-codex.md @@ -1,7 +1,7 @@ --- title: Codex description: "Add Adwarp to the Codex CLI with 1 command, then check that it works." -order: 15 +order: 16 --- The Codex CLI runs Adwarp as a local MCP server with every tool, including the Google Ads tools. [Create a LinkedIn app](./linkedin-app.md) and [authenticate](./install.md#authenticate) first. diff --git a/docs/clients-cursor.md b/docs/clients-cursor.md index c3f41c9..4322e35 100644 --- a/docs/clients-cursor.md +++ b/docs/clients-cursor.md @@ -1,7 +1,7 @@ --- title: Cursor description: "Add Adwarp to Cursor with 1 click or 1 config entry, then check that it works." -order: 16 +order: 17 --- Cursor runs Adwarp as a local MCP server with every tool, including the Google Ads tools. [Create a LinkedIn app](./linkedin-app.md) and [authenticate](./install.md#authenticate) first. diff --git a/docs/config.md b/docs/config.md index b3dca87..677da45 100644 --- a/docs/config.md +++ b/docs/config.md @@ -16,6 +16,8 @@ Local config lives in `~/.adwarp/config.json` (mode 0600, never in the repo). OA Google Ads has its own files in the same folder: `~/.adwarp/google.json` and `~/.adwarp/google-credentials.json`. See [Google Ads](./google-ads.md). +HubSpot audiences read a private app token from `~/.adwarp/hubspot.json` (mode 0600), as `{ "accessToken": "..." }`. See [HubSpot audiences](./hubspot.md). + ## Hosted equivalents A hosted deploy reads environment variables instead of files: @@ -34,6 +36,7 @@ A hosted deploy reads environment variables instead of files: | Variable | Purpose | | --- | --- | +| `ADWARP_HUBSPOT_TOKEN` | HubSpot private app token for [HubSpot audiences](./hubspot.md), used instead of `~/.adwarp/hubspot.json` | | `ADWARP_CHANGELOG_PATH` | override the change journal path | | `ADWARP_NO_CHANGELOG` | set to `1` to turn off change journal auto-capture | | `ADWARP_CREATIVE_WORKER_URL`, `ADWARP_CREATIVE_WORKER_TOKEN` | the [remote creative worker](https://github.com/adwarp-dev/adwarp/tree/main/services/creative-worker) for competitor ad copy | diff --git a/docs/hosted.md b/docs/hosted.md index b0943bf..c9cb3c7 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -49,11 +49,12 @@ These are by design, because they need local resources. Run them through the [lo - `upload_audience_csv` reads a CSV path on the server, not on your machine. - `audience_from_salesforce` needs your local `sf` CLI. +- `audience_from_hubspot` is not registered on the hosted endpoint. Its HubSpot token would come from a server environment variable, so any caller could read that HubSpot account's contacts. - `inspect_competitor_ads` returns Ad Library metadata only, unless the server is configured with the [remote creative worker](https://github.com/adwarp-dev/adwarp/tree/main/services/creative-worker). Nothing launches a browser. - The change journal and lift tools (`log_ad_change`, `list_ad_changes`, `compute_lift`) need the local `~/.adwarp/changelog.jsonl`. Journaling is off on hosted deploys, which have a read-only filesystem. - The Google Ads `gads_*` tools are not registered on the hosted endpoint at all. It serves many callers, each with their own LinkedIn credentials, while Google credentials would come from shared server environment variables. Exposing them would let any caller operate the server's Google Ads account. -[tools.json](./tools.json) says which tools work here. `hosted` is `true` for a tool that works on the hosted endpoint, and `hostedSupport` is `full`, `limited` or `none`. The endpoint registers every LinkedIn tool, but `upload_audience_csv`, `audience_from_salesforce` and the 3 journal tools need your machine, so they are `hosted: false`. `inspect_competitor_ads` is `limited`. `hostedNote` says why. +[tools.json](./tools.json) says which tools work here. `hosted` is `true` for a tool that works on the hosted endpoint, and `hostedSupport` is `full`, `limited` or `none`. The endpoint registers every LinkedIn tool except `audience_from_hubspot`. That tool, `upload_audience_csv`, `audience_from_salesforce` and the 3 journal tools need your machine, so all 6 are `hosted: false`. `inspect_competitor_ads` is `limited`. `hostedNote` says why. ## Put a skill on top diff --git a/docs/hubspot.md b/docs/hubspot.md new file mode 100644 index 0000000..c502e08 --- /dev/null +++ b/docs/hubspot.md @@ -0,0 +1,106 @@ +--- +title: HubSpot audiences +description: "Turn a HubSpot list or a CRM search into a LinkedIn matched audience, using a read-only private app token from your own HubSpot account." +order: 12 +--- + +Adwarp reads contacts and companies from HubSpot with a private app access token that you create in your own HubSpot account. The token only needs read scopes, and Adwarp never writes to HubSpot. + +Point it at a HubSpot list, or describe the records with search filters, and the result becomes a matched audience, closing the loop from "the accounts we track in HubSpot" to "LinkedIn targeting". The `audience_from_hubspot` tool and the `adwarp audience from-hubspot` command do the same thing: + +```bash +adwarp audience from-hubspot -n "Q3 target accounts" --list 123 --dry-run +adwarp audience from-hubspot -n "Q3 target accounts" --list 123 +``` + +## What it does + +- Reads every record: all pages of a list's members, or all pages of a search. +- Contacts become a contact list, matched on email. Adwarp reads only the `email` property, then trims and lowercases it, drops blank, malformed and duplicate addresses, and SHA-256 hashes the rest. Only the hashes leave your machine. +- Companies become a company list, matched on name and website. Adwarp reads `name` and `domain` and turns each domain into an `https://` website URL. +- The records go through the same cleaning, hashing and upload as [`upload_audience_csv`](./tools.md#audiences), so a HubSpot list and the same list exported to a CSV produce the same audience. +- It reports the records read, how many are valid, how many uploaded, and how many were skipped and why: no email, invalid email, duplicate email, no name or domain, or a list member that no longer exists in HubSpot. + +Always run it with `--dry-run` (or `dryRun: true` on the tool) first. A dry run reads HubSpot, shows the counts and a sample of 5 cleaned rows, and uploads nothing. Contact emails in the sample are masked, as in `ad***@example.com`, next to the SHA-256 hash that would upload. + +## Create the private app + +HubSpot lists these under legacy apps, and creating one takes a super admin. + +1. In HubSpot, go to **Development**, then click **Legacy apps** in the left sidebar. +2. Click **Create legacy app**, then choose **Private**. +3. Give it a name, such as "Adwarp audiences". +4. On the **Scopes** tab, click **Add new scope** and select these 3 read scopes: + - `crm.objects.contacts.read` + - `crm.objects.companies.read` + - `crm.lists.read` +5. Click **Create app**. +6. On the app's **Auth** tab, click **Show token**, then copy it. + +Add no other scopes. Adwarp only reads, so a token with read scopes alone is all it needs. + +## Where the token goes + +Save the token in `~/.adwarp/hubspot.json`, next to your other Adwarp credentials: + +```json +{ "accessToken": "your-private-app-token" } +``` + +Then make the file readable only by you: + +```bash +chmod 600 ~/.adwarp/hubspot.json +``` + +Or set the `ADWARP_HUBSPOT_TOKEN` environment variable wherever the MCP server or the CLI runs. When both are set, the environment variable wins. If your install still uses the `~/.liads` folder from before the rename, the file goes there instead. + +HubSpot recommends rotating a private app token every 6 months, from the app's **Auth** tab. Update the file or the variable when you do. + +## Examples + +From a list. HubSpot now calls lists segments: to find a list's id, open **Segments** in HubSpot, hover over the list, and click **Details**. A contact list builds a contact audience and a company list builds a company audience, so `-t` is optional here: + +```bash +adwarp audience from-hubspot -n "Open opportunities" --list 123 --dry-run +``` + +From a search. Each `--filter` is `::`, using the property's internal name in HubSpot, and every filter must match. A search builds a contact audience unless you pass `-t company`: + +```bash +adwarp audience from-hubspot -n "Marketing qualified leads at acme.com" \ + --filter lifecyclestage:EQ:marketingqualifiedlead \ + --filter "email:CONTAINS_TOKEN:*@acme.com" --dry-run + +adwarp audience from-hubspot -n "Large accounts" -t company \ + --filter annualrevenue:GT:10000000 --dry-run +``` + +The operators are HubSpot's: `EQ`, `NEQ`, `LT`, `LTE`, `GT`, `GTE`, `BETWEEN`, `IN`, `NOT_IN`, `HAS_PROPERTY`, `NOT_HAS_PROPERTY`, `CONTAINS_TOKEN` and `NOT_CONTAINS_TOKEN`. `IN` and `NOT_IN` take a comma-separated list, `BETWEEN` takes `low,high`, and `HAS_PROPERTY` and `NOT_HAS_PROPERTY` take no value (`email:HAS_PROPERTY`). For `IN` on a text property, HubSpot expects the values in lowercase. + +Through the MCP tool, the same search is: + +```json +{ + "name": "Marketing qualified leads at acme.com", + "filters": [ + { "property": "lifecyclestage", "operator": "EQ", "value": "marketingqualifiedlead" }, + { "property": "email", "operator": "CONTAINS_TOKEN", "value": "*@acme.com" } + ], + "dryRun": true +} +``` + +Pass `accountId` to choose the ad account, or leave it out to use `defaultAccountId` from your [config](./config.md). + +## Limits + +- **10,000 records per search.** HubSpot's search returns at most 10,000 results for a query, so a broader search stops with an error before anything uploads. Narrow the filters, or save the records as a HubSpot list and use `--list`, which has no such cap. +- **6 filters, all ANDed.** HubSpot allows up to 6 filters in a group. For OR logic, build a list in HubSpot and use that. +- **Request pacing.** HubSpot's search allows 5 requests per second, and a private app gets 100 requests every 10 seconds on Free and Starter. Adwarp paces itself under both and retries when HubSpot answers 429. A list is read 100 members per page and 100 records per batch, so a 300,000 member list takes about 6,000 requests, or roughly 10 minutes. +- **Contacts match on their primary email.** Additional email addresses on a contact are not read. +- **Matched audience rules.** These are the same as any upload: at most 300,000 rows per list, at least 10,000 contacts or 1,000 companies recommended, and about 300 matched members before a campaign can serve. Matching takes up to 48 hours. +- **Refreshing.** Each run creates a new segment. To refresh an audience, run it again and swap the new segment into the campaign's targeting. +- **Local only.** This runs through the local MCP server or the CLI, never the [hosted endpoint](./hosted.md), which does not register the tool. A token stored on a shared server would let any caller read that HubSpot account. + +This needs the Audiences product on your [LinkedIn app](./linkedin-app.md). Adwarp calls HubSpot's date-versioned API, version `2026-09`. diff --git a/docs/index.md b/docs/index.md index f0ee89d..ff68516 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,7 +4,7 @@ description: "The open-source ad agent for LinkedIn Ads and Google Ads. Create c order: 1 --- -Adwarp (formerly Liam) is the open-source ad agent for LinkedIn Ads and Google Ads. You describe the campaign in plain language and Adwarp drafts the audience, the ad groups, and the ads, from Claude Code, Claude Desktop, Cursor, Codex, or other MCP clients that can run a local server, or from a CLI. It covers creation, matched audiences (including building one straight from Salesforce), conversion selection, performance reporting, competitor ad intelligence, and a change journal with before and after lift. +Adwarp (formerly Liam) is the open-source ad agent for LinkedIn Ads and Google Ads. You describe the campaign in plain language and Adwarp drafts the audience, the ad groups, and the ads, from Claude Code, Claude Desktop, Cursor, Codex, or other MCP clients that can run a local server, or from a CLI. It covers creation, matched audiences (including building one straight from Salesforce or HubSpot), conversion selection, performance reporting, competitor ad intelligence, and a change journal with before and after lift. Everything Adwarp creates starts switched off. LinkedIn entities are created as drafts and Google Ads entities as paused. A LinkedIn campaign goes live only when you explicitly ask Adwarp to turn it on or switch it on yourself in Campaign Manager, and Google campaigns can only be switched on in the Google Ads UI. @@ -17,6 +17,7 @@ Once Adwarp is connected, you talk to your assistant in plain language. These ar - "How many VPs of demand gen at US SaaS companies can we reach?" Adwarp resolves the targeting facets and estimates reach before any money is involved. - "Upload this CSV as a matched audience." Adwarp cleans the columns, hashes emails, and converts company domains to URLs. - "Build an audience from Salesforce: every contact on an account flagged as a target." A SOQL query goes straight to a matched audience. +- "Turn our HubSpot target accounts list into a LinkedIn audience." A HubSpot list or search becomes a matched audience of contacts or companies. - "Launch a draft campaign for that audience, $100/day, tracking the demo-booked conversion." The campaign group, campaign, and draft ads are created in a single call. - "What ads is our biggest competitor running right now, and what offers are they pushing?" Adwarp reads the public Ad Library: copy, formats, and EU targeting metadata for any company. - "How did retargeting do last month compared to the month before?" You get a trend report with deltas. diff --git a/docs/platforms-linkedin-ads.md b/docs/platforms-linkedin-ads.md index 564679a..3b91db1 100644 --- a/docs/platforms-linkedin-ads.md +++ b/docs/platforms-linkedin-ads.md @@ -1,7 +1,7 @@ --- title: LinkedIn Ads description: "What Adwarp does on LinkedIn Ads, area by area, with the tools behind each one." -order: 18 +order: 19 --- Adwarp runs LinkedIn Ads through your own developer app, with tools for targeting, audiences, campaigns, reporting, competitor ads and the change journal. Everything it creates starts as a draft. A campaign or a single ad goes live only when you explicitly ask Adwarp to set it ACTIVE, or when you switch it on in Campaign Manager. @@ -23,7 +23,7 @@ LinkedIn's names sit 1 level off from other ad platforms, which trips up everyon - [Accounts](./tools.md#accounts): the ad accounts your app can reach. - [Targeting](./tools.md#targeting): resolve plain-language targeting to facets and entity URNs, and estimate reach before anything is built. -- [Audiences](./tools.md#audiences): upload a CSV or a Salesforce query as a matched audience, then check its match status. +- [Audiences](./tools.md#audiences): upload a CSV, a Salesforce query, or a HubSpot list or search as a matched audience, then check its match status. - [Conversions](./tools.md#conversions): pick the insight-tag conversion a campaign tracks. - [Campaigns](./tools.md#campaigns): create campaign groups, campaigns and ads as drafts, change an ad group in place with `update_campaign`, and pause, resume or archive a single ad with `update_ad`. - [Orchestrator](./tools.md#orchestrator): `launch_from_brief` creates the audience, campaign group, campaign and draft ads in 1 call. @@ -33,4 +33,4 @@ LinkedIn's names sit 1 level off from other ad platforms, which trips up everyon ## Where it runs -The local MCP server and the CLI run every LinkedIn tool. The [hosted endpoint](./hosted.md) registers them too, but CSV uploads, Salesforce audiences and the change journal need your machine, so run those locally. See [Hosted limitations](./hosted.md#hosted-limitations). +The local MCP server and the CLI run every LinkedIn tool. The [hosted endpoint](./hosted.md) registers all of them except HubSpot audiences, but CSV uploads, Salesforce audiences and the change journal need your machine too, so run those locally as well. See [Hosted limitations](./hosted.md#hosted-limitations). diff --git a/docs/safety.md b/docs/safety.md index bff7987..7480b8b 100644 --- a/docs/safety.md +++ b/docs/safety.md @@ -1,7 +1,7 @@ --- title: Safety description: "The rules Adwarp follows so nothing spends without you: drafts-first creation, audience limits, and where secrets live." -order: 12 +order: 13 --- - Every LinkedIn campaign and creative is created as a DRAFT by default, and every Google Ads campaign, ad group, and ad as PAUSED. On LinkedIn, going live is always an explicit request: you ask Adwarp to set a campaign or a single ad ACTIVE, or switch it on yourself in Campaign Manager. On Google, no create schema can express an enabled status, so activation happens in the Google Ads UI. diff --git a/docs/skills.md b/docs/skills.md index 34632bd..18977a5 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -30,7 +30,7 @@ Analysis and monitoring skills are read-only. The operating skills create drafts | --- | --- | --- | | `adwarp-launch` | "Launch a campaign" | a brief turned into a confirmed draft, with house guardrails applied | | `adwarp-experiments` | "Did that change help?" | hypothesis logging plus before and after lift | -| `adwarp-audiences` | "Audit my audiences" | health, dry-run uploads, a Salesforce refresh loop | +| `adwarp-audiences` | "Audit my audiences" | health, dry-run uploads, Salesforce and HubSpot refresh loops | | `adwarp-account-audit` | "Is my account set up right?" | a hygiene scorecard and fix list | ## Google Ads diff --git a/docs/tools.json b/docs/tools.json index e2f0850..023dac4 100644 --- a/docs/tools.json +++ b/docs/tools.json @@ -1,4 +1,92 @@ [ + { + "name": "audience_from_hubspot", + "platform": "linkedin", + "hosted": false, + "hostedSupport": "none", + "hostedNote": "Not registered on the hosted endpoint. Run it on the local MCP server or the CLI.", + "description": "Build a matched audience from HubSpot contacts or companies. Pass exactly 1 source: a HubSpot listId (its members), or filters for a CRM search (all ANDed). Reads every page, cleans the records like a CSV upload, and uploads only SHA256 email hashes for contacts (name and website for companies). Pass dryRun=true first: it returns counts and a masked sample and uploads nothing. Needs a HubSpot private app token on this machine (ADWARP_HUBSPOT_TOKEN or ~/.adwarp/hubspot.json); matching takes up to 48h.", + "inputSchema": { + "type": "object", + "properties": { + "accountId": { + "type": "string", + "description": "LinkedIn ad account id; defaults to config defaultAccountId" + }, + "name": { + "type": "string", + "minLength": 1, + "description": "Audience/segment name" + }, + "type": { + "type": "string", + "enum": [ + "contact", + "company" + ], + "description": "contact or company. With listId it defaults to the list's object type; with filters it defaults to contact." + }, + "listId": { + "type": "string", + "pattern": "^\\d+$", + "description": "HubSpot list id (the ILS list id). Pass this or filters, not both." + }, + "filters": { + "type": "array", + "items": { + "type": "object", + "properties": { + "property": { + "type": "string", + "minLength": 1, + "description": "HubSpot property internal name, e.g. lifecyclestage" + }, + "operator": { + "type": "string", + "enum": [ + "EQ", + "NEQ", + "LT", + "LTE", + "GT", + "GTE", + "BETWEEN", + "IN", + "NOT_IN", + "HAS_PROPERTY", + "NOT_HAS_PROPERTY", + "CONTAINS_TOKEN", + "NOT_CONTAINS_TOKEN" + ] + }, + "value": { + "type": "string", + "description": "Value to compare. IN and NOT_IN take a comma-separated list, BETWEEN takes 'low,high', HAS_PROPERTY and NOT_HAS_PROPERTY take none." + } + }, + "required": [ + "property", + "operator" + ], + "additionalProperties": false + }, + "minItems": 1, + "maxItems": 6, + "description": "CRM search filters, all ANDed (HubSpot allows up to 6 per group). Pass this or listId, not both." + }, + "dryRun": { + "type": "boolean", + "default": false, + "description": "Return counts and a masked sample without uploading anything" + } + }, + "required": [ + "name" + ], + "additionalProperties": false, + "$schema": "http://json-schema.org/draft-07/schema#" + } + }, { "name": "audience_from_salesforce", "platform": "linkedin", diff --git a/docs/tools.md b/docs/tools.md index 1a2b12a..7210646 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -23,6 +23,7 @@ Talk in plain language ("VPs of demand gen at SaaS companies in the US") and Adw - `upload_audience_csv`: cleans a CSV and uploads it as a matched audience. It normalizes column names, drops non-matcher columns, hashes emails, and converts company domains to website URLs. Contact and company lists are both supported. - `audience_from_salesforce`: a SOQL query to a matched audience. See [Salesforce](./salesforce.md). +- `audience_from_hubspot`: a HubSpot list, or a CRM search by property filters, to a matched audience of contacts or companies. Pass `dryRun` first to see the counts and a masked sample. Local only. See [HubSpot](./hubspot.md). - `get_audience_status`: matching status and resolved size. ## Conversions diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 2357c79..7adb0f8 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -14,6 +14,10 @@ import { searchTargeting, estimateAudience, audienceFromSalesforce, + audienceFromHubSpot, + previewHubSpotAudience, + parseHubSpotFilter, + type HubSpotFilter, listConversions, performanceSummary, getPerformance, @@ -172,6 +176,49 @@ audience res.warnings.forEach((w: string) => console.warn(`! ${w}`)); }); +const collect = (value: string, previous: string[]) => [...previous, value]; + +audience + .command("from-hubspot") + .description("Build a matched audience from HubSpot contacts or companies: a list, or CRM search filters") + .requiredOption("-n, --name ", "Audience name") + .option("--list ", "HubSpot list id (its members become the audience)") + .option("--filter ", "CRM search filter, repeatable and ANDed, e.g. lifecyclestage:EQ:customer", collect, []) + .option("-t, --type ", "contact|company (defaults to the list's type, or contact for filters)") + .option("-a, --account ", "Ad account id (defaults to config defaultAccountId)") + .option("--dry-run", "Show counts and a masked sample without uploading") + .action(async (opts: { name: string; list?: string; filter: string[]; type?: "contact" | "company"; account?: string; dryRun?: boolean }) => { + if (opts.type && opts.type !== "contact" && opts.type !== "company") { + throw new Error(`--type must be contact or company, got "${opts.type}".`); + } + const filters: HubSpotFilter[] | undefined = opts.filter.length ? opts.filter.map(parseHubSpotFilter) : undefined; + const input = { name: opts.name, type: opts.type, listId: opts.list, filters }; + const reasons = (r: Record) => + Object.entries(r).map(([k, v]) => `${k} ${v}`).join(", "); + if (opts.dryRun) { + const p = await previewHubSpotAudience(input); + const from = p.source.kind === "list" ? `list ${p.source.listId} "${p.source.listName}"` : "search"; + console.log(`HubSpot ${from} (${p.audienceType})`); + console.log(`read: ${p.read} | valid: ${p.valid} | skipped: ${p.skipped}${p.skipped ? ` (${reasons(p.skippedReasons)})` : ""}`); + p.warnings.forEach((w: string) => console.warn(`! ${w}`)); + if (p.sample.length) { + console.log("\nsample:"); + console.log(p.columns.join(",")); + p.sample.forEach((row) => console.log(p.columns.map((c) => row[c] ?? "").join(","))); + } + return; + } + const linkedinAds = await createLinkedInAds(); + const res = await audienceFromHubSpot(linkedinAds.client, linkedinAds.getToken, { + ...input, + accountId: opts.account ?? (await requireDefaultAccountId()), + }); + console.log(`Read ${res.read} ${res.audienceType === "contact" ? "contacts" : "companies"} from HubSpot.`); + console.log(`Segment ${res.segmentId} (${res.audienceType}), status ${res.status} (${res.uploaded} uploaded).`); + if (res.skipped) console.log(`Skipped ${res.skipped}: ${reasons(res.skippedReasons)}.`); + res.warnings.forEach((w: string) => console.warn(`! ${w}`)); + }); + audience .command("status ") .description("Check a DMP segment's matching status") diff --git a/packages/core/src/audience.ts b/packages/core/src/audience.ts index dacf142..bd52393 100644 --- a/packages/core/src/audience.ts +++ b/packages/core/src/audience.ts @@ -13,7 +13,7 @@ import { RECOMMENDED_MIN_ROWS, type SegmentType, } from "./resources/dmpSegments.js"; -import { cleanAudienceCsv, cleanedToCsv, type AudienceType } from "./audienceCsv.js"; +import { cleanAudienceCsv, cleanedToCsv, type AudienceType, type CleanedCsv } from "./audienceCsv.js"; /** Normalize (trim + lowercase) and SHA256-hash an email, per LinkedIn's spec. */ export function hashEmail(email: string): string { @@ -22,12 +22,17 @@ export function hashEmail(email: string): string { const EMAIL_RE = /^[^@\s]+@[^@\s]+\.[^@\s]+$/; +/** Whether a raw value is shaped like an email address, the check every upload applies. */ +export function isValidEmail(email: string): boolean { + return !!email && EMAIL_RE.test(email); +} + /** Validate, normalize, SHA256-hash, and dedupe a list of raw emails. */ export function hashAndDedupeEmails(emails: string[]): { hashes: string[]; total: number; skipped: number } { const seen = new Set(); let skipped = 0; for (const email of emails) { - if (!email || !EMAIL_RE.test(email)) { + if (!isValidEmail(email)) { skipped += 1; continue; } @@ -139,14 +144,7 @@ async function uploadHashedAudience( const warnings = [...(input.preWarnings ?? [])]; if (hashes.length === 0) throw new Error("No valid emails to upload."); - if (hashes.length > MAX_SEGMENT_ROWS) { - throw new Error(`List has ${hashes.length} emails; LinkedIn caps a single upload at ${MAX_SEGMENT_ROWS}.`); - } - if (hashes.length < RECOMMENDED_MIN_ROWS) { - warnings.push( - `Only ${hashes.length} valid emails. LinkedIn recommends >= ${RECOMMENDED_MIN_ROWS} and needs ~300 matched members to serve. Match rates run ~30-60%, so a small list may never become targetable.`, - ); - } + warnings.push(...audienceSizeWarnings("contact", hashes.length)); if (skipped > 0) warnings.push(`Skipped ${skipped} rows with missing/invalid emails.`); return uploadCsvToSegment(client, getToken, { @@ -164,6 +162,27 @@ async function uploadHashedAudience( /** Min recommended rows for a company list (contacts use RECOMMENDED_MIN_ROWS). */ const RECOMMENDED_MIN_COMPANIES = 1000; +/** + * LinkedIn's list-size rules for a matched audience of `rows` valid rows: throws + * above the per-upload cap, and returns a warning below the recommended minimum. + * Uploads and dry runs both apply it, so a preview fails where the upload would. + */ +export function audienceSizeWarnings(type: AudienceType, rows: number): string[] { + const unit = type === "contact" ? "emails" : "companies"; + if (rows > MAX_SEGMENT_ROWS) { + throw new Error(`List has ${rows} ${unit}; LinkedIn caps a single upload at ${MAX_SEGMENT_ROWS}.`); + } + if (type === "contact" && rows < RECOMMENDED_MIN_ROWS) { + return [ + `Only ${rows} valid emails. LinkedIn recommends >= ${RECOMMENDED_MIN_ROWS} and needs ~300 matched members to serve. Match rates run ~30-60%, so a small list may never become targetable.`, + ]; + } + if (type === "company" && rows < RECOMMENDED_MIN_COMPANIES) { + return [`Only ${rows} companies. LinkedIn recommends >= ${RECOMMENDED_MIN_COMPANIES}.`]; + } + return []; +} + /** * Clean a CSV (fix column names, drop non-matcher columns, turn domains into * website URLs) and upload it as the right list type (auto-detected, or forced @@ -177,7 +196,21 @@ export async function uploadAudienceFromFile( const text = await readFile(input.csvPath, "utf8"); const clean = cleanAudienceCsv(text, { type: input.type }); if (clean.rowCount === 0) throw new Error("No usable rows after cleaning the CSV."); + return uploadCleanedAudience(client, getToken, { accountId: input.accountId, name: input.name, clean }); +} +/** + * Upload a list that has already been cleaned (by cleanAudienceCsv or + * cleanAudienceRows) as the matching segment type. Contacts are SHA256-hashed + * and deduped, so only hashes leave the machine; companies upload name + + * website. Shared by CSV files and CRM sources, so they upload the same way. + */ +export async function uploadCleanedAudience( + client: LinkedInClient, + getToken: TokenProvider, + input: { accountId: string; name: string; clean: CleanedCsv }, +): Promise { + const { clean } = input; if (clean.type === "contact") { const { hashes, total, skipped } = hashAndDedupeEmails(clean.rows.map((r) => r.email ?? "")); const res = await uploadHashedAudience(client, getToken, { @@ -192,10 +225,8 @@ export async function uploadAudienceFromFile( } // company - const warnings = [...clean.warnings]; - if (clean.rowCount < RECOMMENDED_MIN_COMPANIES) { - warnings.push(`Only ${clean.rowCount} companies. LinkedIn recommends >= ${RECOMMENDED_MIN_COMPANIES}.`); - } + if (clean.rowCount === 0) throw new Error("No companies to upload."); + const warnings = [...clean.warnings, ...audienceSizeWarnings("company", clean.rowCount)]; const res = await uploadCsvToSegment(client, getToken, { accountId: input.accountId, name: input.name, diff --git a/packages/core/src/audienceCsv.ts b/packages/core/src/audienceCsv.ts index 491bf23..9684d3c 100644 --- a/packages/core/src/audienceCsv.ts +++ b/packages/core/src/audienceCsv.ts @@ -82,6 +82,15 @@ function detectType(headers: string[]): AudienceType { /** Parse + normalize a CSV into LinkedIn's matched-audience format. */ export function cleanAudienceCsv(text: string, opts: { type?: AudienceType } = {}): CleanedCsv { const parsed = parse(text, { columns: true, skip_empty_lines: true, trim: true }) as Record[]; + return cleanAudienceRows(parsed, opts); +} + +/** + * Normalize already-parsed rows (header -> value) into LinkedIn's matched-audience + * format. The CSV path and CRM sources such as HubSpot share this, so every list + * is cleaned the same way whatever it came from. + */ +export function cleanAudienceRows(parsed: Record[], opts: { type?: AudienceType } = {}): CleanedCsv { const headers = parsed.length ? Object.keys(parsed[0]!) : []; if (headers.length === 0) throw new Error("CSV has no header row."); diff --git a/packages/core/src/hubspot.ts b/packages/core/src/hubspot.ts new file mode 100644 index 0000000..2f717ac --- /dev/null +++ b/packages/core/src/hubspot.ts @@ -0,0 +1,557 @@ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { + ADWARP_DIR, + DEFAULT_MAX_RETRIES, + displayPath, + env, + isRetryableStatus, + retryDelayMs, + sleep, + transportRetryDelayMs, +} from "@adwarp/shared"; +import type { LinkedInClient, TokenProvider } from "./http.js"; +import { audienceSizeWarnings, hashEmail, isValidEmail, uploadCleanedAudience } from "./audience.js"; +import { cleanAudienceRows, type AudienceType, type CleanedCsv } from "./audienceCsv.js"; +import { HubSpotFilterSchema, type HubSpotAudienceInput, type HubSpotFilter } from "./schemas.js"; + +/** + * Reads contacts and companies from HubSpot to feed a LinkedIn matched audience, + * the same way the Salesforce reader does: a HubSpot list (its memberships) or a + * CRM search becomes records, the records go through the CSV cleaning path, and + * the result uploads like any other list. Contacts leave the machine only as + * SHA256 email hashes. + * + * Auth is a HubSpot private app access token the user creates in their own + * account, with the read-only crm.objects.contacts.read, + * crm.objects.companies.read and crm.lists.read scopes. There is no OAuth, and + * nothing here writes to HubSpot. + */ + +export const HUBSPOT_API_BASE = "https://api.hubapi.com"; + +/** + * HubSpot's date-based API version. Pinned deliberately: each version is + * supported for 18 months, and HubSpot ends support for the older v3 paths in + * September 2027, so bumping is a decision, not a default. + */ +export const HUBSPOT_API_VERSION = "2026-09"; + +const CONFIG_PATH = join(ADWARP_DIR, "hubspot.json"); +export const hubspotConfigPath = CONFIG_PATH; + +/** The private app scopes this integration needs, all read-only. */ +export const HUBSPOT_SCOPES = ["crm.objects.contacts.read", "crm.objects.companies.read", "crm.lists.read"] as const; + +const SCOPE_LIST = `${HUBSPOT_SCOPES.slice(0, -1).join(", ")} and ${HUBSPOT_SCOPES[HUBSPOT_SCOPES.length - 1]}`; + +/** HubSpot object type ids for the 2 object types a LinkedIn list can hold. */ +const OBJECT_TYPE_ID: Record = { contact: "0-1", company: "0-2" }; +const OBJECT_PATH: Record = { contact: "contacts", company: "companies" }; +const NOUN: Record = { contact: "contacts", company: "companies" }; + +/** + * The properties read per record: only what LinkedIn matches on. Contacts match + * on email alone, as in the CSV path. Companies match on name and domain; + * HubSpot fills `domain` from the Website URL property. + */ +const PROPERTIES: Record = { contact: ["email"], company: ["name", "domain"] }; + +/** Search: 200 records per page, 10,000 results per query, 5 requests per second. */ +const SEARCH_PAGE_SIZE = 200; +export const HUBSPOT_SEARCH_RESULT_CAP = 10_000; +const SEARCH_INTERVAL_MS = 250; +/** List memberships page size (HubSpot's documented default). */ +const MEMBERSHIP_PAGE_SIZE = 100; +/** Batch read takes at most 100 records per request. */ +const BATCH_READ_SIZE = 100; +/** Stay under 100 requests per 10 seconds, the private app burst limit on Free and Starter. */ +const REQUEST_INTERVAL_MS = 100; + +const SAMPLE_ROWS = 5; + +/* ---------------------------------- auth ------------------------------------ */ + +/** + * The private app access token: ADWARP_HUBSPOT_TOKEN first, then + * `{ "accessToken": "..." }` in hubspot.json in the Adwarp folder. + */ +export async function loadHubSpotToken(): Promise { + const fromEnv = env("ADWARP_HUBSPOT_TOKEN")?.trim(); + if (fromEnv) return fromEnv; + let raw: string; + try { + raw = await readFile(CONFIG_PATH, "utf8"); + } catch { + throw new Error( + `No HubSpot access token. Create a private app in HubSpot with the ${SCOPE_LIST} scopes, then save its access token as { "accessToken": "..." } in ${displayPath(CONFIG_PATH)} (mode 0600), or set ADWARP_HUBSPOT_TOKEN.`, + ); + } + let parsed: { accessToken?: unknown }; + try { + parsed = JSON.parse(raw) as { accessToken?: unknown }; + } catch { + throw new Error(`${displayPath(CONFIG_PATH)} is not valid JSON. Expected { "accessToken": "..." }.`); + } + const token = typeof parsed?.accessToken === "string" ? parsed.accessToken.trim() : ""; + if (!token) throw new Error(`${displayPath(CONFIG_PATH)} must include accessToken.`); + return token; +} + +/* ---------------------------------- http ------------------------------------ */ + +export class HubSpotApiError extends Error { + constructor( + message: string, + readonly status: number, + readonly body: unknown, + ) { + super(message); + this.name = "HubSpotApiError"; + } +} + +interface HubSpotRequest { + method?: "GET" | "POST"; + path: string; + query?: Record; + body?: unknown; +} + +function describeError(status: number, data: unknown, text: string): string { + const body = (data && typeof data === "object" ? data : {}) as { + message?: string; + context?: { missingScopes?: string[] }; + }; + const message = body.message ?? (text || `HubSpot API ${status}`); + if (status === 401) { + return `HubSpot rejected the access token (401): ${message}. Check ADWARP_HUBSPOT_TOKEN or ${displayPath(CONFIG_PATH)}.`; + } + if (status === 403) { + const missing = body.context?.missingScopes?.length ? ` Missing: ${body.context.missingScopes.join(", ")}.` : ""; + return `HubSpot refused the request (403): ${message}. The private app needs the ${SCOPE_LIST} scopes.${missing}`; + } + return `HubSpot API ${status}: ${message}`; +} + +/** + * Every HubSpot call goes through a single client: bearer token, JSON, pacing under + * the private app burst limit, and the shared retry policy on 429 / 5xx. + */ +function hubspotClient(token: string) { + let last = 0; + return async function request(opts: HubSpotRequest, intervalMs = REQUEST_INTERVAL_MS): Promise { + const url = new URL(HUBSPOT_API_BASE + opts.path); + for (const [k, v] of Object.entries(opts.query ?? {})) if (v !== undefined) url.searchParams.set(k, String(v)); + const method = opts.method ?? "GET"; + const headers: Record = { Authorization: `Bearer ${token}` }; + let bodyStr: string | undefined; + if (opts.body !== undefined) { + headers["Content-Type"] = "application/json"; + bodyStr = JSON.stringify(opts.body); + } + + let lastErr: unknown; + for (let attempt = 0; attempt <= DEFAULT_MAX_RETRIES; attempt++) { + const wait = last + intervalMs - Date.now(); + if (wait > 0) await sleep(wait); + last = Date.now(); + + let res: Response; + try { + res = await fetch(url, { method, headers, body: bodyStr }); + } catch (err) { + lastErr = err; + if (attempt < DEFAULT_MAX_RETRIES) { + await sleep(transportRetryDelayMs(attempt)); + continue; + } + throw err; + } + if (isRetryableStatus(res.status) && attempt < DEFAULT_MAX_RETRIES) { + await sleep(retryDelayMs(attempt, res.headers.get("retry-after"))); + continue; + } + const text = await res.text(); + let data: unknown; + try { + data = text ? JSON.parse(text) : undefined; + } catch { + data = text; + } + if (!res.ok) throw new HubSpotApiError(describeError(res.status, data, text), res.status, data); + return data as T; + } + throw lastErr instanceof Error ? lastErr : new Error("HubSpot request failed"); + }; +} + +type HubSpotRequester = ReturnType; + +interface HubSpotRecord { + id: string; + properties?: Record; +} + +interface Paged { + results?: T[]; + total?: number; + paging?: { next?: { after?: string } }; +} + +/* --------------------------------- search ----------------------------------- */ + +/** HubSpot's search filter shape for a single simple filter. */ +export function toHubSpotSearchFilter(filter: HubSpotFilter): Record { + const { property: propertyName, operator } = HubSpotFilterSchema.parse(filter); + if (operator === "HAS_PROPERTY" || operator === "NOT_HAS_PROPERTY") return { propertyName, operator }; + const value = (filter.value ?? "").trim(); + if (!value) throw new Error(`The ${operator} filter on ${propertyName} needs a value.`); + if (operator === "IN" || operator === "NOT_IN") { + const values = value.split(",").map((v) => v.trim()).filter(Boolean); + return { propertyName, operator, values }; + } + if (operator === "BETWEEN") { + const parts = value.split(",").map((v) => v.trim()); + if (parts.length !== 2 || !parts[0] || !parts[1]) { + throw new Error(`The BETWEEN filter on ${propertyName} needs "low,high", got "${value}".`); + } + return { propertyName, operator, value: parts[0], highValue: parts[1] }; + } + return { propertyName, operator, value }; +} + +/** + * Parse a CLI filter, `::`. The value keeps any + * further colons; HAS_PROPERTY and NOT_HAS_PROPERTY take no value. + */ +export function parseHubSpotFilter(spec: string): HubSpotFilter { + const first = spec.indexOf(":"); + if (first <= 0) throw new Error(`Filter "${spec}" should look like ::, e.g. lifecyclestage:EQ:customer.`); + const property = spec.slice(0, first).trim(); + const rest = spec.slice(first + 1); + const second = rest.indexOf(":"); + const operator = (second === -1 ? rest : rest.slice(0, second)).trim().toUpperCase(); + const value = second === -1 ? undefined : rest.slice(second + 1); + const parsed = HubSpotFilterSchema.safeParse({ property, operator, value }); + if (!parsed.success) { + throw new Error( + `Filter "${spec}" has an unknown operator "${operator}". Use any of ${HubSpotFilterSchema.shape.operator.options.join(", ")}.`, + ); + } + toHubSpotSearchFilter(parsed.data); + return parsed.data; +} + +async function searchRecords(request: HubSpotRequester, type: AudienceType, filters: HubSpotFilter[]): Promise { + const body = { + filterGroups: [{ filters: filters.map(toHubSpotSearchFilter) }], + properties: PROPERTIES[type], + limit: SEARCH_PAGE_SIZE, + }; + const records: HubSpotRecord[] = []; + let after: string | undefined; + let first = true; + do { + const page = await request>( + { method: "POST", path: `/crm/objects/${HUBSPOT_API_VERSION}/${OBJECT_PATH[type]}/search`, body: after ? { ...body, after } : body }, + SEARCH_INTERVAL_MS, + ); + if (first && (page.total ?? 0) > HUBSPOT_SEARCH_RESULT_CAP) { + throw new Error( + `This HubSpot search matches ${page.total} ${NOUN[type]}, and HubSpot returns at most ${HUBSPOT_SEARCH_RESULT_CAP} results per search. Narrow the filters, or save the records as a HubSpot list and pass its list id, which has no such cap.`, + ); + } + first = false; + records.push(...(page.results ?? [])); + const next = page.paging?.next?.after; + after = next && next !== after ? next : undefined; + } while (after); + return records; +} + +/* ---------------------------------- lists ----------------------------------- */ + +interface HubSpotList { + listId: string; + name: string; + objectTypeId: string; + processingType?: string; + size?: number; +} + +async function getList(request: HubSpotRequester, listId: string): Promise { + try { + const res = await request<{ list: HubSpotList }>({ path: `/crm/lists/${HUBSPOT_API_VERSION}/${encodeURIComponent(listId)}` }); + return res.list; + } catch (err) { + if (err instanceof HubSpotApiError && err.status === 404) { + throw new Error(`HubSpot has no list with id ${listId}. Find a list's id in HubSpot under Segments: hover the list, then click Details.`); + } + throw err; + } +} + +async function listMemberIds(request: HubSpotRequester, listId: string): Promise { + const ids: string[] = []; + let after: string | undefined; + do { + const page = await request>({ + path: `/crm/lists/${HUBSPOT_API_VERSION}/${encodeURIComponent(listId)}/memberships`, + query: { limit: MEMBERSHIP_PAGE_SIZE, after }, + }); + for (const m of page.results ?? []) if (m.recordId !== undefined) ids.push(String(m.recordId)); + const next = page.paging?.next?.after; + after = next && next !== after ? next : undefined; + } while (after); + return [...new Set(ids)]; +} + +async function batchRead(request: HubSpotRequester, type: AudienceType, ids: string[]): Promise { + const records: HubSpotRecord[] = []; + for (let i = 0; i < ids.length; i += BATCH_READ_SIZE) { + const chunk = ids.slice(i, i + BATCH_READ_SIZE); + const res = await request<{ results?: HubSpotRecord[] }>({ + method: "POST", + path: `/crm/objects/${HUBSPOT_API_VERSION}/${OBJECT_PATH[type]}/batch/read`, + body: { properties: PROPERTIES[type], inputs: chunk.map((id) => ({ id })) }, + }); + records.push(...(res.results ?? [])); + } + return records; +} + +/* --------------------------------- reading ---------------------------------- */ + +export type HubSpotSource = + | { kind: "list"; listId: string; listName: string; processingType?: string } + | { kind: "search"; filters: HubSpotFilter[] }; + +/** HubSpot records read and cleaned into LinkedIn's list format, before any upload. */ +export interface HubSpotAudienceRead { + audienceType: AudienceType; + source: HubSpotSource; + /** Records read from HubSpot. */ + read: number; + /** Rows left to upload: unique valid emails, or companies with a name or website. */ + valid: number; + skipped: number; + skippedReasons: Record; + clean: CleanedCsv; + /** Notes about the HubSpot read itself. Cleaning notes stay on `clean.warnings`. */ + warnings: string[]; +} + +const prop = (r: HubSpotRecord, key: string): string => { + const v = r.properties?.[key]; + return v === null || v === undefined ? "" : String(v); +}; + +/** + * Read a HubSpot list or search fully and clean it. Throws on a missing token, + * a bad source, a list of another object type, or a search over HubSpot's cap. + */ +export async function readHubSpotAudience( + input: Pick, + opts: { token?: string } = {}, +): Promise { + const hasList = !!input.listId; + const hasFilters = !!input.filters?.length; + if (hasList === hasFilters) { + throw new Error("Pass exactly 1 HubSpot source: a list id, or filters for a CRM search."); + } + const request = hubspotClient(opts.token ?? (await loadHubSpotToken())); + + let type: AudienceType; + let source: HubSpotSource; + let records: HubSpotRecord[]; + let notFound = 0; + if (input.listId) { + const list = await getList(request, input.listId); + const listType = (Object.keys(OBJECT_TYPE_ID) as AudienceType[]).find((t) => OBJECT_TYPE_ID[t] === list.objectTypeId); + if (!listType) { + throw new Error( + `HubSpot list ${input.listId} holds object type ${list.objectTypeId}. Only contact (0-1) and company (0-2) lists can become LinkedIn audiences.`, + ); + } + if (input.type && input.type !== listType) { + throw new Error(`HubSpot list ${input.listId} is a ${listType} list, but type is ${input.type}. Drop type or pass ${listType}.`); + } + type = listType; + source = { kind: "list", listId: input.listId, listName: list.name, processingType: list.processingType }; + const ids = await listMemberIds(request, input.listId); + records = await batchRead(request, type, ids); + notFound = Math.max(0, ids.length - records.length); + } else { + type = input.type ?? "contact"; + source = { kind: "search", filters: input.filters! }; + records = await searchRecords(request, type, input.filters!); + } + + const read = records.length + notFound; + const skippedReasons: Record = {}; + const add = (reason: string, n: number) => { + if (n > 0) skippedReasons[reason] = (skippedReasons[reason] ?? 0) + n; + }; + add("not found in HubSpot", notFound); + + let clean: CleanedCsv; + let valid: number; + if (type === "contact") { + clean = records.length + ? cleanAudienceRows(records.map((r) => ({ email: prop(r, "email") })), { type }) + : emptyClean(type); + add("no email", records.length - clean.rowCount); + const seen = new Set(); + let invalid = 0; + for (const row of clean.rows) { + if (!isValidEmail(row.email ?? "")) invalid += 1; + else seen.add(hashEmail(row.email!)); + } + add("invalid email", invalid); + add("duplicate email", clean.rowCount - invalid - seen.size); + valid = seen.size; + } else { + clean = records.length + ? cleanAudienceRows( + records.map((r) => ({ companyname: prop(r, "name").trim(), companywebsite: prop(r, "domain") })), + { type }, + ) + : emptyClean(type); + add("no name or domain", records.length - clean.rowCount); + valid = clean.rowCount; + } + + const skipped = Object.values(skippedReasons).reduce((a, b) => a + b, 0); + const warnings: string[] = []; + if (notFound > 0) warnings.push(`${notFound} list member(s) no longer exist in HubSpot and were skipped.`); + return { audienceType: type, source, read, valid, skipped, skippedReasons, clean, warnings }; +} + +function emptyClean(type: AudienceType): CleanedCsv { + const columns = type === "contact" ? ["email"] : ["companyname", "companywebsite"]; + return { type, columns, rows: [], dropped: [], rowCount: 0, warnings: [] }; +} + +function describeSource(source: HubSpotSource): string { + return source.kind === "list" ? `list ${source.listId} ("${source.listName}")` : "search"; +} + +/** Show the start and the domain of an email, so a sample proves the data without exposing it. */ +export function maskEmail(email: string): string { + const at = email.lastIndexOf("@"); + if (at <= 0) return "***"; + const local = email.slice(0, at); + return `${local.slice(0, local.length > 2 ? 2 : 1)}***${email.slice(at)}`; +} + +/* --------------------------------- results ---------------------------------- */ + +interface HubSpotAudienceCounts { + name: string; + audienceType: AudienceType; + source: HubSpotSource; + read: number; + valid: number; + uploaded: number; + skipped: number; + skippedReasons: Record; + warnings: string[]; +} + +export interface HubSpotAudiencePreview extends HubSpotAudienceCounts { + dryRun: true; + columns: string[]; + /** A few cleaned rows. Emails are masked and shown with the SHA256 hash that would upload. */ + sample: Record[]; +} + +export interface HubSpotAudienceResult extends HubSpotAudienceCounts { + dryRun: false; + segmentId: string; + status: "PROCESSING"; +} + +/** + * Dry run: read and clean the HubSpot records and report counts plus a sample. + * Uploads nothing and needs no LinkedIn credentials. + */ +export async function previewHubSpotAudience( + input: Pick, + opts: { token?: string } = {}, +): Promise { + const r = await readHubSpotAudience(input, opts); + const warnings = [...r.warnings, ...r.clean.warnings]; + if (r.read === 0) warnings.push(`HubSpot ${describeSource(r.source)} returned no ${NOUN[r.audienceType]}. Nothing would upload.`); + else if (r.valid === 0) warnings.push(`None of the ${r.read} ${NOUN[r.audienceType]} can be matched. Nothing would upload.`); + else warnings.push(...audienceSizeWarnings(r.audienceType, r.valid)); + + let columns: string[]; + let sample: Record[]; + if (r.audienceType === "contact") { + columns = ["email", "sha256"]; + const seen = new Set(); + sample = []; + for (const row of r.clean.rows) { + const email = row.email ?? ""; + if (!isValidEmail(email)) continue; + const hash = hashEmail(email); + if (seen.has(hash)) continue; + seen.add(hash); + sample.push({ email: maskEmail(email), sha256: hash }); + if (sample.length === SAMPLE_ROWS) break; + } + } else { + columns = r.clean.columns; + sample = r.clean.rows.slice(0, SAMPLE_ROWS); + } + + return { + dryRun: true, + name: input.name, + audienceType: r.audienceType, + source: r.source, + read: r.read, + valid: r.valid, + uploaded: 0, + skipped: r.skipped, + skippedReasons: r.skippedReasons, + columns, + sample, + warnings, + }; +} + +/** + * Closes the loop: a HubSpot list or CRM search becomes a LinkedIn + * matched-audience DMP segment, through the same cleaning, hashing and upload + * path as a CSV. Reads HubSpot fully before touching LinkedIn. + */ +export async function audienceFromHubSpot( + client: LinkedInClient, + getToken: TokenProvider, + input: Pick & { accountId: string }, + opts: { token?: string } = {}, +): Promise { + const r = await readHubSpotAudience(input, opts); + if (r.read === 0) throw new Error(`HubSpot ${describeSource(r.source)} returned no ${NOUN[r.audienceType]}.`); + if (r.valid === 0) { + const reasons = Object.entries(r.skippedReasons).map(([k, v]) => `${k}: ${v}`).join(", "); + throw new Error(`None of the ${r.read} HubSpot ${NOUN[r.audienceType]} can be matched (${reasons}).`); + } + const res = await uploadCleanedAudience(client, getToken, { accountId: input.accountId, name: input.name, clean: r.clean }); + return { + dryRun: false, + name: input.name, + audienceType: r.audienceType, + source: r.source, + segmentId: res.segmentId, + status: res.status, + read: r.read, + valid: r.valid, + uploaded: res.uploaded, + skipped: r.skipped, + skippedReasons: r.skippedReasons, + warnings: [...r.warnings, ...res.warnings], + }; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index f329801..875f772 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -29,8 +29,9 @@ export * from "./report.js"; // Change journal + lift comparison (local JSONL store). export * from "./changelog.js"; -// Salesforce reader (Phase 3 foundation) +// CRM readers: Salesforce (Phase 3 foundation) and HubSpot, each feeding matched audiences export * from "./salesforce.js"; +export * from "./hubspot.js"; // Competitor ad intelligence: official API and remote creative worker. export * from "./adLibrary.js"; diff --git a/packages/core/src/schemas.ts b/packages/core/src/schemas.ts index 99d75ee..ca426c7 100644 --- a/packages/core/src/schemas.ts +++ b/packages/core/src/schemas.ts @@ -271,6 +271,55 @@ export const SalesforceAudienceSchema = z.object({ }); export type SalesforceAudienceInput = z.infer; +/** HubSpot CRM search operators, as HubSpot's search API names them. */ +export const HubSpotFilterOperatorSchema = z.enum([ + "EQ", + "NEQ", + "LT", + "LTE", + "GT", + "GTE", + "BETWEEN", + "IN", + "NOT_IN", + "HAS_PROPERTY", + "NOT_HAS_PROPERTY", + "CONTAINS_TOKEN", + "NOT_CONTAINS_TOKEN", +]); + +export const HubSpotFilterSchema = z.object({ + property: z.string().min(1).describe("HubSpot property internal name, e.g. lifecyclestage"), + operator: HubSpotFilterOperatorSchema, + value: z + .string() + .optional() + .describe("Value to compare. IN and NOT_IN take a comma-separated list, BETWEEN takes 'low,high', HAS_PROPERTY and NOT_HAS_PROPERTY take none."), +}); +export type HubSpotFilter = z.infer; + +export const HubSpotAudienceSchema = z.object({ + accountId: z.string().optional().describe("LinkedIn ad account id; defaults to config defaultAccountId"), + name: z.string().min(1).describe("Audience/segment name"), + type: z + .enum(["contact", "company"]) + .optional() + .describe("contact or company. With listId it defaults to the list's object type; with filters it defaults to contact."), + listId: z + .string() + .regex(/^\d+$/, "listId is the numeric HubSpot list id") + .optional() + .describe("HubSpot list id (the ILS list id). Pass this or filters, not both."), + filters: z + .array(HubSpotFilterSchema) + .min(1) + .max(6) + .optional() + .describe("CRM search filters, all ANDed (HubSpot allows up to 6 per group). Pass this or listId, not both."), + dryRun: z.boolean().default(false).describe("Return counts and a masked sample without uploading anything"), +}); +export type HubSpotAudienceInput = z.infer; + export const AdLibraryScanSchema = z.object({ advertiser: z.string().optional().describe("Competitor/advertiser display name (accountOwner)"), companyId: z.string().optional().describe("Numeric LinkedIn company id, more precise than name"), diff --git a/packages/core/test/hubspot.test.mjs b/packages/core/test/hubspot.test.mjs new file mode 100644 index 0000000..32bf891 --- /dev/null +++ b/packages/core/test/hubspot.test.mjs @@ -0,0 +1,326 @@ +/** + * HubSpot audiences: reading a CRM search or a list's members, cleaning them like a + * CSV, and uploading only email hashes for contacts. + * + * Runs against dist (`pnpm --filter @adwarp/core build` first). HubSpot answers from a + * stubbed fetch and LinkedIn from a fake client, so no credentials are needed and + * nothing outside this process is called. HOME points at an empty folder, so the + * token file is the test's own. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +const HOME = mkdtempSync(join(tmpdir(), "adwarp-hubspot-")); +process.env.HOME = HOME; +process.env.USERPROFILE = HOME; +delete process.env.ADWARP_HUBSPOT_TOKEN; +process.on("exit", () => rmSync(HOME, { recursive: true, force: true })); + +const { + previewHubSpotAudience, + audienceFromHubSpot, + parseHubSpotFilter, + toHubSpotSearchFilter, + hubspotConfigPath, +} = await import("../dist/index.js"); + +const API = "https://api.hubapi.com"; +const UPLOAD_URL = "https://upload.linkedin.test/list"; +const sha256 = (s) => createHash("sha256").update(s).digest("hex"); +const json = (body, status = 200) => + new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); + +/** + * Replaces fetch for a single test. `handler` answers each call; every call is recorded. + * Anything it does not answer fails the test, so no request can reach a real host. + */ +function stubFetch(t, handler) { + const calls = []; + const saved = globalThis.fetch; + globalThis.fetch = async (input, init = {}) => { + const url = new URL(String(input)); + const call = { + url, + method: init.method ?? "GET", + headers: init.headers ?? {}, + body: typeof init.body === "string" && init.body.startsWith("{") ? JSON.parse(init.body) : init.body, + }; + calls.push(call); + const res = await handler(call); + if (!res) throw new Error(`Unexpected request: ${call.method} ${url}`); + return res; + }; + t.after(() => { + globalThis.fetch = saved; + }); + return calls; +} + +/** Sets the token env var for a single test, or leaves it unset with `undefined`. */ +function withTokenEnv(t, value) { + if (value === undefined) delete process.env.ADWARP_HUBSPOT_TOKEN; + else process.env.ADWARP_HUBSPOT_TOKEN = value; + t.after(() => delete process.env.ADWARP_HUBSPOT_TOKEN); +} + +/** Writes hubspot.json in the test's Adwarp folder for a single test. */ +function withTokenFile(t, contents) { + mkdirSync(join(HOME, ".adwarp"), { recursive: true }); + writeFileSync(hubspotConfigPath, contents, { mode: 0o600 }); + t.after(() => rmSync(hubspotConfigPath, { force: true })); +} + +/** A LinkedIn client that records calls and answers the matched-audience upload flow. */ +function fakeLinkedIn() { + const calls = []; + const client = { + calls, + request: async (req) => { + calls.push(req); + if (req.query?.action === "generateUploadUrl") return { data: { value: UPLOAD_URL }, status: 200 }; + if (req.path === "/dmpSegments" && req.method === "POST") return { data: undefined, status: 201, restliId: "9001" }; + if (req.path === "/dmpSegments/9001/listUploads") return { data: undefined, status: 201 }; + throw new Error(`Unexpected LinkedIn call: ${req.method} ${req.path}`); + }, + }; + return { client, getToken: async () => "li-token" }; +} + +const CUSTOMERS = [{ property: "lifecyclestage", operator: "EQ", value: "customer" }]; + +test("a missing token names the env var, the file and the scopes, and calls nothing", async (t) => { + withTokenEnv(t, undefined); + const calls = stubFetch(t, () => undefined); + const li = fakeLinkedIn(); + + const expected = (err) => { + assert.match(err.message, /ADWARP_HUBSPOT_TOKEN/); + assert.match(err.message, /hubspot\.json/); + assert.match(err.message, /crm\.objects\.contacts\.read, crm\.objects\.companies\.read and crm\.lists\.read/); + return true; + }; + await assert.rejects(previewHubSpotAudience({ name: "Customers", filters: CUSTOMERS }), expected); + await assert.rejects( + audienceFromHubSpot(li.client, li.getToken, { accountId: "1", name: "Customers", filters: CUSTOMERS }), + expected, + ); + assert.equal(calls.length, 0); + assert.equal(li.client.calls.length, 0); +}); + +test("search reads every page with the filters ANDed, and the dry run uploads nothing", async (t) => { + withTokenEnv(t, undefined); + withTokenFile(t, JSON.stringify({ accessToken: "pat-file" })); + const pages = { + undefined: { total: 5, results: [{ id: "1", properties: { email: " Ada@Example.com" } }, { id: "2", properties: { email: null } }], paging: { next: { after: "2" } } }, + 2: { total: 5, results: [{ id: "3", properties: { email: "not-an-email" } }, { id: "4", properties: { email: "ada@example.com" } }], paging: { next: { after: "4" } } }, + 4: { total: 5, results: [{ id: "5", properties: { email: "bob@example.org" } }] }, + }; + const calls = stubFetch(t, (c) => + c.url.href === `${API}/crm/objects/2026-09/contacts/search` && c.method === "POST" ? json(pages[c.body.after]) : undefined, + ); + + const filters = [...CUSTOMERS, { property: "hs_lead_status", operator: "IN", value: "NEW, OPEN" }]; + const preview = await previewHubSpotAudience({ name: "Customers", filters }); + + assert.equal(calls.length, 3); + assert.deepEqual(calls.map((c) => c.body.after), [undefined, "2", "4"]); + for (const c of calls) { + assert.equal(c.headers.Authorization, "Bearer pat-file"); + assert.deepEqual(c.body.filterGroups, [ + { + filters: [ + { propertyName: "lifecyclestage", operator: "EQ", value: "customer" }, + { propertyName: "hs_lead_status", operator: "IN", values: ["NEW", "OPEN"] }, + ], + }, + ]); + assert.deepEqual(c.body.properties, ["email"]); + assert.equal(c.body.limit, 200); + } + + assert.equal(preview.dryRun, true); + assert.equal(preview.audienceType, "contact"); + assert.deepEqual(preview.source, { kind: "search", filters }); + assert.equal(preview.read, 5); + assert.equal(preview.valid, 2); + assert.equal(preview.uploaded, 0); + assert.equal(preview.skipped, 3); + assert.deepEqual(preview.skippedReasons, { "no email": 1, "invalid email": 1, "duplicate email": 1 }); + assert.deepEqual(preview.columns, ["email", "sha256"]); + assert.deepEqual(preview.sample, [ + { email: "ad***@example.com", sha256: sha256("ada@example.com") }, + { email: "bo***@example.org", sha256: sha256("bob@example.org") }, + ]); + assert.ok(!JSON.stringify(preview).includes("ada@example.com"), "the sample never carries a raw email"); + assert.ok(preview.warnings.some((w) => /Only 2 valid emails/.test(w))); +}); + +test("a search over HubSpot's 10,000 result cap stops after the first page", async (t) => { + withTokenEnv(t, "pat-env"); + const calls = stubFetch(t, () => json({ total: 10_001, results: [], paging: { next: { after: "200" } } })); + await assert.rejects(previewHubSpotAudience({ name: "Everyone", filters: CUSTOMERS }), /10001 contacts.*at most 10000/); + assert.equal(calls.length, 1); +}); + +test("a list reads every membership page, batch reads in 100s, and uploads only email hashes", async (t) => { + withTokenEnv(t, "pat-env"); + const ids = Array.from({ length: 150 }, (_, i) => String(i + 1)); + const upload = []; + const calls = stubFetch(t, (c) => { + const path = c.url.pathname; + if (path === "/crm/lists/2026-09/77") { + return json({ list: { listId: "77", name: "Target accounts", objectTypeId: "0-1", processingType: "DYNAMIC" } }); + } + if (path === "/crm/lists/2026-09/77/memberships") { + return c.url.searchParams.get("after") === "p2" + ? json({ results: ids.slice(100).map((recordId) => ({ recordId })) }) + : json({ results: ids.slice(0, 100).map((recordId) => ({ recordId })), paging: { next: { after: "p2" } } }); + } + if (path === "/crm/objects/2026-09/contacts/batch/read" && c.method === "POST") { + // Record 150 was deleted after it joined the list: HubSpot answers 207 without it. + const found = c.body.inputs.filter(({ id }) => id !== "150"); + const body = { status: "COMPLETE", results: found.map(({ id }) => ({ id, properties: { email: `User${id}@Example.com` } })) }; + return json(body, found.length < c.body.inputs.length ? 207 : 200); + } + if (c.url.href === UPLOAD_URL && c.method === "POST") { + upload.push(c); + return new Response(null, { status: 201, headers: { location: "D4E10AQ" } }); + } + return undefined; + }); + const li = fakeLinkedIn(); + + const res = await audienceFromHubSpot(li.client, li.getToken, { accountId: "123", name: "Target accounts", listId: "77" }); + + const hubspot = calls.filter((c) => c.url.origin === API); + assert.deepEqual( + hubspot.map((c) => `${c.method} ${c.url.pathname}${c.url.search}`), + [ + "GET /crm/lists/2026-09/77", + "GET /crm/lists/2026-09/77/memberships?limit=100", + "GET /crm/lists/2026-09/77/memberships?limit=100&after=p2", + "POST /crm/objects/2026-09/contacts/batch/read", + "POST /crm/objects/2026-09/contacts/batch/read", + ], + ); + const reads = hubspot.filter((c) => c.method === "POST"); + assert.deepEqual(reads.map((c) => c.body.inputs.length), [100, 50]); + assert.deepEqual(reads[0].body.properties, ["email"]); + assert.ok(hubspot.every((c) => c.headers.Authorization === "Bearer pat-env")); + + assert.equal(res.dryRun, false); + assert.equal(res.segmentId, "9001"); + assert.equal(res.status, "PROCESSING"); + assert.equal(res.audienceType, "contact"); + assert.deepEqual(res.source, { kind: "list", listId: "77", listName: "Target accounts", processingType: "DYNAMIC" }); + assert.equal(res.read, 150); + assert.equal(res.valid, 149); + assert.equal(res.uploaded, 149); + assert.deepEqual(res.skippedReasons, { "not found in HubSpot": 1 }); + + // Only the hashed CSV leaves for LinkedIn: no raw email anywhere in what was sent. + assert.equal(upload.length, 1); + const [header, ...rows] = upload[0].body.trim().split("\n"); + assert.equal(header, "email"); + assert.equal(rows.length, 149); + assert.ok(rows.every((r) => /^[0-9a-f]{64}$/.test(r))); + assert.ok(rows.includes(sha256("user1@example.com"))); + assert.ok(!JSON.stringify(li.client.calls).includes("@")); + const segment = li.client.calls.find((c) => c.path === "/dmpSegments" && c.body?.name); + assert.equal(segment.body.type, "USER_LIST_UPLOAD"); + assert.equal(segment.body.name, "Target accounts"); +}); + +test("a company list sets the type, turns domains into websites, and refuses a contact type", async (t) => { + withTokenEnv(t, "pat-env"); + const companies = { + 1: { name: "Acme", domain: "acme.com" }, + 2: { name: "", domain: null }, + 3: { name: "Globex", domain: "www.globex.io" }, + }; + const calls = stubFetch(t, (c) => { + const path = c.url.pathname; + if (path === "/crm/lists/2026-09/12") return json({ list: { listId: "12", name: "ABM", objectTypeId: "0-2", processingType: "MANUAL" } }); + if (path === "/crm/lists/2026-09/12/memberships") return json({ results: [{ recordId: "1" }, { recordId: "2" }, { recordId: "3" }] }); + if (path === "/crm/objects/2026-09/companies/batch/read") { + return json({ results: c.body.inputs.map(({ id }) => ({ id, properties: companies[id] })) }); + } + return undefined; + }); + + const preview = await previewHubSpotAudience({ name: "ABM", listId: "12" }); + assert.equal(preview.audienceType, "company"); + assert.equal(preview.read, 3); + assert.equal(preview.valid, 2); + assert.deepEqual(preview.skippedReasons, { "no name or domain": 1 }); + assert.deepEqual(preview.columns, ["companyname", "companywebsite"]); + assert.deepEqual(preview.sample, [ + { companyname: "Acme", companywebsite: "https://acme.com" }, + { companyname: "Globex", companywebsite: "https://globex.io" }, + ]); + assert.deepEqual(calls.find((c) => c.method === "POST").body.properties, ["name", "domain"]); + + await assert.rejects(previewHubSpotAudience({ name: "ABM", listId: "12", type: "contact" }), /company list, but type is contact/); +}); + +test("an empty result: the dry run reports zeros and the upload stops before LinkedIn", async (t) => { + withTokenEnv(t, "pat-env"); + stubFetch(t, (c) => (c.url.pathname === "/crm/objects/2026-09/contacts/search" ? json({ total: 0, results: [] }) : undefined)); + const li = fakeLinkedIn(); + + const preview = await previewHubSpotAudience({ name: "Nobody", filters: CUSTOMERS }); + assert.equal(preview.read, 0); + assert.equal(preview.valid, 0); + assert.deepEqual(preview.sample, []); + assert.ok(preview.warnings.some((w) => /returned no contacts/.test(w))); + + await assert.rejects( + audienceFromHubSpot(li.client, li.getToken, { accountId: "1", name: "Nobody", filters: CUSTOMERS }), + /HubSpot search returned no contacts/, + ); + assert.equal(li.client.calls.length, 0); +}); + +test("exactly 1 source: a list or filters, never both or neither", async (t) => { + withTokenEnv(t, "pat-env"); + const calls = stubFetch(t, () => undefined); + await assert.rejects(previewHubSpotAudience({ name: "x" }), /exactly 1 HubSpot source/); + await assert.rejects(previewHubSpotAudience({ name: "x", listId: "1", filters: CUSTOMERS }), /exactly 1 HubSpot source/); + assert.equal(calls.length, 0); +}); + +test("ADWARP_HUBSPOT_TOKEN wins over hubspot.json, and a 403 names the missing scope", async (t) => { + withTokenEnv(t, "pat-env"); + withTokenFile(t, JSON.stringify({ accessToken: "pat-file" })); + const calls = stubFetch(t, () => + json({ status: "error", message: "This app hasn't been granted all required scopes", category: "MISSING_SCOPES", context: { missingScopes: ["crm.lists.read"] } }, 403), + ); + await assert.rejects(previewHubSpotAudience({ name: "x", listId: "5" }), /403.*Missing: crm\.lists\.read/); + assert.equal(calls[0].headers.Authorization, "Bearer pat-env"); +}); + +test("CLI filters parse into HubSpot search filters", () => { + assert.deepEqual(parseHubSpotFilter("lifecyclestage:eq:customer"), { property: "lifecyclestage", operator: "EQ", value: "customer" }); + assert.deepEqual(parseHubSpotFilter("email:HAS_PROPERTY"), { property: "email", operator: "HAS_PROPERTY", value: undefined }); + const between = parseHubSpotFilter("createdate:BETWEEN:2026-01-01T00:00:00Z,2026-06-30T00:00:00Z"); + assert.deepEqual(toHubSpotSearchFilter(between), { + propertyName: "createdate", + operator: "BETWEEN", + value: "2026-01-01T00:00:00Z", + highValue: "2026-06-30T00:00:00Z", + }); + assert.deepEqual(toHubSpotSearchFilter(parseHubSpotFilter("email:CONTAINS_TOKEN:*@acme.com")), { + propertyName: "email", + operator: "CONTAINS_TOKEN", + value: "*@acme.com", + }); + assert.throws(() => parseHubSpotFilter("lifecyclestage:LIKE:customer"), /unknown operator "LIKE"/); + assert.throws(() => parseHubSpotFilter("lifecyclestage"), /::/); + assert.throws(() => parseHubSpotFilter("industry:IN:"), /needs a value/); + assert.throws(() => parseHubSpotFilter("createdate:BETWEEN:2026-01-01"), /low,high/); +}); diff --git a/packages/mcp/scripts/gen-tools.mjs b/packages/mcp/scripts/gen-tools.mjs index 3ff6825..7bdfd9e 100644 --- a/packages/mcp/scripts/gen-tools.mjs +++ b/packages/mcp/scripts/gen-tools.mjs @@ -20,7 +20,7 @@ const OUT = join(ROOT, "docs", "tools.json"); const HOSTED_ROUTE = join(ROOT, "apps", "web", "app", "api", "[transport]", "route.ts"); /** What the hosted route passes to registerTools. Checked against the route below. */ -const HOSTED_OPTIONS = { google: false }; +const HOSTED_OPTIONS = { google: false, hubspot: false }; /** Registers tools on a fresh server, connects a client in memory, and returns tools/list. */ async function listTools(options) { @@ -43,9 +43,9 @@ async function listTools(options) { // The hosted set is only right while the route still registers with HOSTED_OPTIONS. const route = await readFile(HOSTED_ROUTE, "utf8"); -if (!/registerTools\([^)]*\{\s*google:\s*false\s*\}\s*\)/.test(route)) { +if (!/registerTools\([^)]*\{\s*google:\s*false,\s*hubspot:\s*false\s*\}\s*\)/.test(route)) { throw new Error( - `${relative(ROOT, HOSTED_ROUTE)} no longer calls registerTools(server, { google: false }). ` + + `${relative(ROOT, HOSTED_ROUTE)} no longer calls registerTools(server, { google: false, hubspot: false }). ` + "Update HOSTED_OPTIONS in packages/mcp/scripts/gen-tools.mjs to match it.", ); } diff --git a/packages/mcp/src/tools.ts b/packages/mcp/src/tools.ts index 74fa511..81a8a98 100644 --- a/packages/mcp/src/tools.ts +++ b/packages/mcp/src/tools.ts @@ -12,10 +12,13 @@ import { TargetingSpecSchema, uploadAudienceFromFile, audienceFromSalesforce, + audienceFromHubSpot, + previewHubSpotAudience, listConversions, getDmpSegment, requireDefaultAccountId, SalesforceAudienceSchema, + HubSpotAudienceSchema, createCampaignGroup, createCampaign, updateCampaign, @@ -71,6 +74,13 @@ export interface RegisterToolsOptions { * operate whichever Google Ads account the server is configured with. */ google?: boolean; + /** + * Also register `audience_from_hubspot`. On by default for the local stdio + * server, OFF on the hosted handler for the same reason as Google: its token + * would come from a server environment variable, so on a multi-tenant + * endpoint any caller could read the deploy owner's HubSpot contacts. + */ + hubspot?: boolean; } const JOURNAL_NOTE = @@ -200,6 +210,25 @@ export function registerTools(server: McpServer, options: RegisterToolsOptions = }, ); + if (options.hubspot ?? true) { + server.tool( + "audience_from_hubspot", + "Build a matched audience from HubSpot contacts or companies. Pass exactly 1 source: a HubSpot listId (its members), or filters for a CRM search (all ANDed). Reads every page, cleans the records like a CSV upload, and uploads only SHA256 email hashes for contacts (name and website for companies). Pass dryRun=true first: it returns counts and a masked sample and uploads nothing. Needs a HubSpot private app token on this machine (ADWARP_HUBSPOT_TOKEN or ~/.adwarp/hubspot.json); matching takes up to 48h.", + HubSpotAudienceSchema.shape, + async (args) => { + try { + const input = HubSpotAudienceSchema.parse(args); + if (input.dryRun) return ok(await previewHubSpotAudience(input)); + const linkedinAds = await createLinkedInAds(); + const accountId = input.accountId ?? (await requireDefaultAccountId()); + return ok(await audienceFromHubSpot(linkedinAds.client, linkedinAds.getToken, { ...input, accountId })); + } catch (e) { + return fail(e); + } + }, + ); + } + server.tool( "list_conversions", "List the account's existing conversions (insight tags) so you can select one to track on a campaign. Pass accountId or omit to use the default.", diff --git a/packages/shared/src/env.ts b/packages/shared/src/env.ts index 101efe7..677b5f6 100644 --- a/packages/shared/src/env.ts +++ b/packages/shared/src/env.ts @@ -23,11 +23,15 @@ export const LEGACY_ENV_NAMES = { ADWARP_CREATIVE_WORKER_TOKEN: "LIADS_CREATIVE_WORKER_TOKEN", } as const; -export type EnvName = keyof typeof LEGACY_ENV_NAMES; +/** Variables added after the rename. They have no older name to fall back to. */ +export const ENV_NAMES_SINCE_RENAME = ["ADWARP_HUBSPOT_TOKEN"] as const; + +export type EnvName = keyof typeof LEGACY_ENV_NAMES | (typeof ENV_NAMES_SINCE_RENAME)[number]; /** Reads an Adwarp environment variable, falling back to its pre-rename name. */ export function env(name: EnvName): string | undefined { - return nonEmpty(process.env[name]) ?? nonEmpty(process.env[LEGACY_ENV_NAMES[name]]); + const legacy = (LEGACY_ENV_NAMES as Record)[name]; + return nonEmpty(process.env[name]) ?? (legacy ? nonEmpty(process.env[legacy]) : undefined); } function nonEmpty(value: string | undefined): string | undefined { diff --git a/skills/README.md b/skills/README.md index 51c95aa..50785d9 100644 --- a/skills/README.md +++ b/skills/README.md @@ -35,7 +35,7 @@ switches it on in Campaign Manager. Google campaigns can only be switched on in | --- | --- | --- | | `adwarp-launch` | "Launch a campaign for..." | Brief gathering, reach estimate, house guardrails, confirmed draft creation, activation handoff | | `adwarp-experiments` | "Did that change help?" | Hypothesis logging, before/after lift, honest verdicts with confounds stated | -| `adwarp-audiences` | "Audit my audiences" | Inventory, health (stuck, small, stale, orphaned), dry-run uploads, Salesforce refresh loop | +| `adwarp-audiences` | "Audit my audiences" | Inventory, health (stuck, small, stale, orphaned), dry-run uploads, Salesforce and HubSpot refresh loops | | `adwarp-account-audit` | "Is my account set up right?" | Hygiene scorecard: naming, conversions, toggles, overlap, UTMs, plus a ranked fix list | **Google Ads** diff --git a/skills/adwarp-audiences/SKILL.md b/skills/adwarp-audiences/SKILL.md index 73ab8b0..aa4b743 100644 --- a/skills/adwarp-audiences/SKILL.md +++ b/skills/adwarp-audiences/SKILL.md @@ -1,6 +1,6 @@ --- name: adwarp-audiences -description: Audit and operate LinkedIn matched audiences with Adwarp. Inventory and health checks (stuck matching, under 300 members, stale, orphaned), disciplined CSV uploads with dry runs, and the Salesforce-to-audience refresh loop. Use for "audit my audiences", "audience status", "upload this list", "refresh the audience from Salesforce", "why isn't this audience serving". +description: Audit and operate LinkedIn matched audiences with Adwarp. Inventory and health checks (stuck matching, under 300 members, stale, orphaned), disciplined CSV uploads with dry runs, and the Salesforce-to-audience and HubSpot-to-audience refresh loops. Use for "audit my audiences", "audience status", "upload this list", "refresh the audience from Salesforce", "build an audience from HubSpot", "why isn't this audience serving". --- # Adwarp: audience ops @@ -12,8 +12,9 @@ it is in, and how lists get in and stay fresh. ## How to reach Adwarp Prefer the Adwarp MCP tools if loaded: `get_audience_status`, `upload_audience_csv`, -`audience_from_salesforce`, `list_campaigns`, `list_ad_changes`. CLI: `adwarp audience -status `, `adwarp audience upload`, `adwarp audience from-salesforce`. +`audience_from_salesforce`, `audience_from_hubspot`, `list_campaigns`, `list_ad_changes`. +CLI: `adwarp audience status `, `adwarp audience upload`, `adwarp audience +from-salesforce`, `adwarp audience from-hubspot`. ## Inventory @@ -52,9 +53,19 @@ campaign's targeting happens at campaign creation or in Campaign Manager. Retire old segment from the inventory once the swap is done. Suggest journaling the swap (`log_ad_change`) so lift analysis can see it. +## The HubSpot loop + +`audience_from_hubspot` turns a HubSpot list (its members) or a CRM search (property +filters, all ANDed) into a matched audience of contacts or companies, with a read-only +private app token on the user's machine. It is local only. Run it with `dryRun: true` +first and show the counts (read, valid, skipped with reasons) and the masked sample +before uploading. Prefer a list when the user keeps one: it has no result cap, while a +search stops above 10,000 records. Refreshing works like Salesforce: re-run +it for a fresh segment, report the new segment id, swap it in, and journal the swap. + ## Report format - **Verdict:** 1 line on overall audience health. - **Table:** audience, status, size, age, attached campaigns, flag. -- **Actions:** refreshes to run (with the SOQL or CSV source), attachments or +- **Actions:** refreshes to run (with the SOQL, HubSpot list or CSV source), attachments or retirements, anything blocked waiting on LinkedIn matching.