Skip to content

feat(updater): shared calendar dates, ticker and new-fund checks, guard keeps numeric weights #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daggerok merged 1 commit into from
Oct 26, 2026
Merged

feat(updater): shared calendar dates, ticker and new-fund checks, guard keeps numeric weights #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Each fund carries a derived `metrics` object that powers the catalog columns sha
- `ytd` / `tr1y` - official YTD and 1-year returns -> *YTD Return*, *TR 1Y*
- `cagr3y` / `cagr5y` / `cagr10y` - published annualized 3Y/5Y/10Y figures -> *CAGR 3Y/5Y/10Y*
- `tr3y` / `tr5y` / `tr10y` - cumulative 3Y/5Y/10Y figures `(1 + CAGR)^n - 1` -> *TR 3Y/5Y/10Y*
- `siAnn` - since-inception annualized -> *SI Ann.*; `null` for funds with less than one year of history at the performance date
- `siAnn` - since-inception annualized -> *SI Ann.*; `null` for funds with less than one year of history at the performance date, and every 1, 3, 5 or 10-year figure is `null` before that anniversary of the inception (calendar date, inclusive)
- `dividendYield` - official SSGA Fund Dividend Yield, else an indicated yield (latest distribution x payments per year / price; semi-annual = 2), an estimate derived from SSGA data. HECO, XITK and XSW publish an official `0.00%`, which is kept as `0` (a real zero, not a missing value)
- `dividendYieldBasis` - code of the definition behind `dividendYield`, `null` exactly when `dividendYield` is `null` (table below)
- `secYield` - official 30-day SEC yield when published; unavailable otherwise, never `0`
Expand All @@ -83,11 +83,11 @@ Each fund carries a derived `metrics` object that powers the catalog columns sha

Expense ratio: `terValue` / `ter` are the NET expense ratio (after waivers; the single published figure when there is no waiver), `terGrossValue` / `terGross` the gross ratio. Both come from the product-data workbook ("Gross Expense Ratio", "Net Expense Ratio"), with the fund finder expense ratio as the gross fallback. The `TER` filter uses the net value.

Sources and scope: SPDR has no Yahoo Finance and no SEC EDGAR source. Everything comes from SSGA's own files, so there is no `SKIP_YAHOO`, `EDGAR_FALLBACK` or `SEC_UA` control. Yields are the official 30-day SEC yield and Fund Dividend Yield from SSGA's product-data workbook. When the product-data workbook or the dividend-distribution feed fails or comes back empty, every published fund is kept exactly as published and reported failed; only new funds are built. Identifier columns of the holdings workbooks (Ticker, Identifier, SEDOL) are kept as published text, so CUSIP `037833100` and A-share ticker `002594` keep their leading zeros.
Sources and scope: SPDR has no Yahoo Finance and no SEC EDGAR source. Everything comes from SSGA's own files, so there is no `SKIP_YAHOO`, `EDGAR_FALLBACK` or `SEC_UA` control. Yields are the official 30-day SEC yield and Fund Dividend Yield from SSGA's product-data workbook. When the product-data workbook or the dividend-distribution feed fails or comes back empty, every published fund is kept exactly as published and reported failed; only new funds are built. Identifier columns of the holdings workbooks (Ticker, Identifier, SEDOL) are kept as published text, so CUSIP `037833100` keeps its leading zero. A holdings ticker is published bare only for a U.S. listing: SSGA's Ticker of a line traded in another currency is the local code (ACA is Arcosa in USD and Credit Agricole in EUR), so a line whose Local Currency is not USD gets `-` and is keyed by its Identifier; futures, options and cash lines get `-` too. Dates (`31-Aug-2026`, `Aug 31 2026`, `08/31/2026`) are read as plain calendar text, never through the local time zone.

Commodity trusts such as GLD and GLDM hold metal and publish no holdings workbook. That is valid: they get `meta.json`, NAV, returns and history, with holdings `status: "empty"` (`holdingsStatus` in `index.json`). A fund that published holdings before is never emptied by a transient 404.

Consistency: each fund is fetched and computed completely in memory, then written once (pages, then `meta.json`, then stale pages are removed; every JSON file is written through a temp file and renamed). If any required file of a fund fails to download, that fund keeps its previous `meta.json`, pages and `index.json` row together, so the workflow never publishes a new return next to stale files. A fund without `funds/<T>/meta.json` has `dataFile: null` in `index.json`. A rerun with identical upstream data changes nothing, and `generatedAt` moves only when content moved. Newly discovered funds are announced as `NEW FUNDS: A, B` in the run output and in the job summary. Every request times out after 45 s (headers and body) and is retried per `MAX_RETRIES`; the run stops starting new funds after 25 minutes and still writes the index; it exits non-zero when every selected fund failed.
Consistency: each fund is fetched and computed completely in memory, then written once (pages, then `meta.json`, then stale pages are removed; every JSON file is written through a temp file and renamed). If any required file of a fund fails to download, that fund keeps its previous `meta.json`, pages and `index.json` row together, so the workflow never publishes a new return next to stale files. A fund without `funds/<T>/meta.json` has `dataFile: null` in `index.json`. A rerun with identical upstream data changes nothing, and `generatedAt` moves only when content moved. `NEW FUNDS: A, B` (run output and job summary) names a fund when it is published for the first time, also when its first read failed or it was not selected and a later run publishes it; `NEW FUNDS (not published yet): ...` names new catalog funds that have no data yet. Every request times out after 45 s (headers and body) and is retried per `MAX_RETRIES`; the run stops starting new funds after 25 minutes and still writes the index; it exits non-zero when every selected fund failed.

Returns come from SSGA's own NAV series: `PERFORMANCE_*` filters use month-end NAV returns and `TOTAL_RETURN_*` filters use the separate quarter-end series (3Y/5Y/10Y are annualized CAGR). `DIVIDEND_YIELD` and `SEC_YIELD` filters use the same `metrics` values. There are no market-price or Yahoo estimates in this feed, and no ticker exclusions. Funds not selected for a successful update keep their prior published metadata and data files. `TICKERS` combines with the other filters using AND logic; it does not override them.

Expand All @@ -104,7 +104,7 @@ Every control is in `scripts/update-data.config.json`; the table shows the shipp
| `TER` | `:` | Net expense ratio range in % (strict `min:max`; gross when no waiver is published) |
| `DIVIDEND_YIELD` | `:` | Dividend yield range in % (strict `min:max`); official Fund Dividend Yield, else the indicated yield from the latest distribution; funds without a value fail a bounded range |
| `SEC_YIELD` | `:` | Official 30-day SEC yield range in % (strict `min:max`); funds without a published value fail a bounded range |
| `TICKERS` | empty (all) | Space-, comma- or semicolon-separated ticker allowlist, e.g. `SPY SPYG SPYD SDY XLK`; a ticker that is not in the SPDR catalog is an error |
| `TICKERS` | empty (all) | Space-, comma- or semicolon-separated ticker allowlist, e.g. `SPY SPYG SPYD SDY XLK`; a ticker that is neither in the live SPDR catalog nor a published fund is an error before any other request (nothing is written) |
| `HOLDINGS_PAGE_SIZE` | `250` | Rows in each generated current-holdings JSON page |
| `HISTORY_PAGE_SIZE` | `1000` | Rows in each generated daily-history JSON page |
| `HISTORY_RANGE` | `max` | Window for the generated NAV and premium/discount history: `max`, `Ny` or `Nmo` counted back from the newest row (for example `5y`, `18mo`); SSGA serves the full workbook, so the window trims the generated pages, not the download; older rows already published are kept, a shorter window never deletes history |
Expand Down
88 changes: 84 additions & 4 deletions scripts/update-data.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ import {
setSoftDeadlineMs,
sheetToTable,
toIsoDate,
withListedTickers,
HOLDINGS_GUARD_RELAX,
holdingsWeightCount,
} from './update-data';

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -284,6 +287,17 @@ describe('parsing', () => {
expect(sheet.table.rows.map((r) => r[6])).toEqual(['180135563', '58400000']);
});

test('holdings tickers: a bare symbol only for a U.S. listing; local codes, futures, options and cash get -', () => {
const headers = ['Name', 'Ticker', 'Identifier', 'Weight', 'Local Currency', 'Number of Contracts'];
const rows = [
['ARCOSA INC', 'ACA', '039653100', '1', 'USD', '-'], ['CREDIT AGRICOLE SA', 'ACA', '4262335', '1', 'EUR', '-'], ['TAIWAN SEMICONDUCTOR', '2330', '6889106', '1', 'TWD', '-'],
['S+P500 EMINI FUT DEC26', 'ESZ6', 'ESZ6', '1', 'USD', '4'], ['MICROSOFT CORP NOV26 515 PUT', 'MSFT261120P00515000', 'X', '1', 'USD', '2'], ['U.S. Dollar', 'CASH_USD', 'CASH_USD', '1', 'USD', '-'],
];
expect(withListedTickers({ headers, rows }).rows.map((row) => row[1])).toEqual(['ACA', '-', '-', '-', '-', '-']);
// a fund of SPDR funds publishes no currency: its tickers stay, its futures do not
expect(withListedTickers({ headers: ['Name', 'Ticker', 'Weight'], rows: [['STATE STREET SPDR PORTFOLIO S+', 'SPYM', '50'], ['AUST 10Y BOND FUT DEC26', 'XMZ6', '1']] }).rows.map((row) => row[1])).toEqual(['SPYM', '-']);
});

test('holdings guard: header-only, renamed weight column, broken weights, a bad weight sum and a halved sheet are rejected', () => {
const headers = ['Name', 'Ticker', 'Weight'];
const rows = [{ Name: 'A', Ticker: 'A', Weight: '60%' }, { Name: 'B', Ticker: 'B', Weight: '40%' }];
Expand Down Expand Up @@ -341,9 +355,12 @@ describe('parsing 2', () => {
expect(() => parseProductDataSheet(buildXlsx([['not', 'a', 'header', 'row']]))).toThrow();
});

test('dates: SSGA formats normalize to ISO, garbage becomes null', () => {
for (const ok of ['Aug 31 2026', '08/31/2026', '2026-08-31']) expect(toIsoDate(ok)).toBe('2026-08-31');
for (const bad of ['Feb 30 2026', '', null, '-']) expect(toIsoDate(bad)).toBeNull();
test('dates: SSGA formats normalize to ISO in every time zone, garbage becomes null', () => {
for (const tz of ['Asia/Tokyo', 'America/Los_Angeles', 'UTC']) {
process.env.TZ = tz;
for (const ok of ['Aug 31 2026', '08/31/2026', '2026-08-31', 'Aug 31, 2026', 'August 31 2026', '31-Aug-2026', '08/31/26']) expect(toIsoDate(ok)).toBe('2026-08-31');
for (const bad of ['Feb 30 2026', '', null, '-']) expect(toIsoDate(bad)).toBeNull();
}
});
});

Expand Down Expand Up @@ -435,6 +452,9 @@ describe('metrics 2', () => {
expect(deriveCatalogMetrics({ ...base, inceptionDate: 'Feb 10 2026' }, 10, null).siAnn).toBeNull();
expect(deriveCatalogMetrics({ ...base, inceptionDate: 'Aug 31 2025' }, 10, null).siAnn).toBe(4.2);
expect(deriveCatalogMetrics({ ...base, inceptionDate: 'Sep 01 2025' }, 10, null).siAnn).toBeNull();
// every horizon the fund is too young for is null, not only since inception (calendar anniversary)
const young2 = deriveCatalogMetrics({ ...base, yr1: 3, yr3: 4, yr5: 5, inceptionDate: 'Aug 31 2024' }, 10, null);
expect([young2.tr1y, young2.cagr3y, young2.tr3y, young2.cagr5y, young2.siAnn]).toEqual([3, null, null, null, 4.2]);
});

test('returnsBasis and performanceAsOf travel together at the end of every metrics object', () => {
Expand Down Expand Up @@ -717,7 +737,7 @@ describe('pipeline 3', () => {
test('unknown TICKERS is an error that writes nothing; all selected funds failing is an error', async () => {
const dir = tempRoot();
installMock();
await expect(runMain({ TICKERS: 'XXXX' })).rejects.toThrow(/unknown ticker/);
await expect(runMain({ TICKERS: 'XXXX' })).rejects.toThrow('TICKERS: XXXX not in the SPDR catalog or published feed; nothing was fetched or written');
expect(existsSync(join(dir, 'index.json'))).toBe(false);
installMock({ failNavhist: new Set(['SPY', 'XLK', 'GLD']) });
await expect(runMain({ CONCURRENCY: '3' })).rejects.toThrow(/all 3 selected funds failed/);
Expand Down Expand Up @@ -794,6 +814,66 @@ function statSyncMtime(path: string): number {
const settled = (p: Promise<unknown>, ms = 3000) =>
Promise.race([p.then((v) => (typeof v === 'string' ? v : 'resolved'), () => 'rejected'), new Promise<string>((r) => setTimeout(() => r('hung'), ms))]);

describe('pipeline 4', () => {
test('a relaxed guard still keeps numeric published weights from an all-dash sheet (Q24)', async () => {
const dir = tempRoot();
installMock();
await runMain();
const before = listFiles(dir).map((f) => `${f}:${readFileSync(join(dir, f), 'utf8')}`);
const good = globalThis.fetch;
const dashes = holdingsSheet('SPY').map((r, i) => (i === 4 ? r.map((cell, c) => (c === 2 ? '-' : cell)) : r));
globalThis.fetch = (async (input: unknown, init?: RequestInit) => (String(input).includes('holdings-daily-us-en-spy')
? new Response(new Uint8Array(buildXlsx(dashes)))
: good(input as string, init))) as unknown as typeof fetch;
HOLDINGS_GUARD_RELAX.SPY = ['numeric', 'sum'];
const lines: string[] = [];
try {
await runMain({ TICKERS: 'SPY,XLK' }, lines);
} finally {
delete HOLDINGS_GUARD_RELAX.SPY;
}
expect(listFiles(dir).map((f) => `${f}:${readFileSync(join(dir, f), 'utf8')}`)).toEqual(before);
expect(lines.join('\n')).toMatch(/SPY +failed .*reason=holdings guard: no row has a numeric weight/);
expect(holdingsWeightCount(['Name', 'Weight'], [['A', '-'], ['B', '2']])).toBe(1);
});

test('TICKERS outside the live catalog stops the run before any fund request; a stale catalog-only row is dropped (Q27)', async () => {
const dir = tempRoot();
installMock();
await runMain();
const index = readJson(dir, 'index.json');
index.funds.push({ ...row(dir, 'XLK'), ticker: 'NOPE', dataFile: null });
writeFileSync(join(dir, 'index.json'), JSON.stringify(index));
const before = listFiles(dir).map((f) => `${f}:${readFileSync(join(dir, f), 'utf8')}`);
const requests: string[] = [];
const good = globalThis.fetch;
globalThis.fetch = (async (input: unknown, init?: RequestInit) => { requests.push(String(input)); return good(input as string, init); }) as unknown as typeof fetch;
await expect(runMain({ TICKERS: 'NOPE' })).rejects.toThrow('TICKERS: NOPE not in the SPDR catalog or published feed; nothing was fetched or written');
expect(requests.filter((url) => !url.includes('fundfinder'))).toEqual([]);
expect(listFiles(dir).map((f) => `${f}:${readFileSync(join(dir, f), 'utf8')}`)).toEqual(before);
await runMain({ TICKERS: 'SPY' });
expect(readJson(dir, 'index.json').funds.map((f: any) => f.ticker)).toEqual(['GLD', 'SPY', 'XLK']);
});

test('NEW FUNDS announces a fund on its first successful publish, a new catalog-only fund as not published yet (Q28)', async () => {
const dir = tempRoot();
installMock({ tickers: ['SPY', 'XLK'] });
await runMain();
installMock();
process.env.GITHUB_STEP_SUMMARY = join(dir, '..', `${dir.split('/').pop()}-summary.md`);
dirs.push(process.env.GITHUB_STEP_SUMMARY);
const announced = async (env: Record<string, string>): Promise<string[]> => {
const lines: string[] = [];
await runMain(env, lines);
return lines.filter((line) => /^NEW FUNDS/.test(line));
};
expect(await announced({ TICKERS: 'SPY' })).toEqual(['NEW FUNDS (not published yet): GLD']);
expect(readFileSync(process.env.GITHUB_STEP_SUMMARY, 'utf8')).toContain('### NEW FUNDS\n\nNEW FUNDS (not published yet): GLD\n');
expect(await announced({ TICKERS: 'SPY' })).toEqual([]);
expect(await announced({ TICKERS: 'GLD' })).toEqual(['NEW FUNDS: GLD']);
});
});

describe('network', () => {
test('a stalled connection (headers never arrive) is aborted by the timeout', async () => {
setRequestTimeoutMs(100);
Expand Down
Loading
Loading