From 0ddd8a1a01bed44e3c0d43055d5994a7923341b2 Mon Sep 17 00:00:00 2001 From: Usman Shahid Date: Sun, 24 May 2026 15:32:24 +0400 Subject: [PATCH] chore: cleanup design docs --- docs/api.md | 6163 -------------- .../plans/2026-05-15-harvester-v2.md | 7052 ----------------- .../specs/2026-05-15-harvester-v2-design.md | 409 - 3 files changed, 13624 deletions(-) delete mode 100644 docs/api.md delete mode 100644 docs/superpowers/plans/2026-05-15-harvester-v2.md delete mode 100644 docs/superpowers/specs/2026-05-15-harvester-v2-design.md diff --git a/docs/api.md b/docs/api.md deleted file mode 100644 index edea085..0000000 --- a/docs/api.md +++ /dev/null @@ -1,6163 +0,0 @@ -# **Free Stock Market API and Financial Statements API** - -FMP is your source for the most reliable and accurate Stock Market API and Financial Data API available. Whether you're looking for [real-time stock prices](https://site.financialmodelingprep.com/developer/docs#real-time-full-price-quote), [financial statements](https://site.financialmodelingprep.com/developer/docs#income-statements-financial-statements), or [historical data](https://site.financialmodelingprep.com/developer/docs#earnings-historical-earnings), we offer a comprehensive solution to meet all your financial data needs. - -Our platform provides free stock market data, including audited, standardized, and real-time updates of income statements, balance sheets, and cash flow statements on a quarterly and annual basis. - -We also offer a wide range of historical stock prices, from 1 minute, 15 minutes, 30 minutes, 1 hour and daily intervals, ensuring you have the data you need. - -*Note: When adding the API key to your requests, ensure to use **\&apikey=** if other query parameters already exist in the endpoint.* - -*Any field marked with an \* is required* - -## **Search** - -[Stock Symbol Search API](https://site.financialmodelingprep.com/developer/docs/stable/search-symbol) - -Easily find the ticker symbol of any stock with the FMP Stock Symbol Search API. Search by company name or symbol across multiple global markets. - -**Endpoint:** - -https://financialmodelingprep.com/stable/search-symbol?*query*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| query\* | string | AAPL | -| limit | number | 50 | -| exchange | string | NASDAQ | - -Response -\[ - { - "symbol": "AAPL", - "name": "Apple Inc.", - "currency": "USD", - "exchangeFullName": "NASDAQ Global Select", - "exchange": "NASDAQ" - } -\] - -[Company Name Search API](https://site.financialmodelingprep.com/developer/docs/stable/search-name) - -Search for ticker symbols, company names, and exchange details for equity securities and ETFs listed on various exchanges with the FMP Name Search API. This endpoint is useful for retrieving ticker symbols when you know the full or partial company or asset name but not the symbol identifier. - -**Endpoint:** - -https://financialmodelingprep.com/stable/search-name?*query*\=AA - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| query\* | string | AAPL | -| limit | number | 50 | -| exchange | string | NASDAQ | - -Response -\[ - { - "symbol": "AAGUSD", - "name": "A AG USD", - "currency": "USD", - "exchangeFullName": "CCC", - "exchange": "CRYPTO" - } -\] - -[Stock Screener API](https://site.financialmodelingprep.com/developer/docs/stable/search-company-screener) - -Discover stocks that align with your investment strategy using the FMP Stock Screener API. Filter stocks based on market cap, price, volume, beta, sector, country, and more to identify the best opportunities. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*company-screener* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| marketCapMoreThan | number | 1000000 | -| marketCapLowerThan | number | 1000000000 | -| sector | string | Technology | -| industry | string | Consumer Electronics | -| betaMoreThan | number | 0.5 | -| betaLowerThan | number | 1.5 | -| priceMoreThan | number | 10 | -| priceLowerThan | number | 200 | -| dividendMoreThan | number | 0.5 | -| dividendLowerThan | number | 2 | -| volumeMoreThan | number | 1000 | -| volumeLowerThan | number | 1000000 | -| exchange | string | NASDAQ | -| country | string | US | -| isEtf | boolean | false | -| isFund | boolean | false | -| isActivelyTrading | boolean | true | -| limit | number | 1000 | -| includeAllShareClasses | boolean | false | - -Response -\[ - { - "symbol": "AAPL", - "companyName": "Apple Inc.", - "marketCap": 3435062313000, - "sector": "Technology", - "industry": "Consumer Electronics", - "beta": 1.24, - "price": 225.93, - "lastAnnualDividend": 1, - "volume": 43010091, - "exchange": "NASDAQ Global Select", - "exchangeShortName": "NASDAQ", - "country": "US", - "isEtf": false, - "isFund": false, - "isActivelyTrading": true - } -\] - -## **Directory** - -[Company Symbols List API](https://site.financialmodelingprep.com/developer/docs/stable/company-symbols-list) - -Easily retrieve a comprehensive list of financial symbols with the FMP Company Symbols List API. Access a broad range of stock symbols and other tradable financial instruments from various global exchanges, helping you explore the full range of available securities. - -Response -\[ - { - "symbol": "6898.HK", - "companyName": "China Aluminum Cans Holdings Limited" - } -\] - -[Financial Statement Symbols List API](https://site.financialmodelingprep.com/developer/docs/stable/financial-symbols-list) - -Access a comprehensive list of companies with available financial statements through the FMP Financial Statement Symbols List API. Find companies listed on major global exchanges and obtain up-to-date financial data including income statements, balance sheets, and cash flow statements, are provided. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*financial-statement-symbol-list* - -Response -\[ - { - "symbol": "6898.HK", - "companyName": "China Aluminum Cans Holdings Limited", - "tradingCurrency": "HKD", - "reportingCurrency": "HKD" - } -\] - -[ETF Symbol Search API](https://site.financialmodelingprep.com/developer/docs/stable/etfs-list) - -Quickly find ticker symbols and company names for Exchange Traded Funds (ETFs) using the FMP ETF Symbol Search API. This tool simplifies identifying specific ETFs by their name or ticker. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*etf-list* - -Response -\[ - { - "symbol": "GULF", - "name": "WisdomTree Middle East Dividend Fund" - } -\] -[Actively Trading List API](https://site.financialmodelingprep.com/developer/docs/stable/actively-trading-list) - -List all actively trading companies and financial instruments with the FMP Actively Trading List API. This endpoint allows users to filter and display securities that are currently being traded on public exchanges, ensuring you access real-time market activity. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*actively-trading-list* - -Response -\[ - { - "symbol": "6898.HK", - "name": "China Aluminum Cans Holdings Limited" - } -\] - -[Earnings Transcript List API](https://site.financialmodelingprep.com/developer/docs/stable/earnings-transcript-list) - -Access available earnings transcripts for companies with the FMP Earnings Transcript List API. Retrieve a list of companies with earnings transcripts, along with the total number of transcripts available for each company. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*earnings-transcript-list* - -Response -\[ - { - "symbol": "MCUJF", - "companyName": "Medicure Inc.", - "noOfTranscripts": "16" - } -\] -[Available Exchanges API](https://site.financialmodelingprep.com/developer/docs/stable/available-exchanges) - -Access a complete list of supported stock exchanges using the FMP Available Exchanges API. This API provides a comprehensive overview of global stock exchanges, allowing users to identify where securities are traded and filter data by specific exchanges for further analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*available-exchanges* - -Response -\[ - { - "exchange": "AMEX", - "name": "New York Stock Exchange Arca", - "countryName": "United States of America", - "countryCode": "US", - "symbolSuffix": "N/A", - "delay": "Real-time" - } -\] - -[Available Sectors API](https://site.financialmodelingprep.com/developer/docs/stable/available-sectors) - -Access a complete list of industry sectors using the FMP Available Sectors API. This API helps users categorize and filter companies based on their respective sectors, enabling deeper analysis and more focused queries across different industries. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*available-sectors* - -Response -\[ - { - "sector": "Basic Materials" - } -\] -[Available Industries API](https://site.financialmodelingprep.com/developer/docs/stable/available-industries) - -Access a comprehensive list of industries where stock symbols are available using the FMP Available Industries API. This API helps users filter and categorize companies based on their industry for more focused research and analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*available-industries* - -Response -\[ - { - "industry": "Steel" - } -\] - -[Available Countries API](https://site.financialmodelingprep.com/developer/docs/stable/available-countries) - -Access a comprehensive list of countries where stock symbols are available with the FMP Available Countries API. This API enables users to filter and analyze stock symbols based on the country of origin or the primary market where the securities are traded. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*available-countries* - -#### - -\[ - { - "country": "FK" - } -\] - -## **Analyst** - -[Financial Estimates API](https://site.financialmodelingprep.com/developer/docs/stable/financial-estimates) - -Retrieve analyst financial estimates for stock symbols with the FMP Financial Estimates API. Access projected figures like revenue, earnings per share (EPS), and other key financial metrics as forecasted by industry analysts to inform your investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/analyst-estimates?*symbol*\=AAPL&*period*\=annual&*page*\=0&*limit*\=10 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| period\* | string | annual,quarter | -| page | number | 0 | -| limit | number | 10 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2029-09-28", - "revenueLow": 483092500000, - "revenueHigh": 483093500000, - "revenueAvg": 483093000000, - "ebitdaLow": 155952166036, - "ebitdaHigh": 155952488856, - "ebitdaAvg": 155952327446, - "ebitLow": 140628295747, - "ebitHigh": 140628586847, - "ebitAvg": 140628441297, - "netIncomeLow": 139446957701, - "netIncomeHigh": 157185372990, - "netIncomeAvg": 149150359609, - "sgaExpenseLow": 31694652812, - "sgaExpenseHigh": 31694718420, - "sgaExpenseAvg": 31694685616, - "epsAvg": 9.68, - "epsHigh": 10.20148, - "epsLow": 9.05024, - "numAnalystsRevenue": 16, - "numAnalystsEps": 6 - } -\] - -[Ratings Snapshot API](https://site.financialmodelingprep.com/developer/docs/stable/ratings-snapshot) - -Quickly assess the financial health and performance of companies with the FMP Ratings Snapshot API. This API provides a comprehensive snapshot of financial ratings for stock symbols in our database, based on various key financial ratios. - -**Endpoint:** - -https://financialmodelingprep.com/stable/ratings-snapshot?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 1 | - -Response - -\[ - { - "symbol": "AAPL", - "rating": "A-", - "overallScore": 4, - "discountedCashFlowScore": 3, - "returnOnEquityScore": 5, - "returnOnAssetsScore": 5, - "debtToEquityScore": 4, - "priceToEarningsScore": 2, - "priceToBookScore": 1 - } -\] - -[Historical Ratings API](https://site.financialmodelingprep.com/developer/docs/stable/historical-ratings) - -Track changes in financial performance over time with the FMP Historical Ratings API. This API provides access to historical financial ratings for stock symbols in our database, allowing users to view ratings and key financial metric scores for specific dates. - -**Endpoint:** - -https://financialmodelingprep.com/stable/ratings-historical?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 1 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "rating": "A-", - "overallScore": 4, - "discountedCashFlowScore": 3, - "returnOnEquityScore": 5, - "returnOnAssetsScore": 5, - "debtToEquityScore": 4, - "priceToEarningsScore": 2, - "priceToBookScore": 1 - } -\] - -[Price Target Summary API](https://site.financialmodelingprep.com/developer/docs/stable/price-target-summary) - -Gain insights into analysts' expectations for stock prices with the FMP Price Target Summary API. This API provides access to average price targets from analysts across various timeframes, helping investors assess future stock performance based on expert opinions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/price-target-summary?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "lastMonthCount": 1, - "lastMonthAvgPriceTarget": 200.75, - "lastQuarterCount": 3, - "lastQuarterAvgPriceTarget": 204.2, - "lastYearCount": 48, - "lastYearAvgPriceTarget": 232.99, - "allTimeCount": 167, - "allTimeAvgPriceTarget": 201.21, - "publishers": "\[\\"Benzinga\\",\\"StreetInsider\\",\\"TheFly\\",\\"Pulse 2.0\\",\\"TipRanks Contributor\\",\\"MarketWatch\\",\\"Investing\\",\\"Barrons\\",\\"Investor's Business Daily\\"\]" - } -\] - -[Price Target Consensus API](https://site.financialmodelingprep.com/developer/docs/stable/price-target-consensus) - -Access analysts' consensus price targets with the FMP Price Target Consensus API. This API provides high, low, median, and consensus price targets for stocks, offering investors a comprehensive view of market expectations for future stock prices. - -**Endpoint:** - -https://financialmodelingprep.com/stable/price-target-consensus?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "targetHigh": 300, - "targetLow": 200, - "targetConsensus": 251.7, - "targetMedian": 258 - } -\] - -[Price Target News API](https://site.financialmodelingprep.com/developer/docs/stable/price-target-news) - -Stay informed with real-time updates on analysts' price targets for stocks using the FMP Price Target News API. Access the latest forecasts, stock prices at the time of the update, and direct links to trusted news sources for deeper insights. - -**Endpoint:** - -https://financialmodelingprep.com/stable/price-target-news?*symbol*\=AAPL&*page*\=0&*limit*\=10 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 10 | -| page | number | 0 | - -Response - -\[ - { - "symbol": "AAPL", - "publishedDate": "2025-01-21T01:24:32.000Z", - "newsURL": "https://www.benzinga.com/markets/equities/25/01/43087992/apple-gets-rare-downgrade-from-jefferies-analyst-warns-on-slowing-revenue-growth-missed-forecast", - "newsTitle": "Apple Gets Rare Downgrade From Jefferies, Analyst Warns On Slowing Revenue Growth, Missed Forecasts, And Falling iPhone Demand", - "analystName": "Edison Lee", - "priceTarget": 200.75, - "adjPriceTarget": 200.75, - "priceWhenPosted": 229.98, - "newsPublisher": "Benzinga", - "newsBaseURL": "benzinga.com", - "analystCompany": "Jefferies" - } -\] - -[Price Target Latest News API](https://site.financialmodelingprep.com/developer/docs/stable/price-target-latest-news) - -Stay updated with the most recent analyst price target updates for all stock symbols using the FMP Price Target Latest News API. Get access to detailed forecasts, stock prices at the time of the update, analyst insights, and direct links to news sources for deeper analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/price-target-latest-news?*page*\=0&*limit*\=10 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| limit | number | 10 | -| page | number | 0 | - -Response - -\[ - { - "symbol": "OLN", - "publishedDate": "2025-02-03T18:23:58.000Z", - "newsURL": "https://www.benzinga.com/25/02/43444520/these-analysts-cut-their-forecasts-on-olin-after-q4-earnings", - "newsTitle": "These Analysts Cut Their Forecasts On Olin After Q4 Earnings", - "analystName": "Peter Osterland", - "priceTarget": 32, - "adjPriceTarget": 32, - "priceWhenPosted": 27.76, - "newsPublisher": "Benzinga", - "newsBaseURL": "benzinga.com", - "analystCompany": "Truist Financial" - } -\] - -[Stock Grades API](https://site.financialmodelingprep.com/developer/docs/stable/grades) - -Access the latest stock grades from top analysts and financial institutions with the FMP Grades API. Track grading actions, such as upgrades, downgrades, or maintained ratings, for specific stock symbols, providing valuable insight into how experts evaluate companies over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/grades?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2025-01-31", - "gradingCompany": "Morgan Stanley", - "previousGrade": "Overweight", - "newGrade": "Overweight", - "action": "maintain" - } -\] - -[Historical Stock Grades API](https://site.financialmodelingprep.com/developer/docs/stable/historical-grades) -Globe Flag - -Access a comprehensive record of analyst grades with the FMP Historical Grades API. This tool allows you to track historical changes in analyst ratings for specific stock symbol - -**Endpoint:** - -https://financialmodelingprep.com/stable/grades-historical?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2025-02-01", - "analystRatingsBuy": 8, - "analystRatingsHold": 14, - "analystRatingsSell": 2, - "analystRatingsStrongSell": 2 - } -\] - -[Stock Grades Summary API](https://site.financialmodelingprep.com/developer/docs/stable/grades-summary) - -Quickly access an overall view of analyst ratings with the FMP Grades Summary API. This API provides a consolidated summary of market sentiment for individual stock symbols, including the total number of strong buy, buy, hold, sell, and strong sell ratings. Understand the overall consensus on a stock’s outlook with just a few data points. - -**Endpoint:** - -https://financialmodelingprep.com/stable/grades-consensus?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "strongBuy": 1, - "buy": 29, - "hold": 11, - "sell": 4, - "strongSell": 0, - "consensus": "Buy" - } -\] - -[Stock Grade News API](https://site.financialmodelingprep.com/developer/docs/stable/grade-news) - -Stay informed on the latest analyst grade changes with the FMP Grade News API. This API provides real-time updates on stock rating changes, including the grading company, previous and new grades, and the action taken. Direct links to trusted news sources and stock prices at the time of the update help you stay ahead of market trends and analyst opinions for specific stock symbols. - -**Endpoint:** - -https://financialmodelingprep.com/stable/grades-news?*symbol*\=AAPL&*page*\=0&*limit*\=1 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| page | number | 0 | -| limit | number | 1 | - -Response -\[ - { - "symbol": "AAPL", - "publishedDate": "2025-01-31T10:11:47.000Z", - "newsURL": "https://www.benzinga.com/25/01/43379061/why-apple-shares-are-trading-higher-here-are-20-stocks-moving-premarket", - "newsTitle": "Why Apple Shares Are Trading Higher; Here Are 20 Stocks Moving Premarket", - "newsBaseURL": "benzinga.com", - "newsPublisher": "Benzinga", - "newGrade": "Buy", - "previousGrade": "Hold", - "gradingCompany": "Maxim Group", - "action": "initialise", - "priceWhenPosted": 237.59 - } -\] - -[Stock Grade Latest News API](https://site.financialmodelingprep.com/developer/docs/stable/grade-latest-news) - -Stay informed on the latest stock rating changes with the FMP Grade Latest News API. This API provides the most recent updates on analyst ratings for all stock symbols, including links to the original news sources. Track stock price movements, grading firm actions, and market sentiment shifts in real time, sourced from trusted publishers. - -**Endpoint:** - -https://financialmodelingprep.com/stable/grades-latest-news?*page*\=0&*limit*\=10 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 10 | - -Response - -\[ - { - "symbol": "PYPL", - "publishedDate": "2025-02-04T19:18:04.000Z", - "newsURL": "https://www.benzinga.com/25/02/43475080/paypal-beats-q4-estimates-as-transaction-margins-and-payment-volume-drive-growth-eyes-2025-growth-with-strong-tmd", - "newsTitle": "PayPal Transaction Margins and Payment Volume Drive Growth, Eyes 2025 Growth With Strong TMD Ahead of Investor Day: Analyst", - "newsBaseURL": "benzinga.com", - "newsPublisher": "Benzinga", - "newGrade": "Overweight", - "previousGrade": "Overweight", - "gradingCompany": "J.P. Morgan", - "action": "hold", - "priceWhenPosted": 77.725 - } -\] - -## **Calendar** - -[Dividends Company API](https://site.financialmodelingprep.com/developer/docs/stable/dividends-company) - -Stay informed about upcoming dividend payments with the FMP Dividends Company API. This API provides essential dividend data for individual stock symbols, including record dates, payment dates, declaration dates, and more. - -**Endpoint:** - -https://financialmodelingprep.com/stable/dividends?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-10", - "recordDate": "2025-02-10", - "paymentDate": "2025-02-13", - "declarationDate": "2025-01-30", - "adjDividend": 0.25, - "dividend": 0.25, - "yield": 0.42955326460481097, - "frequency": "Quarterly" - } -\] - -[Dividends Calendar API](https://site.financialmodelingprep.com/developer/docs/stable/dividends-calendar) - -Stay informed on upcoming dividend events with the Dividend Events Calendar API. Access a comprehensive schedule of dividend-related dates for all stocks, including record dates, payment dates, declaration dates, and dividend yields. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*dividends-calendar* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response -\[ - { - "symbol": "1D0.SI", - "date": "2025-02-04", - "recordDate": "", - "paymentDate": "", - "declarationDate": "", - "adjDividend": 0.01, - "dividend": 0.01, - "yield": 6.25, - "frequency": "Semi-Annual" - } -\] - -[Earnings Report API](https://site.financialmodelingprep.com/developer/docs/stable/earnings-company) - -Retrieve in-depth earnings information with the FMP Earnings Report API. Gain access to key financial data for a specific stock symbol, including earnings report dates, EPS estimates, and revenue projections to help you stay on top of company performance. - -**Endpoint:** - -https://financialmodelingprep.com/stable/earnings?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-10-29", - "epsActual": null, - "epsEstimated": null, - "revenueActual": null, - "revenueEstimated": null, - "lastUpdated": "2025-02-04" - } -\] - -[Earnings Calendar API](https://site.financialmodelingprep.com/developer/docs/stable/earnings-calendar) - -Stay informed on upcoming and past earnings announcements with the FMP Earnings Calendar API. Access key data, including announcement dates, estimated earnings per share (EPS), and actual EPS for publicly traded companies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*earnings-calendar* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response - -\[ - { - "symbol": "KEC.NS", - "date": "2024-11-04", - "epsActual": 3.32, - "epsEstimated": 4.97, - "revenueActual": 51133100000, - "revenueEstimated": 44687400000, - "lastUpdated": "2024-12-08" - } -\] - -[IPOs Calendar API](https://site.financialmodelingprep.com/developer/docs/stable/ipos-calendar) - -Access a comprehensive list of all upcoming initial public offerings (IPOs) with the FMP IPO Calendar API. Stay up to date on the latest companies entering the public market, with essential details on IPO dates, company names, expected pricing, and exchange listings. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*ipos-calendar* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response -\[ - { - "symbol": "PEVC", - "date": "2025-02-03", - "daa": "2025-02-03T05:00:00.000Z", - "company": "Pacer Funds Trust", - "exchange": "NYSE", - "actions": "Expected", - "shares": null, - "priceRange": null, - "marketCap": null - } -\] - -[IPOs Disclosure API](https://site.financialmodelingprep.com/developer/docs/stable/ipos-disclosure) - -Access a comprehensive list of disclosure filings for upcoming initial public offerings (IPOs) with the FMP IPO Disclosures API. Stay updated on regulatory filings, including filing dates, effectiveness dates, CIK numbers, and form types, with direct links to official SEC documents. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*ipos-disclosure* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response -\[ - { - "symbol": "SCHM", - "filingDate": "2025-02-03", - "acceptedDate": "2025-02-03", - "effectivenessDate": "2025-02-03", - "cik": "0001454889", - "form": "CERT", - "url": "https://www.sec.gov/Archives/edgar/data/1454889/000114336225000044/SCCR020325.pdf" - } -\] - -[IPOs Prospectus API](https://site.financialmodelingprep.com/developer/docs/stable/ipos-prospectus) - -Access comprehensive information on IPO prospectuses with the FMP IPO Prospectus API. Get key financial details, such as public offering prices, discounts, commissions, proceeds before expenses, and more. This API also provides links to official SEC prospectuses, helping investors stay informed on companies entering the public market. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*ipos-prospectus* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response -\[ - { - "symbol": "ATAK", - "acceptedDate": "2025-02-03", - "filingDate": "2025-02-03", - "ipoDate": "2022-03-20", - "cik": "0001883788", - "pricePublicPerShare": 0.78, - "pricePublicTotal": 4649936.72, - "discountsAndCommissionsPerShare": 0.04, - "discountsAndCommissionsTotal": 254909.67, - "proceedsBeforeExpensesPerShare": 0.74, - "proceedsBeforeExpensesTotal": 4395207.05, - "form": "424B4", - "url": "https://www.sec.gov/Archives/edgar/data/1883788/000149315225004604/form424b4.htm" - } -\] - -[Stock Split Details API](https://site.financialmodelingprep.com/developer/docs/stable/splits-company) - -Access detailed information on stock splits for a specific company using the FMP Stock Split Details API. This API provides essential data, including the split date and the split ratio, helping users understand changes in a company's share structure after a stock split. - -**Endpoint:** - -https://financialmodelingprep.com/stable/splits?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2020-08-31", - "numerator": 4, - "denominator": 1 - } -\] - -[Stock Splits Calendar API](https://site.financialmodelingprep.com/developer/docs/stable/splits-calendar) - -Stay informed about upcoming stock splits with the FMP Stock Splits Calendar API. This API provides essential data on upcoming stock splits across multiple companies, including the split date and ratio, helping you track changes in share structures before they occur. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*splits-calendar* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-05-21 | -| to | date | 2025-08-21 | - -Response - -\[ - { - "symbol": "EYEN", - "date": "2025-02-03", - "numerator": 1, - "denominator": 80 - } -\] - -## **Chart** - -[Stock Chart Light API](https://site.financialmodelingprep.com/developer/docs/stable/historical-price-eod-light) - -Access simplified stock chart data using the FMP Basic Stock Chart API. This API provides essential charting information, including date, price, and trading volume, making it ideal for tracking stock performance with minimal data and creating basic price and volume charts. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/light?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "price": 232.8, - "volume": 44489128 - } -\] - -[Stock Price and Volume Data API](https://site.financialmodelingprep.com/developer/docs/stable/historical-price-eod-full) - -Access full price and volume data for any stock symbol using the FMP Comprehensive Stock Price and Volume Data API. Get detailed insights, including open, high, low, close prices, trading volume, price changes, percentage changes, and volume-weighted average price (VWAP). - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/full?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "open": 227.2, - "high": 233.13, - "low": 226.65, - "close": 232.8, - "volume": 44489128, - "change": 5.6, - "changePercent": 2.46479, - "vwap": 230.86 - } -\] - -[Unadjusted Stock Price API](https://site.financialmodelingprep.com/developer/docs/stable/historical-price-eod-non-split-adjusted) - -Access stock price and volume data without adjustments for stock splits with the FMP Unadjusted Stock Price Chart API. Get accurate insights into stock performance, including open, high, low, and close prices, along with trading volume, without split-related changes. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/non-split-adjusted?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "adjOpen": 227.2, - "adjHigh": 233.13, - "adjLow": 226.65, - "adjClose": 232.8, - "volume": 44489128 - } -\] - -[Dividend Adjusted Price Chart API](https://site.financialmodelingprep.com/developer/docs/stable/historical-price-eod-dividend-adjusted) - -Analyze stock performance with dividend adjustments using the FMP Dividend-Adjusted Price Chart API. Access end-of-day price and volume data that accounts for dividend payouts, offering a more comprehensive view of stock trends over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/dividend-adjusted?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "adjOpen": 227.2, - "adjHigh": 233.13, - "adjLow": 226.65, - "adjClose": 232.8, - "volume": 44489128 - } -\] - -[1 Min Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-1-min) - -Access precise intraday stock price and volume data with the FMP 1-Minute Interval Stock Chart API. Retrieve real-time or historical stock data in 1-minute intervals, including key information such as open, high, low, and close prices, and trading volume for each minute. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1min?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response -\[ - { - "date": "2025-02-04 15:59:00", - "open": 233.01, - "low": 232.72, - "high": 233.13, - "close": 232.79, - "volume": 720121 - } -\] - -[5 Min Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-5-min) - -Access stock price and volume data with the FMP 5-Minute Interval Stock Chart API. Retrieve detailed stock data in 5-minute intervals, including open, high, low, and close prices, along with trading volume for each 5-minute period. This API is perfect for short-term trading analysis and building intraday charts. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/5min?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response -\[ - { - "date": "2025-02-04 15:55:00", - "open": 232.87, - "low": 232.72, - "high": 233.13, - "close": 232.79, - "volume": 1555040 - } -\] - -[15 Min Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-15-min) - -Access stock price and volume data with the FMP 15-Minute Interval Stock Chart API. Retrieve detailed stock data in 15-minute intervals, including open, high, low, close prices, and trading volume. This API is ideal for creating intraday charts and analyzing medium-term price trends during the trading day. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/15min?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response - -\[ - { - "date": "2025-02-04 15:45:00", - "open": 232.25, - "low": 232.18, - "high": 233.13, - "close": 232.79, - "volume": 2535629 - } -\] - -[30 Min Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-30-min) - -Access stock price and volume data with the FMP 30-Minute Interval Stock Chart API. Retrieve essential stock data in 30-minute intervals, including open, high, low, close prices, and trading volume. This API is perfect for creating intraday charts and tracking medium-term price movements for more strategic trading decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/30min?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response -\[ - { - "date": "2025-02-04 15:30:00", - "open": 232.29, - "low": 232.01, - "high": 233.13, - "close": 232.79, - "volume": 3476320 - } -\] - -[1 Hour Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-1-hour) - -Track stock price movements over hourly intervals with the FMP 1-Hour Interval Stock Chart API. Access essential stock price and volume data, including open, high, low, and close prices for each hour, to analyze broader intraday trends with precision. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1hour?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response -\[ - { - "date": "2025-02-04 15:30:00", - "open": 232.29, - "low": 232.01, - "high": 233.13, - "close": 232.37, - "volume": 15079381 - } -\] - -[4 Hour Interval Stock Chart API](https://site.financialmodelingprep.com/developer/docs/stable/intraday-4-hour) - -Analyze stock price movements over extended intraday periods with the FMP 4-Hour Interval Stock Chart API. Access key stock price and volume data in 4-hour intervals, perfect for tracking longer intraday trends and understanding broader market movements. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/4hour?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | -| nonadjusted | boolean | false | - -Response -\[ - { - "date": "2025-02-04 12:30:00", - "open": 231.79, - "low": 231.37, - "high": 233.13, - "close": 232.37, - "volume": 23781913 - } -\] - -## **Company** - -[Company Profile Data API](https://site.financialmodelingprep.com/developer/docs/stable/profile-symbol) - -Access detailed company profile data with the FMP Company Profile Data API. This API provides key financial and operational information for a specific stock symbol, including the company's market capitalization, stock price, industry, and much more. - -**Endpoint:** - -https://financialmodelingprep.com/stable/profile?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "symbol": "AAPL", - "price": 232.8, - "marketCap": 3500823120000, - "beta": 1.24, - "lastDividend": 0.99, - "range": "164.08-260.1", - "change": 4.79, - "changePercentage": 2.1008, - "volume": 0, - "averageVolume": 50542058, - "companyName": "Apple Inc.", - "currency": "USD", - "cik": "0000320193", - "isin": "US0378331005", - "cusip": "037833100", - "exchangeFullName": "NASDAQ Global Select", - "exchange": "NASDAQ", - "industry": "Consumer Electronics", - "website": "https://www.apple.com", - "description": "Apple Inc. designs, manufactures, and markets smartphones, personal computers, tablets, wearables, and accessories worldwide. The company offers iPhone, a line of smartphones; Mac, a line of personal computers; iPad, a line of multi-purpose tablets; and wearables, home, and accessories comprising AirPods, Apple TV, Apple Watch, Beats products, and HomePod. It also provides AppleCare support and cloud services; and operates various platforms, including the App Store that allow customers to discov...", - "ceo": "Mr. Timothy D. Cook", - "sector": "Technology", - "country": "US", - "fullTimeEmployees": "164000", - "phone": "(408) 996-1010", - "address": "One Apple Park Way", - "city": "Cupertino", - "state": "CA", - "zip": "95014", - "image": "https://images.financialmodelingprep.com/symbol/AAPL.png", - "ipoDate": "1980-12-12", - "defaultImage": false, - "isEtf": false, - "isActivelyTrading": true, - "isAdr": false, - "isFund": false - } -\] - -[Company Notes API](https://site.financialmodelingprep.com/developer/docs/stable/company-notes) - -Retrieve detailed information about company-issued notes with the FMP Company Notes API. Access essential data such as CIK number, stock symbol, note title, and the exchange where the notes are listed. - -**Endpoint:** - -https://financialmodelingprep.com/stable/company-notes?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "cik": "0000320193", - "symbol": "AAPL", - "title": "1.000% Notes due 2022", - "exchange": "NASDAQ" - } -\] - -[Company Employee Count API](https://site.financialmodelingprep.com/developer/docs/stable/employee-count) - -Retrieve detailed workforce information for companies, including employee count, reporting period, and filing date. The FMP Company Employee Count API also provides direct links to official SEC documents for further verification and in-depth research. - -**Endpoint:** - -https://financialmodelingprep.com/stable/employee-count?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response - -\[ - { - "symbol": "AAPL", - "cik": "0000320193", - "acceptanceTime": "2024-11-01 06:01:36", - "periodOfReport": "2024-09-28", - "companyName": "Apple Inc.", - "formType": "10-K", - "filingDate": "2024-11-01", - "employeeCount": 164000, - "source": "https://www.sec.gov/Archives/edgar/data/320193/000032019324000123/0000320193-24-000123-index.htm" - } -\] -[Company Historical Employee Count API](https://site.financialmodelingprep.com/developer/docs/stable/historical-employee-count) - -Access historical employee count data for a company based on specific reporting periods. The FMP Company Historical Employee Count API provides insights into how a company’s workforce has evolved over time, allowing users to analyze growth trends and operational changes. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-employee-count?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | - -Response -\[ - { - "symbol": "AAPL", - "cik": "0000320193", - "acceptanceTime": "2024-11-01 06:01:36", - "periodOfReport": "2024-09-28", - "companyName": "Apple Inc.", - "formType": "10-K", - "filingDate": "2024-11-01", - "employeeCount": 164000, - "source": "https://www.sec.gov/Archives/edgar/data/320193/000032019324000123/0000320193-24-000123-index.htm" - } -\] - -[Company Market Cap API](https://site.financialmodelingprep.com/developer/docs/stable/market-cap) - -Retrieve the market capitalization for a specific company on any given date using the FMP Company Market Capitalization API. This API provides essential data to assess the size and value of a company in the stock market, helping users gauge its overall market standing. - -**Endpoint:** - -https://financialmodelingprep.com/stable/market-capitalization?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "marketCap": 3500823120000 - } -\] - -[Batch Market Cap API](https://site.financialmodelingprep.com/developer/docs/stable/batch-market-cap) - -Retrieve market capitalization data for multiple companies in a single request with the FMP Batch Market Capitalization API. This API allows users to compare the market size of various companies simultaneously, streamlining the analysis of company valuations. - -**Endpoint:** - -https://financialmodelingprep.com/stable/market-capitalization-batch?*symbols*\=AAPL,MSFT,GOOG - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbols\* | string | AAPL,MSFT,GOOG | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "marketCap": 3500823120000 - } -\] - -[Historical Market Cap API](https://site.financialmodelingprep.com/developer/docs/stable/historical-market-cap) - -Access historical market capitalization data for a company using the FMP Historical Market Capitalization API. This API helps track the changes in market value over time, enabling long-term assessments of a company's growth or decline. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-market-capitalization?*symbol*\=AAPL - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 100 | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-02-29", - "marketCap": 2784608472000 - } -\] - -[Company Share Float & Liquidity API](https://site.financialmodelingprep.com/developer/docs/stable/shares-float) - -Understand the liquidity and volatility of a stock with the FMP Company Share Float and Liquidity API. Access the total number of publicly traded shares for any company to make informed investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/shares-float?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2025-02-04 17:01:35", - "freeFloat": 99.9095, - "floatShares": 15024290700, - "outstandingShares": 15037900000 - } -\] - -[All Shares Float API](https://site.financialmodelingprep.com/developer/docs/stable/all-shares-float) - -Access comprehensive shares float data for all available companies with the FMP All Shares Float API. Retrieve critical information such as free float, float shares, and outstanding shares to analyze liquidity across a wide range of companies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/shares-float-all?*page*\=0&*limit*\=1000 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| limit | number | 1000 | -| page | number | 0 | - -Response -\[ - { - "symbol": "6898.HK", - "date": "2025-02-04 17:27:01", - "freeFloat": 33.2536, - "floatShares": 318128880, - "outstandingShares": 956675009 - } -\] - -[Latest Mergers & Acquisitions API](https://site.financialmodelingprep.com/developer/docs/stable/latest-mergers-acquisitions) - -Access real-time data on the latest mergers and acquisitions with the FMP Latest Mergers and Acquisitions API. This API provides key information such as the transaction date, company names, and links to detailed filing information for further analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/mergers-acquisitions-latest?*page*\=0&*limit*\=100 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 100 | - -Response - -\[ - { - "symbol": "NLOK", - "companyName": "NortonLifeLock Inc.", - "cik": "0000849399", - "targetedCompanyName": "MoneyLion Inc.", - "targetedCik": "0001807846", - "targetedSymbol": "ML", - "transactionDate": "2025-02-03", - "acceptedDate": "2025-02-03 06:01:10", - "link": "https://www.sec.gov/Archives/edgar/data/849399/000114036125002752/ny20039778x6\_s4.htm" - } -\] - -[Search Mergers & Acquisitions API](https://site.financialmodelingprep.com/developer/docs/stable/search-mergers-acquisitions) - -Search for specific mergers and acquisitions data with the FMP Search Mergers and Acquisitions API. Retrieve detailed information on M\&A activity, including acquiring and targeted companies, transaction dates, and links to official SEC filings. - -**Endpoint:** - -https://financialmodelingprep.com/stable/mergers-acquisitions-search?*name*\=Apple - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| name\* | string | Apple | - -Response -\[ - { - "symbol": "PEGY", - "companyName": "Pineapple Energy Inc.", - "cik": "0000022701", - "targetedCompanyName": "Communications Systems, Inc.", - "targetedCik": "0000022701", - "targetedSymbol": "JCS", - "transactionDate": "2021-11-12", - "acceptedDate": "2021-11-12 09:54:22", - "link": "https://www.sec.gov/Archives/edgar/data/22701/000089710121000932/a211292\_s-4.htm" - } -\] - -[Company Executives API](https://site.financialmodelingprep.com/developer/docs/stable/company-executives) - -Retrieve detailed information on company executives with the FMP Company Executives API. This API provides essential data about key executives, including their name, title, compensation, and other demographic details such as gender and year of birth. - -**Endpoint:** - -https://financialmodelingprep.com/stable/key-executives?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| active | string | true | - -Response - -\[ - { - "title": "Vice President of Worldwide Sales", - "name": "Mr. Michael Fenger", - "pay": null, - "currencyPay": "USD", - "gender": "male", - "yearBorn": null, - "active": null - } -\] - -[Executive Compensation API](https://site.financialmodelingprep.com/developer/docs/stable/executive-compensation) - -Retrieve comprehensive compensation data for company executives with the FMP Executive Compensation API. This API provides detailed information on salaries, stock awards, total compensation, and other relevant financial data, including filing details and links to official documents. - -**Endpoint:** - -https://financialmodelingprep.com/stable/governance-executive-compensation?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "cik": "0000320193", - "symbol": "AAPL", - "companyName": "Apple Inc.", - "filingDate": "2025-01-10", - "acceptedDate": "2025-01-10 16:31:18", - "nameAndPosition": "Kate Adams Senior Vice President, General Counsel and Secretary", - "year": 2023, - "salary": 1000000, - "bonus": 0, - "stockAward": 22323641, - "optionAward": 0, - "incentivePlanCompensation": 3571150, - "allOtherCompensation": 46914, - "total": 26941705, - "link": "https://www.sec.gov/Archives/edgar/data/320193/000130817925000008/0001308179-25-000008-index.htm" - } -\] - -[Executive Compensation Benchmark API](https://site.financialmodelingprep.com/developer/docs/stable/executive-compensation-benchmark) - -Gain access to average executive compensation data across various industries with the FMP Executive Compensation Benchmark API. This API provides essential insights for comparing executive pay by industry, helping you understand compensation trends and benchmarks. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*executive-compensation-benchmark* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| year | string | 2024 | - -Response -\[ - { - "industryTitle": "ABRASIVE, ASBESTOS & MISC NONMETALLIC MINERAL PRODS", - "year": 2023, - "averageCompensation": 694313.1666666666 - } -\] - -## **Commitment Of Traders** - -[COT Report API](https://site.financialmodelingprep.com/developer/docs/stable/cot-report) - -Access comprehensive Commitment of Traders (COT) reports with the FMP COT Report API. This API provides detailed information about long and short positions across various sectors, helping you assess market sentiment and track positions in commodities, indices, and financial instruments. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*commitment-of-traders-report* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | - -Response - -\[ - { - "symbol": "KC", - "date": "2024-02-27 00:00:00", - "name": "Coffee (KC)", - "sector": "SOFTS", - "marketAndExchangeNames": "COFFEE C \- ICE FUTURES U.S.", - "cftcContractMarketCode": "083731", - "cftcMarketCode": "ICUS", - "cftcRegionCode": "1", - "cftcCommodityCode": "83", - "openInterestAll": 209453, - "noncommPositionsLongAll": 75330, - "noncommPositionsShortAll": 23630, - "noncommPositionsSpreadAll": 47072, - "commPositionsLongAll": 79690, - "commPositionsShortAll": 132114, - "totReptPositionsLongAll": 202092, - "totReptPositionsShortAll": 202816, - "nonreptPositionsLongAll": 7361, - "nonreptPositionsShortAll": 6637, - "openInterestOld": 179986, - "noncommPositionsLongOld": 75483, - "noncommPositionsShortOld": 35395, - "noncommPositionsSpreadOld": 27067, - "commPositionsLongOld": 70693, - "commPositionsShortOld": 111666, - "totReptPositionsLongOld": 173243, - "totReptPositionsShortOld": 174128, - "nonreptPositionsLongOld": 6743, - "nonreptPositionsShortOld": 5858, - "openInterestOther": 29467, - "noncommPositionsLongOther": 18754, - "noncommPositionsShortOther": 7142, - "noncommPositionsSpreadOther": 1098, - "commPositionsLongOther": 8997, - "commPositionsShortOther": 20448, - "totReptPositionsLongOther": 28849, - "totReptPositionsShortOther": 28688, - "nonreptPositionsLongOther": 618, - "nonreptPositionsShortOther": 779, - "changeInOpenInterestAll": 2957, - "changeInNoncommLongAll": \-3545, - "changeInNoncommShortAll": 618, - "changeInNoncommSpeadAll": 1575, - "changeInCommLongAll": 4978, - "changeInCommShortAll": 802, - "changeInTotReptLongAll": 3008, - "changeInTotReptShortAll": 2995, - "changeInNonreptLongAll": \-51, - "changeInNonreptShortAll": \-38, - "pctOfOpenInterestAll": 100, - "pctOfOiNoncommLongAll": 36, - "pctOfOiNoncommShortAll": 11.3, - "pctOfOiNoncommSpreadAll": 22.5, - "pctOfOiCommLongAll": 38, - "pctOfOiCommShortAll": 63.1, - "pctOfOiTotReptLongAll": 96.5, - "pctOfOiTotReptShortAll": 96.8, - "pctOfOiNonreptLongAll": 3.5, - "pctOfOiNonreptShortAll": 3.2, - "pctOfOpenInterestOl": 100, - "pctOfOiNoncommLongOl": 41.9, - "pctOfOiNoncommShortOl": 19.7, - "pctOfOiNoncommSpreadOl": 15, - "pctOfOiCommLongOl": 39.3, - "pctOfOiCommShortOl": 62, - "pctOfOiTotReptLongOl": 96.3, - "pctOfOiTotReptShortOl": 96.7, - "pctOfOiNonreptLongOl": 3.7, - "pctOfOiNonreptShortOl": 3.3, - "pctOfOpenInterestOther": 100, - "pctOfOiNoncommLongOther": 63.6, - "pctOfOiNoncommShortOther": 24.2, - "pctOfOiNoncommSpreadOther": 3.7, - "pctOfOiCommLongOther": 30.5, - "pctOfOiCommShortOther": 69.4, - "pctOfOiTotReptLongOther": 97.9, - "pctOfOiTotReptShortOther": 97.4, - "pctOfOiNonreptLongOther": 2.1, - "pctOfOiNonreptShortOther": 2.6, - "tradersTotAll": 357, - "tradersNoncommLongAll": 132, - "tradersNoncommShortAll": 77, - "tradersNoncommSpreadAll": 94, - "tradersCommLongAll": 106, - "tradersCommShortAll": 119, - "tradersTotReptLongAll": 286, - "tradersTotReptShortAll": 250, - "tradersTotOl": 351, - "tradersNoncommLongOl": 136, - "tradersNoncommShortOl": 72, - "tradersNoncommSpeadOl": 88, - "tradersCommLongOl": 94, - "tradersCommShortOl": 114, - "tradersTotReptLongOl": 269, - "tradersTotReptShortOl": 239, - "tradersTotOther": 164, - "tradersNoncommLongOther": 31, - "tradersNoncommShortOther": 34, - "tradersNoncommSpreadOther": 16, - "tradersCommLongOther": 59, - "tradersCommShortOther": 68, - "tradersTotReptLongOther": 102, - "tradersTotReptShortOther": 106, - "concGrossLe4TdrLongAll": 16, - "concGrossLe4TdrShortAll": 23.7, - "concGrossLe8TdrLongAll": 25.8, - "concGrossLe8TdrShortAll": 38.9, - "concNetLe4TdrLongAll": 9.8, - "concNetLe4TdrShortAll": 16.2, - "concNetLe8TdrLongAll": 17.7, - "concNetLe8TdrShortAll": 25.4, - "concGrossLe4TdrLongOl": 13.6, - "concGrossLe4TdrShortOl": 24.7, - "concGrossLe8TdrLongOl": 23.2, - "concGrossLe8TdrShortOl": 40.3, - "concNetLe4TdrLongOl": 11.3, - "concNetLe4TdrShortOl": 18.2, - "concNetLe8TdrLongOl": 20.3, - "concNetLe8TdrShortOl": 31.9, - "concGrossLe4TdrLongOther": 68.2, - "concGrossLe4TdrShortOther": 29.1, - "concGrossLe8TdrLongOther": 77.8, - "concGrossLe8TdrShortOther": 47.3, - "concNetLe4TdrLongOther": 64.7, - "concNetLe4TdrShortOther": 26.7, - "concNetLe8TdrLongOther": 73.9, - "concNetLe8TdrShortOther": 44.2, - "contractUnits": "(CONTRACTS OF 37,500 POUNDS)" - } -\] - -[COT Analysis By Dates API](https://site.financialmodelingprep.com/developer/docs/stable/cot-report-analysis) - -Gain in-depth insights into market sentiment with the FMP COT Report Analysis API. Analyze the Commitment of Traders (COT) reports for a specific date range to evaluate market dynamics, sentiment, and potential reversals across various sectors. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*commitment-of-traders-analysis* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol | string | AAPL | -| from | date | 2024-01-01 | -| to | date | 2024-03-01 | - -Response -\[ - { - "symbol": "B6", - "date": "2024-02-27 00:00:00", - "name": "British Pound (B6)", - "sector": "CURRENCIES", - "exchange": "BRITISH POUND \- CHICAGO MERCANTILE EXCHANGE", - "currentLongMarketSituation": 66.85, - "currentShortMarketSituation": 33.15, - "marketSituation": "Bullish", - "previousLongMarketSituation": 67.97, - "previousShortMarketSituation": 32.03, - "previousMarketSituation": "Bullish", - "netPostion": 46358, - "previousNetPosition": 46312, - "changeInNetPosition": 0.1, - "marketSentiment": "Increasing Bullish", - "reversalTrend": false - } -\] - -[COT Report List API](https://site.financialmodelingprep.com/developer/docs/stable/cot-report-list) - -Access a comprehensive list of available Commitment of Traders (COT) reports by commodity or futures contract using the FMP COT Report List API. This API provides an overview of different market segments, allowing users to retrieve and explore COT reports for a wide variety of commodities and financial instruments. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*commitment-of-traders-list* - -Response -\[ - { - "symbol": "NG", - "name": "Natural Gas (NG)" - } -\] - -## **Discounted Cash Flow** - -[DCF Valuation API](https://site.financialmodelingprep.com/developer/docs/stable/dcf-advanced) - -Estimate the intrinsic value of a company with the FMP Discounted Cash Flow Valuation API. Calculate the DCF valuation based on expected future cash flows and discount rates. - -**Endpoint:** - -https://financialmodelingprep.com/stable/discounted-cash-flow?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "dcf": 147.2669883190846, - "Stock Price": 231.795 - } -\] - -[Levered DCF API](https://site.financialmodelingprep.com/developer/docs/stable/dcf-levered) - -Analyze a company’s value with the FMP Levered Discounted Cash Flow (DCF) API, which incorporates the impact of debt. This API provides post-debt company valuation, offering investors a more accurate measure of a company's true worth by accounting for its debt obligations. - -**Endpoint:** - -https://financialmodelingprep.com/stable/levered-discounted-cash-flow?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "symbol": "AAPL", - "date": "2025-02-04", - "dcf": 147.2669883190846, - "Stock Price": 231.795 - } -\] - -[Custom DCF Advanced API](https://site.financialmodelingprep.com/developer/docs/stable/custom-dcf-advanced) - -Run a tailored Discounted Cash Flow (DCF) analysis using the FMP Custom DCF Advanced API. With detailed inputs, this API allows users to fine-tune their assumptions and variables, offering a more personalized and precise valuation for a company. - -**Endpoint:** - -https://financialmodelingprep.com/stable/custom-discounted-cash-flow?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| revenueGrowthPct | number | 0.1094119804597946 | -| ebitdaPct | number | 0.31273548388 | -| depreciationAndAmortizationPct | number | 0.0345531631720999 | -| cashAndShortTermInvestmentsPct | number | 0.2344222126801843 | -| receivablesPct | number | 0.1533770531229388 | -| inventoriesPct | number | 0.0155245674227653 | -| payablePct | number | 0.1614868903169657 | -| ebitPct | number | 0.2781823207138459 | -| capitalExpenditurePct | number | 0.0306025847141713 | -| operatingCashFlowPct | number | 0.2886333485760204 | -| sellingGeneralAndAdministrativeExpensesPct | number | 0.0662854095187211 | -| taxRate | number | 0.14919579658453103 | -| longTermGrowthRate | number | 4 | -| costOfDebt | number | 3.64 | -| costOfEquity | number | 9.51168 | -| marketRiskPremium | number | 4.72 | -| beta | number | 1.244 | -| riskFreeRate | number | 3.64 | - -Response -\[ - { - "year": "2029", - "symbol": "AAPL", - "revenue": 657173266965, - "revenuePercentage": 10.94, - "ebitda": 205521399637, - "ebitdaPercentage": 31.27, - "ebit": 182813984515, - "ebitPercentage": 27.82, - "depreciation": 22707415125, - "depreciationPercentage": 3.46, - "totalCash": 154056011356, - "totalCashPercentage": 23.44, - "receivables": 100795299078, - "receivablesPercentage": 15.34, - "inventories": 10202330691, - "inventoriesPercentage": 1.55, - "payable": 106124867281, - "payablePercentage": 16.15, - "capitalExpenditure": 20111200574, - "capitalExpenditurePercentage": 3.06, - "price": 232.8, - "beta": 1.244, - "dilutedSharesOutstanding": 15408095000, - "costofDebt": 3.64, - "taxRate": 24.09, - "afterTaxCostOfDebt": 2.76, - "riskFreeRate": 3.64, - "marketRiskPremium": 4.72, - "costOfEquity": 9.51, - "totalDebt": 106629000000, - "totalEquity": 3587004516000, - "totalCapital": 3693633516000, - "debtWeighting": 2.89, - "equityWeighting": 97.11, - "wacc": 9.33, - "taxRateCash": 14919580, - "ebiat": 155538906468, - "ufcf": 197876962552, - "sumPvUfcf": 616840860880, - "longTermGrowthRate": 4, - "terminalValue": 3863553224578, - "presentTerminalValue": 2473772391290, - "enterpriseValue": 3090613252170, - "netDebt": 76686000000, - "equityValue": 3013927252170, - "equityValuePerShare": 195.61, - "freeCashFlowT1": 205792041054 - } -\] - -[Custom DCF Levered API](https://site.financialmodelingprep.com/developer/docs/stable/custom-dcf-levered) - -Run a tailored Discounted Cash Flow (DCF) analysis using the FMP Custom DCF Advanced API. With detailed inputs, this API allows users to fine-tune their assumptions and variables, offering a more personalized and precise valuation for a company. - -**Endpoint:** - -https://financialmodelingprep.com/stable/custom-levered-discounted-cash-flow?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| revenueGrowthPct | number | 0.1094119804597946 | -| ebitdaPct | number | 0.31273548388 | -| depreciationAndAmortizationPct | number | 0.0345531631720999 | -| cashAndShortTermInvestmentsPct | number | 0.2344222126801843 | -| receivablesPct | number | 0.1533770531229388 | -| inventoriesPct | number | 0.0155245674227653 | -| payablePct | number | 0.1614868903169657 | -| ebitPct | number | 0.2781823207138459 | -| capitalExpenditurePct | number | 0.0306025847141713 | -| operatingCashFlowPct | number | 0.2886333485760204 | -| sellingGeneralAndAdministrativeExpensesPct | number | 0.0662854095187211 | -| taxRate | number | 0.14919579658453103 | -| longTermGrowthRate | number | 4 | -| costOfDebt | number | 3.64 | -| costOfEquity | number | 9.51168 | -| marketRiskPremium | number | 4.72 | -| beta | number | 1.244 | -| riskFreeRate | number | 3.64 | - -Response -\[ - { - "year": "2029", - "symbol": "AAPL", - "revenue": 657173266965, - "revenuePercentage": 10.94, - "capitalExpenditure": 20111200574, - "capitalExpenditurePercentage": 3.06, - "price": 232.8, - "beta": 1.244, - "dilutedSharesOutstanding": 15408095000, - "costofDebt": 3.64, - "taxRate": 24.09, - "afterTaxCostOfDebt": 2.76, - "riskFreeRate": 3.64, - "marketRiskPremium": 4.72, - "costOfEquity": 9.51, - "totalDebt": 106629000000, - "totalEquity": 3587004516000, - "totalCapital": 3693633516000, - "debtWeighting": 2.89, - "equityWeighting": 97.11, - "wacc": 9.33, - "operatingCashFlow": 189682120638, - "pvLfcf": 134327365439, - "sumPvLfcf": 652368547936, - "longTermGrowthRate": 4, - "freeCashFlow": 209793321212, - "terminalValue": 4096220460472, - "presentTerminalValue": 2622745564702, - "enterpriseValue": 3275114112638, - "netDebt": 76686000000, - "equityValue": 3198428112638, - "equityValuePerShare": 207.58, - "freeCashFlowT1": 218185054060, - "operatingCashFlowPercentage": 28.86 - } -\] - -## **Economics** - -[Treasury Rates API](https://site.financialmodelingprep.com/developer/docs/stable/treasury-rates) - -Access real-time and historical Treasury rates for all maturities with the FMP Treasury Rates API. Track key benchmarks for interest rates across the economy. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*treasury-rates* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "date": "2024-02-29", - "month1": 5.53, - "month2": 5.5, - "month3": 5.45, - "month6": 5.3, - "year1": 5.01, - "year2": 4.64, - "year3": 4.43, - "year5": 4.26, - "year7": 4.28, - "year10": 4.25, - "year20": 4.51, - "year30": 4.38 - } -\] - -[Economics Indicators API](https://site.financialmodelingprep.com/developer/docs/stable/economics-indicators) - -Access real-time and historical economic data for key indicators like GDP, unemployment, and inflation with the FMP Economic Indicators API. Use this data to measure economic performance and identify growth trends. - -**Endpoint:** - -https://financialmodelingprep.com/stable/economic-indicators?*name*\=GDP - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| name\* | string | GDP,realGDP,nominalPotentialGDP,realGDPPerCapita,federalFunds,CPI,inflationRate,inflation,retailSales,consumerSentiment,durableGoods,unemploymentRate,totalNonfarmPayroll,initialClaims,industrialProductionTotalIndex,newPrivatelyOwnedHousingUnitsStartedTotalUnits,totalVehicleSales,retailMoneyFunds,smoothedUSRecessionProbabilities,3MonthOr90DayRatesAndYieldsCertificatesOfDeposit,commercialBankInterestRateOnCreditCardPlansAllAccounts,30YearFixedRateMortgageAverage,15YearFixedRateMortgageAverage | -| from | date | 2024-07-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "name": "GDP", - "date": "2024-01-01", - "value": 28624.069 - } -\] - -[Economic Data Releases Calendar API](https://site.financialmodelingprep.com/developer/docs/stable/economics-calendar) - -Stay informed with the FMP Economic Data Releases Calendar API. Access a comprehensive calendar of upcoming economic data releases to prepare for market impacts and make informed investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*economic-calendar* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response -\[ - { - "date": "2024-03-01 03:35:00", - "country": "JP", - "event": "3-Month Bill Auction", - "currency": "JPY", - "previous": \-0.112, - "estimate": null, - "actual": \-0.096, - "change": 0.016, - "impact": "Low", - "changePercentage": 14.286 - } -\] - -[Market Risk Premium API](https://site.financialmodelingprep.com/developer/docs/stable/market-risk-premium) - -Access the market risk premium for specific dates with the FMP Market Risk Premium API. Use this key financial metric to assess the additional return expected from investing in the stock market over a risk-free investment. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*market-risk-premium* - -Response -\[ - { - "country": "Zimbabwe", - "continent": "Africa", - "countryRiskPremium": 13.17, - "totalEquityRiskPremium": 17.77 - } -\] - -## **Etf And Mutual Funds** - -[ETF & Fund Holdings API](https://site.financialmodelingprep.com/developer/docs/stable/holdings) - -Get a detailed breakdown of the assets held within ETFs and mutual funds using the FMP ETF & Fund Holdings API. Access real-time data on the specific securities and their weights in the portfolio, providing insights into asset composition and fund strategies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/etf/holdings?*symbol*\=SPY - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -Response -\[ - { - "symbol": "SPY", - "asset": "AAPL", - "name": "APPLE INC", - "isin": "US0378331005", - "securityCusip": "037833100", - "sharesNumber": 188106081, - "weightPercentage": 7.137, - "marketValue": 44744793487.47, - "updatedAt": "2025-01-16 05:01:09", - "updated": "2025-02-04 19:02:31" - } -\] - -[ETF & Mutual Fund Information API](https://site.financialmodelingprep.com/developer/docs/stable/information) - -Access comprehensive data on ETFs and mutual funds with the FMP ETF & Mutual Fund Information API. Retrieve essential details such as ticker symbol, fund name, expense ratio, assets under management, and more. - -**Endpoint:** - -https://financialmodelingprep.com/stable/etf/info?*symbol*\=SPY - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -Response -\[ - { - "symbol": "SPY", - "name": "SPDR S\&P 500 ETF Trust", - "description": "The Trust seeks to achieve its investment objective by holding a portfolio of the common stocks that are included in the index (the “Portfolio”), with the weight of each stock in the Portfolio substantially corresponding to the weight of such stock in the index.", - "isin": "US78462F1030", - "assetClass": "Equity", - "securityCusip": "78462F103", - "domicile": "US", - "website": "https://www.ssga.com/us/en/institutional/etfs/spdr-sp-500-etf-trust-spy", - "etfCompany": "SPDR", - "expenseRatio": 0.0945, - "assetsUnderManagement": 633120180000, - "avgVolume": 46396400, - "inceptionDate": "1993-01-22", - "nav": 603.64, - "navCurrency": "USD", - "holdingsCount": 503, - "updatedAt": "2024-12-03T20:32:48.873Z", - "sectorsList": \[ - { - "industry": "Basic Materials", - "exposure": 1.97 - }, - { - "industry": "Communication Services", - "exposure": 8.87 - }, - { - "industry": "Consumer Cyclical", - "exposure": 9.84 - } - \] - } -\] - -[ETF & Fund Country Allocation API](https://site.financialmodelingprep.com/developer/docs/stable/country-weighting) - -Gain insight into how ETFs and mutual funds distribute assets across different countries with the FMP ETF & Fund Country Allocation API. This tool provides detailed information on the percentage of assets allocated to various regions, helping you make informed investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/etf/country-weightings?*symbol*\=SPY - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -Response -\[ - { - "country": "United States", - "weightPercentage": "97.29%" - } -\] - -[ETF Asset Exposure API](https://site.financialmodelingprep.com/developer/docs/stable/etf-asset-exposure) - -Discover which ETFs hold specific stocks with the FMP ETF Asset Exposure API. Access detailed information on market value, share numbers, and weight percentages for assets within ETFs. - -**Endpoint:** - -https://financialmodelingprep.com/stable/etf/asset-exposure?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -Response -\[ - { - "symbol": "ZECP", - "asset": "AAPL", - "sharesNumber": 5482, - "weightPercentage": 5.86, - "marketValue": 0 - } -\] - -[ETF Sector Weighting API](https://site.financialmodelingprep.com/developer/docs/stable/sector-weighting) - -The FMP ETF Sector Weighting API provides a breakdown of the percentage of an ETF's assets that are invested in each sector. For example, an investor may want to invest in an ETF that has a high exposure to the technology sector if they believe that the technology sector is poised for growth. - -**Endpoint:** - -https://financialmodelingprep.com/stable/etf/sector-weightings?*symbol*\=SPY - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -\[ - { - "symbol": "SPY", - "sector": "Basic Materials", - "weightPercentage": 1.97 - } -\] - -[Mutual Fund & ETF Disclosure API](https://site.financialmodelingprep.com/developer/docs/stable/latest-disclosures) - -Access the latest disclosures from mutual funds and ETFs with the FMP Mutual Fund & ETF Disclosure API. This API provides updates on filings, changes in holdings, and other critical disclosure data for mutual funds and ETFs. - -**Endpoint:** - -https://financialmodelingprep.com/stable/funds/disclosure-holders-latest?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | SPY | - -Response -\[ - { - "cik": "0000106444", - "holder": "VANGUARD FIXED INCOME SECURITIES FUNDS", - "shares": 67030000, - "dateReported": "2024-07-31", - "change": 0, - "weightPercent": 0.03840197 - } -\] - -[Mutual Fund Disclosures API](https://site.financialmodelingprep.com/developer/docs/stable/mutual-fund-disclosures) - -Access comprehensive disclosure data for mutual funds with the FMP Mutual Fund Disclosures API. Analyze recent filings, balance sheets, and financial reports to gain insights into mutual fund portfolios. - -**Endpoint:** - -https://financialmodelingprep.com/stable/funds/disclosure?*symbol*\=VWO&*year*\=2023&*quarter*\=4 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | VWO | -| year\* | string | 2023 | -| quarter\* | string | 4 | -| cik | string | 0000857489 | - -Response -\[ - { - "cik": "0000857489", - "date": "2023-10-31", - "acceptedDate": "2023-12-28 09:26:13", - "symbol": "000089.SZ", - "name": "Shenzhen Airport Co Ltd", - "lei": "3003009W045RIKRBZI44", - "title": "SHENZ AIRPORT-A", - "cusip": "N/A", - "isin": "CNE000000VK1", - "balance": 2438784, - "units": "NS", - "cur\_cd": "CNY", - "valUsd": 2255873.6, - "pctVal": 0.0023838966190458215, - "payoffProfile": "Long", - "assetCat": "EC", - "issuerCat": "CORP", - "invCountry": "CN", - "isRestrictedSec": "N", - "fairValLevel": "2", - "isCashCollateral": "N", - "isNonCashCollateral": "N", - "isLoanByFund": "N" - } -\] - -## **Commodity** - -[Commodities List API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-list) - -Access an extensive list of tracked commodities across various sectors, including energy, metals, and agricultural products. The FMP Commodities List API provides essential data on tradable commodities, giving investors the ability to explore market options in real-time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*commodities-list* - -Response -\[ - { - "symbol": "HEUSX", - "name": "Lean Hogs Futures", - "exchange": null, - "tradeMonth": "Dec", - "currency": "USX" - } -\] - -[Commodities Quote API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-quote) - -Access real-time price quotes for all commodities traded worldwide with the FMP Global Commodities Quotes API. Track market movements and identify investment opportunities with comprehensive price data. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | - -Response - -\[ - { - "symbol": "GCUSD", - "name": "Gold Futures", - "price": 3375.3, - "changePercentage": \-0.65635, - "change": \-22.3, - "volume": 170936, - "dayLow": 3355.2, - "dayHigh": 3401.1, - "yearHigh": 3509.9, - "yearLow": 2354.6, - "marketCap": null, - "priceAvg50": 3358.706, - "priceAvg200": 3054.501, - "exchange": "COMMODITY", - "open": 3398.6, - "previousClose": 3397.6, - "timestamp": 1753372205 - } -\] - -[Commodities Quote Short API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-quote-short) - -Get fast and accurate quotes for commodities with the FMP Commodities Quick Quote API. Instantly access the current price, recent changes, and trading volume for various commodities in real-time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote-short?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | - -Response -\[ - { - "symbol": "GCUSD", - "price": 3375.3, - "change": \-22.3, - "volume": 170936 - } -\] - -[Light Chart API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-historical-price-eod-light) - -Access historical end-of-day prices for various commodities with the FMP Historical Commodities Price API. Analyze past price movements, trading volume, and trends to support informed decision-making. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/light?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "symbol": "GCUSD", - "date": "2025-07-24", - "price": 3373.8, - "volume": 174758 - } -\] - -[Full Chart API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-historical-price-eod-full) - -Access full historical end-of-day price data for commodities with the FMP Comprehensive Commodities Price API. This API enables users to analyze long-term price trends, patterns, and market movements in great detail. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/full?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "symbol": "GCUSD", - "date": "2025-07-24", - "open": 3398.6, - "high": 3401.1, - "low": 3355.2, - "close": 3373.8, - "volume": 174758, - "change": \-24.8, - "changePercent": \-0.72971223, - "vwap": 3376.7 - } -\] - -[1-Minute Interval Commodities Chart API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-intraday-1-min) - -Track real-time, short-term price movements for commodities with the FMP 1-Minute Interval Commodities Chart API. This API provides detailed 1-minute interval data, enabling precise monitoring of intraday market changes. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1min?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:18:00", - "open": 3374.5, - "low": 3373.7, - "high": 3374.5, - "close": 3374, - "volume": 123 - } -\] - -[5-Minute Interval Commodities Chart API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-intraday-5-min) - -Monitor short-term price movements with the FMP 5-Minute Interval Commodities Chart API. This API provides detailed 5-minute interval data, enabling users to track near-term price trends for more strategic trading and investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/5min?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:15:00", - "open": 3374, - "low": 3374, - "high": 3374.8, - "close": 3374.4, - "volume": 193 - } -\] - -[1-Hour Interval Commodities Chart API](https://site.financialmodelingprep.com/developer/docs/stable/commodities-intraday-1-hour) - -Monitor hourly price movements and trends with the FMP 1-Hour Interval Commodities Chart API. This API provides hourly data, offering a detailed look at price fluctuations throughout the trading day to support mid-term trading strategies and market analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1hour?*symbol*\=GCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | GCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "date": "2025-07-24 11:30:00", - "open": 3378.4, - "low": 3373.1, - "high": 3378.8, - "close": 3374.4, - "volume": 7108 - } -\] - -## **Crypto** - -[Cryptocurrency List API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-list) - -Access a comprehensive list of all cryptocurrencies traded on exchanges worldwide with the FMP Cryptocurrencies Overview API. Get detailed information on each cryptocurrency to inform your investment strategies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*cryptocurrency-list* - -#### - -Response -\[ - { - "symbol": "ALIENUSD", - "name": "Alien Inu USD", - "exchange": "CCC", - "icoDate": "2021-11-22", - "circulatingSupply": 0, - "totalSupply": null - } -\] - -[Full Cryptocurrency Quote API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-quote) - -Access real-time quotes for all cryptocurrencies with the FMP Full Cryptocurrency Quote API. Obtain comprehensive price data including current, high, low, and open prices. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | - -Response -\[ - { - "symbol": "BTCUSD", - "name": "Bitcoin USD", - "price": 118741.16, - "changePercentage": \-0.03193323, - "change": \-37.93, - "volume": 75302985728, - "dayLow": 117435.22, - "dayHigh": 119535.45, - "yearHigh": 123091.61, - "yearLow": 49121.24, - "marketCap": 2344693699320, - "priceAvg50": 109824.32, - "priceAvg200": 98161.086, - "exchange": "CRYPTO", - "open": 118779.09, - "previousClose": 118779.09, - "timestamp": 1753374602 - } -\] - -[Cryptocurrency Quote Short API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-quote-short) - -Access real-time cryptocurrency quotes with the FMP Cryptocurrency Quick Quote API. Get a concise overview of current crypto prices, changes, and trading volume for a wide range of digital assets. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote-short?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | - -Response -\[ - { - "symbol": "BTCUSD", - "price": 118741.16, - "change": \-37.93, - "volume": 75302985728 - } -\] - -[All Cryptocurrencies Quotes API](https://site.financialmodelingprep.com/developer/docs/stable/all-cryptocurrency-quotes) - -Access live price data for a wide range of cryptocurrencies with the FMP Real-Time Cryptocurrency Batch Quotes API. Get real-time updates on prices, market changes, and trading volumes for digital assets in a single request. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*batch-crypto-quotes* - -Parameter - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| short | boolean | true | - -Response -\[ - { - "symbol": "00USD", - "price": 0.01755108, - "change": 0.00035108, - "volume": 3719492.41 - } -\] - -[Historical Cryptocurrency Light Chart API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-historical-price-eod-light) - -Access historical end-of-day prices for a variety of cryptocurrencies with the Historical Cryptocurrency Price Snapshot API. Track trends in price and trading volume over time to better understand market behavior. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/light?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "symbol": "BTCUSD", - "date": "2025-07-24", - "price": 118741.16, - "volume": 75302985728 - } -\] - -[Historical Cryptocurrency Full Chart API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-historical-price-eod-full) - -Access comprehensive end-of-day (EOD) price data for cryptocurrencies with the Full Historical Cryptocurrency Data API. Analyze long-term price trends, market movements, and trading volumes to inform strategic decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/full?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "symbol": "BTCUSD", - "date": "2025-07-24", - "open": 118779.09, - "high": 119535.45, - "low": 117435.22, - "close": 118741.16, - "volume": 75302985728, - "change": \-37.93, - "changePercent": \-0.03193323, - "vwap": 118570.61 - } -\] - -[1-Minute Interval Cryptocurrency Data API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-intraday-1-min) - -Get real-time, 1-minute interval price data for cryptocurrencies with the 1-Minute Cryptocurrency Intraday Data API. Monitor short-term price fluctuations and trading volume to stay updated on market movements. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1min?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:29:00", - "open": 118797.96, - "low": 118760.42, - "high": 118818.11, - "close": 118784.04, - "volume": 52293740.08888889 - } -\] - -[5-Minute Interval Cryptocurrency Data API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-intraday-5-min) - -Analyze short-term price trends with the 5-Minute Interval Cryptocurrency Data API. Access real-time, intraday price data for cryptocurrencies to monitor rapid market movements and optimize trading strategies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/5min?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "date": "2025-07-24 12:25:00", - "open": 118988.32, - "low": 118797.03, - "high": 118997.22, - "close": 118797.03, - "volume": 208601161.95555556 - } -\] - -[1-Hour Interval Cryptocurrency Data API](https://site.financialmodelingprep.com/developer/docs/stable/cryptocurrency-intraday-1-hour) - -Access detailed 1-hour intraday price data for cryptocurrencies with the 1-Hour Interval Cryptocurrency Data API. Track hourly price movements to gain insights into market trends and make informed trading decisions throughout the day. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1hour?*symbol*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | BTCUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:00:00", - "open": 119189.36, - "low": 118768.68, - "high": 119272.88, - "close": 118797.03, - "volume": 1493617925.6888888 - } -\] - -## **Forex** - -[Forex Currency Pairs API](https://site.financialmodelingprep.com/developer/docs/stable/forex-list) - -Access a comprehensive list of all currency pairs traded on the forex market with the FMP Forex Currency Pairs API. Analyze and track the performance of currency pairs to make informed investment decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*forex-list* - -Response - -\[ - { - "symbol": "ARSMXN", - "fromCurrency": "ARS", - "toCurrency": "MXN", - "fromName": "Argentine Peso", - "toName": "Mexican Peso" - } -\] - -[Forex Quote API](https://site.financialmodelingprep.com/developer/docs/stable/forex-quote) - -Access real-time forex quotes for currency pairs with the Forex Quote API. Retrieve up-to-date information on exchange rates and price changes to help monitor market movements. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | - -Response -\[ - { - "symbol": "EURUSD", - "name": "EUR/USD", - "price": 1.17598, - "changePercentage": \-0.14754, - "change": \-0.0017376, - "volume": 184065, - "dayLow": 1.17371, - "dayHigh": 1.17911, - "yearHigh": 1.18303, - "yearLow": 1.01838, - "marketCap": null, - "priceAvg50": 1.15244, - "priceAvg200": 1.08866, - "exchange": "FOREX", - "open": 1.17744, - "previousClose": 1.17772, - "timestamp": 1753374603 - } -\] - -[Forex Short Quote API](https://site.financialmodelingprep.com/developer/docs/stable/forex-quote-short) - -Quickly access concise forex pair quotes with the Forex Quote Snapshot API. Get a fast look at live currency exchange rates, price changes, and volume in real time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote-short?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | - -Response -\[ - { - "symbol": "EURUSD", - "price": 1.17598, - "change": \-0.0017376, - "volume": 184065 - } -\] - -[Batch Forex Quotes API](https://site.financialmodelingprep.com/developer/docs/stable/all-forex-quotes) - -Easily access real-time quotes for multiple forex pairs simultaneously with the Batch Forex Quotes API. Stay updated on global currency exchange rates and monitor price changes across different markets. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*batch-forex-quotes* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| short | boolean | true | - -Response -\[ - { - "symbol": "AEDAUD", - "price": 0.41372, - "change": 0.00153892, - "volume": 0 - } -\] - -[Historical Forex Light Chart API](https://site.financialmodelingprep.com/developer/docs/stable/forex-historical-price-eod-light) - -Access historical end-of-day forex prices with the Historical Forex Light Chart API. Track long-term price trends across different currency pairs to enhance your trading and analysis strategies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/light?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "symbol": "EURUSD", - "date": "2025-07-24", - "price": 1.17639, - "volume": 182290 - } -\] - -[Historical Forex Full Chart API](https://site.financialmodelingprep.com/developer/docs/stable/forex-historical-price-eod-full) - -Access comprehensive historical end-of-day forex price data with the Full Historical Forex Chart API. Gain detailed insights into currency pair movements, including open, high, low, close (OHLC) prices, volume, and percentage changes. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/full?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "symbol": "EURUSD", - "date": "2025-07-24", - "open": 1.17744, - "high": 1.17911, - "low": 1.17371, - "close": 1.17639, - "volume": 182290, - "change": \-0.00105, - "changePercent": \-0.08917652, - "vwap": 1.18 - } -\] - -[1-Minute Interval Forex Chart API](https://site.financialmodelingprep.com/developer/docs/stable/forex-intraday-1-min) - -Access real-time 1-minute intraday forex data with the 1-Minute Forex Interval Chart API. Track short-term price movements for precise, up-to-the-minute insights on currency pair fluctuations. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1min?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:29:00", - "open": 1.17582, - "low": 1.17582, - "high": 1.17599, - "close": 1.17598, - "volume": 184 - } -\] - -[5-Minute Interval Forex Chart API](https://site.financialmodelingprep.com/developer/docs/stable/forex-intraday-5-min) - -Track short-term forex trends with the 5-Minute Forex Interval Chart API. Access detailed 5-minute intraday data to monitor currency pair price movements and market conditions in near real-time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/5min?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:25:00", - "open": 1.17612, - "low": 1.17571, - "high": 1.17613, - "close": 1.17578, - "volume": 873 - } -\] - -[1-Hour Interval Forex Chart API](https://site.financialmodelingprep.com/developer/docs/stable/forex-intraday-1-hour) - -Track forex price movements over the trading day with the 1-Hour Forex Interval Chart API. This tool provides hourly intraday data for currency pairs, giving a detailed view of trends and market shifts. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1hour?*symbol*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | EURUSD | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response -\[ - { - "date": "2025-07-24 12:00:00", - "open": 1.17639, - "low": 1.17571, - "high": 1.1773, - "close": 1.17578, - "volume": 4909 - } -\] - -## **Statements** - -[Income Statement API](https://site.financialmodelingprep.com/developer/docs/stable/income-statement) - -Access real-time income statement data for public companies, private companies, and ETFs with the FMP Real-Time Income Statements API. Track profitability, compare competitors, and identify business trends with up-to-date financial data. - -**Endpoint:** - -https://financialmodelingprep.com/stable/income-statement?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "date": "2024-09-28", - "symbol": "AAPL", - "reportedCurrency": "USD", - "cik": "0000320193", - "filingDate": "2024-11-01", - "acceptedDate": "2024-11-01 06:01:36", - "fiscalYear": "2024", - "period": "FY", - "revenue": 391035000000, - "costOfRevenue": 210352000000, - "grossProfit": 180683000000, - "researchAndDevelopmentExpenses": 31370000000, - "generalAndAdministrativeExpenses": 0, - "sellingAndMarketingExpenses": 0, - "sellingGeneralAndAdministrativeExpenses": 26097000000, - "otherExpenses": 0, - "operatingExpenses": 57467000000, - "costAndExpenses": 267819000000, - "netInterestIncome": 0, - "interestIncome": 0, - "interestExpense": 0, - "depreciationAndAmortization": 11445000000, - "ebitda": 134661000000, - "ebit": 123216000000, - "nonOperatingIncomeExcludingInterest": 0, - "operatingIncome": 123216000000, - "totalOtherIncomeExpensesNet": 269000000, - "incomeBeforeTax": 123485000000, - "incomeTaxExpense": 29749000000, - "netIncomeFromContinuingOperations": 93736000000, - "netIncomeFromDiscontinuedOperations": 0, - "otherAdjustmentsToNetIncome": 0, - "netIncome": 93736000000, - "netIncomeDeductions": 0, - "bottomLineNetIncome": 93736000000, - "eps": 6.11, - "epsDiluted": 6.08, - "weightedAverageShsOut": 15343783000, - "weightedAverageShsOutDil": 15408095000 - } -\] - -[Balance Sheet Statement API](https://site.financialmodelingprep.com/developer/docs/stable/balance-sheet-statement) - -Access detailed balance sheet statements for publicly traded companies with the Balance Sheet Data API. Analyze assets, liabilities, and shareholder equity to gain insights into a company's financial health. - -**Endpoint:** - -https://financialmodelingprep.com/stable/balance-sheet-statement?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response: -\[ - { - "date": "2024-09-28", - "symbol": "AAPL", - "reportedCurrency": "USD", - "cik": "0000320193", - "filingDate": "2024-11-01", - "acceptedDate": "2024-11-01 06:01:36", - "fiscalYear": "2024", - "period": "FY", - "cashAndCashEquivalents": 29943000000, - "shortTermInvestments": 35228000000, - "cashAndShortTermInvestments": 65171000000, - "netReceivables": 66243000000, - "accountsReceivables": 33410000000, - "otherReceivables": 32833000000, - "inventory": 7286000000, - "prepaids": 0, - "otherCurrentAssets": 14287000000, - "totalCurrentAssets": 152987000000, - "propertyPlantEquipmentNet": 45680000000, - "goodwill": 0, - "intangibleAssets": 0, - "goodwillAndIntangibleAssets": 0, - "longTermInvestments": 91479000000, - "taxAssets": 19499000000, - "otherNonCurrentAssets": 55335000000, - "totalNonCurrentAssets": 211993000000, - "otherAssets": 0, - "totalAssets": 364980000000, - "totalPayables": 95561000000, - "accountPayables": 68960000000, - "otherPayables": 26601000000, - "accruedExpenses": 0, - "shortTermDebt": 20879000000, - "capitalLeaseObligationsCurrent": 1632000000, - "taxPayables": 26601000000, - "deferredRevenue": 8249000000, - "otherCurrentLiabilities": 50071000000, - "totalCurrentLiabilities": 176392000000, - "longTermDebt": 85750000000, - "deferredRevenueNonCurrent": 10798000000, - "deferredTaxLiabilitiesNonCurrent": 0, - "otherNonCurrentLiabilities": 35090000000, - "totalNonCurrentLiabilities": 131638000000, - "otherLiabilities": 0, - "capitalLeaseObligations": 12430000000, - "totalLiabilities": 308030000000, - "treasuryStock": 0, - "preferredStock": 0, - "commonStock": 83276000000, - "retainedEarnings": \-19154000000, - "additionalPaidInCapital": 0, - "accumulatedOtherComprehensiveIncomeLoss": \-7172000000, - "otherTotalStockholdersEquity": 0, - "totalStockholdersEquity": 56950000000, - "totalEquity": 56950000000, - "minorityInterest": 0, - "totalLiabilitiesAndTotalEquity": 364980000000, - "totalInvestments": 126707000000, - "totalDebt": 106629000000, - "netDebt": 76686000000 - } -\] - -[Cash Flow Statement API](https://site.financialmodelingprep.com/developer/docs/stable/cashflow-statement) - -Gain insights into a company's cash flow activities with the Cash Flow Statements API. Analyze cash generated and used from operations, investments, and financing activities to evaluate the financial health and sustainability of a business. - -**Endpoint:** - -https://financialmodelingprep.com/stable/cash-flow-statement?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "date": "2024-09-28", - "symbol": "AAPL", - "reportedCurrency": "USD", - "cik": "0000320193", - "filingDate": "2024-11-01", - "acceptedDate": "2024-11-01 06:01:36", - "fiscalYear": "2024", - "period": "FY", - "netIncome": 93736000000, - "depreciationAndAmortization": 11445000000, - "deferredIncomeTax": 0, - "stockBasedCompensation": 11688000000, - "changeInWorkingCapital": 3651000000, - "accountsReceivables": \-5144000000, - "inventory": \-1046000000, - "accountsPayables": 6020000000, - "otherWorkingCapital": 3821000000, - "otherNonCashItems": \-2266000000, - "netCashProvidedByOperatingActivities": 118254000000, - "investmentsInPropertyPlantAndEquipment": \-9447000000, - "acquisitionsNet": 0, - "purchasesOfInvestments": \-48656000000, - "salesMaturitiesOfInvestments": 62346000000, - "otherInvestingActivities": \-1308000000, - "netCashProvidedByInvestingActivities": 2935000000, - "netDebtIssuance": \-5998000000, - "longTermNetDebtIssuance": \-9958000000, - "shortTermNetDebtIssuance": 3960000000, - "netStockIssuance": \-94949000000, - "netCommonStockIssuance": \-94949000000, - "commonStockIssuance": 0, - "commonStockRepurchased": \-94949000000, - "netPreferredStockIssuance": 0, - "netDividendsPaid": \-15234000000, - "commonDividendsPaid": \-15234000000, - "preferredDividendsPaid": 0, - "otherFinancingActivities": \-5802000000, - "netCashProvidedByFinancingActivities": \-121983000000, - "effectOfForexChangesOnCash": 0, - "netChangeInCash": \-794000000, - "cashAtEndOfPeriod": 29943000000, - "cashAtBeginningOfPeriod": 30737000000, - "operatingCashFlow": 118254000000, - "capitalExpenditure": \-9447000000, - "freeCashFlow": 108807000000, - "incomeTaxesPaid": 26102000000, - "interestPaid": 0 - } -\] - -[Key Metrics API](https://site.financialmodelingprep.com/developer/docs/stable/key-metrics) - -Access essential financial metrics for a company with the FMP Financial Key Metrics API. Evaluate revenue, net income, P/E ratio, and more to assess performance and compare it to competitors. - -**Endpoint:** - -https://financialmodelingprep.com/stable/key-metrics?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "marketCap": 3495160329570, - "enterpriseValue": 3571846329570, - "evToSales": 9.134339201273542, - "evToOperatingCashFlow": 30.204866893043786, - "evToFreeCashFlow": 32.82735788662494, - "evToEBITDA": 26.524727497716487, - "netDebtToEBITDA": 0.5694744580836323, - "currentRatio": 0.8673125765340832, - "incomeQuality": 1.2615643936161134, - "grahamNumber": 22.587017267616833, - "grahamNetNet": \-12.352478525015636, - "taxBurden": 0.7590881483581001, - "interestBurden": 1.0021831580314244, - "workingCapital": \-23405000000, - "investedCapital": 22275000000, - "returnOnAssets": 0.25682503150857583, - "operatingReturnOnAssets": 0.3434290787011036, - "returnOnTangibleAssets": 0.25682503150857583, - "returnOnEquity": 1.6459350307287095, - "returnOnInvestedCapital": 0.4430708117427921, - "returnOnCapitalEmployed": 0.6533607652660827, - "earningsYield": 0.026818798327209237, - "freeCashFlowYield": 0.03113076074921754, - "capexToOperatingCashFlow": 0.07988736110406414, - "capexToDepreciation": 0.8254259501965924, - "capexToRevenue": 0.02415896275269477, - "salesGeneralAndAdministrativeToRevenue": 0, - "researchAndDevelopementToRevenue": 0.08022299794136074, - "stockBasedCompensationToRevenue": 0.02988990755303234, - "intangiblesToTotalAssets": 0, - "averageReceivables": 63614000000, - "averagePayables": 65785500000, - "averageInventory": 6808500000, - "daysOfSalesOutstanding": 61.83255974529134, - "daysOfPayablesOutstanding": 119.65847721913745, - "daysOfInventoryOutstanding": 12.642570548414087, - "operatingCycle": 74.47513029370543, - "cashConversionCycle": \-45.18334692543202, - "freeCashFlowToEquity": 32121000000, - "freeCashFlowToFirm": 117192805288.09166, - "tangibleAssetValue": 56950000000, - "netCurrentAssetValue": \-155043000000 - } -\] - -[Financial Ratios API](https://site.financialmodelingprep.com/developer/docs/stable/metrics-ratios) - -Analyze a company's financial performance using the Financial Ratios API. This API provides detailed profitability, liquidity, and efficiency ratios, enabling users to assess a company's operational and financial health across various metrics. - -**Endpoint:** - -https://financialmodelingprep.com/stable/ratios?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response - -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "grossProfitMargin": 0.4620634981523393, - "ebitMargin": 0.31510222870075566, - "ebitdaMargin": 0.3443707085043538, - "operatingProfitMargin": 0.31510222870075566, - "pretaxProfitMargin": 0.3157901466620635, - "continuousOperationsProfitMargin": 0.23971255769943867, - "netProfitMargin": 0.23971255769943867, - "bottomLineProfitMargin": 0.23971255769943867, - "receivablesTurnover": 5.903038811648023, - "payablesTurnover": 3.0503480278422272, - "inventoryTurnover": 28.870710952511665, - "fixedAssetTurnover": 8.560310858143607, - "assetTurnover": 1.0713874732862074, - "currentRatio": 0.8673125765340832, - "quickRatio": 0.8260068483831466, - "solvencyRatio": 0.3414634938155374, - "cashRatio": 0.16975259648963673, - "priceToEarningsRatio": 37.287278415656736, - "priceToEarningsGrowthRatio": \-45.93792700808932, - "forwardPriceToEarningsGrowthRatio": \-45.93792700808932, - "priceToBookRatio": 61.37243774486391, - "priceToSalesRatio": 8.93822887866815, - "priceToFreeCashFlowRatio": 32.12256867269569, - "priceToOperatingCashFlowRatio": 29.55638142954995, - "debtToAssetsRatio": 0.29215025480848267, - "debtToEquityRatio": 1.872326602282704, - "debtToCapitalRatio": 0.6518501763673821, - "longTermDebtToCapitalRatio": 0.6009110021023125, - "financialLeverageRatio": 6.408779631255487, - "workingCapitalTurnoverRatio": \-31.099932397502684, - "operatingCashFlowRatio": 0.6704045534944896, - "operatingCashFlowSalesRatio": 0.3024128274962599, - "freeCashFlowOperatingCashFlowRatio": 0.9201126388959359, - "debtServiceCoverageRatio": 5.024761722304708, - "interestCoverageRatio": 0, - "shortTermOperatingCashFlowCoverageRatio": 5.663777000814215, - "operatingCashFlowCoverageRatio": 1.109022873702276, - "capitalExpenditureCoverageRatio": 12.517624642743728, - "dividendPaidAndCapexCoverageRatio": 4.7912969490701345, - "dividendPayoutRatio": 0.16252026969360758, - "dividendYield": 0.0043585983369965175, - "dividendYieldPercentage": 0.43585983369965176, - "revenuePerShare": 25.484914639368924, - "netIncomePerShare": 6.109054070954992, - "interestDebtPerShare": 6.949329249507765, - "cashPerShare": 4.247388013764271, - "bookValuePerShare": 3.711600978715614, - "tangibleBookValuePerShare": 3.711600978715614, - "shareholdersEquityPerShare": 3.711600978715614, - "operatingCashFlowPerShare": 7.706965094592383, - "capexPerShare": 0.6156891035281195, - "freeCashFlowPerShare": 7.091275991064264, - "netIncomePerEBT": 0.7590881483581001, - "ebtPerEbit": 1.0021831580314244, - "priceToFairValue": 61.37243774486391, - "debtToMarketCap": 0.03050761336980449, - "effectiveTaxRate": 0.24091185164189982, - "enterpriseValueMultiple": 26.524727497716487 - } -\] - -[Financial Scores API](https://site.financialmodelingprep.com/developer/docs/stable/financial-scores) - -Assess a company's financial strength using the Financial Health Scores API. This API provides key metrics such as the Altman Z-Score and Piotroski Score, giving users insights into a company’s overall financial health and stability. - -**Endpoint:** - -https://financialmodelingprep.com/stable/financial-scores?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response -\[ - { - "symbol": "AAPL", - "reportedCurrency": "USD", - "altmanZScore": 9.322985825443649, - "piotroskiScore": 8, - "workingCapital": \-11125000000, - "totalAssets": 344085000000, - "retainedEarnings": \-11221000000, - "ebit": 125675000000, - "marketCap": 3259495258000, - "totalLiabilities": 277327000000, - "revenue": 395760000000 - } -\] - -[Owner Earnings API](https://site.financialmodelingprep.com/developer/docs/stable/owner-earnings) - -Retrieve a company's owner earnings with the Owner Earnings API, which provides a more accurate representation of cash available to shareholders by adjusting net income. This metric is crucial for evaluating a company’s profitability from the perspective of investors. - -**Endpoint:** - -https://financialmodelingprep.com/stable/owner-earnings?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | - -Response - -\[ - { - "symbol": "AAPL", - "reportedCurrency": "USD", - "fiscalYear": "2025", - "period": "Q1", - "date": "2024-12-28", - "averagePPE": 0.13969, - "maintenanceCapex": \-2279964750, - "ownersEarnings": 27655035250, - "growthCapex": \-660035250, - "ownersEarningsPerShare": 1.83 - } -\] - -[Enterprise Values API](https://site.financialmodelingprep.com/developer/docs/stable/enterprise-values) - -Access a company's enterprise value using the Enterprise Values API. This metric offers a comprehensive view of a company's total market value by combining both its equity (market capitalization) and debt, providing a better understanding of its worth. - -**Endpoint:** - -https://financialmodelingprep.com/stable/enterprise-values?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "stockPrice": 227.79, - "numberOfShares": 15343783000, - "marketCapitalization": 3495160329570, - "minusCashAndCashEquivalents": 29943000000, - "addTotalDebt": 106629000000, - "enterpriseValue": 3571846329570 - } -\] - -[Income Statement Growth API](https://site.financialmodelingprep.com/developer/docs/stable/income-statement-growth) - -Track key financial growth metrics with the Income Statement Growth API. Analyze how revenue, profits, and expenses have evolved over time, offering insights into a company’s financial health and operational efficiency. - -**Endpoint:** - -https://financialmodelingprep.com/stable/income-statement-growth?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "growthRevenue": 0.020219940775141214, - "growthCostOfRevenue": \-0.017675600199872046, - "growthGrossProfit": 0.06819471705252206, - "growthGrossProfitRatio": 0.04776303446712012, - "growthResearchAndDevelopmentExpenses": 0.04863780712017383, - "growthGeneralAndAdministrativeExpenses": 0, - "growthSellingAndMarketingExpenses": 0, - "growthOtherExpenses": \-1, - "growthOperatingExpenses": 0.04776924900176856, - "growthCostAndExpenses": \-0.004331112631234571, - "growthInterestIncome": \-1, - "growthInterestExpense": \-1, - "growthDepreciationAndAmortization": \-0.006424168764649709, - "growthEBITDA": 0.07026704816404387, - "growthOperatingIncome": 0.07799581805933456, - "growthIncomeBeforeTax": 0.08571604417246959, - "growthIncomeTaxExpense": 0.7770145152619318, - "growthNetIncome": \-0.033599670086086914, - "growthEPS": \-0.008116883116883088, - "growthEPSDiluted": \-0.008156606851549727, - "growthWeightedAverageShsOut": \-0.02543458616683152, - "growthWeightedAverageShsOutDil": \-0.02557791606880283, - "growthEBIT": 0.0471407082579099, - "growthNonOperatingIncomeExcludingInterest": 1, - "growthNetInterestIncome": 1, - "growthTotalOtherIncomeExpensesNet": 1.4761061946902654, - "growthNetIncomeFromContinuingOperations": \-0.033599670086086914, - "growthOtherAdjustmentsToNetIncome": 0, - "growthNetIncomeDeductions": 0 - } -\] - -[Balance Sheet Statement Growth API](https://site.financialmodelingprep.com/developer/docs/stable/balance-sheet-statement-growth) - -Analyze the growth of key balance sheet items over time with the Balance Sheet Statement Growth API. Track changes in assets, liabilities, and equity to understand the financial evolution of a company. - -**Endpoint:** - -https://financialmodelingprep.com/stable/balance-sheet-statement-growth?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "growthCashAndCashEquivalents": \-0.0007341898882029034, - "growthShortTermInvestments": 0.11516302627413738, - "growthCashAndShortTermInvestments": 0.058744212492892536, - "growthNetReceivables": 0.08621792243994425, - "growthInventory": 0.15084504817564365, - "growthOtherCurrentAssets": \-0.02776454576386526, - "growthTotalCurrentAssets": 0.06562138667929733, - "growthPropertyPlantEquipmentNet": \-0.15992349565984992, - "growthGoodwill": 0, - "growthIntangibleAssets": 0, - "growthGoodwillAndIntangibleAssets": 0, - "growthLongTermInvestments": \-0.09015953214513049, - "growthTaxAssets": 0.09225857046829487, - "growthOtherNonCurrentAssets": 0.5266933370120016, - "growthTotalNonCurrentAssets": 0.014238076328719674, - "growthOtherAssets": 0, - "growthTotalAssets": 0.035160515396374756, - "growthAccountPayables": 0.1014039066617687, - "growthShortTermDebt": 0.32087050041121024, - "growthTaxPayables": 2.01632838190271, - "growthDeferredRevenue": 0.023322168465450935, - "growthOtherCurrentLiabilities": \-0.1254584832500786, - "growthTotalCurrentLiabilities": 0.21391802240757563, - "growthLongTermDebt": \-0.10003043628845205, - "growthDeferredRevenueNonCurrent": 0, - "growthDeferredTaxLiabilitiesNonCurrent": 0, - "growthOtherNonCurrentLiabilities": \-0.09048495373370312, - "growthTotalNonCurrentLiabilities": \-0.09295867814151548, - "growthOtherLiabilities": 0, - "growthTotalLiabilities": 0.060574238130816666, - "growthPreferredStock": 0, - "growthCommonStock": 0.12821763398905328, - "growthRetainedEarnings": \-88.50467289719626, - "growthAccumulatedOtherComprehensiveIncomeLoss": 0.3737338456164862, - "growthOthertotalStockholdersEquity": 0, - "growthTotalStockholdersEquity": \-0.0836095645737457, - "growthMinorityInterest": 0, - "growthTotalEquity": \-0.0836095645737457, - "growthTotalLiabilitiesAndStockholdersEquity": 0.035160515396374756, - "growthTotalInvestments": \-0.04107194211936368, - "growthTotalDebt": \-0.0401393489845888, - "growthNetDebt": \-0.05469472282829777, - "growthAccountsReceivables": 0.13223532601328453, - "growthOtherReceivables": 0.04307907360930203, - "growthPrepaids": 0, - "growthTotalPayables": 0.5262653527335452, - "growthOtherPayables": 0, - "growthAccruedExpenses": 0, - "growthCapitalLeaseObligationsCurrent": 0.03619047619047619, - "growthAdditionalPaidInCapital": 0, - "growthTreasuryStock": 0 - } -\] - -[Cashflow Statement Growth API](https://site.financialmodelingprep.com/developer/docs/stable/cashflow-statement-growth) - -Measure the growth rate of a company’s cash flow with the FMP Cashflow Statement Growth API. Determine how quickly a company’s cash flow is increasing or decreasing over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/cash-flow-statement-growth?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "growthNetIncome": \-0.033599670086086914, - "growthDepreciationAndAmortization": \-0.006424168764649709, - "growthDeferredIncomeTax": 0, - "growthStockBasedCompensation": 0.07892550540016616, - "growthChangeInWorkingCapital": 1.555116314429071, - "growthAccountsReceivables": \-2.0473933649289098, - "growthInventory": 0.3535228677379481, - "growthAccountsPayables": 4.1868713605082055, - "growthOtherWorkingCapital": 2.4402563136072373, - "growthOtherNonCashItems": \-0.017512348450830714, - "growthNetCashProvidedByOperatingActivites": 0.06975566069312394, - "growthInvestmentsInPropertyPlantAndEquipment": 0.13796879277306323, - "growthAcquisitionsNet": 0, - "growthPurchasesOfInvestments": \-0.6486294175448107, - "growthSalesMaturitiesOfInvestments": 0.3698202750801951, - "growthOtherInvestingActivites": 0.02169035153328347, - "growthNetCashUsedForInvestingActivites": \-0.2078272604588394, - "growthDebtRepayment": \-0.012662502110417018, - "growthCommonStockIssued": 0, - "growthCommonStockRepurchased": \-0.2243584784010316, - "growthDividendsPaid": \-0.013910149750415973, - "growthOtherFinancingActivites": 0.03493013972055888, - "growthNetCashUsedProvidedByFinancingActivities": \-0.12439163778482412, - "growthEffectOfForexChangesOnCash": 0, - "growthNetChangeInCash": \-1.1378472222222222, - "growthCashAtEndOfPeriod": \-0.02583205908188828, - "growthCashAtBeginningOfPeriod": 0.23061216319013492, - "growthOperatingCashFlow": 0.06975566069312394, - "growthCapitalExpenditure": 0.13796879277306323, - "growthFreeCashFlow": 0.092615279562982, - "growthNetDebtIssuance": 0.3942026057973942, - "growthLongTermNetDebtIssuance": \-0.6812426135404356, - "growthShortTermNetDebtIssuance": 1.995475113122172, - "growthNetStockIssuance": \-0.2243584784010316, - "growthPreferredDividendsPaid": \-0.013910149750415973, - "growthIncomeTaxesPaid": 0.3973981476524439, - "growthInterestPaid": \-1 - } -\] - -[Financial Statement Growth API](https://site.financialmodelingprep.com/developer/docs/stable/financial-statement-growth) - -Analyze the growth of key financial statement items across income, balance sheet, and cash flow statements with the Financial Statement Growth API. Track changes over time to understand trends in financial performance. - -**Endpoint:** - -https://financialmodelingprep.com/stable/financial-growth?*symbol*\=AAPL - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 5 | -| period | string | Q1,Q2,Q3,Q4,FY,annual,quarter | - -Response -\[ - { - "symbol": "AAPL", - "date": "2024-09-28", - "fiscalYear": "2024", - "period": "FY", - "reportedCurrency": "USD", - "revenueGrowth": 0.020219940775141214, - "grossProfitGrowth": 0.06819471705252206, - "ebitgrowth": 0.07799581805933456, - "operatingIncomeGrowth": 0.07799581805933456, - "netIncomeGrowth": \-0.033599670086086914, - "epsgrowth": \-0.008116883116883088, - "epsdilutedGrowth": \-0.008156606851549727, - "weightedAverageSharesGrowth": \-0.02543458616683152, - "weightedAverageSharesDilutedGrowth": \-0.02557791606880283, - "dividendsPerShareGrowth": 0.040371570095532654, - "operatingCashFlowGrowth": 0.06975566069312394, - "receivablesGrowth": 0.08621792243994425, - "inventoryGrowth": 0.15084504817564365, - "assetGrowth": 0.035160515396374756, - "bookValueperShareGrowth": \-0.059693251557224776, - "debtGrowth": \-0.0401393489845888, - "rdexpenseGrowth": 0.04863780712017383, - "sgaexpensesGrowth": 0.04672709770575967, - "freeCashFlowGrowth": 0.092615279562982, - "tenYRevenueGrowthPerShare": 2.3937532854122625, - "fiveYRevenueGrowthPerShare": 0.8093292228858464, - "threeYRevenueGrowthPerShare": 0.163506592883552, - "tenYOperatingCFGrowthPerShare": 2.1417809176982403, - "fiveYOperatingCFGrowthPerShare": 1.051533221923415, - "threeYOperatingCFGrowthPerShare": 0.23720294833900227, - "tenYNetIncomeGrowthPerShare": 2.76381558093543, - "fiveYNetIncomeGrowthPerShare": 1.0421744314966246, - "threeYNetIncomeGrowthPerShare": 0.07761907162786884, - "tenYShareholdersEquityGrowthPerShare": \-0.19003774225234785, - "fiveYShareholdersEquityGrowthPerShare": \-0.24235004889283715, - "threeYShareholdersEquityGrowthPerShare": \-0.017459858915902907, - "tenYDividendperShareGrowthPerShare": 1.1722201809466772, - "fiveYDividendperShareGrowthPerShare": 0.29890046876764864, - "threeYDividendperShareGrowthPerShare": 0.14617932692103452, - "ebitdaGrowth": null, - "growthCapitalExpenditure": null, - "tenYBottomLineNetIncomeGrowthPerShare": null, - "fiveYBottomLineNetIncomeGrowthPerShare": null, - "threeYBottomLineNetIncomeGrowthPerShare": null - } -\] - -[Revenue Product Segmentation API](https://site.financialmodelingprep.com/developer/docs/stable/revenue-product-segmentation) - -Access detailed revenue breakdowns by product line with the Revenue Product Segmentation API. Understand which products drive a company's earnings and get insights into the performance of individual product segments. - -**Endpoint:** - -https://financialmodelingprep.com/stable/revenue-product-segmentation?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| period | string | annual,quarter | -| structure | string | flat | - -Response -\[ - { - "symbol": "AAPL", - "fiscalYear": 2024, - "period": "FY", - "reportedCurrency": null, - "date": "2024-09-28", - "data": { - "Mac": 29984000000, - "Service": 96169000000, - "Wearables, Home and Accessories": 37005000000, - "iPad": 26694000000, - "iPhone": 201183000000 - } - } -\] - -[Revenue Geographic Segments API](https://site.financialmodelingprep.com/developer/docs/stable/revenue-geographic-segments) - -Access detailed revenue breakdowns by geographic region with the Revenue Geographic Segments API. Analyze how different regions contribute to a company’s total revenue and identify key markets for growth. - -**Endpoint:** - -https://financialmodelingprep.com/stable/revenue-geographic-segmentation?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| period | string | annual,quarter | -| structure | string | flat | - -Response -\[ - { - "symbol": "AAPL", - "fiscalYear": 2024, - "period": "FY", - "reportedCurrency": null, - "date": "2024-09-28", - "data": { - "Americas Segment": 167045000000, - "Europe Segment": 101328000000, - "Greater China Segment": 66952000000, - "Japan Segment": 25052000000, - "Rest of Asia Pacific": 30658000000 - } - } -\] - -## **Form 13F** - -[Institutional Ownership Filings API](https://site.financialmodelingprep.com/developer/docs/stable/latest-filings) - -Stay up to date with the most recent SEC filings related to institutional ownership using the Institutional Ownership Filings API. This tool allows you to track the latest reports and disclosures from institutional investors, giving you a real-time view of major holdings and regulatory submissions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/latest?*page*\=0&*limit*\=100 - -Query Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 100 | - -Response -\[ - { - "cik": "0001963967", - "name": "CPA ASSET MANAGEMENT LLC", - "date": "2024-12-31", - "filingDate": "2025-02-04 00:00:00", - "acceptedDate": "2025-02-04 17:28:36", - "formType": "13F-HR", - "link": "https://www.sec.gov/Archives/edgar/data/1963967/000196396725000001/0001963967-25-000001-index.htm", - "finalLink": "https://www.sec.gov/Archives/edgar/data/1963967/000196396725000001/boc2024q413f.xml" - } -\] - -[Filings Extract API](https://site.financialmodelingprep.com/developer/docs/stable/filings-extract) - -The SEC Filings Extract API allows users to extract detailed data directly from official SEC filings. This API provides access to key information such as company shares, security details, and filing links, making it easier to analyze corporate disclosures. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/extract?*cik*\=0001388838&*year*\=2023&*quarter*\=3 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| cik\* | string | 0001388838 | -| year\* | string | 2023 | -| quarter\* | string | 3 | - -Response -\[ - { - "date": "2023-09-30", - "filingDate": "2023-11-13", - "acceptedDate": "2023-11-13", - "cik": "0001388838", - "securityCusip": "674215207", - "symbol": "CHRD", - "nameOfIssuer": "CHORD ENERGY CORPORATION", - "shares": 13280, - "titleOfClass": "COM NEW", - "sharesType": "SH", - "putCallShare": "", - "value": 2152290, - "link": "https://www.sec.gov/Archives/edgar/data/1388838/000117266123003760/0001172661-23-003760-index.htm", - "finalLink": "https://www.sec.gov/Archives/edgar/data/1388838/000117266123003760/infotable.xml" - } -\] -[Form 13F Filings Dates API](https://site.financialmodelingprep.com/developer/docs/stable/form-13f-filings-dates) - -The Form 13F Filings Dates API allows you to retrieve dates associated with Form 13F filings by institutional investors. This is crucial for tracking stock holdings of institutional investors at specific points in time, providing valuable insights into their investment strategies. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/dates?*cik*\=0001067983 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| cik\* | string | 0001067983 | - -Response - -\[ - { - "date": "2024-09-30", - "year": 2024, - "quarter": 3 - } -\] - -[Filings Extract With Analytics By Holder API](https://site.financialmodelingprep.com/developer/docs/stable/filings-extract-with-analytics-by-holder) - -The Filings Extract With Analytics By Holder API provides an analytical breakdown of institutional filings. This API offers insight into stock movements, strategies, and portfolio changes by major institutional holders, helping you understand their investment behavior and track significant changes in stock ownership. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/extract-analytics/holder?*symbol*\=AAPL&*year*\=2023&*quarter*\=3&*page*\=0&*limit*\=10 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| year\* | string | 2023 | -| quarter\* | string | 3 | -| page | number | 0 | -| limit | number | 10 | - -Response - -\[ - { - "date": "2023-09-30", - "cik": "0000102909", - "filingDate": "2023-12-18", - "investorName": "VANGUARD GROUP INC", - "symbol": "AAPL", - "securityName": "APPLE INC", - "typeOfSecurity": "COM", - "securityCusip": "037833100", - "sharesType": "SH", - "putCallShare": "Share", - "investmentDiscretion": "SOLE", - "industryTitle": "ELECTRONIC COMPUTERS", - "weight": 5.4673, - "lastWeight": 5.996, - "changeInWeight": \-0.5287, - "changeInWeightPercentage": \-8.8175, - "marketValue": 222572509140, - "lastMarketValue": 252876459509, - "changeInMarketValue": \-30303950369, - "changeInMarketValuePercentage": \-11.9837, - "sharesNumber": 1299997133, - "lastSharesNumber": 1303688506, - "changeInSharesNumber": \-3691373, - "changeInSharesNumberPercentage": \-0.2831, - "quarterEndPrice": 171.21, - "avgPricePaid": 95.86, - "isNew": false, - "isSoldOut": false, - "ownership": 8.3336, - "lastOwnership": 8.305, - "changeInOwnership": 0.0286, - "changeInOwnershipPercentage": 0.3445, - "holdingPeriod": 42, - "firstAdded": "2013-06-30", - "performance": \-29671950396, - "performancePercentage": \-11.7338, - "lastPerformance": 38078179274, - "changeInPerformance": \-67750129670, - "isCountedForPerformance": true - } -\] - -[Holder Performance Summary API](https://site.financialmodelingprep.com/developer/docs/stable/holder-performance-summary) - -The Holder Performance Summary API provides insights into the performance of institutional investors based on their stock holdings. This data helps track how well institutional holders are performing, their portfolio changes, and how their performance compares to benchmarks like the S\&P 500\. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/holder-performance-summary?*cik*\=0001067983&*page*\=0 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| cik\* | string | 0001067983 | -| page | number | 0 | - -Response - -\[ - { - "date": "2024-09-30", - "cik": "0001067983", - "investorName": "BERKSHIRE HATHAWAY INC", - "portfolioSize": 40, - "securitiesAdded": 3, - "securitiesRemoved": 4, - "marketValue": 266378900503, - "previousMarketValue": 279969062343, - "changeInMarketValue": \-13590161840, - "changeInMarketValuePercentage": \-4.8542, - "averageHoldingPeriod": 18, - "averageHoldingPeriodTop10": 31, - "averageHoldingPeriodTop20": 27, - "turnover": 0.175, - "turnoverAlternateSell": 13.9726, - "turnoverAlternateBuy": 1.1974, - "performance": 17707926874, - "performancePercentage": 6.325, - "lastPerformance": 38318168662, - "changeInPerformance": \-20610241788, - "performance1year": 89877376224, - "performancePercentage1year": 28.5368, - "performance3year": 91730847239, - "performancePercentage3year": 31.2597, - "performance5year": 157058602844, - "performancePercentage5year": 73.1617, - "performanceSinceInception": 182067479115, - "performanceSinceInceptionPercentage": 198.2138, - "performanceRelativeToSP500Percentage": 6.325, - "performance1yearRelativeToSP500Percentage": 28.5368, - "performance3yearRelativeToSP500Percentage": 36.5632, - "performance5yearRelativeToSP500Percentage": 36.1296, - "performanceSinceInceptionRelativeToSP500Percentage": 37.0968 - } -\] - -[Holders Industry Breakdown API](https://site.financialmodelingprep.com/developer/docs/stable/holders-industry-breakdown) - -The Holders Industry Breakdown API provides an overview of the sectors and industries that institutional holders are investing in. This API helps analyze how institutional investors distribute their holdings across different industries and track changes in their investment strategies over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/holder-industry-breakdown?*cik*\=0001067983&*year*\=2023&*quarter*\=3 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| cik\* | string | 0001067983 | -| year\* | string | 2023 | -| quarter\* | string | 3 | - -Response -\[ - { - "date": "2023-09-30", - "cik": "0001067983", - "investorName": "BERKSHIRE HATHAWAY INC", - "industryTitle": "ELECTRONIC COMPUTERS", - "weight": 49.7704, - "lastWeight": 51.0035, - "changeInWeight": \-1.2332, - "changeInWeightPercentage": \-2.4178, - "performance": \-20838154294, - "performancePercentage": \-178.2938, - "lastPerformance": 26615340304, - "changeInPerformance": \-47453494598 - } -\] - -[Positions Summary API](https://site.financialmodelingprep.com/developer/docs/stable/positions-summary) - -The Positions Summary API provides a comprehensive snapshot of institutional holdings for a specific stock symbol. It tracks key metrics like the number of investors holding the stock, changes in the number of shares, total investment value, and ownership percentages over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/symbol-positions-summary?*symbol*\=AAPL&*year*\=2023&*quarter*\=3 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| year\* | string | 2023 | -| quarter\* | string | 3 | - -Response - -\[ - { - "symbol": "AAPL", - "cik": "0000320193", - "date": "2023-09-30", - "investorsHolding": 4805, - "lastInvestorsHolding": 4749, - "investorsHoldingChange": 56, - "numberOf13Fshares": 9247670386, - "lastNumberOf13Fshares": 9345671472, - "numberOf13FsharesChange": \-98001086, - "totalInvested": 1613733330618, - "lastTotalInvested": 1825154796061, - "totalInvestedChange": \-211421465443, - "ownershipPercent": 59.2821, - "lastOwnershipPercent": 59.5356, - "ownershipPercentChange": \-0.2535, - "newPositions": 158, - "lastNewPositions": 188, - "newPositionsChange": \-30, - "increasedPositions": 1921, - "lastIncreasedPositions": 1775, - "increasedPositionsChange": 146, - "closedPositions": 156, - "lastClosedPositions": 122, - "closedPositionsChange": 34, - "reducedPositions": 2375, - "lastReducedPositions": 2506, - "reducedPositionsChange": \-131, - "totalCalls": 173528138, - "lastTotalCalls": 198746782, - "totalCallsChange": \-25218644, - "totalPuts": 192878290, - "lastTotalPuts": 177007062, - "totalPutsChange": 15871228, - "putCallRatio": 1.1115, - "lastPutCallRatio": 0.8906, - "putCallRatioChange": 22.0894 - } -\] - -[Industry Performance Summary API](https://site.financialmodelingprep.com/developer/docs/stable/industry-summary) - -The Industry Performance Summary API provides an overview of how various industries are performing financially. By analyzing the value of industries over a specific period, this API helps investors and analysts understand the health of entire sectors and make informed decisions about sector-based investments. - -**Endpoint:** - -https://financialmodelingprep.com/stable/institutional-ownership/industry-summary?*year*\=2023&*quarter*\=3 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| year\* | string | 2023 | -| quarter\* | string | 3 | - -Response - -\[ - { - "industryTitle": "ABRASIVE, ASBESTOS & MISC NONMETALLIC MINERAL PRODS", - "industryValue": 10979226300, - "date": "2023-09-30" - } -\] - -## **Indexes** - -[Stock Market Indexes List API](https://site.financialmodelingprep.com/developer/docs/stable/indexes-list) - -Retrieve a comprehensive list of stock market indexes across global exchanges using the FMP Stock Market Indexes List API. This API provides essential information such as the symbol, name, exchange, and currency for each index, helping analysts and investors keep track of various market benchmarks. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*index-list* - -Response - -\[ - { - "symbol": "^TTIN", - "name": "S\&P/TSX Capped Industrials Index", - "exchange": "TSX", - "currency": "CAD" - } -\] - -[Index Quote API](https://site.financialmodelingprep.com/developer/docs/stable/index-quote) - -Access real-time stock index quotes with the Stock Index Quote API. Stay updated with the latest price changes, daily highs and lows, volume, and other key metrics for major stock indices around the world. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | - -Response - -\[ - { - "symbol": "^GSPC", - "name": "S\&P 500", - "price": 6366.13, - "changePercentage": 0.11354, - "change": 7.22, - "volume": 1498664000, - "dayLow": 6360.57, - "dayHigh": 6379.54, - "yearHigh": 6379.54, - "yearLow": 4835.04, - "marketCap": 0, - "priceAvg50": 6068.663, - "priceAvg200": 5880.0864, - "exchange": "INDEX", - "open": 6368.6, - "previousClose": 6358.91, - "timestamp": 1753374601 - } -\] - -[Index Short Quote API](https://site.financialmodelingprep.com/developer/docs/stable/index-quote-short) - -Access concise stock index quotes with the Stock Index Short Quote API. This API provides a snapshot of the current price, change, and volume for stock indexes, making it ideal for users who need a quick overview of market movements. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote-short?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | - -Response - -\[ - { - "symbol": "^GSPC", - "price": 6366.13, - "change": 7.22, - "volume": 1498664000 - } -\] - -[All Index Quotes API](https://site.financialmodelingprep.com/developer/docs/stable/all-index-quotes) - -The All Index Quotes API provides real-time quotes for a wide range of stock indexes, from major market benchmarks to niche indexes. This API allows users to track market performance across multiple indexes in a single request, giving them a broad view of the financial markets. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*batch-index-quotes* - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| short | boolean | true | - -Response - -\[ - { - "symbol": "^DJBGIE", - "price": 4155.76, - "change": 1.09, - "volume": 0 - } -\] - -[Historical Index Light Chart API](https://site.financialmodelingprep.com/developer/docs/stable/index-historical-price-eod-light) - -Retrieve end-of-day historical prices for stock indexes using the Historical Price Data API. This API provides essential data such as date, price, and volume, enabling detailed analysis of price movements over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/light?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "symbol": "^GSPC", - "date": "2025-07-24", - "price": 6365.77, - "volume": 1499302000 - } -\] - -[Historical Index Full Chart API](https://site.financialmodelingprep.com/developer/docs/stable/index-historical-price-eod-full) - -Access full historical end-of-day prices for stock indexes using the Detailed Historical Price Data API. This API provides comprehensive information, including open, high, low, close prices, volume, and additional metrics for detailed financial analysis. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-price-eod/full?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "symbol": "^GSPC", - "date": "2025-07-24", - "open": 6368.6, - "high": 6379.54, - "low": 6360.57, - "close": 6365.77, - "volume": 1499302000, - "change": \-2.83, - "changePercent": \-0.04443677, - "vwap": 6368.63 - } -\] - -[1-Minute Interval Index Price API](https://site.financialmodelingprep.com/developer/docs/stable/index-intraday-1-min) - -Retrieve 1-minute interval intraday data for stock indexes using the Intraday 1-Minute Price Data API. This API provides granular price information, helping users track short-term price movements and trading volume within each minute. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1min?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "date": "2025-07-24 12:29:00", - "open": 6365.34, - "low": 6365.34, - "high": 6366.09, - "close": 6366.09, - "volume": 4428000 - } -\] - -[5-Minute Interval Index Price API](https://site.financialmodelingprep.com/developer/docs/stable/index-intraday-5-min) - -Retrieve 5-minute interval intraday price data for stock indexes using the Intraday 5-Minute Price Data API. This API provides crucial insights into price movements and trading volume within 5-minute windows, ideal for traders who require short-term data. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/5min?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "date": "2025-07-24 12:30:00", - "open": 6366.18, - "low": 6365.57, - "high": 6366.18, - "close": 6365.69, - "volume": 1574690 - } -\] - -[1-Hour Interval Index Price API](https://site.financialmodelingprep.com/developer/docs/stable/index-intraday-1-hour) - -Access 1-hour interval intraday data for stock indexes using the Intraday 1-Hour Price Data API. This API provides detailed price movements and volume within hourly intervals, making it ideal for tracking medium-term market trends during the trading day. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-chart/1hour?*symbol*\=^GSPC - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | ^GSPC | -| from | date | 2025-04-25 | -| to | date | 2025-07-25 | - -Response - -\[ - { - "date": "2025-07-24 12:30:00", - "open": 6366.18, - "low": 6365.57, - "high": 6366.18, - "close": 6365.69, - "volume": 1574690 - } -\] - -## **Insider Trades** - -[Latest Insider Trading API](https://site.financialmodelingprep.com/developer/docs/stable/latest-insider-trade) - -Access the latest insider trading activity using the Latest Insider Trading API. Track which company insiders are buying or selling stocks and analyze their transactions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/insider-trading/latest?*page*\=0&*limit*\=100 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| date | date | 2025-04-24 | -| page | number | 0 | -| limit | number | 100 | - -Response - -\[ - { - "symbol": "APA", - "filingDate": "2025-02-04", - "transactionDate": "2025-02-01", - "reportingCik": "0001380034", - "companyCik": "0001841666", - "transactionType": "M-Exempt", - "securitiesOwned": 104398, - "reportingName": "Hoyt Rebecca A", - "typeOfOwner": "officer: Sr. VP, Chief Acct Officer", - "acquisitionOrDisposition": "A", - "directOrIndirect": "D", - "formType": "4", - "securitiesTransacted": 3450, - "price": 0, - "securityName": "Common Stock", - "url": "https://www.sec.gov/Archives/edgar/data/1841666/000194906025000035/0001949060-25-000035-index.htm" - } -\] - -[Search Insider Trades API](https://site.financialmodelingprep.com/developer/docs/stable/search-insider-trades) - -Search insider trading activity by company or symbol using the Search Insider Trades API. Find specific trades made by corporate insiders, including executives and directors. - -**Endpoint:** - -https://financialmodelingprep.com/stable/insider-trading/search?*page*\=0&*limit*\=100 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol | string | AAPL | -| page | number | 0 | -| limit | number | 100 | -| reportingCik | string | 0001496686 | -| companyCik | string | 0000320193 | -| transactionType | string | S-Sale | - -Response - -\[ - { - "symbol": "AAPL", - "filingDate": "2025-02-04", - "transactionDate": "2025-02-03", - "reportingCik": "0001214128", - "companyCik": "0000320193", - "transactionType": "S-Sale", - "securitiesOwned": 4159576, - "reportingName": "LEVINSON ARTHUR D", - "typeOfOwner": "director", - "acquisitionOrDisposition": "D", - "directOrIndirect": "D", - "formType": "4", - "securitiesTransacted": 1516, - "price": 226.3501, - "securityName": "Common Stock", - "url": "https://www.sec.gov/Archives/edgar/data/320193/000032019325000019/0000320193-25-000019-index.htm" - } -\] - -[Search Insider Trades by Reporting Name API](https://site.financialmodelingprep.com/developer/docs/stable/search-reporting-name) - -Search for insider trading activity by reporting name using the Search Insider Trades by Reporting Name API. Track trading activities of specific individuals or groups involved in corporate insider transactions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/insider-trading/reporting-name?*name*\=Zuckerberg - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| name\* | string | Zuckerberg | - -Response - -\[ - { - "reportingCik": "0001548760", - "reportingName": "Zuckerberg Mark" - } -\] - -[All Insider Transaction Types API](https://site.financialmodelingprep.com/developer/docs/stable/all-transaction-types) - -Access a comprehensive list of insider transaction types with the All Insider Transaction Types API. This API provides details on various transaction actions, including purchases, sales, and other corporate actions involving insider trading. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*insider-trading-transaction-type* - -Response - -\[ - { - "transactionType": "A-Award" - } -\] - -[Insider Trade Statistics API](https://site.financialmodelingprep.com/developer/docs/stable/insider-trade-statistics) - -Analyze insider trading activity with the Insider Trade Statistics API. This API provides key statistics on insider transactions, including total purchases, sales, and trends for specific companies or stock symbols. - -**Endpoint:** - -https://financialmodelingprep.com/stable/insider-trading/statistics?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - { - "symbol": "AAPL", - "cik": "0000320193", - "year": 2024, - "quarter": 4, - "acquiredTransactions": 6, - "disposedTransactions": 38, - "acquiredDisposedRatio": 0.1579, - "totalAcquired": 994544, - "totalDisposed": 2297088, - "averageAcquired": 165757.3333, - "averageDisposed": 60449.6842, - "totalPurchases": 0, - "totalSales": 22 - } -\] - -[Acquisition Ownership API](https://site.financialmodelingprep.com/developer/docs/stable/acquisition-ownership) - -Track changes in stock ownership during acquisitions using the Acquisition Ownership API. This API provides detailed information on how mergers, takeovers, or beneficial ownership changes impact the stock ownership structure of a company. - -**Endpoint:** - -https://financialmodelingprep.com/stable/acquisition-of-beneficial-ownership?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| limit | number | 2000 | - -Response - -\[ - { - "cik": "0000320193", - "symbol": "AAPL", - "filingDate": "2024-02-14", - "acceptedDate": "2024-02-14", - "cusip": "037833100", - "nameOfReportingPerson": "National Indemnity Company", - "citizenshipOrPlaceOfOrganization": "State of Nebraska", - "soleVotingPower": "0", - "sharedVotingPower": "755059877", - "soleDispositivePower": "0", - "sharedDispositivePower": "755059877", - "amountBeneficiallyOwned": "755059877", - "percentOfClass": "4.8", - "typeOfReportingPerson": "IC, EP, IN, CO", - "url": "https://www.sec.gov/Archives/edgar/data/320193/000119312524036431/d751537dsc13ga.htm" - } -\] - -## **Market Performance** - -[Market Sector Performance Snapshot API](https://site.financialmodelingprep.com/developer/docs/stable/sector-performance-snapshot) - -Get a snapshot of sector performance using the Market Sector Performance Snapshot API. Analyze how different industries are performing in the market based on average changes across sectors. - -**Endpoint:** - -https://financialmodelingprep.com/stable/sector-performance-snapshot?*date*\=2024-02-01 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| date\* | string | 2024-02-01 | -| exchange | string | NASDAQ | -| sector | string | Energy | - -Response - -\[ - { - "date": "2024-02-01", - "sector": "Basic Materials", - "exchange": "NASDAQ", - "averageChange": \-0.31481377464310634 - } -\] - -[Industry Performance Snapshot API](https://site.financialmodelingprep.com/developer/docs/stable/industry-performance-snapshot) - -Access detailed performance data by industry using the Industry Performance Snapshot API. Analyze trends, movements, and daily performance metrics for specific industries across various stock exchanges. - -**Endpoint:** - -https://financialmodelingprep.com/stable/industry-performance-snapshot?*date*\=2024-02-01 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| date\* | string | 2024-02-01 | -| exchange | string | NASDAQ | -| industry | string | Biotechnology | - -Response - -\[ - { - "date": "2024-02-01", - "industry": "Advertising Agencies", - "exchange": "NASDAQ", - "averageChange": 3.8660194344955996 - } -\] - -[Historical Market Sector Performance API](https://site.financialmodelingprep.com/developer/docs/stable/historical-sector-performance) - -Access historical sector performance data using the Historical Market Sector Performance API. Review how different sectors have performed over time across various stock exchanges. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-sector-performance?*sector*\=Energy - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | string | 2024-02-01 | -| exchange | string | NASDAQ | -| sector\* | string | Energy | -| to | string | 2024-03-01 | - -Response - -\[ - { - "date": "2024-02-01", - "sector": "Energy", - "exchange": "NASDAQ", - "averageChange": 0.6397534025664513 - } -\] - -[Historical Industry Performance API](https://site.financialmodelingprep.com/developer/docs/stable/historical-industry-performance) - -Access historical performance data for industries using the Historical Industry Performance API. Track long-term trends and analyze how different industries have evolved over time across various stock exchanges. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-industry-performance?*industry*\=Biotechnology - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| industry\* | string | Biotechnology | -| exchange | string | NASDAQ | -| from | string | 2024-02-01 | -| to | string | 2024-03-01 | - -Response - -\[ - { - "date": "2024-02-01", - "industry": "Biotechnology", - "exchange": "NASDAQ", - "averageChange": 1.1479066960358322 - } -\] - -[Sector PE Snapshot API](https://site.financialmodelingprep.com/developer/docs/stable/sector-pe-snapshot) - -Retrieve the price-to-earnings (P/E) ratios for various sectors using the Sector P/E Snapshot API. Compare valuation levels across sectors to better understand market valuations. - -**Endpoint:** - -https://financialmodelingprep.com/stable/sector-pe-snapshot?*date*\=2024-02-01 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| date\* | string | 2024-02-01 | -| exchange | string | NASDAQ | -| sector | string | Energy | - -Response - -\[ - { - "date": "2024-02-01", - "sector": "Basic Materials", - "exchange": "NASDAQ", - "pe": 15.687711758428254 - } -\] - -[Industry PE Snapshot API](https://site.financialmodelingprep.com/developer/docs/stable/industry-pe-snapshot) - -View price-to-earnings (P/E) ratios for different industries using the Industry P/E Snapshot API. Analyze valuation levels across various industries to understand how each is priced relative to its earnings. - -**Endpoint:** - -https://financialmodelingprep.com/stable/industry-pe-snapshot?*date*\=2024-02-01 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| date\* | string | 2024-02-01 | -| exchange | string | NASDAQ | -| industry | string | Biotechnology | - -Response - -\[ - { - "date": "2024-02-01", - "industry": "Advertising Agencies", - "exchange": "NASDAQ", - "pe": 71.09601665201151 - } -\] - -[Historical Sector PE API](https://site.financialmodelingprep.com/developer/docs/stable/historical-sector-pe) - -Access historical price-to-earnings (P/E) ratios for various sectors using the Historical Sector P/E API. Analyze how sector valuations have evolved over time to understand long-term trends and market shifts. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-sector-pe?*sector*\=Energy - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | string | 2024-02-01 | -| exchange | string | NASDAQ | -| sector\* | string | Energy | -| to | string | 2024-03-01 | - -Response - -\[ - { - "date": "2024-02-01", - "sector": "Energy", - "exchange": "NASDAQ", - "pe": 14.411400922841464 - } -\] - -[Historical Industry PE API](https://site.financialmodelingprep.com/developer/docs/stable/historical-industry-pe) - -Access historical price-to-earnings (P/E) ratios by industry using the Historical Industry P/E API. Track valuation trends across various industries to understand how market sentiment and valuations have evolved over time. - -**Endpoint:** - -https://financialmodelingprep.com/stable/historical-industry-pe?*industry*\=Biotechnology - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| industry\* | string | Biotechnology | -| exchange | string | NASDAQ | -| from | string | 2024-02-01 | -| to | string | 2024-03-01 | - -Response -\[ - { - "date": "2024-02-01", - "industry": "Biotechnology", - "exchange": "NASDAQ", - "pe": 10.181600321811821 - } -\] - -[Biggest Stock Gainers API](https://site.financialmodelingprep.com/developer/docs/stable/biggest-gainers) - -Track the stocks with the largest price increases using the Top Stock Gainers API. Identify the companies that are leading the market with significant price surges, offering potential growth opportunities. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*biggest-gainers* - -Response - -\[ - { - "symbol": "LTRY", - "price": 0.5876, - "name": "Lottery.com Inc.", - "change": 0.2756, - "changesPercentage": 88.3333, - "exchange": "NASDAQ" - } -\] - -[Biggest Stock Losers API](https://site.financialmodelingprep.com/developer/docs/stable/biggest-losers) - -Access data on the stocks with the largest price drops using the Biggest Stock Losers API. Identify companies experiencing significant declines and track the stocks that are falling the fastest in the market. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*biggest-losers* - -Response - -\[ - { - "symbol": "IDEX", - "price": 0.0021, - "name": "Ideanomics, Inc.", - "change": \-0.0029, - "changesPercentage": \-58, - "exchange": "NASDAQ" - } -\] - -[Top Traded Stocks API](https://site.financialmodelingprep.com/developer/docs/stable/most-active) - -View the most actively traded stocks using the Top Traded Stocks API. Identify the companies experiencing the highest trading volumes in the market and track where the most trading activity is happening. - -**Endpoint:** - -https://financialmodelingprep.com/stable/*most-actives* - -Response - -\[ - { - "symbol": "LUCY", - "price": 5.03, - "name": "Innovative Eyewear, Inc.", - "change": \-0.01, - "changesPercentage": \-0.1984, - "exchange": "NASDAQ" - } -\] - -## **News** - -[FMP Articles API](https://site.financialmodelingprep.com/developer/docs/stable/fmp-articles) - -Access the latest articles from Financial Modeling Prep with the FMP Articles API. Get comprehensive updates including headlines, snippets, and publication URLs. - -**Endpoint:** - -https://financialmodelingprep.com/stable/fmp-articles?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "title": "Merck Shares Plunge 8% as Weak Guidance Overshadows Strong Revenue Growth", - "date": "2025-02-04 09:33:00", - "content": "\\Merck & Co (NYSE:MRK)\ saw its stock sink over 8% in pre-market today after delivering mixed fourth-quarter results, with earnings missing expectations, revenue exceeding forecasts, and full-year guidance coming in below analyst estimates.\\\n\For Q4, the pharmaceutical giant reported adjusted earnings per share (EPS) of $1.72, falling short of the $1.81 consensus estimate. However, revenue climbed 7% year-over-year to $1...", - "tickers": "NYSE:MRK", - "image": "https://cdn.financialmodelingprep.com/images/fmp-1738679603793.jpg", - "link": "https://financialmodelingprep.com/market-news/fmp-merck-shares-plunge-8-as-weak-guidance-overshadows-strong-revenue-growth", - "author": "Davit Kirakosyan", - "site": "Financial Modeling Prep" - } -\] - -[General News API](https://site.financialmodelingprep.com/developer/docs/stable/general-news) - -Access the latest general news articles from a variety of sources with the FMP General News API. Obtain headlines, snippets, and publication URLs for comprehensive news coverage. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/general-latest?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "symbol": null, - "publishedDate": "2025-02-03 23:51:37", - "publisher": "CNBC", - "title": "Asia tech stocks rise after Trump pauses tariffs on China and Mexico", - "image": "https://images.financialmodelingprep.com/news/asia-tech-stocks-rise-after-trump-pauses-tariffs-on-20250203.jpg", - "site": "cnbc.com", - "text": "Gains in Asian tech companies were broad-based, with stocks in Japan, South Korea and Hong Kong advancing. Semiconductor players Advantest and Lasertec led gains among Japanese tech stocks.", - "url": "https://www.cnbc.com/2025/02/04/asia-tech-stocks-rise-after-trump-pauses-tariffs-on-china-and-mexico.html" - } -\] - -[Press Releases API](https://site.financialmodelingprep.com/developer/docs/stable/press-releases) - -Access official company press releases with the FMP Press Releases API. Get real-time updates on corporate announcements, earnings reports, mergers, and more. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/press-releases-latest?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "symbol": "LNW", - "publishedDate": "2025-02-03 23:32:00", - "publisher": "PRNewsWire", - "title": "Rosen Law Firm Encourages Light & Wonder, Inc. Investors to Inquire About Securities Class Action Investigation \- LNW", - "image": "https://images.financialmodelingprep.com/news/rosen-law-firm-encourages-light-wonder-inc-investors-to-20250203.jpg", - "site": "prnewswire.com", - "text": "NEW YORK , Feb. 3, 2025 /PRNewswire/ \-- Why: Rosen Law Firm, a global investor rights law firm, continues to investigate potential securities claims on behalf of shareholders of Light & Wonder, Inc. (NASDAQ: LNW) resulting from allegations that Light & Wonder may have issued materially misleading business information to the investing public. So What: If you purchased Light & Wonder securities you may be entitled to compensation without payment of any out of pocket fees or costs through a contingency fee arrangement.", - "url": "https://www.prnewswire.com/news-releases/rosen-law-firm-encourages-light--wonder-inc-investors-to-inquire-about-securities-class-action-investigation--lnw-302366877.html" - } -\] - -[Stock News API](https://site.financialmodelingprep.com/developer/docs/stable/stock-news) - -Stay informed with the latest stock market news using the FMP Stock News Feed API. Access headlines, snippets, publication URLs, and ticker symbols for the most recent articles from a variety of sources. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/stock-latest?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "symbol": "INSG", - "publishedDate": "2025-02-03 23:53:40", - "publisher": "Seeking Alpha", - "title": "Q4 Earnings Release Looms For Inseego, But Don't Expect Miracles", - "image": "https://images.financialmodelingprep.com/news/q4-earnings-release-looms-for-inseego-but-dont-expect-20250203.jpg", - "site": "seekingalpha.com", - "text": "Inseego's Q3 beat was largely due to a one-time debt restructuring gain, not sustainable earnings growth, raising concerns about future performance. The sale of its telematics business for $52 million allows INSG to focus on North America, but it remains to be seen if this was wise. Despite improved margins and reduced debt, Inseego's revenue growth is insufficient, and its high stock price remains unjustifiable for new investors.", - "url": "https://seekingalpha.com/article/4754485-inseego-stock-q4-earnings-preview-monitor-growth-margins-closely" - } -\] - -[Crypto News API](https://site.financialmodelingprep.com/developer/docs/stable/crypto-news) - -Stay informed with the latest cryptocurrency news using the FMP Crypto News API. Access a curated list of articles from various sources, including headlines, snippets, and publication URLs. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/crypto-latest?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "symbol": "BTCUSD", - "publishedDate": "2025-02-03 23:32:19", - "publisher": "Coingape", - "title": "Crypto Prices Today Feb 4: BTC & Altcoins Recover Amid Pause On Trump's Tariffs", - "image": "https://images.financialmodelingprep.com/news/crypto-prices-today-feb-4-btc-altcoins-recover-amid-20250203.webp", - "site": "coingape.com", - "text": "Crypto prices today have shown signs of recovery as U.S. President Donald Trump's newly announced import tariffs on Canada and Mexico were paused for 30 days. Bitcoin (BTC) price regained its value, hitting a $102K high amid broader market recovery.", - "url": "https://coingape.com/crypto-prices-today-feb-4-btc-altcoins-recover-amid-pause-on-trumps-tariffs/" - } -\] - -[Forex News API](https://site.financialmodelingprep.com/developer/docs/stable/forex-news) - -Stay updated with the latest forex news articles from various sources using the FMP Forex News API. Access headlines, snippets, and publication URLs for comprehensive market insights. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/forex-latest?*page*\=0&*limit*\=20 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -\[ - { - "symbol": "XAUUSD", - "publishedDate": "2025-02-03 23:55:44", - "publisher": "FX Street", - "title": "United Arab Emirates Gold price today: Gold steadies, according to FXStreet data", - "image": "https://images.financialmodelingprep.com/news/united-arab-emirates-gold-price-today-gold-steadies-according-20250203.jpg", - "site": "fxstreet.com", - "text": "Gold prices remained broadly unchanged in United Arab Emirates on Tuesday, according to data compiled by FXStreet.", - "url": "https://www.fxstreet.com/news/united-arab-emirates-gold-price-today-gold-steadies-according-to-fxstreet-data-202502040455" - } -\] - -[Search Press Releases API](https://site.financialmodelingprep.com/developer/docs/stable/search-press-releases) - -Search for company press releases with the FMP Search Press Releases API. Find specific corporate announcements and updates by entering a stock symbol or company name. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/press-releases?*symbols*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbols\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - { - "symbol": "AAPL", - "publishedDate": "2025-01-30 16:30:00", - "publisher": "Business Wire", - "title": "Apple reports first quarter results", - "image": "https://images.financialmodelingprep.com/news/apple-reports-first-quarter-results-20250130.jpg", - "site": "businesswire.com", - "text": "CUPERTINO, Calif.--(BUSINESS WIRE)--Apple® today announced financial results for its fiscal 2025 first quarter ended December 28, 2024\. The Company posted quarterly revenue of $124.3 billion, up 4 percent year over year, and quarterly diluted earnings per share of $2.40, up 10 percent year over year. “Today Apple is reporting our best quarter ever, with revenue of $124.3 billion, up 4 percent from a year ago,” said Tim Cook, Apple's CEO. “We were thrilled to bring customers our best-ever lineup.", - "url": "https://www.businesswire.com/news/home/20250130261281/en/Apple-reports-first-quarter-results/" - } -\] - -[Search Stock News API](https://site.financialmodelingprep.com/developer/docs/stable/search-stock-news) - -Search for stock-related news using the FMP Search Stock News API. Find specific stock news by entering a ticker symbol or company name to track the latest developments. - -**Endpoint:** - -[https://financialmodelingprep.com/stable/news/stock?*symbols*\=AAPL](https://financialmodelingprep.com/stable/news/stock?symbols=AAPL) - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbols\* | string | AAPL | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - - { - - "symbol": "AAPL", - - "publishedDate": "2025-02-03 21:05:14", - - "publisher": "Zacks Investment Research", - - "title": "Apple & China Tariffs: A Closer Look", - - "image": "https://images.financialmodelingprep.com/news/apple-china-tariffs-a-closer-look-20250203.jpg", - - "site": "zacks.com", - - "text": "Tariffs have been the talk of the town over recent weeks, regularly overshadowing other important developments and causing volatility spikes.", - - "url": "https://www.zacks.com/stock/news/2408814/apple-china-tariffs-a-closer-look?cid=CS-STOCKNEWSAPI-FT-stocks\_in\_the\_news-2408814" - - } - -\] - -[Search Crypto News API](https://site.financialmodelingprep.com/developer/docs/stable/search-crypto-news) - -Search for cryptocurrency news using the FMP Search Crypto News API. Retrieve news related to specific coins or tokens by entering their name or symbol. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/crypto?*symbols*\=BTCUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbols\* | string | BTCUSD | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - - { - - "symbol": "BTCUSD", - - "publishedDate": "2025-02-03 23:32:19", - - "publisher": "Coingape", - - "title": "Crypto Prices Today Feb 4: BTC & Altcoins Recover Amid Pause On Trump's Tariffs", - - "image": "https://images.financialmodelingprep.com/news/crypto-prices-today-feb-4-btc-altcoins-recover-amid-20250203.webp", - - "site": "coingape.com", - - "text": "Crypto prices today have shown signs of recovery as U.S. President Donald Trump's newly announced import tariffs on Canada and Mexico were paused for 30 days. Bitcoin (BTC) price regained its value, hitting a $102K high amid broader market recovery.", - - "url": "https://coingape.com/crypto-prices-today-feb-4-btc-altcoins-recover-amid-pause-on-trumps-tariffs/" - - } - -\] - -[Search Forex News API](https://site.financialmodelingprep.com/developer/docs/stable/search-forex-news) - -Search for foreign exchange news using the FMP Search Forex News API. Find targeted news on specific currency pairs by entering their symbols for focused updates. - -**Endpoint:** - -https://financialmodelingprep.com/stable/news/forex?*symbols*\=EURUSD - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbols\* | string | EURUSD | -| from | date | 2025-04-24 | -| to | date | 2025-07-25 | -| page | number | 0 | -| limit | number | 20 | - -Response - -\[ - - { - - "symbol": "EURUSD", - - "publishedDate": "2025-02-03 18:43:01", - - "publisher": "FX Street", - - "title": "EUR/USD trims losses but still sheds weight", - - "image": "https://images.financialmodelingprep.com/news/eurusd-trims-losses-but-still-sheds-weight-20250203.jpg", - - "site": "fxstreet.com", - - "text": "EUR/USD dropped sharply following fresh tariff threats from US President Donald Trump, impacting the markets. However, significant declines in global risk markets eased as the Trump administration offered 30-day concessions on impending tariffs for Canada and Mexico.", - - "url": "https://www.fxstreet.com/news/eur-usd-trims-losses-but-still-sheds-weight-202502032343" - - } - -\] - -## **Technical Indicators** - -[Simple Moving Average API](https://site.financialmodelingprep.com/developer/docs/stable/simple-moving-average) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/sma?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "sma": 231.215 - - } - -\] - -[Exponential Moving Average API](https://site.financialmodelingprep.com/developer/docs/stable/exponential-moving-average) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/ema?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "ema": 232.8406611792779 - - } - -\] - -[Weighted Moving Average API](https://site.financialmodelingprep.com/developer/docs/stable/weighted-moving-average) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/wma?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "wma": 233.04745454545454 - - } - -\] - -[Double Exponential Moving Average API](https://site.financialmodelingprep.com/developer/docs/stable/double-exponential-moving-average) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/dema?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "dema": 232.10592058582725 - - } - -\] - -[Triple Exponential Moving Average API](https://site.financialmodelingprep.com/developer/docs/stable/triple-exponential-moving-average) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/tema?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "tema": 233.66383715917516 - - } - -\] - -[Relative Strength Index API](https://site.financialmodelingprep.com/developer/docs/stable/relative-strength-index) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/rsi?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "rsi": 47.64507340768903 - - } - -\] - -[Standard Deviation API](https://site.financialmodelingprep.com/developer/docs/stable/standard-deviation) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/standarddeviation?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "standardDeviation": 6.139182763202282 - - } - -\] - -[Williams API](https://site.financialmodelingprep.com/developer/docs/stable/williams) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/williams?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "williams": \-52.51824817518242 - - } - -\] - -[Average Directional Index API](https://site.financialmodelingprep.com/developer/docs/stable/average-directional-index) - -**Endpoint:** - -https://financialmodelingprep.com/stable/technical-indicators/adx?*symbol*\=AAPL&*periodLength*\=10&*timeframe*\=1day - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | -| periodLength\* | number | 10 | -| timeframe\* | string | 1min,5min,15min,30min,1hour,4hour,1day | -| from | date | 2025-04-24 | -| to | date | 2025-07-24 | - -Response - -\[ - - { - - "date": "2025-02-04 00:00:00", - - "open": 227.2, - - "high": 233.13, - - "low": 226.65, - - "close": 232.8, - - "volume": 44489128, - - "adx": 26.414065772772613 - - } - -\] - -## **Quote** - -[Stock Quote API](https://site.financialmodelingprep.com/developer/docs/stable/quote) - -Access real-time stock quotes with the FMP Stock Quote API. Get up-to-the-minute prices, changes, and volume data for individual stocks. - -**Endpoint:** - -https://financialmodelingprep.com/stable/quote?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "name": "Apple Inc.", - - "price": 232.8, - - "changePercentage": 2.1008, - - "change": 4.79, - - "volume": 44489128, - - "dayLow": 226.65, - - "dayHigh": 233.13, - - "yearHigh": 260.1, - - "yearLow": 164.08, - - "marketCap": 3500823120000, - - "priceAvg50": 240.2278, - - "priceAvg200": 219.98755, - - "exchange": "NASDAQ", - - "open": 227.2, - - "previousClose": 228.01, - - "timestamp": 1738702801 - - } - -\] - -[Aftermarket Trade API](https://site.financialmodelingprep.com/developer/docs/stable/aftermarket-trade) - -Track real-time trading activity occurring after regular market hours with the FMP Aftermarket Trade API. Access key details such as trade prices, sizes, and timestamps for trades executed during the post-market session. - -**Endpoint:** - -https://financialmodelingprep.com/stable/aftermarket-trade?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "price": 232.53, - - "tradeSize": 132, - - "timestamp": 1738715334311 - - } - -\] - -[Aftermarket Quote API](https://site.financialmodelingprep.com/developer/docs/stable/aftermarket-quote) - -Access real-time aftermarket quotes for stocks with the FMP Aftermarket Quote API. Track bid and ask prices, volume, and other relevant data outside of regular trading hours. - -**Endpoint:** - -https://financialmodelingprep.com/stable/aftermarket-quote?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "bidSize": 1, - - "bidPrice": 232.45, - - "askSize": 3, - - "askPrice": 232.64, - - "volume": 41647042, - - "timestamp": 1738715334311 - - } - -\] - -[Stock Price Change API](https://site.financialmodelingprep.com/developer/docs/stable/quote-change) - -Track stock price fluctuations in real-time with the FMP Stock Price Change API. Monitor percentage and value changes over various time periods, including daily, weekly, monthly, and long-term. - -**Endpoint:** - -https://financialmodelingprep.com/stable/stock-price-change?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "1D": 2.1008, - - "5D": \-2.45946, - - "1M": \-4.33925, - - "3M": 4.86014, - - "6M": 5.88556, - - "ytd": \-4.53147, - - "1Y": 24.04092, - - "3Y": 35.04264, - - "5Y": 192.05871, - - "10Y": 678.8558, - - "max": 181279.04168 - - } - -\] - -## **Senate** - -[Latest Senate Financial Disclosures API](https://site.financialmodelingprep.com/developer/docs/stable/senate-latest) - -Access the latest financial disclosures from U.S. Senate members with the FMP Latest Senate Financial Disclosures API. Track recent trades, asset ownership, and transaction details for enhanced transparency in government financial activities. - -**Endpoint:** - -https://financialmodelingprep.com/stable/senate-latest?*page*\=0&*limit*\=100 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 100 | - -Response - -\[ - - { - - "symbol": "LRN", - - "disclosureDate": "2025-01-31", - - "transactionDate": "2025-01-02", - - "firstName": "Markwayne", - - "lastName": "Mullin", - - "office": "Markwayne Mullin", - - "district": "OK", - - "owner": "Self", - - "assetDescription": "Stride Inc", - - "assetType": "Stock", - - "type": "Purchase", - - "amount": "$15,001 \- $50,000", - - "comment": "", - - "link": "https://efdsearch.senate.gov/search/view/ptr/446c7588-5f97-42c0-8983-3ca975b91793/" - - } - -\] - -[Latest House Financial Disclosures API](https://site.financialmodelingprep.com/developer/docs/stable/house-latest) - -Access real-time financial disclosures from U.S. House members with the FMP Latest House Financial Disclosures API. Track recent trades, asset ownership, and financial holdings for enhanced visibility into political figures' financial activities. - -**Endpoint:** - -https://financialmodelingprep.com/stable/house-latest?*page*\=0&*limit*\=100 - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| page | number | 0 | -| limit | number | 100 | - -Response - -\[ - - { - - "symbol": "$VIRTUALUSD", - - "disclosureDate": "2025-02-03", - - "transactionDate": "2025-01-03", - - "firstName": "Michael", - - "lastName": "Collins", - - "office": "Michael Collins", - - "district": "GA10", - - "owner": "", - - "assetDescription": "VIRTUALS PROTOCOL", - - "assetType": "Cryptocurrency", - - "type": "Purchase", - - "amount": "$1,001 \- $15,000", - - "capitalGainsOver200USD": "False", - - "comment": "", - - "link": "https://disclosures-clerk.house.gov/public\_disc/ptr-pdfs/2025/20026696.pdf" - - } - -\] - -[Senate Trading Activity API](https://site.financialmodelingprep.com/developer/docs/stable/senate-trading) - -Monitor the trading activity of US Senators with the FMP Senate Trading Activity API. Access detailed information on trades made by Senators, including trade dates, assets, amounts, and potential conflicts of interest. - -**Endpoint:** - -https://financialmodelingprep.com/stable/senate-trades?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "disclosureDate": "2025-01-08", - - "transactionDate": "2024-12-19", - - "firstName": "Sheldon", - - "lastName": "Whitehouse", - - "office": "Sheldon Whitehouse", - - "district": "RI", - - "owner": "Self", - - "assetDescription": "Apple Inc", - - "assetType": "Stock", - - "type": "Sale (Partial)", - - "amount": "$15,001 \- $50,000", - - "capitalGainsOver200USD": "False", - - "comment": "--", - - "link": "https://efdsearch.senate.gov/search/view/ptr/70c80513-d89a-4382-afa6-d80f6c1fcbf1/" - - } - -\] - -[Senate Trades By Name API](https://site.financialmodelingprep.com/developer/docs/stable/senate-trading-by-name) - -**Endpoint:** - -https://financialmodelingprep.com/stable/senate-trades-by-name?*name*\=Jerry - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| name\* | string | Jerry | - -Response - -\[ - - { - - "symbol": "BRK/B", - - "disclosureDate": "2025-01-18", - - "transactionDate": "2024-12-16", - - "firstName": "Jerry", - - "lastName": "Moran", - - "office": "Jerry Moran", - - "district": "KS", - - "owner": "Self", - - "assetDescription": "Berkshire Hathaway Inc", - - "assetType": "Stock", - - "type": "Purchase", - - "amount": "$1,001 \- $15,000", - - "capitalGainsOver200USD": "False", - - "comment": "", - - "link": "https://efdsearch.senate.gov/search/view/ptr/e37322e3-0829-4e3c-9faf-7a4a1a957e09/" - - } - -\] - -[U.S. House Trades API](https://site.financialmodelingprep.com/developer/docs/stable/house-trading) - -Track the financial trades made by U.S. House members and their families with the FMP U.S. House Trades API. Access real-time information on stock sales, purchases, and other investment activities to gain insight into their financial decisions. - -**Endpoint:** - -https://financialmodelingprep.com/stable/house-trades?*symbol*\=AAPL - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| symbol\* | string | AAPL | - -Response - -\[ - - { - - "symbol": "AAPL", - - "disclosureDate": "2025-01-20", - - "transactionDate": "2024-12-31", - - "firstName": "Nancy", - - "lastName": "Pelosi", - - "office": "Nancy Pelosi", - - "district": "CA11", - - "owner": "Spouse", - - "assetDescription": "Apple Inc", - - "assetType": "Stock", - - "type": "Sale", - - "amount": "$10,000,001 \- $25,000,000", - - "capitalGainsOver200USD": "False", - - "comment": "", - - "link": "https://disclosures-clerk.house.gov/public\_disc/ptr-pdfs/2025/20026590.pdf" - - } - -\] - -[House Trades By Name API](https://site.financialmodelingprep.com/developer/docs/stable/house-trading-by-name) - -**Endpoint:** - -https://financialmodelingprep.com/stable/house-trades-by-name?*name*\=James - -Parameters - -| Query Parameter | Type | Example | -| :---- | :---- | :---- | -| name\* | string | James | - -Response - -\[ - - { - - "symbol": "LUV", - - "disclosureDate": "2025-01-13", - - "transactionDate": "2024-12-31", - - "firstName": "James", - - "lastName": "Comer", - - "office": "James Comer", - - "district": "KY01", - - "owner": "", - - "assetDescription": "Southwest Airlines Co", - - "assetType": "Stock", - - "type": "Sale", - - "amount": "$1,001 \- $15,000", - - "capitalGainsOver200USD": "False", - - "comment": "", - - "link": "https://disclosures-clerk.house.gov/public\_disc/ptr-pdfs/2025/20018054.pdf" - - } - -\] - diff --git a/docs/superpowers/plans/2026-05-15-harvester-v2.md b/docs/superpowers/plans/2026-05-15-harvester-v2.md deleted file mode 100644 index aee943d..0000000 --- a/docs/superpowers/plans/2026-05-15-harvester-v2.md +++ /dev/null @@ -1,7052 +0,0 @@ -# Harvester v2 Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build the long-running `aiofmp harvest` CLI that proactively warms the local Parquet cache via category-specific harvest strategies (P1 discover-then-fill, P2 walk-back-until-checkpoint, P3 gap-fill, P4 snapshot overwrite) to stay under FMP's starter-plan 20 GB/month bandwidth cap. - -**Architecture:** A new `aiofmp/harvester/` package with a `HarvesterManager` that spawns one `asyncio.Task` per enabled category. Shared `FmpClient` + `CachedClient`, shared SQLite `StateStore` for checkpoints/bandwidth/symbol-catalogs, shared `BudgetTracker`. P3 categories use the existing `CachedClient` `DATE_RANGE` gap-fill. P1 uses `earnings_calendar` as a discovery probe. P2 needs a new `PAGE_WALK` `TemporalPattern` registered in the existing `CachedClient` registry. P4 uses a thin `SnapshotStore` helper writing single-row parquet under `("snapshot/", entity)`. - -**Tech Stack:** Python 3.10+, `aiohttp`, `asyncio`, `click`, `pytest` + `pytest-asyncio`, `pyarrow` (already in), `pyyaml` (new), built-in `sqlite3`. - -**Spec:** `docs/superpowers/specs/2026-05-15-harvester-v2-design.md` - -**File structure (new files this plan creates):** - -``` -aiofmp/harvester/ - __init__.py - cli.py - manager.py - config.py - state.py - budget.py - catalog.py - base.py - gap_fill_base.py (DRY base for P3 categories) - categories/ - __init__.py - chart_eod.py - chart_intraday.py - commodities_eod.py - commodities_intraday.py - forex_eod.py - forex_intraday.py - indexes_eod.py - indexes_intraday.py - news.py - economics.py - technical_indicators.py - statements.py - analyst_estimates.py - analyst_snapshots.py - insider_trades.py - form13f.py - dcf.py - -aiofmp/cachedclient/ - snapshot_store.py (new) - -tests/test_harvester/ - __init__.py - conftest.py - test_config.py - test_state.py - test_budget.py - test_catalog.py - test_base.py - test_manager.py - test_gap_fill_base.py - test_chart_eod.py - test_chart_intraday.py - test_commodities_eod.py - test_forex_eod.py - test_indexes_eod.py - test_news.py - test_economics.py - test_technical_indicators.py - test_statements.py - test_analyst_estimates.py - test_analyst_snapshots.py - test_insider_trades.py - test_form13f.py - test_dcf.py - test_cli.py - -tests/test_cachedclient/ - test_snapshot_store.py (new) - test_page_walk_registry.py (new) -``` - -**Files modified (existing):** - -- `pyproject.toml` — add `pyyaml` dependency -- `aiofmp/base.py` — `FMPBudgetError`, `on_response_size` callback, harvester-category `ContextVar` -- `aiofmp/cachedclient/gap.py` — no changes (already done) -- `aiofmp/cachedclient/registry.py` — `TemporalPattern.PAGE_WALK`, new `CacheableEndpoint` fields, register PAGE_WALK endpoints, register chart endpoints under `commodity`/`forex`/`indexes` categories -- `aiofmp/cachedclient/proxy.py` — read-only handling for `PAGE_WALK` keys -- `aiofmp/cli.py` — wire `harvest` and `harvest-status` subcommands -- `aiofmp/__init__.py` — re-export `FMPBudgetError` - -**Conventions used throughout this plan:** - -- Tests use `pytest` + `pytest-asyncio` with class-based grouping (matches existing `tests/test_cachedclient/test_gap.py`). -- Async fixtures use the `async def` form with `pytest_asyncio.fixture`. -- `asyncio_mode = "auto"` is set globally in `pyproject.toml` already, so test functions can be `async def` without decorators. -- File timestamps: UTC, ISO-8601 (`datetime.now(UTC).isoformat()`). -- Run commands assume `uv run` is the test runner. -- Each commit message uses Conventional Commits (`feat(harvester): ...`, `test(harvester): ...`, `chore: ...`) per the repo's release automation. - ---- - -## Phase 0 — Setup - -### Task 1: Add `pyyaml` dependency - -**Files:** -- Modify: `pyproject.toml` - -- [ ] **Step 1: Add `pyyaml` to dependencies** - -In `pyproject.toml` under `[project] dependencies = [...]`, add `"pyyaml>=6.0"`. The list should look like: - -```toml -dependencies = [ - "aiohttp>=3.12.15", - "fastmcp>=2.0.0", - "click>=8.0.0", - "pyarrow>=14.0.0", - "pyyaml>=6.0", - "pytest>=8.4.1", - "pytest-asyncio>=1.1.0", - "pytest-cov>=5.0.0", - "ruff>=0.13.2", -] -``` - -- [ ] **Step 2: Sync the lockfile** - -Run: `uv sync` -Expected: `uv` resolves and installs `pyyaml`. No error. - -- [ ] **Step 3: Verify import** - -Run: `uv run python -c "import yaml; print(yaml.__version__)"` -Expected: prints a 6.x version, no traceback. - -- [ ] **Step 4: Commit** - -```bash -git add pyproject.toml uv.lock -git commit -m "chore(harvester): add pyyaml dependency" -``` - ---- - -### Task 2: Skeleton package + test directory - -**Files:** -- Create: `aiofmp/harvester/__init__.py` -- Create: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/__init__.py` -- Create: `tests/test_harvester/conftest.py` - -- [ ] **Step 1: Create `aiofmp/harvester/__init__.py` with public exports placeholder** - -```python -"""Harvester — long-running CLI that proactively warms the local Parquet cache.""" - -__all__: list[str] = [] -``` - -- [ ] **Step 2: Create `aiofmp/harvester/categories/__init__.py`** - -```python -"""Category-specific harvester implementations.""" -``` - -- [ ] **Step 3: Create `tests/test_harvester/__init__.py` (empty)** - -```python -``` - -- [ ] **Step 4: Create `tests/test_harvester/conftest.py` with shared fixtures** - -```python -"""Shared fixtures for harvester tests.""" - -from __future__ import annotations - -import asyncio -from datetime import UTC, datetime -from pathlib import Path -from typing import Any - -import pytest_asyncio - - -@pytest_asyncio.fixture -async def tmp_state_dir(tmp_path: Path) -> Path: - """Temp directory that mimics the harvester's state_dir layout.""" - state_dir = tmp_path / "aiofmp_cache" - state_dir.mkdir() - return state_dir - - -def utc_now_iso() -> str: - """Return current UTC time as ISO-8601 string.""" - return datetime.now(UTC).isoformat() -``` - -- [ ] **Step 5: Verify test discovery** - -Run: `uv run pytest tests/test_harvester -q` -Expected: `no tests ran in ...` (no tests yet, no collection errors). - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/__init__.py aiofmp/harvester/categories/__init__.py tests/test_harvester/__init__.py tests/test_harvester/conftest.py -git commit -m "feat(harvester): scaffold harvester package and test dir" -``` - ---- - -## Phase 1 — Foundation - -### Task 3: Config dataclasses + interval parser - -**Files:** -- Create: `aiofmp/harvester/config.py` -- Create: `tests/test_harvester/test_config.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_config.py`: - -```python -"""Tests for the harvester config module.""" - -from __future__ import annotations - -import pytest - -from aiofmp.harvester.config import ( - BudgetConfig, - CategoryConfig, - DiscoveryConfig, - HarvestConfig, - RetryConfig, - parse_interval, -) - - -class TestParseInterval: - @pytest.mark.parametrize( - "spec,expected_seconds", - [ - ("30s", 30), - ("5m", 300), - ("30m", 1800), - ("1h", 3600), - ("4h", 14400), - ("24h", 86400), - ("1d", 86400), - ("7d", 604800), - ("30D", 2592000), # case-insensitive - ], - ) - def test_valid_specs(self, spec: str, expected_seconds: int) -> None: - assert parse_interval(spec) == expected_seconds - - @pytest.mark.parametrize("spec", ["", "30", "30x", "abc", "30 m", "-5m"]) - def test_invalid_specs(self, spec: str) -> None: - with pytest.raises(ValueError): - parse_interval(spec) - - -class TestHarvestConfig: - def test_defaults(self) -> None: - cfg = HarvestConfig() - assert cfg.state_dir == "~/.aiofmp/cache" - assert cfg.log_level == "INFO" - assert cfg.budget.monthly_soft_cap_gb == 18 - assert cfg.budget.monthly_hard_cap_gb == 20 - assert cfg.retry.on_429.backoff_seconds == [60, 120, 240, 480] - assert cfg.discovery.refresh_interval == "7d" - assert cfg.categories == {} - - def test_category_config_interval_seconds(self) -> None: - cc = CategoryConfig(enabled=True, interval="6h", extra={}) - assert cc.interval_seconds == 6 * 3600 - - def test_retry_defaults(self) -> None: - r = RetryConfig() - assert r.on_429.max_attempts == 4 - assert r.on_5xx.max_attempts == 3 - assert r.on_5xx.backoff_seconds == [10, 30, 60] - - def test_discovery_refresh_seconds(self) -> None: - d = DiscoveryConfig(refresh_interval="7d") - assert d.refresh_interval_seconds == 7 * 86400 - - def test_budget_defaults(self) -> None: - b = BudgetConfig() - assert b.soft_cap_behavior == "pause_until_next_month" -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_config.py -v` -Expected: ImportError — `aiofmp.harvester.config` not found. - -- [ ] **Step 3: Create `aiofmp/harvester/config.py`** - -```python -"""Harvester configuration dataclasses + YAML schema. - -Defines the structured representation of `harvester.yaml` and the -`parse_interval` helper that turns strings like ``"6h"`` into seconds. -""" - -from __future__ import annotations - -import re -from dataclasses import dataclass, field -from typing import Any - -_INTERVAL_RE = re.compile(r"^(\d+)([smhd])$", re.IGNORECASE) -_INTERVAL_UNIT_SECONDS: dict[str, int] = { - "s": 1, - "m": 60, - "h": 3600, - "d": 86400, -} - - -def parse_interval(spec: str) -> int: - """Parse an interval spec like ``"30s"``, ``"5m"``, ``"6h"``, ``"7d"`` into seconds. - - Raises ``ValueError`` for any other input (empty, missing unit, unknown unit, negative). - """ - if not isinstance(spec, str): - raise ValueError(f"interval must be a string, got {type(spec).__name__}") - match = _INTERVAL_RE.match(spec) - if not match: - raise ValueError(f"invalid interval spec: {spec!r}") - n = int(match.group(1)) - unit = match.group(2).lower() - if n <= 0: - raise ValueError(f"interval must be positive: {spec!r}") - return n * _INTERVAL_UNIT_SECONDS[unit] - - -@dataclass -class RetryPolicy: - backoff_seconds: list[int] - max_attempts: int - - -@dataclass -class RetryConfig: - on_429: RetryPolicy = field( - default_factory=lambda: RetryPolicy(backoff_seconds=[60, 120, 240, 480], max_attempts=4) - ) - on_5xx: RetryPolicy = field( - default_factory=lambda: RetryPolicy(backoff_seconds=[10, 30, 60], max_attempts=3) - ) - - -@dataclass -class BudgetConfig: - monthly_soft_cap_gb: int = 18 - monthly_hard_cap_gb: int = 20 - soft_cap_behavior: str = "pause_until_next_month" # | "warn_only" - - -@dataclass -class DiscoveryConfig: - refresh_interval: str = "7d" - - @property - def refresh_interval_seconds(self) -> int: - return parse_interval(self.refresh_interval) - - -@dataclass -class CategoryConfig: - """Generic per-category config; specific keys live in ``extra``. - - Common fields (``enabled``, ``interval``) are split out so the manager - can inspect them without knowing the category. All other YAML keys go - into ``extra`` and are interpreted by the concrete category harvester. - """ - - enabled: bool - interval: str - extra: dict[str, Any] - - @property - def interval_seconds(self) -> int: - return parse_interval(self.interval) - - -@dataclass -class HarvestConfig: - state_dir: str = "~/.aiofmp/cache" - log_level: str = "INFO" - shutdown_grace_seconds: int = 30 - budget: BudgetConfig = field(default_factory=BudgetConfig) - retry: RetryConfig = field(default_factory=RetryConfig) - discovery: DiscoveryConfig = field(default_factory=DiscoveryConfig) - categories: dict[str, CategoryConfig] = field(default_factory=dict) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_config.py -v` -Expected: all 12 tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/config.py tests/test_harvester/test_config.py -git commit -m "feat(harvester): config dataclasses and interval parser" -``` - ---- - -### Task 4: YAML loader for `HarvestConfig` - -**Files:** -- Modify: `aiofmp/harvester/config.py` -- Modify: `tests/test_harvester/test_config.py` - -- [ ] **Step 1: Append failing tests to `test_config.py`** - -Append to `tests/test_harvester/test_config.py`: - -```python -from textwrap import dedent - -from aiofmp.harvester.config import load_config_from_yaml - - -class TestLoadConfigFromYaml: - def test_minimal(self, tmp_path) -> None: - path = tmp_path / "h.yaml" - path.write_text("state_dir: /tmp/x\n") - cfg = load_config_from_yaml(path) - assert cfg.state_dir == "/tmp/x" - assert cfg.log_level == "INFO" - assert cfg.budget.monthly_soft_cap_gb == 18 - - def test_full(self, tmp_path) -> None: - path = tmp_path / "h.yaml" - path.write_text(dedent(""" - state_dir: /var/aiofmp - log_level: DEBUG - budget: - monthly_soft_cap_gb: 10 - monthly_hard_cap_gb: 12 - soft_cap_behavior: warn_only - retry: - on_429: - backoff_seconds: [30, 60] - max_attempts: 2 - on_5xx: - backoff_seconds: [5] - max_attempts: 1 - discovery: - refresh_interval: 3d - categories: - statements: - enabled: true - interval: 6h - periods: [annual, quarter] - initial_limit: 40 - chart_eod: - enabled: false - interval: 24h - """).strip() + "\n") - cfg = load_config_from_yaml(path) - assert cfg.state_dir == "/var/aiofmp" - assert cfg.log_level == "DEBUG" - assert cfg.budget.monthly_soft_cap_gb == 10 - assert cfg.budget.soft_cap_behavior == "warn_only" - assert cfg.retry.on_429.backoff_seconds == [30, 60] - assert cfg.retry.on_429.max_attempts == 2 - assert cfg.discovery.refresh_interval == "3d" - assert "statements" in cfg.categories - st = cfg.categories["statements"] - assert st.enabled is True - assert st.interval == "6h" - assert st.extra["periods"] == ["annual", "quarter"] - assert st.extra["initial_limit"] == 40 - assert cfg.categories["chart_eod"].enabled is False - - def test_missing_file(self, tmp_path) -> None: - import pytest - with pytest.raises(FileNotFoundError): - load_config_from_yaml(tmp_path / "nope.yaml") - - def test_bad_yaml(self, tmp_path) -> None: - import pytest - path = tmp_path / "bad.yaml" - path.write_text("state_dir: [unclosed\n") - with pytest.raises(ValueError, match="yaml"): - load_config_from_yaml(path) - - def test_category_missing_required(self, tmp_path) -> None: - import pytest - path = tmp_path / "h.yaml" - path.write_text(dedent(""" - categories: - statements: - enabled: true - # interval missing - """).strip() + "\n") - with pytest.raises(ValueError, match="interval"): - load_config_from_yaml(path) -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_config.py::TestLoadConfigFromYaml -v` -Expected: ImportError — `load_config_from_yaml` not defined. - -- [ ] **Step 3: Add `load_config_from_yaml` to `aiofmp/harvester/config.py`** - -Append at the bottom of `aiofmp/harvester/config.py`: - -```python -import os -from pathlib import Path - -import yaml - - -def _parse_retry_policy(raw: dict[str, Any] | None, default: RetryPolicy) -> RetryPolicy: - if raw is None: - return default - return RetryPolicy( - backoff_seconds=list(raw.get("backoff_seconds", default.backoff_seconds)), - max_attempts=int(raw.get("max_attempts", default.max_attempts)), - ) - - -def _parse_categories(raw: dict[str, Any] | None) -> dict[str, CategoryConfig]: - if not raw: - return {} - out: dict[str, CategoryConfig] = {} - for name, body in raw.items(): - if not isinstance(body, dict): - raise ValueError(f"category {name!r} must be a mapping, got {type(body).__name__}") - if "interval" not in body: - raise ValueError(f"category {name!r} is missing required field: interval") - if "enabled" not in body: - raise ValueError(f"category {name!r} is missing required field: enabled") - extra = {k: v for k, v in body.items() if k not in ("enabled", "interval")} - out[name] = CategoryConfig( - enabled=bool(body["enabled"]), - interval=str(body["interval"]), - extra=extra, - ) - # Validate interval format early - parse_interval(out[name].interval) - return out - - -def load_config_from_yaml(path: str | Path) -> HarvestConfig: - """Parse a YAML file into a HarvestConfig. Raises FileNotFoundError / ValueError on issues.""" - p = Path(os.path.expanduser(str(path))) - if not p.exists(): - raise FileNotFoundError(f"config file not found: {p}") - try: - with open(p, encoding="utf-8") as fh: - raw = yaml.safe_load(fh) or {} - except yaml.YAMLError as e: - raise ValueError(f"yaml parse error: {e}") from e - - defaults = HarvestConfig() - - budget_raw = raw.get("budget") or {} - budget = BudgetConfig( - monthly_soft_cap_gb=int(budget_raw.get("monthly_soft_cap_gb", defaults.budget.monthly_soft_cap_gb)), - monthly_hard_cap_gb=int(budget_raw.get("monthly_hard_cap_gb", defaults.budget.monthly_hard_cap_gb)), - soft_cap_behavior=str(budget_raw.get("soft_cap_behavior", defaults.budget.soft_cap_behavior)), - ) - - retry_raw = raw.get("retry") or {} - retry = RetryConfig( - on_429=_parse_retry_policy(retry_raw.get("on_429"), defaults.retry.on_429), - on_5xx=_parse_retry_policy(retry_raw.get("on_5xx"), defaults.retry.on_5xx), - ) - - discovery_raw = raw.get("discovery") or {} - discovery = DiscoveryConfig( - refresh_interval=str(discovery_raw.get("refresh_interval", defaults.discovery.refresh_interval)), - ) - # Validate discovery interval early - parse_interval(discovery.refresh_interval) - - return HarvestConfig( - state_dir=str(raw.get("state_dir", defaults.state_dir)), - log_level=str(raw.get("log_level", defaults.log_level)), - shutdown_grace_seconds=int(raw.get("shutdown_grace_seconds", defaults.shutdown_grace_seconds)), - budget=budget, - retry=retry, - discovery=discovery, - categories=_parse_categories(raw.get("categories")), - ) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_config.py -v` -Expected: all tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/config.py tests/test_harvester/test_config.py -git commit -m "feat(harvester): YAML loader for HarvestConfig" -``` - ---- - -### Task 5: StateStore (SQLite schema + CRUD) - -**Files:** -- Create: `aiofmp/harvester/state.py` -- Create: `tests/test_harvester/test_state.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_state.py`: - -```python -"""Tests for the harvester state store.""" - -from __future__ import annotations - -from datetime import UTC, datetime, timedelta -from pathlib import Path - -import pytest - -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -def store(tmp_path: Path) -> StateStore: - s = StateStore(tmp_path / "harvester.sqlite") - s.initialize() - return s - - -class TestSchema: - def test_initialize_idempotent(self, tmp_path: Path) -> None: - path = tmp_path / "harvester.sqlite" - StateStore(path).initialize() - StateStore(path).initialize() # must not error - assert path.exists() - - -class TestCategoryRuns: - def test_record_run_lifecycle(self, store: StateStore) -> None: - started = datetime.now(UTC) - store.record_run_start("statements", started) - store.record_run_finish( - "statements", - started, - status=RunStatus.OK, - items_attempted=10, - items_succeeded=10, - bytes_estimate=1024, - ) - latest = store.get_latest_run("statements") - assert latest is not None - assert latest.status == RunStatus.OK - assert latest.items_attempted == 10 - assert latest.items_succeeded == 10 - assert latest.bytes_estimate == 1024 - - def test_get_latest_run_none(self, store: StateStore) -> None: - assert store.get_latest_run("never_ran") is None - - -class TestCheckpoints: - def test_upsert_and_get(self, store: StateStore) -> None: - store.set_checkpoint("statements", "AAPL", "2024-12-31") - assert store.get_checkpoint("statements", "AAPL") == "2024-12-31" - store.set_checkpoint("statements", "AAPL", "2025-01-15") - assert store.get_checkpoint("statements", "AAPL") == "2025-01-15" - - def test_missing_checkpoint(self, store: StateStore) -> None: - assert store.get_checkpoint("statements", "MSFT") is None - - -class TestSymbolCatalog: - def test_upsert_and_list(self, store: StateStore) -> None: - store.upsert_symbol("financial_symbols", "AAPL", {"name": "Apple"}) - store.upsert_symbol("financial_symbols", "MSFT", {"name": "Microsoft"}) - symbols = store.list_symbols("financial_symbols") - assert set(symbols) == {"AAPL", "MSFT"} - - def test_replace_universe(self, store: StateStore) -> None: - store.upsert_symbol("commodities", "GCUSD", {}) - store.replace_universe("commodities", [("CLUSD", {}), ("SIUSD", {})]) - assert set(store.list_symbols("commodities")) == {"CLUSD", "SIUSD"} - - def test_refresh_timestamp(self, store: StateStore) -> None: - assert store.get_last_refresh("indexes") is None - ts = datetime.now(UTC) - store.set_last_refresh("indexes", ts) - got = store.get_last_refresh("indexes") - assert got is not None - assert abs((got - ts).total_seconds()) < 1 - - -class TestBandwidthLedger: - def test_add_and_get(self, store: StateStore) -> None: - store.add_bytes("2026-05", "statements", 1024) - store.add_bytes("2026-05", "statements", 2048) - store.add_bytes("2026-05", "news", 512) - store.add_bytes("2026-04", "statements", 9999) - assert store.get_month_bytes("2026-05") == 1024 + 2048 + 512 - assert store.get_category_month_bytes("2026-05", "statements") == 3072 - assert store.get_month_bytes("2026-04") == 9999 -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_state.py -v` -Expected: ImportError — `aiofmp.harvester.state` missing. - -- [ ] **Step 3: Create `aiofmp/harvester/state.py`** - -```python -"""SQLite-backed state store for the harvester. - -Holds: - - category_runs: per-cycle bookkeeping - - checkpoints: last-seen-date per (category, scope) - - symbol_catalog: cached symbol universes - - catalog_refresh: when each universe was last refreshed - - bandwidth_ledger: bytes consumed per (month, category) - -Schema migrations are linear and idempotent; tracked via PRAGMA user_version. -""" - -from __future__ import annotations - -import json -import sqlite3 -from dataclasses import dataclass -from datetime import UTC, datetime -from enum import Enum -from pathlib import Path -from typing import Any - -SCHEMA_VERSION = 1 - - -class RunStatus(str, Enum): - RUNNING = "running" - OK = "ok" - PARTIAL = "partial" - ERROR = "error" - PAUSED_FOR_BUDGET = "paused_for_budget" - - -@dataclass(frozen=True) -class CategoryRun: - category: str - started_at: str - finished_at: str | None - status: RunStatus - items_attempted: int - items_succeeded: int - bytes_estimate: int - error: str | None - - -class StateStore: - """Thin synchronous wrapper around a single SQLite file. - - All methods are synchronous. Callers running inside asyncio loops should - wrap hot paths with ``asyncio.to_thread`` if blocking becomes a concern. - In practice the harvester's state writes are sub-millisecond. - """ - - def __init__(self, path: Path | str) -> None: - self._path = Path(path) - - def initialize(self) -> None: - self._path.parent.mkdir(parents=True, exist_ok=True) - with self._connect() as conn: - cur = conn.cursor() - cur.execute("PRAGMA user_version") - current_version = cur.fetchone()[0] - if current_version < 1: - self._apply_v1(conn) - conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}") - conn.commit() - - def _connect(self) -> sqlite3.Connection: - conn = sqlite3.connect(self._path) - conn.row_factory = sqlite3.Row - conn.execute("PRAGMA foreign_keys = ON") - return conn - - def _apply_v1(self, conn: sqlite3.Connection) -> None: - conn.executescript( - """ - CREATE TABLE IF NOT EXISTS category_runs ( - category TEXT NOT NULL, - started_at TEXT NOT NULL, - finished_at TEXT, - status TEXT NOT NULL, - items_attempted INTEGER NOT NULL DEFAULT 0, - items_succeeded INTEGER NOT NULL DEFAULT 0, - bytes_estimate INTEGER NOT NULL DEFAULT 0, - error TEXT, - PRIMARY KEY (category, started_at) - ); - CREATE INDEX IF NOT EXISTS idx_category_runs_started - ON category_runs(category, started_at DESC); - - CREATE TABLE IF NOT EXISTS checkpoints ( - category TEXT NOT NULL, - scope TEXT NOT NULL, - checkpoint TEXT NOT NULL, - updated_at TEXT NOT NULL, - PRIMARY KEY (category, scope) - ); - - CREATE TABLE IF NOT EXISTS symbol_catalog ( - universe TEXT NOT NULL, - symbol TEXT NOT NULL, - payload_json TEXT, - discovered_at TEXT NOT NULL, - PRIMARY KEY (universe, symbol) - ); - - CREATE TABLE IF NOT EXISTS catalog_refresh ( - universe TEXT PRIMARY KEY, - last_refreshed_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS bandwidth_ledger ( - month TEXT NOT NULL, - category TEXT NOT NULL, - bytes INTEGER NOT NULL DEFAULT 0, - PRIMARY KEY (month, category) - ); - """ - ) - - # ----- category_runs ----- - - def record_run_start(self, category: str, started_at: datetime) -> None: - with self._connect() as conn: - conn.execute( - "INSERT INTO category_runs (category, started_at, status) VALUES (?, ?, ?)", - (category, started_at.isoformat(), RunStatus.RUNNING.value), - ) - conn.commit() - - def record_run_finish( - self, - category: str, - started_at: datetime, - *, - status: RunStatus, - items_attempted: int = 0, - items_succeeded: int = 0, - bytes_estimate: int = 0, - error: str | None = None, - ) -> None: - with self._connect() as conn: - conn.execute( - """ - UPDATE category_runs - SET finished_at = ?, - status = ?, - items_attempted = ?, - items_succeeded = ?, - bytes_estimate = ?, - error = ? - WHERE category = ? AND started_at = ? - """, - ( - datetime.now(UTC).isoformat(), - status.value, - items_attempted, - items_succeeded, - bytes_estimate, - error, - category, - started_at.isoformat(), - ), - ) - conn.commit() - - def get_latest_run(self, category: str) -> CategoryRun | None: - with self._connect() as conn: - row = conn.execute( - "SELECT * FROM category_runs WHERE category = ? ORDER BY started_at DESC LIMIT 1", - (category,), - ).fetchone() - if row is None: - return None - return CategoryRun( - category=row["category"], - started_at=row["started_at"], - finished_at=row["finished_at"], - status=RunStatus(row["status"]), - items_attempted=row["items_attempted"], - items_succeeded=row["items_succeeded"], - bytes_estimate=row["bytes_estimate"], - error=row["error"], - ) - - # ----- checkpoints ----- - - def set_checkpoint(self, category: str, scope: str, checkpoint: str) -> None: - with self._connect() as conn: - conn.execute( - """ - INSERT INTO checkpoints (category, scope, checkpoint, updated_at) - VALUES (?, ?, ?, ?) - ON CONFLICT(category, scope) DO UPDATE SET - checkpoint = excluded.checkpoint, - updated_at = excluded.updated_at - """, - (category, scope, checkpoint, datetime.now(UTC).isoformat()), - ) - conn.commit() - - def get_checkpoint(self, category: str, scope: str) -> str | None: - with self._connect() as conn: - row = conn.execute( - "SELECT checkpoint FROM checkpoints WHERE category = ? AND scope = ?", - (category, scope), - ).fetchone() - return row["checkpoint"] if row else None - - # ----- symbol_catalog ----- - - def upsert_symbol(self, universe: str, symbol: str, payload: dict[str, Any]) -> None: - with self._connect() as conn: - conn.execute( - """ - INSERT INTO symbol_catalog (universe, symbol, payload_json, discovered_at) - VALUES (?, ?, ?, ?) - ON CONFLICT(universe, symbol) DO UPDATE SET - payload_json = excluded.payload_json - """, - (universe, symbol, json.dumps(payload), datetime.now(UTC).isoformat()), - ) - conn.commit() - - def replace_universe(self, universe: str, symbols: list[tuple[str, dict[str, Any]]]) -> None: - """Atomically replace all symbols for a universe.""" - with self._connect() as conn: - conn.execute("DELETE FROM symbol_catalog WHERE universe = ?", (universe,)) - now_iso = datetime.now(UTC).isoformat() - conn.executemany( - """ - INSERT INTO symbol_catalog (universe, symbol, payload_json, discovered_at) - VALUES (?, ?, ?, ?) - """, - [(universe, sym, json.dumps(p), now_iso) for sym, p in symbols], - ) - conn.commit() - - def list_symbols(self, universe: str) -> list[str]: - with self._connect() as conn: - rows = conn.execute( - "SELECT symbol FROM symbol_catalog WHERE universe = ? ORDER BY symbol", - (universe,), - ).fetchall() - return [r["symbol"] for r in rows] - - def set_last_refresh(self, universe: str, when: datetime) -> None: - with self._connect() as conn: - conn.execute( - """ - INSERT INTO catalog_refresh (universe, last_refreshed_at) - VALUES (?, ?) - ON CONFLICT(universe) DO UPDATE SET last_refreshed_at = excluded.last_refreshed_at - """, - (universe, when.isoformat()), - ) - conn.commit() - - def get_last_refresh(self, universe: str) -> datetime | None: - with self._connect() as conn: - row = conn.execute( - "SELECT last_refreshed_at FROM catalog_refresh WHERE universe = ?", - (universe,), - ).fetchone() - if row is None: - return None - return datetime.fromisoformat(row["last_refreshed_at"]) - - # ----- bandwidth ledger ----- - - def add_bytes(self, month: str, category: str, n_bytes: int) -> None: - with self._connect() as conn: - conn.execute( - """ - INSERT INTO bandwidth_ledger (month, category, bytes) - VALUES (?, ?, ?) - ON CONFLICT(month, category) DO UPDATE SET bytes = bytes + excluded.bytes - """, - (month, category, n_bytes), - ) - conn.commit() - - def get_month_bytes(self, month: str) -> int: - with self._connect() as conn: - row = conn.execute( - "SELECT COALESCE(SUM(bytes), 0) AS total FROM bandwidth_ledger WHERE month = ?", - (month,), - ).fetchone() - return int(row["total"]) - - def get_category_month_bytes(self, month: str, category: str) -> int: - with self._connect() as conn: - row = conn.execute( - "SELECT bytes FROM bandwidth_ledger WHERE month = ? AND category = ?", - (month, category), - ).fetchone() - return int(row["bytes"]) if row else 0 -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_state.py -v` -Expected: all tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/state.py tests/test_harvester/test_state.py -git commit -m "feat(harvester): SQLite state store with schema migrations" -``` - ---- - -### Task 6: Bandwidth callback + `FMPBudgetError` in base client - -**Files:** -- Modify: `aiofmp/base.py` -- Modify: `aiofmp/__init__.py` -- Create: `tests/test_harvester/test_base_client_hooks.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_base_client_hooks.py`: - -```python -"""Tests for the FMPBaseClient bandwidth hook and FMPBudgetError.""" - -from __future__ import annotations - -import asyncio -import contextvars -from typing import Any -from unittest.mock import AsyncMock, MagicMock, patch - -import pytest - -from aiofmp.base import ( - FMPBaseClient, - FMPBudgetError, - FMPError, - current_harvest_category, -) - - -def test_fmp_budget_error_is_fmp_error_subclass() -> None: - assert issubclass(FMPBudgetError, FMPError) - - -def test_current_harvest_category_default_none() -> None: - assert current_harvest_category.get() is None - - -@pytest.mark.asyncio -async def test_on_response_size_callback_invoked() -> None: - """When a callback is set, the client invokes it with category + byte count.""" - captured: list[tuple[str | None, int]] = [] - - def callback(category: str | None, byte_count: int) -> None: - captured.append((category, byte_count)) - - client = FMPBaseClient(api_key="test_key") - client.on_response_size = callback - - body = b'{"data": "abcdefg"}' - fake_resp = MagicMock() - fake_resp.status = 200 - fake_resp.read = AsyncMock(return_value=body) - - token = current_harvest_category.set("statements") - try: - result = await client._handle_response(fake_resp) - finally: - current_harvest_category.reset(token) - - assert result == {"data": "abcdefg"} - assert len(captured) == 1 - assert captured[0][0] == "statements" - assert captured[0][1] == len(body) - - -@pytest.mark.asyncio -async def test_on_response_size_none_category_when_unset() -> None: - captured: list[tuple[str | None, int]] = [] - client = FMPBaseClient(api_key="test_key") - client.on_response_size = lambda cat, n: captured.append((cat, n)) - - fake_resp = MagicMock() - fake_resp.status = 200 - fake_resp.read = AsyncMock(return_value=b'{"x": 1}') - fake_resp.json = AsyncMock(return_value={"x": 1}) - - await client._handle_response(fake_resp) - - assert captured[0][0] is None # no ContextVar set - - -@pytest.mark.asyncio -async def test_callback_exceptions_are_logged_not_raised() -> None: - """A misbehaving callback must not break API calls.""" - client = FMPBaseClient(api_key="test_key") - - def bad_callback(category: str | None, byte_count: int) -> None: - raise RuntimeError("boom") - - client.on_response_size = bad_callback - - fake_resp = MagicMock() - fake_resp.status = 200 - fake_resp.read = AsyncMock(return_value=b"{}") - fake_resp.json = AsyncMock(return_value={}) - - # Should not raise - result = await client._handle_response(fake_resp) - assert result == {} -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_base_client_hooks.py -v` -Expected: ImportError — `FMPBudgetError` / `current_harvest_category` not defined. - -- [ ] **Step 3: Modify `aiofmp/base.py`** - -At the top of `aiofmp/base.py`, add imports and the new exports: - -```python -import asyncio -import contextvars -import logging -from collections.abc import Callable -from typing import Any -from urllib.parse import urlencode - -import aiohttp -``` - -Below the existing `FMPResponseError` class, add `FMPBudgetError`: - -```python -class FMPBudgetError(FMPError): - """Raised when the harvester's monthly hard bandwidth cap is exceeded.""" - - pass -``` - -Just before the `class FMPBaseClient:` line, add the `ContextVar`: - -```python -current_harvest_category: contextvars.ContextVar[str | None] = contextvars.ContextVar( - "current_harvest_category", default=None -) -``` - -In `FMPBaseClient.__init__`, add a public attribute: - -```python -# Append after self._request_semaphore line -self.on_response_size: Callable[[str | None, int], None] | None = None -``` - -Modify `_handle_response` to read the body via `read()`, compute byte count, invoke the callback, then JSON-decode: - -```python -async def _handle_response(self, response: aiohttp.ClientResponse) -> Any: - if response.status == 200: - try: - raw = await response.read() - byte_count = len(raw) - cb = self.on_response_size - if cb is not None: - try: - cb(current_harvest_category.get(), byte_count) - except Exception: - logger.exception("on_response_size callback raised; ignoring") - - import json # local import keeps top of file unchanged - data = json.loads(raw) if raw else None - - if isinstance(data, dict) and "Error Message" in data: - raise FMPResponseError(f"API Error: {data['Error Message']}") - return data - except FMPError: - raise - except Exception as e: - raise FMPError(f"Failed to parse response: {e}") from e - - elif response.status == 401: - raise FMPAuthenticationError("Invalid API key or authentication failed") - elif response.status == 429: - raise FMPRateLimitError("Rate limit exceeded") - elif response.status >= 500: - raise FMPError(f"Server error: {response.status}") - else: - raise FMPError(f"HTTP {response.status}: {response.reason}") -``` - -- [ ] **Step 4: Re-export from `aiofmp/__init__.py`** - -In `aiofmp/__init__.py`, update the imports from `.base`: - -```python -from .base import ( - FMPAuthenticationError, - FMPBaseClient, - FMPBudgetError, - FMPError, - FMPRateLimitError, - FMPResponseError, - current_harvest_category, -) -``` - -And add to `__all__`: - -```python -__all__ = [ - "FmpClient", - "FMPError", - "FMPAuthenticationError", - "FMPRateLimitError", - "FMPResponseError", - "FMPBudgetError", -] -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_base_client_hooks.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Re-run existing tests to confirm no regression** - -Run: `uv run pytest tests/test_cachedclient tests/test_search.py -q` -Expected: existing tests still pass. - -- [ ] **Step 7: Commit** - -```bash -git add aiofmp/base.py aiofmp/__init__.py tests/test_harvester/test_base_client_hooks.py -git commit -m "feat(harvester): on_response_size hook, current_harvest_category contextvar, FMPBudgetError" -``` - ---- - -### Task 7: BudgetTracker - -**Files:** -- Create: `aiofmp/harvester/budget.py` -- Create: `tests/test_harvester/test_budget.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_budget.py`: - -```python -"""Tests for the harvester budget tracker.""" - -from __future__ import annotations - -from datetime import UTC, datetime -from pathlib import Path - -import pytest - -from aiofmp.base import FMPBudgetError, current_harvest_category -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import BudgetConfig -from aiofmp.harvester.state import StateStore - -GB = 1024 ** 3 - - -@pytest.fixture -def store(tmp_path: Path) -> StateStore: - s = StateStore(tmp_path / "h.sqlite") - s.initialize() - return s - - -def test_record_bytes_attributes_to_current_category(store: StateStore) -> None: - tracker = BudgetTracker(store, BudgetConfig()) - token = current_harvest_category.set("statements") - try: - tracker.record_bytes(category=current_harvest_category.get(), byte_count=1024) - finally: - current_harvest_category.reset(token) - month = datetime.now(UTC).strftime("%Y-%m") - assert store.get_category_month_bytes(month, "statements") == 1024 - - -def test_record_bytes_none_category_attributed_to_user(store: StateStore) -> None: - tracker = BudgetTracker(store, BudgetConfig()) - tracker.record_bytes(category=None, byte_count=512) - month = datetime.now(UTC).strftime("%Y-%m") - assert store.get_category_month_bytes(month, "user") == 512 - - -def test_is_paused_below_soft_cap(store: StateStore) -> None: - tracker = BudgetTracker(store, BudgetConfig(monthly_soft_cap_gb=18)) - assert tracker.is_paused("statements") is False - - -def test_is_paused_at_soft_cap(store: StateStore) -> None: - cfg = BudgetConfig(monthly_soft_cap_gb=1, monthly_hard_cap_gb=2) - tracker = BudgetTracker(store, cfg) - tracker.record_bytes(category="statements", byte_count=1 * GB) - assert tracker.is_paused("statements") is True - - -def test_is_paused_warn_only(store: StateStore) -> None: - cfg = BudgetConfig(monthly_soft_cap_gb=1, monthly_hard_cap_gb=2, soft_cap_behavior="warn_only") - tracker = BudgetTracker(store, cfg) - tracker.record_bytes(category="statements", byte_count=1 * GB) - assert tracker.is_paused("statements") is False # warn only — never pauses - - -def test_check_hard_cap_raises(store: StateStore) -> None: - cfg = BudgetConfig(monthly_soft_cap_gb=1, monthly_hard_cap_gb=2) - tracker = BudgetTracker(store, cfg) - tracker.record_bytes(category="statements", byte_count=2 * GB) - with pytest.raises(FMPBudgetError, match="hard cap"): - tracker.check_hard_cap("statements") - - -def test_check_hard_cap_only_for_harvester_categories(store: StateStore) -> None: - cfg = BudgetConfig(monthly_soft_cap_gb=1, monthly_hard_cap_gb=2) - tracker = BudgetTracker(store, cfg) - tracker.record_bytes(category="user", byte_count=2 * GB) - # No exception even though hard cap is breached, because category is 'user' - tracker.check_hard_cap(None) - - -def test_month_key_format(store: StateStore) -> None: - tracker = BudgetTracker(store, BudgetConfig()) - key = tracker.current_month_key() - assert len(key) == 7 and key[4] == "-" -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_budget.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/budget.py`** - -```python -"""Bandwidth budget tracker for the harvester. - -Reads/writes the ``bandwidth_ledger`` table in the StateStore. Provides: - - ``record_bytes(category, byte_count)``: attribute bytes to a category - (or the synthetic ``"user"`` category when called from user code). - - ``is_paused(category)``: True if the monthly soft cap is reached and the - configured behavior is ``pause_until_next_month``. - - ``check_hard_cap(category)``: raise ``FMPBudgetError`` when the monthly - hard cap is reached, for harvester-initiated requests only. -""" - -from __future__ import annotations - -import logging -from datetime import UTC, datetime - -from aiofmp.base import FMPBudgetError -from aiofmp.harvester.config import BudgetConfig -from aiofmp.harvester.state import StateStore - -logger = logging.getLogger(__name__) - -GB = 1024 ** 3 - - -class BudgetTracker: - def __init__(self, store: StateStore, config: BudgetConfig) -> None: - self._store = store - self._config = config - - def current_month_key(self) -> str: - return datetime.now(UTC).strftime("%Y-%m") - - def record_bytes(self, category: str | None, byte_count: int) -> None: - attribution = category if category else "user" - self._store.add_bytes(self.current_month_key(), attribution, byte_count) - - def is_paused(self, category: str) -> bool: - if self._config.soft_cap_behavior != "pause_until_next_month": - return False - total = self._store.get_month_bytes(self.current_month_key()) - return total >= self._config.monthly_soft_cap_gb * GB - - def check_hard_cap(self, category: str | None) -> None: - if category is None or category == "user": - return - total = self._store.get_month_bytes(self.current_month_key()) - if total >= self._config.monthly_hard_cap_gb * GB: - raise FMPBudgetError( - f"monthly hard cap ({self._config.monthly_hard_cap_gb} GB) reached for " - f"category {category!r}; total this month: {total / GB:.2f} GB" - ) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_budget.py -v` -Expected: all 8 tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/budget.py tests/test_harvester/test_budget.py -git commit -m "feat(harvester): BudgetTracker (soft/hard cap, ledger attribution)" -``` - ---- - -### Task 8: SymbolCatalog - -**Files:** -- Create: `aiofmp/harvester/catalog.py` -- Create: `tests/test_harvester/test_catalog.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_catalog.py`: - -```python -"""Tests for the symbol catalog.""" - -from __future__ import annotations - -from datetime import UTC, datetime, timedelta -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.catalog import SymbolCatalog -from aiofmp.harvester.state import StateStore - - -@pytest.fixture -def store(tmp_path: Path) -> StateStore: - s = StateStore(tmp_path / "h.sqlite") - s.initialize() - return s - - -def make_fake_fmp() -> MagicMock: - fmp = MagicMock() - fmp.directory = MagicMock() - fmp.directory.financial_symbols = AsyncMock( - return_value=[{"symbol": "AAPL", "companyName": "Apple"}, {"symbol": "MSFT", "companyName": "MS"}] - ) - fmp.directory.actively_trading = AsyncMock( - return_value=[{"symbol": "AAPL"}, {"symbol": "MSFT"}, {"symbol": "TSLA"}] - ) - fmp.directory.etf_list = AsyncMock(return_value=[{"symbol": "SPY"}]) - - fmp.commodity = MagicMock() - fmp.commodity.commodities_list = AsyncMock(return_value=[{"symbol": "GCUSD"}, {"symbol": "CLUSD"}]) - - fmp.forex = MagicMock() - fmp.forex.forex_list = AsyncMock(return_value=[{"symbol": "EURUSD"}, {"symbol": "GBPUSD"}]) - - fmp.indexes = MagicMock() - fmp.indexes.index_list = AsyncMock(return_value=[{"symbol": "^GSPC"}, {"symbol": "^DJI"}]) - return fmp - - -class TestSymbolCatalog: - @pytest.mark.asyncio - async def test_first_call_discovers(self, store: StateStore) -> None: - fmp = make_fake_fmp() - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=86400) - result = await catalog.symbols("financial_symbols") - assert set(result) == {"AAPL", "MSFT"} - fmp.directory.financial_symbols.assert_awaited_once() - - @pytest.mark.asyncio - async def test_second_call_uses_cache(self, store: StateStore) -> None: - fmp = make_fake_fmp() - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=86400) - await catalog.symbols("financial_symbols") - await catalog.symbols("financial_symbols") - fmp.directory.financial_symbols.assert_awaited_once() - - @pytest.mark.asyncio - async def test_stale_cache_triggers_refresh(self, store: StateStore) -> None: - fmp = make_fake_fmp() - # 1 second refresh - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=1) - await catalog.symbols("financial_symbols") - # Forge an old refresh timestamp - store.set_last_refresh("financial_symbols", datetime.now(UTC) - timedelta(hours=2)) - await catalog.symbols("financial_symbols") - assert fmp.directory.financial_symbols.await_count == 2 - - @pytest.mark.asyncio - async def test_all_six_universes(self, store: StateStore) -> None: - fmp = make_fake_fmp() - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=86400) - assert set(await catalog.symbols("financial_symbols")) == {"AAPL", "MSFT"} - assert set(await catalog.symbols("actively_trading")) == {"AAPL", "MSFT", "TSLA"} - assert set(await catalog.symbols("commodities")) == {"GCUSD", "CLUSD"} - assert set(await catalog.symbols("forex_pairs")) == {"EURUSD", "GBPUSD"} - assert set(await catalog.symbols("indexes")) == {"^GSPC", "^DJI"} - assert set(await catalog.symbols("etf_list")) == {"SPY"} - - @pytest.mark.asyncio - async def test_unknown_universe_raises(self, store: StateStore) -> None: - fmp = make_fake_fmp() - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=86400) - with pytest.raises(ValueError, match="unknown universe"): - await catalog.symbols("crypto") - - @pytest.mark.asyncio - async def test_concurrent_refresh_is_single_shot(self, store: StateStore) -> None: - """Two parallel calls during discovery must result in one API call.""" - import asyncio - fmp = make_fake_fmp() - catalog = SymbolCatalog(store, fmp, refresh_interval_seconds=86400) - await asyncio.gather( - catalog.symbols("commodities"), - catalog.symbols("commodities"), - catalog.symbols("commodities"), - ) - fmp.commodity.commodities_list.assert_awaited_once() -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_catalog.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/catalog.py`** - -```python -"""Symbol-universe catalog backed by SQLite. - -Lazily discovers each universe on first access; refreshes when older -than ``refresh_interval_seconds``. All discovery calls run through the -shared FmpClient (not CachedClient — directory endpoints are not in the -cache registry today). -""" - -from __future__ import annotations - -import asyncio -import logging -from datetime import UTC, datetime -from typing import Any - -from aiofmp.harvester.state import StateStore - -logger = logging.getLogger(__name__) - -# Maps universe name -> (fmp_client.., ...) -_UNIVERSE_SPECS: dict[str, tuple[str, str]] = { - "financial_symbols": ("directory", "financial_symbols"), - "actively_trading": ("directory", "actively_trading"), - "etf_list": ("directory", "etf_list"), - "commodities": ("commodity", "commodities_list"), - "forex_pairs": ("forex", "forex_list"), - "indexes": ("indexes", "index_list"), -} - - -class SymbolCatalog: - def __init__(self, store: StateStore, fmp_client: Any, refresh_interval_seconds: int) -> None: - self._store = store - self._fmp = fmp_client - self._refresh_seconds = refresh_interval_seconds - self._locks: dict[str, asyncio.Lock] = {} - - def _lock(self, universe: str) -> asyncio.Lock: - if universe not in self._locks: - self._locks[universe] = asyncio.Lock() - return self._locks[universe] - - async def symbols(self, universe: str) -> list[str]: - if universe not in _UNIVERSE_SPECS: - raise ValueError(f"unknown universe: {universe!r}") - async with self._lock(universe): - if self._is_stale(universe): - await self._refresh(universe) - return self._store.list_symbols(universe) - - def _is_stale(self, universe: str) -> bool: - last = self._store.get_last_refresh(universe) - if last is None: - return True - age = (datetime.now(UTC) - last).total_seconds() - return age >= self._refresh_seconds - - async def _refresh(self, universe: str) -> None: - cat_attr, method_name = _UNIVERSE_SPECS[universe] - category = getattr(self._fmp, cat_attr) - method = getattr(category, method_name) - logger.info("Refreshing symbol universe %s via %s.%s", universe, cat_attr, method_name) - records = await method() - if not isinstance(records, list): - logger.warning("Universe %s returned non-list (%s); keeping prior cache", universe, type(records).__name__) - return - rows: list[tuple[str, dict[str, Any]]] = [] - for r in records: - if not isinstance(r, dict): - continue - symbol = r.get("symbol") - if not symbol: - continue - rows.append((str(symbol), r)) - self._store.replace_universe(universe, rows) - self._store.set_last_refresh(universe, datetime.now(UTC)) - logger.info("Universe %s refreshed: %d symbols", universe, len(rows)) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_catalog.py -v` -Expected: all 6 tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/catalog.py tests/test_harvester/test_catalog.py -git commit -m "feat(harvester): SymbolCatalog with lazy discovery and TTL refresh" -``` - ---- - -### Task 9: CategoryHarvester abstract base class - -**Files:** -- Create: `aiofmp/harvester/base.py` -- Create: `tests/test_harvester/test_base.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_base.py`: - -```python -"""Tests for the CategoryHarvester abstract base class.""" - -from __future__ import annotations - -import asyncio -from datetime import UTC, datetime -from pathlib import Path -from typing import Any - -import pytest - -from aiofmp.base import FMPBudgetError, FMPRateLimitError, current_harvest_category -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig, RetryPolicy -from aiofmp.harvester.state import RunStatus, StateStore - - -class FakeCategory(CategoryHarvester): - """Minimal concrete subclass for testing the base class.""" - - def __init__(self, name, config, store, budget, retry, *, behavior="ok"): - super().__init__(name, config, store, budget, retry) - self.behavior = behavior - self.run_count = 0 - self.items = 0 - - async def run_cycle(self) -> RunOutcome: - self.run_count += 1 - if self.behavior == "ok": - self.items += 3 - return RunOutcome(status=RunStatus.OK, items_attempted=3, items_succeeded=3) - if self.behavior == "raise": - raise RuntimeError("boom") - if self.behavior == "429_then_ok": - if self.run_count == 1: - raise FMPRateLimitError("rl") - return RunOutcome(status=RunStatus.OK, items_attempted=1, items_succeeded=1) - if self.behavior == "always_429": - raise FMPRateLimitError("rl") - if self.behavior == "budget": - raise FMPBudgetError("hard cap") - raise AssertionError(f"unknown behavior {self.behavior!r}") - - -@pytest.fixture -def store(tmp_path: Path) -> StateStore: - s = StateStore(tmp_path / "h.sqlite") - s.initialize() - return s - - -def _make(store: StateStore, behavior: str = "ok"): - config = CategoryConfig(enabled=True, interval="1s", extra={}) - budget = BudgetTracker(store, BudgetConfig()) - retry = RetryConfig( - on_429=RetryPolicy(backoff_seconds=[0, 0], max_attempts=2), - on_5xx=RetryPolicy(backoff_seconds=[0], max_attempts=1), - ) - return FakeCategory("fake", config, store, budget, retry, behavior=behavior) - - -class TestCategoryHarvester: - @pytest.mark.asyncio - async def test_ok_run_records_state(self, store: StateStore) -> None: - h = _make(store, "ok") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.OK - assert latest.items_attempted == 3 - assert latest.items_succeeded == 3 - assert h.run_count == 1 - - @pytest.mark.asyncio - async def test_exception_marks_error(self, store: StateStore) -> None: - h = _make(store, "raise") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.ERROR - assert latest.error is not None and "boom" in latest.error - - @pytest.mark.asyncio - async def test_429_then_ok_retries(self, store: StateStore) -> None: - h = _make(store, "429_then_ok") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.OK - assert h.run_count == 2 - - @pytest.mark.asyncio - async def test_always_429_marks_partial(self, store: StateStore) -> None: - h = _make(store, "always_429") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.PARTIAL - - @pytest.mark.asyncio - async def test_budget_error_marks_partial(self, store: StateStore) -> None: - h = _make(store, "budget") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.PARTIAL - - @pytest.mark.asyncio - async def test_sets_contextvar_during_cycle(self, store: StateStore) -> None: - seen: list[str | None] = [] - - class CaptureCategory(FakeCategory): - async def run_cycle(self) -> RunOutcome: - seen.append(current_harvest_category.get()) - return RunOutcome(status=RunStatus.OK) - - config = CategoryConfig(enabled=True, interval="1s", extra={}) - budget = BudgetTracker(store, BudgetConfig()) - retry = RetryConfig() - h = CaptureCategory("fake", config, store, budget, retry) - await h._run_once_and_record() - assert seen == ["fake"] - assert current_harvest_category.get() is None - - @pytest.mark.asyncio - async def test_paused_category_skips_run(self, store: StateStore) -> None: - cfg = CategoryConfig(enabled=True, interval="1s", extra={}) - budget_cfg = BudgetConfig(monthly_soft_cap_gb=0, monthly_hard_cap_gb=1) - budget = BudgetTracker(store, budget_cfg) - budget.record_bytes("fake", 1) - retry = RetryConfig() - h = FakeCategory("fake", cfg, store, budget, retry, behavior="ok") - await h._run_once_and_record() - latest = store.get_latest_run("fake") - assert latest is not None - assert latest.status == RunStatus.PAUSED_FOR_BUDGET - assert h.run_count == 0 - - @pytest.mark.asyncio - async def test_run_forever_respects_stop_event(self, store: StateStore) -> None: - h = _make(store, "ok") - stop_event = asyncio.Event() - task = asyncio.create_task(h.run_forever(stop_event)) - await asyncio.sleep(0.05) - stop_event.set() - await asyncio.wait_for(task, timeout=2.0) - assert h.run_count >= 1 -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_base.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/base.py`** - -```python -"""CategoryHarvester abstract base class. - -Concrete categories override ``run_cycle()`` and return a ``RunOutcome``. -The base class handles the run/sleep loop, retry/backoff for ``FMPRateLimitError``, -state-store bookkeeping, budget pause-before-run, and the ``current_harvest_category`` -ContextVar plumbing so byte-counting attributes every request to the right category. -""" - -from __future__ import annotations - -import abc -import asyncio -import logging -from dataclasses import dataclass -from datetime import UTC, datetime -from typing import Any - -from aiofmp.base import ( - FMPBudgetError, - FMPError, - FMPRateLimitError, - current_harvest_category, -) -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import CategoryConfig, RetryConfig, RetryPolicy -from aiofmp.harvester.state import RunStatus, StateStore - -logger = logging.getLogger(__name__) - - -@dataclass -class RunOutcome: - """Result of a single run_cycle() invocation.""" - - status: RunStatus - items_attempted: int = 0 - items_succeeded: int = 0 - bytes_estimate: int = 0 - error: str | None = None - - -class CategoryHarvester(abc.ABC): - """Abstract base for one harvester category.""" - - def __init__( - self, - name: str, - config: CategoryConfig, - state: StateStore, - budget: BudgetTracker, - retry: RetryConfig, - ) -> None: - self.name = name - self.config = config - self.state = state - self.budget = budget - self.retry = retry - - @abc.abstractmethod - async def run_cycle(self) -> RunOutcome: - """Execute one harvest cycle. Must be overridden by subclasses.""" - raise NotImplementedError - - async def run_forever(self, stop_event: asyncio.Event) -> None: - """Main loop: run a cycle each ``interval_seconds`` until stop_event fires.""" - interval = self.config.interval_seconds - while not stop_event.is_set(): - await self._run_once_and_record() - try: - await asyncio.wait_for(stop_event.wait(), timeout=interval) - except asyncio.TimeoutError: - pass - - async def _run_once_and_record(self) -> None: - """Run one cycle with state bookkeeping and budget checks.""" - started = datetime.now(UTC) - self.state.record_run_start(self.name, started) - - if self.budget.is_paused(self.name): - logger.warning("Category %s paused for budget; skipping cycle", self.name) - self.state.record_run_finish( - self.name, started, status=RunStatus.PAUSED_FOR_BUDGET - ) - return - - token = current_harvest_category.set(self.name) - try: - outcome = await self._run_cycle_with_retry() - except Exception as e: - logger.exception("Unhandled error in %s.run_cycle", self.name) - self.state.record_run_finish( - self.name, started, status=RunStatus.ERROR, error=f"{type(e).__name__}: {e}" - ) - return - finally: - current_harvest_category.reset(token) - - self.state.record_run_finish( - self.name, - started, - status=outcome.status, - items_attempted=outcome.items_attempted, - items_succeeded=outcome.items_succeeded, - bytes_estimate=outcome.bytes_estimate, - error=outcome.error, - ) - - async def _run_cycle_with_retry(self) -> RunOutcome: - """Wrap a single run_cycle() with the retry/backoff policy for 429.""" - policy: RetryPolicy = self.retry.on_429 - last_exc: BaseException | None = None - for attempt in range(policy.max_attempts): - try: - return await self.run_cycle() - except FMPRateLimitError as e: - last_exc = e - if attempt + 1 >= policy.max_attempts: - break - delay = policy.backoff_seconds[min(attempt, len(policy.backoff_seconds) - 1)] - logger.warning( - "%s hit 429 (attempt %d/%d); sleeping %ds", - self.name, attempt + 1, policy.max_attempts, delay, - ) - if delay > 0: - await asyncio.sleep(delay) - except FMPBudgetError as e: - logger.warning("%s hit budget hard cap: %s", self.name, e) - return RunOutcome(status=RunStatus.PARTIAL, error=str(e)) - return RunOutcome( - status=RunStatus.PARTIAL, - error=f"rate-limit retries exhausted: {last_exc}", - ) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_base.py -v` -Expected: all 8 tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/base.py tests/test_harvester/test_base.py -git commit -m "feat(harvester): CategoryHarvester ABC with retry/budget/state plumbing" -``` - ---- - -### Task 10: HarvesterManager (lifecycle + signal handling) - -**Files:** -- Create: `aiofmp/harvester/manager.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_manager.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_manager.py`: - -```python -"""Tests for HarvesterManager.""" - -from __future__ import annotations - -import asyncio -from pathlib import Path -from unittest.mock import MagicMock - -import pytest - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.config import ( - CategoryConfig, - HarvestConfig, -) -from aiofmp.harvester.manager import HarvesterManager -from aiofmp.harvester.state import RunStatus - - -class CountingCategory(CategoryHarvester): - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self.cycle_count = 0 - - async def run_cycle(self) -> RunOutcome: - self.cycle_count += 1 - return RunOutcome(status=RunStatus.OK, items_attempted=1, items_succeeded=1) - - -class StuckCategory(CategoryHarvester): - async def run_cycle(self) -> RunOutcome: - await asyncio.sleep(60) - return RunOutcome(status=RunStatus.OK) - - -@pytest.fixture -def cfg(tmp_path: Path) -> HarvestConfig: - return HarvestConfig( - state_dir=str(tmp_path), - shutdown_grace_seconds=2, - categories={ - "alpha": CategoryConfig(enabled=True, interval="1s", extra={}), - "beta": CategoryConfig(enabled=True, interval="1s", extra={}), - "gamma": CategoryConfig(enabled=False, interval="1s", extra={}), - }, - ) - - -def _patch_builder(mgr: HarvesterManager, cls: type[CategoryHarvester]) -> dict: - instances: dict[str, CategoryHarvester] = {} - - def build(name, c): - if name not in ("alpha", "beta"): - raise KeyError(name) - inst = cls(name, c, mgr.state, mgr.budget, mgr.config.retry) - instances[name] = inst - return inst - - mgr._build_category = build - return instances - - -class TestHarvesterManager: - @pytest.mark.asyncio - async def test_spawns_only_enabled(self, cfg: HarvestConfig) -> None: - mgr = HarvesterManager(cfg, fmp_client=MagicMock(), cached_client=MagicMock()) - _patch_builder(mgr, CountingCategory) - await mgr.start() - task = asyncio.create_task(mgr.run_until_stopped()) - await asyncio.sleep(0.05) - mgr.request_stop() - await asyncio.wait_for(task, timeout=3.0) - await mgr.close() - assert set(mgr._categories.keys()) == {"alpha", "beta"} - - @pytest.mark.asyncio - async def test_categories_run_concurrently(self, cfg: HarvestConfig) -> None: - mgr = HarvesterManager(cfg, fmp_client=MagicMock(), cached_client=MagicMock()) - instances = _patch_builder(mgr, CountingCategory) - await mgr.start() - task = asyncio.create_task(mgr.run_until_stopped()) - await asyncio.sleep(0.1) - mgr.request_stop() - await asyncio.wait_for(task, timeout=3.0) - await mgr.close() - assert instances["alpha"].cycle_count >= 1 - assert instances["beta"].cycle_count >= 1 - - @pytest.mark.asyncio - async def test_stop_event_finishes_cleanly(self, cfg: HarvestConfig) -> None: - mgr = HarvesterManager(cfg, fmp_client=MagicMock(), cached_client=MagicMock()) - _patch_builder(mgr, CountingCategory) - await mgr.start() - task = asyncio.create_task(mgr.run_until_stopped()) - mgr.request_stop() - await asyncio.wait_for(task, timeout=cfg.shutdown_grace_seconds + 2) - await mgr.close() - - @pytest.mark.asyncio - async def test_grace_timeout_cancels(self, cfg: HarvestConfig) -> None: - cfg.shutdown_grace_seconds = 1 - mgr = HarvesterManager(cfg, fmp_client=MagicMock(), cached_client=MagicMock()) - _patch_builder(mgr, StuckCategory) - await mgr.start() - task = asyncio.create_task(mgr.run_until_stopped()) - await asyncio.sleep(0.05) - mgr.request_stop() - await asyncio.wait_for(task, timeout=4.0) - await mgr.close() - - @pytest.mark.asyncio - async def test_unknown_category_in_config_skipped(self, cfg: HarvestConfig) -> None: - cfg.categories["nonexistent"] = CategoryConfig(enabled=True, interval="1s", extra={}) - mgr = HarvesterManager(cfg, fmp_client=MagicMock(), cached_client=MagicMock()) - _patch_builder(mgr, CountingCategory) - await mgr.start() - assert "nonexistent" not in mgr._categories - await mgr.close() -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_manager.py -v` -Expected: ImportError. - -- [ ] **Step 3: Replace `aiofmp/harvester/categories/__init__.py`** - -Overwrite with a registry binding: - -```python -"""Category-specific harvester implementations. - -Concrete category modules call ``register_category(name, factory)`` at import -time. ``build_category(name, cfg, manager)`` then dispatches via the registry. -""" - -from __future__ import annotations - -from collections.abc import Callable -from typing import TYPE_CHECKING - -if TYPE_CHECKING: - from aiofmp.harvester.base import CategoryHarvester - from aiofmp.harvester.config import CategoryConfig - from aiofmp.harvester.manager import HarvesterManager - -CategoryFactory = Callable[..., "CategoryHarvester"] - -_REGISTRY: dict[str, CategoryFactory] = {} - - -def register_category(name: str, factory: CategoryFactory) -> None: - _REGISTRY[name] = factory - - -def build_category(name: str, cfg: "CategoryConfig", manager: "HarvesterManager") -> "CategoryHarvester": - if name not in _REGISTRY: - raise KeyError(name) - return _REGISTRY[name](cfg, manager) - - -def registered_names() -> list[str]: - return sorted(_REGISTRY.keys()) -``` - -- [ ] **Step 4: Create `aiofmp/harvester/manager.py`** - -```python -"""HarvesterManager — orchestrates category tasks and lifecycle.""" - -from __future__ import annotations - -import asyncio -import logging -import os -import signal -from pathlib import Path -from typing import Any - -from aiofmp.harvester.base import CategoryHarvester -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.catalog import SymbolCatalog -from aiofmp.harvester.config import CategoryConfig, HarvestConfig -from aiofmp.harvester.state import StateStore - -logger = logging.getLogger(__name__) - - -class HarvesterManager: - """Loads enabled categories, runs them as concurrent asyncio tasks until stopped.""" - - def __init__(self, config: HarvestConfig, fmp_client: Any, cached_client: Any) -> None: - self.config = config - self.fmp_client = fmp_client - self.cached_client = cached_client - - state_dir = Path(os.path.expanduser(config.state_dir)) - state_dir.mkdir(parents=True, exist_ok=True) - self.state = StateStore(state_dir / "harvester.sqlite") - self.state.initialize() - - self.budget = BudgetTracker(self.state, config.budget) - self.catalog = SymbolCatalog( - self.state, fmp_client, config.discovery.refresh_interval_seconds - ) - - self._stop_event = asyncio.Event() - self._categories: dict[str, CategoryHarvester] = {} - self._tasks: list[asyncio.Task[None]] = [] - - async def start(self) -> None: - """Construct enabled category harvesters. Does not start tasks.""" - for name, cat_cfg in self.config.categories.items(): - if not cat_cfg.enabled: - logger.info("Category %s is disabled; skipping", name) - continue - try: - self._categories[name] = self._build_category(name, cat_cfg) - except KeyError: - logger.warning("Unknown category in config: %s (skipping)", name) - - def _build_category(self, name: str, cat_cfg: CategoryConfig) -> CategoryHarvester: - from aiofmp.harvester.categories import build_category - return build_category(name, cat_cfg, self) - - def request_stop(self) -> None: - self._stop_event.set() - - def install_signal_handlers(self) -> None: - loop = asyncio.get_event_loop() - try: - loop.add_signal_handler(signal.SIGINT, self.request_stop) - loop.add_signal_handler(signal.SIGTERM, self.request_stop) - except NotImplementedError: - signal.signal(signal.SIGINT, lambda *_: self.request_stop()) - - async def run_until_stopped(self) -> None: - if not self._categories: - logger.warning("No enabled categories; manager will idle until stop_event.") - await self._stop_event.wait() - return - - for name, cat in self._categories.items(): - self._tasks.append( - asyncio.create_task(cat.run_forever(self._stop_event), name=f"harvester:{name}") - ) - await self._stop_event.wait() - - logger.info("Stop requested; waiting for %d categor(y/ies) (grace %ds)", - len(self._tasks), self.config.shutdown_grace_seconds) - done, pending = await asyncio.wait( - self._tasks, timeout=self.config.shutdown_grace_seconds - ) - if pending: - logger.warning("%d task(s) did not stop within grace; cancelling", len(pending)) - for t in pending: - t.cancel() - await asyncio.gather(*pending, return_exceptions=True) - - async def close(self) -> None: - return None -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_manager.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/manager.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_manager.py -git commit -m "feat(harvester): HarvesterManager lifecycle and signal handling" -``` - ---- - -## Phase 2 — CachedClient extensions - -### Task 11: `TemporalPattern.PAGE_WALK` + supporting registry fields - -**Files:** -- Modify: `aiofmp/cachedclient/registry.py` -- Modify: `aiofmp/cachedclient/proxy.py` -- Create: `tests/test_cachedclient/test_page_walk_registry.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_cachedclient/test_page_walk_registry.py`: - -```python -"""Tests for the PAGE_WALK pattern in the cached client registry.""" - -from __future__ import annotations - -from datetime import date -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest -import pytest_asyncio - -from aiofmp.cachedclient.proxy import CachedCategoryProxy -from aiofmp.cachedclient.registry import ( - CacheableEndpoint, - EndpointRegistry, - TemporalPattern, - build_default_registry, -) -from aiofmp.cachedclient.storage.parquet import ParquetStorage - - -def test_temporal_pattern_page_walk_exists() -> None: - assert TemporalPattern.PAGE_WALK.value == "page_walk" - - -def test_page_walk_endpoint_fields() -> None: - ep = CacheableEndpoint( - category="analyst", - method="financial_estimates", - api_endpoint="analyst-estimates", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=["symbol"], - extra_key_args=["period"], - page_param="page", - default_page_size=100, - walk_date_field="date", - call_params=["symbol", "period", "page", "limit"], - ) - assert ep.page_param == "page" - assert ep.default_page_size == 100 - assert ep.walk_date_field == "date" - - -def test_default_registry_includes_page_walk_endpoints() -> None: - reg = build_default_registry() - # The harvester adds these in a later task; we only require the field exists. - # No PAGE_WALK endpoints are pre-registered by build_default_registry in v1. - assert reg.get("chart", "historical_price_full") is not None - - -@pytest_asyncio.fixture -async def storage(tmp_path: Path) -> ParquetStorage: - s = ParquetStorage(tmp_path) - await s.initialize() - return s - - -class TestPageWalkProxyReadOnly: - @pytest.mark.asyncio - async def test_user_call_returns_stored_records(self, storage: ParquetStorage) -> None: - """When a user calls a PAGE_WALK-registered method, the proxy returns stored records - without contacting the upstream API.""" - reg = EndpointRegistry() - reg.register( - CacheableEndpoint( - category="analyst", - method="financial_estimates", - api_endpoint="analyst-estimates", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=["symbol"], - extra_key_args=["period"], - page_param="page", - walk_date_field="date", - call_params=["symbol", "period", "page", "limit"], - ) - ) - - # Pre-seed storage as if the harvester had written it - await storage.write( - ("analyst-estimates", "AAPL", "annual"), - [ - {"symbol": "AAPL", "date": "2025-12-31", "revenueAvg": 500}, - {"symbol": "AAPL", "date": "2024-12-31", "revenueAvg": 400}, - {"symbol": "AAPL", "date": "2023-12-31", "revenueAvg": 300}, - ], - date_field="date", - ) - - real_category = MagicMock() - # If the proxy mistakenly calls upstream, this will be invoked: - real_category.financial_estimates = AsyncMock(side_effect=AssertionError("upstream called")) - - proxy = CachedCategoryProxy(real_category, "analyst", storage, reg) - records = await proxy.financial_estimates(symbol="AAPL", period="annual", page=0, limit=2) - - # Returns the requested limit slice (newest first by date convention) - assert len(records) == 2 - assert records[0]["date"] == "2025-12-31" - assert records[1]["date"] == "2024-12-31" - - @pytest.mark.asyncio - async def test_user_call_returns_empty_when_no_storage(self, storage: ParquetStorage) -> None: - reg = EndpointRegistry() - reg.register( - CacheableEndpoint( - category="analyst", - method="financial_estimates", - api_endpoint="analyst-estimates", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=["symbol"], - extra_key_args=["period"], - page_param="page", - walk_date_field="date", - call_params=["symbol", "period", "page", "limit"], - ) - ) - real_category = MagicMock() - real_category.financial_estimates = AsyncMock(side_effect=AssertionError("upstream called")) - proxy = CachedCategoryProxy(real_category, "analyst", storage, reg) - records = await proxy.financial_estimates(symbol="MSFT", period="annual") - assert records == [] -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_cachedclient/test_page_walk_registry.py -v` -Expected: failures — `TemporalPattern.PAGE_WALK` doesn't exist, `page_param` / `walk_date_field` / `default_page_size` fields missing. - -- [ ] **Step 3: Add `PAGE_WALK` to `TemporalPattern`** - -In `aiofmp/cachedclient/registry.py`, extend the enum: - -```python -class TemporalPattern(str, Enum): - """How an endpoint's time-series data is parameterized.""" - - DATE_RANGE = "date_range" # Pattern A: from/to date params - PERIOD_BASED = "period_based" # Pattern B: limit/period params - PAGE_WALK = "page_walk" # Harvester-only writes; user reads are passthrough-to-storage -``` - -- [ ] **Step 4: Extend `CacheableEndpoint` with the three new fields** - -In `aiofmp/cachedclient/registry.py`, add to the dataclass: - -```python -@dataclass(frozen=True) -class CacheableEndpoint: - # ... existing fields ... - # PAGE_WALK fields (only used when pattern == PAGE_WALK) - page_param: str = "page" - default_page_size: int = 100 - walk_date_field: str = "date" -``` - -(These have sensible defaults so existing DATE_RANGE / PERIOD_BASED registrations don't need changes.) - -- [ ] **Step 5: Add PAGE_WALK handling to the proxy** - -In `aiofmp/cachedclient/proxy.py`, extend `_cached_fetch` to handle the new pattern. Modify the existing `if/elif` ladder: - -```python -async def _cached_fetch( - self, - meta: CacheableEndpoint, - real_method: Callable[..., Any], - args: tuple[Any, ...], - kwargs: dict[str, Any], -) -> list[dict[str, Any]]: - bound_args = _bind_args(meta, args, kwargs) - storage_key = _derive_storage_key(meta, bound_args) - - async with self._get_lock(storage_key): - if meta.pattern == TemporalPattern.DATE_RANGE: - return await self._fetch_date_range(meta, real_method, bound_args, storage_key) - elif meta.pattern == TemporalPattern.PERIOD_BASED: - return await self._fetch_period_based(meta, real_method, args, kwargs, bound_args, storage_key) - elif meta.pattern == TemporalPattern.PAGE_WALK: - return await self._read_page_walk(meta, bound_args, storage_key) - else: - return await real_method(*args, **kwargs) - - -async def _read_page_walk( - self, - meta: CacheableEndpoint, - bound_args: dict[str, Any], - storage_key: tuple[str, ...], -) -> list[dict[str, Any]]: - """Read-only path for PAGE_WALK-registered endpoints. - - PAGE_WALK keys are written by the harvester only. User calls return whatever - is stored, sorted by ``walk_date_field`` descending, sliced by ``page``/``limit``. - """ - records = await self._storage.read(storage_key) - if not records: - return [] - field = meta.walk_date_field - records.sort(key=lambda r: str(r.get(field, "")), reverse=True) - page = bound_args.get(meta.page_param, 0) or 0 - limit_val = bound_args.get(meta.limit_param) if meta.limit_param else None - if limit_val is None: - limit_val = meta.default_page_size - start = page * limit_val - end = start + limit_val - return records[start:end] -``` - -- [ ] **Step 6: Run tests to confirm pass** - -Run: `uv run pytest tests/test_cachedclient/test_page_walk_registry.py -v` -Expected: all tests pass. - -- [ ] **Step 7: Run the full existing cached-client test suite to confirm no regression** - -Run: `uv run pytest tests/test_cachedclient -q` -Expected: every previously passing test still passes. - -- [ ] **Step 8: Commit** - -```bash -git add aiofmp/cachedclient/registry.py aiofmp/cachedclient/proxy.py tests/test_cachedclient/test_page_walk_registry.py -git commit -m "feat(cachedclient): add PAGE_WALK temporal pattern (read-only for users)" -``` - ---- - -### Task 12: SnapshotStore for P4 snapshot-overwrite pattern - -**Files:** -- Create: `aiofmp/cachedclient/snapshot_store.py` -- Modify: `aiofmp/cachedclient/__init__.py` -- Create: `tests/test_cachedclient/test_snapshot_store.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_cachedclient/test_snapshot_store.py`: - -```python -"""Tests for SnapshotStore (P4 snapshot-overwrite pattern).""" - -from __future__ import annotations - -from pathlib import Path - -import pytest -import pytest_asyncio - -from aiofmp.cachedclient.snapshot_store import SnapshotStore -from aiofmp.cachedclient.storage.parquet import ParquetStorage - - -@pytest_asyncio.fixture -async def storage(tmp_path: Path) -> ParquetStorage: - s = ParquetStorage(tmp_path) - await s.initialize() - return s - - -class TestSnapshotStore: - @pytest.mark.asyncio - async def test_write_and_read_single_row(self, storage: ParquetStorage) -> None: - ss = SnapshotStore(storage) - await ss.write("discounted-cash-flow", "AAPL", {"symbol": "AAPL", "dcf": 147.27, "Stock Price": 231.79}) - row = await ss.read("discounted-cash-flow", "AAPL") - assert row is not None - assert row["symbol"] == "AAPL" - assert row["dcf"] == 147.27 - - @pytest.mark.asyncio - async def test_write_overwrites_previous(self, storage: ParquetStorage) -> None: - ss = SnapshotStore(storage) - await ss.write("ratings-snapshot", "AAPL", {"symbol": "AAPL", "rating": "A-", "score": 4}) - await ss.write("ratings-snapshot", "AAPL", {"symbol": "AAPL", "rating": "A", "score": 5}) - row = await ss.read("ratings-snapshot", "AAPL") - assert row is not None - assert row["rating"] == "A" - assert row["score"] == 5 - - @pytest.mark.asyncio - async def test_read_missing_returns_none(self, storage: ParquetStorage) -> None: - ss = SnapshotStore(storage) - assert await ss.read("ratings-snapshot", "ZZZ") is None - - @pytest.mark.asyncio - async def test_write_empty_dict_skipped(self, storage: ParquetStorage) -> None: - ss = SnapshotStore(storage) - await ss.write("ratings-snapshot", "AAPL", {}) - assert await ss.read("ratings-snapshot", "AAPL") is None - - @pytest.mark.asyncio - async def test_list_entities(self, storage: ParquetStorage) -> None: - ss = SnapshotStore(storage) - await ss.write("discounted-cash-flow", "AAPL", {"x": 1}) - await ss.write("discounted-cash-flow", "MSFT", {"x": 2}) - await ss.write("levered-discounted-cash-flow", "AAPL", {"x": 3}) - entities = await ss.list_entities("discounted-cash-flow") - assert set(entities) == {"AAPL", "MSFT"} - entities2 = await ss.list_entities("levered-discounted-cash-flow") - assert set(entities2) == {"AAPL"} -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_cachedclient/test_snapshot_store.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/cachedclient/snapshot_store.py`** - -```python -"""SnapshotStore — single-row-per-entity storage on top of a StorageBackend. - -Used by the P4 snapshot-overwrite pattern (e.g. analyst ratings, DCF). -Storage keys take the shape ``("snapshot/", entity)``. -Each write replaces the previous row; reads return the latest dict or None. -""" - -from __future__ import annotations - -from typing import Any - -from aiofmp.cachedclient.storage.base import StorageBackend - - -class SnapshotStore: - """Thin wrapper that stores a single dict-row per (endpoint, entity) key.""" - - def __init__(self, storage: StorageBackend) -> None: - self._storage = storage - - def _key(self, endpoint: str, entity: str) -> tuple[str, ...]: - return (f"snapshot/{endpoint}", entity) - - async def write(self, endpoint: str, entity: str, payload: dict[str, Any]) -> None: - if not payload: - return - await self._storage.write(self._key(endpoint, entity), [payload], date_field="date") - - async def read(self, endpoint: str, entity: str) -> dict[str, Any] | None: - records = await self._storage.read(self._key(endpoint, entity)) - if not records: - return None - return records[0] - - async def list_entities(self, endpoint: str) -> list[str]: - """List all entities stored under ``snapshot/``.""" - prefix = (f"snapshot/{endpoint}",) - keys = await self._storage.list_keys(prefix=prefix) - entities: list[str] = [] - for k in keys: - # key shape is (f"snapshot/{endpoint}", entity) - if len(k) >= 2: - entities.append(k[1]) - return entities -``` - -- [ ] **Step 4: Re-export from `aiofmp/cachedclient/__init__.py`** - -Append to `aiofmp/cachedclient/__init__.py`: - -```python -from .snapshot_store import SnapshotStore # noqa: E402 - -__all__ = [*__all__, "SnapshotStore"] if "__all__" in dir() else ["SnapshotStore"] -``` - -(If `__all__` doesn't exist, simply add `from .snapshot_store import SnapshotStore` near the top with the other imports.) - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_cachedclient/test_snapshot_store.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/cachedclient/snapshot_store.py aiofmp/cachedclient/__init__.py tests/test_cachedclient/test_snapshot_store.py -git commit -m "feat(cachedclient): SnapshotStore for P4 snapshot-overwrite pattern" -``` - ---- - -### Task 13: Register chart endpoints under `commodity`/`forex`/`indexes` categories - -**Files:** -- Modify: `aiofmp/cachedclient/registry.py` -- Modify: existing `tests/test_cachedclient/test_registry.py` (if it exists) OR create new `tests/test_cachedclient/test_multi_category_registration.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_cachedclient/test_multi_category_registration.py`: - -```python -"""Tests that historical-price endpoints are registered under commodity/forex/indexes too.""" - -from __future__ import annotations - -from aiofmp.cachedclient.gap import DateParamType -from aiofmp.cachedclient.registry import TemporalPattern, build_default_registry - - -class TestMultiCategoryRegistration: - def test_commodity_historical_price_full(self) -> None: - reg = build_default_registry() - ep = reg.get("commodity", "historical_price_full") - assert ep is not None - assert ep.pattern == TemporalPattern.DATE_RANGE - assert ep.api_endpoint == "historical-price-eod/full" - assert ep.entity_key_args == ["symbol"] - assert ep.date_param_type == DateParamType.STRING - - def test_commodity_intraday(self) -> None: - reg = build_default_registry() - for m, ep_path in [ - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - ep = reg.get("commodity", m) - assert ep is not None, f"missing commodity.{m}" - assert ep.api_endpoint == ep_path - assert ep.date_param_type == DateParamType.STRING - - def test_forex_historical_and_intraday(self) -> None: - reg = build_default_registry() - for m, ep_path in [ - ("historical_price_full", "historical-price-eod/full"), - ("historical_price_light", "historical-price-eod/light"), - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - ep = reg.get("forex", m) - assert ep is not None, f"missing forex.{m}" - assert ep.api_endpoint == ep_path - assert ep.date_param_type == DateParamType.STRING - - def test_indexes_uses_date_obj(self) -> None: - reg = build_default_registry() - for m, ep_path in [ - ("historical_price_eod_full", "historical-price-eod/full"), - ("historical_price_eod_light", "historical-price-eod/light"), - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - ep = reg.get("indexes", m) - assert ep is not None, f"missing indexes.{m}" - assert ep.api_endpoint == ep_path - assert ep.date_param_type == DateParamType.DATE_OBJ - - def test_same_storage_key_across_categories(self) -> None: - """commodity.historical_price_full and chart.historical_price_full must produce - the same storage key for the same symbol — that's the whole point of multi-registration.""" - from aiofmp.cachedclient.proxy import _derive_storage_key # private helper - - reg = build_default_registry() - chart_ep = reg.get("chart", "historical_price_full") - commodity_ep = reg.get("commodity", "historical_price_full") - assert chart_ep is not None and commodity_ep is not None - bound = {"symbol": "GCUSD", "from_date": "2024-01-01", "to_date": "2024-12-31"} - assert _derive_storage_key(chart_ep, bound) == _derive_storage_key(commodity_ep, bound) -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_cachedclient/test_multi_category_registration.py -v` -Expected: failures — these registrations don't exist yet. - -- [ ] **Step 3: Extend `build_default_registry()` in `aiofmp/cachedclient/registry.py`** - -Append to `build_default_registry()` BEFORE the `return registry` line, after all existing `chart` registrations: - -```python - # ========================================================================= - # Multi-category sharing: commodity / forex / indexes use the same FMP - # wire endpoints as chart.* but expose their own SDK methods. Register - # the same metadata under those categories so user-driven calls hit the - # same cache. Storage keys start with api_endpoint, so they share parquet. - # ========================================================================= - - # --- Commodity (str dates) --- - for method, endpoint in [ - ("historical_price_light", "historical-price-eod/light"), - ("historical_price_full", "historical-price-eod/full"), - ]: - registry.register( - CacheableEndpoint( - category="commodity", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - date_param_type=DateParamType.STRING, - call_params=["symbol", "from_date", "to_date"], - ) - ) - for method, endpoint in [ - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - registry.register( - CacheableEndpoint( - category="commodity", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - response_date_format="%Y-%m-%d %H:%M:%S", - date_param_type=DateParamType.STRING, - call_params=["symbol", "from_date", "to_date"], - ) - ) - - # --- Forex (str dates) --- - for method, endpoint in [ - ("historical_price_light", "historical-price-eod/light"), - ("historical_price_full", "historical-price-eod/full"), - ]: - registry.register( - CacheableEndpoint( - category="forex", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - date_param_type=DateParamType.STRING, - call_params=["symbol", "from_date", "to_date"], - ) - ) - for method, endpoint in [ - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - registry.register( - CacheableEndpoint( - category="forex", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - response_date_format="%Y-%m-%d %H:%M:%S", - date_param_type=DateParamType.STRING, - call_params=["symbol", "from_date", "to_date"], - ) - ) - - # --- Indexes (date objects) --- - for method, endpoint in [ - ("historical_price_eod_light", "historical-price-eod/light"), - ("historical_price_eod_full", "historical-price-eod/full"), - ]: - registry.register( - CacheableEndpoint( - category="indexes", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - date_param_type=DateParamType.DATE_OBJ, - call_params=["symbol", "from_date", "to_date"], - ) - ) - for method, endpoint in [ - ("intraday_1min", "historical-chart/1min"), - ("intraday_5min", "historical-chart/5min"), - ("intraday_1hour", "historical-chart/1hour"), - ]: - registry.register( - CacheableEndpoint( - category="indexes", - method=method, - api_endpoint=endpoint, - pattern=TemporalPattern.DATE_RANGE, - entity_key_args=["symbol"], - response_date_format="%Y-%m-%d %H:%M:%S", - date_param_type=DateParamType.DATE_OBJ, - call_params=["symbol", "from_date", "to_date"], - ) - ) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_cachedclient/test_multi_category_registration.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 5: Confirm no regression on the cached-client suite** - -Run: `uv run pytest tests/test_cachedclient -q` -Expected: every previously passing test still passes. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/cachedclient/registry.py tests/test_cachedclient/test_multi_category_registration.py -git commit -m "feat(cachedclient): register chart endpoints under commodity/forex/indexes" -``` - ---- - -### Task 14: Register PAGE_WALK endpoints for analyst / insider-trading / form 13F - -**Files:** -- Modify: `aiofmp/cachedclient/registry.py` -- Modify: `tests/test_cachedclient/test_page_walk_registry.py` - -- [ ] **Step 1: Append failing tests to `test_page_walk_registry.py`** - -Append to `tests/test_cachedclient/test_page_walk_registry.py`: - -```python -class TestPageWalkEndpointRegistrations: - def test_analyst_financial_estimates_registered(self) -> None: - reg = build_default_registry() - ep = reg.get("analyst", "financial_estimates") - assert ep is not None - assert ep.pattern == TemporalPattern.PAGE_WALK - assert ep.api_endpoint == "analyst-estimates" - assert ep.entity_key_args == ["symbol"] - assert ep.extra_key_args == ["period"] - assert ep.page_param == "page" - assert ep.limit_param == "limit" - assert ep.walk_date_field == "date" - - def test_insider_trades_latest_registered(self) -> None: - reg = build_default_registry() - ep = reg.get("insider_trades", "latest_insider_trades") - assert ep is not None - assert ep.pattern == TemporalPattern.PAGE_WALK - assert ep.api_endpoint == "insider-trading/latest" - assert ep.entity_key_args == [] - assert ep.page_param == "page" - assert ep.walk_date_field == "filingDate" - - def test_form13f_latest_filings_registered(self) -> None: - reg = build_default_registry() - ep = reg.get("form13f", "latest_filings") - assert ep is not None - assert ep.pattern == TemporalPattern.PAGE_WALK - assert ep.api_endpoint == "institutional-ownership/latest" - assert ep.walk_date_field == "acceptedDate" -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_cachedclient/test_page_walk_registry.py::TestPageWalkEndpointRegistrations -v` -Expected: failures — endpoints not registered. - -- [ ] **Step 3: Register PAGE_WALK endpoints in `build_default_registry`** - -Append to `build_default_registry()` BEFORE the `return registry` line: - -```python - # ========================================================================= - # Pattern PAGE_WALK: harvester-write, user-read paginated endpoints - # ========================================================================= - - # --- Analyst estimates (per-symbol walk) --- - registry.register( - CacheableEndpoint( - category="analyst", - method="financial_estimates", - api_endpoint="analyst-estimates", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=["symbol"], - extra_key_args=["period"], - page_param="page", - limit_param="limit", - walk_date_field="date", - default_page_size=100, - call_params=["symbol", "period", "page", "limit"], - ) - ) - - # --- Insider trades (global walk; per-symbol shard on write) --- - registry.register( - CacheableEndpoint( - category="insider_trades", - method="latest_insider_trades", - api_endpoint="insider-trading/latest", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=[], - page_param="page", - limit_param="limit", - walk_date_field="filingDate", - default_page_size=100, - call_params=["page", "limit", "trade_date"], - ) - ) - - # --- Form 13F (global walk; per-CIK shard on write) --- - registry.register( - CacheableEndpoint( - category="form13f", - method="latest_filings", - api_endpoint="institutional-ownership/latest", - pattern=TemporalPattern.PAGE_WALK, - entity_key_args=[], - page_param="page", - limit_param="limit", - walk_date_field="acceptedDate", - default_page_size=100, - call_params=["page", "limit", "trade_date"], - ) - ) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_cachedclient/test_page_walk_registry.py -v` -Expected: all tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/cachedclient/registry.py tests/test_cachedclient/test_page_walk_registry.py -git commit -m "feat(cachedclient): register PAGE_WALK endpoints for analyst/insider/form13f" -``` - ---- - -## Phase 3 — Pattern P3 categories (gap-fill via CachedClient) - -### Task 15: `GapFillHarvester` shared base for all P3 categories - -**Files:** -- Create: `aiofmp/harvester/gap_fill_base.py` -- Create: `tests/test_harvester/test_gap_fill_base.py` - -This is a DRY base used by `chart_eod`, `chart_intraday`, `commodities_eod`, `commodities_intraday`, `forex_eod`, `forex_intraday`, `indexes_eod`, `indexes_intraday`, `news`, `economics`, and `technical_indicators`. Each concrete category just configures the base with a symbol universe and the SDK method to call per symbol; CachedClient gap-fill does the rest. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_gap_fill_base.py`: - -```python -"""Tests for GapFillHarvester (DRY base for P3 categories).""" - -from __future__ import annotations - -from datetime import date, timedelta -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -def store(tmp_path: Path) -> StateStore: - s = StateStore(tmp_path / "h.sqlite") - s.initialize() - return s - - -@pytest.fixture -def catalog() -> MagicMock: - cat = MagicMock() - cat.symbols = AsyncMock(return_value=["AAPL", "MSFT", "GOOGL"]) - return cat - - -@pytest.fixture -def cached_client() -> MagicMock: - """A fake CachedClient that records each call and returns canned data.""" - cc = MagicMock() - cc.chart = MagicMock() - cc.chart.historical_price_full = AsyncMock(return_value=[{"date": "2025-01-15", "close": 100}]) - cc.chart.historical_price_light = AsyncMock(return_value=[{"date": "2025-01-15", "close": 100}]) - return cc - - -def make_harvester( - *, - store: StateStore, - catalog: MagicMock, - cached_client: MagicMock, - extra: dict[str, Any] | None = None, - targets: list[GapFillTarget] | None = None, -) -> GapFillHarvester: - cfg = CategoryConfig(enabled=True, interval="1d", extra=extra or {}) - budget = BudgetTracker(store, BudgetConfig()) - retry = RetryConfig() - - if targets is None: - targets = [ - GapFillTarget( - category_attr="chart", - method_name="historical_price_full", - use_date_obj=False, - ) - ] - - return GapFillHarvester( - name="test_gap", - config=cfg, - state=store, - budget=budget, - retry=retry, - symbol_catalog=catalog, - symbol_universe="actively_trading", - cached_client=cached_client, - targets=targets, - backfill_seconds=10 * 86400, # 10 days - ) - - -class TestGapFillHarvester: - @pytest.mark.asyncio - async def test_iterates_symbols_and_calls_target(self, store, catalog, cached_client) -> None: - h = make_harvester(store=store, catalog=catalog, cached_client=cached_client) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - assert outcome.items_attempted == 3 - assert outcome.items_succeeded == 3 - assert cached_client.chart.historical_price_full.await_count == 3 - - @pytest.mark.asyncio - async def test_uses_string_dates(self, store, catalog, cached_client) -> None: - h = make_harvester(store=store, catalog=catalog, cached_client=cached_client) - await h.run_cycle() - first_call = cached_client.chart.historical_price_full.await_args_list[0] - # called as historical_price_full(symbol, from_date, to_date) - assert isinstance(first_call.args[0], str) - assert isinstance(first_call.args[1], str) - assert isinstance(first_call.args[2], str) - - @pytest.mark.asyncio - async def test_uses_date_objects_when_configured(self, store, catalog, cached_client) -> None: - cached_client.indexes = MagicMock() - cached_client.indexes.historical_price_eod_full = AsyncMock(return_value=[]) - targets = [ - GapFillTarget( - category_attr="indexes", - method_name="historical_price_eod_full", - use_date_obj=True, - ) - ] - h = make_harvester( - store=store, catalog=catalog, cached_client=cached_client, targets=targets - ) - await h.run_cycle() - first_call = cached_client.indexes.historical_price_eod_full.await_args_list[0] - assert isinstance(first_call.args[1], date) - assert isinstance(first_call.args[2], date) - - @pytest.mark.asyncio - async def test_per_symbol_error_does_not_kill_cycle(self, store, catalog, cached_client) -> None: - cached_client.chart.historical_price_full = AsyncMock( - side_effect=[ - [{"date": "2025-01-15"}], - RuntimeError("symbol explodes"), - [{"date": "2025-01-15"}], - ] - ) - h = make_harvester(store=store, catalog=catalog, cached_client=cached_client) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.PARTIAL - assert outcome.items_attempted == 3 - assert outcome.items_succeeded == 2 - - @pytest.mark.asyncio - async def test_multiple_variants(self, store, catalog, cached_client) -> None: - targets = [ - GapFillTarget(category_attr="chart", method_name="historical_price_full", use_date_obj=False), - GapFillTarget(category_attr="chart", method_name="historical_price_light", use_date_obj=False), - ] - h = make_harvester( - store=store, catalog=catalog, cached_client=cached_client, targets=targets, - ) - outcome = await h.run_cycle() - assert outcome.items_attempted == 6 # 3 symbols * 2 variants - assert cached_client.chart.historical_price_full.await_count == 3 - assert cached_client.chart.historical_price_light.await_count == 3 - - @pytest.mark.asyncio - async def test_empty_universe_is_ok(self, store, cached_client) -> None: - empty_catalog = MagicMock() - empty_catalog.symbols = AsyncMock(return_value=[]) - h = make_harvester(store=store, catalog=empty_catalog, cached_client=cached_client) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - assert outcome.items_attempted == 0 -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_gap_fill_base.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/gap_fill_base.py`** - -```python -"""GapFillHarvester — DRY base for P3 categories that delegate to CachedClient gap-fill.""" - -from __future__ import annotations - -import asyncio -import logging -from dataclasses import dataclass -from datetime import UTC, date, datetime, timedelta -from typing import Any - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - -logger = logging.getLogger(__name__) - - -@dataclass(frozen=True) -class GapFillTarget: - """One (category_attr, method_name) pair to call per symbol.""" - - category_attr: str # e.g. "chart", "commodity", "forex", "indexes" - method_name: str # e.g. "historical_price_full" - use_date_obj: bool # True for indexes.*, False elsewhere - - -class GapFillHarvester(CategoryHarvester): - """Generic P3 harvester: for each symbol in a universe, call each target - method with ``(symbol, from_date, to_date)`` and let CachedClient gap-fill. - """ - - def __init__( - self, - name: str, - config: CategoryConfig, - state: StateStore, - budget: BudgetTracker, - retry: RetryConfig, - *, - symbol_catalog: Any, - symbol_universe: str, - cached_client: Any, - targets: list[GapFillTarget], - backfill_seconds: int, - ) -> None: - super().__init__(name, config, state, budget, retry) - self._catalog = symbol_catalog - self._universe = symbol_universe - self._cached = cached_client - self._targets = list(targets) - self._backfill_seconds = backfill_seconds - - async def run_cycle(self) -> RunOutcome: - symbols = await self._catalog.symbols(self._universe) - if not symbols: - return RunOutcome(status=RunStatus.OK) - - today = date.today() - from_date = today - timedelta(seconds=self._backfill_seconds) - - attempted = 0 - succeeded = 0 - for symbol in symbols: - for target in self._targets: - attempted += 1 - try: - await self._call_target(target, symbol, from_date, today) - succeeded += 1 - except Exception as exc: - logger.warning( - "%s: %s.%s(%s) failed: %s", - self.name, target.category_attr, target.method_name, symbol, exc, - ) - - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - async def _call_target( - self, - target: GapFillTarget, - symbol: str, - from_d: date, - to_d: date, - ) -> None: - category = getattr(self._cached, target.category_attr) - method = getattr(category, target.method_name) - if target.use_date_obj: - await method(symbol, from_d, to_d) - else: - await method(symbol, from_d.isoformat(), to_d.isoformat()) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_gap_fill_base.py -v` -Expected: all 6 tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/gap_fill_base.py tests/test_harvester/test_gap_fill_base.py -git commit -m "feat(harvester): GapFillHarvester DRY base for P3 categories" -``` - ---- - -### Task 16: `chart_eod` category - -**Files:** -- Create: `aiofmp/harvester/categories/chart_eod.py` -- Create: `tests/test_harvester/test_chart_eod.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_chart_eod.py`: - -```python -"""Tests for chart_eod category.""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.categories.chart_eod import build_chart_eod -from aiofmp.harvester.config import CategoryConfig - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - """A fake manager exposing the attributes a category factory consumes.""" - from aiofmp.harvester.budget import BudgetTracker - from aiofmp.harvester.config import BudgetConfig, RetryConfig - from aiofmp.harvester.state import StateStore - - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite") - m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock() - m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL"]) - m.cached_client = MagicMock() - m.cached_client.chart = MagicMock() - m.cached_client.chart.historical_price_full = AsyncMock(return_value=[{"date": "2025-01-15"}]) - m.cached_client.chart.historical_price_light = AsyncMock(return_value=[]) - return m - - -class TestChartEod: - @pytest.mark.asyncio - async def test_default_variants(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={"backfill_years": 5}) - h = build_chart_eod(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 1 # 1 symbol * 1 default variant - manager.cached_client.chart.historical_price_full.assert_awaited_once() - - @pytest.mark.asyncio - async def test_multiple_variants(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, - interval="24h", - extra={"variants": ["historical_price_full", "historical_price_light"]}, - ) - h = build_chart_eod(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 2 - manager.cached_client.chart.historical_price_full.assert_awaited_once() - manager.cached_client.chart.historical_price_light.assert_awaited_once() - - @pytest.mark.asyncio - async def test_custom_symbol_universe(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, - interval="24h", - extra={"symbol_universe": "financial_symbols"}, - ) - h = build_chart_eod(cfg, manager) - await h.run_cycle() - manager.catalog.symbols.assert_awaited_with("financial_symbols") - - @pytest.mark.asyncio - async def test_registers_at_import(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "chart_eod" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_chart_eod.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/chart_eod.py`** - -```python -"""chart_eod category — P3 gap-fill over actively_trading.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -_DEFAULT_VARIANTS = ["historical_price_full"] - - -def build_chart_eod(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - variants = list(cfg.extra.get("variants", _DEFAULT_VARIANTS)) - backfill_years = int(cfg.extra.get("backfill_years", 10)) - universe = str(cfg.extra.get("symbol_universe", "actively_trading")) - - targets = [ - GapFillTarget(category_attr="chart", method_name=v, use_date_obj=False) - for v in variants - ] - return GapFillHarvester( - name="chart_eod", - config=cfg, - state=manager.state, - budget=manager.budget, - retry=manager.config.retry, - symbol_catalog=manager.catalog, - symbol_universe=universe, - cached_client=manager.cached_client, - targets=targets, - backfill_seconds=backfill_years * 365 * 86400, - ) - - -register_category("chart_eod", build_chart_eod) -``` - -- [ ] **Step 4: Import the category in the categories package so registration runs** - -Modify `aiofmp/harvester/categories/__init__.py` — append at the BOTTOM (after `registered_names` is defined): - -```python -# Import concrete category modules so their register_category() calls fire. -from aiofmp.harvester.categories import chart_eod # noqa: E402, F401 -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_chart_eod.py -v` -Expected: all 4 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/chart_eod.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_chart_eod.py -git commit -m "feat(harvester): chart_eod category (P3 gap-fill)" -``` - ---- - -### Task 17: `chart_intraday`, `commodities_eod`/`_intraday`, `forex_eod`/`_intraday`, `indexes_eod`/`_intraday` - -**Files:** -- Create: `aiofmp/harvester/categories/chart_intraday.py` -- Create: `aiofmp/harvester/categories/commodities_eod.py` -- Create: `aiofmp/harvester/categories/commodities_intraday.py` -- Create: `aiofmp/harvester/categories/forex_eod.py` -- Create: `aiofmp/harvester/categories/forex_intraday.py` -- Create: `aiofmp/harvester/categories/indexes_eod.py` -- Create: `aiofmp/harvester/categories/indexes_intraday.py` -- Modify: `aiofmp/harvester/categories/__init__.py` (register new modules) -- Create: `tests/test_harvester/test_chart_intraday.py` -- Create: `tests/test_harvester/test_commodities_eod.py` -- Create: `tests/test_harvester/test_forex_eod.py` -- Create: `tests/test_harvester/test_indexes_eod.py` - -All seven categories share a near-identical shape: pick a universe, pick a target method, optionally walk multiple timeframes. We write one comprehensive test file per "EOD" variant and one per "intraday" variant (intraday tests cover all three vehicles in one file). - -- [ ] **Step 1: Create `aiofmp/harvester/categories/chart_intraday.py`** - -```python -"""chart_intraday category — P3 gap-fill per (symbol, timeframe).""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -_DEFAULT_TIMEFRAMES = ["1hour"] -_TF_TO_METHOD = { - "1min": "intraday_1min", - "5min": "intraday_5min", - "15min": "intraday_15min", - "30min": "intraday_30min", - "1hour": "intraday_1hour", - "4hour": "intraday_4hour", -} - - -def _build_intraday( - name: str, - category_attr: str, - use_date_obj: bool, - universe: str, - cfg: CategoryConfig, - manager: "HarvesterManager", -) -> GapFillHarvester: - timeframes = list(cfg.extra.get("timeframes", _DEFAULT_TIMEFRAMES)) - for tf in timeframes: - if tf not in _TF_TO_METHOD: - raise ValueError(f"{name}: unknown timeframe {tf!r}; valid: {list(_TF_TO_METHOD)}") - backfill_days = int(cfg.extra.get("backfill_days", 30)) - user_universe = str(cfg.extra.get("symbol_universe", universe)) - - targets = [ - GapFillTarget(category_attr=category_attr, method_name=_TF_TO_METHOD[tf], use_date_obj=use_date_obj) - for tf in timeframes - ] - return GapFillHarvester( - name=name, - config=cfg, - state=manager.state, - budget=manager.budget, - retry=manager.config.retry, - symbol_catalog=manager.catalog, - symbol_universe=user_universe, - cached_client=manager.cached_client, - targets=targets, - backfill_seconds=backfill_days * 86400, - ) - - -def build_chart_intraday(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - return _build_intraday("chart_intraday", "chart", False, "actively_trading", cfg, manager) - - -register_category("chart_intraday", build_chart_intraday) -``` - -- [ ] **Step 2: Create `aiofmp/harvester/categories/commodities_eod.py`** - -```python -"""commodities_eod category — P3 gap-fill over the commodities universe.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -_DEFAULT_VARIANTS = ["historical_price_full"] - - -def build_commodities_eod(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - variants = list(cfg.extra.get("variants", _DEFAULT_VARIANTS)) - backfill_years = int(cfg.extra.get("backfill_years", 10)) - targets = [ - GapFillTarget(category_attr="commodity", method_name=v, use_date_obj=False) - for v in variants - ] - return GapFillHarvester( - name="commodities_eod", - config=cfg, - state=manager.state, - budget=manager.budget, - retry=manager.config.retry, - symbol_catalog=manager.catalog, - symbol_universe="commodities", - cached_client=manager.cached_client, - targets=targets, - backfill_seconds=backfill_years * 365 * 86400, - ) - - -register_category("commodities_eod", build_commodities_eod) -``` - -- [ ] **Step 3: Create `aiofmp/harvester/categories/commodities_intraday.py`** - -```python -"""commodities_intraday category — P3 gap-fill per (symbol, timeframe).""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.categories.chart_intraday import _build_intraday # reuse helper -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -def build_commodities_intraday(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - return _build_intraday("commodities_intraday", "commodity", False, "commodities", cfg, manager) - - -register_category("commodities_intraday", build_commodities_intraday) -``` - -- [ ] **Step 4: Create `aiofmp/harvester/categories/forex_eod.py`** - -```python -"""forex_eod category — P3 gap-fill over the forex_pairs universe.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -_DEFAULT_VARIANTS = ["historical_price_full"] - - -def build_forex_eod(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - variants = list(cfg.extra.get("variants", _DEFAULT_VARIANTS)) - backfill_years = int(cfg.extra.get("backfill_years", 10)) - targets = [ - GapFillTarget(category_attr="forex", method_name=v, use_date_obj=False) - for v in variants - ] - return GapFillHarvester( - name="forex_eod", - config=cfg, - state=manager.state, - budget=manager.budget, - retry=manager.config.retry, - symbol_catalog=manager.catalog, - symbol_universe="forex_pairs", - cached_client=manager.cached_client, - targets=targets, - backfill_seconds=backfill_years * 365 * 86400, - ) - - -register_category("forex_eod", build_forex_eod) -``` - -- [ ] **Step 5: Create `aiofmp/harvester/categories/forex_intraday.py`** - -```python -"""forex_intraday category.""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.categories.chart_intraday import _build_intraday -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -def build_forex_intraday(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - return _build_intraday("forex_intraday", "forex", False, "forex_pairs", cfg, manager) - - -register_category("forex_intraday", build_forex_intraday) -``` - -- [ ] **Step 6: Create `aiofmp/harvester/categories/indexes_eod.py`** - -```python -"""indexes_eod category — P3 gap-fill over the indexes universe (DATE_OBJ params).""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester, GapFillTarget - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -_DEFAULT_VARIANTS = ["historical_price_eod_full"] - - -def build_indexes_eod(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - variants = list(cfg.extra.get("variants", _DEFAULT_VARIANTS)) - backfill_years = int(cfg.extra.get("backfill_years", 10)) - targets = [ - GapFillTarget(category_attr="indexes", method_name=v, use_date_obj=True) - for v in variants - ] - return GapFillHarvester( - name="indexes_eod", - config=cfg, - state=manager.state, - budget=manager.budget, - retry=manager.config.retry, - symbol_catalog=manager.catalog, - symbol_universe="indexes", - cached_client=manager.cached_client, - targets=targets, - backfill_seconds=backfill_years * 365 * 86400, - ) - - -register_category("indexes_eod", build_indexes_eod) -``` - -- [ ] **Step 7: Create `aiofmp/harvester/categories/indexes_intraday.py`** - -```python -"""indexes_intraday category (DATE_OBJ params).""" - -from __future__ import annotations - -from typing import TYPE_CHECKING - -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.categories.chart_intraday import _build_intraday -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.gap_fill_base import GapFillHarvester - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - - -def build_indexes_intraday(cfg: CategoryConfig, manager: "HarvesterManager") -> GapFillHarvester: - return _build_intraday("indexes_intraday", "indexes", True, "indexes", cfg, manager) - - -register_category("indexes_intraday", build_indexes_intraday) -``` - -- [ ] **Step 8: Wire all new modules into the registry init** - -Replace the bottom of `aiofmp/harvester/categories/__init__.py` with the full import list: - -```python -# Import concrete category modules so their register_category() calls fire. -from aiofmp.harvester.categories import ( # noqa: E402, F401 - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, -) -``` - -- [ ] **Step 9: Create `tests/test_harvester/test_chart_intraday.py`** - -```python -"""Tests for chart_intraday and its sister intraday categories.""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -def _fake_manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite") - m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock() - m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["X"]) - m.cached_client = MagicMock() - # Set up every (category, method) the tests touch: - for cat in ("chart", "commodity", "forex", "indexes"): - c = MagicMock() - for method in ("intraday_1min", "intraday_5min", "intraday_15min", "intraday_30min", - "intraday_1hour", "intraday_4hour"): - setattr(c, method, AsyncMock(return_value=[])) - setattr(m.cached_client, cat, c) - return m - - -class TestChartIntradayFamily: - @pytest.mark.asyncio - async def test_chart_intraday_default_timeframe(self, tmp_path: Path) -> None: - from aiofmp.harvester.categories.chart_intraday import build_chart_intraday - mgr = _fake_manager(tmp_path) - cfg = CategoryConfig(enabled=True, interval="4h", extra={}) - h = build_chart_intraday(cfg, mgr) - await h.run_cycle() - mgr.cached_client.chart.intraday_1hour.assert_awaited_once() - - @pytest.mark.asyncio - async def test_chart_intraday_multiple_timeframes(self, tmp_path: Path) -> None: - from aiofmp.harvester.categories.chart_intraday import build_chart_intraday - mgr = _fake_manager(tmp_path) - cfg = CategoryConfig(enabled=True, interval="4h", extra={"timeframes": ["1hour", "5min"]}) - h = build_chart_intraday(cfg, mgr) - await h.run_cycle() - mgr.cached_client.chart.intraday_1hour.assert_awaited_once() - mgr.cached_client.chart.intraday_5min.assert_awaited_once() - - @pytest.mark.asyncio - async def test_chart_intraday_invalid_timeframe_raises(self, tmp_path: Path) -> None: - from aiofmp.harvester.categories.chart_intraday import build_chart_intraday - mgr = _fake_manager(tmp_path) - cfg = CategoryConfig(enabled=True, interval="4h", extra={"timeframes": ["weekly"]}) - with pytest.raises(ValueError, match="weekly"): - build_chart_intraday(cfg, mgr) - - @pytest.mark.asyncio - async def test_commodities_intraday(self, tmp_path: Path) -> None: - from aiofmp.harvester.categories.commodities_intraday import build_commodities_intraday - mgr = _fake_manager(tmp_path) - mgr.catalog.symbols = AsyncMock(return_value=["GCUSD"]) - cfg = CategoryConfig(enabled=True, interval="4h", extra={}) - h = build_commodities_intraday(cfg, mgr) - await h.run_cycle() - mgr.catalog.symbols.assert_awaited_with("commodities") - mgr.cached_client.commodity.intraday_1hour.assert_awaited_once() - - @pytest.mark.asyncio - async def test_forex_intraday(self, tmp_path: Path) -> None: - from aiofmp.harvester.categories.forex_intraday import build_forex_intraday - mgr = _fake_manager(tmp_path) - mgr.catalog.symbols = AsyncMock(return_value=["EURUSD"]) - cfg = CategoryConfig(enabled=True, interval="4h", extra={}) - h = build_forex_intraday(cfg, mgr) - await h.run_cycle() - mgr.catalog.symbols.assert_awaited_with("forex_pairs") - mgr.cached_client.forex.intraday_1hour.assert_awaited_once() - - @pytest.mark.asyncio - async def test_indexes_intraday_uses_date_obj(self, tmp_path: Path) -> None: - from datetime import date - from aiofmp.harvester.categories.indexes_intraday import build_indexes_intraday - mgr = _fake_manager(tmp_path) - mgr.catalog.symbols = AsyncMock(return_value=["^GSPC"]) - cfg = CategoryConfig(enabled=True, interval="4h", extra={}) - h = build_indexes_intraday(cfg, mgr) - await h.run_cycle() - mgr.catalog.symbols.assert_awaited_with("indexes") - first_call = mgr.cached_client.indexes.intraday_1hour.await_args_list[0] - assert isinstance(first_call.args[1], date) - assert isinstance(first_call.args[2], date) -``` - -- [ ] **Step 10: Create `tests/test_harvester/test_commodities_eod.py`** - -```python -"""Tests for commodities_eod.""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.commodities_eod import build_commodities_eod -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["GCUSD", "CLUSD"]) - m.cached_client = MagicMock() - m.cached_client.commodity = MagicMock() - m.cached_client.commodity.historical_price_full = AsyncMock(return_value=[]) - return m - - -class TestCommoditiesEod: - @pytest.mark.asyncio - async def test_iterates_commodities(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_commodities_eod(cfg, manager) - outcome = await h.run_cycle() - manager.catalog.symbols.assert_awaited_with("commodities") - assert outcome.items_attempted == 2 - assert manager.cached_client.commodity.historical_price_full.await_count == 2 - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "commodities_eod" in _REGISTRY -``` - -- [ ] **Step 11: Create `tests/test_harvester/test_forex_eod.py`** - -```python -"""Tests for forex_eod.""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.forex_eod import build_forex_eod -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["EURUSD"]) - m.cached_client = MagicMock() - m.cached_client.forex = MagicMock() - m.cached_client.forex.historical_price_full = AsyncMock(return_value=[]) - return m - - -class TestForexEod: - @pytest.mark.asyncio - async def test_iterates_pairs(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_forex_eod(cfg, manager) - outcome = await h.run_cycle() - manager.catalog.symbols.assert_awaited_with("forex_pairs") - assert outcome.items_attempted == 1 - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "forex_eod" in _REGISTRY -``` - -- [ ] **Step 12: Create `tests/test_harvester/test_indexes_eod.py`** - -```python -"""Tests for indexes_eod (uses date objects).""" - -from __future__ import annotations - -from datetime import date -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.indexes_eod import build_indexes_eod -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["^GSPC"]) - m.cached_client = MagicMock() - m.cached_client.indexes = MagicMock() - m.cached_client.indexes.historical_price_eod_full = AsyncMock(return_value=[]) - return m - - -class TestIndexesEod: - @pytest.mark.asyncio - async def test_uses_date_objects(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_indexes_eod(cfg, manager) - await h.run_cycle() - call = manager.cached_client.indexes.historical_price_eod_full.await_args_list[0] - assert isinstance(call.args[1], date) - assert isinstance(call.args[2], date) - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "indexes_eod" in _REGISTRY -``` - -- [ ] **Step 13: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_chart_intraday.py tests/test_harvester/test_commodities_eod.py tests/test_harvester/test_forex_eod.py tests/test_harvester/test_indexes_eod.py -v` -Expected: all tests pass. - -- [ ] **Step 14: Commit** - -```bash -git add aiofmp/harvester/categories/chart_intraday.py \ - aiofmp/harvester/categories/commodities_eod.py \ - aiofmp/harvester/categories/commodities_intraday.py \ - aiofmp/harvester/categories/forex_eod.py \ - aiofmp/harvester/categories/forex_intraday.py \ - aiofmp/harvester/categories/indexes_eod.py \ - aiofmp/harvester/categories/indexes_intraday.py \ - aiofmp/harvester/categories/__init__.py \ - tests/test_harvester/test_chart_intraday.py \ - tests/test_harvester/test_commodities_eod.py \ - tests/test_harvester/test_forex_eod.py \ - tests/test_harvester/test_indexes_eod.py -git commit -m "feat(harvester): intraday + commodities/forex/indexes P3 categories" -``` - ---- - -### Task 18: `news` category (P3, multi-variant, no per-symbol iteration) - -**Files:** -- Create: `aiofmp/harvester/categories/news.py` -- Modify: `aiofmp/harvester/categories/__init__.py` (add import) -- Create: `tests/test_harvester/test_news.py` - -`news` differs slightly from the chart family: it's date-range with NO symbol (entity_key_args=[]). The harvester just calls each variant with `(from_date, to_date)` and lets CachedClient gap-fill. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_news.py`: - -```python -"""Tests for the news category.""" - -from __future__ import annotations - -from datetime import date -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.news import build_news -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.cached_client = MagicMock() - m.cached_client.news = MagicMock() - for v in ("general_news", "press_releases", "stock_news", "crypto_news", "forex_news"): - setattr(m.cached_client.news, v, AsyncMock(return_value=[])) - return m - - -class TestNews: - @pytest.mark.asyncio - async def test_default_variants(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="30m", extra={}) - h = build_news(cfg, manager) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - # five variants, each called once - for v in ("general_news", "press_releases", "stock_news", "crypto_news", "forex_news"): - getattr(manager.cached_client.news, v).assert_awaited_once() - - @pytest.mark.asyncio - async def test_custom_variants(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, interval="30m", - extra={"variants": ["general_news"], "backfill_days_initial": 7, "page_size": 50}, - ) - h = build_news(cfg, manager) - await h.run_cycle() - call = manager.cached_client.news.general_news.await_args_list[0] - # called as general_news(page=0, limit=50, from_date=, to_date=) - assert call.kwargs["limit"] == 50 - assert isinstance(call.kwargs["from_date"], date) - assert isinstance(call.kwargs["to_date"], date) - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "news" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_news.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/news.py`** - -```python -"""news category — P3 over no entity (the call has from/to but no symbol).""" - -from __future__ import annotations - -import logging -from datetime import UTC, date, datetime, timedelta -from typing import TYPE_CHECKING - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_DEFAULT_VARIANTS = ["general_news", "press_releases", "stock_news", "crypto_news", "forex_news"] - - -class NewsHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("news", cfg, manager.state, manager.budget, manager.config.retry) - self._cached = manager.cached_client - self._variants = list(cfg.extra.get("variants", _DEFAULT_VARIANTS)) - self._backfill_days = int(cfg.extra.get("backfill_days_initial", 30)) - self._page_size = int(cfg.extra.get("page_size", 100)) - - async def run_cycle(self) -> RunOutcome: - today = date.today() - from_date = today - timedelta(days=self._backfill_days) - attempted = 0 - succeeded = 0 - for v in self._variants: - attempted += 1 - method = getattr(self._cached.news, v) - try: - await method(page=0, limit=self._page_size, from_date=from_date, to_date=today) - succeeded += 1 - except Exception as exc: - logger.warning("news.%s failed: %s", v, exc) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_news(cfg: CategoryConfig, manager: "HarvesterManager") -> NewsHarvester: - return NewsHarvester(cfg, manager) - - -register_category("news", build_news) -``` - -- [ ] **Step 4: Register the module** - -Update `aiofmp/harvester/categories/__init__.py` to include `news` in the import block at the bottom: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - news, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_news.py -v` -Expected: all 3 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/news.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_news.py -git commit -m "feat(harvester): news category (P3 multi-variant)" -``` - ---- - -### Task 19: `economics` category (P3, treasury rates + indicator list) - -**Files:** -- Create: `aiofmp/harvester/categories/economics.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_economics.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_economics.py`: - -```python -"""Tests for the economics category.""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.economics import build_economics -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.cached_client = MagicMock() - m.cached_client.economics = MagicMock() - m.cached_client.economics.treasury_rates = AsyncMock(return_value=[]) - m.cached_client.economics.economic_indicators = AsyncMock(return_value=[]) - return m - - -class TestEconomics: - @pytest.mark.asyncio - async def test_default_indicators(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_economics(cfg, manager) - outcome = await h.run_cycle() - # 1 treasury_rates call + 5 default indicators - assert outcome.items_attempted == 6 - manager.cached_client.economics.treasury_rates.assert_awaited_once() - assert manager.cached_client.economics.economic_indicators.await_count == 5 - - @pytest.mark.asyncio - async def test_custom_indicators(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, interval="24h", - extra={"indicators": ["GDP", "CPI"], "backfill_years": 3}, - ) - h = build_economics(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 3 # treasury + 2 indicators - names_called = [c.args[0] for c in manager.cached_client.economics.economic_indicators.await_args_list] - assert set(names_called) == {"GDP", "CPI"} - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "economics" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_economics.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/economics.py`** - -```python -"""economics category — P3 (treasury_rates + per-indicator).""" - -from __future__ import annotations - -import logging -from datetime import date, timedelta -from typing import TYPE_CHECKING - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_DEFAULT_INDICATORS = ["GDP", "CPI", "UNRATE", "FEDFUNDS", "DFF"] - - -class EconomicsHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("economics", cfg, manager.state, manager.budget, manager.config.retry) - self._cached = manager.cached_client - self._indicators = list(cfg.extra.get("indicators", _DEFAULT_INDICATORS)) - self._backfill_years = int(cfg.extra.get("backfill_years", 10)) - - async def run_cycle(self) -> RunOutcome: - today = date.today() - from_date = today - timedelta(days=self._backfill_years * 365) - from_s, to_s = from_date.isoformat(), today.isoformat() - attempted = 0 - succeeded = 0 - - attempted += 1 - try: - await self._cached.economics.treasury_rates(from_s, to_s) - succeeded += 1 - except Exception as exc: - logger.warning("economics.treasury_rates failed: %s", exc) - - for ind in self._indicators: - attempted += 1 - try: - await self._cached.economics.economic_indicators(ind, from_s, to_s) - succeeded += 1 - except Exception as exc: - logger.warning("economics.economic_indicators(%s) failed: %s", ind, exc) - - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_economics(cfg: CategoryConfig, manager: "HarvesterManager") -> EconomicsHarvester: - return EconomicsHarvester(cfg, manager) - - -register_category("economics", build_economics) -``` - -- [ ] **Step 4: Register the module** - -Update the import block at the bottom of `aiofmp/harvester/categories/__init__.py` to include `economics`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - news, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_economics.py -v` -Expected: all 3 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/economics.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_economics.py -git commit -m "feat(harvester): economics category (treasury + indicators)" -``` - ---- - -### Task 20: `technical_indicators` category (P3, off by default) - -**Files:** -- Create: `aiofmp/harvester/categories/technical_indicators.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_technical_indicators.py` - -Indicators are configured as a list of `{method, period_length, timeframe}` dicts. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_technical_indicators.py`: - -```python -"""Tests for the technical_indicators category.""" - -from __future__ import annotations - -from datetime import date -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.technical_indicators import build_technical_indicators -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL", "MSFT"]) - m.cached_client = MagicMock() - ti = MagicMock() - ti.simple_moving_average = AsyncMock(return_value=[]) - ti.relative_strength_index = AsyncMock(return_value=[]) - m.cached_client.technical_indicators = ti - return m - - -class TestTechnicalIndicators: - @pytest.mark.asyncio - async def test_empty_indicators_is_ok(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={"indicators": []}) - h = build_technical_indicators(cfg, manager) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - assert outcome.items_attempted == 0 - - @pytest.mark.asyncio - async def test_one_indicator_two_symbols(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, interval="24h", - extra={ - "indicators": [ - {"method": "simple_moving_average", "period_length": 20, "timeframe": "1day"} - ], - }, - ) - h = build_technical_indicators(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 2 - assert manager.cached_client.technical_indicators.simple_moving_average.await_count == 2 - first = manager.cached_client.technical_indicators.simple_moving_average.await_args_list[0] - assert first.args[0] in ("AAPL", "MSFT") # symbol - assert first.args[1] == 20 # period_length - assert first.args[2] == "1day" # timeframe - assert isinstance(first.args[3], date) # from_date - assert isinstance(first.args[4], date) # to_date - - @pytest.mark.asyncio - async def test_multiple_indicators(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, interval="24h", - extra={ - "indicators": [ - {"method": "simple_moving_average", "period_length": 20, "timeframe": "1day"}, - {"method": "relative_strength_index", "period_length": 14, "timeframe": "1day"}, - ], - }, - ) - h = build_technical_indicators(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 4 # 2 symbols * 2 indicators - - @pytest.mark.asyncio - async def test_unknown_method_raises_at_build(self, manager: MagicMock) -> None: - cfg = CategoryConfig( - enabled=True, interval="24h", - extra={"indicators": [{"method": "moonshot", "period_length": 1, "timeframe": "1day"}]}, - ) - with pytest.raises(ValueError, match="moonshot"): - build_technical_indicators(cfg, manager) - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "technical_indicators" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_technical_indicators.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/technical_indicators.py`** - -```python -"""technical_indicators category — P3 per (symbol, indicator, period, timeframe).""" - -from __future__ import annotations - -import logging -from datetime import date, timedelta -from typing import TYPE_CHECKING, Any - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_VALID_METHODS = { - "simple_moving_average", - "exponential_moving_average", - "weighted_moving_average", - "double_exponential_moving_average", - "triple_exponential_moving_average", - "relative_strength_index", - "standard_deviation", - "williams_percent_r", - "average_directional_index", -} - - -class TechnicalIndicatorsHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("technical_indicators", cfg, manager.state, manager.budget, manager.config.retry) - self._catalog = manager.catalog - self._cached = manager.cached_client - self._universe = str(cfg.extra.get("symbol_universe", "actively_trading")) - self._backfill_years = int(cfg.extra.get("backfill_years", 5)) - - indicators: list[dict[str, Any]] = list(cfg.extra.get("indicators", [])) - for ind in indicators: - if ind.get("method") not in _VALID_METHODS: - raise ValueError(f"technical_indicators: unknown method {ind.get('method')!r}") - self._indicators = indicators - - async def run_cycle(self) -> RunOutcome: - if not self._indicators: - return RunOutcome(status=RunStatus.OK) - symbols = await self._catalog.symbols(self._universe) - today = date.today() - from_date = today - timedelta(days=self._backfill_years * 365) - attempted = 0 - succeeded = 0 - for symbol in symbols: - for ind in self._indicators: - attempted += 1 - method_name = ind["method"] - period_length = int(ind["period_length"]) - timeframe = str(ind["timeframe"]) - method = getattr(self._cached.technical_indicators, method_name) - try: - await method(symbol, period_length, timeframe, from_date, today) - succeeded += 1 - except Exception as exc: - logger.warning( - "technical_indicators.%s(%s, %d, %s) failed: %s", - method_name, symbol, period_length, timeframe, exc, - ) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_technical_indicators(cfg: CategoryConfig, manager: "HarvesterManager") -> TechnicalIndicatorsHarvester: - return TechnicalIndicatorsHarvester(cfg, manager) - - -register_category("technical_indicators", build_technical_indicators) -``` - -- [ ] **Step 4: Register the module** - -Add `technical_indicators` to the import block in `aiofmp/harvester/categories/__init__.py`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - news, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_technical_indicators.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/technical_indicators.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_technical_indicators.py -git commit -m "feat(harvester): technical_indicators category (off by default)" -``` - ---- - -## Phase 4 — Pattern P1: statements - -### Task 21: `statements` category (discover-then-fill + monthly safety-net) - -**Files:** -- Create: `aiofmp/harvester/categories/statements.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_statements.py` - -The statements category uses two cooperating run-modes inside one harvester: -1. **Incremental cycle** (every `interval`): call `earnings_calendar(from=last_run, to=today)` → unique symbols → re-call the 13 statement endpoints per touched symbol with `limit=incremental_limit`. -2. **Safety-net cycle** (every `safety_net_interval`): enumerate every symbol in `financial_symbols` and call all 13 statement endpoints with `limit=incremental_limit`. Bookkept as a separate `category_runs` row with `category="statements_safetynet"`. - -The checkpoint `("statements", "global")` stores the ISO date of the most recent earnings_calendar discovery. - -The 13 statement endpoints fall into three sub-shapes: -- 10 with `(period, limit)`: income, balance, cash flow, key_metrics, ratios, enterprise_values, 4× growth -- 1 with `limit` only: owner_earnings -- 2 with `period` only (no limit): revenue_product_segmentation, revenue_geographic_segmentation - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_statements.py`: - -```python -"""Tests for the statements category (P1).""" - -from __future__ import annotations - -from datetime import UTC, datetime, timedelta -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.statements import ( - PERIOD_AND_LIMIT_ENDPOINTS, - PERIOD_ONLY_ENDPOINTS, - LIMIT_ONLY_ENDPOINTS, - StatementsHarvester, - build_statements, -) -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -def _wire_statement_methods(stmts: MagicMock) -> None: - for m in PERIOD_AND_LIMIT_ENDPOINTS + LIMIT_ONLY_ENDPOINTS + PERIOD_ONLY_ENDPOINTS: - setattr(stmts, m, AsyncMock(return_value=[])) - - -@pytest.fixture -def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL", "MSFT"]) - m.cached_client = MagicMock() - m.cached_client.calendar = MagicMock() - m.cached_client.calendar.earnings_calendar = AsyncMock(return_value=[]) - m.cached_client.statements = MagicMock() - _wire_statement_methods(m.cached_client.statements) - return m - - -def _expected_calls_per_symbol(periods: list[str]) -> int: - # 10 period+limit endpoints × len(periods) - n = len(PERIOD_AND_LIMIT_ENDPOINTS) * len(periods) - # 1 limit-only endpoint - n += len(LIMIT_ONLY_ENDPOINTS) - # 2 period-only endpoints × len(periods) - n += len(PERIOD_ONLY_ENDPOINTS) * len(periods) - return n - - -class TestStatementsFirstRun: - @pytest.mark.asyncio - async def test_first_run_uses_initial_limit_and_full_universe(self, manager: MagicMock) -> None: - """With no checkpoint set, first run iterates the financial_symbols universe with initial_limit.""" - cfg = CategoryConfig(enabled=True, interval="6h", extra={ - "periods": ["annual", "quarter"], - "initial_limit": 40, - "incremental_limit": 2, - }) - h = build_statements(cfg, manager) - outcome = await h.run_cycle() - - manager.catalog.symbols.assert_awaited_with("financial_symbols") - expected = _expected_calls_per_symbol(["annual", "quarter"]) * 2 # 2 symbols - assert outcome.items_attempted == expected - # Verify limit=40 was passed to a period+limit endpoint - first = manager.cached_client.statements.income_statement.await_args_list[0] - assert first.kwargs.get("limit") == 40 - - @pytest.mark.asyncio - async def test_first_run_sets_checkpoint(self, manager: MagicMock) -> None: - cfg = CategoryConfig(enabled=True, interval="6h", extra={"periods": ["annual"]}) - h = build_statements(cfg, manager) - await h.run_cycle() - ckpt = manager.state.get_checkpoint("statements", "global") - assert ckpt is not None - # Looks like a date string YYYY-MM-DD - assert len(ckpt) == 10 and ckpt[4] == "-" - - -class TestStatementsIncremental: - @pytest.mark.asyncio - async def test_uses_earnings_calendar_when_checkpoint_exists(self, manager: MagicMock) -> None: - manager.state.set_checkpoint("statements", "global", "2026-04-01") - manager.cached_client.calendar.earnings_calendar = AsyncMock( - return_value=[ - {"symbol": "AAPL", "date": "2026-04-25"}, - {"symbol": "AAPL", "date": "2026-04-26"}, # duplicate symbol - {"symbol": "TSLA", "date": "2026-04-28"}, - ] - ) - cfg = CategoryConfig(enabled=True, interval="6h", extra={ - "periods": ["annual"], "incremental_limit": 2, - }) - h = build_statements(cfg, manager) - outcome = await h.run_cycle() - - manager.cached_client.calendar.earnings_calendar.assert_awaited_once() - # 2 unique symbols (AAPL, TSLA) - expected = _expected_calls_per_symbol(["annual"]) * 2 - assert outcome.items_attempted == expected - # Verify limit=2 was used - first = manager.cached_client.statements.income_statement.await_args_list[0] - assert first.kwargs.get("limit") == 2 - - @pytest.mark.asyncio - async def test_no_filings_in_window_is_noop(self, manager: MagicMock) -> None: - manager.state.set_checkpoint("statements", "global", "2026-04-01") - manager.cached_client.calendar.earnings_calendar = AsyncMock(return_value=[]) - cfg = CategoryConfig(enabled=True, interval="6h", extra={"periods": ["annual"]}) - h = build_statements(cfg, manager) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - assert outcome.items_attempted == 0 - manager.cached_client.statements.income_statement.assert_not_awaited() - - -class TestStatementsSafetyNet: - @pytest.mark.asyncio - async def test_safety_net_triggers_after_interval(self, manager: MagicMock) -> None: - # set an old "last safety-net" checkpoint - manager.state.set_checkpoint( - "statements_safetynet", - "global", - (datetime.now(UTC) - timedelta(days=40)).date().isoformat(), - ) - manager.state.set_checkpoint("statements", "global", "2026-05-10") - cfg = CategoryConfig(enabled=True, interval="6h", extra={ - "periods": ["annual"], "safety_net_interval": "30d", - }) - h = build_statements(cfg, manager) - outcome = await h.run_cycle() - - # Safety-net ran: should have iterated full universe (2 symbols) - expected = _expected_calls_per_symbol(["annual"]) * 2 - assert outcome.items_attempted == expected - # New safety-net checkpoint should be today - new_ckpt = manager.state.get_checkpoint("statements_safetynet", "global") - from datetime import date - assert new_ckpt == date.today().isoformat() - - @pytest.mark.asyncio - async def test_safety_net_skipped_when_recent(self, manager: MagicMock) -> None: - manager.state.set_checkpoint( - "statements_safetynet", - "global", - (datetime.now(UTC) - timedelta(days=5)).date().isoformat(), - ) - manager.state.set_checkpoint("statements", "global", "2026-05-10") - manager.cached_client.calendar.earnings_calendar = AsyncMock(return_value=[]) - cfg = CategoryConfig(enabled=True, interval="6h", extra={"periods": ["annual"]}) - h = build_statements(cfg, manager) - outcome = await h.run_cycle() - # No safety net, no filings discovered → 0 attempted - assert outcome.items_attempted == 0 - - -class TestRegistration: - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "statements" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_statements.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/statements.py`** - -```python -"""statements category — Pattern P1 (discover-then-fill) + monthly safety-net sweep.""" - -from __future__ import annotations - -import logging -from datetime import UTC, date, datetime, timedelta -from typing import TYPE_CHECKING - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig, parse_interval -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - - -# The 13 statement endpoints split by SDK signature shape. -PERIOD_AND_LIMIT_ENDPOINTS: list[str] = [ - "income_statement", - "balance_sheet_statement", - "cash_flow_statement", - "key_metrics", - "financial_ratios", - "enterprise_values", - "income_statement_growth", - "balance_sheet_statement_growth", - "cash_flow_statement_growth", - "financial_statement_growth", -] - -LIMIT_ONLY_ENDPOINTS: list[str] = [ - "owner_earnings", -] - -PERIOD_ONLY_ENDPOINTS: list[str] = [ - "revenue_product_segmentation", - "revenue_geographic_segmentation", -] - - -class StatementsHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("statements", cfg, manager.state, manager.budget, manager.config.retry) - self._catalog = manager.catalog - self._cached = manager.cached_client - self._periods: list[str] = list(cfg.extra.get("periods", ["annual", "quarter"])) - self._initial_limit = int(cfg.extra.get("initial_limit", 40)) - self._incremental_limit = int(cfg.extra.get("incremental_limit", 2)) - self._safety_net_seconds = parse_interval(str(cfg.extra.get("safety_net_interval", "30d"))) - - async def run_cycle(self) -> RunOutcome: - today = date.today() - if self._should_run_safety_net(today): - outcome = await self._run_safety_net(today) - # Mark the safety-net last-run in a separate scope - self.state.set_checkpoint("statements_safetynet", "global", today.isoformat()) - # Also bump primary checkpoint so incremental window doesn't span both - self.state.set_checkpoint("statements", "global", today.isoformat()) - return outcome - - return await self._run_incremental(today) - - def _should_run_safety_net(self, today: date) -> bool: - last = self.state.get_checkpoint("statements_safetynet", "global") - if last is None: - return True # never ran a safety-net before → run one now - try: - last_d = date.fromisoformat(last) - except ValueError: - return True - age_sec = (today - last_d).days * 86400 - return age_sec >= self._safety_net_seconds - - async def _run_safety_net(self, today: date) -> RunOutcome: - symbols = await self._catalog.symbols("financial_symbols") - return await self._iterate_symbols(symbols, self._incremental_limit) - - async def _run_incremental(self, today: date) -> RunOutcome: - last = self.state.get_checkpoint("statements", "global") - if last is None: - # First run ever (and not yet a safety-net run): treat as full backfill - symbols = await self._catalog.symbols("financial_symbols") - outcome = await self._iterate_symbols(symbols, self._initial_limit) - self.state.set_checkpoint("statements", "global", today.isoformat()) - return outcome - - try: - from_date = date.fromisoformat(last) - except ValueError: - from_date = today - timedelta(days=30) - - from_s, to_s = from_date.isoformat(), today.isoformat() - records = await self._cached.calendar.earnings_calendar(from_s, to_s) - - symbols = sorted({r["symbol"] for r in records if r.get("symbol")}) - outcome = await self._iterate_symbols(symbols, self._incremental_limit) - self.state.set_checkpoint("statements", "global", today.isoformat()) - return outcome - - async def _iterate_symbols(self, symbols: list[str], limit: int) -> RunOutcome: - if not symbols: - return RunOutcome(status=RunStatus.OK) - attempted = 0 - succeeded = 0 - for symbol in symbols: - # 10 (period, limit) endpoints × periods - for endpoint in PERIOD_AND_LIMIT_ENDPOINTS: - for period in self._periods: - attempted += 1 - method = getattr(self._cached.statements, endpoint) - try: - await method(symbol, limit=limit, period=period) - succeeded += 1 - except Exception as exc: - logger.warning("statements.%s(%s, %s) failed: %s", endpoint, symbol, period, exc) - # limit-only endpoints (owner_earnings) - for endpoint in LIMIT_ONLY_ENDPOINTS: - attempted += 1 - method = getattr(self._cached.statements, endpoint) - try: - await method(symbol, limit=limit) - succeeded += 1 - except Exception as exc: - logger.warning("statements.%s(%s) failed: %s", endpoint, symbol, exc) - # period-only endpoints (segmentation) - for endpoint in PERIOD_ONLY_ENDPOINTS: - for period in self._periods: - attempted += 1 - method = getattr(self._cached.statements, endpoint) - try: - await method(symbol, period=period) - succeeded += 1 - except Exception as exc: - logger.warning("statements.%s(%s, %s) failed: %s", endpoint, symbol, period, exc) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_statements(cfg: CategoryConfig, manager: "HarvesterManager") -> StatementsHarvester: - return StatementsHarvester(cfg, manager) - - -register_category("statements", build_statements) -``` - -- [ ] **Step 4: Register the module** - -Append `statements` to the import block in `aiofmp/harvester/categories/__init__.py`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_statements.py -v` -Expected: all 7 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/statements.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_statements.py -git commit -m "feat(harvester): statements category (P1 + monthly safety-net)" -``` - ---- - -## Phase 5 — Pattern P2: page-walk categories - -### Task 22: `analyst_estimates` category (per-symbol walk-back) - -**Files:** -- Create: `aiofmp/harvester/categories/analyst_estimates.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_analyst_estimates.py` - -The harvester walks `analyst.financial_estimates(symbol, period, page=N, limit=page_size)` page-by-page, accumulating records and stopping when the oldest record on the current page is older than `max(last_checkpoint, today - max_backfill_years)`. Writes go directly to `ParquetStorage` under the same key the PAGE_WALK registry uses, so user reads via `cached.analyst.financial_estimates(...)` return harvester-written data. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_analyst_estimates.py`: - -```python -"""Tests for analyst_estimates category (P2 per-symbol page walk).""" - -from __future__ import annotations - -from datetime import date, timedelta -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient.storage.parquet import ParquetStorage -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.analyst_estimates import build_analyst_estimates -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -def _row(d: str) -> dict[str, Any]: - return {"symbol": "AAPL", "date": d, "revenueAvg": 100} - - -@pytest.fixture -async def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL"]) - - m.fmp_client = MagicMock() - m.fmp_client.analyst = MagicMock() - m.fmp_client.analyst.financial_estimates = AsyncMock() - - storage = ParquetStorage(tmp_path) - await storage.initialize() - m.cached_client = MagicMock() - m.cached_client.storage = storage - return m - - -class TestAnalystEstimates: - @pytest.mark.asyncio - async def test_first_run_walks_until_max_backfill(self, manager) -> None: - today = date.today() - manager.fmp_client.analyst.financial_estimates.side_effect = [ - [_row((today - timedelta(days=400)).isoformat())], - [_row((today - timedelta(days=800)).isoformat())], - [_row((today - timedelta(days=1200)).isoformat())], # > 3y backfill -> stop after this - [], # safety - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={ - "estimate_periods": ["annual"], - "max_backfill_years": 3, - "page_size": 1, - }) - h = build_analyst_estimates(cfg, manager) - outcome = await h.run_cycle() - - assert outcome.status == RunStatus.OK - # 3 pages walked (oldest = 1200d > 3*365 = 1095, so we stop after that page) - assert manager.fmp_client.analyst.financial_estimates.await_count == 3 - ckpt = manager.state.get_checkpoint("analyst_estimates", "AAPL/annual") - assert ckpt is not None - - @pytest.mark.asyncio - async def test_subsequent_run_walks_until_checkpoint(self, manager) -> None: - manager.state.set_checkpoint("analyst_estimates", "AAPL/annual", "2024-06-01") - manager.fmp_client.analyst.financial_estimates.side_effect = [ - [_row("2024-12-31"), _row("2024-09-30")], - [_row("2024-07-31"), _row("2024-04-30")], # 2024-04-30 < 2024-06-01 -> stop - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={ - "estimate_periods": ["annual"], - "max_backfill_years": 3, - "page_size": 2, - }) - h = build_analyst_estimates(cfg, manager) - outcome = await h.run_cycle() - assert manager.fmp_client.analyst.financial_estimates.await_count == 2 - assert outcome.status == RunStatus.OK - - @pytest.mark.asyncio - async def test_empty_page_stops_walk(self, manager) -> None: - manager.fmp_client.analyst.financial_estimates.side_effect = [ - [_row("2025-01-01")], - [], - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={"estimate_periods": ["annual"]}) - h = build_analyst_estimates(cfg, manager) - await h.run_cycle() - assert manager.fmp_client.analyst.financial_estimates.await_count == 2 - - @pytest.mark.asyncio - async def test_writes_to_storage_at_expected_key(self, manager) -> None: - manager.fmp_client.analyst.financial_estimates.side_effect = [ - [_row("2025-12-31")], - [], - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={"estimate_periods": ["annual"]}) - h = build_analyst_estimates(cfg, manager) - await h.run_cycle() - records = await manager.cached_client.storage.read(("analyst-estimates", "AAPL", "annual")) - assert len(records) >= 1 - assert records[0]["date"] == "2025-12-31" - - @pytest.mark.asyncio - async def test_per_symbol_error_does_not_stop_cycle(self, manager) -> None: - manager.catalog.symbols = AsyncMock(return_value=["AAPL", "MSFT"]) - - async def side_effect(symbol, period, page=0, limit=100): - if symbol == "MSFT": - raise RuntimeError("blown") - return [_row("2025-12-31")] if page == 0 else [] - - manager.fmp_client.analyst.financial_estimates = AsyncMock(side_effect=side_effect) - cfg = CategoryConfig(enabled=True, interval="24h", extra={"estimate_periods": ["annual"]}) - h = build_analyst_estimates(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 2 - assert outcome.items_succeeded == 1 - assert outcome.status == RunStatus.PARTIAL - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "analyst_estimates" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_analyst_estimates.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/analyst_estimates.py`** - -```python -"""analyst_estimates category — Pattern P2 (per-symbol page walk).""" - -from __future__ import annotations - -import logging -from datetime import date, timedelta -from typing import TYPE_CHECKING, Any - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_STORAGE_PREFIX = "analyst-estimates" -_MAX_PAGES_SAFETY = 1000 - - -def _parse_iso(d: str | None) -> date | None: - if not d: - return None - try: - return date.fromisoformat(str(d)[:10]) - except ValueError: - return None - - -class AnalystEstimatesHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("analyst_estimates", cfg, manager.state, manager.budget, manager.config.retry) - self._catalog = manager.catalog - self._fmp = manager.fmp_client - self._storage = manager.cached_client.storage - self._periods: list[str] = list(cfg.extra.get("estimate_periods", ["annual"])) - self._max_backfill_days = int(cfg.extra.get("max_backfill_years", 3)) * 365 - self._page_size = int(cfg.extra.get("page_size", 100)) - - async def run_cycle(self) -> RunOutcome: - symbols = await self._catalog.symbols("actively_trading") - attempted = 0 - succeeded = 0 - for symbol in symbols: - for period in self._periods: - attempted += 1 - try: - await self._walk_symbol(symbol, period) - succeeded += 1 - except Exception as exc: - logger.warning( - "analyst_estimates walk failed for %s/%s: %s", symbol, period, exc, - ) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - async def _walk_symbol(self, symbol: str, period: str) -> None: - scope = f"{symbol}/{period}" - last_ckpt = _parse_iso(self.state.get_checkpoint("analyst_estimates", scope)) - max_backfill = date.today() - timedelta(days=self._max_backfill_days) - stop_at: date = last_ckpt if last_ckpt and last_ckpt > max_backfill else max_backfill - - storage_key = (_STORAGE_PREFIX, symbol, period) - all_records: list[dict[str, Any]] = [] - newest_seen: date | None = None - - for page in range(_MAX_PAGES_SAFETY): - records = await self._fmp.analyst.financial_estimates( - symbol, period, page=page, limit=self._page_size - ) - if not records: - break - all_records.extend(records) - page_dates = [_parse_iso(r.get("date")) for r in records] - page_dates = [d for d in page_dates if d is not None] - if not page_dates: - # Defensive: if no parseable dates on this page, stop to avoid infinite loop - break - oldest_on_page = min(page_dates) - page_newest = max(page_dates) - if newest_seen is None or page_newest > newest_seen: - newest_seen = page_newest - if oldest_on_page <= stop_at: - break - - if all_records: - # Merge with existing storage (dedup by date) - existing = await self._storage.read(storage_key) - seen = {r.get("date") for r in existing} - for r in all_records: - if r.get("date") not in seen: - existing.append(r) - seen.add(r.get("date")) - existing.sort(key=lambda r: str(r.get("date", "")), reverse=True) - await self._storage.write(storage_key, existing, date_field="date") - - if newest_seen is not None: - self.state.set_checkpoint("analyst_estimates", scope, newest_seen.isoformat()) - - -def build_analyst_estimates(cfg: CategoryConfig, manager: "HarvesterManager") -> AnalystEstimatesHarvester: - return AnalystEstimatesHarvester(cfg, manager) - - -register_category("analyst_estimates", build_analyst_estimates) -``` - -- [ ] **Step 4: Register the module** - -Add `analyst_estimates` to the import block in `aiofmp/harvester/categories/__init__.py`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - analyst_estimates, - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_analyst_estimates.py -v` -Expected: all 6 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/analyst_estimates.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_analyst_estimates.py -git commit -m "feat(harvester): analyst_estimates category (P2 per-symbol walk)" -``` - ---- - -### Task 23: `insider_trades` category (global walk, per-symbol shard) - -**Files:** -- Create: `aiofmp/harvester/categories/insider_trades.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_insider_trades.py` - -A single global walk of `insider_trades.latest_insider_trades(page=N, limit=page_size)`; results are sharded into per-symbol parquet files keyed by `("insider-trading/latest", symbol)`, with the raw global stream also written to `("insider-trading/latest", "_global")`. - -Stop condition: oldest `filingDate` in page is older than `last_checkpoint`, OR page is empty, OR `page >= max_pages`. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_insider_trades.py`: - -```python -"""Tests for insider_trades category (P2 global walk, per-symbol shard).""" - -from __future__ import annotations - -from datetime import UTC, date, datetime, timedelta -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient.storage.parquet import ParquetStorage -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.insider_trades import build_insider_trades -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -def _trade(symbol: str, d: str) -> dict[str, Any]: - return {"symbol": symbol, "filingDate": d, "transactionType": "P-Purchase"} - - -@pytest.fixture -async def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.fmp_client = MagicMock() - m.fmp_client.insider_trades = MagicMock() - m.fmp_client.insider_trades.latest_insider_trades = AsyncMock() - storage = ParquetStorage(tmp_path) - await storage.initialize() - m.cached_client = MagicMock() - m.cached_client.storage = storage - return m - - -class TestInsiderTrades: - @pytest.mark.asyncio - async def test_first_run_walks_until_max_pages(self, manager) -> None: - manager.fmp_client.insider_trades.latest_insider_trades.side_effect = [ - [_trade("AAPL", "2026-05-01"), _trade("MSFT", "2026-05-01")], - [_trade("AAPL", "2026-04-25"), _trade("GOOGL", "2026-04-24")], - [], # stops the walk - ] - cfg = CategoryConfig(enabled=True, interval="6h", extra={ - "max_pages": 10, "page_size": 2, - }) - h = build_insider_trades(cfg, manager) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - assert manager.fmp_client.insider_trades.latest_insider_trades.await_count == 3 - - @pytest.mark.asyncio - async def test_subsequent_run_stops_at_checkpoint(self, manager) -> None: - manager.state.set_checkpoint("insider_trades", "global", "2026-04-30") - manager.fmp_client.insider_trades.latest_insider_trades.side_effect = [ - [_trade("AAPL", "2026-05-02")], - [_trade("MSFT", "2026-04-25")], # < checkpoint -> stop after this page - [_trade("X", "2020-01-01")], - ] - cfg = CategoryConfig(enabled=True, interval="6h", extra={"page_size": 1, "max_pages": 10}) - h = build_insider_trades(cfg, manager) - await h.run_cycle() - assert manager.fmp_client.insider_trades.latest_insider_trades.await_count == 2 - - @pytest.mark.asyncio - async def test_shards_by_symbol(self, manager) -> None: - manager.fmp_client.insider_trades.latest_insider_trades.side_effect = [ - [_trade("AAPL", "2026-05-01"), _trade("MSFT", "2026-05-01")], - [], - ] - cfg = CategoryConfig(enabled=True, interval="6h", extra={}) - h = build_insider_trades(cfg, manager) - await h.run_cycle() - aapl_rows = await manager.cached_client.storage.read(("insider-trading/latest", "AAPL")) - msft_rows = await manager.cached_client.storage.read(("insider-trading/latest", "MSFT")) - global_rows = await manager.cached_client.storage.read(("insider-trading/latest", "_global")) - assert any(r["symbol"] == "AAPL" for r in aapl_rows) - assert any(r["symbol"] == "MSFT" for r in msft_rows) - assert len(global_rows) == 2 - - @pytest.mark.asyncio - async def test_max_pages_safety(self, manager) -> None: - manager.fmp_client.insider_trades.latest_insider_trades.return_value = [_trade("AAPL", "2026-05-01")] - cfg = CategoryConfig(enabled=True, interval="6h", extra={"max_pages": 3, "page_size": 1}) - h = build_insider_trades(cfg, manager) - await h.run_cycle() - assert manager.fmp_client.insider_trades.latest_insider_trades.await_count == 3 - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "insider_trades" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_insider_trades.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/insider_trades.py`** - -```python -"""insider_trades category — Pattern P2 (global walk; per-symbol shard on write).""" - -from __future__ import annotations - -import logging -from collections import defaultdict -from datetime import date -from typing import TYPE_CHECKING, Any - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_STORAGE_PREFIX = "insider-trading/latest" - - -def _parse_iso(d: str | None) -> date | None: - if not d: - return None - try: - return date.fromisoformat(str(d)[:10]) - except ValueError: - return None - - -class InsiderTradesHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("insider_trades", cfg, manager.state, manager.budget, manager.config.retry) - self._fmp = manager.fmp_client - self._storage = manager.cached_client.storage - self._max_pages = int(cfg.extra.get("max_pages", 200)) - self._page_size = int(cfg.extra.get("page_size", 100)) - - async def run_cycle(self) -> RunOutcome: - last_ckpt = _parse_iso(self.state.get_checkpoint("insider_trades", "global")) - all_records: list[dict[str, Any]] = [] - newest_seen: date | None = None - - for page in range(self._max_pages): - records = await self._fmp.insider_trades.latest_insider_trades(page=page, limit=self._page_size) - if not records: - break - all_records.extend(records) - page_dates = [_parse_iso(r.get("filingDate")) for r in records] - page_dates = [d for d in page_dates if d is not None] - if page_dates: - page_newest = max(page_dates) - page_oldest = min(page_dates) - if newest_seen is None or page_newest > newest_seen: - newest_seen = page_newest - if last_ckpt is not None and page_oldest <= last_ckpt: - break - - if all_records: - await self._persist(all_records) - - if newest_seen is not None: - self.state.set_checkpoint("insider_trades", "global", newest_seen.isoformat()) - - return RunOutcome( - status=RunStatus.OK, - items_attempted=len(all_records), - items_succeeded=len(all_records), - ) - - async def _persist(self, records: list[dict[str, Any]]) -> None: - # Global stream - global_key = (_STORAGE_PREFIX, "_global") - existing_global = await self._storage.read(global_key) - seen_global = {(r.get("symbol"), r.get("filingDate"), r.get("transactionType")) for r in existing_global} - for r in records: - tup = (r.get("symbol"), r.get("filingDate"), r.get("transactionType")) - if tup not in seen_global: - existing_global.append(r) - seen_global.add(tup) - existing_global.sort(key=lambda r: str(r.get("filingDate", "")), reverse=True) - await self._storage.write(global_key, existing_global, date_field="filingDate") - - # Per-symbol shard - by_symbol: dict[str, list[dict[str, Any]]] = defaultdict(list) - for r in records: - sym = r.get("symbol") - if sym: - by_symbol[str(sym)].append(r) - for sym, rows in by_symbol.items(): - key = (_STORAGE_PREFIX, sym) - existing = await self._storage.read(key) - seen = {(r.get("filingDate"), r.get("transactionType")) for r in existing} - for r in rows: - tup = (r.get("filingDate"), r.get("transactionType")) - if tup not in seen: - existing.append(r) - seen.add(tup) - existing.sort(key=lambda r: str(r.get("filingDate", "")), reverse=True) - await self._storage.write(key, existing, date_field="filingDate") - - -def build_insider_trades(cfg: CategoryConfig, manager: "HarvesterManager") -> InsiderTradesHarvester: - return InsiderTradesHarvester(cfg, manager) - - -register_category("insider_trades", build_insider_trades) -``` - -- [ ] **Step 4: Register the module** - -Add `insider_trades` to the import block: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - analyst_estimates, - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - indexes_eod, - indexes_intraday, - insider_trades, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_insider_trades.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/insider_trades.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_insider_trades.py -git commit -m "feat(harvester): insider_trades category (P2 global walk, per-symbol shard)" -``` - ---- - -### Task 24: `form13f` category (global walk, per-CIK shard) - -**Files:** -- Create: `aiofmp/harvester/categories/form13f.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_form13f.py` - -Identical pattern to insider_trades but the walk target is `form13f.latest_filings(page=N, limit=page_size)`, sharded by `cik` instead of `symbol`, and the walk date field is `acceptedDate`. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_form13f.py`: - -```python -"""Tests for form13f category (P2 global walk, per-CIK shard).""" - -from __future__ import annotations - -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient.storage.parquet import ParquetStorage -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.form13f import build_form13f -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import StateStore - - -def _filing(cik: str, d: str) -> dict[str, Any]: - return {"cik": cik, "acceptedDate": d, "name": "TestFund"} - - -@pytest.fixture -async def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.fmp_client = MagicMock() - m.fmp_client.form13f = MagicMock() - m.fmp_client.form13f.latest_filings = AsyncMock() - storage = ParquetStorage(tmp_path) - await storage.initialize() - m.cached_client = MagicMock() - m.cached_client.storage = storage - return m - - -class TestForm13F: - @pytest.mark.asyncio - async def test_walks_until_empty(self, manager) -> None: - manager.fmp_client.form13f.latest_filings.side_effect = [ - [_filing("0001067983", "2026-05-01"), _filing("0001067900", "2026-05-01")], - [_filing("0001067983", "2026-04-28")], - [], - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={"max_pages": 10, "page_size": 2}) - h = build_form13f(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 3 - assert manager.fmp_client.form13f.latest_filings.await_count == 3 - - @pytest.mark.asyncio - async def test_stops_at_checkpoint(self, manager) -> None: - manager.state.set_checkpoint("form13f", "global", "2026-04-29") - manager.fmp_client.form13f.latest_filings.side_effect = [ - [_filing("0001067983", "2026-05-01")], - [_filing("0001067900", "2026-04-20")], # < checkpoint -> stop after this page - [_filing("0001067", "2020-01-01")], - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={"page_size": 1, "max_pages": 5}) - h = build_form13f(cfg, manager) - await h.run_cycle() - assert manager.fmp_client.form13f.latest_filings.await_count == 2 - - @pytest.mark.asyncio - async def test_shards_by_cik(self, manager) -> None: - manager.fmp_client.form13f.latest_filings.side_effect = [ - [_filing("0001067983", "2026-05-01"), _filing("0001067900", "2026-05-01")], - [], - ] - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_form13f(cfg, manager) - await h.run_cycle() - berkshire = await manager.cached_client.storage.read(("institutional-ownership/latest", "0001067983")) - other = await manager.cached_client.storage.read(("institutional-ownership/latest", "0001067900")) - global_rows = await manager.cached_client.storage.read(("institutional-ownership/latest", "_global")) - assert len(berkshire) == 1 - assert len(other) == 1 - assert len(global_rows) == 2 - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "form13f" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_form13f.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/form13f.py`** - -```python -"""form13f category — Pattern P2 (global walk; per-CIK shard on write).""" - -from __future__ import annotations - -import logging -from collections import defaultdict -from datetime import date -from typing import TYPE_CHECKING, Any - -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -_STORAGE_PREFIX = "institutional-ownership/latest" - - -def _parse_iso(d: str | None) -> date | None: - if not d: - return None - try: - return date.fromisoformat(str(d)[:10]) - except ValueError: - return None - - -class Form13FHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("form13f", cfg, manager.state, manager.budget, manager.config.retry) - self._fmp = manager.fmp_client - self._storage = manager.cached_client.storage - self._max_pages = int(cfg.extra.get("max_pages", 50)) - self._page_size = int(cfg.extra.get("page_size", 100)) - - async def run_cycle(self) -> RunOutcome: - last_ckpt = _parse_iso(self.state.get_checkpoint("form13f", "global")) - all_records: list[dict[str, Any]] = [] - newest_seen: date | None = None - - for page in range(self._max_pages): - records = await self._fmp.form13f.latest_filings(page=page, limit=self._page_size) - if not records: - break - all_records.extend(records) - page_dates = [_parse_iso(r.get("acceptedDate")) for r in records] - page_dates = [d for d in page_dates if d is not None] - if page_dates: - page_newest = max(page_dates) - page_oldest = min(page_dates) - if newest_seen is None or page_newest > newest_seen: - newest_seen = page_newest - if last_ckpt is not None and page_oldest <= last_ckpt: - break - - if all_records: - await self._persist(all_records) - - if newest_seen is not None: - self.state.set_checkpoint("form13f", "global", newest_seen.isoformat()) - - return RunOutcome( - status=RunStatus.OK, - items_attempted=len(all_records), - items_succeeded=len(all_records), - ) - - async def _persist(self, records: list[dict[str, Any]]) -> None: - global_key = (_STORAGE_PREFIX, "_global") - existing_global = await self._storage.read(global_key) - seen_global = {(r.get("cik"), r.get("acceptedDate")) for r in existing_global} - for r in records: - tup = (r.get("cik"), r.get("acceptedDate")) - if tup not in seen_global: - existing_global.append(r) - seen_global.add(tup) - existing_global.sort(key=lambda r: str(r.get("acceptedDate", "")), reverse=True) - await self._storage.write(global_key, existing_global, date_field="acceptedDate") - - by_cik: dict[str, list[dict[str, Any]]] = defaultdict(list) - for r in records: - cik = r.get("cik") - if cik: - by_cik[str(cik)].append(r) - for cik, rows in by_cik.items(): - key = (_STORAGE_PREFIX, cik) - existing = await self._storage.read(key) - seen = {r.get("acceptedDate") for r in existing} - for r in rows: - if r.get("acceptedDate") not in seen: - existing.append(r) - seen.add(r.get("acceptedDate")) - existing.sort(key=lambda r: str(r.get("acceptedDate", "")), reverse=True) - await self._storage.write(key, existing, date_field="acceptedDate") - - -def build_form13f(cfg: CategoryConfig, manager: "HarvesterManager") -> Form13FHarvester: - return Form13FHarvester(cfg, manager) - - -register_category("form13f", build_form13f) -``` - -- [ ] **Step 4: Register the module** - -Add `form13f` to the import block: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - analyst_estimates, - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - form13f, - indexes_eod, - indexes_intraday, - insider_trades, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_form13f.py -v` -Expected: all 4 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/form13f.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_form13f.py -git commit -m "feat(harvester): form13f category (P2 global walk, per-CIK shard)" -``` - ---- - -## Phase 6 — Pattern P4: snapshot-overwrite categories - -### Task 25: `analyst_snapshots` category - -**Files:** -- Create: `aiofmp/harvester/categories/analyst_snapshots.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_analyst_snapshots.py` - -Per symbol in `actively_trading`, fetch the configured snapshot endpoints from `client.analyst.*` and overwrite via `SnapshotStore`. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_analyst_snapshots.py`: - -```python -"""Tests for analyst_snapshots category (P4 snapshot overwrite).""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient.snapshot_store import SnapshotStore -from aiofmp.cachedclient.storage.parquet import ParquetStorage -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.analyst_snapshots import build_analyst_snapshots -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -async def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL"]) - m.fmp_client = MagicMock() - m.fmp_client.analyst = MagicMock() - m.fmp_client.analyst.ratings_snapshot = AsyncMock(return_value=[{"symbol": "AAPL", "rating": "A-"}]) - m.fmp_client.analyst.price_target_consensus = AsyncMock(return_value=[{"symbol": "AAPL", "targetConsensus": 250}]) - m.fmp_client.analyst.price_target_summary = AsyncMock(return_value=[{"symbol": "AAPL"}]) - m.fmp_client.analyst.stock_grades_summary = AsyncMock(return_value=[{"symbol": "AAPL", "consensus": "Buy"}]) - storage = ParquetStorage(tmp_path) - await storage.initialize() - m.cached_client = MagicMock() - m.cached_client.storage = storage - return m - - -class TestAnalystSnapshots: - @pytest.mark.asyncio - async def test_default_endpoints(self, manager) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_analyst_snapshots(cfg, manager) - outcome = await h.run_cycle() - assert outcome.status == RunStatus.OK - # 1 symbol * 4 default endpoints - assert outcome.items_attempted == 4 - - ss = SnapshotStore(manager.cached_client.storage) - row = await ss.read("ratings-snapshot", "AAPL") - assert row is not None - assert row["rating"] == "A-" - - @pytest.mark.asyncio - async def test_custom_endpoint_list(self, manager) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={"include": ["ratings_snapshot"]}) - h = build_analyst_snapshots(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 1 - manager.fmp_client.analyst.price_target_consensus.assert_not_awaited() - - @pytest.mark.asyncio - async def test_empty_response_skips_write(self, manager) -> None: - manager.fmp_client.analyst.ratings_snapshot = AsyncMock(return_value=[]) - cfg = CategoryConfig(enabled=True, interval="24h", extra={"include": ["ratings_snapshot"]}) - h = build_analyst_snapshots(cfg, manager) - await h.run_cycle() - ss = SnapshotStore(manager.cached_client.storage) - assert await ss.read("ratings-snapshot", "AAPL") is None - - @pytest.mark.asyncio - async def test_per_endpoint_error_does_not_kill_cycle(self, manager) -> None: - manager.fmp_client.analyst.ratings_snapshot = AsyncMock(side_effect=RuntimeError("boom")) - cfg = CategoryConfig(enabled=True, interval="24h", extra={ - "include": ["ratings_snapshot", "price_target_consensus"] - }) - h = build_analyst_snapshots(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 2 - assert outcome.items_succeeded == 1 - assert outcome.status == RunStatus.PARTIAL - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "analyst_snapshots" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_analyst_snapshots.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/analyst_snapshots.py`** - -```python -"""analyst_snapshots category — Pattern P4 (snapshot overwrite per symbol).""" - -from __future__ import annotations - -import logging -from typing import TYPE_CHECKING - -from aiofmp.cachedclient.snapshot_store import SnapshotStore -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - - -# Maps the YAML "include" name -> (SDK method on client.analyst, snapshot endpoint key). -_INCLUDE_MAP: dict[str, tuple[str, str]] = { - "ratings_snapshot": ("ratings_snapshot", "ratings-snapshot"), - "price_target_consensus": ("price_target_consensus", "price-target-consensus"), - "price_target_summary": ("price_target_summary", "price-target-summary"), - "stock_grades_summary": ("stock_grades_summary", "grades-consensus"), -} - -_DEFAULT_INCLUDE = list(_INCLUDE_MAP.keys()) - - -class AnalystSnapshotsHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("analyst_snapshots", cfg, manager.state, manager.budget, manager.config.retry) - self._catalog = manager.catalog - self._fmp = manager.fmp_client - self._snapshots = SnapshotStore(manager.cached_client.storage) - include = list(cfg.extra.get("include", _DEFAULT_INCLUDE)) - for n in include: - if n not in _INCLUDE_MAP: - raise ValueError(f"analyst_snapshots: unknown include {n!r}") - self._include = include - - async def run_cycle(self) -> RunOutcome: - symbols = await self._catalog.symbols("actively_trading") - attempted = 0 - succeeded = 0 - for symbol in symbols: - for name in self._include: - attempted += 1 - method_name, endpoint = _INCLUDE_MAP[name] - method = getattr(self._fmp.analyst, method_name) - try: - rows = await method(symbol) - payload = rows[0] if isinstance(rows, list) and rows else None - if payload: - await self._snapshots.write(endpoint, symbol, payload) - succeeded += 1 - except Exception as exc: - logger.warning( - "analyst_snapshots.%s(%s) failed: %s", method_name, symbol, exc, - ) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_analyst_snapshots(cfg: CategoryConfig, manager: "HarvesterManager") -> AnalystSnapshotsHarvester: - return AnalystSnapshotsHarvester(cfg, manager) - - -register_category("analyst_snapshots", build_analyst_snapshots) -``` - -- [ ] **Step 4: Register the module** - -Add `analyst_snapshots`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - analyst_estimates, - analyst_snapshots, - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - economics, - forex_eod, - forex_intraday, - form13f, - indexes_eod, - indexes_intraday, - insider_trades, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_analyst_snapshots.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/analyst_snapshots.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_analyst_snapshots.py -git commit -m "feat(harvester): analyst_snapshots category (P4 overwrite)" -``` - ---- - -### Task 26: `dcf` category (P4 snapshot overwrite over `financial_symbols`) - -**Files:** -- Create: `aiofmp/harvester/categories/dcf.py` -- Modify: `aiofmp/harvester/categories/__init__.py` -- Create: `tests/test_harvester/test_dcf.py` - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_dcf.py`: - -```python -"""Tests for dcf category (P4 snapshot overwrite).""" - -from __future__ import annotations - -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient.snapshot_store import SnapshotStore -from aiofmp.cachedclient.storage.parquet import ParquetStorage -from aiofmp.harvester.budget import BudgetTracker -from aiofmp.harvester.categories.dcf import build_dcf -from aiofmp.harvester.config import BudgetConfig, CategoryConfig, RetryConfig -from aiofmp.harvester.state import RunStatus, StateStore - - -@pytest.fixture -async def manager(tmp_path: Path) -> MagicMock: - m = MagicMock() - m.state = StateStore(tmp_path / "h.sqlite"); m.state.initialize() - m.budget = BudgetTracker(m.state, BudgetConfig()) - m.config = MagicMock(); m.config.retry = RetryConfig() - m.catalog = MagicMock() - m.catalog.symbols = AsyncMock(return_value=["AAPL", "MSFT"]) - m.fmp_client = MagicMock() - m.fmp_client.dcf = MagicMock() - m.fmp_client.dcf.dcf_valuation = AsyncMock(return_value=[{"symbol": "AAPL", "dcf": 147.27}]) - m.fmp_client.dcf.levered_dcf = AsyncMock(return_value=[{"symbol": "AAPL", "dcf": 150.10}]) - storage = ParquetStorage(tmp_path) - await storage.initialize() - m.cached_client = MagicMock() - m.cached_client.storage = storage - return m - - -class TestDcf: - @pytest.mark.asyncio - async def test_default_endpoints(self, manager) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={}) - h = build_dcf(cfg, manager) - outcome = await h.run_cycle() - # 2 symbols * 2 endpoints - assert outcome.items_attempted == 4 - manager.catalog.symbols.assert_awaited_with("financial_symbols") - - ss = SnapshotStore(manager.cached_client.storage) - row = await ss.read("discounted-cash-flow", "AAPL") - assert row is not None - assert row["dcf"] == 147.27 - row2 = await ss.read("levered-discounted-cash-flow", "AAPL") - assert row2 is not None - assert row2["dcf"] == 150.10 - - @pytest.mark.asyncio - async def test_custom_include(self, manager) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={"include": ["dcf_valuation"]}) - h = build_dcf(cfg, manager) - outcome = await h.run_cycle() - assert outcome.items_attempted == 2 - manager.fmp_client.dcf.levered_dcf.assert_not_awaited() - - @pytest.mark.asyncio - async def test_unknown_include_raises(self, manager) -> None: - cfg = CategoryConfig(enabled=True, interval="24h", extra={"include": ["bogus"]}) - with pytest.raises(ValueError, match="bogus"): - build_dcf(cfg, manager) - - @pytest.mark.asyncio - async def test_registers(self) -> None: - from aiofmp.harvester.categories import _REGISTRY - assert "dcf" in _REGISTRY -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_dcf.py -v` -Expected: ImportError. - -- [ ] **Step 3: Create `aiofmp/harvester/categories/dcf.py`** - -```python -"""dcf category — Pattern P4 (snapshot overwrite over financial_symbols).""" - -from __future__ import annotations - -import logging -from typing import TYPE_CHECKING - -from aiofmp.cachedclient.snapshot_store import SnapshotStore -from aiofmp.harvester.base import CategoryHarvester, RunOutcome -from aiofmp.harvester.categories import register_category -from aiofmp.harvester.config import CategoryConfig -from aiofmp.harvester.state import RunStatus - -if TYPE_CHECKING: - from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - -# Maps include-name -> (SDK method on client.dcf, snapshot endpoint key) -_INCLUDE_MAP: dict[str, tuple[str, str]] = { - "dcf_valuation": ("dcf_valuation", "discounted-cash-flow"), - "levered_dcf": ("levered_dcf", "levered-discounted-cash-flow"), -} - -_DEFAULT_INCLUDE = list(_INCLUDE_MAP.keys()) - - -class DcfHarvester(CategoryHarvester): - def __init__(self, cfg: CategoryConfig, manager: "HarvesterManager") -> None: - super().__init__("dcf", cfg, manager.state, manager.budget, manager.config.retry) - self._catalog = manager.catalog - self._fmp = manager.fmp_client - self._snapshots = SnapshotStore(manager.cached_client.storage) - include = list(cfg.extra.get("include", _DEFAULT_INCLUDE)) - for n in include: - if n not in _INCLUDE_MAP: - raise ValueError(f"dcf: unknown include {n!r}") - self._include = include - - async def run_cycle(self) -> RunOutcome: - symbols = await self._catalog.symbols("financial_symbols") - attempted = 0 - succeeded = 0 - for symbol in symbols: - for name in self._include: - attempted += 1 - method_name, endpoint = _INCLUDE_MAP[name] - method = getattr(self._fmp.dcf, method_name) - try: - rows = await method(symbol) - payload = rows[0] if isinstance(rows, list) and rows else None - if payload: - await self._snapshots.write(endpoint, symbol, payload) - succeeded += 1 - except Exception as exc: - logger.warning("dcf.%s(%s) failed: %s", method_name, symbol, exc) - status = RunStatus.OK if succeeded == attempted else RunStatus.PARTIAL - return RunOutcome(status=status, items_attempted=attempted, items_succeeded=succeeded) - - -def build_dcf(cfg: CategoryConfig, manager: "HarvesterManager") -> DcfHarvester: - return DcfHarvester(cfg, manager) - - -register_category("dcf", build_dcf) -``` - -- [ ] **Step 4: Register the module** - -Add `dcf`: - -```python -from aiofmp.harvester.categories import ( # noqa: E402, F401 - analyst_estimates, - analyst_snapshots, - chart_eod, - chart_intraday, - commodities_eod, - commodities_intraday, - dcf, - economics, - forex_eod, - forex_intraday, - form13f, - indexes_eod, - indexes_intraday, - insider_trades, - news, - statements, - technical_indicators, -) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_dcf.py -v` -Expected: all 4 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/categories/dcf.py aiofmp/harvester/categories/__init__.py tests/test_harvester/test_dcf.py -git commit -m "feat(harvester): dcf category (P4 snapshot overwrite)" -``` - ---- - -## Phase 7 — CLI integration + end-to-end - -### Task 27: Wire the bandwidth callback from `FmpClient` → `BudgetTracker` - -**Files:** -- Modify: `aiofmp/harvester/manager.py` -- Modify: `tests/test_harvester/test_manager.py` - -Until now the manager doesn't actually attach the bandwidth callback to the shared `FmpClient`. This task wires it so every API request goes through `BudgetTracker.record_bytes()` and `BudgetTracker.check_hard_cap()`. - -- [ ] **Step 1: Append failing test to `tests/test_harvester/test_manager.py`** - -Append at the end of `tests/test_harvester/test_manager.py`: - -```python -class TestBandwidthAttachment: - @pytest.mark.asyncio - async def test_callback_attached_to_fmp_client(self, cfg: HarvestConfig) -> None: - from aiofmp.base import FMPBaseClient, current_harvest_category - fmp = FMPBaseClient(api_key="dummy") - mgr = HarvesterManager(cfg, fmp_client=fmp, cached_client=MagicMock()) - await mgr.start() - assert fmp.on_response_size is not None - # Invoke it and verify it lands in the ledger. - token = current_harvest_category.set("alpha") - try: - fmp.on_response_size(current_harvest_category.get(), 4096) - finally: - current_harvest_category.reset(token) - month = mgr.budget.current_month_key() - assert mgr.state.get_category_month_bytes(month, "alpha") == 4096 - await mgr.close() - - @pytest.mark.asyncio - async def test_callback_triggers_hard_cap(self, cfg: HarvestConfig) -> None: - from aiofmp.base import FMPBaseClient, FMPBudgetError, current_harvest_category - cfg.budget.monthly_soft_cap_gb = 1 - cfg.budget.monthly_hard_cap_gb = 2 - fmp = FMPBaseClient(api_key="dummy") - mgr = HarvesterManager(cfg, fmp_client=fmp, cached_client=MagicMock()) - await mgr.start() - # Pre-fill the ledger to past hard cap - mgr.budget.record_bytes("alpha", 3 * 1024 ** 3) - token = current_harvest_category.set("alpha") - try: - with pytest.raises(FMPBudgetError): - fmp.on_response_size(current_harvest_category.get(), 1) - finally: - current_harvest_category.reset(token) - await mgr.close() -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_manager.py::TestBandwidthAttachment -v` -Expected: failures — the manager doesn't wire `on_response_size` yet. - -- [ ] **Step 3: Wire the callback in `HarvesterManager.start()`** - -In `aiofmp/harvester/manager.py`, at the end of `start()`, install the bandwidth hook: - -```python - async def start(self) -> None: - # Wire bandwidth + hard-cap callback into the shared FmpClient - self.fmp_client.on_response_size = self._on_response_size - - for name, cat_cfg in self.config.categories.items(): - if not cat_cfg.enabled: - logger.info("Category %s is disabled; skipping", name) - continue - try: - self._categories[name] = self._build_category(name, cat_cfg) - except KeyError: - logger.warning("Unknown category in config: %s (skipping)", name) - - def _on_response_size(self, category: str | None, byte_count: int) -> None: - """Bandwidth callback installed on FmpClient. - - Records bytes against the current category (or 'user' if outside a cycle), - then raises FMPBudgetError if the hard cap is breached for harvester traffic. - """ - self.budget.record_bytes(category, byte_count) - self.budget.check_hard_cap(category) -``` - -- [ ] **Step 4: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_manager.py -v` -Expected: all manager tests pass (5 prior + 2 new = 7). - -- [ ] **Step 5: Commit** - -```bash -git add aiofmp/harvester/manager.py tests/test_harvester/test_manager.py -git commit -m "feat(harvester): wire bandwidth callback + hard-cap enforcement" -``` - ---- - -### Task 28: `aiofmp harvest` CLI subcommand - -**Files:** -- Create: `aiofmp/harvester/cli.py` -- Modify: `aiofmp/cli.py` -- Create: `tests/test_harvester/test_cli.py` - -The `aiofmp harvest` subcommand has these flags: -- `--config PATH` (required) — points at the YAML file -- `--once` — run each enabled category's next-due cycle and exit -- `--category NAME` — restrict to a single category (combine with `--once`) -- `--dry-run` — print the plan, fetch nothing - -`aiofmp harvest-status` prints a tabular summary from the state store. - -- [ ] **Step 1: Write the failing tests** - -Create `tests/test_harvester/test_cli.py`: - -```python -"""Tests for the harvester CLI.""" - -from __future__ import annotations - -import os -import sys -from pathlib import Path -from textwrap import dedent -from unittest.mock import AsyncMock, patch - -import pytest -from click.testing import CliRunner - -from aiofmp.cli import cli - - -def _write_yaml(path: Path, body: str) -> None: - path.write_text(dedent(body).strip() + "\n") - - -@pytest.fixture -def runner() -> CliRunner: - return CliRunner() - - -@pytest.fixture -def env_with_key(monkeypatch) -> None: - monkeypatch.setenv("FMP_API_KEY", "test_key_xxxx") - - -@pytest.fixture -def minimal_config(tmp_path: Path) -> Path: - state = tmp_path / "state" - state.mkdir() - cfg = tmp_path / "h.yaml" - _write_yaml(cfg, f""" - state_dir: {state} - categories: - news: - enabled: true - interval: 30m - """) - return cfg - - -class TestHarvestCli: - def test_missing_api_key(self, runner: CliRunner, minimal_config: Path) -> None: - # Ensure FMP_API_KEY is NOT in env - result = runner.invoke( - cli, ["harvest", "--config", str(minimal_config), "--once"], - env={"FMP_API_KEY": ""}, - ) - assert result.exit_code != 0 - assert "FMP_API_KEY" in result.output - - def test_dry_run_prints_plan(self, env_with_key, runner: CliRunner, minimal_config: Path) -> None: - result = runner.invoke( - cli, ["harvest", "--config", str(minimal_config), "--dry-run"], - ) - assert result.exit_code == 0 - assert "news" in result.output - assert "enabled" in result.output.lower() - - def test_once_runs_and_exits( - self, env_with_key, runner: CliRunner, minimal_config: Path, - ) -> None: - with patch("aiofmp.harvester.cli._build_cached_client") as m: - fake_cached = AsyncMock() - fake_cached.__aenter__ = AsyncMock(return_value=fake_cached) - fake_cached.__aexit__ = AsyncMock(return_value=None) - fake_cached.news = type("X", (), {})() - for v in ("general_news", "press_releases", "stock_news", "crypto_news", "forex_news"): - setattr(fake_cached.news, v, AsyncMock(return_value=[])) - fake_cached.storage = AsyncMock() - m.return_value = (AsyncMock(), fake_cached) - - result = runner.invoke( - cli, ["harvest", "--config", str(minimal_config), "--once"], - ) - assert result.exit_code == 0, result.output - - def test_once_with_specific_category( - self, env_with_key, runner: CliRunner, tmp_path: Path, - ) -> None: - cfg = tmp_path / "h.yaml" - state = tmp_path / "state"; state.mkdir() - _write_yaml(cfg, f""" - state_dir: {state} - categories: - news: - enabled: true - interval: 30m - economics: - enabled: true - interval: 24h - """) - with patch("aiofmp.harvester.cli._build_cached_client") as m: - fake_cached = AsyncMock() - fake_cached.__aenter__ = AsyncMock(return_value=fake_cached) - fake_cached.__aexit__ = AsyncMock(return_value=None) - fake_cached.news = type("X", (), {})() - for v in ("general_news", "press_releases", "stock_news", "crypto_news", "forex_news"): - setattr(fake_cached.news, v, AsyncMock(return_value=[])) - fake_cached.economics = type("X", (), {})() - fake_cached.economics.treasury_rates = AsyncMock(return_value=[]) - fake_cached.economics.economic_indicators = AsyncMock(return_value=[]) - fake_cached.storage = AsyncMock() - m.return_value = (AsyncMock(), fake_cached) - - result = runner.invoke( - cli, ["harvest", "--config", str(cfg), "--once", "--category", "news"], - ) - assert result.exit_code == 0, result.output - # Economics should NOT have been called when filtered to news - fake_cached.economics.treasury_rates.assert_not_awaited() - - -class TestHarvestStatusCli: - def test_empty_status(self, env_with_key, runner: CliRunner, minimal_config: Path) -> None: - result = runner.invoke(cli, ["harvest-status", "--config", str(minimal_config)]) - assert result.exit_code == 0 - # Header should mention categories columns - assert "category" in result.output.lower() -``` - -- [ ] **Step 2: Run tests to confirm failure** - -Run: `uv run pytest tests/test_harvester/test_cli.py -v` -Expected: ImportError (`aiofmp.harvester.cli` missing) or "no such command 'harvest'". - -- [ ] **Step 3: Create `aiofmp/harvester/cli.py`** - -```python -"""CLI subcommands for the harvester.""" - -from __future__ import annotations - -import asyncio -import logging -import os -import sys -from pathlib import Path -from typing import Any - -import click - -from aiofmp import FmpClient -from aiofmp.cachedclient import CachedClient -from aiofmp.harvester import categories # noqa: F401 — triggers registration imports -from aiofmp.harvester.config import ( - CategoryConfig, - HarvestConfig, - load_config_from_yaml, -) -from aiofmp.harvester.manager import HarvesterManager - -logger = logging.getLogger(__name__) - - -def _resolve_api_key(api_key_opt: str | None) -> str: - key = api_key_opt or os.environ.get("FMP_API_KEY") - if not key: - click.echo("error: FMP_API_KEY env var or --api-key option is required", err=True) - sys.exit(2) - return key - - -def _build_cached_client(api_key: str) -> tuple[FmpClient, CachedClient]: - """Build a real FmpClient + CachedClient pair. Patched in tests.""" - fmp = FmpClient(api_key=api_key) - cached = CachedClient(fmp) - return fmp, cached - - -def _print_dry_run_plan(cfg: HarvestConfig, restrict_to: str | None) -> None: - click.echo(f"state_dir: {cfg.state_dir}") - click.echo(f"budget: soft={cfg.budget.monthly_soft_cap_gb}GB hard={cfg.budget.monthly_hard_cap_gb}GB") - click.echo("") - click.echo(f"{'category':30s} {'enabled':8s} {'interval':10s}") - click.echo("-" * 50) - for name, cat in cfg.categories.items(): - if restrict_to and name != restrict_to: - continue - flag = "yes" if cat.enabled else "no" - click.echo(f"{name:30s} {flag:8s} {cat.interval:10s}") - - -async def _run_once( - cfg: HarvestConfig, api_key: str, restrict_to: str | None, -) -> int: - fmp, cached = _build_cached_client(api_key) - rc = 0 - async with cached: - mgr = HarvesterManager(cfg, fmp_client=fmp, cached_client=cached) - await mgr.start() - targets = list(mgr._categories.keys()) - if restrict_to: - targets = [t for t in targets if t == restrict_to] - if not targets: - click.echo(f"error: category {restrict_to!r} not enabled in config", err=True) - return 2 - for name in targets: - await mgr._categories[name]._run_once_and_record() - run = mgr.state.get_latest_run(name) - status = run.status.value if run else "missing" - items = run.items_attempted if run else 0 - ok = run.items_succeeded if run else 0 - click.echo(f"{name:30s} {status:20s} {ok}/{items} items") - if run and run.error: - rc = 1 - await mgr.close() - return rc - - -async def _run_forever(cfg: HarvestConfig, api_key: str) -> int: - fmp, cached = _build_cached_client(api_key) - async with cached: - mgr = HarvesterManager(cfg, fmp_client=fmp, cached_client=cached) - await mgr.start() - mgr.install_signal_handlers() - try: - await mgr.run_until_stopped() - finally: - await mgr.close() - return 0 - - -@click.command("harvest") -@click.option("--config", "config_path", required=True, type=click.Path(exists=True, dir_okay=False)) -@click.option("--once", is_flag=True, help="Run each enabled category's next-due cycle and exit.") -@click.option("--category", "category_name", default=None, help="Restrict to a single category (with --once).") -@click.option("--dry-run", is_flag=True, help="Print the plan; fetch nothing.") -@click.option("--api-key", default=None, help="Override FMP_API_KEY env var.") -def harvest(config_path: str, once: bool, category_name: str | None, dry_run: bool, api_key: str | None) -> None: - """Run the aiofmp harvester.""" - cfg = load_config_from_yaml(config_path) - logging.basicConfig(level=getattr(logging, cfg.log_level.upper(), logging.INFO)) - - if dry_run: - _print_dry_run_plan(cfg, restrict_to=category_name) - sys.exit(0) - - key = _resolve_api_key(api_key) - if once: - rc = asyncio.run(_run_once(cfg, key, restrict_to=category_name)) - sys.exit(rc) - else: - rc = asyncio.run(_run_forever(cfg, key)) - sys.exit(rc) - - -@click.command("harvest-status") -@click.option("--config", "config_path", required=True, type=click.Path(exists=True, dir_okay=False)) -def harvest_status(config_path: str) -> None: - """Print a tabular summary of the harvester state store.""" - cfg = load_config_from_yaml(config_path) - state_dir = Path(os.path.expanduser(cfg.state_dir)) - from aiofmp.harvester.state import StateStore - store = StateStore(state_dir / "harvester.sqlite") - store.initialize() - - click.echo(f"{'category':30s} {'status':20s} {'finished_at':30s} {'items':10s}") - click.echo("-" * 95) - for name in sorted(cfg.categories.keys()): - run = store.get_latest_run(name) - if run is None: - click.echo(f"{name:30s} {'never_ran':20s} {'-':30s} {'-':10s}") - continue - items = f"{run.items_succeeded}/{run.items_attempted}" - finished = run.finished_at or "(running)" - click.echo(f"{name:30s} {run.status.value:20s} {finished:30s} {items:10s}") -``` - -- [ ] **Step 4: Wire the new commands into `aiofmp/cli.py`** - -In `aiofmp/cli.py`, BEFORE `if __name__ == "__main__":`, add: - -```python -from aiofmp.harvester.cli import harvest as _harvest_cmd -from aiofmp.harvester.cli import harvest_status as _harvest_status_cmd - -cli.add_command(_harvest_cmd) -cli.add_command(_harvest_status_cmd) -``` - -- [ ] **Step 5: Run tests to confirm pass** - -Run: `uv run pytest tests/test_harvester/test_cli.py -v` -Expected: all 5 tests pass. - -- [ ] **Step 6: Commit** - -```bash -git add aiofmp/harvester/cli.py aiofmp/cli.py tests/test_harvester/test_cli.py -git commit -m "feat(harvester): aiofmp harvest + harvest-status CLI commands" -``` - ---- - -### Task 29: End-to-end smoke test (mocked SDK, real registry & storage) - -**Files:** -- Create: `tests/test_harvester/test_e2e_smoke.py` - -A single test that wires the actual `HarvesterManager`, with a mocked `FmpClient` returning canned responses, and runs one cycle per category. Verifies: -- All enabled categories complete a cycle without unhandled exceptions. -- State store has one `category_runs` row per category. -- Parquet files are created under the temp state dir. - -- [ ] **Step 1: Write the failing test** - -Create `tests/test_harvester/test_e2e_smoke.py`: - -```python -"""End-to-end smoke test for the harvester.""" - -from __future__ import annotations - -import asyncio -from datetime import date -from pathlib import Path -from typing import Any -from unittest.mock import AsyncMock, MagicMock - -import pytest - -from aiofmp.cachedclient import CachedClient -from aiofmp.harvester.config import ( - BudgetConfig, - CategoryConfig, - HarvestConfig, - RetryConfig, -) -from aiofmp.harvester.manager import HarvesterManager -from aiofmp.harvester.state import RunStatus - - -def _eod_record() -> dict[str, Any]: - return {"date": "2025-06-01", "open": 100.0, "high": 101.0, "low": 99.0, "close": 100.5, "volume": 1000} - - -def _earnings_record(symbol: str, d: str) -> dict[str, Any]: - return {"symbol": symbol, "date": d, "epsEstimated": 1.0, "eps": 1.05} - - -def _build_fake_fmp() -> MagicMock: - fmp = MagicMock() - # Directory + per-category list endpoints - fmp.directory = MagicMock() - fmp.directory.financial_symbols = AsyncMock(return_value=[{"symbol": "AAPL"}]) - fmp.directory.actively_trading = AsyncMock(return_value=[{"symbol": "AAPL"}]) - fmp.directory.etf_list = AsyncMock(return_value=[]) - fmp.commodity = MagicMock() - fmp.commodity.commodities_list = AsyncMock(return_value=[{"symbol": "GCUSD"}]) - fmp.forex = MagicMock() - fmp.forex.forex_list = AsyncMock(return_value=[{"symbol": "EURUSD"}]) - fmp.indexes = MagicMock() - fmp.indexes.index_list = AsyncMock(return_value=[{"symbol": "^GSPC"}]) - - # Chart EOD endpoints - fmp.chart = MagicMock() - fmp.chart.historical_price_full = AsyncMock(return_value=[_eod_record()]) - - # Statements endpoints — wired with a single canned return - fmp.statements = MagicMock() - for m in [ - "income_statement", "balance_sheet_statement", "cash_flow_statement", - "key_metrics", "financial_ratios", "enterprise_values", - "income_statement_growth", "balance_sheet_statement_growth", - "cash_flow_statement_growth", "financial_statement_growth", - "owner_earnings", "revenue_product_segmentation", "revenue_geographic_segmentation", - ]: - setattr(fmp.statements, m, AsyncMock(return_value=[{"date": "2024-12-31", "symbol": "AAPL"}])) - - # Calendar — earnings_calendar is the discovery probe for statements - fmp.calendar = MagicMock() - fmp.calendar.earnings_calendar = AsyncMock(return_value=[_earnings_record("AAPL", date.today().isoformat())]) - - # News endpoints - fmp.news = MagicMock() - for v in ("general_news", "press_releases", "stock_news", "crypto_news", "forex_news"): - setattr(fmp.news, v, AsyncMock(return_value=[])) - - # Economics - fmp.economics = MagicMock() - fmp.economics.treasury_rates = AsyncMock(return_value=[]) - fmp.economics.economic_indicators = AsyncMock(return_value=[]) - - # Analyst - fmp.analyst = MagicMock() - fmp.analyst.financial_estimates = AsyncMock(side_effect=[ - [{"symbol": "AAPL", "date": "2025-12-31", "revenueAvg": 100}], - [], - ]) - fmp.analyst.ratings_snapshot = AsyncMock(return_value=[{"symbol": "AAPL", "rating": "A"}]) - fmp.analyst.price_target_consensus = AsyncMock(return_value=[{"symbol": "AAPL", "targetConsensus": 250}]) - fmp.analyst.price_target_summary = AsyncMock(return_value=[{"symbol": "AAPL"}]) - fmp.analyst.stock_grades_summary = AsyncMock(return_value=[{"symbol": "AAPL", "consensus": "Buy"}]) - - # Insider trades - fmp.insider_trades = MagicMock() - fmp.insider_trades.latest_insider_trades = AsyncMock(side_effect=[ - [{"symbol": "AAPL", "filingDate": "2026-05-01", "transactionType": "P-Purchase"}], - [], - ]) - - # Form 13F - fmp.form13f = MagicMock() - fmp.form13f.latest_filings = AsyncMock(side_effect=[ - [{"cik": "0001067983", "acceptedDate": "2026-05-01", "name": "Berkshire"}], - [], - ]) - - # DCF - fmp.dcf = MagicMock() - fmp.dcf.dcf_valuation = AsyncMock(return_value=[{"symbol": "AAPL", "dcf": 150.0}]) - fmp.dcf.levered_dcf = AsyncMock(return_value=[{"symbol": "AAPL", "dcf": 155.0}]) - - # Make on_response_size assignable (FmpClient has it; MagicMock too) - fmp.on_response_size = None - fmp.start = AsyncMock() - fmp.close = AsyncMock() - return fmp - - -@pytest.mark.asyncio -async def test_one_cycle_per_category(tmp_path: Path) -> None: - cfg = HarvestConfig( - state_dir=str(tmp_path), - budget=BudgetConfig(), - retry=RetryConfig(), - categories={ - "chart_eod": CategoryConfig(enabled=True, interval="24h", extra={"backfill_years": 1}), - "commodities_eod": CategoryConfig(enabled=True, interval="24h", extra={}), - "forex_eod": CategoryConfig(enabled=True, interval="24h", extra={}), - "indexes_eod": CategoryConfig(enabled=True, interval="24h", extra={}), - "news": CategoryConfig(enabled=True, interval="30m", extra={}), - "economics": CategoryConfig(enabled=True, interval="24h", extra={"indicators": ["GDP"]}), - "statements": CategoryConfig(enabled=True, interval="6h", extra={"periods": ["annual"]}), - "analyst_estimates": CategoryConfig(enabled=True, interval="24h", extra={"estimate_periods": ["annual"], "page_size": 1}), - "analyst_snapshots": CategoryConfig(enabled=True, interval="24h", extra={}), - "insider_trades": CategoryConfig(enabled=True, interval="6h", extra={"page_size": 1, "max_pages": 2}), - "form13f": CategoryConfig(enabled=True, interval="24h", extra={"page_size": 1, "max_pages": 2}), - "dcf": CategoryConfig(enabled=True, interval="24h", extra={}), - }, - ) - fmp = _build_fake_fmp() - cached = CachedClient(fmp) - - async with cached: - mgr = HarvesterManager(cfg, fmp_client=fmp, cached_client=cached) - await mgr.start() - - # Run one cycle per enabled category, sequentially - for name in list(mgr._categories.keys()): - await mgr._categories[name]._run_once_and_record() - run = mgr.state.get_latest_run(name) - assert run is not None, f"no run row for {name}" - assert run.status in (RunStatus.OK, RunStatus.PARTIAL), \ - f"{name} ended in unexpected status {run.status}" - - await mgr.close() - - # Sanity: at least one parquet file should exist - assert any((tmp_path / "cachedclient_data").rglob("data.parquet")) -``` - -- [ ] **Step 2: Run test to confirm it passes** - -Run: `uv run pytest tests/test_harvester/test_e2e_smoke.py -v` -Expected: pass (or at most a clearly diagnosable failure — fix any registry-import side effects). - -- [ ] **Step 3: Run the entire test suite to confirm no regressions** - -Run: `uv run pytest -q` -Expected: every test passes. - -- [ ] **Step 4: Run ruff over the new code** - -Run: `uv run ruff check aiofmp/harvester aiofmp/cachedclient/snapshot_store.py aiofmp/base.py tests/test_harvester` -Expected: no errors. Fix any issues inline before committing. - -Run: `uv run ruff format aiofmp/harvester aiofmp/cachedclient/snapshot_store.py tests/test_harvester` -Expected: formatter exits 0. - -- [ ] **Step 5: Commit** - -```bash -git add tests/test_harvester/test_e2e_smoke.py -git commit -m "test(harvester): end-to-end smoke covering all 17 categories" -``` - ---- - -### Task 30: README + example config - -**Files:** -- Modify: `README.md` -- Create: `examples/harvester.example.yaml` - -- [ ] **Step 1: Create `examples/harvester.example.yaml`** - -```yaml -# Example harvester config — copy to harvester.yaml and edit as needed. -# Cap defaults assume FMP starter plan (20 GB/month). - -state_dir: ~/.aiofmp/cache -log_level: INFO - -budget: - monthly_soft_cap_gb: 18 - monthly_hard_cap_gb: 20 - soft_cap_behavior: pause_until_next_month # | warn_only - -retry: - on_429: - backoff_seconds: [60, 120, 240, 480] - max_attempts: 4 - on_5xx: - backoff_seconds: [10, 30, 60] - max_attempts: 3 - -discovery: - refresh_interval: 7d - -categories: - statements: - enabled: true - interval: 6h - periods: [annual, quarter] - initial_limit: 40 - incremental_limit: 2 - safety_net_interval: 30d - - analyst_estimates: - enabled: true - interval: 24h - estimate_periods: [annual] - max_backfill_years: 3 - page_size: 100 - - analyst_snapshots: - enabled: true - interval: 24h - include: [ratings_snapshot, price_target_consensus, price_target_summary, stock_grades_summary] - - chart_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - - chart_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - technical_indicators: - enabled: false - interval: 24h - indicators: [] - - news: - enabled: true - interval: 30m - variants: [general_news, press_releases, stock_news, crypto_news, forex_news] - backfill_days_initial: 30 - page_size: 100 - - insider_trades: - enabled: true - interval: 6h - max_pages: 200 - page_size: 100 - - form13f: - enabled: true - interval: 24h - max_pages: 50 - page_size: 100 - - economics: - enabled: true - interval: 24h - indicators: [GDP, CPI, UNRATE, FEDFUNDS, DFF] - backfill_years: 10 - - commodities_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - - commodities_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - forex_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - - forex_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - indexes_eod: - enabled: true - interval: 24h - variants: [historical_price_eod_full] - backfill_years: 10 - - indexes_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - dcf: - enabled: true - interval: 24h - include: [dcf_valuation, levered_dcf] -``` - -- [ ] **Step 2: Add a "Harvester" section to README.md** - -Open `README.md` and append (just before the existing "## Configuration" section) the following: - -```markdown -### Harvester - -`aiofmp harvest` is a long-running CLI that proactively warms the local Parquet cache. It runs one async task per enabled category, each on its own configurable interval, and uses category-specific strategies to minimise redundant fetches (target: stay under 20 GB/month on FMP starter plan). - -**Quick start:** - -```bash -cp examples/harvester.example.yaml ~/harvester.yaml -export FMP_API_KEY=your_key_here - -# Print the plan, fetch nothing -aiofmp harvest --config ~/harvester.yaml --dry-run - -# One cycle per enabled category, then exit -aiofmp harvest --config ~/harvester.yaml --once - -# Run a single category once -aiofmp harvest --config ~/harvester.yaml --once --category statements - -# Run forever (Ctrl-C to stop cleanly) -aiofmp harvest --config ~/harvester.yaml - -# Inspect state -aiofmp harvest-status --config ~/harvester.yaml -``` - -**State:** SQLite at `/harvester.sqlite` (checkpoints, bandwidth ledger, symbol catalogs). Parquet under `/cachedclient_data/`. - -See `docs/superpowers/specs/2026-05-15-harvester-v2-design.md` for the full design. -``` - -- [ ] **Step 3: Verify the docs render cleanly** - -Run: `uv run python -c "from pathlib import Path; print(Path('README.md').read_text(encoding='utf-8').count('Harvester'))"` -Expected: at least 1. - -- [ ] **Step 4: Commit** - -```bash -git add README.md examples/harvester.example.yaml -git commit -m "docs(harvester): README section + example config" -``` - ---- - -### Task 31: Full-suite final pass + tag-ready summary - -**Files:** -- (Verification only) - -- [ ] **Step 1: Run the whole suite** - -Run: `uv run pytest -q --tb=short` -Expected: all tests pass. No skipped tests beyond the existing optional integration tests. - -- [ ] **Step 2: Run ruff over the whole tree** - -Run: `uv run ruff check . && uv run ruff format --check .` -Expected: clean. - -- [ ] **Step 3: Sanity: import the package and list categories** - -Run: `uv run python -c "from aiofmp.harvester.categories import registered_names; print('\n'.join(registered_names()))"` -Expected: lists 17 categories (`chart_eod`, `chart_intraday`, `commodities_eod`, `commodities_intraday`, `forex_eod`, `forex_intraday`, `indexes_eod`, `indexes_intraday`, `news`, `economics`, `statements`, `analyst_estimates`, `analyst_snapshots`, `insider_trades`, `form13f`, `technical_indicators`, `dcf`). - -- [ ] **Step 4: Sanity: `--help` works** - -Run: `uv run aiofmp harvest --help` -Expected: shows the `--config`, `--once`, `--category`, `--dry-run`, `--api-key` options. - -Run: `uv run aiofmp harvest-status --help` -Expected: shows the `--config` option. - -- [ ] **Step 5: Done — print final summary** - -Run: `git log --oneline 6b52a86..HEAD` -Expected: a clean linear history of the harvester implementation commits. - -No code change for this step — this is the final verification gate. diff --git a/docs/superpowers/specs/2026-05-15-harvester-v2-design.md b/docs/superpowers/specs/2026-05-15-harvester-v2-design.md deleted file mode 100644 index 0de8731..0000000 --- a/docs/superpowers/specs/2026-05-15-harvester-v2-design.md +++ /dev/null @@ -1,409 +0,0 @@ -# Harvester v2 — Design - -**Date:** 2026-05-15 -**Status:** Approved (pending user review of written spec) -**Owners:** codemug -**Related code:** `aiofmp/cachedclient/`, `aiofmp/base.py`, `aiofmp/cli.py` - -## 1. Goal & non-goals - -**Goal.** A long-running CLI process (`aiofmp harvest --config harvester.yaml`) that proactively warms the local Parquet cache with FMP data, choosing per-category strategies that minimize redundant fetches and stay under the 20 GB/month FMP starter-plan cap. - -**Non-goals.** - -- Not a data analytics or query tool. Reads/queries remain the user's job via `CachedClient`. -- Not a generic scheduler. It only knows how to harvest FMP categories. -- Does not modify on-demand `CachedClient` semantics. User-driven queries behave identically whether the harvester is running or not. -- Does not auto-promote new categories without code changes. Adding a category is a typed, reviewed addition. - -## 2. Architecture - -``` -aiofmp harvest --config harvester.yaml - │ - ▼ -HarvesterManager - ├─ load YAML → HarvestConfig - ├─ create one FmpClient + CachedClient (shared, reused across categories) - ├─ create StateStore (sqlite3 at /harvester.sqlite) - ├─ create BudgetTracker (reads/writes to StateStore) - ├─ create SymbolCatalog (reads/writes to StateStore) - ├─ register SIGINT/SIGTERM → stop_event - └─ spawn one asyncio.Task per enabled category - -CategoryHarvester (abstract base, one running task per concrete subclass): - while not stop_event.is_set(): - await wait_until(next_due, stop_event) - if await budget.is_paused(category): continue - await run_cycle() # category-specific logic - state.record_run(category, status, items, bytes) - next_due = now + interval - - on CancelledError: - flush partial checkpoint - mark current run 'partial' - re-raise - -Manager: await asyncio.gather(*tasks, return_exceptions=True), then close. -``` - -**Module layout (new):** - -``` -aiofmp/harvester/ - __init__.py (public exports) - cli.py (click subcommand wiring) - manager.py (HarvesterManager, lifecycle, signal handling) - config.py (YAML schema → dataclasses, interval parsing) - state.py (StateStore — sqlite3 wrapper, schema migrations) - budget.py (BudgetTracker) - catalog.py (SymbolCatalog — discover and persist symbol universes) - base.py (CategoryHarvester ABC, retry/backoff, cycle bookkeeping) - categories/ - statements.py - analyst_estimates.py - analyst_snapshots.py - insider_trades.py - form13f.py - chart_eod.py - chart_intraday.py - technical_indicators.py - news.py - economics.py - commodities_eod.py - commodities_intraday.py - forex_eod.py - forex_intraday.py - indexes_eod.py - indexes_intraday.py - dcf.py -``` - -**Touched code outside `harvester/`:** - -- `aiofmp/cli.py` — wire `aiofmp harvest` and `aiofmp harvest-status` subcommands. -- `aiofmp/cachedclient/registry.py` — add `TemporalPattern.PAGE_WALK`, supporting fields on `CacheableEndpoint`, and register the new endpoints (analyst estimates, insider trades, form 13F). Also register the historical-price-eod and historical-chart endpoints under the `commodity`, `forex`, and `indexes` categories so user-driven calls to those category methods hit the same cache as the existing `chart.*` registrations (storage key is keyed off `api_endpoint`, so all four categories share parquet location). -- `aiofmp/cachedclient/proxy.py` — read-only handling for `PAGE_WALK` keys when called from user code (no implicit writes). -- `aiofmp/base.py` — emit response-size estimate to a callback so `BudgetTracker` can attribute bytes per category; add `FMPBudgetError` exception raised when the monthly hard cap is exceeded. - -No changes to existing parquet layout or `StorageBackend` interface. - -## 3. Symbol catalog - -Six "universes" are sourced from FMP's directory and per-category list endpoints. Each universe is discovered lazily on first use, cached in `symbol_catalog` (SQLite), and refreshed when older than `discovery.refresh_interval` (default 7 days). - -| Universe | FMP endpoint | Used by | -| --- | --- | --- | -| `financial_symbols` | `directory.financial_symbols()` | statements, dcf | -| `actively_trading` | `directory.actively_trading()` | analyst_estimates, analyst_snapshots, chart_eod, chart_intraday, technical_indicators, insider_trades sharding | -| `commodities` | `commodity.commodities_list()` | commodities_eod, commodities_intraday | -| `forex_pairs` | `forex.forex_list()` | forex_eod, forex_intraday | -| `indexes` | `indexes.index_list()` | indexes_eod, indexes_intraday | -| `etf_list` | `directory.etf_list()` | (no category harvests ETFs yet; kept for future) | - -`SymbolCatalog.symbols(universe: str) -> list[str]` returns the cached list and triggers a refresh if expired. Directory endpoints themselves are passthrough in `CachedClient` today — `symbol_catalog` is their de facto cache. - -## 4. The four harvest patterns - -| Pattern | Idea | First run | Subsequent runs | -| --- | --- | --- | --- | -| **P1 Discover-then-fill** | Use a "what changed since X" endpoint to find work, then fetch only that. | Full backfill for every catalog symbol. | Hit the discovery endpoint with `from = last_run`. Only touched symbols get fetched. | -| **P2 Walk-back-until-checkpoint** | Page from newest backwards until first record < checkpoint or max_backfill reached. | Walk pages until oldest record < `today - max_backfill`. | Walk until oldest record in page < `last_run`. | -| **P3 Gap-fill (CachedClient native)** | Request the full window; `CachedClient` gap detection fetches only what's missing. | First call fetches the full window. | Subsequent calls fetch only the tail (or any holes). | -| **P4 Snapshot overwrite** | Fetch latest snapshot per symbol; replace stored copy. | Same as subsequent: fetch + replace. | Fetch + replace. | - -## 5. Per-category strategies - -| Category | Pattern | Discovery / walk endpoint | Persistence | Default interval | -| --- | --- | --- | --- | --- | -| **statements** | P1 | `calendar.earnings_calendar(from=last_run, to=now)` returns symbols with filings in the window. For each touched symbol, call all 13 statement endpoints. Specifics: the 10 period+limit endpoints (income, balance, cash flow, key_metrics, ratios, enterprise_values, 4× growth) run for both `annual` and `quarter` with `limit=incremental_limit` (default 2); `owner_earnings` runs once with `limit=incremental_limit` (no period param); the 2 segmentation endpoints run once per `period`. First run uses `limit=initial_limit` (default 40). Plus monthly safety-net sweep (Section 6). | `CachedClient` `PERIOD_BASED` (existing) | 6h | -| **analyst_estimates** | P2 | Per symbol in `actively_trading`, walk `analyst.financial_estimates(symbol, period='annual', page=N, limit=page_size)` until first record date < `min(last_run, today - max_backfill_years)`. | New `PAGE_WALK` registry entry, key `("analyst-estimates", symbol, "annual")`. | 24h | -| **analyst_snapshots** | P4 | Per symbol in `actively_trading`: `ratings_snapshot`, `price_target_consensus`, `price_target_summary`, `stock_grades_summary`. | New simple snapshot storage: key `("snapshot/", symbol)` → single-row parquet, overwritten each cycle. | 24h | -| **chart_eod** | P3 | Per symbol in `actively_trading`: `cached.chart.historical_price_full(symbol, today - backfill_years, today)`. Variants list also supports `historical_price_light` / `historical_price_unadjusted` / `historical_price_dividend_adjusted`. | `CachedClient` `DATE_RANGE` (existing) | 24h | -| **chart_intraday** *(off by default)* | P3 | Per `(symbol, timeframe)` for symbols in `actively_trading`: `cached.chart.intraday_(symbol, today - backfill_days, today)`. | `CachedClient` `DATE_RANGE` (existing) | 4h | -| **technical_indicators** *(off by default)* | P3 | Per `(symbol, indicator, period_length, timeframe)`: `cached.technical_indicators.(...)`. Configurable list. | `CachedClient` `DATE_RANGE` (existing) | 24h | -| **news** | P3 | For each variant in `[general_news, press_releases, stock_news, crypto_news, forex_news]`: `cached.news.(from = today - backfill_days_initial, to = now, limit = page_size)`. The `CachedClient` `DATE_RANGE` gap detector handles the incremental tail — after the first cycle each subsequent cycle only fetches the new days. The harvester walks `page` until a returned batch is empty or its newest record falls in the existing cache window. | `CachedClient` `DATE_RANGE` (existing) | 30m | -| **insider_trades** | P2 (global walk, per-symbol shard on write) | `insider_trades.latest_insider_trades(page=N, limit=page_size)` until oldest `filingDate < last_run` or page index ≥ `max_pages`. | New `PAGE_WALK` entry. Records sharded by `symbol` into key `("insider-trading/latest", symbol)`. A second key `("insider-trading/latest", "_global")` holds the raw global stream for completeness. | 6h | -| **form13f** | P2 (global walk, per-CIK shard on write) | `form13f.latest_filings(page=N, limit=page_size)` until oldest `acceptedDate < last_run` or page index ≥ `max_pages`. | New `PAGE_WALK` entry. Records sharded by `cik` into key `("institutional-ownership/latest", cik)`. Per-CIK detail extracts (`filings_extract`) are NOT harvested in v1 — fetched on-demand by users. | 24h | -| **economics** | P3 | `cached.economics.treasury_rates(today - backfill_years, today)` plus per-indicator `cached.economics.economic_indicators(name, today - backfill_years, today)`. Indicators come from config (default: `GDP`, `CPI`, `UNRATE`, `FEDFUNDS`, `DFF`). | `CachedClient` `DATE_RANGE` (existing) | 24h | -| **commodities_eod** | P3 | For each symbol in `commodities` universe (~50 symbols), `cached.commodity.historical_price_full(symbol, today - backfill_years, today)`. Variants list also supports `historical_price_light`. | `CachedClient` `DATE_RANGE` (new registration under `commodity` category, shared parquet with `chart.historical_price_full`) | 24h | -| **commodities_intraday** *(off by default)* | P3 | Per `(symbol, timeframe)` for symbols in `commodities` universe: `cached.commodity.intraday_(symbol, today - backfill_days, today)`. | `CachedClient` `DATE_RANGE` (new registration under `commodity` category) | 4h | -| **forex_eod** | P3 | For each symbol in `forex_pairs` universe (~200–300 pairs), `cached.forex.historical_price_full(symbol, today - backfill_years, today)`. Variants list also supports `historical_price_light`. | `CachedClient` `DATE_RANGE` (new registration under `forex` category) | 24h | -| **forex_intraday** *(off by default)* | P3 | Per `(symbol, timeframe)` for symbols in `forex_pairs` universe: `cached.forex.intraday_(symbol, today - backfill_days, today)`. | `CachedClient` `DATE_RANGE` (new registration under `forex` category) | 4h | -| **indexes_eod** | P3 | For each symbol in `indexes` universe (~100–200), `cached.indexes.historical_price_eod_full(symbol, today - backfill_years, today)`. Variants list also supports `historical_price_eod_light`. | `CachedClient` `DATE_RANGE` (new registration under `indexes` category, `DATE_OBJ` date type) | 24h | -| **indexes_intraday** *(off by default)* | P3 | Per `(symbol, timeframe)` for symbols in `indexes` universe: `cached.indexes.intraday_(symbol, today - backfill_days, today)`. | `CachedClient` `DATE_RANGE` (new registration under `indexes` category, `DATE_OBJ`) | 4h | -| **dcf** | P4 | Per symbol in `financial_symbols`: `dcf.dcf_valuation(symbol)` and `dcf.levered_dcf(symbol)`. Both return one-row point-in-time snapshots that change only when underlying statements change. `custom_dcf_*` is NOT harvested (user-parameterized). | Snapshot store: key `("snapshot/", symbol)` → single-row parquet, overwritten each cycle. | 24h | - -### New CachedClient surface area - -- Add `TemporalPattern.PAGE_WALK = "page_walk"` and supporting `CacheableEndpoint` fields: `page_param: str`, `walk_date_field: str`, `default_page_size: int`. -- `PAGE_WALK` keys are **harvester-write, user-read**. The proxy: - - On normal user call to a `PAGE_WALK`-registered method: read all stored records, filter to the call's `limit`/`page`/`symbol` filter, return. Never writes. - - The harvester does not call the proxy for these endpoints; it calls the raw `FmpClient` method and writes directly to `StorageBackend` via the same key the proxy would read from. -- Snapshot (P4) storage: a tiny helper `SnapshotStore` writes one-row parquet under `("snapshot/", entity)` and is read by the proxy via passthrough today (no change to user-facing semantics). - -## 6. Monthly safety-net sweep (statements) - -In addition to the incremental P1 flow, statements run a full safety-net sweep every `safety_net_interval` (default `30d`). During a safety-net cycle the category ignores `last_run` for the earnings-calendar probe and instead enumerates every symbol in `financial_symbols`, calling each statement endpoint with `limit=incremental_limit` (default 2). This catches filings missed by earnings-calendar discovery (e.g. non-US issuers, funds without earnings calls). - -The safety-net cycle is bookkept as a separate `category_runs` row (category=`statements_safetynet`) so its bandwidth and timing are observable independently. - -## 7. Config schema (YAML) - -```yaml -# harvester.yaml -state_dir: ~/.aiofmp/cache # parquet + harvester.sqlite both live here -log_level: INFO # DEBUG | INFO | WARNING | ERROR - -budget: - monthly_soft_cap_gb: 18 - monthly_hard_cap_gb: 20 - soft_cap_behavior: pause_until_next_month # | warn_only - -retry: - on_429: - backoff_seconds: [60, 120, 240, 480] - max_attempts: 4 - on_5xx: - backoff_seconds: [10, 30, 60] - max_attempts: 3 - -discovery: - refresh_interval: 7d # how often to refresh symbol catalogs - -categories: - statements: - enabled: true - interval: 6h - periods: [annual, quarter] - initial_limit: 40 # first-time per symbol - incremental_limit: 2 # per discovery hit - safety_net_interval: 30d # full sweep cadence - - analyst_estimates: - enabled: true - interval: 24h - estimate_periods: [annual] - max_backfill_years: 3 - page_size: 100 - - analyst_snapshots: - enabled: true - interval: 24h - include: [ratings_snapshot, price_target_consensus, price_target_summary, stock_grades_summary] - - chart_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - symbol_universe: actively_trading - - chart_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - symbol_universe: actively_trading - - technical_indicators: - enabled: false - interval: 24h - indicators: [] # [{method: simple_moving_average, period_length: 20, timeframe: 1day}, ...] - symbol_universe: actively_trading - - news: - enabled: true - interval: 30m - variants: [general_news, press_releases, stock_news, crypto_news, forex_news] - backfill_days_initial: 30 - page_size: 100 - - insider_trades: - enabled: true - interval: 6h - max_pages: 200 - page_size: 100 - - form13f: - enabled: true - interval: 24h - max_pages: 50 - page_size: 100 - - economics: - enabled: true - interval: 24h - indicators: [GDP, CPI, UNRATE, FEDFUNDS, DFF] - backfill_years: 10 - - commodities_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - - commodities_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - forex_eod: - enabled: true - interval: 24h - variants: [historical_price_full] - backfill_years: 10 - - forex_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - indexes_eod: - enabled: true - interval: 24h - variants: [historical_price_eod_full] - backfill_years: 10 - - indexes_intraday: - enabled: false - interval: 4h - timeframes: [1hour] - backfill_days: 30 - - dcf: - enabled: true - interval: 24h - include: [dcf_valuation, levered_dcf] -``` - -**Interval parsing.** `Ns`, `Nm`, `Nh`, `Nd` (case-insensitive). Validation rejects anything else. - -## 8. State store (SQLite) - -`/harvester.sqlite`: - -```sql -CREATE TABLE category_runs ( - category TEXT NOT NULL, - started_at TEXT NOT NULL, -- ISO8601 UTC - finished_at TEXT, - status TEXT NOT NULL, -- 'running' | 'ok' | 'partial' | 'error' | 'paused_for_budget' - items_attempted INTEGER NOT NULL DEFAULT 0, - items_succeeded INTEGER NOT NULL DEFAULT 0, - bytes_estimate INTEGER NOT NULL DEFAULT 0, - error TEXT, - PRIMARY KEY (category, started_at) -); - -CREATE TABLE checkpoints ( - category TEXT NOT NULL, - scope TEXT NOT NULL, -- 'global' | '' | '/' | '' | etc. - checkpoint TEXT NOT NULL, -- ISO8601 date or datetime - updated_at TEXT NOT NULL, - PRIMARY KEY (category, scope) -); - -CREATE TABLE symbol_catalog ( - universe TEXT NOT NULL, -- 'financial_symbols' | 'actively_trading' | 'commodities' - -- | 'forex_pairs' | 'indexes' | 'etf_list' - symbol TEXT NOT NULL, - payload_json TEXT, -- whole row from FMP - discovered_at TEXT NOT NULL, - PRIMARY KEY (universe, symbol) -); - -CREATE TABLE catalog_refresh ( - universe TEXT PRIMARY KEY, - last_refreshed_at TEXT NOT NULL -); - -CREATE TABLE bandwidth_ledger ( - month TEXT NOT NULL, -- 'YYYY-MM' UTC - category TEXT NOT NULL, - bytes INTEGER NOT NULL DEFAULT 0, - PRIMARY KEY (month, category) -); - -CREATE INDEX idx_category_runs_started ON category_runs(category, started_at DESC); -``` - -Schema version table + simple linear migrations live alongside in `state.py`. - -## 9. Failure & budget behavior - -- **429 (rate limit).** Caught in `CategoryHarvester._run_with_backoff`. Sleeps `retry.on_429.backoff_seconds[attempt]` then retries. Exhaustion → mark current cycle `partial`, persist whatever checkpoint progress was made, schedule next cycle normally. -- **5xx.** Same retry ladder via `retry.on_5xx`. -- **Other exceptions during a per-item fetch.** Logged, counted in `items_attempted` but not `items_succeeded`, cycle continues — one bad symbol does not kill the category. -- **Cycle-wide exception (config bug, SDK bug).** Caught by `CategoryHarvester.run()`, logged with traceback, cycle marked `error`. Task keeps running and will retry on next due time. -- **Soft cap.** Before each cycle, `BudgetTracker.is_paused(category)` sums `bandwidth_ledger` for the current UTC month. If ≥ `monthly_soft_cap_gb` → log a warning, write a `paused_for_budget` row to `category_runs`, return. Resumes naturally on the first cycle of the next UTC month (month rollover detection happens at the top of each `wait_until`). -- **Hard cap.** A hot per-request check inside `FMPBaseClient` — when accumulated month bytes ≥ `monthly_hard_cap_gb`, raise `FMPBudgetError` (new exception in `aiofmp/base.py`, subclass of `FMPError`). The category catches it, marks the cycle `partial`, and sleeps until next month. The hard-cap check applies to harvester-initiated requests only (those with a category context); user-driven `CachedClient` queries are never blocked, but they are still tallied under the `'user'` category in `bandwidth_ledger` for visibility. - -## 10. Byte accounting - -`FMPBaseClient._handle_response` is extended with an optional `on_response_size(category: str | None, byte_count: int)` callback. The harvester registers a callback per CategoryHarvester via a `contextvars.ContextVar[str]` set at the top of each cycle so concurrent categories attribute bytes correctly. Byte count is `len(raw_body)` after `response.read()` — close enough for soft/hard cap decisions; chunked responses rarely carry an accurate Content-Length. - -User-driven `CachedClient` calls (not harvester-initiated) write to a `bandwidth_ledger` row with `category = 'user'` so the harvester's budget isn't polluted by ad-hoc queries. - -## 11. CLI - -```bash -aiofmp harvest --config harvester.yaml # run forever -aiofmp harvest --config harvester.yaml --once # one cycle per category, then exit -aiofmp harvest --config harvester.yaml --category news --once # one-shot single category -aiofmp harvest --config harvester.yaml --dry-run # print plan, fetch nothing -aiofmp harvest-status --config harvester.yaml # summary table from state store -``` - -`--config` defaults to `./harvester.yaml`, then `~/.aiofmp/harvester.yaml`. - -`--once` and `--dry-run` exit code: 0 on success, 1 on any category error. - -## 12. Lifecycle - -1. `cli.harvest` loads YAML → validates → builds `HarvestConfig`. -2. `HarvesterManager.__init__(config)`, then `await manager.start()`: - - opens SQLite, applies schema migrations, - - constructs shared `FmpClient` and `CachedClient`, enters async context, - - constructs `BudgetTracker` and `SymbolCatalog`, - - registers SIGINT/SIGTERM handlers that set `manager.stop_event`. -3. Manager builds the list of enabled `CategoryHarvester` instances and spawns one task each. -4. `await asyncio.gather(*tasks, return_exceptions=True)` — main loop. -5. On stop: every task observes `stop_event`, finishes its current cycle iteration (or catches `CancelledError` if cancellation is forced after a grace timeout), flushes state, returns. -6. `manager.close()` closes the CachedClient context and SQLite. Process exits 0. - -Grace timeout: configurable `shutdown_grace_seconds` (default 30s). After grace, tasks are cancelled. - -## 13. Testing strategy - -- **Unit tests per category** (`tests/test_harvester/test_.py`): - - Mock the FMP SDK methods with deterministic responses. - - Assert correct call sequence, checkpoint advancement, retry behavior, per-pattern stop conditions. - - Parameterize the P1–P4 patterns over their representative categories. -- **StateStore tests:** in-memory SQLite, schema migration idempotency, checkpoint upsert semantics, bandwidth-ledger month rollover. -- **BudgetTracker tests:** soft/hard cap behavior, attribution under concurrent categories (uses a fake `FMPBaseClient` that emits a stream of fake byte counts). -- **SymbolCatalog tests:** discovery, refresh-when-stale, never-refresh-when-fresh, persistence across process restarts. -- **Manager tests:** SIGINT path, grace timeout cancellation, partial-checkpoint persistence on cancel. -- **CLI smoke tests:** `--dry-run` prints expected plan, `--once --category news` runs a single category against mocked SDK and asserts state-store side effects. -- **Optional integration test** (skipped unless `FMP_API_KEY` set): `aiofmp harvest --config tiny.yaml --once --category news` against the real FMP API; assert files appear under a temporary `state_dir`. - -Target ≥80% line coverage on the new `aiofmp/harvester/` package. - -## 14. Risks & open items - -- **Earnings-calendar accuracy for statements discovery.** Filings released without an earnings call may be missed by the P1 incremental flow. Mitigated by the monthly safety-net sweep (Section 6). -- **Global walks fan out into per-entity shards.** Insider trades and form 13F are walked globally but stored per-symbol/per-CIK. A user querying the "global" stream via the SDK will get the passthrough live response; the harvester's per-entity shards are the cache. This is by design but worth a comment in the registry. -- **PAGE_WALK is harvester-write-only.** Documented explicitly in the registry and in the user-facing docs for `CachedClient`. -- **Byte counting is approximate.** Acceptable for soft/hard cap decisions; not suitable for FMP-side billing reconciliation. -- **Concurrency.** Up to N category tasks fight for the shared `FmpClient` semaphore (default `max_concurrent_requests = 10`). If this turns out to be too aggressive, lower it via `FmpClient(max_concurrent_requests=...)` exposed through config in a follow-up. -- **Stop-vs-finish under load.** Categories with very long cycles (initial full sweep) may not respect `stop_event` quickly. The cooperative cancellation contract is: check `stop_event` between symbols, not within a single SDK call. Grace timeout escalates to `CancelledError`. - -## 15. Open extension points (deferred, not v1) - -- Multi-tier symbol watchlists (tier1 = full harvest, tier2 = statements only, etc.). -- HTTP server exposing `/healthz` and `/metrics`. -- Form 13F per-CIK extract auto-harvest. -- Live structured logging (JSONL to disk). -- Resume-from-checkpoint inside a single cycle (e.g. resume statements sweep at symbol N+1).