UNDER THE HOOD · METHODOLOGY & PIPELINE

Data ingestion pipelines, feature calculation rules, ML validation, and honest limitations.

Daily Execution Pipeline (GitHub Actions)

Ingest ──────► Parquet Lake (Git) ──► Feature Engineering ──► ML Model & Verdict
  │                                           │                      │
Equities (30 tickers × 2y)            Trading-Day Calendar      Logistic P(AI up 5d)
FRED (Incremental OAS & VIX)          6 Z-Scored Gauges         Permanent Ledger Append
Manual CapEx Seed (Quarterly)         Bubble Temperature        Out-of-Sample Scoreboard
                                      Regime Classification     Static Site & Digest Gen

The Git repository functions as the database. Every daily automated run appends and deduplicates Parquet rows, ensuring full historical repeatability and zero external database dependencies.

Data Sources

Source	Cadence	Details & Fallbacks
Yahoo Finance → Stooq	Daily	22-name AI complex (NVDA, AVGO, AMD, TSM, MSFT, GOOGL, META, AMZN, AAPL, ORCL, CRM, NOW, PLTR, SMCI, ANET, MRVL, VRT, ETN, PWR, DLR, EQIX, MU), 4 power/utility leaders (CEG, VST, NEE, TLN), benchmarks (SPY, RSP, QQQ, ^SOX, ^VIX, ^TNX, BTC-USD). Backfill: 750 sessions.
FRED Public Graph Endpoint	Daily	US High-Yield OAS (`BAMLH0A0HYM2`) and VIX close (`VIXCLS`). Pulled incrementally using fast HTTP retrieval to prevent socket stalls. Yahoo ^VIX serves as immediate daily fallback.
Manual CapEx CSV	Quarterly	Combined Big Tech CapEx ($B) and Nvidia YoY revenue growth in `data/manual/hyperscaler_capex.csv`. Fresh through 2026Q2 with staleness age surfaced on dashboard.

Key Design Principles

Trading-Day Calendar Filter: Crypto (BTC) trades 24/7/365, while equities trade ~252 sessions a year. BubbleWatch filters market dates to active equity sessions so 20d, 60d, and 250d lookback windows represent genuine trading sessions rather than empty weekend pads.
Compounded Equal-Weight Returns: Baskets are built by compounding daily mean percentage returns rather than averaging raw share prices, ensuring constituent IPOs or listings (e.g. TLN) do not distort the index.
Strict Chronological Split: Supervised machine learning models train on the first 80% of historical sessions and validate on the final 20% out-of-sample test set. No data shuffling or forward leakage.
Cryptographic Ledger: Daily forecasts are hashed by feature content and committed to an append-only JSONL ledger before outcomes occur. Once the 5-day horizon elapses, actual basket moves join back for evaluation.

Honest Limitations

Single Generative AI Cycle In-Sample: The current generative AI buildout has only one major boom cycle in recent market data. Rolling statistics inevitably overlap this period. Treat temperature as an ordinal scale (rising/falling, cool/hot) rather than an absolute law.
Five-Day Direction Noise: Short-term 5-day equity market direction is largely dominated by micro news events. Out-of-sample Brier scores will hover near 0.24–0.26. The value lies in honest probability calibration, not claims of psychic forecasting.
Fundamentals Update Latency: CapEx and revenue figures update once every 90 days following quarterly 10-Q earnings releases.
Regulatory & Legal Status: BubbleWatch is an educational open-source software project. It does not provide financial or trading advice. Review the complete Legal Disclaimers, Data Disclosures & Terms of Service.