NDX-100 v2: Regime-Aware Cross-Sectional Ranking System

Architecture Document

Status: Design Phase
Scope: Independent model + codebase from Semiconductors TFT
Goal: Discover conditional alpha pockets across NASDAQ-100

Paradigm Shift from v1 (Pooled Model)

Old Approach (Failed)

Multi-sector data (Consumer + Software + Semis + Mega-cap)
         ↓
    Single model
         ↓
   Conflicting gradients
         ↓
   Poor generalization

New Approach (v2)

NDX-100 tickers
       ↓
Relative feature engineering (vs sector, vs QQQ)
       ↓
Sector embeddings + Regime embeddings
       ↓
Cross-sectional ranking (rank within each date)
       ↓
Per-sector, per-regime evaluation
       ↓
"NO EDGE" decision (don't trade if no signal)

Key Design Decisions

1. Data Integrity (Strict Temporal Isolation)

Window Building:

For each DATE:
  For each TICKER in NDX-100:
    Get ticker's historical features (last 60 days)
    Get ticker's TARGET (return next 10 days)
    Store: (features, ticker_id, sector_id, regime_id, target)

Critical Rules:

• ✓ Each ticker gets its own target (NOT shared)
• ✓ Features cut-off before target date
• ✓ No forward-looking information in features
• ✓ Regime labels computed from historical data only
• ✗ NO confidence filtering during training (avoid selection bias)

2. Relative Features (Mandatory)

Why: Markets are relative. "NVDA +5% vs AMD -2%" is more predictive than absolute movements.

Feature Set:

All features z-scored cross-sectionally (mean=0, std=1 per date within NDX-100)

3. Sector Embeddings

NDX-100 Sectors (11 groups):

Technology:       AAPL, MSFT, NVDA, AVGO, ORCL, CRM, ACN, AMD, CSCO, IBM, INTC
Healthcare:       LLY, UNH, JNJ, ABBV, MRK, TMO, ABT, DHR, PFE, AMGN, BSX, SYK, ISRG, GILD, MDT, VRTX, REGN, ELV, HUM, CI, CVS, MCK, CAH, ZBH, BDX, BAX, DXCM, IDXX, RMD, EW, ALGN, HOLX, PODD
Consumer:         AMZN, TSLA, HD, MCD, NKE, SBUX, LOW, BKNG, CMG, TJX, ORLY, AZO, ROST, DHI, LEN, PHM, F, GM, APTV, NVR, TOL, KMX, AN, GRMN, RL, YUM, DRI
Financials:       BRK.B, JPM, V, MA, BAC, WFC, GS, MS, BLK, SCHW, AXP, C, SPGI, ICE, CB, PGR, AON, MMC, MET, PRU, AFL, TRV, ALL, HIG, AIG, COF, USB, PNC, TFC, FI, FISV
Industrials:      RTX, HON, UPS, CAT, DE, LMT, BA, GE, MMM, NOC, GD, LHX, TDG, PH, EMR, ETN, ROK, AME, FTV, CARR, OTIS, CTAS, CSX, NSC, UNP, FDX, GWW, ITW, PCAR, WAB, IR, VRSK, CPRT, ROP, IDEX, IEX, SWK, PNR, MAS, JCI, FAST, AXON, ODFL
Communication:    META, GOOG, GOOGL, NFLX, DIS, CMCSA, VZ, T, TMUS, EA, TTWO, WBD, FOXA, FOX, OMC, IPG
Energy:           XOM, CVX, COP, EOG, SLB, MPC, PSX, VLO, PXD, DVN, HAL, OXY, FANG, BKR, APA, MRO, HES, CTRA, EQT, TRGP, KMI, OKE, WMB
Real Estate:      AMT, PLD, CCI, EQIX, PSA, O, SBAC, DLR, SPG, AVB, EQR, MAA, UDR, CPT, EXR, VICI, WY, IRM, ARE, VTR, BXP
Utilities:        NEE, DUK, SO, D, AEP, EXC, SRE, PCG, ED, XEL, ETR, FE, WEC, CMS, NI, AEE
Materials:        LIN, APD, SHW, FCX, NEM, NUE, VMC, MLM, CF, MOS, ALB, ECL, PPG, EMN, LYB, DOW, DD
Staples:          PG, KO, PEP, WMT, COST, PM, MO, KHC, GIS, K, HSY, MDLZ, STZ, CL, CHD, CLX, EL, HRL, SJM, CPB, MKC, BG, CAG, LW, ADM, TSN

Embedding Logic:

sector_id = get_sector(ticker)  # 0-10
sector_embedding = nn.Embedding(11, 32)(sector_id)  # 11 sectors → 32-dim
# Sector embedding learned during training
# Captures sector-specific patterns (e.g., tech momentum vs utility mean-reversion)

4. Regime Embeddings

Regime Detection (5 regimes):

Embedding Logic:

regime_id = detect_regime(qqq_return, market_vol)  # 0-4
regime_embedding = nn.Embedding(5, 32)(regime_id)  # 5 regimes → 32-dim
# Regime embedding learned during training
# Captures regime-specific strategies (momentum in bull, mean-reversion in bear)

Model Architecture

Input Pipeline

(ticker_features_60d, sector_id, regime_id, ticker_id) 
    ↓
Feature projection (relative features)
    ↓
[Features (96-dim) ⊕ Sector embedding (32-dim) ⊕ Regime embedding (32-dim)]
    ↓
Total: 160-dim input to Transformer

Core TFT with Routing

class NDX100V2(nn.Module):
    def __init__(self, n_sectors=11, n_regimes=5):
        # Sector-specific attention heads
        self.sector_heads = nn.ModuleDict({
            sector_name: TransformerHead(...) for sector_name in SECTORS
        })
        
        # Regime gating
        self.regime_gate = nn.Linear(32, n_sectors)  # Which head to use?
        
        # Shared backbone (light)
        self.backbone = TFTEncoder(d_model=128, num_heads=4, num_layers=2)
        
        # Multi-head ranking outputs
        self.ranking_heads = nn.ModuleDict({
            sector: nn.Linear(128, 1) for sector in SECTORS
        })
    
    def forward(self, x, sector_id, regime_id, ticker_id):
        # Backbone processes all
        backbone_out = self.backbone(x)  # (batch, seq, 128)
        
        # Regime gate selects which sector head to emphasize
        gate = self.regime_gate(regime_embedding)  # (batch, n_sectors)
        gate = torch.softmax(gate, dim=1)
        
        # Sector-specific ranking
        sector_name = SECTORS[sector_id]
        logits = self.ranking_heads[sector_name](backbone_out)
        
        # Output: single ranking score per stock
        return logits.squeeze(-1)  # (batch,)

Loss Function (Cross-Sectional Ranking)

NOT regression loss. NOT pairwise contrastive.

Ranking Loss: Directly optimize Spearman correlation

def ranking_loss(pred, target, n_stocks_per_date=100):
    """
    For each date:
      - Rank predictions
      - Rank actual returns
      - Compute Spearman correlation
      - Loss = 1 - correlation
    """
    batch_size = pred.shape[0]
    n_dates = batch_size // n_stocks_per_date
    
    total_loss = 0.0
    for d in range(n_dates):
        start = d * n_stocks_per_date
        end = (d + 1) * n_stocks_per_date
        
        pred_d = pred[start:end]
        target_d = target[start:end]
        
        # Rank predictions and targets
        pred_ranks = torch.argsort(torch.argsort(pred_d.float()))
        target_ranks = torch.argsort(torch.argsort(target_d.float()))
        
        # Normalize ranks to [0, 1]
        pred_ranks_norm = pred_ranks.float() / (n_stocks_per_date + 1)
        target_ranks_norm = target_ranks.float() / (n_stocks_per_date + 1)
        
        # Correlation loss (minimize 1 - correlation)
        correlation = torch.nn.functional.cosine_similarity(
            pred_ranks_norm.unsqueeze(0),
            target_ranks_norm.unsqueeze(0)
        )
        loss_d = 1.0 - correlation
        total_loss += loss_d
    
    return total_loss / max(1, n_dates)

Training Configuration

Hyperparameters (Conservative, Non-Optimized)

TRAIN_CONFIG = {
    "batch_size": 128,              # Never > 256 (ranking precision)
    "epochs": 100,                  # More data = more epochs
    "learning_rate": 5e-4,          # Gentle learning
    "weight_decay": 1e-4,
    "gradient_clip": 1.0,
    "dropout": 0.2,                 # More regularization
    "d_model": 128,
    "num_heads": 4,
    "num_attention_layers": 3,      # Slightly deeper for complexity
    "sequence_length": 60,          # 60-day lookback
    "scheduler": "CosineAnnealingLR",
    "T_max": 100,
}

# NO mixed precision (AMP/FP16)
# NO confidence filtering
# NO data augmentation

Training Loop

for epoch in range(epochs):
    for batch in train_loader:
        X, sector_id, regime_id, ticker_id, target = batch
        
        # Forward
        pred = model(X, sector_id, regime_id, ticker_id)
        loss = ranking_loss(pred, target)
        
        # Backward
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()
        
        # Log
        if batch_idx % 100 == 0:
            print(f"Epoch {epoch} | Batch {batch_idx} | Loss: {loss:.4f}")

Evaluation Framework (Strict)

1. Global Metrics

Rank IC (all tickers):  Spearman(pred, actual)
Sharpe:                 mean_return / std_return
Win rate:               % of dates with positive P&L

2. Per-Sector Metrics

For each sector:
  - Rank IC within sector
  - Number of dates > 0.05 IC threshold
  - Average decile spread
  - Sector-specific Sharpe

3. Per-Regime Metrics

For each regime:
  - Rank IC in that regime
  - Does edge exist in this regime?
  - Regime-specific Sharpe

4. "NO EDGE" Capability

def should_trade(rank_ic_today, p_value):
    """
    Decision rule:
    - If Rank IC < 0.02 AND p_value > 0.05: NO EDGE
    - Return: can_trade (bool)
    """
    if rank_ic_today < 0.02:
        return False  # Not enough signal
    if p_value > 0.05:
        return False  # Not significant
    return True

Data Pipeline (Modular)

File Structure

ndx100_v2/
├── __init__.py
├── data/
│   ├── feature_engineering.py    # Relative features
│   ├── regime_detection.py       # 5-regime classification
│   ├── window_builder.py         # Strict ticker-specific targets
│   └── data_loader.py            # PyTorch DataLoader
├── model/
│   ├── architecture.py           # NDX100V2Model
│   ├── loss.py                   # ranking_loss()
│   └── config.py                 # TRAIN_CONFIG
├── training/
│   ├── train.py                  # Training loop
│   ├── validate.py               # Walk-forward validation
│   └── inference.py              # Prediction
├── validation/
│   ├── rank_ic.py                # Per-sector, per-regime IC
│   ├── regime_analysis.py        # Does edge exist per regime?
│   ├── ablation.py               # Sector contribution
│   └── no_edge_detector.py       # Signal quality check
└── README.md                      # Full documentation

Success Criteria

✅ Must Pass

1. All walk-forward folds: Rank IC > 0.03 (lower threshold than semis due to 100 tickers)
2. Per-sector IC: > 0.02 IC in at least 5/11 sectors
3. Per-regime analysis: Identify where edge exists (e.g., "strong in Tech bull markets, weak in Utilities")
4. "NO EDGE" accuracy: Can accurately say "don't trade today"
5. Sector-specific heads: Each sector head should learn different patterns
6. No data leakage: Strict temporal isolation verified

⚠️ Nice to Have

• Rank IC > 0.05 globally (excellent)
• Sharpe > 0.3 in paper trading
• Consistent edge across 80% of regimes
• <2 bps advantage over momentum baseline

Comparison to Semis Baseline

Timeline

Total: 9-12 days to discovery phase

Key Principles

✅ No universal alpha assumption. Edge is conditional (by sector, by regime).
✅ Strict data integrity. Each ticker's target is only its own return.
✅ Ranking focus. Not trying to predict exact returns, just rank correctly.
✅ Testable hypothesis. Each sector head has a learned hypothesis.
✅ Regime awareness. Different strategies for different markets.
✅ Can say no. "NO EDGE" is a valid decision.
✅ Separate from semis. Independent codebase, deployable separately.

Next Steps

1. Implement ndx100_v2/data/feature_engineering.py with relative features
2. Implement sector/regime detection and embeddings
3. Build strict window builder (ticker-specific targets)
4. Train on NDX-100 data (2023-2026)
5. Validate with final_validation_suite adapted for 100 tickers
6. Report per-sector, per-regime edge existence