Semiconductors TFT Ranking Engine Strategy

Date: 2026-05-30
Status: Implementation & Training In Progress
Goal: Convert weak forecasting model → production-ready ranking engine

Executive Summary

Previous Verdict: Semiconductors TFT showed only marginal signal (Rank IC +0.0391, just barely above 2σ significance), with economic returns too small to survive trading costs.

New Approach: Pivot from return forecasting → cross-sectional ranking within semiconductors-only universe. Focus on maximizing relative ranking power, not absolute return prediction.

Target Metrics (This Iteration):

• Rank IC: >0.05 (from 0.0391)
• Decile Spread: >0.20% daily (from 0.08%)
• Portfolio Sharpe: >0.5 (from ~0.2)
• Turnover: <5% daily (confidence filtering + regime awareness)

1. Semiconductors-Only Universe

Tickers (14 total):

NVDA, AMD, AVGO, QCOM, MU, AMAT, LRCX, KLAC,
MRVL, ON, ADI, MCHP, NXPI, INTC

Why Narrow Universe Improves Signal:

Result: Model no longer averages contradictory signals from Consumer + Software + Mega-cap.

2. New Feature Engineering

Problem with Old Features:

• 20 absolute features (SMA ratios, RSI, etc.)
• Markets are relative, not absolute
• NVDA up 1% could be -5% vs sector or +5% vs peer momentum

New Feature Tiers:

TIER 0: Cross-Sectional Relative (HIGHEST SIGNAL)

• ret5_vs_sector — Stock return relative to SMH sector ETF
• ret10_vs_sector — 10-day relative return (primary)
• ret20_vs_sector — 20-day relative return
• relative_momentum_rank — Percentile rank within semis universe
• relative_volume_rank — Volume expansion rank (who's hot?)
• relative_volatility_rank — Vol rank (who's volatile?)

Why: Captures relative strength. NVDA -1% but AMD -5% = NVDA is winner.

TIER 1: Cross-Sectional Absolute

• cross_sectional_return_rank — Absolute return rank among semis
• cross_sectional_volume_rank — Absolute volume rank

Why: Objective ranking without sector normalization.

TIER 2: Volume Structure

• volume_over_sma20 — Current vol / 20-day average
• rolling_volume_percentile — Vol percentile in rolling window
• volume_zscore — Z-scored volume

Why: Volume expansion = institutional buying = continuation signal.

TIER 3: Volatility Structure

• stock_vol_over_sector_vol — Relative volatility vs semis
• volatility_expansion_flag — Is vol increasing?
• volatility_compression_flag — Is vol decreasing?

Why: Regime detection (high vol = different trading dynamics than low vol).

TIER 4: Microstructure

• overnight_gap_continuation — Gap direction correlated with next day?
• opening_range_expansion — Opening range larger than recent average?
• gap_fill_probability — How far through daily range did we close?

Why: Daily bar aggregation hides intraday momentum that persists.

TIER 5: Core Technical (Minimal)

• sma_200_ratio, sma_50_ratio, sma_14_ratio — Trend confirmation
• rsi_14 — Overbought/oversold (ranks better than predicts)
• sentiment_z — Normalized sentiment score

Result: 20+ relative features → model sees rankings, not just prices.

3. Regime-Aware Predictions

Problem: Model averages contradictory regimes

• In high-vol panic: small moves are noise
• In low-vol momentum: small moves are signal
• Model can't distinguish → weak overall signal

Solution: Regime Embeddings + Conditional Predictions

Regimes Detected:

Model Behavior by Regime:

LOW_VOL Regime:
  - Focus on absolute value (mean reversion)
  - Ranking = contrarian (lowest ranked = highest upside)
  - Confidence: LOWER (moves are noise)

NORMAL Regime:
  - Standard momentum ranking
  - Ranking = momentum (highest momentum ranks = continue)
  - Confidence: MEDIUM

HIGH_VOL Regime:
  - Focus on relative strength (who survives?)
  - Ranking = relative safety (highest ranked = less disaster)
  - Confidence: MEDIUM-HIGH (trends are clear)

MOMENTUM_EXPANSION Regime:
  - Trend following strong
  - Ranking = exactly as predicted
  - Confidence: HIGH (everyone agrees)

COMPRESSION Regime:
  - Anti-momentum (revert)
  - Ranking = inverted
  - Confidence: VERY HIGH (clear signal)

Implementation: Regime_id embedding in TFT (same as ticker_id embedding).

4. Ranking Loss, Not Regression Loss

Old Approach: Regression Loss

Loss = MSE(predicted_return, actual_return)
Problem: Penalizes slightly-wrong predictions equally with very-wrong
         A model predicting [0.5%, 0.4%, 0.3%] for [1%, 0%, -1%] loses heavily
         but ranking is correct (1st > 2nd > 3rd)

New Approach: Combined Loss

Total Loss = 0.7 * RankingLoss + 0.3 * RegressionLoss

RankingLoss = -Spearman_Correlation(predicted_ranks, actual_ranks)
  Directly optimizes ranking (whether top prediction actually has top return)
  Ignores absolute magnitudes (1% vs 0.1% doesn't matter)

RegressionLoss = MSE(prediction, actual)
  Keeps absolute scale reasonable (prevents collapse to [0,0,0])
  Downweighted since ranking is more important for portfolio

Result: Model optimizes for "top predicted stocks > bottom predicted stocks"
        But maintains price-level sanity for position sizing.

Why It Matters:

5. Turnover Reduction Strategies

Problem: Previous validation assumed daily rebalancing = 100% turnover = 0.1% cost daily = kills returns

Solutions:

A. Confidence Filtering

# Only rebalance if confidence > 60th percentile
pred_confidence = |prediction| / std(predictions)
filter = confidence > percentile(confidence, 60)
weights[~filter] = 0  # Hold previous positions
weights = weights / sum(weights)

Result: 40% of days skip rebalance → 60% turnover reduction

B. Weekly Rebalance Only

# Update portfolio once per week instead of daily
# Holds positions 5 trading days per change
# Reduces rebalancing activity 5x

Result: 80% turnover reduction vs daily

C. Holding Period Tuning

Test 3 holding periods:
- 7d:  More frequent, higher costs
- 10d: Matches ret10 horizon (best fit)
- 14d: Fewer trades, but stale predictions

Result: Identify optimal cost/stale-prediction tradeoff

D. Position Limits

# No single stock > 15% of portfolio
weights = np.clip(weights, 0, 0.15)
weights = weights / sum(weights)

Result: Prevents overconcentration → stable Sharpe

6. Portfolio Construction

Strategy: Long Top Decile, Short Bottom Decile

Each rebalance (weekly):
  1. Get TFT predictions for all 14 semis
  2. Rank by prediction
  3. Long: top 1-2 stocks (highest predicted return)
  4. Short: bottom 1-2 stocks (lowest predicted return)
  5. Size via softmax(predictions) capped at 15% each
  6. Hold for 10 days (ret10 horizon)
  7. Rebalance week 2

Cost Model:

Per rebalance:
  - Execution: 0.05% slippage (market impact)
  - Round-trip: 0.1% transaction cost
  - Total round-trip: 0.15% per 2-week cycle
  - Annual: 52 weeks ÷ 2 = 26 cycles × 0.15% = 3.9% annual cost

Gross decile spread needed: 0.20% daily × 10 days = 2% per cycle
After costs: 2% - 0.15% = 1.85% per cycle
Annualized: 1.85% × 26 = 48% (if constant)

Realistic (decay over time): 15-25% annualized Sharpe

Metrics to Track:

• Daily Rank IC (rolling 20d)
• Weekly portfolio return
• Turnover (% positions changed)
• Sharpe ratio (rolling 60d)
• Drawdown duration
• By-regime performance

7. Event-Aware Features (Future Enhancement)

Not yet implemented but ready for:

- days_to_earnings: Proximity to NVDA/AMD earnings
- post_earnings_drift: Stock behavior after earnings (PED anomaly)
- FOMC_window: Within 5 days of FOMC meeting?
- CPI_release: Economic data announcement nearby?
- fab_news: New fab announced? Yield problem? (manual signal)

Benefit: Semiconductors are announcement-driven (more than most sectors).

8. Feature Importance & Gating Analysis

Goal: Understand what model actually learned

After training, extract feature gating weights:

# For each sample, gating sigmoid = feature importance
# Output: Top 5 features should be clear (>0.7 weight)
# Bad sign: Uniform gating (all ~0.5) = can't learn

Expected top features:
  1. relative_momentum_rank (should be obvious)
  2. volume_expansion_flag (momentum confirmation)
  3. ret10_vs_sector (main ranking feature)
  4. volatility_compression_flag (regime detection)
  5. rsi_14 (overbought/oversold catch)

Outcome: If top 5 features are sensible → signal is explainable, not spurious.

9. Validation Metrics by Horizon

Horizon Performance:

Decision: Focus only on ret10 (10-day) predictions. Drop ret5 & ret20.

10. Success Criteria (Production Readiness)

Phase 1: Signal Quality

✅ PASS if:
  - Rank IC (ret10) > 0.05 (vs old 0.0391)
  - Z-score > 2.5σ vs Monte Carlo random
  - Consistent across time periods
  - Positive in 70%+ of rolling windows

Phase 2: Economic Viability

✅ PASS if:
  - Decile spread > 0.20% daily (after costs)
  - Portfolio Sharpe > 0.5 annualized
  - Max drawdown < 15%
  - Win rate > 55% daily

Phase 3: Stability

✅ PASS if:
  - Rolling 20d Rank IC doesn't decay
  - Regime-specific performance consistent
  - Out-of-sample (test set) ≈ in-sample (validation)
  - No data leakage detected

Phase 4: Interpretability

✅ PASS if:
  - Top 5 gating features are sensible (momentum, volume, relative strength)
  - Feature importance doesn't change drastically by regime
  - Model predictions correlate with fundamental semiconductor factors

Implementation Roadmap

Total Time to Tradable: ~40 minutes

Files Generated This Session

/home/aimikamirai/projects/mm-dgx-tft/
  ├─ semis_signal_pipeline.py          (Feature engineering)
  ├─ semis_ranking_train.py            (Training + validation)
  ├─ semis_portfolio_backtest.py        (Portfolio simulation)
  ├─ sessions/2026-05-30/
  │  ├─ SEMIS_RANKING_ENGINE_STRATEGY.md (This file)
  │  └─ SEMIS_RESULTS_FINAL.md          (Results, generated after training)

Outputs Directory:

/tmp/
  ├─ semis_tft_results.json            (Validation metrics)
  ├─ semis_backtest_results.json        (Portfolio backtest)
  ├─ semis_pipeline_config.json         (Feature configuration)
  └─ semis_tft_best.pth                (Trained model weights)

Expected Outcome

If all 10 improvements work as designed:

Signal Improvement:

• Rank IC: 0.0391 → 0.045-0.055 (+25-40%)
• Z-score: 2.82σ → 3.2-3.8σ (stronger significance)

Economic Improvement:

• Decile Spread: 0.08% → 0.20-0.25% (+150-200%)
• Turnover: 100% → 20% daily (weekly rebalance)
• Costs: 0.1% → 0.03% per day
• Net P&L: -0.02% → +0.15%+ per day
• Annualized Sharpe: 0.2 → 0.6-0.8

Viability:

• FROM: Marginal, barely above noise
• TO: Tradable, small but consistent edge

Next Steps (Post-Training)

1. Review validation outputs (Rank IC by horizon)
2. If Rank IC > 0.045: Proceed to backtest
3. If backtest Sharpe > 0.4: Ready for live testing on small account
4. If stable for 60+ days: Scale to full position size

If any metric misses target → iterate on:

• Expand universe to 20 semis? (add mid-cap players)
• Increase model capacity (d_model=256, 4 layers)?
• Add sector momentum features?
• Adjust confidence filter (30% vs 60%)?

Generated: 2026-05-30 UTC
Strategy Owner: Anand
Status: 🔄 IMPLEMENTATION IN PROGRESS