Codex LD-TM LSTM Implementation, Architecture, Model Design, and Runbook

Date: 2026-04-22

Implementation name: LD-TM LSTM, Long-term Deep Temporal Model

Status: implemented locally. Azure deployment is intentionally deferred.

Executive Summary

LD-TM is a daily, multi-horizon prediction pipeline for the DGX trading system. It trains one LSTM model per ticker, writes inference outputs into PostgreSQL, fills actual results later, exposes prediction history through a Streamlit dashboard, and supplies live database context to a local LLM query layer.

The current implementation covers 103 tickers, including NDX-style equities plus TQQQ and SQQQ. The latest observed snapshot set contains 103 ticker rows for 2026-04-22. The database currently holds 104 snapshot rows across 103 tickers, with first observed snapshot date 2026-04-21 and latest snapshot date 2026-04-22.

Observed local performance from ldtm_run_log:

Training aggregate:

Inference aggregate:

Current observed container memory snapshot:

GPU query result:

The GB10 runtime reports GPU memory as N/A through nvidia-smi, so the orchestrator uses a fallback concurrency model for MIG-like or virtual GPU reporting.

Implementation Inventory

Architecture

market_data_daily
       |
       v
OHLCV feature builder
       |
       v
30-trading-day normalized windows
       |
       v
LD-TM LSTM model per ticker
       |
       +--> trainer.py -> checkpoints -> ldtm_run_log
       |
       +--> predict.py -> ldtm_run_log
                         |
                         v
                 snapshot_writer.py
                         |
                         v
              ldtm_daily_snapshots
                         |
                         +--> snapshot_fillback.py -> actuals, errors, accuracy
                         |
                         +--> ldtm_accuracy_30d view
                         |
                         +--> dashboard/app.py
                         |
                         +--> llm/llm_query.py
                         |
                         +--> optional export_to_blob.py

Runtime services:

AI Model Design

LD-TM uses one trained model per ticker. The model consumes the most recent 30 valid trading days and predicts three future closing prices:

Input tensor:

X shape = (batch_size, 30, 11)

The 11 features are:

Network architecture:

Input: (B, 30, 11)
  |
  v
LSTM, hidden_size=128, num_layers=2, dropout=0.2, batch_first=True
  |
  v
Last hidden state: (B, 128)
  |
  +--> Head next_day:    Linear(128, 64) -> ReLU -> Linear(64, 1)
  +--> Head next_monday: Linear(128, 64) -> ReLU -> Linear(64, 1)
  +--> Head one_month:   Linear(128, 64) -> ReLU -> Linear(64, 1)

Training settings:

Mathematical Concepts and Formulas

Daily log return:

ret1_t = ln(close_t / close_(t-1))

Five-day log return:

ret5_t = ln(close_t / close_(t-5))

Moving average for window length k:

MA_k(t) = (1 / k) * sum(close_(t-i), i = 0 to k-1)

RSI:

RS_t = avg_gain_t / (avg_loss_t + epsilon)
RSI_t = 100 - 100 / (1 + RS_t)

Per-window min-max normalization:

x_scaled[t, f] = (x[t, f] - min_f) / (max_f - min_f + epsilon)

Future-close label scaling:

y_scaled = (future_close - close_min) / (close_max - close_min + epsilon)

Prediction inverse transform:

pred_close = y_pred_scaled * (close_max - close_min) + close_min

Multi-horizon training loss:

loss = MSE(y_hat_next_day, y_next_day)
     + MSE(y_hat_next_monday, y_next_monday)
     + MSE(y_hat_one_month, y_one_month)

Percent error:

pct_error = ((predicted_close - actual_close) / actual_close) * 100

Next-day direction:

next_day_direction_pred =
    "UP"   if next_day_close_pred > run_date_close
    "DOWN" otherwise

Direction accuracy:

direction_accuracy_pct =
    100 * count(next_day_direction_correct = true)
        / count(next_day_direction_correct is not null)

Implied one-month return:

implied_1m_return_pct =
    ((one_month_close_pred / next_day_close_pred) - 1) * 100

PostgreSQL implementation:

ROUND((((one_month_close_pred / NULLIF(next_day_close_pred, 0)) - 1) * 100)::numeric, 2)

The explicit ::numeric cast fixed the PostgreSQL error:

function round(double precision, integer) does not exist

Database Tables and Views

market_data_daily

Expected upstream table:

Primary key:

(ticker, date)

ldtm_run_log

Run ledger for training and inference:

ldtm_daily_snapshots

Daily prediction and evaluation table, one row per (run_date, ticker).

Observed latest sample outputs for 2026-04-22:

ldtm_accuracy_30d

Dashboard view for recent accuracy:

At the time of this document, the view exists but evaluation fields are mostly empty because actuals have not yet matured for the newest predictions.

Outputs Returned

Training final output:

[LDTM] RESULT: ticker=AAPL epochs=... val_loss=... duration=...s

Inference JSON:

{
  "ticker": "AAPL",
  "next_day_close": 268.55,
  "next_monday_close": 268.74,
  "one_month_close": 275.05
}

Snapshot writer output:

[snapshot_writer] Wrote N snapshots for YYYY-MM-DD

Fill-back output:

[snapshot_fillback] Updated N rows

Dashboard URL:

http://localhost:8501

Dashboard tabs:

Current Prediction Distribution

For latest run date 2026-04-22:

Top implied one-month returns:

Lowest implied one-month returns:

These are model outputs, not trading advice.

Scheduler Design

run_ldtm_monthly_retrain.sh is scheduled every Sunday at 1 AM, but exits unless the day of month is 1 through 7.

Runbook

1. Start from project root

cd /home/aimikamirai/projects/dgx-trading-system

2. Confirm .env

Required variables:

DB_HOST
DB_PORT
DB_NAME
DB_USER
DB_PASSWORD

Optional variables:

AZURE_BLOB_CONN_STR
AZURE_BLOB_CONTAINER
AZURE_BLOB_URL
LLM_BASE_URL
LLM_API_KEY
LLM_MODEL

Azure deployment is not required. Blob export remains optional and should be treated as inactive until AZURE_BLOB_CONN_STR exists.

3. Build model image

docker build -t model-ldtm ./model/ldtm

4. Initialize LD-TM run log

bash model/ldtm/run_ldtm.sh --init-db

5. Initialize snapshot schema

docker exec -i trading-postgres psql -U postgres -d trading < model/ldtm/snapshots_schema.sql

6. Train one ticker

bash model/ldtm/run_ldtm.sh --ticker AAPL --mode train --epochs 100 --env_file .env

7. Infer one ticker

bash model/ldtm/run_ldtm.sh --ticker AAPL --mode infer --env_file .env

8. Train and infer one ticker

bash model/ldtm/run_ldtm.sh --ticker AAPL --mode both --env_file .env

9. Train or infer a group

bash model/ldtm/run_ldtm.sh --group mega_cap --mode train --parallel 4 --env_file .env

Available groups:

mega_cap, semis, software, internet, consumer, healthcare, industrial,
telecom, financials, hardware, intl, etfs

10. Infer all tickers

bash model/ldtm/run_ldtm.sh --all --mode infer --parallel 16 --env_file .env

11. Run the daily inference pipeline manually

bash schedule/run_ldtm_infer.sh

This performs:

all ticker inference -> snapshot writer -> snapshot fill-back

12. Build and run dashboard

docker build -t trading-dashboard ./dashboard
docker run -d --name trading-dashboard-test \
  --network host --env-file .env \
  -e DATA_SOURCE=db \
  trading-dashboard

Open:

http://localhost:8501

Health check:

curl -s http://localhost:8501/healthz

13. Run LLM query

The LLM query layer expects an OpenAI-compatible endpoint:

http://localhost:8000/v1

Example:

docker compose --profile llm run --rm llm-query \
  python llm_query.py --group mega_cap \
  --question "Which tickers have the strongest 1-month signal and why?"

14. Install cron jobs

Ensure scripts are executable:

chmod +x schedule/run_ldtm_infer.sh \
  schedule/run_ldtm_canary_retrain.sh \
  schedule/run_ldtm_monthly_retrain.sh \
  schedule/run_blob_export.sh

Install:

bash schedule/install_cron.sh

Verify:

crontab -l

15. Optional Blob export

Do not deploy to Azure right now. When Blob export is desired later:

1. Add AZURE_BLOB_CONN_STR to .env.
2. Optionally set AZURE_BLOB_CONTAINER.
3. Run:

docker compose --profile blob run --rm blob-export

Operations and Troubleshooting

PostgreSQL ROUND error

Symptom:

function round(double precision, integer) does not exist

Fix:

ROUND((some_double_expression)::numeric, 2)

No snapshot rows

Check latest inference logs:

SELECT ticker, mode, status, next_day_close, run_at
FROM ldtm_run_log
WHERE mode = 'infer'
ORDER BY run_at DESC
LIMIT 20;

Accuracy is empty

This is expected immediately after first snapshots. ldtm_accuracy_30d needs future market data to fill next_day_actual, next_monday_actual, and one_month_actual.

Dashboard shows no data

Check:

SELECT COUNT(*), MAX(run_date) FROM ldtm_daily_snapshots;

Then confirm DATA_SOURCE=db and DB env vars are available inside the dashboard container.

LLM query fails

Check:

docker ps

Confirm the local LLM endpoint is up:

trtllm-server on port 8000

Performance Notes and Efficiency Plan

The current LD-TM model is small enough that per-ticker inference is fast, averaging about 0.605 seconds. Training is dominated by per-ticker dataset loading, DataLoader overhead, Docker startup overhead, and epoch count before early stopping. Average training time is 44.78 seconds per ticker, with median 35.71 seconds and P90 84.08 seconds.

The dashboard memory footprint is small, around 40 MiB at the observed moment. The local LLM server is the major resident memory consumer at about 2.91 GiB CPU/container memory, not counting GPU allocation that may not be reported by the GB10 runtime.

The orchestrator estimates GPU slots as:

slots = max(1, min(16, vram_mib / 600))

When GPU memory is N/A, it uses:

fallback_slots_per_gpu = 4

Recommended improvements:

FP16 AMP is already used in training when CUDA is available. Flash Attention is not the primary lever for LD-TM because the model is an LSTM, not a Transformer attention stack. Flash Attention is more relevant to the local Mistral/TensorRT-LLM service. For LD-TM, better acceleration paths are FP16 inference, warm model execution, ONNX export, TensorRT engines where supported, and a persistent batched service.

Design Notes and Risks

1. Per-window normalization makes the model robust to price scale, but predictions can exceed the local min/max regime.
2. Accuracy metrics are intentionally sparse until actual market closes mature.
3. Top and bottom implied returns are raw model signals and need accuracy weighting before becoming trading candidates.
4. One model per ticker is operationally simple and isolates ticker behavior, but it prevents cross-sectional learning.
5. The local LLM is an explanation and query layer, not the source of prediction truth.
6. Blob export is ready as an optional static publishing path, but Azure deployment is paused by request.

Recommended Next Steps

1. Let fill-back run for several trading days before judging direction accuracy.
2. Add a dashboard confidence score blending implied return, direction accuracy, evaluated days, and average absolute percent error.
3. Build a persistent inference service to reduce all-ticker inference time.
4. Add a feature cache table or parquet cache for engineered features.
5. Add benchmark logging for GPU memory, CPU memory, wall-clock per stage, and queue time.
6. Add a model card row per ticker with last train date, epochs, validation loss, inference latency, and latest signal.
7. Keep Azure deployment deferred until local accuracy, cost, and security posture are satisfactory.