Back to Blog
ai-infralocal LLMsautomated publishingFastAPIDockerfswatchGit operations

Automated Content Publishing Pipeline with Local LLMs

Built an automated content publishing pipeline using local LLMs that processes raw session notes into structured markdown and publishes them to a blog GitHub repository if they meet publishability criteria.

April 20, 2026·1 min read

Summary

Built an automated content publishing pipeline using local LLMs that processes raw session notes into structured markdown and publishes them to a blog GitHub repository if they meet publishability criteria.

What I Did

Core pipeline uses two local LLM models (Qwen 32B and Llama 3 8B) for extracting structure and formatting content. Pipeline stages include extraction, markdown writing, cleaning, metadata detection, and synchronization to a Git repo. FastAPI service provides HTTP endpoints for liveness checks and publishing. Docker containerizes the service with necessary dependencies and volume mounts for file sharing. fswatch monitors a directory for new files, triggers pipeline processing, and handles Git operations.

Key Technical Findings

  • Core pipeline uses two local LLM models (Qwen 32B and Llama 3 8B) for extracting structure and formatting content.
  • Pipeline stages include extraction, markdown writing, cleaning, metadata detection, and synchronization to a Git repo.
  • FastAPI service provides HTTP endpoints for liveness checks and publishing.
  • Docker containerizes the service with necessary dependencies and volume mounts for file sharing.
  • fswatch monitors a directory for new files, triggers pipeline processing, and handles Git operations.

Commands

  • docker build -t publish-service .
  • docker run -d --name publish-service --restart unless-stopped -p 8088:8088 -v "$HOME/Documents/blogs:/app/inbox" -v "$(pwd)/drafts:/app/drafts" -v "$(pwd)/published:/app/published" -v "$(pwd)/processed:/app/processed" -v "/Users/mikamirai/projects/auto-publsh-blog:/Users/mikamirai/projects/auto-publsh-blog" publish-service
  • bash watcher/watch-blogs.sh
  • curl http://localhost:8088/health
  • curl -X POST http://localhost:8088/publish-file -H "Content-Type: application/json" -d '{"path": "inbox/day3.md"}'

Lessons Learned

  • Git operations must run where credentials are configured.
  • Volume mounts facilitate sharing state between container and host in development.
  • Debouncing file write events and using a state file prevent double-processing.
  • Parsing JSON responses from the pipeline provides awareness of processing outcomes.
  • Separating logic into distinct modules improves testability and extensibility.