Local AI & LLM Capabilities — User Guide
This guide covers what’s running on the self-hosted AI stack (CT103), how to access it, and which tool fits which task.
The Stack
- Hardware: Tesla P100 (16GB, compute) + Tesla P40 (24GB, compute + hardware transcode)
- Ollama — runs the actual models (llama3.2, qwen3:14b, qwen3.6:27b, mistral-small3.2:24b)
- llama-swap — hot-swaps GGUF models on demand for advanced/experimental use
- SmarterRouter — the recommended entry point for everything. Automatically profiles and picks the best available model for a given request instead of you choosing manually. OpenAI-compatible API at
ai.xinle.biz/router/ - OpenWebUI — the chat interface for interactive use, already wired to SmarterRouter:
ai.xinle.biz - AnythingLLM — for RAG / knowledge-base work (feeding it documents and asking questions against them), not general chat
- ComfyUI — local image generation:
ai.xinle.biz/comfyui/
How to Access It
| Need | Use | Where |
|---|---|---|
| Quick chat / drafting | OpenWebUI | ai.xinle.biz |
| Ask questions against a specific set of documents | AnythingLLM | ai.xinle.biz/ollama/ (via AnythingLLM UI) |
| Programmatic access (scripts, n8n, automation) | SmarterRouter API (OpenAI-compatible) | ai.xinle.biz/router/ |
| Image generation | ComfyUI | ai.xinle.biz/comfyui/ |
| Text-to-speech / speech-to-text | openedai-speech | ai.xinle.biz/tts/ |
Recommendation: Always route through SmarterRouter
Rather than calling a specific model directly, point automation (n8n workflows, scripts, future WordPress MCP integrations) at SmarterRouter’s endpoint. It handles model selection automatically based on the task profile, so you don’t have to hand-pick a model per request — and it’s already the standard everything else in the stack uses.
Known Model Limitations
qwen3.6:27bis documented as impractical on this hardware (MoE architecture causes 180s+ timeouts) — avoid for interactive useqwen3:14bis the recommended general-purpose default- No dedicated code-generation model is pulled yet — flagged for a future addition (e.g. a Qwen Coder variant) for RPG Maker MZ / Phoenix project scripting work
Performance & Cost Monitoring
Live GPU utilization, power draw, VRAM, temperature, and transcode load are tracked in Grafana:
CT103 AI/GPU Performance Dashboard →
Note: the power-cost panel currently uses a placeholder rate ($0.14/kWh) — update once the actual utility rate is provided.