Ark310

Local LLM Setup — AI skill for Claude Code

AI community

Runbook and PowerShell automation to host two isolated local LLMs (KYC analyst and KB-chatbot reasoning) on one 8 GB RTX GPU: Ollama on localhost behind a per-user bcrypt Basic-auth Caddy gateway, sin.

How to install Local LLM Setup

This entry records only its repository, not the path inside it, so there is no exact command to give. Open Ark310/local-llm-setup and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Local LLM Setup does

Alternatives in AI

Bot On Anything — A large model-based chatbot builder that can quickly integrate AI models (including ChatGPT, Claude, Gemini) i 4.2k ★
Chatwise Releases — The fastest AI Chatbot for any LLM 1.4k ★
Rb Setup — First-time setup 1.3k ★

README

Local LLM Setup: Two Isolated Models on One 8 GB GPU

Runbook and PowerShell automation for hosting two separate local LLM workloads (a KYC-analyst model and a knowledge-base chatbot's reasoning model) on one Windows gaming PC, with an authenticated, per-user LAN gateway in front of Ollama.

![PowerShell](https://img.shields.io/badge/PowerShell-5.1-5391FE?logo=powershell&logoColor=white) ![Ollama](https://img.shields.io/badge/Ollama-local%20LLM-000000?logo=ollama) ![Caddy](https://img.shields.io/badge/Caddy-basic--auth%20gateway-1F88C0?logo=caddy) ![Qwen](https://img.shields.io/badge/model-qwen2.5%3A7b--instruct-6f42c1) ![GPU](https://img.shields.io/badge/GPU-RTX%205060%20Ti%208GB-76B900?logo=nvidia) ![License](https://img.shields.io/badge/license-MIT-green) ![Built with Claude Code](https://img.shields.io/badge/Built%20with-Claude%20Code%20%2B%20Codex-d97757?logo=anthropic)

🧭 Part of **Abdul Raqeeb Khatri's portfolio**: [📂 Hub](https://github.com/Ark310/portfolio) · [🌐 Site](https://ark310.github.io) · [💼 Experience](https://github.com/Ark310/experience)

Overview

A team wanted a *local* reasoning model for its internal knowledge-base chatbot, served from a shared AI PC on the office LAN. The catch: the same PC (AMD Ryzen, **RTX 5060 Ti 8 GB**, 32 GB RAM) already hosts a **KYC-analyst model** whose prompts, test data and routes must never mix with the chatbot's. Two 7B-class models don't fit in 8 GB of VRAM together, and raw Ollama has no authentication.

This repo holds the rules, the design guide and the scripts that make the setup repeatable:

One model server, two named models, one active at a time (OLLAMA_MAX_LOADED_MODELS=1, ollama stop to switch).
Ollama stays on localhost. Colleagues reach the reasoning model only through a Caddy reverse proxy with per-user Basic auth.
Hard separation between the KYC and chatbot folders, prompts, test data, logs and routes,