laoposkj

Multi Modal Agent TS — DevOps skill for Claude Code

DevOps community

TypeScript multimodal AI agent: GPT-4o / Claude / Gemini + Whisper + Ollama (LLaVA).

How to install Multi Modal Agent TS

This entry records only its repository, not the path inside it, so there is no exact command to give. Open laoposkj/multi-modal-agent-ts and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Multi Modal Agent TS does

TypeScript multimodal AI agent: GPT-4o / Claude / Gemini + Whisper + Ollama (LLaVA). REST API, streaming, Docker. Vision, audio & text in one flow.

Alternatives in DevOps

Claudable — Claudable is an open-source web builder that leverages local CLI agents, such as Claude Code, Codex, Gemini CL 4k ★
Azure Functions TypeScript — Azure Functions for TypeScript 1.8k ★
Codeg — Aggregate and browse AI coding agent sessions (Claude Code, Codex, Gemini CLI, etc.) in one place 850 ★

README

multi-modal-agent-ts

**Vision + text + audio TypeScript AI agent** Clone → `npm install` → `npm run dev` (no build step) → process any input modality. Use `npm run build` before `npm start` or when importing from `./dist`.

[![TypeScript](https://img.shields.io/badge/TypeScript-5.8-blue.svg)](https://www.typescriptlang.org/) [![Vercel AI SDK](https://img.shields.io/badge/Vercel%20AI%20SDK-6.0-black)](https://sdk.vercel.ai/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Node.js](https://img.shields.io/badge/Node.js-20+-green)](https://nodejs.org/)

What is this?

`multi-modal-agent-ts` is a **TypeScript agent that can combine images, audio, and text** in one flow. It uses multimodal chat models (OpenAI GPT‑4o, Anthropic Claude 3.x, Google Gemini) via the [Vercel AI SDK](https://sdk.vercel.ai/), OpenAI **Whisper** for speech-to-text (`whisper-api`), optional **local Whisper** via `@xenova/transformers` (`whisper-local`), and **Ollama** (e.g. LLaVA) for local vision without an API key (`local/llava`).

You can:

Analyze images — paths, https URLs, or buffers; optional imageDetail for OpenAI.
Transcribe audio — file path, URL, or buffer; Whisper API or optional local ASR.
Fuse modalities — one user message with text + transcripts + images.
Stream answers — processStream() with transcript / vision / answer chunks.
Run an HTTP API — POST /process with JSON or multipart uploads.

Installation

cd multi-modal-agent-ts
npm install

npm run dev — runs the API with tsx (TypeScript directly; no dist/ needed).
npm run build — needed for npm start, Docker, or importing from ./dist/....

Requires **Node.js 20+**. For video frame/audio extraction and microphone capture helpers, install **ffmpeg** and use a normal audio device (see [Audio / mic](#audio--mic)).

Copy `.env.example` to `.env` and set keys for the provid