Multi Modal Agent Ts

multi-modal-agent-ts

**Vision + text + audio TypeScript AI agent** Clone → `npm install` → `npm run dev` (no build step) → process any input modality. Use `npm run build` before `npm start` or when importing from `./dist`.

[](https://www.typescriptlang.org/) [](https://sdk.vercel.ai/) [](https://opensource.org/licenses/MIT) [](https://nodejs.org/)

---

What is this?

`multi-modal-agent-ts` is a **TypeScript agent that can combine images, audio, and text** in one flow. It uses multimodal chat models (OpenAI GPT‑4o, Anthropic Claude 3.x, Google Gemini) via the [Vercel AI SDK](https://sdk.vercel.ai/), OpenAI **Whisper** for speech-to-text (`whisper-api`), optional **local Whisper** via `@xenova/transformers` (`whisper-local`), and **Ollama** (e.g. LLaVA) for local vision without an API key (`local/llava`).

You can:

• Analyze images — paths, https URLs, or buffers; optional imageDetail for OpenAI.
• Transcribe audio — file path, URL, or buffer; Whisper API or optional local ASR.
• Fuse modalities — one user message with text + transcripts + images.
• Stream answers — processStream() with transcript / vision / answer chunks.
• Run an HTTP API — POST /process with JSON or multipart uploads.

---

Installation

cd multi-modal-agent-ts
npm install

• npm run dev — runs the API with tsx (TypeScript directly; no dist/ needed).
• npm run build — needed for npm start, Docker, or importing from ./dist/....

Requires **Node.js 20+**. For video frame/audio extraction and microphone capture helpers, install **ffmpeg** and use a normal audio device (see [Audio / mic](#audio--mic)).

Copy `.env.example` to `.env` and set keys for the provid