Site To Md

site-to-md

[](https://badge.fury.io/rb/site-to-md) [](https://github.com/tmaier/site-to-md/actions/workflows/ci.yml?query=branch%3Amain) [](LICENSE.txt)

This command-line tool aggregates content from HTML pages into a single, streamlined markdown file. It removes unnecessary HTML elements to reduce token usage and provides an easily uploadable format for AI tools like Claude AI or ChatGPT.

This tool can also be used to create a [llms.txt or llms-full.txt](https://llmstxt.org/). For this use case, consider to add this tool to the build pipeline of your satic website.

Features

• Converts all HTML files in a directory (and subdirectories) to markdown
• Extracts content from tag (falls back to if not found)
• Preserves document structure with frontmatter metadata
• Maintains proper markdown formatting for:
  • Headers
  • Lists
  • Tables
  • Code blocks
  • Links
  • And more...
• Command-line interface for easy integration

Installation

gem install site-to-md

Or add to your Gemfile:

gem 'site-to-md'

Usage

Command Line

Basic usage:

site-to-md convert path/to/site

Specify output file:

site-to-md convert path/to/site -o output.md

Get help:

site-to-md help

Docker Image

You can use the [site-to-md tool via a Docker image](https://github.com/tmaier/site-to-md/pkgs/container/site-to-md), making it convenient to include in your build pipeline for static websites.

Example: GitLab CI

Here's an example GitLab CI configuration. This configuration includes a job `llms-full-txt` that uses the Docker image to convert HTML files in the public folder and generates the llms-full.txt file in the same folder. This

llms-full-txt:
  image: ghcr.io/tmaier/site-to-m