Private Ml Sdk

Private ML SDK

A secure and verifiable solution for running Large Language Models (LLMs) in Trusted Execution Environments (TEEs), leveraging NVIDIA GPU TEE and Intel TDX technologies.


Overview

Private ML SDK provides a secure environment for running LLM workloads with guaranteed privacy and security, preventing unauthorized access to both the model and user data during inference operations. The solution leverages NVIDIA's TEE GPU technology (H100/H200/B100) and Intel CPUs with TDX support to ensure that AI model execution and data processing remain fully protected within secure enclaves.

Key features:

• Tamper-proof data processing
• Secure execution environment
• Open source and reproducible builds
• Verifiable execution results
• Nearly native speed performance (up to 99% efficiency)

Architecture

The system consists of several core components:

• Secure Compute Environment: TEE-based execution environment
• Remote Attestation: Verification of the TEE environment
• Secure Communication: End-to-end encryption between users and LLM
• Key Management Service (KMS): Key management service to manage keys for encryption and decryption

Getting Started

Build the TDX guest image

Prerequisites:

• Install Docker:

  curl -fsSL https://get.docker.com -o get-docker.sh
  sudo sh get-docker.sh

• Add the current user to the docker group:

  sudo usermod -aG docker $USER
  newgrp docker  # Apply group changes without logout

• Verify Docker installation:

  docker --version
  docker run hello-world

Clone the repository and build the TDX guest image:

git clone https://github.com/nearai/private-ml-sdk --recursive
cd private-ml-sdk/
./build.sh

If everything goes well, you should see the images files in `private-ml-sdk/images/`.

There are two image directories:

• dstack-nvidia-*/: the production image without developer tools.
• `dstack