RoboCAT

[](https://discord.gg/qUtxnK2NMf)

ROBOTCAT


RoboCAT is a self-improving foundation agent for robotic manipulation developed by DeepMind Robotics. The model architecture of RoboCAT is similar to the RT-1 model. It follows a tokenization approach where robotics images, proprioception, and future actions are tokenized.

• Help with the implementation in the Agora discord,

---

Install

`pip install robotcat` 

---

Basic Usage

import torch 
from robocat.model import RoboCat

model = RoboCat()

video = torch.randn(2, 3, 6, 224, 224)
instructions = [
    "bring me that apple"
]

result = model.forward(video, instructions)
print(result)

Generate Data

• RoboCat can self generate data with images and or video, to use simply use the generator classes -> then tokenize and or tensorize using your custom strategy

#for images
from robotcat import ImageDataGenerator

generator = ImageDataGenerator()

prompt = "Robot picking up cup in kitchen"
generator.generate(prompt)

and or for videos

from robocat import VideoDataGenerator, RoboCat

generator = VideoDataGenerator()
model = RoboCat()

prompt = "Robot picking up cup in kitchen in first person"
video = generator.generate(prompt)
video = transform_to_tensors(video) #(batch, frames, actions, bins)

model(video, prompt)

---

Architecture

The architecture consists of the following key components:

1. Tokenizer: RoboCAT learns a tokenizer for robotics images. It tokenizes proprioception and future actions in a straightforward manner, enabling the prediction of future action tokens.

2. Transformer Model: The tokenized inputs are fed into a Transformer model. The Transformer model predicts future action sequences based on the input tokens. This allows the model to perform a wide range of robotic tasks using a unified interface.

3. Action Spaces