What it can do

RT-1 is able to perform real-time tasks based on an image sequence with natural language instructions.

What’s cool?

• was able to perform 700 tasks with a 97% success rate
• can operate at 3Hz

Overall Architecture

The RT-1 architecture consists of the following components:

• Universal Sentence Encoder
• EfficientNet
• Feature-wise Linear Modulation (FiLM)
• TokenLearner
• Positional Encoding
• Transformer

Universal Sentence Encoder

We first want a way to encode the natural language instruction, so RT-1 adopts the method from Universal Sentence Encoder to do this:

Universal Sentence Encoder

This paper proposes two primary methods for sentence embedding:

• Transformer based
• Deep Average Network

Whats’s good about this encoder?