Deploying robots in data centers and advanced manufacturing brings us back to three customer requirements: get useful work running quickly, handle equipment that is only available on site, and keep sensitive data under customer control.
A robot that needs weeks of task-specific data collection can slow a critical timeline. A policy that depends on cloud inference may be unusable where external model APIs are prohibited.
We built Shepherd-0, a robot learning system that learns unseen manipulation tasks from a few demonstrations in minutes and can train and operate entirely locally. It brings together pretrained models, the Shepherd Adapt posttraining pipeline, and local robot execution.
01 / THE MODEL
Model architecture
At the core of Shepherd-0 is a pretrained manipulation model that takes visual tokens from camera observations and language tokens describing the task. A flow-matching head generates action sequences containing both delta end effector poses and delta joint-angle poses. A separate text-output pathway generates language. The segmentation head combines shared visual features with a separate language query to predict pixel-level segmentation masks.
We jointly pretrain the model on diverse manipulation demonstrations, reasoning tasks through the language head, and image segmentation tasks through the segmentation head. Our core pretraining data combines real-world demonstrations with simulation generated for our bimanual embodiment.
Actions relative to the starting pose
For each arm, let T0 = (R0, p0) and q0 denote the measured end-effector pose and joint angles at the start of an action chunk. At waypoint k, Tk = (Rk, pk) and qk are the target pose and joint angles. We define:
The flow-matching head predicts a sequence of per-arm action targets:
H is the prediction horizon, ρ6 encodes a rotation using its first two columns, and gk is absolute gripper closure. Both arms use the same definitions with their own measured starting poses.
Every waypoint uses the same chunk-start reference. Translation is expressed in the starting end-effector frame, giving tool motion a common representation across embodiments. For training data from other embodiments, we supervise only the delta end effector outputs.
02 / POSTTRAINING
Shepherd Adapt
Customer-specific operations are often underrepresented in pretraining, and collection time is limited. We built Shepherd Adapt, a pipeline that takes a new manipulation task from first demonstration to a deployed policy on the robot in under 30 minutes.
Our labeling model follows labeling instructions set at the start of data collection to generate language and segmentation labels. Labeling runs in parallel with data collection. The annotated demonstrations then feed training, local deployment, and the robot’s inference stack.
These annotations enable multi-head supervision: the action, language, and segmentation heads learn jointly from each annotated demonstration.
03 / EVALUATION
Few-shot learning
With ten demonstrations, we can posttrain Shepherd-0 for a new task in about one minute. We evaluate few-shot adaptation on six simulation tasks, alongside real-world cable insertion, cable routing, and fuse holder handling.
In simulation, each model receives ten demonstrations per task and fifty total gradient steps, with a learning rate of 2e-5 and batch size 128. Cable insertion uses 300 training steps because performance at 50 steps was low. Cable routing uses 300 training steps. All demonstrations come from an initial round of data collection; no DAgger data is collected.
We compare action-only adaptation with multi-head supervision, which jointly trains the action, language, and segmentation heads using annotations from Shepherd Adapt.
View results as a table
| Task | Shepherd-0 — multi-head supervision | Shepherd-0 — action only | π0.5 | ABC Official200k | MolmoAct2 Full FT |
|---|---|---|---|---|---|
| Cable insertion (300 steps) | 60.0% | 65.0% | 55.0% | 10.0% | 0.0% |
| Cable routing (300 steps) | 95.0% | 75.0% | 80.0% | 0.0% | 20.0% |
| Fuse holder handling | 62.5% | 43.8% | 50.0% | 25.0% | 18.8% |
| Parts sorting (sim) | 83.3% | 84.2% | 75.0% | 0.0% | 0.8% |
| Handoff objects between grippers (sim) | 80.0% | 85.0% | 55.0% | 0.0% | 0.0% |
| Bottle binning (sim) | 97.6% | 97.6% | 84.1% | 43.9% | 0.0% |
| Plate racking (upright) (sim) | 20.4% | 22.4% | 8.2% | 2.0% | 0.0% |
| Pouring beads (sim) | 36.5% | 0.0% | 0.0% | 0.0% | 0.0% |
| Place object relative to another object (sim) | 55.0% | 0.0% | 20.0% | 0.0% | 0.0% |
Shepherd-0 achieves state-of-the-art few-shot performance among the open-weight models evaluated in our setup. Multi-head supervision powered by Shepherd Adapt clearly helps on more challenging, high-mix tasks. Most evaluated models still struggle as task variation increases; on lower-mix tasks, Shepherd-0 generally reaches reasonable performance from just ten demonstrations.
Shepherd-0 performs well on both real-world and simulation tasks.
04 / THE ROBOT SYSTEM
Putting it together
We posttrain Shepherd-0 for manipulation tasks and deploy the adapted policies as callable subtask capabilities. shepherd-0-reasoning runs the procedure, maintaining memory and task status as it coordinates the robot. It calls SLAM-based navigation to reach a location, then invokes Shepherd-0 for manipulation.
Navigation and manipulation outcomes update task status and inform the next step. The system communicates with people when required by the procedure, or when it needs help or confirmation to continue.
Together, embodiment-focused pretraining and richer posttraining supervision help a robot learn unfamiliar work with less customer time and data. Local deployment must cover the full workflow, including annotation wherever external APIs are prohibited.
05 / LOOKING AHEAD
What’s next
Despite the faster iteration cycles enabled by Shepherd-0, reaching production-level reliability for a new customer or task still requires humans in the loop.
Our next goal is a continuously learning system that improves task performance on its own. We are building toward a loop in which robots learn from execution, adapt their policies, and improve quickly without manual intervention.