Octo

Thanks hu-po. love to see more robotics papers

This seems wrong to me: [Task] + [Observation 1] -> [Readout 1] + [Observation 2] -> [Readout 2] + ... + [Observation n] -> [Readout n] Is it how it is fed to the model, as a growing historical context? You probably don't need an entire history of motion to accomplish the [Task], you can just view any subrange of the process as a subtask [Task] + [Observation at Step n] + [Summary of interactions prior to Step n], decreasing the historical context and allowing more compute budget for action refinement context. So initial can probably be reframed as separate subtasks, with potentially much smaller context thus requiring less time to calculate: Fame 1 [Task] + [Observation 1] -> [Readout 1] Frame 2 [Task] + [Observation 2] + [Summarised difference between predicted outcome of Readout 1 and Observation 2 (to account for possible errors)] -> [Readout 2] .... [Task] + [Observation n] + [Summarised difference between predicted outcome of Readout n-1 and Observation n (to account for possible errors)] -> [Readout n] This way the model would be able to self-adjust based on the environment and the state of the robot, f.e. for the arbitrary step n: the robot was supposed to be in the state [Readout n-1], but instead is in state [Observation n], suggesting, that there was an outside force applied to the "limb" that wasn't accounted for, thus [Readout n] should be adjusted to counteract it. In practice, if the robot is tasked to push an object, and miscalculated the opposing force (ie, difference between pushing a round object vs pushing a cubic object (friction), or difference between pushing uphill or downhill (objects on a slope), or lifting/putting down, or some actuators have some slack/backlash/etc), and after first "frame" of motion, the actual position of the "pushed" object can differ from prediction, and that can be adjusted for on "per subtask" basis. It would also allow for "easing", making motion smoother. Sorry for my mumbling...

Problem with the robotics not providing feedback, it’s the end game. That’s how they deploy.

Summary starts at 1:29:47

Agree that the path forward is using video of humans to train