Jekyll2026-10-29T19:45:05+00:00https://leo01110111.github.io/log/feed.xmlblank | LogA simple, whitespace theme for academics. Based on [*folio](https://github.com/bogoli/-folio) design. Hand Tracking and Voice Commands for Collecting BC Data in Sim2026-10-29T19:39:35+00:002026-10-29T19:39:35+00:00https://leo01110111.github.io/log/153935I experimented with using a hand tracker and voice-activated commands (start, pause, resume) to collect behavior cloning data in simulation. The webcam tracks my hands and the sim’s bimanual arms follow, while spoken commands control the recording.

It was too hard. Controlling two arms in 3D through a webcam, with no feedback in my hands, was far less precise than the leader arms.

It did help me understand why the wrist cams matter so much more than the ego cam for control. While I was teleoperating, the wrist views showed me what the gripper was doing at the moment of contact, and the ego view could not. The wrist cam sees the contact, and the ego cam mostly sees the scene around it.

That makes me wonder whether the biggest opportunity for improving robots is a touch-native foundation model. If vision close to the contact is what matters most, then the signal the robot lacks is touch itself.

]]>Can Astra Zero-Shot Dexterous Manipulation?2026-10-24T01:08:40+00:002026-10-24T01:08:40+00:00https://leo01110111.github.io/log/210840I gave GPT 6 Astra a very hard dexterous hand task, Dexverse’s long horizon coffee making benchmark. The robot had to:

  1. Grab a mug from a stand.
  2. Pour a carton of milk into the mug
  3. Put the mug under the coffee machine
  4. Spin the dial of the coffee machine to turn it on.

I attached how Astra did after spending \$92 worth of tokens, and an example of what successful completion looks like. (I didn’t actually spend \$92 on this. I just went through 6 resets on the plus subscription)

The lesson I got from this is that Astra is unable to zero shot dexterous manipulation tasks. A harness must be made.

Astra’s attempt:

Successful completion:

The methodology of the model is a state machine approach where each stage would contain an IK target that the arm and fingers would go towards. Grasping finger positions were tuned by searching over values.

The original prompt:

I’m investigating whether Astra is a viable source of generating robot training data. This is the experiment I want to run: without using the provided training data, solve a long horizon dexverse task. Log the amount of time it took for you to solve it (wall clock time) and log the number of tokens used to solve each task. Feel free to use subagents. My hypothesis is that you will take 30 minutes and a whole session’s worth of tokens on the plus subscription. My guess for cost would be 1 usd for one task. I don’t think the solution you come up with will be robust to augmentations. (You should prove me wrong here). It went without saying that I think you will solve it. Also make your solution look like a human did it. Don’t give me weird IK waypoint solutions that move the hand in weird way. Use the UR10e with WUJI hands. Log this hypothesis in an experiment markdown for me to track my belief state as we do more experiments. Create any skills that are helpful in informing future agents in solving robot benchmarks like the one you’re doing now

]]>1233422026-10-18T19:33:42+00:002026-10-18T19:33:42+00:00https://leo01110111.github.io/log/123342If we train a transformer to predict the next token of dolphin spectrograms, could build a machine that speaks dolphin? Can an LLM co-trained with human language translate between dolphin and human?

]]>0310372026-10-18T10:10:42+00:002026-10-18T10:10:42+00:00https://leo01110111.github.io/log/031037Learned about MCAP and protobuf. MCAP strings together protobuf messages which store data like videos and pose frames. Foxglove is a pretty cool tool to visualize MCAPs, which can be seen in the video–had to copy/paste code from claude to get the hand frames remapped into a different format to get it visualized. MCAP is used by Open100k for their egocentric data. I wonder why they decided to go with MCAP and not some other kind of organization. MCAP is also used by ROS2 systems to store data.

]]>0252282026-10-18T09:52:28+00:002026-10-18T09:52:28+00:00https://leo01110111.github.io/log/025228Started an engineering log. Small progress, small takes, small ideas — tweet-sized, posted from the terminal with bin/log.

]]>