⌘ Tech
Sundae Robotics 11: Beyond the Gripper — Teaching Robots Dexterity Through Touch, Vision, and Play

- When
- Sunday, November 22 · 2:00 PM – 5:00 PM
- Where
- San Francisco
- Listed by
- Lu.ma — f.inc
🤖🍨 Grab a sundae and join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, and builders working at the frontier of physical intelligence.
Sundae Robotics 12Dexterous Manipulation, Tactile Perception & Robot LearningFeatured Talk: Beyond the Gripper — Teaching Robots Dexterity Through Touch, Vision, and Play
Beyond the Gripper: Teaching Robots Dexterity Through Touch, Vision, and Play
Keynote: Akash SharmaResearch Scientist, Amazon Frontier AI & Robotics · CMU Robotics Institute PhD
Robots that impress in demos still mostly use parallel-jaw grippers, and true in-hand dexterity remains out of reach. The two standard routes each face fundamental limitations: demonstrations often force humans to act like robots, while simulation cannot faithfully model touch or real-world vision.
In this talk, Akash will present four ingredients that aim to close those gaps.
First, he will introduce PTLD, which learns tactile policies without ever simulating a tactile sensor by distilling a privileged simulation-trained teacher on real hardware.
He will then discuss INVAR, which learns visual invariance once from clean and randomized simulation pairs, allowing policies trained on clean simulation to transfer zero-shot.
Next, Akash will present CARD, which retargets human video, glove, and motion-capture demonstrations according to what the hand touches rather than simply how the hand moves.
Finally, he will discuss DSP, which turns minutes of human play into goal-conditioned dexterity through autonomous play, world models, and distillation.
Together, these works point toward a common recipe for scaling dexterous manipulation: let humans be humans, use a privileged teacher, scale with the robot's own experience, and distill what is learned into a deployable policy.
Akash Sharma is a Research Scientist at Amazon Frontier AI & Robotics (FAR), where he works with Jitendra Malik on robot hands and dexterous manipulation.
He received his PhD from the Carnegie Mellon University Robotics Institute, where he was advised by Prof. Michael Kaess. During his PhD, he was also a visiting researcher with the FAIR Robotics team at Meta, where he worked closely with Mustafa Mukadam and Francois Hogan.
Akash's research focuses on robot perception using varied sensing modalities, particularly vision and touch, to enable dexterous manipulation. He is also broadly interested in state estimation and reconstruction in dynamic environments.
His recent work explores how tactile sensing, simulation, human demonstrations, and large-scale robot experience can be combined to overcome limitations of conventional dexterous learning pipelines. His work on PTLD develops a sim-to-real approach for tactile dexterity that avoids tactile simulation by using privileged real-world sensing to distill information into deployable tactile policies.
Akash has also worked extensively on general-purpose tactile representations. Sparsh, Sparsh-skin, and Sparsh-X explore self-supervised representations across vision-based tactile sensors, distributed tactile skin, and multimodal touch, while DexterityGen investigates combining reinforcement-learned dexterous primitives with human guidance for complex manipulation.
His broader research spans multi-fingered manipulation, human-to-robot skill transfer, tactile perception, sim-to-real learning, object-centric manipulation, state estimation, and scalable data generation for robot learning.
Before his PhD, Akash received a Master's in Robotics from Carnegie Mellon University, where he worked on object-based SLAM for robot navigation. He received his bachelor's degree in Electronics and Communication Engineering from Sri Jayachamarajendra College of Engineering.
Topics
• Moving beyond parallel-jaw grippers toward true in-hand dexterity
• Vision and touch for multi-fingered robot manipulation
• Why conventional human demonstrations struggle to scale to dexterous hands
• Why tacti…

