Sep 2023 – Current

[NYU] Soft Robotics Perception Research

Mechanical Engineering Researcher

Prepared for Publication (Tentative Title):

• Ke Yang, Qinsong Guo, Hanwen Zhao, Chen Feng, "Contact Reconstruction of Soft Robot via Audiovisual Proprioception," IROS 2025.

This research is based on Toward Zero-Shot Sim-to-Real Transfer Learning for Pneumatic Soft Robot 3D Proprioceptive Sensing (link). The project explores audiovisual multimodal perception for accurate geometry and contact force estimation of a soft robotic gripper, especially at large bend angles and visually occluded contact locations. The gripper has an internal camera and a microphone array. 

My contributions:

• Designed and fabricated an exoskeleton system to actuate the soft robotic gripper, utilizing CAD and advanced manufacturing to balance dexterity and payload.

• Collected real-world point cloud, image, and audio data.

• Generated parameterized 3d point clouds of the gripper for training using Numpy. 

• Tested and optimized transformer-based multimodal proprioception algorithms in PyTorch.

By January 2024, the soft robotic fingers have reached the final development phase, with active data collection and model training. The aim is to predict the geometry of the robotic finger using image and audio data from internal sensors, addressing challenges in proprioceptive sensing by minimizing labor-intensive data collection and enhancing hardware transferability. The research is prepared for submission in Fall 2024.