KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association
INSTITUTE FOR ANTHROPOMATICS AND ROBOTICS - INTELLIGENT PROCESS CONTROL AND ROBOTICS, LAB (IAR-IPR) KARLSRUHE INSTITUTE OF TECHNOLOGY
www.kit.edu
IROS 2015, Hamburg
Multi-Modal Robot Skins: Proximity Servoing and its Applications
Stefan Escaida Navarro and Björn Hein
Workshop “See and Touch: 1st Workshop on multimodal sensor-based robot control for HRI and soft manipulation” at IROS 2015
Karlsruhe Institute of Technology 2
Outline
Introduction State of the Art Proximity Servoing
Preshaping and Grasping
Haptic Exploration
Teleoperation
(Collision Prediction and Detection)
Contour Following/Collision Avoidance
Conclusions and Outlook
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 3 Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Introduction
Karlsruhe Institute of Technology 4
What is Proximity Sensing?
What is the role of proximity sensing in the perception taxonomy? It shares treats with haptics: Perception is “local” and can only be interpreted together with proprioception, i.e. with use of the kinematic chain A spatial resolution is implemented in the skin of the robot (on the robot’s exterior and grippers) As example from nature, the human skin is also locally sensitive to heat radiation
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 5
Why Proximity Sensing?
2D and 3D camera systems are inadequate for the detection and modeling of events in the near proximity of the robot
Perception of camera systems is encumbered by occlusions, extraneous light influence and shadows → problematic for safety Tactile perception can only detect the presence of obstacles when the contact has already occurred, aka “too late”
Every touch event has a prior proximity event Good complement All in all: important for safety because it closes a still existing perception gap
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 6
Capacitive Proximity Sensing with Spatial Resolution
Advantages: Insensitivity regarding lighting and visibility conditions Insensitivity regarding optical surface properties Wide area around the module is covered (sensing is not oriented)
Disadvantages:
(Sensitivity to object material properties, geometry and size) Can be used to identify object?
Signal to distance non-linearity Signal processing overhead due to demodulation ( Latency)
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 7 Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
State of the Art
Karlsruhe Institute of Technology 8
State of the Art (excerpt)
Cheung and Lumelsky 1992 [1]
Wistort et al. 2008 [2]
Preshaping with adjustment of hand pose and fingers (Hsiao et al., IR, and Wistort et al., capacitive) Collision Avoidance in C-Space (Lumelsky and Cheung, IR) Low latency/high speed preshaping and collision avoidance by Shimojo Labs (IR) Not many of the approaches also consider the tactile modality or all possible DoFs
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Hsiao et al., 2009 [3]
Shimojo Labs, 2011 [4]
Shimojo Labs, 2013 [5]
Karlsruhe Institute of Technology 9
Self-Developed Sensor Modules (2012-2015)
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Large Area Module: • 4x4cm • 1 proximity cell • 1 tactile cell • Detection range: ~10cm • Up to 16 modules on
one channel
Sensor array: • 3x16 = 48 sensors • ~25 FPS • ~100ms latency
The arrangement of sensors as a regular grid delivers proximity and tactile images
Karlsruhe Institute of Technology 10 Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Proximity Servoing
Karlsruhe Institute of Technology 11
Proximity Servoing: Closed-loop position or velocity control using proximity sensors
Most common control goal: achieve a configuration where sensor values are equilibrated Use methods from CV to extract features from proximity images
Proximity Servoing is a base for many applications: Preshaping and Grasping, Haptic Exploration, Obstacle Avoidance, etc.
Proximity Servoing
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 12
Safe Zone
Proximity Servoing – Challenges
„Rear lights in the fog“-problem (©Franz Heger) Reduced max. velocity due small distances and non-negligible reaction times (latency)
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
• No signal/ no feedback for control
• Target is too far away
• Signal is weak/noisy • Signal is too close to
the detection threshold
• Signal is ok/good • Distance to target is
safe
• Signal is very good • Distance to target is
not safe
Detection Threshold Distance to Target
Safe Zone
Collision
Some latency
Negligible latency
Karlsruhe Institute of Technology 13 Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Applications
Karlsruhe Institute of Technology 14
Preshaping [8]
Challenge when grasping with tactile-only sensors non-negligible contact force has to be established for perception
Preshaping means to find a suitable configuration for an gripper and its fingers -before touch is established-, that favors robustness when grasping Preshaping is proximity servoing
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
An industrial gripper equipped with 2x2 CPTS in its jaws
Karlsruhe Institute of Technology 15
6D Preshaping using CTPS in Gripper
Calibration of the sensors of the two jaws to equal output under same conditions High resolution/low noise is given at small distances (<1mm)
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 16
6D Preshaping using CTPS in Gripper
Grouping of measured sensor values of modules for gradient detection: Implementation of closed-loop control “Gradient Descent” results in preshaping Comparison of lowest distance/highest proximity values of the groups
Grouping for translational and rotational adjustments:
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 17
6D Preshaping using CPTS in Gripper
Simultaneous adjustment in all 6 DoF Continuous adaption while closing the gripper
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 18
6D Preshaping Evaluation with different materials
Alignment always successful Higher deviation with wood and plastic compared to aluminum Good repeat accuracy Low derivation at end of phase two
Wood: Front Back
Alum.: Front Back
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 19
Proximity + Haptic Exploration
Haptic Exploration deals with acquiring object and world information through a systematic exploration strategy using touch Combined and complementary tactile- and proximity-based approach
Based on proximity servoing/preshaping
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 20
Telemanipulation [9]
Display proximity values acquired inside the gripper as forces on a haptic input device to assist the user in telemanipulation Additionally, assist the user with partial autonomy
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 21
Implementation using a cascade-controller Inner loop is a force-position controller
The user opposes full resistance, i.e. maintains effector at 0 pose Full force display and no robot movement User opposes no resistance Effector pose corresponds to proximity gradient and robot moves accordingly
Outer loop is velocity controller like before
Telemanipulation – Cascade Controller
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 22
Telemanipulation – Video
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 23
Telemanipulation -- Results
Workload and intuitiveness was assessed in a user study against a baseline of “perfect” vision
Workload reduction is possible when operating with partial autonomy Complementary proximity information considered to be useful Small improvement in performance Details: Talk for the paper
Novel approach: Manipulation of moving objects
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 24
Contact-Prediction (Depth Tracking) [8]
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Contact time given by tactile modality
Karlsruhe Institute of Technology 25
6D Contour Following/Obstacle Avoidance
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Problem statement:
Starting from pose S, follow a path to goal G parallel to an obstacle’s surface first detected at pose D, as long as the obstacle is in the way to the goal. Otherwise, choose a direct path (from S to D and E to G)
An end-effector equipped with an array of proximity sensors is used to demonstrate contour following/obstacle avoidance
Karlsruhe Institute of Technology 26
Closed-Loop Control and Curvature Prediction
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Reactive control approach: Track obstacle tangent plane P over time to plan next increment Get misalignment from gradients in proximity image and use them as feedback for distance and orientation controllers
Predictive control approach:
Track curvature of obstacle (1𝑟𝑟, vertical
and horizontal) over time Use curvature to take a shortcut 𝑇𝑇 that is the chord of the corresponding circle If necessary, pre-align with rotation 𝜑𝜑
1𝑟𝑟 vertical 1
𝑟𝑟 horizontal
Karlsruhe Institute of Technology 27
Preliminary Results
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Curvature prediction should deliver speedup when alignment constraints are tight Correction path corresponds to object shape Implementation of HRI should be straight forward
Karlsruhe Institute of Technology 28 Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Conclusions and Outlook
Karlsruhe Institute of Technology 29
Conclusions
It is purposeful to treat proximity sensing similar to tactile sensing Spatial resolution Image Processing It shares treats with “traditional” haptics
Proximity Servoing has applications in many scenarios Often tactile modality can be used complementarily (Grasping, Haptic Exploration)
Many measurement principles exist with advantages and disadvantages It still is unclear whether there is a “best practice” solution
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 30
Outlook, future work
Reduce latency (“Rear lights in the fog”-problem) Grasping smaller skin patches (almost there, work in progress) Big sensor networks challenge for the future
Sensor module design (work in progress)
Modularity: signal processing in each module Increase spatial resolution
If possible, without loosing sensing range Add features to the sensors
Receive mode for better detection of insulating materials, etc. Shear-force measurements (tactile mode)
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 31
Thank your for your kind attention! Questions?
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 32
References [1] E. Cheung and V. Lumelsky, “A sensitive skin system for motion control of robot arm manipulators,” Robotics and Autonomous Systems, vol. 10, no. 1, pp. 9 – 32, 1992. [2] R. Wistort and J. R. Smith, “Electric field servoing for robotic manipulation,” in Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ International Conference on, sept. 2008, pp. 494 –499. [3] K. Hsiao, P. Nangeroni, M. Huber, A. Saxena, A. Y Ng, “Reactive Grasping Using Optical Proximity Sensors,” in Intelligent Robots and Systems (IROS), 2009 IEEE/RSJ International Conference on, 2009, pp. 2098–2105. [4] K. Terada, Y. Suzuki, H. Hasegawa, S. Sone, A. Ming, M. Ishikawa, and M. Shimojo, “Development of omni-directional and fastresponsive net-structure proximity sensor,” in International Conference on Intelligent Robots and Systems, 2011. [5] Keisuke Koyama, Hiroaki Hasegawa, Yosuke Suzuki, Aiguo Ming, and Makoto Shimojo, “Pre-shaping for Various Objects by the Robot Hand Equipped with Resistor Network Structure Proximity Sensors,” in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, 2013, pp. 4027-4033 [6] D. Göger, M. Blankertz, and H. Wörn, “A Tactile Proximity Sensor,” in IEEE Sensors 2010, 2010, pp. 589 –594. [7] S. Escaida Navarro, M. Marufo, Y. Ding, S. Puls, D. Göger, B. Hein, and H. Wörn, “Methods for Safe Human-Robot-Interaction Using Capacitive Tactile Proximity Sensors,” in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, 2013, pp. 1149–1154. [8] S. Escaida Navarro, M. Schonert, B. Hein, and H. Wörn, “6D Proximity Servoing for Preshaping and Haptic Exploration using Capacitive Tactile Proximity Sensors,” in Intelligent Robots and Systems (IROS), 2014 IEEE/RSJ International Conference on, 2014, pp. 7–14. [9] S. Escaida Navarro, F. Heger, F. Putze, T. Beyl, T. Schultz and B. Hein, “Telemanipulation with Force-based Feedback of Proximity Fields,” in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, 2015
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Karlsruhe Institute of Technology 33
Sensor Circuit and Modules in Our Group
Stefan Escaida Navarro -- Multi-Modal Robot Skins: Proximity Servoing and its Applications
Principled sensor circuit design from D. Göger, M. Blankertz, and H. Wörn: “A Tactile Proximity Sensor,” in IEEE Senors 2010
Proximity Mode (self-capacitance)
Tactile Mode (mutual-capacitance)