Machine vision based situation

classification in micromanipulation

Seminar presentation

6.11.2015

Janne Venäläinen

Contents

• Micromanipulation

– overview

– motivation

– challenges in automatization

• Situation identification

– for more autonomous microrobots

MicromanipulationMicro-objects

• Manipulation of small

objects (< 1mm)

Example objects

Micro-opto-electro-mechanical system (MOEMS)

Figure from (Gauthier & Régnier, 2011)

Living cells

Figure from (Wilson, 1900)

Example applications

Digital light processing device

Figure by David Pape, 2007Biological research, e.g. cancer research

Figure from (Ammi & Ferreira, 2006)

MicromanipulationManipulators

MEMS-micromanipulator

Figure from (Sariola et al., 2008)

Biological micromanipulator

Figure from (Lu et al., 2007)

Micromanipulator in SEM

Figure from (Marturi et al., 2013)

Automatic micromanipulationMotivation

Increasing market

of MEMS products:

2015: $40 billion →

2025: $200 billion(Pryputniewicz, 2012)

Requirement for complex

MEMS with incompatible

batch manufacturing

processes

Need for automatic

micromanipulation

Increasing interest

in single-cell reseach

Low yield with manual work:

• Fragice samples

• Hand-tremor

• Liquid flow

• Long training phase

Need for automatic

biomicromanipulation

Challenges in micromanipulation• Unpredictable

– Sticking problem

– Physical models are difficult to apply (Gauthier & Régnier, 2011)

• Difficult to retrieve parameters for a specific application

• Valid only in equilibrium

Scaling effect

Figure from (Arai et al., 1995)

200 µm 100 µm

200 µm

Feedback control is required!

Open loop control not enough,

like it might be in macroscale robots

Challenges in micromanipulation

• Visual feedback for manual / automatic control– Light microscope

– Scanning electron microscope (SEM)

• Better resolution, worse– depth of field (DOF)– working distance

• Especially in bio-micromanipulation

• 100x → few hundred µm

• Short working distance+

Large imaging equipment=

Limited space for other sensors etc.

• SEM– Better resolution & DOF

– Sample preparation MEMS-micromanipulator

Figure from (Sariola et al., 2008)Light microscope vs. SEM

Figure from (Goldstein et al., 2008)

Previous work in automatic

micromanipulation• Limited to

review by (Banerjee & Gupta, 2013)

– visual servoing

– path planning

– using low-level data (e.g.coordinate values of objects and tools)

• However, understanding situations is required for more autonomous microrobots.

• No situation identification in micromanipulation– Automatic image

annotation (Zhang et al. 2012)

– Automatic video labelling• Based on human action

recognition (Ronald Poppe, 2010)

?

Sticking problem occurred!

Pick object again.

Reposition object.

Reposition object

without picking.

Vibrate to release.

Situation identification &

Previous work cont.• Detection methods in

micromanipulation include

– Template matching (Ammi & Ferreira, 2006); (Lu et al., 2007); (Oyang et al., 2007)

– Hough transform(Oussalah et al., 2005); (Tanaka et al., 2008); (Pawashe and Sitti, 2006)

– Depth from focus(Tamadazte et al., 2011); (Sjövall et al. 2006); (Wang & Cho, 2008)

– Image moments(Sjövall et al., 2006); (Wang & Cho, 2008)

– Watershed(Inoue et al., 2005);(Marturi et al., 2013)

– Active contour model(Ammi & Ferreira, 2006);(Inoue et al., 2005)

– Depends on application

– Evaluate and select

Situation identification flow

• Assess supervised ML classification methods

• Depends on application

• Select the one with best performance

– Naive bayes

– Tree learning

– SVM

– Ensemble learning

– Neural networks

Summary

• Challenges in micromanipulation

– unpredictable

– challenges in obtaining feedback

• Solution:

– abstraction of low-level data: situations

→ visual feedback

→ image processing

→ feature engineering

→ classification

References 1 / 2

• Mehdi Ammi and Antoine Ferreira. Biological cell injection visual and haptic interface. Advanced Robotics, 20(3):283–304, 2006.

• Arai, Fumihito, et al. "Micro manipulation based on micro physics-strategy based on attractive force reduction and stress measurement." Intelligent Robots and Systems 95.'Human Robot Interaction and Cooperative Robots', Proceedings. 1995 IEEE/RSJ International Conference on. Vol. 2. IEEE, 1995.

• Ashis Gopal Banerjee and Suneet K Gupta. Research in automated planning and control for micromanipulation. Automation Science and Engineering, IEEE Transactions on, 10(3):485–495, 2013.

• Michaël Gauthier and Stéphane Régnier. Robotic micro-assembly. John Wiley & Sons, 2011.

• Joseph Goldstein, Dale E Newbury, Patrick Echlin, David C Joy, Alton D Romig Jr, Charles E Lyman, Charles Fiori, and Eric Lifshin. Scanning electron microscopy and X-ray microanalysis: a text for biologists, materials scientists, and geologists. Springer Science & Business Media, 2012.

• Kenji Inoue, Tatsuo Arai, Tamio Tanikawa, and Kohtaro Ohba. Dexterous micromanipulation supporting cell and tissue engineering. In Micro-NanoMechatronics and Human Science, 2005 IEEE International Symposium On, pages 197–202. IEEE, 2005.

• Lu, Zhe, et al. "A micromanipulation system with dynamic force-feedback for automatic batch microinjection." Journal of micromechanics and microengineering 17.2 (2007): 314.

• Marturi, Naresh, Sounkalo Dembélé, and Nadine Piat. "Depth and Shape estimation from focus in scanning electron microscope for micromanipulation." Control, Automation, Robotics and Embedded Systems (CARE), 2013 International Conference on. IEEE, 2013.

• Mourad Oussalah, BP Amavasai, F Caparrelli, A Selvan, M Boissenin, JR Travis, and S Meikle. Machine vision methods for autonomous micro-robotic systems. Kybernetes, 34(9/10):1421–1439, 2005.

• PR Ouyang, WJ Zhang, Madan M Gupta, and W Zhao. Overview of the development of a visual based automated bio-micromanipulation system. Mechatronics, 17(10):578–588, 2007.

• Chytra Pawashe and Metin Sitti. Two-dimensional vision-based autonomous microparticle manipulation using a nanoprobe. Journal of Micromechatronics, 3(3):285–306, 2006.

References 2 / 2

• Ronald Poppe. A survey on vision-based human action recognition. Image and vision computing, 28(6):976–990, 2010.

• Ryszard J Pryputniewicz. Current trends and future directions in mems. Experimental mechanics, 52(3):289–303, 2012.

• Sariola, Veikko, et al. "Experimental study on droplet based hybrid microhandling using high speed camera." Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ International Conference on. IEEE, 2008.

• Sami Sjövall, Jukka Laitinen, Petteri Korhonen, Veikko Sariola, and Quan Zhou. Positioning and quality inspection of micro components using machine vision and 6dof micro handling system. In 6th International Conference on Machine Automation. IMCA, 2006.

• Brahim Tamadazte, Nadine Le-Fort Piat, and Sounkalo Dembélé. Robotic micromanipulation and microassembly using monoview and multiscale visual servoing. Mechatronics, IEEE/ASME Transactions on, 16(2):277–287, 2011.

• Yoshio Tanaka, Hiroyuki Kawada, Ken Hirano, Mitsuru Ishikawa, and Hiroyuki Kitajima. Automated manipulation of non-spherical micro-objects using optical tweezers combined with image processing techniques. Optics express, 16(19):15115–15122, 2008.

• Junping Wang and Hyungsuck Cho. Micropeg and hole alignment using image moments based visual servoing method. Industrial Electronics, IEEE Transactions on, 55(3):1286–1294, 2008.

• Wilson, Edmund Beecher. The cell in development and inheritance. Macmillan, 1900.

• Dengsheng Zhang, Md Monirul Islam, and Guojun Lu. A review on automatic image annotation techniques. Pattern Recognition, 45(1):346–362, 2012.