Post on 06-Jan-2022
transcript
National Institute of Advanced Industrial Science and Technology
MR Fitting Simulation and Clothes Modeling for Wheelchair Users
○Ryosuke Ichikari, Masaki Onishi, and Takeshi Kurata AIST, Japan
2014/4/18-20 KJMR2014@Jeonju (KIST Jeonbuk Branch), Korea
National Institute of Advanced Industrial Science and Technology
Backgrounds • Demands for improving quality of life (QoL) of
people with disabilities • Wheelchair users’ problems for purchasing clothes
– Difficulties of trial fitting – Lack of items specially designed for the users
Clothes fabrication is normally based on standing position
⇒Research development for supporting clothes fabrication for wheelchair users is required
2 Fashion show showing clothes for the wheelchair users
Images from http://www.rehab.go.jp/rehanews/japanese/webnews/201302/news_201302_6.html
National Institute of Advanced Industrial Science and Technology
Research Project System
3
Project members consist of AIST, National Rehabilitation Center for Persons with Disabilities, and Bunka Fashion College (all in Japan)
Non-contactMeasurement by RGB-D Camera
Modeling/Capturing Clothes
MR Fitting Simulation
Contact-typeMeasurement
Pattern Makingby Apparel CAD
Design (Photo) ofthe Dedicated DummyMeasurement
of the Wheel-chair User
Virtual FittingActual ClothesFabrication
Visualizingthe measuredresults Utilizing data from
fitting results (size, personal preference, etc.)
Utilizing the measured data for the clothes fabrication
Feedback of data obtained duringclothes fabrication
National Institute of Advanced Industrial Science and Technology
MR(Mixed Reality) Fitting Simulation
4
• Needs for MR fitting simulation from the field – Virtual fitting without physically-taxing operations – Visualizing the fitting result from various viewpoints – Simple configuration at a low price
• Computer-generated clothes are superimposed onto the users body by MR technology.
• Camera position is not restricted by the real positing, it can be changed virtually in the 3D-CG space.
National Institute of Advanced Industrial Science and Technology
Related Works • MR-based virtual fitting systems in the industry
– EON Reality Eon Interective Mirror Visualizing virtual fitting results with simple cloth simulation in real time on a display which represents mirror
– DNP Fitting system demonstrated in UNIQLO. This shows color variations of the clothes by changing color of the clothes in the display.
• Researches related to virtual fitting or clothes fabrication – Stefan et.al. (ISMAR2011):
Fitting simulation based pre-captured clothes with various poses – Igarashi et.al.(UIST2002):
Interactive simulation of sewing the parts of clothes and fitting on CG dolls
5
No work have focused on the fitting simulation with sitting positions
National Institute of Advanced Industrial Science and Technology
System Requirements and Prototype To be beneficial for both users and system owners Prototype Measuring users’ body for actual fabrications ✔ Trial fitting without forcing users to get off from the wheelchair ✔ Improve the quality of the experience of trial fitting ✔
6
Requirements for system configurations Simple configuration at a moderate price ✔
Adopting contact-less measurement for the users ✔
Requirements for MR functions Allowing to check their appearance from various view-points ✔ Composition of the clothes without visual paradox ✔ Accurate registration between CG clothes and real body ✔ Tracking the user motion and deformation of the clothes On going Practical method for preparing clothes models On going
National Institute of Advanced Industrial Science and Technology
MR System Based on 3D Reconstruction of Objects in the Real World
• Visualization of output from the non- contact measuring method
• Inputting the sequence from a fixed RGB-D camera – Library:OpenNI(RGB-D camera)
OpenGL(Rendering CG space) • MR composition in the 3D-CG space where
real objects in the world are reconstructed in – Enabling virtual camera movement
• CG Clothes are prepared by capturing real clothes and superimposed onto the image of the user. – Utilizing multi-path rendering for rigorous compositions without
visual paradox
• . 7
RGB-D camera: Asus Xtion
National Institute of Advanced Industrial Science and Technology
How to Render the MR Space • Real-Objects: Importing RGB-D sequence and rendered as
point cloud in the 3D-CG space (with OpenNI, OpenGL) • Clothes: Rendering mesh of the clothes generated by RGB-D
sequence in the same space file format of meshes: Collada(.dae)
8
CG model of the clothes Rendered RGB-D
sequence
Composition result
National Institute of Advanced Industrial Science and Technology
Rigorous Composition without Visual Paradox • Geometric consistency is important
for MR-based visualization e.g. ) If inner clothes under superimposed outer clothes is visible, it may cause visual paradox.
• Rendering Method – Keeping the silhouette of the clothes – If the silhouette of the body is unnaturally protruded from the
clothes, these area should be deleted – Necessary parts (face, hand, foot) are preserved in the “clean-
up” process – The deleted area and occluded area of RGB-D camera are filled
with the pre-captured background image or inpainting method ⇒Composition process is implemented using multipath-rendering
9
Example of bad case
National Institute of Advanced Industrial Science and Technology
Fulfilling the holes or the deleted areas • The holes and the deleted areas are fulfilled with
pre-captured background images or inpainting method*
10 *A. Telea, "An image inpainting technique based on the fast marching method," J. Graphics Tools, 2004
Input RGB-D image (rendered)
Background RGB-D Image(rendered)
After composition
National Institute of Advanced Industrial Science and Technology
Topics about Preparing Clothes Model and Deformation
11
• How to prepare CG model of clothes – Simulation based: Using output of 3D sewing simulation from
apparel CAD
– Capturing based:Scanning actual clothes by 3D scanning.
• Deformation of the clothes model according to the shape of body and movement of joint angles
• Consideration of expansion and contraction(stretching) of the material
• Measuring the shape and pose of the users and their tracking.
• How to utilize the result of the simulation for the actual fabrication – Compatibility with the apparel CAD
National Institute of Advanced Industrial Science and Technology
Pattern→3D clothes model
(Sewing Simulation)
Conceptual-Flow
Clothes model→Fitting results
(Fitting Simulation)
Pattern data
Torso data
3D clothes model
Shape data
Sitting Pose data Fitting result
Rigorous Composition
of the CG clothes
MR composition result
Contact-type measurement
Contact-less body capture
Real scene capture
Prototype
3D scanning of real clothes by RGB-D camera
Mesh Deformation of scanned mesh by
skeleton and measured data
National Institute of Advanced Industrial Science and Technology
3D Scanning with RGB-D Camera • ReconstructMe: A 3D scanning software based on the
SLAM with RGB-D camera – Outputs: Triangle mesh with vertex color
13
National Institute of Advanced Industrial Science and Technology
Pipeline of the Sawing Emulation • Preparing the CG model of the clothes
– Adding pattern information
• Captured mesh is separated into parts which imitates pattern of the clothes – No UV coordinate and texture at this time
• Example of the operating pipeline with CG software (Autodesk Maya)
– Meth separation – UV space definition by planner/cylindrical projection (Mesh Lab.) – Baking(Projecting) vertex color onto texture
14
National Institute of Advanced Industrial Science and Technology
Mesh Separation and UV Space Definition • Imitating pattern of the real clothes by UV space
definition – (e.g.) Separating into four parts (front, back, left, right) – Capability for accessing/modifying each parts
separately
15
National Institute of Advanced Industrial Science and Technology
Approximate Fitting Simulation • Deformation to deal with variation of the shape of body
– Shape deformation: Changing vertex position along normal direction
– Estimating size of real body and clothes for the deformation by ellipse approximation
• Deformation to deal with variation of the joint angles – Rigorous clothes simulation is not employed in the prototype – Employing skeleton based mesh deformation
• Pros: Calculate deformation of the clothes in real-time • Cons:
– Quality of the result depends on the mesh weight – Not physically aculeate (gravity, volume preservation, stretch
property, etc.)
16
National Institute of Advanced Industrial Science and Technology
Example of the Shape Deformation • Shape deformation is done by moving vertex position
along the normal while keeping the UV coordinate
17
Normal of a Mesh
Mask for the deformation (target: waist)
National Institute of Advanced Industrial Science and Technology
Example of Skinning and Deformation of Captured Mesh in Maya
• Definition of joints (skeleton) • Definition of mesh weight • Skeleton based mesh deformation
18
National Institute of Advanced Industrial Science and Technology
Demo
19
National Institute of Advanced Industrial Science and Technology
Conclusions
20
• Proposed MR fitting simulation system which aims for supporting wheelchair users to purchase clothes
• Utilized RGB-D camera for changing virtual viewpoints in the MR space
• Rigorous rendering pipeline keeps geometric consistency
• Proposed methods for preparing clothe model and deformation of the model – Clothe model creation using RGB-D camera based
scanning method – Mesh deformation to deal with the variety of the
shapes and the joint angles
National Institute of Advanced Industrial Science and Technology
Future Works
• Integration with contact-type measuring method for measuring occluded parts
• Capturing from back-side of the users • Automatic tracking of the skeleton from RGB-D
sequence • Measuring or representing material stretching • Feasibility study of the service with the proposed
system
21