PowerPoint Presentation
Violin Pedagogy for Finger and Bow Placement using Augmented RealityFrancois de SorbierHiroyuki ShiinoHideo SaitoIntroductionOverview of our systemViolin extraction and 3D registrationVirtual advisingConclusionOutlineViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier2Violin is a beautiful instrument but one of the most complicatedNo fret on the fingerboardNo help for the position the bow on stringsMotivation
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier3IntroductionMusic JacketVibro-tactile feedbackGuide the bowing arm
Guitar playing supportTracking with markerGuide with virtual hand Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier4Previous works
Y. Motokawa, H. Saito. Support system for guitar playing using augmented reality display. In Proceedings of the 5th IEEE and ACM ISMAR, 243-244, 2006van der Linden, J., Schoonderwaldt, E. and Bird, J. Good Vibrations: Guiding Body Movements with Vibrotactile Feedback.Proceedings of the Third International Workshop on Physicality, 13-18, 2009IntroductionOverlay virtual information on the violinVirtual fretsGuides for the bow and fingersSound analysisViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier5Our objectiveIntroduction
No intrusive deviceNo markerReal time feedback
Marker-free violin trackingusing a RGBD cameraViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier6Our constraintsIntroductionTracking of the violinFeedback displayed on the screenConstant view of the violinVirtual informationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier7Overview of our system
Overview of our systemFeatures detection and extractionMany occlusions caused by the playerThe surface has a poor textureThe material is highly specular
Difficult to use features in this context
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier8Marker less tracking
Overview of our systemUse Kinect for tracking the violinDepth values for the pose estimation
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier9Depth-based pose estimation
Overview of our systemViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier10WorkflowOverview of our systemColorDepthSegmentationViolin detectionViolin extractionDatabaseRegistrationVirtual informationdisplayedDetect the brown color in the imageRemove noiseMany parts are missingOcclusionsSpecular materialStrings and fingerboardNot enough for tracking
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier11Color segmentationViolin extraction and 3D registration
Get the 3D points from extracted colorCompute the corresponding plane equationOptimized with RANSACAlign a 3D volume to the planeTypical dimensions of a violinKeep the 3D points in itViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier12Extended segmentation
Violin extraction and 3D registrationIterative Closest Point algorithmCompare input points with a modelSlow if too many pointsInaccurate if not enoughOur proposed solutionIncrease the number of modelsReduce the number of points per modelFast retrieval with a plane equation comparisonViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier133D registration with ICPViolin extraction and 3D registrationOffline phase25 modelsCompare the plane equationsStore candidate if the difference is big enoughStore also the plane equation
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier14Creating the databaseViolin extraction and 3D registration
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier15Result of the trackingViolin extraction and 3D registration
Real time (21 milliseconds)Pose also estimated using markersViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier16Evaluation of the trackingViolin extraction and 3D registrationRx(deg)Ry(deg)Rz(deg)T(mm)Minimum error0.120.250.200.22Maximum error13.298.277.8932.1Average error3.072.692.787.20Use the pose estimationLocation manually defined during the capture of the models Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier17Display the fretsVirtual advising
Emphasize the string to be pressedDisplay a red dot at the junction of the string and the fretDefine where the finger has to press the stringViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier18Fingering advicesVirtual advisingSuggest the violinist to play a given noteDisplay the fret/string to be pressedAnalyze the sound obtainedIf fingering is considered correctAdvice about the position of the bow given the difference of pitch (OK / LOW / HIGH)Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier19Other advicesVirtual advisingViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier20ResultsVirtual advising
Performed by confirmed playerBowing is correctCompute the difference of pitch on each fret with the expected oneViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier21Evaluation of the resultsFret number123456789averageDifference of pitch11.114.112.012.413.415.812.813.919.213.8Virtual advisingOriginal marker-free method for virtual advising on a violinMethod based on several pre-computed modelsReal-timeAccurateDisplay virtual guides on the fingerboardAnalyze the note played for further advicesViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier22ConclusionsVirtual advising22Perform a user based analysisValidate or improve our approachUse different kind of displaySee-through HMDSpatial augmented realityApply to other similar instrumentsJapanese shamisenViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier23Future works
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier24QuestionsThank you for your attention