MusicBot 5000: Design and Implementation of a Virtual Reality Interface for Playing...

MusicBot 5000 Design and

Implementation of a Virtual Reality

Interface for Playing Real-World

Instruments

by

Rebecca Moffat

Rachel Peters

Paul Schellenberg

Isaac Wiebe

Benjamin White

Final report submitted in partial satisfaction of the requirement s for the degree of

Bachelor of Science

in

Electrical and Computer Engineering

in the

Faculty of Engineering

of the

University of Manitoba

Faculty Advisors

Dr Ian Jeffrey

Dr Derek Oliver

March 17th 2017

copy Copyright 2017 Rebecca Moffat Rachel Peters Paul Schellenberg Isaac Wiebe and Benjamin White

i

Abstract

This report details the design and implementation of a virtual reality interface for playing

real-world instruments which we fondly refer to as the MusicBot 5000 The system consists of

three main modules the user interface the web server and the instrument robot The user

interface displays playable instruments in non-immersive virtual reality and translates user hand

gestures into HTTP requests which are sent to the web server The instrument robot receives

instructions from the web server and operates peripheral instruments accordingly

The goal of this project was to design and implement a system that allows users to play

instruments remotely with an interface that closely mimics the actual playing of the instruments

The MusicBot 5000 met or exceeded all design specifications set out for this project with

peripheral instruments including a xylophone a drum and a cowbell

ii

Contributions

Rebecca Rachel Paul Benjamin Isaac

Virtual instrument design amp hitbox

setup

Hand gesture recognition

Game controllerVR logic

Virtual menus amp hands

Communication protocol

Metronome

Looping

Amplifier circuit design amp testing

Web server amp web app

Mount design

Microcontroller setup

Perfboard setup amp soldering

Final report

Legend lead contributed

iii

Acknowledgements

We would first like to thank our advisors Dr Ian Jeffrey and Dr Derek Oliver for their

open doors and open minds Thank you to Sandra Komishon for donating the xylophone used in

this project We would also like to thank Cory Smit and Zoran Trajkoski from the machine shop

for making our designs come to life We would also like to express our appreciation to systems

integration specialist Dr Ahmad Byagowi for his technical advice and guidance in this project

Thanks must also be extended to Aidan Topping for her technical writing feedback on multiple

drafts Finally we could not have accomplished this without our colleagues who helped with

testing our system and gave us invaluable feedback

iv

Table of Contents

Abstract i

Contributions ii

Acknowledgements iii

List of Figures vi

List of Tables vii

List of Abbreviations viii

Chapter 1 Introduction 1

11 Performance Metrics 2

Chapter 2 User Interface Module 5

21 Virtual Instruments 5

22 Gesture Recognition 8

221 Detectors 9

222 Finger-Pointing Gesture 9

223 Open Palm Gesture 10

23 Virtual Hands 11

24 Looping 14

25 Metronome 14

27 Communication 15

28 Game Controller 15

29 Summary 16

Chapter 3 Web Server Module 17

31 Server Architecture 17

32 Web Application 18

Chapter 4 Instrument Robot Module 21

41 Actuators 21

411 Solenoids 22

412 ROB-11015 23

42 Amplifier Circuits 23

v

43 Solenoid Mounts 25

44 Mallets 27

45 Microcontroller 28

46 Summary 29

Chapter 5 System Integration 30

51 Client to Web Server Communication 30

52 Webserver to Instrument Robot Communication 30

53 Summary 31

Chapter 6 System Validation 32

61 Objective System Verification 32

611 Systematic Testing 33

62 Subjective Testing 35

Chapter 7 Conclusion 38

References 39

Appendix A Final Budget 40

Appendix B User Testing Documentation 41

Appendix C Source Control 43

vi

List of Figures

Figure 1-1 System block diagram 1

Figure 2-1 Program flow for virtual instrument note playing 6

Figure 2-2 A plan view of the drum collision zones 7

Figure 2-3 A cross section of the collision zones in the virtual drum 8

Figure 2-4 Virtual xylophone in the virtual environment 8

Figure 2-5 Finger-pointing gesture 10

Figure 2-6 Open palm gesture 11

Figure 2-7 Left-hand menu 12

Figure 2-8 Right-hand menu 13

Figure 2-9 Surroundings selection menu 13

Figure 2-10 Program flow for metronome 15

Figure 3-1 Web server flow chart 18

Figure 3-2 Login screen for the web app 19

Figure 3-3 Xylophone interface on the web app 19

Figure 3-4 Drum kit interface on the web 20

Figure 4-1 Diagram of solenoid with no current applied to coil shaft out 22

Figure 4-2 Diagram of solenoid with current applied to coil shaft pulled in 22

Figure 4-3 ROB-11015 solenoid [8] 23

Figure 4-4 MultiSim schematic of circuit with single Darlington pair and solenoid load 24

Figure 4-5 MultiSim simulation oscilloscope output 25

Figure 4-6 Solenoid mount for xylophone (rear view) 26

Figure 4-7 Solenoid mount for drum (front view) 26

Figure 4-8 Cowbell mount 27

Figure 4-9 3D model of a mallet 28

Figure 6-1 Round Trip Time for HTTP Packets 34

Figure 6-2 Gesture Validation Tests 35

Figure 6-3 Test subject previous VR experience 36

Figure 6-4 User feedback on intuitiveness at beginning of session 36

Figure 6-5 User feedback on intuitiveness at end of session 37

vii

List of Tables

Table 1-1 Proposed and Achieved Specifications 3

Table 2-1 Finger-pointing gesture palm direction detector 10

Table 2-2 Finger-pointing finger extension detector 10

Table 2-3 Open palm gesture palm direction detector 11

Table 2-4 Open palm gesture finger extension detector 11

Table 2-5 Virtual hands object priority 14


viii

List of Abbreviations

Abbreviation Description

3D Three dimensional

ASCII American standard code for information interchange

BPM Beats per minute

CAD Canadian Dollars

CSS Cascading style sheet

API Application program interface

DC Direct current

GPIO General purpose inputoutput

HTML Hyper-text markup language

HTTP Hyper-text transfer protocol

I2C Inter-integrated circuit

IC Integrated circuit

IP Internet protocol

LAN Local area network

PC Personal computer

RESTRESTful Representational state transfer

USB Universal serial bus

VR Virtual reality

Wi-Fi IEEE 80211x

1

Chapter 1 Introduction

The goal of this project is to design and implement a system that allows users to play

real-world instruments remotely The objective of this system is to provide users with the same

feeling as playing a real instrument and allow for multiple users to play the same set of

instruments simultaneously To fulfill these requirements a novel creation was designed and

implemented the MusicBot 5000

This report covers the details of the final design of the MusicBot 5000 The bulk of the

report is devoted to the technical aspects including the design of the subsystems and their

integration into a final system However the final integrated system is novel to the point that user

testing of the system was deemed critical to determine the effectiveness of the design The

remainder of this report is dedicated to the results of the testing

The MusicBot 5000 project is broken up into three components divided by the main

functionalities in the system The first subsystem is the user interface where the user can play

virtual representations of instruments Second is the web server that receives commands from the

user over the internet and passes instructions to the instrument robot1 Finally the instrument

robot receives instructions from the webserver and plays the corresponding physical instruments

according to user input A high-level representation of these subsystems and their interactions is

depicted in Figure 1-1

Figure 1-1 System block diagram

The desired functionality of the MusicBot 5000 includes the following features

1 To allow users to play real instruments through virtual reality

2 To allow users to select through virtual reality which instrument they will play

3 To allow for real time play and the feeling of playing real instruments

4 To allow for multiple simultaneous users of the system

1 Instrument robot is not a humanoid robot

2

Since the user is supposed to feel like they are playing the instruments the system is to be

controlled by user hand gestures which are captured by a LeapMotion controller This system

should also provide additional music environment features such as a metronome and the ability

to create and play repeating loops of music

In the user interface the userrsquos gestures are turned from movements in 3D space into

commands for the system These commands allow users to select which instrument they would

like to play choose which environment they would like to play music in record a music loop and

start a metronome

The webserver sits between the user interface and the instrument robot This module

receives HTTP requests from the user interface over the internet and decodes what note is meant

to be played The instrument robot receives instructions from the web server and activates the

appropriate actuator This results in a note being played on the desired instrument

11 Performance Metrics

The performance metrics for this project are based on system features system

intuitiveness and system speed System features include playable instruments as well as looping

and playback features System intuitiveness relates to gesture recognition and how easily users

can navigate the system System speed relates to how fast the system reacts to user gestures and

how fast notes can be played consecutively All performance metrics are shown in Table 1-1

3

Table 1-1 Proposed and Achieved Specifications

Project Feature Proposed Specification

Proposal Notes Achieved Specification

Outcome

Supported Platforms

Windows PC

Windows PC Met

Simultaneous loop limit

1 Loop One instrument loop playing back while user plays a different instrument

1 Loop Met

Easy to play Exit survey responses

Test users will be given a survey to gauge how intuitive the experience was

Initial 3735 Final 4735 (1 being not intuitive 5 being the most intuitive)

Met

Recognized gestures

6 gestures recognized

Xylophone drums record loop pause loop start loop instrument selection

All proposed actions are possible with additional actions recognized

Surpassed

Recognition Accuracy

5 Missed Gesture Rate


Surpassed

Number of notes played simultaneously

2 on each instrument simultaneously

All notes can be played with multiple users

Surpassed

Successive notes The same note can be played twice per second

Maximum actuation speed 125Hz

Surpassed

Latency from VR to analog instrument

250 milliseconds

35 Milliseconds

Surpassed

The system has two prominent features the ability to play multiple musical instruments

and the ability to loop a single track The system was specified to be able to play two different

instruments a xylophone and a snare drum In addition to these instruments a cowbell was

added To assist a user with keeping time a software metronome has also been added This

metronome can keep time from 40 to 208 beats per minute Another feature is looping playing

back a user defined note sequence with exact timing

Hand gestures are the primary way users interact with the system Two gestures were

required to meet the specifications of the MusicBot 5000 an open palm for identifying menus

and a finger-pointingrdquo gesture to load the drum sticks or xylophone mallets in VR The system is

specified to have above a 95 identification rate for both gestures to reduce the learning curve

for the user

4

The remaining performance metrics are related to system speed Since playing an

instrument requires aural feedback a large delay between a userrsquos hand gesture and the system

responding would be a poor experience for the user To resolve this the maximum latency from

the virtual reality to the instrument was specified to be 250ms To ensure songs can be played at

an appropriate speed it was required that a single note can be hit at least twice per second The

system must also have the ability to play at least two notes simultaneously to support blocked

intervals2

Although it is not technically a performance metric this project had to adhere to a strict

budget of 500 CAD The details of the budget for the MusicBot 5000 are outlined in Appendix A

The final design and implementation of the MusicBot 5000 has met or surpassed every

goal that is outlined in Table 1-1 Testing and verification of these goals is covered in Chapter 6

2 The musical concept of playing two or more notes simultaneously

5

Chapter 2 User Interface Module

This chapter provides a description of the design and functionality of the virtual reality

user interface3 for the MusicBot 5000 project

The user interface has three key tasks displaying a playable virtual instrument to the

user detecting which note has been played on the virtual instrument and controlling the looping

module The virtual instruments are depictions of the remote physical instruments within the

virtual environment The virtual environment detects when a note is played and sends a message

to the instrument robot so that the corresponding real note is played The VR communication

system sends and receives messages from the MusicBot instruments as outlined in Section 27

These messages are sent over the internet using HTTP

As laid out in the goals of this project the experience of playing virtual instruments

should closely mirror the experience of playing real instruments Using a standard keyboard or

controller does not provide this experience and so a LeapMotion controller was chosen to

provide user input A LeapMotion controller is a device that tracks the position orientations and

configuration of a users hands and fingers in 3D space This allows the user to interact naturally

with the virtual environment

The virtual environment was programmed in the Unity game engine Unity is a full-

featured game engine that is free for non-commercial use This means that rather than

programming the physics data management and graphics that are a part of every virtual

environment the development can focus on solving problems that are unique to the MusicBot

5000 Unity was chosen for three reasons firstly it is free and so it would have no impact on the

budget Secondly one of the group members had previous experience with Unity which allowed

development to begin without delay Finally the LeapMotion team has made assets available in

Unity that allow developers to integrate the LeapMotion technology with Unity software These

assets include 3D models of hands that respond to the input from the LeapMotion controller This

means that the LeapMotion controller can be used without the additional development work of

creating tools to integrate the it with Unity

21 Virtual Instruments

The virtual instruments are virtual objects that the user will interact with to play real

notes on the physical instruments These instruments detect collisions with the virtual mallets

wielded by the user and create messages that are sent to the physical instruments which play the

3 The user interface is not ldquoimmersiverdquo VR as the user does not wear a head-mounted display such as the

Oculus Rift Nonetheless this environment is still virtual and so still qualifies as virtual reality

6

desired note To play a note a user first selects an instrument from the instrument selection menu

Once their chosen instrument is selected it appears in front of them The userrsquos hands are tracked

in real-time and they can now play a note on the instrument by striking one of the virtual keys

with their virtual mallet Once a key has been struck the virtual environment sends a message to

the web server module discussed in Chapter 3 where the message is processed so that the

instrument robot can play the appropriate note

The first step in playing a physical note is detecting a collision In Unity developers are

provided with callback functions for three types of collision events the beginning of contact

between two objects the end of contact and two objects staying in contact Once the beginning of

a collision has been detected by the Unity engine a message is sent to the communications

module that is then forwarded to the instrument robot To detect different notes each ldquoplayablerdquo

area has its own collision mesh A collision mesh is a 3D model in the virtual environment that

can be attached to in-game objects In every frame the Unity engine checks through all objects

with collision meshes to determine if any other objects with collision meshes have taken part in

one of the aforementioned collision events Figure 2-1 shows how the note controller registers

collision events and plays a note

Figure 2-1 Program flow for virtual instrument note playing

Once a collision with a virtual mallet has been detected the virtual instrument signals the

communication module of the game controller to send a message to the instrument robot This

message contains two parts an instrument code and a note name For example the instrument

code for the xylophone is lsquoxrsquo and the note names on the xylophone are simply the name of the

7

note in English notation The code to play an E in the second octave is lsquoxE2rsquo while for an E in

the third octave the code is lsquoxE3rsquo4

There are two code components to each instrument an instrument controller and many

instances of a note controller The instrument controller acts as a parent for the note controller

receiving note names when a collision is detected and then prepending the instrument code

On the implementation of the xylophone each key on the virtual xylophone has a

collision mesh and a note controller This allows every key to be uniquely identified and generate

its own message The collision meshes initially mirrored the physical geometry of each key

however it was found that this led to adjacent notes playing if key strikes were not perfectly

precise To fix this each collision mesh was made thinner This still allows notes to be played as

expected and remedies the double note bug

The implementation of the virtual snare drum is more complicated than the virtual

xylophone This is due to the nature of a physical drum where different areas on a drumhead

produce different sounds when hit To accomplish this two collision meshes are present on the

drum with the smaller central hit zone slightly raised This is illustrated in Figure 2-2 and Figure

2-3 When a drumstick enters one of the hit zones the zone not collided with is deactivated Since

the outer zone still exists within the inner zone this deactivation prevents double notes from

being played When the drumstick leaves the hit zone the deactivated collision mesh is

reactivated

Figure 2-2 A plan view of the drum collision zones

4 The xylophone contains notes from the second and third musical octaves

8

Figure 2-3 A cross section of the collision zones in the virtual drum

While the design of the underlying software structure of the instruments is key to the final

product the instruments also need geometry that can be seen by the user To that end 3D models

were found online [1] [2] [3] to represent the instruments in the virtual environment Figure 2-4

shows the virtual xylophone

Figure 2-4 Virtual xylophone in the virtual environment

Once the models were imported into the Unity project collision meshes and the instrument- and

note controller scripts were integrated with the model so that they could be played

22 Gesture Recognition

The system must recognize the userrsquos hand gestures to control the virtual environment

The LeapMotion controller comes with an implemented set of basic gesture recognition tools

This section outlines how these tools can be combined to create an environment that recognizes

and responds to different user hand gestures Subsequently this section outlines the choices of

hand gestures that are required to control the system

9

221 Detectors

To recognize hand gestures LeapMotion has implemented a basic set of detectors to

recognize when the userrsquos hands satisfy various criteria These basic detectors are easily

accessible through LeapMotionrsquos Unity software libraries Developers can create more

complicated custom detectors by combining sets of these basic detectors In this project three

LeapMotion detectors were used the palm direction detector finger extension detector and the

logic gate detector

The palm direction detector is triggered when the palm is pointed in a specified direction

This detector includes two sensitivity variables These sensitivity variables allow the users to be

less precise with their palm direction to activate and deactivate the detector This range is

implemented by giving the detector specific angles to act as a margin of error for the triggering

palm direction One of the variables specifies the activation angle and the other specifies the

deactivation angle LeapMotion named these variables the lsquoOnAnglersquo and lsquoOffAnglersquo

respectively [4]

The finger extension detector detects when specified fingers are extended or not The

detector is set to monitor the state of fingers which can be one of three options lsquoextendedrsquo lsquonot

extendedrsquo or lsquoeitherrsquo [5] The detector is also able to monitor a minimum and maximum finger

extension count [5]

The logic gate detector does not detect specific features of the userrsquos hands but it is

activated when various other detectors are activated in a specific combination The logic gate

detector can emulate an lsquoANDrsquo gate an lsquoORrsquo gate or the negated versions of these gates This

detector allows for the recognition of very specific gestures by combining palm direction and

finger extensions

These detectors were used to recognize different types of gestures a finger-pointing

gestures and an open palm gesture These gestures are used in different combinations to control

the virtual environment

222 Finger-Pointing Gesture

The finger-pointing gesture triggers the drumsticks to appear Initially the drumsticks

were going to be activated through a fist gesture However the LeapMotion controller proved to

be incapable of detecting lateral rotations of the fist The extended finger of this gesture helps the

LeapMotion controller orient the virtual hands and allows finer tracking than the fist gesture

10

Figure 2-5 shows the modeled hands in the virtual environment that satisfy the constraints for this

finger-pointing gesture

Figure 2-5 Finger-pointing gesture

Both the palm direction and the finger extension detectors were used to detect the finger-pointing

gesture Table 2-1 and Table 2-2 show the parameters used for the two respective detectors

Table 2-1 Finger-pointing gesture palm direction detector

Detector Direction OnAngle OffAngle

Palm Direction In-and-Downwards 40 60

Table 2-2 Finger-pointing finger extension detector

Detector Thumb Index Middle Ring Pinky Minimum

Extended

Maximum

Extended

Finger

Extension

Not-

Extended

Extended Either Not-

Extended

Either 1 3

These two detectors are passed through a logic gate detector in the lsquoANDrsquo configuration so that

only a simultaneous activation of the two tabulated detectors can trigger the finger-pointing

gesture

223 Open Palm Gesture

The open palm gesture is used to control the menus When the user opens their palm and

faces it towards the camera the menu opens This gesture was picked for the menu because it is

an intuitive way to create a menu user interface Figure 2-6 shows the modeled hands in the

virtual environment that satisfy the constraints for the open palm gesture

11

Figure 2-6 Open palm gesture

Similar to the finger pointing gesture Table 2-3 and Table 2-4 show the parameters used

for the palm direction detector and the finger extension detector for the open palm gesture

Table 2-3 Open palm gesture palm direction detector


Palm Direction Camera-Facing 30 60

Table 2-4 Open palm gesture finger extension detector


Extended

Maximum

Extended

Finger

Extension

Extended Extended Extended Extended Either 4 5

These two detectors are also passed through a logic gate detector in the lsquoANDrsquo configuration to

combine these two detectors

23 Virtual Hands

The virtual hands module consists of all the game objects that are used to interact with

the virtual environment The module was designed to give the user full control of the virtual

environment without the requirement of a keyboard or mouse There are five different objects in

this module the left-hand menu the right-hand menu environment selection menu the

drumsticks and the mallets

The in-game menus were designed around and inspired by a painterrsquos palette Controlling

the system is as easy and unobtrusive as changing paint colors while painting To open the menus

12

the user opens their palm and faces it toward their face To select an option the user must then

use the index finger on their opposite hand to press the desired button When a button is activated

the buttonrsquos outer ring will turn from black to orange

The left-hand menu as shown in Figure 2-7 allows the user to select the instrument they

would like to play When the user performs the open palm gesture with their left hand three

buttons appear to the right of the hand The top button shows the drum set the middle button

shows the xylophone and the bottom button hides the selected instrument Selecting an

instrument will hide any other visible instrument

Figure 2-7 Left-hand menu

The right-hand menu as shown in Figure 2-8 allows the user to control the looping and

metronome mechanics of the virtual environment Users can also select the virtual surroundings

they in which they would like to play Similar to the left-hand menu the right-hand menu is

activated by performing the open palm gesture with the right hand Four buttons appear in the

menu a record loop button a play loop button a metronome button and an surroundings

selection button Looping functionality can be triggered by toggling the record loop button and

the play loop button The metronome can be activated or deactivated by toggling the metronome

button When activated a slider will appear above the metronome button and a display will

appear in the palm of the userrsquos hand The slider can be activated by the userrsquos index finger to

select the desired tempo of the metronome The display will show the current tempo for the

metronome

13

Figure 2-8 Right-hand menu

The surroundings selection menu can be opened by selecting the lsquoENVrsquo button on the

right-hand menu The surroundings selection menu will then appear in front of the user with three

different scenes as shown in Figure 2-9 To select their surroundings the user must reach out and

touch the desired scene

Figure 2-9 Surroundings selection menu

The drumsticks appear when the user performs the finger-pointing gesture The sticks are

mapped to the movement of the userrsquos finger as opposed to statically mapped with respect to the

userrsquos palm This mapping helps to generate the feeling of lsquofinger-drummingrsquo

The game objects within this module cannot interact with each other If they could

drumsticks would appear and cause interference when selecting an option in a menu To prevent

interference menus and drumsticks cannot exist at the same time The objects are also given a

priority outlined in Table 2-5

14

Table 2-5 Virtual hands object priority

Priority Object

1 Environment Selection Menu

2 Hand Menus

3 Drumsticks amp Mallets

If multiple objects attempt to activate at the same time the object with the highest priority is

chosen and the rest are ignored

24 Looping

Looping is the process of having the instruments play a user defined sequence of notes

without constant user input on an infinite loop This feature allows the user to begin recording a

sequence of notes by pressing the lsquoRECrsquo button in VR Software saves the notes as well as the

delays between notes in the game controller as the user plays Recording is stopped by pressing

the lsquoRECrsquo button a second time The recording starts on the first note played and ends on the last

note This eliminates blank space at the beginning and end of the loop The recording of a new

loop overwrites previous loop data therefore a maximum of one loop can be stored at a time

In order to start or stop playback of the recorded loop the play button on the right-hand

menu must be pressed During the playback period the game controller sends the recorded notes

at the specified delays to the server which are then sent to the microcontroller to play on the

physical instruments Users can also play new notes while the loop is playing

25 Metronome

As outlined in Chapter 1 this project is to provide features to create a complete virtual

music environment As such a metronome has been implemented in the virtual environment A

metronome is a device that helps a user play music at a steady tempo To help the user the

metronome clicks at a constant tempo The user may then associate a click with a beat to help

keep time A software metronome was designed and implemented in this project with an integral

range of 40 to 208 BPM

The metronome triggers a Unity sound event that causes the PC to play a sound at a

constant rate In Figure 2-10 the flowchart of the metronome is shown

15

Figure 2-10 Program flow for metronome

If the metronome is active the user can specify the rate at which the metronome clicks through a

slider that can be selected from the right-hand menu If the metronome is inactive then no clicks

will be heard

27 Communication

The communication module is the subsystem of the project that connects the virtual

environment to the instrument robot The communication takes place over the internet messages

are passed from the virtual environment to the web server using HTTP requests This method

allows for two distinct favourable features First multiple users can play the same real

instruments from different client machines and second the client machines can be truly remote

If the client machines can connect to the webserver they can be anywhere

28 Game Controller

The game controller is the central hub of the software modules in the VR system All

inter-module communication is routed through the game controller This simplifies dataflow and

increases system modularity and scalability The game controller also controls what elements to

display to the user at a given time When a new instrument is chosen through the menu system

the game controller is queried and spawns an instance of the desired instrument at the appropriate

location The music loop data is also stored in the game controller

16

29 Summary

The VR system is the principle portion that the user will interact with Using data

gathered by the LeapMotion controller users are able to use intuitive hand gestures to play virtual

representations of instruments Notes played in the VR are then sent to the instrument robot over

the internet allowing for remote operation of the instrument robot

17

Chapter 3 Web Server Module

A web server is a piece of software that accepts HTTP requests as input and processes

them The MusicBot 5000 utilizes a webserver to enable multiplayer and remote locality system

enhancements The nature and versatility of the internet made a web server the logical choice to

receive information from the VR and output to the instrument robot via I2C I2C is the required

communication protocol for the instrument robot as outlined in Section 52

The web client serves as an additional interface that people can use to interact with the

instruments if the virtual reality is occupied or unavailable in addition to assisting in the testing

of the server It can be accessed from any internet enabled device with a browser installed The

web client uses the same communication protocol as the virtual reality which is discussed in

Section 21 It does not require any additional server configuration as it produces the same

messages as playing the instrument in virtual reality The web client accepts a user defined

username which is stored along with data about their usage of the instruments

31 Server Architecture

The NodeJS framework was chosen to implement the web server because the base

framework can be expanded to allow for direct I2C communication NodeJS is compatible with

the model view controller framework which centres around three main principles The first is

that the view is the user interface Second the model interfaces with the back end which can be a

database another server or a number of other modules including an instrument robot Last the

controller is the bridge between the model and the view All HTTP requests from the view are

routed through the controller which transfers them to the appropriate model The view for the

MusicBot 5000 is either the virtual reality or the web application discussed in the following

section These views send HTTP GET requests to the controller which route the data from the

requests to the correct model based on the information in the URL The model then outputs the

correct I2C packet to the instrument robot Figure 3-1 outlines how the web server communicates

with the client

18

Figure 3-1 Web server flow chart

This server uses a RESTful API to handle incoming requests This means that the server

waits until a client sends a request routes it to the proper function and sends a response back

This API allows the server to receive customized requests in this instance the server receives a

custom username and note name This custom username is sent in every request as RESTful

servers do not maintain connections

I2C packets were constructed using the node-mcp23017 library There are two different

open source NodeJS libraries that are built for the MCP23017 IC but the node-mcp23017 library

was selected because it allows for the configuration of GPIO pins as output or input The address

of each MCP23017 is configured in the web server as well The server is always started in debug

mode so if the addressing or the output configuration is unsuccessful there will be immediate

text indication written to the console

32 Web Application

A web application was created so that the instruments can be played by users who do not

currently have access to the virtual reality but have access to an internet connected device Given

that web applications are a very common user interface this asset was put into the final design

This web application provides a different way to interact with the system

19

The web application accepts the users choice of username which is constrained to fifteen

characters This username is not stored on the server but is stored on the client and sent in the

URL of every request Figure 3-2 shows the login page of the MusicBot 5000 web application

Figure 3-2 Login screen for the web app

Figure 3-3 shows the web application interface to interact with the xylophone In order to

play a note users need only to press the corresponding note on the screen The web application

then sends HTTP requests to the server telling the server which note to play on which instrument

Figure 3-3 Xylophone interface on the web app

Users can also play the drum kit by selecting the drum kit button seen in the top left

corner of Figure 3-4 When the drum kit has been selected the user can press buttons that

correspond to the four solenoids on the drum Since users can be more accurate with their motions

20

on the web client than in the VR users can select all four solenoids individually in the VR users

only have two hit zones There is also a button for the cowbell Finally there is a button to begin

a drum roll If a roll is currently in progress the button changes to stop the roll

Figure 3-4 Drum kit interface on the web

The web application was written using HTML CSS and JavaScript which allows it to

seamlessly integrate with the RESTful web server The general styling is from Twitters Bootstrap

library which is designed to be visually appealing and minimalistic Some CSS adjustments were

made to change the length width and shape of certain xylophone keys and hit zones to better

approximate the instruments The drum was made to be round and the hit zones were made to fit

in the correct sections of the drum

The web application sends an HTTP GET request whenever a key is pressed These

requests are identical to those sent by the virtual reality This request is met with an HTTP

response from the server It is standard for web applications to wait for a response before

continuing A small text field is included below the xylophone which allows the user to see the

status of their request This field changes from the name of their note to success upon a 200

OK response

The webserver serves as a communication channel between the user interface module and

the instrument robot module

21

Chapter 4 Instrument Robot Module

In order to translate collision events in the virtual environment into playing a physical

instrument hardware in the form of actuators amplifier circuits mounts to hold the actuators in

place and a microcontroller are required These components make up the instrument robot

subsection of this project This chapter discusses the choice of actuators and how they work the

design and construction of the amplifier circuits the design of mounts and mallets for the

actuators and the role of the microcontroller

41 Actuators

The purpose of the actuators is to strike the real-world instruments The actuators must be

able to respond to input at a rate of at least two strikes per second so that they can play the

instruments at a reasonable tempo in real-time which a primary requirement of the system as

discussed in Chapter 1 To meet this requirement one actuator per note is necessary Having an

actuator move to hit successive notes creates too much latency in the system to have a real-time

response and would not allow for simultaneously playing two notes which is a specification of

this project

The actuators used for this project are pull type solenoids with a spring Solenoids are

linear on-off actuators that pull in when current is put across the coil inside due to the induced

magnetic field A spring in the solenoid returns the striker to the original position when current is

turned off The simplicity of the actuation from solenoids allows for the actuators to be placed

above the desired actuation point In addition the low cost per solenoid allows for one actuator

per note on more than one instrument

The first actuators that were considered for this project were servomotors Servomotors

are rotary actuators that use position feedback to control actuation [6] The idea for the setup was

that the servomotors would have mallets attached onto the rotating shaft and then when the motor

would rotate the mallet would move downward to strike the instruments The servomotors would

be mounted onto a platform in front of the instruments to ensure they would be at the correct

height for actuation Servomotors were deemed unfeasible due to price issues mechanical

complexity and relatively large actuation time Instead solenoids were chosen as the actuators

The main disadvantage of using solenoids rather than servo motors is that dynamics5

cannot be implemented as a feature To create the loudest and least dampened sound from the

instruments the solenoids in this project require a very specific actuation time of five

5 In music dynamics refer to how loud or soft an instrument is being played

22

milliseconds As a consequence the solenoids cannot implement more than one volume

Implementation of dynamics is discussed in Chapter 7 as a possible expansion for future work

411 Solenoids

Solenoids are actuators that work by converting electrical power into linear mechanical

power Solenoids consist of a soft-iron shaft that has a wire coiled around it held within a soft-

iron frame The shaft is separated from the base of the frame creating an air gap Once current is

applied across the coil the induced magnetic field works to reduce the air gap and the shaft is

pulled toward the frame until the air gap is removed [7] For a solenoid to function as an actuator

there needs to be a return spring placed within the system

Figure 4-1 shows a diagram of the solenoid while no current is being applied to the coil

and Figure 4-2 shows the solenoid once current has been applied across the coil

Figure 4-1 Diagram of solenoid with no current applied to coil shaft out

Figure 4-2 Diagram of solenoid with current applied to coil shaft pulled in

The potential energy stored in the compressed spring moves the shaft back to its original position

once current is removed from the coil to restore the air gap This allows the solenoid to act as an

on-off linear actuator

23

412 ROB-11015

Most low-cost solenoids are pull type meaning that when power is applied to them the

pin is retracted This presents the need for an active low setup as the MusicBot 5000rsquos solenoids

create sound when extended Having an active low solenoid setup presents two main issues an

extremely high-current power source is required and the solenoids become very hot when active

for long periods

The ROB-11015 is a pull solenoid with an extra attachment on the bottom that gets

pushed out when the top pulls in This combines the low-cost aspect of a pull solenoid with the

ability to have a piece push out and have active actuation of a push solenoid making it perfect for

this project

Another advantage of the ROB-11015 is its small size Figure 4-3 shows the solenoid

next to an American quarter for scale

Figure 4-3 ROB-11015 solenoid [8]

Since the setup for the xylophone requires one solenoid per key the solenoid must be much

narrower than a single xylophone key The ROB-11015 is 12mm in width whereas a xylophone

key is 36mm in width with a space of 6mm between keys This leaves enough space for a bracket

to mount one solenoid above each key

42 Amplifier Circuits

The microcontroller is unable to power the solenoids directly from its internal power

Attempting to do so would cause it to malfunction reboot or even break permanently This can

be rectified using Darlington Transistor arrays which are found in the ULN2803 This IC accepts

a digital input with low current on one side which allows up to 500mA to flow through the

adjacent pin to ground There is a common pin which is set to positive for inductive loads such as

24

solenoids to suppress kick-back voltage [9] A low current digital signal is applied to the input pin

of the ULN2803 from the microcontroller which activates the load on the adjacent pin

The load in this case is a solenoid When the source voltage for solenoids is DC they can

be roughly approximated by a resistance value In the case of the ROB-11015 the value used for

simulation in MultiSim was 455Ohms This resistance value was determined by using Ohmrsquos

Law with the current rating of 11A and the voltage rating of 5V given in the datasheet [9] Figure

4-4 contains the schematic of the circuit used to simulate the ULN2803 connected to a ROB-

11015 solenoid

Figure 4-4 MultiSim schematic of circuit with single Darlington pair and solenoid load

The circuit uses a square wave source to simulate the output of the microcontroller since

it will send on-off pulses to indicate when a note is played The oscilloscope measures both the

input voltage from the source as well as the output voltage across the solenoid with respect to

ground The output of the oscilloscope in this circuit is shown in Figure 4-5

25

Figure 4-5 MultiSim simulation oscilloscope output

The output of the virtual oscilloscope shows the desired functionality for this application when

this circuit was built on a breadboard the same output was observed on the physical oscilloscope

43 Solenoid Mounts

Since solenoids provide linear and not rotary actuation like servo motors the solenoids

need to be positioned above the instruments rather than in front of them Frames are required to

hold the solenoids above the instruments Furthermore the mounting frames need to allow for

easy adjustment of each individual actuator to allow their position to be fine-tuned to create the

optimal sound This section discusses the design choices made in the creation of solenoid mounts

for the xylophone drum and cowbell

The mounting frames for the drum and xylophone are similar in design and have one

adjustable piece across the top of the instruments that moves all the solenoids up and down

together Figure 4-6 shows the rear view of the xylophone mount

26

Figure 4-6 Solenoid mount for xylophone (rear view)

The wooden stands on either side of the instrument have slits down the centre to allow

the wooden piece where the solenoids are mounted to be adjusted both up or down and sideways

The large screws that can be seen in the figure above can be loosened and tightened to allow for

this adjustment

The drum mount as show in Figure 4-7 differs from the xylophone mount in three main

ways First there are risers under the snare drum to keep the snares out of contact with the base

This also allows for better drum resonance There are also some cut-outs in the top bar to

accommodate the rim of the drum Another difference between the drum and xylophone mount is

that the xylophone supports eleven solenoids whereas the drum only requires four The figure also

shows the custom mounting brackets that are used to hold the solenoids above the instrument

Since they wrap around the body of the solenoid horizontally this allows for slight vertical

adjustment of each solenoid individually This is also present on the xylophone frame

Figure 4-7 Solenoid mount for drum (front view)

27

Finally the cowbell mount is unique in that the cowbell must be held off any surface to

produce the correct sound As such the frame must mount both the cowbell as well as the

solenoid In addition the cowbell is struck from the side rather than the top like the xylophone

and drum Figure 4-8 shows the final design for the cowbell mount with accommodations that

allow the aforementioned requirements to be met

Figure 4-8 Cowbell mount

The mount has a removable rod to hold the cowbell off the base so that the correct sounds can be

produced Furthermore a vertical wooden piece to mount the solenoid allows it to actuate

horizontally The mounting location also has multiple hole sets so that the exact vertical position

of the solenoid can be adjusted as required

44 Mallets

A mallet attachment was created to strike the instruments because the solenoid pins are

made of metal Hitting a drum or xylophone with metal not only creates the incorrect sound in the

short term it can also cause wear and tear that will alter the sound of the instrument permanently

Plastic was chosen to create the mallets because it can be 3D printed in the exact shape that is

desired it will not damage the instruments in the long term and it will also create the correct

sound The mallet was created in two sections shown in Figure 4-9 a half sphere base and a

cylinder with a cut-out to allow for insertion onto the solenoid pin

28

Figure 4-9 3D model of a mallet

Holding the mallet at the orientation shown in the figure above and holding the solenoid

so the pin sits vertical the mallet can slide horizontally onto the end of the solenoid pin head The

mallet was 3D printed in blue plastic and two-part epoxy was used to affix it onto the end of the

solenoid

45 Microcontroller

A microcontroller was required to receive messages from the virtual reality and relay

them to the instruments The scope of requirements for this project are entirely met by the

Raspberry Pi which is currently implemented in the system The initial selection for these

requirements was an Arduino Mega which was chosen specifically because it has 54 GPIO pins

This number of GPIO pins allows for the control of more solenoids Software was written to

accept input via USB and output to the solenoids via GPIO pins This microcontroller was used

as an intermediate system to verify the functionality of the instrument robot and the virtual

reality

The microcontroller was switched from an Arduino to a Raspberry Pi so that the

instrument robot is able to host a webserver This change allows the virtual reality to

communicate with the instruments over Wi-Fi This web server module is standalone receiving

inputs over Wi-Fi and outputting via I2C bus

The Raspberry Pi was selected for this project because it has the amenities of a full

computer including an installed Linux distribution called Raspbian with the addition of GPIO

pins and an I2C bus Initially the functionality of the webserver was tested by outputting via the

built-in GPIO pins on the Raspberry Pi To allow for expansion in the project in the form of

additional instruments it was determined that the seventeen built-in GPIO pins would not be

29

enough to control the desired number of solenoids A communication module was added to allow

the Raspberry Pi to interface with the instrument robot via I2C instead of using GPIO output

46 Summary

The instrument robot is the system that facilitates the playing of the physical instruments

Solenoids mounted above the instruments with mallet attachments are used as actuators to strike

the xylophone drum and cowbell The solenoids are supplied sufficient current via amplifier

circuits that amplify the signals coming from the Raspberry Pi The Raspberry Pi controls the

physical instruments with messages coming from the web server module

30

Chapter 5 System Integration

Once the three main subsystems of this project namely the user interface the web server

and the instrument robot were fully functional individually they were combined to form the

completed MusicBot 5000 A key component in the full integration of the system was the

communication between the various subsystems

51 Client to Web Server Communication

The communication between the client and the web server is HTTP requests over Wi-Fi

These requests are sent by the virtual reality or the web application and received by the web

server which processes the request and sends an HTTP response The server waits until a request

is received at which point it processes the request and returns an HTTP response based on that

request There are many different types of responses but this server uses only two 200 OK and

400 BAD REQUEST

The expected URL structure of a request is httpIPAddressportnote$note$username

In this structure lsquoIPAddressportrsquo is the address of the server on the LAN This is where the

requests are directed The next field $note is a string variable that indicates which note is to be

played in the format of three ASCII characters The format of these characters is discussed in

Section 21 The next field $username is a string variable that contains the username selected by

the client For the virtual reality this username is hardcoded as VR User but for the web client

this can be chosen by the user

Upon receipt of the URL within an HTTP GET request the server will process the note

string and route the resulting instruction to the correct instrument The successful routing of the

request ends when the server responds to the client with an HTTP response of 200 OK If the

URL is invalid or the request is not a GET request the server will respond with an HTTP

response of 400 BAD REQUEST The user will never receive a 400 error if they are using the

server as intended

52 Webserver to Instrument Robot Communication

The webserver outputs instructions over I2C to the instrument robot This I2C is used to

address eight I2C GPIO expanding ICs (MCP23017) that create a GPIO bus with 128 GPIO pins

The MCP23017 has two output busses each with eight GPIO pins for a total of sixteen GPIO

pins per chip The chip has external power ground and receives one I2C clock and one data line

both which are standard communication lines on an I2C bus Each IC is provided with a unique

address using pins A2 A1 and A0 These pins are addressed using binary 0 (ground) and 1

(VDD)

31

This GPIO expander is modular so anywhere between one and eight ICs can be

addressed using one I2C bus This specific IC was selected because it has sixteen GPIO pins

which is the maximum number of pins of any GPIO expander on the market The other options of

GPIO expanders have only eight output pins

53 Summary

In summary when the user plays a note the VR or web application sends an HTTP

request to the web server This request contains details about which note is to be played and who

is playing the note The data in the request is processed by the server and output to the instrument

using I2C packets that specify which note is to be played These packets are then decoded

amplified and used as electric signals to control the solenoids

32

Chapter 6 System Validation

The specifications that were achieved for this project were validated either through user

testing responses or a systematic testing procedure This chapter outlines the methods used for

testing final specification values as well as the results of those tests Table 1-1 containing the

details of the project specifications is repeated in this section for readerrsquos convenience




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes

The same note can be played twice per second


Surpassed


250 milliseconds

35 Milliseconds Surpassed

61 Objective System Verification

While some of the specifications found in Table 1-1 required testing other specifications

could be validated by inspection These are the elements of the design that do not relate to user

experience or hard time constraints Firstly this system is to be playable on personal computers

running the Windows 10 operating system This requirement was necessary because the Unity

assets provided by the LeapMotion team are only compatible with Windows computers Since the

system works on Windows PCs it is safe to say that this specification has been met

33

Next according to proposed specifications there was to be software support for one loop

on the VR client This requirement was also met as the VR client supports exactly one loop

Another system specification was that the loops were to have a maximum length of twenty

seconds This time was chosen because at the time the loops were to be stored on the

microcontroller attached to the instrument robot This specification has been surpassed the loop

is stored on the VR client and is theoretically of unlimited length In practice it is limited by the

clientrsquos available memory

An additional design specification was that two notes should be able to be played

simultaneously This was accomplished for certain values of lsquosimultaneousrsquo Messages from the

VR are sent out sequentially so two simultaneous notes will have a short delay between them

However this delay is so small as to be negligible

One final objectively observable specification is the number of gestures recognized by

the system This specification has also changed since the design was first conceived During the

initial design the concept was that each action the system was to perform would have its own

gesture However this has been modified in the final design It was found that the LeapMotion

controller could not handle the fine tracking required for these gestures Instead a menu system

was implemented that facilitates all desired interactions This resulted in only two gestures being

needed an open palm gesture to open each menu and a finger-pointing gesture to make

selections The finger-pointing gesture is also used to summon drumsticks or mallets so that the

user can play the virtual instruments

611 Systematic Testing

There are three metrics that needed systematic testing The server latency the gesture

recognition accuracy and the successive note rate To test the server latency the Wireshark

packet capture tool was used to obtain HTTP statistics on the Web Server From these statistics

the average round trip time was calculated and found to be 35ms The round trip time as a

function of HTTP packet number is shown in Figure 6-5

34

Figure 6-1 Round Trip Time for HTTP Packets

In this test the server has an IP address of 192168233000 and the client machine that observed

these packets has an IP address of 1921682253624 Packets have a round trip time that ranges

from 210ms to 2ms

To verify the gesture recognition rate the gesture recognition software was tested on its

ability to recognize the implemented gestures on the left and right hands exclusively as well as

both hands simultaneously Therefore the two implemented gestures one to activate the

drumstick and one to activate the menu underwent three tests each with twenty trials per test

Figure 6-2 shows that the gesture recognition was found to be successful 9767 of the time

35

Figure 6-2 Gesture Validation Tests

To conduct these trials as objectively as possible each trial commenced without any

hands visible to the LeapMotion controller The tester would then insert their hands in the view of

the LeapMotion controller perform the gesture and then remove their hands from view It was

found that all failed recognitions were because the LeapMotion software failed to identify if the

hand was the testerrsquos left or right hand This observation is solidified as the tests with the gestures

performed on both hands simultaneously were 100 successful as seen in Figure 6-2

The successive note rate was tested by playing a note multiple times and logging the

time of when the notes were played The frequency of successive notes was found to be 125 Hz

with eight milliseconds per sample This frequency of successive notes surpasses the original

specification of the system which was twice per second The systematic tests were all successful

in proving that the MusicBot 5000 meets all proposed performance metrics

62 Subjective Testing

User testing was conducted in order to validate the intuitiveness of the user interface In

total eleven test subjects with varying VR and LeapMotion controller experience contributed to

the results of the test A Google Forms survey was used and the test subjects anonymously filled

out the form after the test session which lasted 20 minutes The full survey used for testing can

be found in Appendix B In order to evaluate the previous experience of the users which will

affect how intuitive they find the system they were asked before the session to evaluate their

36

experience with VR and the LeapMotion controller Figure 6-3 shows the results of the previous

VR experience of the test subjects

Figure 6-3 Test subject previous VR experience

The scale of the above figure varies from 1 which means no experience to 5 which

means very experienced The majority of test subjects felt they had between no previous

experience and middling previous experience with virtual reality The intuitiveness of the system

also depends on whether a subject has used the LeapMotion controller which is used to track the

hand gestures for the VR Less than 20 of test subjects had previously used the LeapMotion

controller The responses to these two questions show that the test subjects in this experiment

were relatively inexperienced with the matter being tested

The test subjects were given a set of tasks such as completing a scale on the xylophone

or creating a loop At the end of the test subjects were asked to evaluate the intuitiveness of the

experience both at the beginning of their test as well as at the end of the session Figure 6-4

shows the user feedback on the intuitiveness of the system at the beginning of the session

Figure 6-4 User feedback on intuitiveness at beginning of session

37

The scale ranges from 1 which was indicated on the test to mean ldquoNot intuitive at allrdquo to

5 which was indicated on the test to mean ldquoI knew what to do at all timesrdquo No users indicated

that the system was not intuitive however only two indicated they felt they knew what to do at all

times This is to be expected since the test subjects have a low overall experience with the

fundamental control elements present in the system Since a user often needs time to adapt to a

new system subjects were also asked how intuitive they found the system at the end of the

session Figure 6-5 the user feedback on the intuitiveness of the system at the end of the session

Figure 6-5 User feedback on intuitiveness at end of session

The scale for the responses in the above figure is the same as in Figure 6-4 After the end

of the testing session the intuitiveness rating was markedly improved Another metric that was

considered to be important due to the novel nature of this project was the enjoyment of the users

This reflects on the quality of the environment in addition to the intuitiveness of the system At

the end of the sessions each user was asked whether they had fun The subjects unanimously

responded that the experience was fun With an initial average intuitiveness rating of 3735 a

final average intuitiveness of 4735 and a 100 fun rate the specification of being easy to use

was met

38

Chapter 7 Conclusion

In conclusion the design and implementation of the MusicBot 5000 was a resounding

success The system is functioning at a level that has surpassed proposed technical specifications

outlined Table 1-1 The creation and integration of the virtual reality web server and instrument

robot have successfully established an enjoyable user experience

The VR allows users to control real world instruments by interacting with virtual

instruments It implements loops changes to the surroundings and a metronome with variable

speed The virtual reality is designed to have intuitive user interactions that allow users to start

playing immediately after being introduced to the system

The web server acts as a portal to the instrument robot It accepts requests from the client

for different notes to be played and instructs the instrument robot to play those notes In addition

the web server hosts a web application that allows the user to interact with the instrument robot

using a browser The instrument robot receives input telling it which notes to play and it

subsequently plays those notes

The project also allows future expansion to improve on system functionality Non-trivial

future work could include adding a method for haptic feedback and adding a head-mounted

display These additions would create an experience that more closely mimics playing an

instrument thus making the environment more immersive In addition a different actuation

system could be implemented to enable dynamics in the system This would also require changes

to the software so that the speed of strokes could be recognized by the VR and implemented by

the instrument robot

Together these modules create the MusicBot 5000 a unique interactive experience The

MusicBot 5000 is a successful implementation of virtual reality control of a hardware system and

a stride toward the widespread virtual reality control of hardware systems

39

References

[1] Joey Sketchup 3D Warehouse Drum Kit Trimble Inc 26 March 2014 [Online] Available

https3dwarehousesketchupcommodel3f07dd64edc69713a1e0605f15b1d68bDrum-Kit

[Accessed 11 November 2016]

[2] Natedogg Sketchup 3D Warehouse Xylophone Trimble Inc 14 March 2014 [Online]

Available

https3dwarehousesketchupcommodelcb2f93d1bf03a3cc349aecb1a766079cXylophone

[Accessed 15 October 2016]

[3] H L Sketchup 3D Warehouse Cowbell Trimble Inc 24 March 2014 [Online] Available

https3dwarehousesketchupcommodel28952cba43bd2d19cb833ee48ab7383aCowbell


[4] LeapMotion PalmDirectionDetector [Online] Available

httpsdeveloperleapmotioncomdocumentationunityunityUnityPalmDirectionDetectorhtml

[Accessed 11 March 2017]

[5] LeapMotion ExtendedFingerDetector [Online] Available

httpsdeveloperleapmotioncomdocumentationunityunityUnityExtendedFingerDetectorhtml


[6] F Reed How Servo Motors Work Jamesco 6 September 2012 [Online] Available

httpwwwjamecocomjamecoworkshophowitworkshow-servo-motors-workhtml [Accessed

17 March 2017]

[7] P Hills Solenoids and Actuators 18 03 2003 [Online] Available

httphomepageswhichnet~paulhillsSolenoidsSolenoidsBodyhtml [Accessed 13 02 2017]

[8] Sparkfun SparkFun [Online] Available sparkfuncomproducts11015 [Accessed 3 02 2017]

[9] SHENZEN ZOHEN ELECTRIC APPLIANCES Co Ltd ROB 11015 DataSheet [Online]

Available httpscdnsparkfuncomdatasheetsRoboticsZHO-0420S-

05A4520SPECIFICATIONpdf [Accessed 04 02 2017]

40

Appendix A Final Budget

Initially the entire $500 budget was allocated to different portions of the project but

thanks to gracious donations from family friends and advisors the project was approximately

$200 under budget The xylophone was donated by a group memberrsquos family friend Sandra

Komishon and a second LeapMotion controller was donated by Dr Ian Jeffrey Several of the

purchased parts went unused because they did not meet the required specifications of this project

Table A-1 outlines how the budget was spent on the MusicBot 5000

Table A- 1

Part Name Part Number Quantity Total Cost (CAD) Comments

Xylophone NA 1 0 Donated

Raspberry Pi NA 1 0 Donated

Snare Drum NA 1 0 Donated

Cowbell NA 1 0 Donated

Wi-Fi USB Dongle NA 1 0 Donated

LeapMotion

Controller

NA 2 6588 One was

donated

Solenoid ROB-11015 25 18508

I2C GPIO Expander MCP23017 6 1038

28 Pin IC Socket ED281DT 10 489

20 Pin Connection

Header

924327-28-10-1 2 1272 Unused

Tinned Perf Board SBB2805-1 2 1006

JST Connection Header BM05B-GHS-TBT 6 474 Unused

Total 29375

41

Appendix B User Testing Documentation

The survey questions used to evaluate the system during user testing are found below

The first two questions are to gauge the experience of the user with virtual reality in general and

the LeapMotion controller in particular The next several questions ask if the user was able to

complete the tasks in the test Next the user is asked to gauge the intuitiveness of the system at

the beginning and end of the system Finally the user is asked if they had fun

42

43

Appendix C Source Control

All code for this project was uploaded to GitHub This allowed for easy collaboration and

source control The code on GitHub is organized into two folders RPi and VR The RPi folder

contains the code required to run the server and host the web application on the Raspberry Pi

The source files for the web application are also contained in this folder The VR folder contains

the Unity project for the MusicBot 5000 virtual reality user interface

The repository can be found at httpsgithubcomMusicBot5000Musicbot5000

i

Abstract

This report details the design and implementation of a virtual reality interface for playing

real-world instruments which we fondly refer to as the MusicBot 5000 The system consists of

three main modules the user interface the web server and the instrument robot The user

interface displays playable instruments in non-immersive virtual reality and translates user hand

gestures into HTTP requests which are sent to the web server The instrument robot receives

instructions from the web server and operates peripheral instruments accordingly

The goal of this project was to design and implement a system that allows users to play

instruments remotely with an interface that closely mimics the actual playing of the instruments

The MusicBot 5000 met or exceeded all design specifications set out for this project with

peripheral instruments including a xylophone a drum and a cowbell

ii

Contributions



setup





Metronome

Looping



Mount design



Final report


iii

Acknowledgements









iv

Table of Contents

Abstract i

Contributions ii


List of Figures vi

List of Tables vii







221 Detectors 9



23 Virtual Hands 11

24 Looping 14

25 Metronome 14

27 Communication 15


29 Summary 16





41 Actuators 21

411 Solenoids 22

412 ROB-11015 23


v


44 Mallets 27


46 Summary 29




53 Summary 31






References 39




vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








ii

Contributions



setup





Metronome

Looping



Mount design



Final report


iii

Acknowledgements









iv

Table of Contents

Abstract i

Contributions ii


List of Figures vi

List of Tables vii







221 Detectors 9



23 Virtual Hands 11

24 Looping 14

25 Metronome 14

27 Communication 15


29 Summary 16





41 Actuators 21

411 Solenoids 22

412 ROB-11015 23


v


44 Mallets 27


46 Summary 29




53 Summary 31






References 39




vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








iii

Acknowledgements









iv

Table of Contents

Abstract i

Contributions ii


List of Figures vi

List of Tables vii







221 Detectors 9



23 Virtual Hands 11

24 Looping 14

25 Metronome 14

27 Communication 15


29 Summary 16





41 Actuators 21

411 Solenoids 22

412 ROB-11015 23


v


44 Mallets 27


46 Summary 29




53 Summary 31






References 39




vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








iv

Table of Contents

Abstract i

Contributions ii


List of Figures vi

List of Tables vii







221 Detectors 9



23 Virtual Hands 11

24 Looping 14

25 Metronome 14

27 Communication 15


29 Summary 16





41 Actuators 21

411 Solenoids 22

412 ROB-11015 23


v


44 Mallets 27


46 Summary 29




53 Summary 31






References 39




vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








v


44 Mallets 27


46 Summary 29




53 Summary 31






References 39




vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








vi

List of Figures






























vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








vii

List of Tables








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








viii









DC Direct current











VR Virtual reality

Wi-Fi IEEE 80211x

1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








1


























2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








2








start a metronome











3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








3




Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed



Surpassed


250 milliseconds

35 Milliseconds

Surpassed











for the user

4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








4







intervals2






5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








5


































6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








6






















7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








7



















reactivated



8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








8















9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








9

221 Detectors






logic gate detector







respectively [4]




extension count [5]





finger extensions









10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








10











Extended

Maximum

Extended

Finger

Extension

Not-

Extended


Extended

Either 1 3



gesture






11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








11









Extended

Maximum

Extended

Finger

Extension




23 Virtual Hands








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








12




















metronome

13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








13














14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








14


Priority Object


2 Hand Menus




24 Looping












25 Metronome









15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








15




will be heard

27 Communication







28 Game Controller







16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








16

29 Summary





17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








17


























with the client

18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








18













32 Web Application





19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








19












20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








20
















OK response



21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








21








41 Actuators







this project


















22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








22



411 Solenoids














23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








23

412 ROB-11015





for long periods




this project














24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








24








11015 solenoid






25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








25




43 Solenoid Mounts










26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








26





this adjustment










27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








27











44 Mallets








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








28





solenoid

45 Microcontroller








reality










29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








29



46 Summary






30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








30












400 BAD REQUEST













server as intended








(VDD)

31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








31





53 Summary






32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








32









Outcome

Supported Platforms

Windows PC

Windows PC Met



1 Loop Met




Met

Recognized gestures




Surpassed




Surpassed




Surpassed

Successive notes



Surpassed


250 milliseconds









33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








33



























34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








34




from 210ms to 2ms






35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








35




















36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








36
















37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








37
















was met

38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








38

























39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








39

References





Available














17 March 2017]







40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








40








Table A- 1







LeapMotion

Controller

NA 2 6588 One was

donated




20 Pin Connection

Header

924327-28-10-1 2 1272 Unused



Total 29375

41







42

43








41







42

43








42

43








43








Date post:	22-Apr-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

MusicBot 5000: Design and Implementation of a Virtual Reality Interface for Playing...

Documents