LOS TRICERAFLOPSAdrian Buckley
Evan Defibaugh
John Duhamel
Alex Kaiser
Darnell Parker
Craig Riggins
T Q
~Q
Project Objectives
- Relay real time Nintendo 64 session data between multiple hosts for network play
- Provide an interface for a Microsoft Kinect frontend in place of tradition controller
- Utilize non-obtrusive electronics- Minimize cost and maximize functionality- Power consumption is a secondary concern
System Architecture
Client 1 Client 2Server
Nintendo 64
ServerClient 1 Client 2
Player 1 Television
Player 2 Television
Player 1Controller Signal
Player 2Controller Signal
Player 2Controller Signal
Player 1Controller Signal
N64 RCASignal
N64 RCASignal
N64 RCASignalN64 RCA
SignalN64 RCA
Signal
Client Architecture
Regular N64 Controllers
TelevisionARM Cortex
MCU
N64 Controller Signal
KinectUSB Signal
DigitalRCA Signal
DigitalController Signal
Ethernet IO
Microsoft Kinect
Analog RCASignal
MCU takes input from 4 N64 controllers Microsoft Kinect can track 2 people
Server Architecture
ARM Cortex MCU
DigitalController Signal
DigitalRCA Signal
Ethernet IO
Nintendo 64
N64 ControllerSignal
AnalogRCA Signal
Can communicate with 1-4 clients Moves controller signal from client to N64
and RCA signal from N64 to client
Unified Hardware
N64 Controllers / Microsoft Kinect Television
ARM Cortex MCU
Controller Signal
RCA Signal
Controller Signal
Ethernet IO
RCA Signal
Nintendo 64
ControllerSignal
RCA Signal
1 hardware architecture will be able to operate as either the client or the server
Advanced System Architecture
Client 2Client 1 / Server
Nintendo 64
Client 1 / Server
Client 2
Player 1 Television
Player 2 Television
Player 2Controller Signal
Player 2Controller Signal
Player 1Controller Signal
N64 RCASignal
N64 RCASignalN64 RCA
SignalN64 RCA
Signal
Kinect Integrated Peripheral
Kinect Processor Linux Microcontroller
XBOX Kinect ProcessorLinux
Microcontroller
Kinect Sensor
I/O2 Cameras (RGB and IR Depth)IR Output4 Array MicrophoneUSB Port
Possible Implementations
ROS: BeagleBone VS C# 2010: Netduino VS C# 2010: PC
Implementing Kinect via ROS on Beaglebone
ROS (Robot Operating System) is an open source project focusing on interfacing microprocessors with robots (in our case the Kinect and a peripheral Linux microcontroller)
Kinect Processor
Control Node
The Netduino operates on Microsoft Visual Studios C# 2010 via Net. Micro workforce
Implementing Kinect via VS C# 2010 on Netduino
Minor Hiccups in Progression
Using the Robot OS ROS tutorial has
incorrect/missing info
Open source, thus compatibility issues from node to node
Unintuitive to navigate
Using the Netduino• Netduino requires
Net. Micro framework, where Kinect requires Net. Framework
• Trouble converting from standard to micro
There is no problems implementing Kinect via PC
Last Resort and Additional info
As a last resort the Kinect can be routed through a Windows PC for processing then sent to microcontroller.
More ROS infohttp://www.ros.org/wiki/kinect
(Install ROS)
http://www.ros.org/wiki/ROS/Tutorials
(ROS Tutorial)
http://beagleboard.org/bone
(Beaglebone Description)
http://www.ros.org/browse/list.php
(ROS software)
More Net. Micro workforcehttp://docs.com/BTN4 (Installing OpenNI)
http://www.netduino.com/netduino/schematic.pdf (Netduino Schematics)
http://informatix.miloush.net/Microframework/Articles/WpfEmulators.aspx
(.net micro install)
http://informatix.miloush.net/microframework/Articles/Reflector.aspx (.net micro reflector)
http://netduino.com/netduino/specs.htm
(Netduino Specs)
Gesture and Speech Algorithms
Joint Data is returned in X, Y, Z, and W (confidence) data
Evaluate position of joints relative to another and trigger button pressed event
For audio grammar files must be written Compare text output of audio and set trigger
for the controller or the sensor as being used Output control trigger and button triggers
Kinect Processor Interface
The Processor will output triggers in the form of buttons and sensor/controller triggers for both players one and two
The triggers will be relayed to the Linux microcontroller that will convert button trigger to N64 controller output format and use the control trigger to decide if forwarding the N64 controller data or sensor converted data
N64 Controller Tutorial
How the N64 Controller Sends Its Data
The N64 controller only has three pins: Ground, data, and VCC. It takes a +3.6V supply.
The N64 uses a protocol in which bits are sent serially to and from the controller as 4µs-wide pulses. ‘Low’ is 3 µs low and 1 µs high, and ‘High' is 1 µs low and 3 µs high.
N64 Data Polling
The N64 continually polls the controller using an 8-bit command (0x01), and the controller reports the state of its buttons and analog joystick with a 32-bit response: 16 bits for the buttons, signed 8 bits for horizontal joystick position, and signed 8 for vertical. Each bit packet ends with a stop bit (1).
Button Configuration
The buttons are mapped to their respective bits as follows:
Sample Joystick Positions and respective bits:
Bit 0 1 2 3 4 5 6 7
Button A B Z Start D-Up D-Down D-Left D-Right
Bit 8 9 10 11 12 13 14 15
Button Null Null L R C-Up C-Down C-Left C-Right
Left Extreme 1 0 0 0 0 0 0 1
Left Off Center 1 1 1 1 1 1 1 1
Right Off Center 0 0 0 0 0 0 0 1
Right Extreme 0 1 1 1 1 1 1 1
Up Off Center 0 0 0 0 0 0 0 1
Down Off Center 1 1 1 1 1 1 1 1
Joystick Off-Center Left & Down
The Translator Box The Kinect will take player movement/sound data and
send over button and trigger data to the “translator box,” which then converts that data into the 64-bit struct format shown below.
There will be a physical master switch on the translator box, to switch between Kinect control and N64 controller control. When this switch is set to “Kinect,” the translator will await trigger and button data from the Kinect to process.
If the switch is set to “N64,” the translator will need to keep track of one or two controllers, simultaneously converting their button presses into the shown format, but with the “More Buttons” bit and time delay consistently set to zero.
Button Data Joystick Data Which Player? More Buttons? Time Delay
16 bits 16 bits 1 bit (0 or 1) 1 bit Int (30-bit)
Server to Translator to N64 Another Translator Box is needed to translate the
Server’s data into N64 controller data for the N64. This translator will read in the controller data, and send
the correct button sequence via serial ports to the N64, correctly timed. If the N64 polls the “controller,” but the translator has not received button data from the server, the translator will automatically respond with a 32-bit ‘0’ analog stream and continue doing so until new button data appears.
Audio/Visual Signal
The video output of the Nintendo 64 carries visual information via an analog signal, plus two synchronizing signals for aligning the video signal on a television.
The audio output, also analog, is split into two signals for stereo speakers in a television.
Audio/Visual Waveforms
Video
Audio
Audio/Visual Sampling
The audio and visual output of the Nintendo 64 must be digitized and formatted for input into the server for transport over the internet.
Once the audio/visual information is sent over the internet it must be converted back into an analog signal to be displayed on a player's television.
A Series of Tubes – Hardware
Initial design and testing will be done on an Arduino with an Ethernet shield.Ardiuno is easy to program to send and receive
data packets. After getting a fully functional system with the
Premade hardware we will move on to a custom PCB with a Linux OS.
Our Internet Objectives
We need the data received from the server to be real timeIf there is missing data we just move to the next
data packet.We know that the receiver is ready to receive
packets.
TCP/UDP
TCP/IP uses an established connection between sender and receiver.This requires an acknowledge signal from the
receiver which takes time.
UDP instead sends packets in a continuous stream and doesn’t care if packets are missing on the receiver side.Instead of waiting for acknowledgement UDP just
tags the packet with the correct receiver IP address and sends it on its way.
UDP in depth
UDP sends data with a header, and within the header a 2byte portion telling the receiver how long the data section will be.This 2byte section allows for a maximum of 64Kb
section of data which is more than enough for sending the n64 signal and frames of an RCA signal.
The other parts of the header are receiver IP and port number.
Putting it all Together
Using UDP because we need a real time connection and don’t mind dropped packets.
Using a static IP for our server so that we have something to aim at for our UDP packets.
Using Arduino and Ethernet shield to prototype and onto custome PCB once all kinks are worked out.
ID Task Name Duration Start Finish Predecessors
1 Capstone Project 79 days Mon 1/16/12 Thu 5/3/12
2 Idea Development 5 days Mon 1/16/12 Fri 1/20/12
3 Documentation 73 days Mon 1/23/12 W ed 5/2/12 2
4 Functional Specification 5 days Mon 1/23/12 Fri 1/27/12
5 Funct. Req. Document 28 days Mon 1/30/12 Wed 3/7/12 4
6 Block Diagrams 20 days Mon 1/23/12 Fri 2/17/12
7 Block Diagrams for PDR 6 days Mon 1/23/12 Mon 1/30/12
8 Block Diagrams for CDR 14 days Tue 1/31/12 Fri 2/17/12 7
9 Build User's Manual 73 days Mon 1/23/12 W ed 5/2/12
10 Preliminary version for PDR 6 days Mon 1/23/12 Mon 1/30/12 2
11 Revised version for CDR 19 days Tue 1/31/12 Fri 2/24/12 10
12 Final version for Design Expo 48 days Mon 2/27/12 Wed 5/2/12 11
13 System Engineering 28 days Mon 1/30/12 W ed 3/7/12
14 Kinect Design 21 days Mon 1/30/12 Mon 2/27/12 4
15 N64 Signal Design 21 days Mon 1/30/12 Mon 2/27/12 4
16 Ethernet Design 21 days Mon 1/30/12 Mon 2/27/12 4
17 RCA Design 28 days Mon 1/30/12 Wed 3/7/12 4
18 Integration 37 days Tue 2/28/12 W ed 4/18/12
19 Kinect/N64 Integration 14 days Tue 3/6/12 Wed 4/4/12 15
20 N64 to Server Integration 14 days Tue 2/28/12 Fri 3/16/12 15
21 Server/RCA Integration 14 days Tue 2/28/12 Fri 3/16/12 15
22 PCB Review 30 days Thu 3/8/12 Wed 4/18/12 13
23 Beta Testing 11 days Thu 4/19/12 Thu 5/3/12 22
24
25 Initial Design Objectives Due 1 day Mon 1/23/12 Mon 1/23/12 2
26 Preliminary Design Review 1 day Tue 1/31/12 Tue 1/31/12 10,7,2
27 Critical Design Review 1 day Mon 2/27/12 Mon 2/27/12 11,8
28 Milestone 1 1 day Tue 3/20/12 Tue 3/20/12 13
29 Milestone 2 1 day Tue 4/10/12 Tue 4/10/12 21
30 Technical Reference Manual 73 days Mon 1/23/12 Wed 5/2/12
31 Capstone Expo 1 day Thu 5/3/12 Thu 5/3/12 23,12,18,13,3
11 14 17 20 23 26 29 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 30 3 6 9December 2011 January 2012 February 2012 March 2012 April 2012 May 2012
Project Schedule
Longest Design Period
11 Days for Testing
Technical Reference Manual
37 Days for Integration
Division of LaborTask Adrian Evan John Alex Darnell Craig
Program Mgmt & Documentation
P S S
Kinect Programming S P S
N64 Translation S P S
Ethernet Interface P P
Controller Programming S S P
Audio/Video Signal Conversion
S P S
N64 Gameplay Action P P P P P P
P = Primary S = Secondary
Advanced Parts ListHardware Cost Quantity Extended Cost Possible Substitutions
Television Provided 2 ProvidedXBOX Kinect Provided 2 ProvidedNintendo 64 Provided 2 ProvidedNintendo 64 Controller Provided 3 ProvidedNintendo 64 Controller $19 4 $76Nintendo 64 Controller Cable Extension $10 2 $20Nintendo 64 ROMS Provided 7 ProvidedNetduino $60 2 $120Beagle Bone $90 3 $270 Beagle BoardModem Provided 1 ProvidedParts for Printed Circuit Board $50 1 $50XBOX Kinect / Nintendo 64 Shield $14 2 $28RCA Cable Provided 1 Provided Coax CableCase $10 1 $10PCB Development / Revisions $500 TBD $500
TOTAL ESTIMATED PROJECT COST = $1,074
*NOTE* – ALL EXPENSES PAID TRIP TO LAS VEGAS FOR “ELECTRONICS SEMINAR” NOT INCLUDED
GoalsPrimary:• Control Nintendo 64 game exclusively via an XBOX Kinect
Secondary:• Have one N64 controller communicate with one N64
console over the internet• Control the N64 game using either the Kinect or the N64
controller remotely
Tertiary:• Play a complete N64 game with 2 players at one location
using the Kinect, and the other 2 players at a different location using N64 controllers
• Allow functionality with a variety of different controllers
Milestone 1 Objectives
Kinect Translation • Kinect translator sends simple gesture data to
the Linux µController, in correct format
Nintendo 64• Send properly timed Nintendo 64 analog
signal from the µController to the N64 console
Network Configuration• Send data packets across the network to and
from each user using the µController
Audio/Video• Convert analog input into a digital code
Milestone 2 Objectives
Kinect Determine ideal Kinect gestures for
satisfactory user interface Transmit verbal commands through Kinect
Network Configuration Send and receive N64 and audio/video
signals across server
Audio/Video Output signal from server to television
Risks and Contingency Plan
Risks Syncing the N64 timing sequence with the Netduino
microcontroller’s internal timer Busy internet path/lost data packets Delay between Kinect / N64 input and television
output Data corruption or potential injury resulting from
falling or player collisions while playing with a motion sensitive controller
Solutions Use only N64 controllers (no Kinect) for game control Leave out server / direct connection from
microprocessor to N64 User plays video game at own risk!!
Questions???