Greener Collaboration
California Institute for Telecommunications and Information Technology: Calit2
•! Over 1000 Researchers in Two Buildings Linked via Dedicated Optical Networks
•! International Science, Art and Engineering via Telecommunications and Information Technology
•! Nanotechnology to Wireless, Microbial Genomics to Earth Sciences, Virtual Reality to 4K Digital Cinema and HD videoconferencing
UC Irvine
Preparing for a World in Which Distance is Eliminated… And You Don’t Need to Take off Your Shoes and Belt
UC San Diego
Tom’s Office
Outline--in the Past 15 years
•! Network Engineers have made 10G easy
–! Globally
–! Regionally
–! Locally
•! Computer Engineers have made clusters easy
–! At Universities and Companies
–! In Clouds
•! Visualization Engineers have made graphics easy
–! 2D
–! 3D
–! HD and 4K
•! But Collaboration is still hard!
•! And much more energy is being used by ICT > Airlines
•! So, we need to build Greener Collaboration
Our Vision for the Next Decade--Multiple Rooms for Greener Collaboration
Graphic: Jason Leigh, EVL
Augmented Reality AutoStereo--No Glasses
4K Streaming Video
Gigapixel Seamless Tiled Displays
Includes High-Quality Directional Audio with Echo Cancellation and Session Archiving over multiple 10G Networks
Today: Cisco Telepresence Provides Leading Edge Commercial HD Video Teleconferencing to its Offices
•! 191 Cisco TelePresence in Major Cities Globally
–!US/Canada: 83 CTS 3000, 46 CTS 1000
–!APAC: 17 CTS 3000, 4 CTS 1000
–!Japan: 4 CTS 3000, 2 CTS 1000
–!Europe: 22 CTS 3000, 10 CTS 1000
–!Emerging: 3 CTS 3000
•! Overall Average Utilization is 45%
!! 85,854 TelePresence Meetings Scheduled to Date
!! Weekly Average is 2,263 Meetings
!! 108,736 Hours
!! Average is 1.25 Hours
!! 13,450 Meetings Avoided Travel Average to Date (Based on 8 Participants)
~$107.60 M To Date
!! Cubic Meters of Emissions Saved 16,039,052 (6,775 Cars off the Road)
Source: Cisco 3/22/08
Uses QoS Over Shared Internet ~ 15 mbps
Green Results
Today: Global Lambda Integrated Facility 1 to 10G Dedicated Lambda Infrastructure
Source: Maxine Brown, UIC and Robert Patterson, NCSA
Interconnects Global Researchers
The Role of Visualization in Science, Society & Education
•! Vision and hearing are your dominant senses for the acquisition of information from the everyday world
•! More than 50% of your brain is used for processing visuals and audio
•! Visualization has 5 main roles in society:
–!To use as an instrument to view and understand complex sensor and collected data (like a microscope)
–!To validate algorithmic results (like a computer simulation)
–!To explain these complex results to students, government policy makers, and the general public
–!To inspire the next generation of scientists and engineers
–!To collaborate over distance
10 years of Bringing Scalable Visualization to the Users
ORNL 35Mpixel EVEREST
2004 1999
LLNL 20 Mpixel Wall NCSA 4 MPixel
NSF Alliance PowerWall
TACC 307 Mpixel Stallion NSF TeraGrid
1997 1999
2004 2005
Calit2@UCI 200 Mpixel HiPerWall NSF MRI
EVL 100 Mpixel LambdaVision NSF MRI
2008
Two Orders of Magnitude Growth (Same as 100mb/s-->10Gb/s!
Collaborating with Photons: North American OptIPuter Team OptIPortals
Calit2@UCI
NCMIR-UCSD
Calit2@UCSD Calit2@UCSD Calit2@UCSD
EVL@UIC
SIO-UCSD
Texas A&M
USGS
Calit2@UCSD
VISIT-U Michigan
Calit2@UCSD
iCAIR- Northwestern U
EVL@UIC
EVL@UIC
OptIPuter Scalable Displays Are Used for Multi-Scale Biomedical Imaging
Green: Purkinje Cells Red: Glial Cells Light Blue: Nuclear DNA
Source: Mark
Ellisman, David Lee,
Jason Leigh
Two-Photon Laser Confocal Microscope Montage of 40x36=1440 Images in 3 Channels of a Mid-Sagittal Section
of Rat Cerebellum Acquired Over an 8-hour Period
200 Megapixels!
Scalable Displays Allow Both Global Content and Fine Detail
Allows for Interactive Zooming from Cerebellum to Individual Neurons
Very Large Images Can be Viewed Using CGLX’s TiffViewer
Hubble Space Telescope (Optical)
Spitzer Space Telescope (Infrared)
Source: Falko Kuester, Calit2@UCSD
Using HIPerWall OptIPortals for Humanities and Social Sciences
Software Studies Initiative, Calti2@UCSD
Interface Designs for Cultural Analytics
Research Environment
Jeremy Douglass (top) & Lev Manovich
(bottom)
Second Annual Meeting of the
Humanities, Arts, Science, and Technology Advanced
Collaboratory (HASTAC II)
UC Irvine May 23, 2008
Calit2@UCI 200 Mpixel HIPerWall
Power Consumption: Calit2’s 285 Megapixel 25kW? HiPerSpace OptIPortal
The Calit2 OptIPortals at UCSD and UCI Are Now a 50kW? Gbit/s HD Collaboratory
Calit2@ UCSD wall
Calit2@ UCI wall
NASA Ames Visit Feb. 29, 2008
HiPerVerse: First ! Gigapixel Distributed
OptIPortal- 124 Tiles
Sept. 15, 2008
UCSD cluster: 15 x Quad core Dell XPS with Dual nVIDIA 5600s UCI cluster: 25 x Dual Core Apple G5
On-Line Resources Help You Build Your Own OptIPortal
www.optiputer.net
http://wiki.optiputer.net/optiportal
http://vis.ucsd.edu/~cglx/
www.evl.uic.edu/cavern/sage
More North American OptIPortals
SDSC - UCSD
Adler Planetarium - Chicago
NASA Goddard
U Washington TRECC, Chicago AOSS-U Michigan
DMC-U Michigan
TACC TeraGrid
Even More North American OptIPortals
CICESE, Mexico
Nortel
Florida International U
Sharp Corp. Rincon
Global Lambda Integrated Facility Europe
European OptIPortals
Space Research Institute-Moscow
SARA- Netherlands
Masaryk U- Czech Republic
Zurich, Switzerland
Science Innovation Center Chernogolovka, Russia
Global Lambda Integrated Facility Australia / New Zealand
Australian / New Zealand OptIPortals
U Wellington, NZ
ANU
Monash U
AARNet
CSIRO
U Queensland
U Melbourne
Global Lambda Integrated Facility Northeast Asia
Northeast Asia OptIPortals
KISTI-Korea
AIST-Japan Osaka U-Japan
NCHC-Taiwan
Kyoto-Japan
GIST-Korea Chinese Academy of Sciences
CNIC-China
The StarCAVE as a 25kW? Browser for the Protein Data Bank at UCSD
Calit2’s StarCAVE -25kW?
(25kW is $2.5/hr, $60/day, less than renting a car)
Source: Tom DeFanti, Greg Dawe, Calit2
Connected at 2x10Gb/s to CENIC, NLR, GLIF 5x100Gb/s to local servers 10G Myrinet to all 17 graphics nodes
34 1080p HD Projectors
Passive Polarization-- Optimized the
Polarization Separation and Minimized Attenuation
~10000x3400 pixels/eye
StarCAVE with Floor from Above
The StarCAVE as a 25kW? Browser for the Protein Data Bank at UCSD
Source: Tom DeFanti, Jurgen Schulze, Bob Kooima, Calit2/EVL
•! Next step up: KAUST-Saudi Arabia
•! 24 Sony 4K projectors 100 Million Pixels/eye
•! 240,000 lumens!
•! ~ 75kW?
•! Mechdyne/Iowa State Design based on original 1991 EVL CAVE
•! 10G connected to GLIF!
10m
3m
Calit2/EVL Varrier --~20kW 60 Screen Stereo OptIPortal, no Glasses Needed
Dan Sandin, Greg Dawe, Tom Peterka, Tom DeFanti, Jason Leigh, Jinghua Ge, Javier
Girado, Bob Kooima, Todd Margolis, Lance Long, Alan Verlo, Maxine Brown, Jurgen Schulze, Qian Liu, Ian Kaufman, Bryan Glogowski
Mars Rendered at 46,000 x 23,000 pixels
~10000x4000 pixels/eye
Lenticular 3D Autostereo Alioscopy Display 400w per tile?
New work on Tiled Autostereo displays made by Alioscopy, Inc.
Dan Sandin, Bob Kooima, Andrew
Prudhomme, Tom DeFanti
12kW?
~3600x3000 pixels/eye
Tiled Micropolarized (Xpol) 3D JVC Panels (w/Glasses)
Source: NHK Media Technology, Inc.
21-Panel Xpol LCD Stereo CAVE 9kW?
14000x1500 pixels/eye
Visualization Part for the Next Decade is Close But, How do We Collaborate?
Source: Jason Leigh, EVL
4K Streaming Video
Keio/Calit2 Collaboration: Trans-Pacific 4K Teleconference September 2005
Keio University President Anzai
UCSD Chancellor Fox
Used 1Gbps
Dedicated ----
NTT Sony
SGI Olympus
Mitsubishi Yamaha Toppan
Live 4K Video via JPEG2000 (Sender)
4K motion picture camera
4 x HD-SDI
ADAT (to HDSP 9652)
4K JPEG2000 Encoder
HD-SDI
SW 1-10 Gb/s
JPEG2000 Board
Can also send
uncompressed 4K
Source: Dr. Ohta, Keio University
Live 4K Video via JPEG2000 (Receiver)
ADAT from HDSP 9652
4K JPEG2000 Decoder
JPEG2000 Board
1-10 Gb/s
4K Auditorium
4K Projector
Source: Dr. Ohta, Keio University
Everyday 4K Pre-encoded File Streaming
1Gb/s E
ADAT form HDSP 9652 Encoded File
WAV File
Streaming Server 4K JPEG2000 Decoder
Source: Dr. Ohta, Keio University
CineGrid@AES: October 2006
Keio DMC
Tokyo
CineGrid
International
Networks
LDAC Premiere Theater
UCSD San Diego USC LA
Sync
NTT JPEG2000 Servers
Sony 4K
Audio
CineGrid
California
Networks
ProTools Audio Server
Yamaha Mixers
Sync
DVTS Sony DV
NTT JPEG2000
CODEC and Server
Olympus 4K Camera
CineGrid @ AES 2006 Keio Wagner Society String Ensemble
Holland Fest (June 20-22, 2007) on CineGrid
•! “ERA LA NOTTE” Star soprano Anna Maria Antonacci sang solo madrigals
•! 4K transmission
–! JPEG2000 Compressed (500Mb/s) via IRNC/C(ON)2/CAVEwave to Calit2 on Wednesday
–! Uncompressed via IRNC/JGN2 to Keio on Friday (8bGb/s)
CineGrid @ GLIF 2007 September 2007
Source: Maxine Brown, OptIPuter Project Manager
Can Optical Fiber Replace Airline Travel for Greening
Collaborations?
The OptIPuter Creates an OptIPlatform: Enabling 4K Video & Data-Intensive Collaboration
www.evl.uic.edu/cavern/sage
“OptIPlanet: The OptIPuter
Global Collaboratory” – Special Section of
Future Generations Computer Systems, Volume 25, Issue 2, February 2009
Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST
Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
EVL’s SAGE International GLIF Telepresence Infrastructure
EVL Chicago
SARA Amsterdam
Masaryk University
Brno
Russian Academy Sciences Moscow
Calit2 UCSD
AIST Tokyo
Osaka University
Osaka
GIST Korea
KISTI Korea
CNIC China
NCHC Taiwan
USGS Sioux Falls
University Michigan
Ann Arbor
NCSA & TRECC Urbana
www.evl.uic.edu/cavern/optiplanet
U Queensland, Australia
U Osaka, Japan
Masaryk U, Czech
Republc
Space Research Institute, Moscow
Bandwidth Challenge: International Sites Connected via SAGE November 2008
Display clients equipped with
HD video cameras
There Are Now 36 High Resolution Scalable Displays Using EVL’s SAGE Around the World
EVL SC08 Bandwidth Challenge Bandwidth per Site for Tuesday November 18, 2008
Gb/s
18:32:17 18:59:58
Global Visualcasting: Collaborative Remote Visualization Over High Speed Networks
Source: Jason Leigh and Luc Renambot
Terabit LAN Latency Compensation needed for N-Way SAGE 4K and Data Collaboration!!
4K Uncompressed Streaming from NTT Yokosuka to Calit2 February 2009
4K Uncompressed Streaming from NTT Yokosuka to Calit2 February 2009!
•! Two VLANs in two different 10G physical routes!
!"!
Yokosuka! Calit2!
VLAN 2132
VLAN 3790
VLAN 2132
VLAN 3790
Shorter route (via JGN2)
Longer route (via IEEAF)
Time difference is about 150 ms.
VLAN 2132
RTT=119 ms
VLAN 3790
RTT= 420 ms
Telepresence for the Next Decade is Close Too! But how much Power?
Source: Jason Leigh, EVL
Augmented Reality No Glasses
4K Streaming Video
Gigapixel Wall Paper
1 GigaPixel x 3 bytes/pixel @ 30fps = 1 Terabit/sec
~125kW? display + 200W? per router hop @10G+Transmission kW?
Shipping in 1Q2008 Juniper M320
Cisco 12416
Juniper MX240
Cisco 7604
Slot Capacity, Gpbs 20 10 (SIP-601) 40 20 (ES-20)
System Capacity 320 Gbps 280 Gbps 240 Gbps 80 Gbps
Rack Space (height) 1/2 rack Full rack 8.71” 8.75”
Max System Draw 2.78 KW 5.58 KW 1.42 KW 1.30 KW
Energy Efficiency (EER) 58 Gbps/KW 25 Gbps/KW 85 Gbps/KW 31 Gbps/KW
ECR, Watts/10Gbps 173.75 398.57 118.33 325.00
Feature Set Multiservice edge Carrier Ethernet
How many Watts per 10Gb/s?
Juniper Data from 2008
The Planet is Already Committed to a Dangerous Level of Warming
Temperature Threshold Range that Initiates the Climate-Tipping
V. Ramanathan and Y. Feng, Scripps Institution of Oceanography, UCSD September 23, 2008
http://www.pnas.org/content/105/38/14245.abstract!
Additional Warming over 1750 Level
90% of the Additional 1.6 Degree Warming Will Occur in the 21st
Century
The Intergovernmental Panel on Climate Change (IPCC)
Recommends a 25-40% Reduction Below 1990 Levels by 2020
•! On September 27, 2006, California Governor Schwarzenegger signed the California Global Warming Solutions Act of 2006 –! Requires Reduction of GHG to 1990 Levels by 2020
–! 15% Reduction from 2008 Levels
–! 4 Tons of CO2-equiv. for Every Person in California
•! The European Union Requires Reduction of GHG by 2020 to 20% Below 1990 Levels (12/12/2008) (Amsterdam by 40%!)
•! Australia has Pledged to Cut by 2020 its GHG Emissions 5% from 2000 Levels via the World's Broadest Cap &Trade Scheme (12/15/08) [~5% Below 1990 Levels]
•! Neither the U.S. or Canada has an Official Target Yet
–! President Obama Has Endorsed the California 2020 Goal
www.ipcc.ch
ICT is a Critical Element in Achieving Greenhouse Gas Emission Reduction Targets
Applications of ICT could enable emissions reductions
of 7.8 Gt CO2e in 2020, or 15% of business as usual emissions.
But it must keep its own growing footprint in check and overcome a number of hurdles
if it expects to deliver on this potential.
www.smart2020.org
The Global ICT Carbon Footprint Roughly the Same as the Aviation Industry Today
www.smart2020.org
But ICT is Growing at 6% Annually!
ICT Industry is Already Acting to Reduce Carbon Footprint
Electricity Usage by U.S. Data Centers: Emission Reductions are Underway
Source: Silicon Valley Leadership Group Report July 29, 2008 https://microsite.accenture.com/svlgreport/Documents/pdf/SVLG_Report.pdf
Universities Need to Lead Too!
•! Compute energy/rack : 2 kW (2000) to 30kW (in 2010)
•! Cooling and power issues now a major factor in campus clusters
•! But academic clusters are often small: departmental closets
•! Energy use of departmental computers is increasing fast
creating crises of space, power, and cooling
•! Unfortunately, almost nothing is known about how to make
these shared virtual clusters energy efficient, since there has
been no financial motivation to do so"
The GreenLight Project: Instrumenting the Energy Cost of Cluster Computing
•! Focus on 5 Communities with At-Scale Computing Needs:
–! Metagenomics
–! Ocean Observing
–! Microscopy
–! Bioinformatics
–! Digital Media
•! Measure, Monitor, & Web Publish Real-Time Sensor Outputs
–! Via Service-oriented Architectures
–! Allow Researchers Anywhere To Study Computing Energy Cost
–! Enable Scientists To Explore Tactics For Maximizing Work/Watt
•! Develop Middleware that Automates Optimal Choice of Compute/RAM Power Strategies for Desired Greenness
Earthquake Testing
UCSD Structural Engineering Dept. Conducted Tests May 2007
Answer the Threat to Cluster Deployment with Facts on Work/Watt
•! Computer Architecture –! Rajesh Gupta/CSE
•! Software Architecture –! Amin Vahdat, Ingolf Kruger/CSE
•! CineGrid Exchange –! Tom DeFanti/Calit2
•! Visualization –! Falko Kuster/Structural Engineering
•! Power and Thermal Management –! Tajana Rosing/CSE
•! Analyzing Power Consumption Data –! Jim Hollan/Cog Sci
•! Direct DC Datacenters –! Tom DeFanti, Greg Hidley http://greenlight.calit2.net
MRI
Example: Improve Mass Spectrometry’s Green Efficiency By Matching Algorithms to Specialized Processors
•! Inspect application implements the very computationally intense MS-Alignment algorithm for discovery of unanticipated rare or uncharacterized Post-Translational Modifications
•! Solution: Hardware acceleration with a FPGA-based co-Processor (Convey Architecture)
•! Results:
–! 300x Speedup with hand FPGA coding
–! Increase in work/watt?
–! Increase in work/$ (purchase, life-cycle)?
Zero Carbon GreenLight Experiment: Direct DC-Powered Modular Data Center
•! Concept—Avoid DC to AC to DC Conversion Losses
–! Computers Use DC Power Internally
–! Solar and Fuel Cells Produce DC
–! Both Plug into the AC Power Grid
–! Can We Use DC Directly?
•! DC Generation Can Be Intermittent
–! Depends on Source
–!Solar, Wind, Fuel Cell, Hydro
–! Can Use Sensors to Shut Down or Sleep Computers
–! Can Use Virtualization to Halt/Shift Jobs
•! Experiment Planning Just Starting
–! Collaboration with Sun and LBNL
–! NSF GreenLight Year 2 and Year 3 Funds
Source: Tom DeFanti, Calit2; GreenLight PI
DC Fuel Cell 2800kWatts Sun Box <200kWatt
DC Power--UCSD is Installing Zero Carbon Emission Solar and Fuel Cell DC Electricity Generators
San Diego’s Point Loma Wastewater Treatment Plant Produces Waste Methane
UCSD 2.8 Megawatt Fuel Cell Power Plant Uses Methane
2 Megawatts of Solar Power Cells
Being Installed
Available 2010
UCSD is Studying a Global Demonstration Project for Sea Water Cooling
•! UCSD is uniquely located to use cold seawater from one of only 40 deep shoreline sites in the world. It can supply cold water essential for air conditioning laboratories and computer rooms
•! Initial study of La Jolla underwater trench suggests a seawater cooling system could produce savings of $4M/yr and 100 million gallons of fresh water per year.
Toward “Zero Carbon” ICT Green Cloud Computing and Storage
•! Purchasing Green Power Locally is Expensive with Significant Transmission Line Losses
–! Demand for Green Power Within Cities is Growing Dramatically
–! ICT Facilities Don’t Need To Be Located In Cities
•! But Most Renewable Energy Sites are Very Remote and Impractical to Connect to Electrical Grid
–! Can be Easily Reached by an Optical Network
–! Provide Independence from Electrical Utility
–! Savings in Transmission Line Losses (Up To 15% Alone)
–! Plus Carbon Offsets Can Pay for Moving ICT Facilities to Renewable Energy Site
•! Calit2 is Discussing Partnering with Canada
–! Move a GreenLight Facility to Hydro Site in British Columbia,
–! Link by 10Gbps Optical Fiber to Calit2—Offer to Remote Users
Source: Bill St. Arnaud, CANARIE, Canada
More: Application of ICT Can Lead to a 5-Fold Greater Decrease in GHGs Than its Own Carbon Footprint
Major Opportunities for the United States*
–! Smart Electrical Grids
–! Smart Transportation Systems
–! Smart Buildings
–! Virtual Meetings
* Smart 2020 United States Report Addendum
www.smart2020.org
While the ICT sector plans to significantly step up the energy efficiency of its products and services,
ICT’s largest influence will be by enabling energy efficiencies in other sectors, an opportunity
that could deliver carbon savings five times larger than the total emissions from the entire ICT sector in 2020.
--Smart 2020 Report
Conclusion: Vision for the Next Decade is Very Close; Advanced Communcations Can Improve International
Collaboration and Reduce Carbon Emissions!
Source: Jason Leigh, EVL
Is this the Killer Application?
To be Energy Efficient, We Must Think about Koala-Style Computing and other “Smart” Ways
Size Your Brain Power, Visualization, Storage, Sleep Cycles, and Communications to Your Problem
Thank You Very Much!
•! Our planning, research, and education efforts are made possible, in major
part, by funding from:
–! US National Science Foundation (NSF) awards ANI-0225642, EIA-0115809,
SCI-0441094, and CNS 0821155
–! State of California, Calit2 UCSD Division
–! State of Illinois I-WIRE Program, and major UIC cost sharing
•! University of Illinois at Chicago, Argonne National Laboratory, and
Northwestern University for StarLight networking and management
•! National Lambda Rail, Pacific Wave and CENIC
•! NTT Network Innovations Lab
•! Cisco Systems, Inc.
•! Pacific Interface, Inc.
•! Darkstrand, Inc.
•! KAUST-US
•! Sharp Labs of America