Your Results From Disk to Device(Preview)
Amit ChourasiaMona Wong-BarnumMichael L. Norman
San Diego Supercomputer Center, UCSD
Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013
Stream Encode Explore Disseminate My Experiments
Computation Cycle
Compute/IngestPost ProcessAnalyze & Save
Monitor?
Validate?
Abstract?
Derived Products
Monitoring
ValidationAbstraction
Direct Access Direct Sharing
Not All Members Have HPC
Access
Security Implications
On HPC
Derived Products
Concise derived products• Plots & Text Documents• Image Sequences & Videos
Share with the research group1. Download derived products2. Share with collaborators
a. Email snippets b. Set up a webpage
– Add content– Email
Derived Visual Content Survey
Derived content for a single research group in respective domain
Pitfalls in sharing derived content
Download Upload
Round Trip
Video Encoding
Complexity Web Setup Process
Duplication Scalability
Direct AccessNot All Members Have HPC Access
Direct SharingSecurity
Implications On HPC
Gaps
Download Upload
Video Encoding Web Setup Process
Duplication Scalability
Direct Transfer Easy Sharing Reusable Ubiquitous
SeedMe.org
Why not use existing tools?Features SeedMe YouTube & Vimeo FlickrSharing Public, Group, Private Public, Private Public
Upload Method Command Line, API, Web API, Web API, Web
Meta Data Rich Limited Limited
Download Support Yes Vimeo only No
Copyright Unencumbered Encumbered Encumbered
Still Images Yes No Yes
Periodic Image Update Yes No No
Image Sequence to Video Yes No No
Frame Rate Arbitrary 29.97 fps Unknown
Video Resolution Arbitrary Upto 4K Upto HD
Desirable Limiting Crippling
SeedMe Architecture
Ingestion (Atomic or Recurring)
Instant Processing
(Text, Images)
NotificationResponsive
Content Access
Batch Processing
(Videos)
SeedMe Hardware
Ingest Temp
Storage (10TB) Encode Cluster (16 nodes) :
Apple Compressor + ffmpeg
Web Cluster (2-8 nodes) : Drupal
Mirrored Storage (50 TB) : ZFS
Desktops
Phones
Tablets
20 gbe
20 gbe
20 gbe
Ingestion (Atomic or Recurring)
Instant Processing
(Text, Images)
NotificationResponsive
Content Access
Batch Processing
(Videos)
Compressor Encoding Benchmark
SeedMe Collection
Ticker* Plots Image Sequences* Videos
* Recurring: periodic update
Meta Data
Access: Public, Group, Private
Collaborators
Create New CollectionSet Access Control (Public, Group, Private)
Specify collaborators* and meta data
Upload/Update contenthttp/https, ftp, Globus Online
Interactively on Web, Command Line or REST Services
View Collection on Web
SeedMe: Interaction Process
SeedMe: Interaction Modality
• Command Line Interface (using curl)• Web Interface (WYSIWYG)• RESTful API (integrate your tool chain)– JSON
Content
Videos
Permissions
Plot Files
Doc Files
Authentication
Text Ticker
Image Sequences
Meta Data
{ "email": ”[email protected]", "api_key": "XXXXXXXXXXX", ”view_permission": "public", ”viewer_emails": ["[email protected]", "[email protected]"], "title": "simulation XYZ", "description": "input conditions", "credits": "Visservices group, SDSC", "key_values": { “key1”: "v1”, "key2": "v2”}, "expire_date": "2013-05-01", "tags": [ "x", "y", "z"],
"ticker": ["time step 100", "time step 200"], "files":{ ”docs" : ["a.pdf", ”b.txt"],
"plots" : ["a.png", "b.jpg", "c.tiff"], "image_sequences" : { "ash": ["ash1.jpg", "ash2.jpg", "ash3.jpg"], "air": ["air1.jpg", "air2.jpg", "air3.jpg"] }, "movies": ["m1.mov", "m2.mov »] }}
Application Scenarios
• Monitor progress with text tickers and images• Encode and stream videos from image
sequences• Share & collect visualization results• Use the collections for education and
outreach
Content accessible on most devices
Work in progress
How it Works Examples #1 & #2
Periodic update at simulation run time from a Magneto Hydrodynamics SimulationCourtesy of Mahidhar Tatineni and Homa Karimabadi, UCSD
Future Work
• Production SeedMe Service• Example Scripts & Documentation• Globus integration with SeedMe
Acknowledgements
National Science FoundationThis material is based upon work supported by the National Science
Foundation under Grant No. OCI-1235505
Andrew Ferbert, Michael Dwyer, Kristen Levy & Doug Weimer, SDSC : Hardware/software setup
Mahidhar Tatineni, SDSC & Homa Karimabadi, UCSD: in-situ testsApple Inc. : Provided test hardware/software on loan during project
conceptualization phase
Sign up InvitationFor notification and early access
(we won’t spam)
Contact: amit @ sdsc.edu
Persistence?
Results
Move to own
resource
Move to XSEDE
resource
Move to YouTube/Flickr, Vimeo
Move to Dropbox/C
loud
Freemium
PURGE