TelePresence Microscopy at CSIRO Minerals
Colin MacRae
CSIRO Minerals - Microbeam Laboratorywww.minerals.csiro.au/microbeamlab
Remote Access and automation Workshop 2005
Talk outline
What and why? History of TelePresence at CSIRO Minerals 3rd generation TelePresence system
System architecture Control interfaces Video streaming technology Security features Administration Modular installation of new functions
Conclusions, where to from here?
Remote Access and automation Workshop 2005
What is TelePresence? Tele-conferencing
Video + Audio
Tele-control Microscope controls On-line data collection + analysis Tele-control is provided by some
manufacturers
Ideally, TelePresence should be: Interoperable with many platforms ‘Shrink-wrapped’ software
• Looks professional, with easy to use interfaces
Fast and responsive Stable and secure
Remote Access and automation Workshop 2005
Why do we want TelePresence? Internal clients (Brisbane, Hobart, Perth) External clients (Melbourne, Sydney, Brisbane) For analyses, our clients must either:
Mail / courier samples, and wait for our report (slow) Travel to our laboratory (expensive)
TelePresence can help Provide results to client in real-time, and also allows instant
feedback from client Attract new clients
CSIRO
Marine Research
CSIRO
Minerals
Comalco & RioTinto
CSIRO
Minerals, Exploration & MiningBHP-Billiton
Remote Access and automation Workshop 2005
What do we want to connect?
Scanning Electron Microscopes Electron Microprobes & Proton Microprobe X-ray detectors –EDS & WDS Cathodoluminescence systems Optical microscopes
Remote Access and automation Workshop 2005
What do we want to offer
Clear fast images On-line microanalysis On-line x-ray spectra Large area optical views Streaming maps Security Voice over IP Data storage Data
analysis/presentation software
Remote Access and automation Workshop 2005
What do we need from TelePresence?
Provision of: Real-time video from multiple sources (SEM, room camera) Streaming EDS spectra and other data ( + controls) Control of microscope stage + column (lower priority)
Minimal hardware requirementsPC or laptop (no workstations)ADSL or cable modem, ~1Mbit/s (not Internet2, GRID)
‘Shrink-wrap’ softwareEasy to set-upVery easy to learn and use, with generic controls/interfaceVery, very stableExtraordinary securityLow maintenance
Must not affect normal operation of instruments!
Remote Access and automation Workshop 2005
TelePresence history
TelePresence Microscopy 1999 Nestor J. Zaluzec, Argonne National
Laboratory Web-based, ‘pervasive’ Web based notebook Video with “GTS” streaming
technology
Remote Access and automation Workshop 2005
TelePresence history
TelePresence Microscopy 1999 Nestor J. Zaluzec, Argonne National
Laboratory Web-based, ‘pervasive’
Java system 2001 Platform independent, ‘safe’ code MJPEG video (TCP-IP) Web based notebook with
embedded Java applets for Images, Spectra and maps
Successful, but limited
Remote Access and automation Workshop 2005
TelePresence historyTelePresence Microscopy - 1999
Nestor J. Zaluzec, Argonne National Laboratory
Web-based, ‘pervasive’
Java system - 2001 Platform independent, ‘safe’ code MJPEG video Web based notebook with embedded
Java applets for Images, Spectra and maps
Successful, but limited
C++ system – 2004 Widely used language, with broad
library support MPEG-4 video compression in software,
changeable video size New, secure architecture
Remote Access and automation Workshop 2005
Internet protocols: UDP versus TCP
RouterTCP packet
Sender
acknowledgement
Lost packet?
UDP packet
TCP
UDP
TCP = Transport Control Protocol
UDP = User Datagram Protocol
Remote Access and automation Workshop 2005
Video latency Definition:
Latency = time displayed – time captured What do we want?
<500ms for focussing (or better) <200ms preferred
Causes of latency1. Video compression / decompression
• Hardware accelerated rendering (done)• Hardware compression (cheap MPEG-4 hardware compression
coming soon)
2. Router delays (congestion)• Ultra-broadband internet (eg GRID)• Reduce stream size (better codec)
3. inter-router delays (speed / distance)• More bandwidth wont help• Use faster internet protocol (i.e. TCP UDP)
Remote Access and automation Workshop 2005
Another real life problem : The Firewall
Client
Host
Server
‘DMZ’
Client network
Internet
Video
Our networkFirewall Firewall
port 8080 (UDP), from specified addresses
port 8080 (TCP)
Remote Access and automation Workshop 2005
Comparison of platforms
Web browser
Java C++
Trusted? Somewhat Yes Yes, with security
Platform independent?
Yes Yes No
Functionality HighModerateLow
Execution speed HighHighLow
Development pain ModerateHighLow
The vibe Yeah groovySo soPoor
Remote Access and automation Workshop 2005
Password
System architecture
Client
Server
Drivers
Host
Video
(UDP stream)
Server
Server
commands & data
(TCP socket)
Instrument specific
Generic interface
JEOL 8900RElectron Microprobe
FEI QuantaFEG-Environmental
SEM
Hitachi S-5000In-Lens Cold FEG
SEM
PIXEProton Microprobe
Remote Access and automation Workshop 2005
DirectShow video streaming
‘GraphEdit’ component of Microsoft DirectX software development kit (SDK)
Video capture
Timing system
(frame-rate)
Compression codec
Network transmission
sockets
Remote Access and automation Workshop 2005
DirectShow video streaming Very easy to configure and re-configure ‘graphs’ Other stream designs are possible, eg:
Saving stream to HDD Motion detection triggering Adaptable to as yet unforseen requirements
All components are upgradeable and replaceable New codecs can be adopted as they become available ‘Future proof’
We didn’t have to write any of these components (only wire them together)
Some DirectShow components are hardware-accelerated
Fast video rendering
Remote Access and automation Workshop 2005
TelePresence interfaces
Video Camera control
Tilt-pan-zoom controlsUsed in conjunction with video
window
EDX spectrometersGeneric, easy to use interface Fast (2-5 updates/s)
Text streams Miscellaneous others Stage and Electron
optical System
Remote Access and automation Workshop 2005
Video streaming technology
Microsoft DirectShow Widely supported standard
with pre-compiled modules Hardware-accelerated
rendering
MPEG-4 encoding 320 240 at 25 fps is
~500kbit/s (~ADSL) (~12 Mbit/s by MJPEG) (~59 Mbit/s for raw video)
UDP transmission Lowest possible latency over
IP (~200ms)
Themeda Australis
Kangaroo grass
Variations in humidity cause grass to spiral and “dig” into the soil
Remote Access and automation Workshop 2005
Video cameras
Canon VC-C1 and VC-C3 cameras
450 x 350 pixels at 30 frames/s.
±90º pan, ±25º tilt, 10 zoom
Good low-light performance
Remote Access and automation Workshop 2005
Bench-top camera
‘ELMO’ camera XGA video, 20 frames/s www.elmousa.com/presentation
Remote Access and automation Workshop 2005
Client feedback
Problem: Video stream is one-way. How do clients tell us what they want to see?
Verbal feedback (telephone) “that thing, over to the left. Up,
say, half an inch…” Not very effective
Graphical feedback Video overlay marker Visible to all other clients and the
operator More tools to follow
Remote Access and automation Workshop 2005
Security Firewalls
Blocks all direct communication with servers and drivers
Host and servers are in separate ‘DMZ’ network, not mounted on CSIRO file system
Socket encryption: Sockets are wrapped with ‘Rijndael’ cipher,
NIST advanced encryption standard (AES 2001).
• Certified for all levels of classification, up to and including ‘top-secret’.
Hackers must break AES to read data, send false data, or issue commands
Remote Access and automation Workshop 2005
On-line help
Remote Access and automation Workshop 2005
Administration
Remote Access and automation Workshop 2005
Web based administration
Remote Access and automation Workshop 2005
Software based Administration tools
Remote Access and automation Workshop 2005
Administration tools – Modular
New instruments can be added and
configured
Remote Access and automation Workshop 2005
Summary
TelePresence allows Microscopy facilities providing a alternative service to clients
High-quality real-time video is achievable over cable modem using MPEG-4 codec's and UDP transmission
Internet security is ensured using firewalls and encryption of communications
Architecture is extensible and applicable to any instrument for which DLLs or control code are available
Remote Access and automation Workshop 2005
Acknowledgements CSIRO Minerals
Aaron TorpyNick Wilson
CSIRO ITSTony SandersonAndrew Smith
Engineering Firm CymontkowskiMarc Cymontkowski
Further informationTPM web reference
http://minerals.tpm.csiro.au/software/telepresence/
Remote Access and automation Workshop 2005
Questions?