Voice over Internet Protocol (VoIP) in a Control CenterEnvironment

Joseph L. Pirani 1

NASA Marshall Space Flight Center, Huntsville, Alabama, 35812

Steven G. Calvelage2

Colsa Corporation, Marshall Space Flight Center, Huntsville, Alabama, 35812

The technology of transmitting voice over data networks has been available for over 10years. Mass market VoIP services for consumers to make and receive standard telephonecalls over broadband Internet networks have grown in the last 5 years. While operationalcosts are less with VoIP implementations as opposed to time division multiplexing (TDM)based voice switches, is it still advantageous to convert a mission control center’s voicesystem to this newer technology? Marshall Space Flight Center (MSFC) HuntsvilleOperations Support Center (HOSC) has converted its mission voice services to a commercialproduct that utilizes VoIP technology. Results from this testing, design, and installation haveshown unique considerations that must be addressed before user operations. There are manyfactors to consider for a control center voice design. Technology advantages anddisadvantages were investigated as they refer to cost. There were integration concerns whichcould lead to complex failure scenarios but simpler integration for the missioninfrastructure. MSFC HOSC will benefit from this voice conversion with less productreplacement cost, less operations cost and a more integrated mission services environment.

I. Introduction

VOICE over Internet Protocol (VoIP) has changed the world of telephony. Businesses are now phasing outtraditional phone lines in favor of their data network lines. With new construction or remodeling, it is lessexpensive to just pull data cable, particularly Category 5 cable, to run voice and data in all locations in a

building. Businesses with multiple locations can consolidate phone services with Internet telephony for less costthan traditional phone service at each location. Residential VoIP services have grown with more broadbandavailability throughout the country.

There are advantages and disadvantages of using VoIP services. The main advantage is the cost. Oneaspect of that cost is the network infrastructure. The same network that carries data also carries voice traffic. Thatequates to less wiring in new building installations or remodeling and less electronic infrastructure because of theconverged service. Because residential VoIP services are not taxed or assessed regulatory fees, the service is muchless expensive than traditional phone service. PC to PC calls are usually free. Another advantage is portability.With a laptop, you can always log into your VoIP account and make or receive telephone calls with the sameassigned telephone number. This could be an advantage over the portability of cell phones if you are in a differentcountry as there would be no new fees for ‘roaming’ or connecting to another subscriber cell network. Thisportability also allows businesses to consolidate their multiple locations with a single phone network. It also worksin the reverse example; a business can have a virtual presence in another geographic location. Your business canhave a Houston, Texas phone number, but the number will connect to a phone in Huntsville, Alabama.

There are also disadvantages to VoIP services. The main disadvantage is service during a power outage.Plain old telephone service (POTS) phones are powered through the current supplied through the phone line. YourVoIP equipment is locally powered and susceptible to an interruption in power. The business must also add the costof a generator or uninterruptible power supply to the VoIP service if it expects voice service during power outages.While portability is an advantage, it also leads to a disadvantage of emergency calls. Traditional 911 services cantrace the location of an emergency call and divert to the nearest call center. VoIP calls are made between IP

1 Lead, Ground Systems Development Branch, EO50, Mission Operations Laboratory.2 Communications Engineer, Ground Systems Operations Branch, EO60, Mission Operations Laboratory.

American Institute of Aeronautics and Astronautics

https://ntrs.nasa.gov/search.jsp?R=20100020881 2020-02-26T07:24:39+00:00Z

addresses and not physical locations. The technology is not there to pinpoint the actual location of an IP addresswhere a VoIP call originates. One of the earlier disadvantages of sharing voice and data traffic on the Internet wasvoice quality. During large data transfers, voice service would suffer because all IP traffic is the same. There wasno way to prioritize which data packets were more important for timely delivery. Voice quality has improved andmajor networking vendors have introduced equipment to route and prioritize voice traffic. VoIP services andinstallations have increased in the last 5 years. However, residential broadband Internet service though DigitalSubscriber Line (DSL) or television cable providers is still not as reliable as the local telephone company. So,customers are still weighing the cost versus availability options in deciding to switch from POTS to VoIP. So, thisis one of many choices that an Information Technology (IT) organization must make when providing voice servicesto a business or campus.

II. History of VoIP at NASA Marshall Space Flight CenterIn 2002, Marshall Space Flight Center (MSFC) had to replace their Private Area Branch Exchange (PABX)

phone system. This system had been in operation since the mid 1980’s. VoIP was chosen as a means to replacestandard telephone service in a selected area. This was needed to gather existing phone switch hardware for spareparts. As the PABX switch was no longer being supported by the vendor, it was imperative that enough parts couldbe gathered for maintenance and repairs until such time that a new replacement system could be ordered andinstalled.

VoIP was a viable commercial option at the time. During this timeframe, MSFC was upgrading itsnetworking infrastructure to category 5 cabling and 100BASE-TX Ethernet standards. One area of buildings on thecampus was undergoing facility renovations. Users in these buildings were selected to use the new VoIP technologyto test the service and user reactions to the new phones. This experiment concluded at the end of the fiscal year2002. Many of the disadvantages of VoIP mentioned earlier had real effects on the project. In an attempt to makethe new voice services as close to reliable as the current PABX system, it was imperative to increase the networkreliability. Redundancy was designed into the network. Core and distribution routers were redundantly installed tohandle a failure of any one router. However, this redundancy was not installed at the end user switches where userdevices, such as personal computers or VoIP telephones, were connected. Uninterruptible Power Supplies (UPS)were priced to ensure no power interruptions to phone service. Core and distribution routers were connected to UPSpower. End user switches were not connected to UPS power. The network vendor had newer switches that couldprovide line power to a network attached device. These switches were significantly more expensive. Also, VoIPphones were much more expensive than digital telephone devices.

Extrapolating the costs of adding reliable power to a VoIP system with the cost of new phone units acrossthe entire campus, VoIP services in 2002 would have been twice as much cost as simply buying another PABX. So,the VoIP experiment ended at MSFC for normal telephone service.

III. Voice Services at the Huntsville Operations Support Center (HOSC)The HOSC, located at the Marshall Space Flight Center, supports the International Space Station (ISS)

Payload Operations Integration Center (POIC). The POIC manages the execution of on-orbit ISS payloads andpayload support systems in coordination with the Mission Control Center in Houston (MCC-H), the distributedInternational Partner Payload Control Centers, Tele-science Support Centers, and payload-unique facilitiesthroughout the world. The POIC provides capabilities required by the POIC Cadre to manage and integrate payloadoperations. The POIC also provides capabilities required by individual payload users to operate and control theirpayloads and experiments. In addition to telemetry and command processing and video distribution, the POICprovides mission voice communication. The POIC is the source of payload voice communications to all payloadusers.

The circuit-switched HOSC voice system, the Enhanced Voice Distribution System (EVoDS), was installedin 1999 utilizing existing keysets that had been in use since the late 1980’s. The system provided standard voiceconferencing services via HOSC operator configuration control of keysets. It also provides record and playbackcapability for internal HOSC users. Remote facilities and users have the capability to connect to the HOSC via T1interface. The system does provide standard Air-to-Ground and Space to Ground voice loop enable and disablecapability for remote sites and keysets.

In 2000, the Missions Operations Lab (MOL) embarked on a joint development with industry to provide anInternet based voice distribution capability for the mission science community. During this timeframe, the HOSCbegan delivering more remote services. The paradigm had shifted from payload operations during Shuttle missions

American Institute of Aeronautics and Astronautics

with a large contingent of scientists and payload operators located in the HOSC to 24-hour, on-orbit science withscientists and payload operators located anywhere. It was impractical for these users to relocate to Huntsville formission increments which could last six months or more. In May 2002, the MOL released an Internet VoiceDistribution System (IVoDS) for remote payloads users. In addition to VoIP conferencing servers, connections tothe HOSC circuit-switched voice system, the IVoDS architecture also included a virtual private network (VPN)connection to the HOSC as this voice traffic traversed the general Internet. Figure 1 shows the general voicearchitecture of the HOSC as VoIP services were added in 2002.

Crew + OpsJSC Voice Loops

Mission ControlCenter

MSFC Payload Operations &Integration Center Remote Payload User Sites

Existing EVoDSTelephony System

Dedicated Circuit-

MSFC Switched T-1 Lines

VoiceSwitch

EVoDS Keysets

IVoDS Server New IVoDSAnd Gateways Voice over IP System

:

arch,, ResePublic IP

etworks

IVoDS PC’s

Figure 1: HOSC Voice Architecture circa 2002 1

Although the cost for such a development was relatively high, the ISS program realized that the savings of utilizingthis technology versus the travel costs plus installing dedicated voice circuits to remote locations was well worth theinitial cost. Also, the advantages of this cutting edge technology outweigh the disadvantages of potential quality ofservice issues with the Internet. A comprehensive testing program, which lasted over six months and involvedschools and NASA National Educators Workshop teachers, provided feedback and the initial baseline for the voicequality when the service was introduced to ISS operations. 2

In 2007, research showed that VoIP technology had progressed to allow for a commercial-off-the-shelf(COTS) system. A COTS replacement system was purchased and installed for POIC remote users. This latesttechnology opened the doors for allowing voice to practically anywhere via a laptop with Internet connectivity.Another paradigm shift was occurring as research was beginning to prove that using a total IP voice distributionswitch was a viable option for a control center environment. However, the EVODS has gone well beyond end ofservice agreements with its vendor. Also, VoIP had improved as a commercial product since 2002, and many userswanted better performance and features similar to that of a traditional voice switch keyset. It was past time for theHOSC to upgrade its voice services for the POIC.

In 2007, NASA finalized an Agency purchase for mission voice switches. The Mission Operations VoiceEnhancement (MOVE) would replace existing mission voice systems with COTS products to standardize thesesystems throughout NASA. However, some of HOSC’s requirements were not met with MOVE, particularly the ITsecurity requirements, as the “one size fits all” philosophy could not keep up with the evolving mission supportphilosophy of the POIC. As mentioned earlier, the POIC supports many remote users. The paradigm shift ofdelivering services was no longer confined to users in a control room environment. As the MOVE system was timedivision multiplexed (TDM) based and common to all Centers being delivered a new system, implementing new

American Institute of Aeronautics and Astronautics

requirements could take years. A server based IP voice system was more flexible and more responsive to thePOIC’s changing requirements and user base. The MOVE system is a purely TDM solution and contractually inplace for 10 years. Since it meets NASA rigid static configurations required for Mission Critical Real time mannedspace flight requirements, it does not necessarily facilitate less static environments such as space science researchand payload management. The HOSC had to adapt its mission voice communications services to IP based voicesolutions to meet the constantly changing customer base.

IV. The Future of VoIP at the HOSC

The Voice over IP Mission Conferencing environment considers many of the same support requirements aswould be for a telephony design and implementation. First, the HOSC team had to determine how large thisMission Conferencing VoIP system should be to handle the ISS mission at MSFC. The POIC supports 100concurrent users accessing up to 500 unique conferences, with over 700 external participants joining in the form ofT1 DS0 digital signaling connections. Using original EVoDS telephony requirements and enhancing theserequirements with existing remote IVoDS requirements, Table 1 lists a blend of telephony and IT requirements forthe system.

950 real time conferences Access control to restricted conferences

Individual user ID’s Encryption and hardened informationtechnology security (ITS), with accountlock out

Individual user preferences Lightweight Directory Access Protocol(LDAP) Interoperable

Multiple headsets, with external Speakers andFoot Switches

Multiple compressor- decompressor(CODEC’s) support

Sidetone Gain and attenuation controls

Non blocking Rapid reconfiguration

Special signaling controls and support System alarm and status

External interfacing via a channelized T1

Table 1: Mission Voice Conferencing IVoDS Requirements. This listing is not all inclusive

American Institute of Aeronautics and Astronautics

Unlike a traditional telephone system, telephony included, the Mission Voice Conferencing systems allow usersaccess to hundreds of conferences at the push of a button: no dialing, no call set up, no waiting. Specification fortranscoding dissimilar streams and media termination point bridging were not included for the Mission VoiceConferencing system versus being mandatory features of a telephony system.

A major design consideration for either a telephony or Mission Voice Conferencing system is theunderlying network infrastructure. The

4edica0d kT11 tnter^armeGt}vnty t76dt'cated Private IP'Flelwark; POIC determined early on that the

Ex[CrnaIN6i CCi t ^i Mission Voice services warranted

-se- t dedicated switches and LAN segments t to the core routing engine to connect the

gaga; t ; user devices, and the conferencing and

i^ sA t ; gateway servers. This decision

F.x V10 s' t vp^^ r eliminates the risk of traffic competitiont Corife"reneing' rt kid' - i `

N,o' Imo' for available bandwidth and priorities.-ion

;I Voics IJse"rst Voce YF& This is not a luxury for deploying at Syste`_m I

,IntemalrVoice : t d telephony solution, which relies heavily

! t 1vocis,t _ i on the use of existing infrastructure toTes,IVaDS'

txs iv aos ki

be cost beneficial. Figure 2 shows thet y IVoDS system architecture. Ast i^ mentioned earlier, a disadvantage of

-------------------------------------- _-_-_:_- t ;i VoIP is the quality of voice could sufferihils'sec[lon Includes-

I K11s'sec[lon InclLid60 ri;is sectbn Indudes::, if IP traffic has suddenly increased on

Exlsting Patch' Panels Prime aotl Back;p d 8,1 dedicated' the data network. Mitigation to this risk

Wiring IrTrrasvucture Conference Server rToudrt5c een Keysetsis to implement a set of products and

Test Equipmeni' 7 T1rGa^eway yen a s' S GlCantrol _WfS, h pStaiiistandControY_WJ$ T`detlicatgd' techniques which would prioritize voice

^ ^ ,E[hiarnettSrvi[ches'traffic above other data. This is an

Figure 2: IVoDS System Architecture 2010 added cost to network equipment and islabor intensive. An additional risk is having all traffic at risk during a network outage. HOSC mitigated this risk byhaving a separate local network (LAN) for voice traffic within the control center. The cost was minimal to add thisseparate LAN as opposed to upgrading the existing network to prioritize traffic and to eliminate additional potentialfailure points in the network.

Voice services and systems have to be cognizant of the end to end time as to minimize the effects of delaythat can be perceived as echo. A range of 25 to 150 milliseconds (ms) is generally considered an acceptable range.This is the time measured from speech source to listener ear. The distance between any two users will alwaysinvolve many different types of equipment, of analog and digital communications functions. This is can bedescribed as; (Talker) User Headset (H/S) to Conference/Router to Gateway to LAN/wide area network (WAN) toGateway to Switch to User H/S (Listener). Since every user hears and talks in analog, this sound energy has to beconverted to a digital form at each end preferably with no other conversion in between. This conversion should takeplace in very close proximity to the actual ear and mouth. The digital conversion is described as Mu Law per theITUG.711 3 standard in which voice is sampled 8000 times per second with each sample representing 8 bits, whichyields the highest speech quality. Once digitized, the voice packets conform to standard digital communicationsarchitectures.

The challenge for a Control Center and its voice users is that generally the spoken conversations are fromone to many versus a point to point connection. A portion of those many are located in the same proximity or room,while others could be located thousands of miles away. Prudent network planning would be used in the ControlCenter design and implemented to limit the delay between all users to < 150ms 4. Legacy voice conferencingsystems, which are usually TDM switches, overcome this challenge through the use of dedicated wiring and circuitsthat operate at high speeds. This works well but does not allow itself to collaborative networking. Dedicated TDMsystems generally meet or exceed delay times <80ms including any terrestrial WAN links. In an all or near all IPsystem, new considerations are mandatory such as the Open Systems Interconnection model for layeredcommunications and computer networking as well as other networking protocols such as TCP/IP, UDP, andSignaling Standards. Special considerations are given to the general make up of co-located and inter-room listeners,where visual contact as well as ambient sounds can be observed. The goal in designing an IP voice system to meeta Control Room environment is to be standards based and to have minimal software customization to replicatelegacy time division multiplexed functions.

American Institute of Aeronautics and Astronautics

A commercial system was implemented in early 2010 within thePOIC. The newest IVoDS service has minimal levied developmentalrequirements to meet the delay expectation of 60 to 120ms. Comparing

0 the differences between the standard HOSC IVoDS service that providesvoice services to remote science community, and the latest delayintolerant IVoDS, the round trip delays are quite contrasting. One waypacket delivery dropped from 210ms to 120ms. Expecting delays of120ms is worst case and it generally averages around 100ms. The biggest

' I offender of delay is induced by the Operating System (OS) and-.^ application software routines. The network (layer 3 and down) only plays

u_►^ ^_ _^ a small role of a few milliseconds. This is due to implementing a network

Figure 3: HOSC IVoDS Keyset that is 100Base-T from network interface card to switch and gigabit over

Image courtesv of Quintron Svstems fiber between the core routers. As described earlier, traffic is segmentedto its own network and dedicated switches. The analog to digital

conversion takes place in a commercial universal serial bus (USB) attached sound card in a traditional pistol gripstyle with integrated push-to-talk (PTT) function. Another option to achieve reductions in voice processinghandling is in the form of developing a custom digital signal processor and network interface that performs a veryspecific function. This type of effort would utilize an existing user PC to connect the add-on custom hardware. Noactivities were performed to test the custom digital serial processor idea that the voice processing was expected toperform around the 80ms range.

The user keysets, as shown in Figure 3, have been developed using a light weight Unix OS running on acompact dual core, single board computer package. Many of the usual OS functions have been removed andtailored to focus on digital voice processing with some input-output management to operate a Touch Screen devicefor user interactions. Pursuing the customized OS provided another equally important benefit. By removing most ofthe services running on a standard OS, the IVoDS Keysets proved very secure. By reporting few, if any,vulnerabilities on all IT security scans, the keyset is an invisible (non reactive) device hanging on the network. It istruly an appliance running on the network that only requires two ports for secure encrypted voice communicationsbetween any user and the conferencing server.

With the high security provided by the new IVoDS keysets, the system integrates with all current networkand institutional services, such as Active Directory, network timing protocol (NTP), Enterprise network operationscenter (NOC) monitoring and control, voice recorders and external voice lines and connectivity. Additionally, it iswell suited to integrate with future requirements including single sign-on and two factor authentication. Theforward looking plan for the Mission Voice over IP services is a candidate for the contingency operations andmobile computing environments.

Figure 4: HOSC IVoDS User Interface

American Institute of Aeronautics and Astronautics

V. Operations Paradigm Shift

An evaluation of operational support skills was done prior to selecting the new local IVoDS service into thePOIC. Traditional TDM telephone systems require specialized tools and technical skills to maintain and operate.The new system brings established skills that currently reside in the facility, such as network engineers, systemsmanagement and administration, account management and IT security. Contrasting the skills above with a TDMsystem, these are not normally integrated with many of the standard services, and requires talent not seen muchtoday such as the telecommunications engineer, electronics technician and field electronics technicians. Skills totroubleshoot electronics circuits to component or sub component level have been replaced by systems that are eitherreplaced in whole or broken down to assemblies like line replaceable units; hard disk drives, power supplies,motherboards, memory modules and Peripheral Component Interconnect (PCI) cards. The new IVoDS is nodifferent in terms of maintenance as to what would be expected to operate and maintain a personal computer orserver. These are the skills sets of today and tomorrow. Similarly, the users are expected to operate the new userdevices, which resemble any other computer or workstation located on the POIC consoles. Figure 4 shows theIVoDS user interface. From an operations day to day management perspective, it is truly a point and clickenvironment, compared to the old days of single command style line entry.

The new system was implemented in parallel with the existing system for some length of time, in order forsystems validation, and training to occur. Particularly high on the list of items to accomplish was a short parallelreal time operations period. This is driven by the need for a user acclimation period enabled both the end user andsupport personnel to acquire the knowledge base to operate the new system proficiently. There was also a concernof induced human errors while maintaining multiple voice systems, as skill sets were different for each system. Aminor facility concern consisted of sustained burdens on the facility as the legacy system power and coolingconsumption levels were near the maximum safe parameters. The older TDM systems consume large amounts ofpower and require much more cooling than the new IVoDS. A short 2-3 month period was targeted for the POICparallel operations.

With the new system in place, it is clear that operations and maintenance will not add additional complexityto the Ground Operations model. As shown, the new voice system simplifies the operations model by replacing aunique TDM voice system with an IP voice system that has the same IT footprint as other control center servers.The new system provides the mission voice users with a flexible yet highly available service. IVoDS now supportsthe internal and external customers of the POIC and HOSC.

HOSC collaborated with the Johnson Space Center (JSC) Mission Operations Directorate (MOD) for manyprojects to find common solutions to both centers. As part of that collaboration, the POIC has tried to lead inimplementing more cutting edge solutions to provide a proof of concept for NASA’s main mission control center atJSC. MOD is also getting more requests for mission voice loops outside of their control center environment. TheHOSC is showing the benefits and difficulties of remote servicing of voice services to MOD. Their implementationshould be easier and with less risk, benefiting from the experiences of implementing VoIP services at the HOSC.

VI. Conclusion

MSFC HOSC was in desperate need to replace its mission voice system. The EVODS system was wellbeyond its end of serviceable life. The HOSC has annual power maintenance on its electrical grid. Coming out ofthis service window each year, it was always a risk whether EVODS would survive the power cycle. Now, with anew system with more modern technology and a smaller footprint, the HOSC will start to see operations andmaintenance savings. The first immediate savings will be space. The new IVoDS system resides in the samecomputer room as the rest of the control center servers. EVODS, with its many racks, interfaces and wiringinfrastructure, along with residing in its own room is a methodology of the past. This will be, obviously, less drawon the total facility power consumption steering the HOSC towards a greener business model.

The users quickly picked up on the use and manipulation of the keysets to customize it to their preference.Some of these user specific settings included loop location on the key frame matrix, individual volume control,screen angle, brightness and color saturation. A survey was given to each user after group and individual training onthe new system. Table 2 summarizes the overall results from the collected data concerning three targeted areas ofthe service and keysets. Obviously, the new keysets are a paradigm shift from the push button keysets from overtwenty years ago. Each user now controls customization features of the keyset to allow him to operate as he feelsmost efficient. These customized configurations are saved for each user as the keyset and system is accessedthrough individual login.

Voice over IP introduces a new chapter in mission operations. Service costs are reduced and users havemore flexibility over their voice services. With appropriate permissions, users outside of the control center can now

American Institute of Aeronautics and Astronautics

participate in mission operations as society has become more dependent on mobility and flexibility ofcommunications.

50 ML

Table 2: IVoDS User Survey Results

References

1 Chamberlain, J., Bradford, B., Best, S., and Nichols, K., “Utilization of Internet Protocol Base Voice Systems in RemotePayload Operations,” SpaceOps 2002, Houston, Texas, page 5

2 Chamberlain, J., Bradford, B., Best, S., and Nichols, K., “Utilization of Internet Protocol Base Voice Systems inRemote Payload Operations,” SpaceOps 2002, Houston, Texas, page 5

3International Telecommunications Union, “G.711: Pulse code modulation (PCM) of voice frequencies,” URL:http://www.itu.int/rec/T-REC-G.711/e [cited 14 February 2010].

4 Chong and Mathews, “Comparative analysis of traditional telephone and voice-over-Internet protocol (VoIP) systems,”Electronics and the Environment, 2004. Conference Record. 2004 IEEE International Symposium. 10-13 May 2004. On page(s):106 – 111.

8American Institute of Aeronautics and Astronautics