Automating The Teleport - Kratos Defense/media/... · For Automating the Teleport, WTA invited...

World Teleport Association www.worldteleport.org © 2018 World Teleport Association. All rights reserved. December 13, 2018

STRATEGY | TECHNOLOGY | RESULTS US$1,650, free for WTA members

Automating The Teleport

Automating the Teleport Page 2

World Teleport Association www.worldteleport.org

Contents Foreword ............................................................................................................................... 4 Catching Up with the Telephone Company ............................................................. 4 Executive Summary .......................................................................................................... 9 What to Automate – and Why .................................................................................... 16 Who Can Automate Your Operations? .................................................................... 21 What Do Operators Need to Know About Automation? .................................. 31 The Cost-Benefit Analysis for Automating a Process ....................................... 38 Lessons Learned – The Hard Way ............................................................................ 42 How to Avoid Deployment Nightmares ................................................................. 47 What Do Operators Want Next from Automation.............................................. 50 About the Report ............................................................................................................ 54 About World Teleport Association........................................................................... 54

SPONSORED BY

You may also be interested in these other reports from World Teleport Association.

Visit https://www.worldteleport.org/store/default.aspx to learn more.

https://www.worldteleport.org/store/default.aspx



Automating the Teleport is also made possible by the financial support of WTA’s Industry Leaders…

…And Industry Patrons:



Foreword The explosion in satellite capacity is dramatically changing the dynamics in the space industry. The teleport which sits in between the terrestrial and space segment is being driven to become more innovative and operationally efficient to meet these new bandwidth demands

The current environment is changing quickly and moving from traditional teleports working with wideband FSS satellites to HTS gateways supporting a huge increase in bandwidth and an exponential growth in services. The critical question is: how will the teleport increase efficiencies to support more capacity, more customers, more services, more flexibility, and more dynamic usage, all at a lower cost per service?

This WTA report shares insights from thought leaders in the industry on how automation will play a critical role in transforming the teleport from traditional antenna farms that provide satellite access to data centers with dishes that layer on value-added capabilities and services. Based on interviews with executives, it’s clear that technology will play a critical role in automating processes and operations in the teleport.

This reinforces what we are hearing from our customers – some of the world’s largest satellite and teleport operators. They are looking for ways to optimize their operations with more automated, intelligent and robust solutions that will allow them to seamlessly scale while reducing costs and improving QoS.

In the larger terrestrial and wireless telecoms markets, virtualization, cloud infrastructure and SDN (Software Defined Networking) are enabling greater efficiency and more dynamic services. The satellite industry is following their lead and striving for the same efficiency gains and dynamic service models. Some of the key areas enabling these gains include:

• Virtualizing ground infrastructure, including the transition from analog to digital teleport IF and RF processing, thus lowering costs and increasing operational flexibility by taking advantage of cloud-based technologies and infrastructure.

• Expanding the use of software-based infrastructure (over hardware-based solutions) to improve scalability, while enabling faster and more dynamic services.

• Automating network and RF management solutions that cost-effectively scale from small, simple, single wide-beam networks to some of the world’s largest and most complex satellite networks in operation today.



• Optimizing service delivery and operations efficiently through the integration of the entire infrastructure, including rolling up legacy M&C and NMS systems into a single management solution that is connected with the customer’s OSS systems. This will enable the automation of not only the tracking of the health and SLAs of customer services, but also the process of identifying the root cause of alarms and support automated recovery.

Kratos has been developing industry leading innovations and enterprise technology in support of teleports and ground stations for the last 30 years, and we are excited about the changes we see coming in the industry. This original research from the WTA provides valuable insights into the role of automation and how it will improve operational efficiency to keep pace with changes in space.

Stuart Daughtridge Vice President, Advanced Technology



Catching Up with the Telephone Company For decades, the mainstream telecommunications industry has gone through wave after wave of automation, from manual to automated switching, analog to digital systems, person-to-person customer engagement to voice processing and online portals.

Through most of that time, however, the teleport and satellite industry remained stubbornly manual. Antennas were pointed and polarized by hand and through telephone calls to the NOC. Tapes were mounted for playout. Service quality was analyzed by eye and managed with fingers on control panels. Of course, systems were digitized – but the underlying processes that delivered service were in the hands of highly trained people 24 hours per day. It all worked because the number of services on an antenna was small and seldom changed, and the high costs of capacity kept the number of customers small as well. Those days are gone. An executive interviewed in 2016 for the WTA report The Teleport of Tomorrow put it this way: “If you are serving a few hundred or few thousand terminals now, you should expect to serve hundreds of thousands in future. With lower-cost capacity and terminals, the demand will increase substantially. If you are a teleport, don’t plan on doubling the terminals and capacity you manage: plan on seeing it increase 10-fold.” These trends are now bringing a sharp change to ground segment operations. The antenna farms of the past are now data centers with

https://worldteleport.site-ym.com/store/ViewProduct.aspx?id=5696259&hhSearchTerms=%2522teleport+and+tomorrow%2522



dishes. Technology vendors are delivering hardware and software automation solutions that substantially reduce the need for manual intervention. They monitor devices and networks and not only send alerts when technical faults occur but indicate what services and customers are affected and when SLAs will be violated. They automate the pointing of antennas and eliminate the need for NOC involvement. They ingest, format, encode, schedule and move content across internal servers, cloud platforms and CDNs. They respond automatically to outages and reconfigure networks on the fly. In the process, they save money, improve quality, speed turnaround and improve profitability. Methodology For Automating the Teleport, WTA invited technology experts, teleport and satellite executives to weigh in on the most transformative automation technologies of today and the technologies that we can expect to see in the near future. They shared their views of what works, what’s hype, the challenges of entry into automated operations, and the resulting rewards.

WTA thanks the following individuals for contributing their time and expertise to the project: Keith Frost Media and Data Networks Director Arqiva Satellite & Media, UK Guy White Director Teleport Engineering and Operations COMSAT, Inc., USA Roger Franklin President & CEO Crystal, USA Ken Fuller CTO Globecast America, USA

Brad Adam President/CEO Hyperlink Inc., Canada Alvaro Sanchez CEO Integrasys S.A, Spain Stuart Daughtridge Vice President of Advanced Technology Kratos, USA Scott Mumford Group Head of Satellite Services Liquid Telecommunications Ltd., UK



Nick Leake Head of Satellite Networks Optus Satellite, Australia Viplob Syngal Director Connectivity Solutions, ScheduAll NetInsight, UK David K. Chan VP, Sales & Marketing Quintech Electronics & Communications, Inc., USA Steven Soenens VP Product Marketing Skyline Communications, Belgium

Toni Lee Rudnicki VP Global Marketing Speedcast Limited, USA Tomaz Lovsin CTO STN - Satellite Telecommunications Network PLC, Slovenia David Grooms Business Development Manager U.S. Electrodynamics, Inc. – USEI, USA Greg Dolan Chief Operating Officer Xytech Systems Corporation, USA



Executive Summary Software automation and “service orchestration” are helping teleport operators improve the efficiency and scale of technical operations. Improving technical operations is good, but the real pay-off is on the commercial side. Faster and more automated workflows make for better QOS and enable teleports to add new revenues without expanding headcount.

Service orchestration orchestrates internal and external hardware and software to make the best use of in-house resources and leverage capacity from third-party providers to meet peak demand and enable faster turn-up. Operators can also reduce risk by using third-party cloud providers to quickly test, add, change and remove services without investment risk. Who Can Automate Your Operations? NMS and M&C systems are the major automation platforms for teleport operations, but there is a range of other systems supporting operations. Below is a sample in alphabetical order, of the leading companies: Network Management and Control

• Crystal provides automation software for monitoring, control and metadata management of the end-to-end broadcast through OTT/TVE. Key products used in the satellite/teleport space include Crystal Control M&C, Crystal Spectrum Monitoring and Recording, Crystal Carrier ID, and Crystal Video Data Analyzer.

• DataPath’s MaxView Enterprise is a monitor and control software solution, installed and supported on more than 1,000 systems in 40 countries. It provides a single, consistent interface to manage all equipment, elements and service applications within a network across all locations.

• Kratos offers end-to-end monitoring and control across satellite and terrestrial networks. Relevant products include Compass for monitor & control, NeuralStar for centralized management of hybrid networks, NeuralStar SQM for service management, Monics for carrier monitoring and interference mitigation, satID for signal geolocation, SatGuard for VSAT monitoring , SigX for interference cancellation and SpectralNet for the transport of RF over IP.



• IPswitch’s WhatsUp Gold Network Monitoring Software for cloud or on-premises visibility of the status of network devices, systems and applications. Network monitoring Layer 2/3 discovery of devices and any networked server

• Skyline Communications’ Dataminer is a multi-vendor network management & OSS solution for broadcast, telecoms and satellite. It provides a comprehensive NMS but the product suite also provides IT service orchestration, customer booking and workflows, and terminal management.

• WorldCast Systems software centralizes and unifies monitoring and management of all connected devices across multiple sites. It brings all the required modules within one interface and optimizes monitoring and maintenance operations.

IT Service Orchestration, Operational Support Systems

• ServiceNow is one of the world’s largest IT service management platforms, which increases enterprise agility by enabling IT to automate manual tasks involving systems and applications outside the ServiceNow environment.

Video, IP and Baseband Quality Assurance

• TAG Video Systems provides specialized IP monitoring and multi-viewer solutions for video on standard and mobile devices.

• Telestream provides a large family of software tools and solutions for media and OTT service production and management, such as VantageVidchecker for automated quality control and correction of file-based media.

Customer Booking, Media and Production Workflows • Net Insight provides online software including ScheduALL, a popular

booking system for OU services; ScheduLINK for transmission management and optimization; and NIMBRA VSION, an end-to-end service orchestration solution.

• Xytech Systems’ MediaPulse creates a seamless environment for media enterprise and transmission workflows, with separate planning, scheduling and financial management tools, reducing redundant data entry and reducing touchpoints for all workflows.



Terminal/VSAT Management • Integrasys provides Satmotion Pocket VSAT Auto Commisioning,

which minimizes deployment time, effort and interference; Controlsat Carrier Monitoring; Alust, an evolution of Satmotion Pocketthat allows accurate installation during remote commissioning; and Satmotion SNG for SNG auto lineups.

What Do Operators Need to Know About Automation? Automation systems can reduce outages and improve QOS – or they can risk creating large, single points of failure. Engineering design and careful planning of implementation are critical to success. On a high level, automation presents an opportunity to rethink aspects of your operations from the NOC to terminal provisioning, and to achieve new levels of efficiency and quality. Proprietary vs. Open Some M&C platforms are proprietary and only work with specific technology, which offers the benefit of simplicity. Others are open, multi-vendor platforms allowing anyone to integrate new APIs and create custom views from that data without additional coding. The risk inherent in these platforms is that, with all information centralized in a few dashboards, there can be too much information and complexity for operators to manage. An IP Media World For media-focused teleports, the industry has changed from monitoring a video wall with hundreds of thumbnails to a data-driven understanding of how services are performing. IT, IP and Cloud Orchestration For a growing number of operators, the next step is about moving beyond command-and-control of the network and adding orchestration of cloud services and applications to support virtualized operations and services. It can also mean integrating the OSS (Operational Support System) and billing systems with network management.



Operator Feedback on Specific Systems For this report, we asked operators to share their views of today’s automation solutions from the “big four” M&C and NMS; Compass, Crystal, DataPath MaxView and Dataminer. The Cost-Benefit Analysis for Automating a Process When you do the cost-benefit analysis for automating a particular process or set of processes, what factors should you consider and what tips the balance toward automation? M&C Systems The payback for M&C systems may be hard to quantify since the payback is in performance and mean-time to restoration rather than cost savings. Areas worth considering in your analysis may include eliminating of time-consuming manual processes, the avoidance of customer refunds or credits due to errors, and higher QOS and customer retention. Trouble Ticketing and IT Support Procedures Manual ticketing processes can take hours, which automation can turn into seconds, while ensuring that trouble tickets are accurate and complete by automating the workflow process. Booking and Asset Management Systems Reducing the time required to book a circuit from New York to Tokyo from an hour to 5 minutes could save US$50 a booking. Booking errors that require last-minute re-routes can add 5-10% to operating costs. A resource scheduling and asset management system can also reduce under-utilization of your inventory, be it satellite or ground facilities. Remote Site Installation and Management Automating the field install process and integrating it into provisioning and billing and reseller requests can drive down an operator’s break-even point, making it possible to activate hundreds of terminals per day. The cost-benefit analysis can also be based upon freeing labor hours, increasing customer satisfaction and time-savings in response to customer calls. Lessons Learned – The Hard Way “Garbage in, garbage out” is one of the oldest rules in computer processing. What are the lessons operators have learned in automating mission-critical processes?



Identify the Desired Outcomes The processes to be automated must be sufficiently understood and documented to be turned into code, and the reduction in manual work must be sufficient to justify the time and expense of deployment. User-Friendly Interfaces are Important User-friendliness is not a “soft” benefit – it can spell the difference between a system that minimizes operator errors and one that actively generates them. User-friendliness is even more critical when you will allow customers access to the automation system. Carefully Map Human Interaction Automation pull tasks out of the labor chain – but human interaction is still required to input information or respond to the system’s output. Any human interaction can create a dead-end point for the system, and these must be carefully thought out, drawing on the expertise of future users. Map the Logic Behind the Software You need to retain a higher-level understanding of the system’s logic and sustain that understanding in your organization. Individual network operators on staff may need to understand more deeply than before how the systems are being managed, and what their capabilities are. Understanding and Validating Your Data Converting manual or semi-digital processes into automation puts a premium on data quality. Respondents focused on these aspects of data preparation: • Metadata Taxonomy. You need to begin with a comprehensive and

vetted taxonomy to identify what systems own each part of the data. • Nomenclature. In combining systems and allowing data to be shared

among them, it is vital to make sure that every product, endpoint and element of the systems has a unique and unambiguous name.

• Taking Out the Garbage. For a client portal, the software has to be smart enough to filter out bad data as it is entered.

• Preserve Historical Data. For applications like monitor & control, the ability to take an historical look at your systems is essential.

• Internal Alignment and Resources. Organizations can drag out the deployment of an automation system for years when there is no internal alignment. Make sure your organization and affected



stakeholders are prepared to make it work by addressing their concerns and aligning their incentives early in the process.

• Compatibility with Other Systems. It is paramount to review the ability of new systems to interface with your core network.

• Testing. You must stress-test systems and play scenarios for worst-case situations to ensure the new system will not overload operators with information they cannot use.

How to Avoid Deployment Nightmares Teleport executives and tech suppliers offered a number of recommendations for staging the implementation of a system. 1. Plan It the Right Way

An effective automation plan includes all human interactions, business systems and decisions required. Everyone who will have a role in the system’s success needs to be involved. In planning, work through scenarios and logic for the most common actions. This will also reduce training time, because operators understand how the automation logic should work, and where to look to fix or recover.

2. Succeed One Step at a Time

Break the project into manageable bite-sized pieces and prioritize easy wins so teams can buy in early. Initial steps for automation should be assistive in nature, helping people do their job rather than reducing headcount.

3. Train for the Right Results

Any significant process change usually means skills need to be upgraded. All operators and suppliers we interviewed agree that it is essential to invest in training and skills with automation projects. Automation often removes first-level support tasks, increasing the importance of more highly skilled and trained technical support.

4. Be Agile

Several operators advocate the Agile development process, which can quickly develop and deliver a solution, then rapidly improve it through testing and user feedback. It puts software in users’ hands rapidly, knowing you’ll need to change it, but gets you closer to the reality of the job faster.



What Do Operators Want Next from Automation? Looking to future needs, operators reported interest in more unified management, open interfaces, easier integration, tools for hybrid cloud management, and new technologies like artificial intelligence. Unified NMS Architecture When it comes to operating multiple M&C systems, trying to wrap another M&C system around them becomes very challenging. Operators say they want unified architectures to manage their devices and networks, from SLA and QOS to service management, commissioning, and lineup. Virtualized Environments and Common Platforms Another goal some operators share – and are in various stages of implementing – is to build architectures that help teleports support virtualized software environments to enable better data-center economics and operational efficiencies from the playout to CDN delivery. Service Management and IT Orchestration Tools Software suppliers continue to add APIs, allowing easier exchange of data between applications, but more time and work is needed to allow seamless integration of control systems. Operators would also like to see is more tools to help them leverage openware applications. AI and Machine Learning to Manage Complex Data As operation of hybrid satellite multi-cloud networks grows, the amount of network information outpaces the capacity of people to manage it. Artificial intelligence (AI) or machine learning could help by running many calculations and checking the patterns of equipment and software status. End-to-End Automation Another area on product roadmaps is further integration with back office systems such as IVRS, plan maintenance, CRM, inventory databases, topology databases and help desk systems. New IP Media Protocols A technical development that could offer big opportunities for improved automation of media services is Networked Media Open Specifications (NMOS) protocols. NMOS is an IP media networking family of open specifications in development by media companies and tech suppliers to make media standards easily interoperable with IP standards and protocols.



What to Automate – and Why

Teleport operators and their automation technology suppliers described key areas where software automation is helping them improve the efficiency and scale of technical operations. These include network and asset management, provisioning, booking, trouble ticketing, SLA reporting, installation and site management, IT orchestration and more.

Improving technical operations is good, but the real pay-off is on the commercial side. “The biggest opportunity is shifting from being reactive to proactive, enabling faster delivery of services and at lower cost,” says a European teleport executive. Faster and more automated workflows make for better QOS and enable teleports to add new revenues without expanding headcount. Service “Orchestration” In broadcast and other markets, there is much disruption from new cloud-based players delivering video and data services in a nearly 100% software environment. This has met customer demand for flexible, shorter-term contracts without investment in hardware, and the ability to add and change service quickly to meet fickle consumer demand. The operator’s challenge, says a European executive, is how to move from traditional broadcast to a world where you can provision like an IT desk and stitch together resources for short term or long-term services. That is where Service Automation or Service Orchestration comes into play.

Teleport operators and tech suppliers say that service automation enables new commercial possibilities. It orchestrates internal and external hardware and software to make the best use of in-house resources and leverage capacity from third-party providers to meet peak demand and enable faster turn-up. Operators can also reduce risk by using third-party cloud providers to quickly test, add, change and remove services without investment risk.



What is Orchestration? Orchestration is often discussed in the context of service-oriented architecture, virtualization, provisioning, converged infrastructure and dynamic data center topics. Orchestration in this sense is about aligning the business request with the applications, data, and infrastructure. It defines the policies and service levels through automated workflows, provisioning, and change management. This creates an application-aligned infrastructure that can be scaled up or down based on the needs of each application. Orchestration also provides centralized management of the resource pool, including billing, metering, and chargeback for consumption. For example, orchestration reduces the time and effort for deploying multiple instances of a single application. And as the requirement for more resources or a new application is triggered, automated tools now can perform tasks that previously could only be done by multiple administrators operating on their individual pieces of the physical stack.

Network Management Systems (NMS) NMS systems have been automated for many years, but the increasing scale of operations, sites, services, devices and integrated networks makes NMS systems more complex and important than ever. For example, a European teleport with hundreds of video channels has to monitor 5,000+ devices at local and remote sites around the world using an all-encompassing NMS. Thanks to the automated setup, the operator achieves extremely fast service, very low cost, scale efficiency and huge labor efficiency gains.



Monitoring & Control (M&C) Assurance management, confidence monitoring, signal quality and service monitoring are key areas where an NMS drives labor efficiency gains, improved service and the ability to scale operation.

One teleport operator uses network monitoring and control software to manage all network infrastructure. “By using the system, we give broadcast customers Virtual Network Operator capability, so they can see their equipment and control their networks without damaging other customer operations within our network of earth stations.”

Confidence Monitoring Automation has transformed confidence monitoring. It allows fewer people to monitor more channels and data signals, to the point where human visual monitoring becomes secondary. Combined with data recording of signal, image, and device data events for later reporting and analysis, automatic monitoring systems improve accuracy and help improve quality of service. Remote Spectrum Monitoring In the past, expensive spectrum monitoring systems could not affordably be placed at remote terminals and polled for site data to, for example, isolate site-specific RF interference. “Automated spectrum monitoring, connected to software where you can pre-configure automated alarms, is a key area for improving quality for us,” explains an operator’s executive, ”because customer expectations for service are the same even as prices have come down in the market.” QOS, SLA Calculation and Reporting In many traditional teleport environments, SLA calculations and reporting can involve a lot of manual work. Determining billing credits or penalties and calculating the impact of service disruptions on service level agreements (SLAs) consumes substantial staff time. Manual tasks run the gamut from reading through trouble tickets and pulling device data to scanning emails, checking contracts and querying billing and accounting.

Automation of service assurance and SLA reporting is an area that is easy to reduce cost and increase QOS, according to interviewees for this report. “The biggest opportunity is to automate daily routine, from service lineup to occasional use sessions,” says an NMS supplier’s executive. “The majority of tasks done in teleports today are manual tasks. Anything that can be done from a PC, we can automate the repetitive tasks.”



Installation and Remote Management In the VSAT service arena, the cost for installation and maintenance of thousands of remotes in the network comprises the single largest cost factor in the network. By achieving cost efficiencies, automation lets providers close business models they could not otherwise close in broadband VSAT markets such as digital divide, mobility and remote connectivity. “An example is a 200,000-school program in Mexico, where we have customers connecting 6,000+ VSAT terminals,” explains a software supplier. “That is a huge project.”

Automating the process reduces costs for remote installs, as well as NOC time and costs. It helps networks deploy faster and achieve time-to-market and revenue acceleration. Poor antenna installs not only require expensive site visits, but can cause RF interference, increase overhead and space capacity utilization, and cause QOS hits. Automated VSAT lineups and installation maximizes the remote link performance, minimizing operational expenses. (See WTA Report: Understanding and Improving the ROI of VSAT Networks.) A teleport improved its field install/service management by fully integrating subcontractors into its online system. Improved remote site management through automation delivers its biggest savings. “The ability to monitor client services from their point of view and make changes without manual intervention at the site level enables a new level of experience for customers,” reports the teleport executive. Occasional Use and SNG VSAT is not the only application to benefit. Occasional-use services require detailed booking, scheduling and operational resources, on top of network engineering, provisioning, billing and other activities, on a dynamic basis. Automation greatly reduces the complexity of stitching together numerous facilities, fiber and satellite networks, and reduces the potential for costly manual errors, double-booking, resource conflicts or schedule errors and overlaps. “The biggest opportunity for us is to automate our occasional-use process and provisioning of OU resources,” says the CTO of a global teleport provider. “Ideally, the system knows what resources are available from its own inventory or through a manual entry by booking, and can execute an automated process that provisions routers, switches, modulators and transmitters.” Another satellite and teleport operator has fully automated its satellite newsgathering (SNG) operations by implementing a web-based environment for booking satellite access across its space fleet. It



developed a web-based customer portal that allows satellite users/SNGs to input their booking reference, then schedule and line up carrier access without calls to the satellite NOC. “Calls to our NOC from SNGs dropped by 80%,” says the operator’s executive.” Assets and Resource Management Automation allows for better tracking and management of inventory and resources in almost any industry. Automating the allocation of network resources in the satellite teleport environment, including on-premises and off-premise (cloud-based) assets, can bring the immediate benefit of better capacity utilization. Asset and resource management is also an underlying prerequisite for automating the provisioning of those resources for services.

By automating front end provisioning, for example, a Pacific Rim operator reduced the number of Tier 1 and Tier 2 tech support interactions. This made possible cost savings of 25-35% from front-end workforce labor. Self-Provisioning An operator in Africa has an online customer portal for its wholesale business that allows its ISP customers to order and self-provision VSAT services for their retail end-user customers. “The big benefit is the reduced number of highly skilled staff needed to run the business,” explains the operator’s executive. “We sign a reseller agreement and do not need physical interaction with our reseller customers. They buy the terminal and can select the SLAs and it automatically starts billing. They can suspend or cancel sites. Customers can operate their own businesses, and the software lets them control their operations instead of relying on our staff. The system also ensures that installers must adhere to predefined signal levels as the VSAT is installed, preventing poor installationss and greatly reducing installations costs.



Who Can Automate Your Operations?

NMS and M&C systems are the major automation platforms for teleport operations, but there is a range of other systems supporting operations including:

• OSS/BSS systems • Spectrum monitoring • RF switch automation • Specialized M&C for HVAC, water, power, security and de-icing • Media production, transcoding and encoding • Workflow • Customer relationship management • Enterprise resource planning

Below is a sample, in alphabetical order, of the leading companies serving the industry and what they can offer. The companies are grouped under the categories for which they are best known but it should be noted that many offer products serving multiple purposes from NMS to service orchestration and media and production booking. Network Management, Monitor & Control NMS solutions include interfaces to antennas, RF, IF, baseband and network devices, as well as software systems from the IT and terrestrial IP network services world.

Crystal https://crystalcc.com

Crystal provides automation software for monitoring, control and metadata management of the end-to-end broadcast through OTT/TVE. Key products used in the satellite/teleport space include:

Crystal Control M&C • Configures, monitors and controls equipment and devices to predict,

detect and react to problems. • Unified dashboard guides operators and engineers toward speedy

analysis, diagnosis and repair • Single, easy-to-understand interface regardless of what company

manufactured each device.

https://crystalcc.com/



• Similar-looking screens for similar devices; key important information presented to operators in a clean view.

• Can be deployed to in-house data infrastructure or the cloud to monitor and control virtual machines.

Crystal Spectrum Monitoring and Recording • Allows hundreds of transmission spectral segments to be periodically

sampled and monitored for user-defined error conditions. • Carriers can be continuously recorded or selectively recorded based

on triggering error events. • Modulator and analyzer-agnostic. • Flexible design for monitoring wide or narrow bands.

Crystal’s Video Metadata Analyzer • Ensures metadata accuracy by capturing and logging video metadata,

including SCTE 104 and SCTE 35 messages, along each streams’ transmission path.

• Checks for the presence of metadata, assures it is properly formatted, ensures it complies with business rules, and presents the analysis, errors and details in an actionable format.

DataPath http://datapath.com/cyber-software/maxview-management-software/

MaxView® Enterprise Monitor and control software solution for government agencies, leading broadcast organizations and satellite operators, installed and supported on more than 1,000 systems in 40 countries on all seven continents. • Provides a single, consistent interface to manage all equipment,

elements and service applications within a network across all locations.

• Enables large customers with thousands of devices and hundreds of sites to reduce operational complexity by having a single view into the network.

• Eliminates the need for separate proprietary management systems, minimizing operational complexity and cost.

• Provides for multi-platform, fully web-enabled access so operators can securely manage from anywhere.

http://datapath.com/cyber-software/maxview-management-software/



• Enhanced analytics engine for network performance insights, helping operators offer maximum uptime and improved SLAs.

• Features include: • Immediately detect and isolate service failures • Automate outage recovery • Provision circuits or services • Centralize control of remote facilities • Produce automatically scheduled and ad hoc reports • Secure access from virtually any web-enabled device • Upgraded analytics including interactive reporting feature • Built-in continuous learning through integrated multi-media

training • Cutting-edge user interface including customizable dashboard • Advanced scheduling capabilities • Expanded library of device drivers

Kratos® http://www.kratoscomms.com/products

Kratos provides enterprise-grade solutions across the ground segment, including: carrier management, RF interference mitigation, network monitoring and control, service quality management, satellite command and control, signal processing and protection, tailored antenna solutions and more. Relevant products include:

Compass® • With over 3,000 installations, Compass is deployed across the United

States and around the globe as a mission-critical monitor and control solution that assures equipment health and uptime.

Monics® • Comprehensive carrier monitoring and interference mitigation

solution that protects the spectrum and signal quality.

NeuralStar® • Centralizes visibility across hybrid networks to provide situational

awareness. NeuralStar is used by some of the most complex and security-conscious networks in the world.

http://www.kratoscomms.com/products



NeuralStar® SQM • Manages services across the entire network and tracks the health and

SLAs related to customers.

SatGuard • Provides VSAT monitoring to identify the terminal IDs of VSATs

causing interference in minutes to accelerate the remediation process.

satID® • Delivers very accurate geolocation results and helps pinpoint the

location of interference, including for VSAT/TDMA terminals.

SigX™ • The appliance cancels satellite and terrestrial interference in real-

time and restores QoS levels without the reliance on the interfering party.

SpectralNet™ • Digitizes and transports RF spectrum over IP networks enabling the

optimization of ground infrastructure to improve flexibility, reliability and reduce costs.

IPswitch https://www.whatsupgold.com

WhatsUp® Gold Network Monitoring Software: for cloud or on-premises visibility of the status of network devices, systems and applications. Network monitoring Layer 2/3 discovery of devices and any networked server. • Leverages topology-aware monitoring that understands network

dependencies, delivering fewer, more intelligent alerts, whether via SMS, email, web or Slack.

• Discovery and mapping tools cover individual IP addresses, seed scan, IP address, or subnets.

• Easy-to-add credentials including SNMP, Windows domain administrator, ADO, Telnet/SSH, VMware, JMX and SMIS.

• Scheduling for ad hoc or recurring discoveries. • Built-in, customizable device roles assign monitors based on device

type.

https://www.whatsupgold.com/



Network Monitoring & Alerting: • Network performance and availability reporting with powerful active,

passive and performance monitors, and intelligent alerting features and dashboards.

• Web-based interface makes reporting easy by quickly collecting, refining and delivering information for IT teams.

• Dashboard and reporting features include: • Sorting, addition or removal of columns with a single click • Timeframe or business hour filters (depending on report

context) • 'Pins' that apply filter choices to multiple reports • An overview dashboard that displays up/down status with red-

green bar charts • Flexible scheduling and email distribution • Report exports in multiple formats

Cloud Monitoring • Tracks cloud resource usage and billing and provide cost-justification

to management. • Automatically discovers, maps and monitors cloud environments

including Amazon Web Services and Azure servers. • Report and alert on the performance metrics on cloud service

providers’ API.

Skyline Communications www.skyline.be

Skyline Communication’s DataMiner® is an AI-powered end-to-end multi-vendor NMS, OSS & orchestration platform for media, broadband and network providers including satellite operators and service providers. DataMiner is deployed by more than 1,000 customers, and 6,000 systems are operational 24x7 spread across 125 countries in all continents. The platform has been embedded by at least two technology suppliers of VSAT hubs and terminals as part of their solution (OEM). DataMiner provides: • A unified and multi-vendor NMS integrating with any documented

API and providing – • A full suite of automated discovery of network elements and

software systems, with zero-touch provisioning on the platform

http://www.skyline.be/



• Automated software updates and configuration management workflows across all infrastructure elements

• Vendor-independent, multi-vendor management, control, monitoring and performance management of antennas, RF and baseband products; end-to-end remote terminals; video acquisition, processing and delivery; IP core and distribution networks; provide data center; and public cloud (multi-cloud).

• A powerful protocol engine, alarm correlation and RCA • Screens and visuals created on the fly by the operator using Visio • Built-in automation engine and editing for applications such as

redundancy, AUPC, carrier ID reporting, beam switching and network roaming

• Geographical maps integration with real-time and historical data • Advanced IP traffic assurance and analysis • Synthetic network and service testing • Built-in contextual dashboards and reporting

• Service booking and lifecycle orchestration for the booking, provisioning and monitoring of media and broadband services (e.g., OU) across physical and virtualized resources and any frequency band, transponder, transport system or network topology.

• Integration with OSS and BSS systems using off-the-shelf integrations, with M2M and bi-directional interfaces and operational workflow automation.

• AI-powered augmented operation using an unsupervised deep-learning engine to forecast, conduct intelligent fault detection and incident analysis, and generate adaptive dashboards.

WorldCast Systems https://www.worldcastconnect.com/en/c108p12/broadcast-networks/worldcast-manager-for-satellite-broadcast-networks

WorldCast Manager software centralizes and unifies monitoring and management of all connected devices across multiple sites. • Brings all the required modules within

one centralized interface. • Optimizes monitoring and maintenance operations with user-friendly

functionalities. • Network Monitoring Solution (NMS) and multi-faceted platform.

https://www.worldcastconnect.com/en/c108p12/broadcast-networks/worldcast-manager-for-satellite-broadcast-networks

https://www.worldcastconnect.com/en/c108p12/broadcast-networks/worldcast-manager-for-satellite-broadcast-networks



• Insightful data that is essential to their decisions and daily assignments.

• Enterprise- grade NMS & OSS system. • Autopilot Configuration Engine guarantees swift startup and fully

automates equipment discovery. • Graphical user interface for administration of the platform IT Service Orchestration, Operational Support Systems Cloud and service orchestration is a rapidly growing, multi-billion-dollar IT market that extends far beyond applications in the teleport business.

ServiceNow https://www.servicenow.com

ServiceNow is one of the world’s largest IT service management platforms. • ServiceNow® Orchestration increases enterprise agility by enabling

IT to automate manual tasks involving systems and applications outside the ServiceNow environment.

• IT can build orchestration workflows by adding various combinations of pre-built orchestration activities for Linux, Unix, Windows, and cloud services and custom orchestration activities.

• Orchestration leverages all the platform features, interconnected application processes, and single system of record in the ServiceNow enterprise IT cloud.

Video, IP and Baseband Quality Assurance Data quality assurance systems span the gamut, depending on vertical industry applications. They can range from broadcasting and OTT to oil & gas, defense and beyond. The vendors below are just a small sample of systems used by some teleport operators and their customers.

TAG Video Systems Ltd. http://www.tagvs.com

Specialized IP monitoring and high-quality multi-viewer solutions for video services. • Software-only approach provides state-of-the-art IP

monitoring and analysis tools. • Multiviewer from anywhere available on standard and mobile devices

displays.

https://www.servicenow.com/

http://www.tagvs.com/



• Software-based IP monitoring solution for all broadcast layers, OTT, UHD or HD video mosaic output with rich data and overlaid alerts.

• Digital video and baseband analysis, SNMP traps, syslog, email alerts with detailed event information and various notification types displayed on multiple mosaic layouts.

Telestream www.telestream.com

Telestream provides a large family of software tools and solutions for media and OTT service production and management, such as VantageVidchecker for automated quality control and correction of file-based media. Customer Booking, Media and Production Workflows Workflow management stretches from booking and transmission management to financial functionality.

Net Insight https://netinsight.net

ScheduALL®: Two decades plus of satellite industry operations. • Booking system popular for

Occasional Use (OU) services that interfaces with accounting, human resource and ERP systems, creating a 360-degree cost control that increases business efficiencies.

ScheduLINK®: • Transmission management and optimization software used by

leading satellite bandwidth providers, television broadcasters, cable networks, mobile uplink providers, teleports and global news organizations.

ScheduALL Circuit Selection™: • Tools to configure transmissions, manage occasional-use (OU) and

bandwidth of complex inventories for terrestrial networks.

ScheduALL Portal™: • Simplifies booking activities, personnel, and other services (such as

OU transmission feeds), by allowing customers to book packaged services or individual resources through the ScheduALL system in real time.

http://www.telestream.com/

https://netinsight.net/



NIMBRA VISION • Provides end-to-end service orchestration so network operators get

a complete picture of the network and service health and performance.

Xytech Systems https://www.xytechsystems.com

MediaPulse creates a seamless environment for media enterprise and transmission workflows, with separate planning, scheduling and financial management tools, reducing redundant data entry and reducing touchpoints for all workflows. • Brings parameter-based management to issues such as transponder

use, antenna use and transmission path engineering plans. Teleports can have an end-to-end management and operation ecosystem integrating with Miranda, Net Insight, Avid, Crystal and other vendors in the transmission space.

• Circuit profiles and specifications allow automated creation of multiple circuits in an order, based upon user configurable parameters in an easy-to-use, modern interface.

• Mobile and cloud-ready.

MediaPulse Transmission: • Operation through automated feed management, device automation,

bandwidth utilization and accurate billing all delivered on any device connected to the Internet.

Terminal/VSAT Management Field-ready technology designed to simplify and speed installation.

Integrasys https://www.integrasys-space.com

Satmotion Pocket VSAT Auto Commissioning • Integrasys Satmotion Pocket is the

unique VSAT auto-commissioning system for minimizing the deployment time, effort and interference intra-satellite such as crosspol and inter-satellite such as adjacent satellite interference (ASI).

https://www.xytechsystems.com/

https://www.integrasys-space.com/



Integrasys Controlsat Carrier Monitoring: • Allows quicker reliable OPEX savings while ensuring quality and

interference-free service. • Fast carrier and spectrum monitoring. • Efficiently manage and operate NOCs worldwide in real time • Easy-to-use interface for saving opex. • Allows fully automated monitoring for multiple satellites and beams

remotely from the main NOC.

Integrasys Alust, an evolution of Integrasys’ Satmotion Pocket technology which allows accurate installation during remote commissioning. An automated network maintenance system minimizes maintenance time, effort and interference. The commissioning system saves up to 88% of the time for VSAT engineers and 90% of NOC support time by automating the commissioning, peaking and pol without NOC calls from remote sites. • Allows installs to be scheduled more quickly so sites can activate

faster, e.g. from hundreds of sites/month to thousands of installs. • Consistent monitoring of a VSAT site after installation to ensure

optimal operation and the minimization of costs caused by service failures and site re-visits.

• Virtual site visits. • Can recover out-of-service or service-degraded terminals. • Operator can set thresholds for Rx and Tx and automate subsequent

system command reactions to rectify problems. • Checks selected terminals, analyzing the Rx and Tx measured values of

co-pol power, cross-pol isolation, adjacent satellite interference and 1dB compression point to detect failures and raise necessary alarms.

Integrasys Satmotion SNG: • SNG auto lineup system for minimizing the occasional use access time,

effort and interference intra-satellite such as crosspol and inter-satellite such as adjacent satellite interference.

Integrasys also has link budget automation tools to simplify the link budget for GEO and is adding a LEO product.



What Do Operators Need to Know About Automation? Automation systems can reduce outages and improve QOS – or they can risk creating large, single points of failure. Engineering design and careful planning of implementation are critical to success. On a high level, automation presents an opportunity to rethink aspects of your operations from the NOC to terminal provisioning, and to achieve new levels of efficiency and quality. Our respondents offered observations and strategies for achieving success with today’s automation solutions and using them to support adoption of new opportunities. A Show That Never Ends Automation of operations is a continuing process, not a one-off project. “It is an evolving process as a facility changes, adapts, grows and shrinks, and the monitoring and control must go along in lock step,” explains a CTO. “There is a great deal of care and feeding. You must make sure tools are provided, updated and returned as operations evolve. It also means making sure your documentation and procedures all flow as part of the automation process.” Proprietary vs. Open Some M&C platforms are proprietary and only work with specific technology. “Equipment-specific NMS or M&C systems can have simple and easy-to-use GUIs and graphics,” says an executive, “but in return, you have a system that only controls a particular piece of equipment and that usually runs proprietary software.” Others are open, multi-vendor platforms allowing anyone to integrate new APIs and create custom views from that data without additional coding. The risk inherent in these platforms is that, with all information centralized in a few dashboards, there can be too much information and complexity for operators to manage. “Many satellite systems have features you do not use, and an open system feeds all of that into your NMS or operator interfaces, so you then need to sort through what’s relevant and not,” says a technical manager. “We use Open Source NMS systems and customize them to the look and feel we need. We also keep up with specialty NMS systems. Ideally you want an NMS customized to your earth stations, and one interface for everyone, but the cost is extremely high to customize. Any change then requires a team trained on the custom system to be retrained, adding training costs.”



Another operator found that the value of open systems far outweighs the requirements they place on the organization. “Our vendor applications need to be open, not fixed source code. We want full API connections between all applications to avoid garbage in/garbage out,” says the operator’s IT leader. The teleport then uses RPA (Robotic Process Automation) and screen scraping to take information out of its older system for input into a new system.

An IP Media World For media-focused teleports, the industry has changed from monitoring a video wall with hundreds of thumbnails to a data-driven understanding of how services are performing. “Today, clients are looking at carrier, video, audio levels and only call up individual feeds that have alarms or problems,” says a media teleport executive. “Now QOS is a measure that can include capacity available to stream, whereas traditionally M&C might only measure present channels that violate alarm parameters.” Today, tracking streaming video end-user QOS requires a measurement and monitoring ecosystem and third-party technologies far beyond the satellite, but which still may need to be integrated into your operation.

IT, IP and Cloud Orchestration For a growing number of operators, the next step is about moving beyond command-and-control of the network and adding orchestration of cloud services and applications to support virtualized operations and services. For example, a large European operator is now extending a major vendor’s NMS system to design and automate new services such as cloud-based



solutions. This will allow them to manage legacy infrastructure with new cloud services in a single system. (See WTA’s report Clear Skies or Stormy Weather? Cloud Services for Teleport Operators.)

It can also mean integrating the OSS (Operational Support System) and billing systems with network management. One executive reports working with traditional hardware providers to virtualize their encoder technology. At the same time, the operator wants to make the most out of Microsoft Azure and Amazon Web Services for flexible cloud service and storage infrastructure, and open-source apps that support virtualized environments for services, so that it can minimize costs for startup and migrate customers to an on-premises implementation if they grow.

“Going open source means you can use the best apps such as Puppet, Lamba and S3 Services offered by Amazon,” explains a technical executive. “Puppet, for example, lets you reduce the cost of maintaining your servers and automate the configuration and management of your infrastructure. With this next-generation open-source server automation tool, you can discover resources within minutes, provision new nodes easily in cloud, hybrid or physical environments, configure a range of setups, and orchestrate changes and events across clusters of nodes.” Operator Feedback on Specific Systems For this report, we asked operators to share their views of today’s automation solutions with respect to:

• Completeness of the technology in meeting their needs • Benefits offered versus initial and running costs • Ease of use and management

The teleport and satellite operators we interviewed all ran at least one of the “big four” M&C and NMS: Compass, Crystal, DataPath MaxView or Dataminer.

Compass • A large satellite operator standardized on the Compass solution for

centralized management of all the modems located at its teleports globally. The Compass system is used for carrier line-up and as a principle method for troubleshooting circuits by the operations staff. Compass is also used in this dynamic environment, where customer services are brought up and taken down rapidly to track the modems and current usage throughout their facilities. The operator also uses NeuralStar SQM to centrally manage all the Compass systems



throughout the teleports to provide visibility into the performance of their customer services, and monitors the RF spectrum and combats cases of interference using the Monics carrier monitoring system and the satID geolocation system.

• For a large cable provider Compass is used at two sites that package and transmit broadcast data. The cable operator uses a distributed architecture where a number of Compass remote site applications are located in the different equipment rooms. Compass is used to physically connect the equipment into the management system across a high speed LAN.

• Compass provides M&C capabilities for this satellite operator to ensure the uptime and health of the mission-critical equipment across the teleports from a central NOC. Using Compass, operators monitor and control equipment, change displays, add devices and perform configuration changes with the GUI. The monitoring staff is also able to eliminate recurring services and configuration issues by automating tasks to simplify operations.

• This regional satellite operator uses Compass to interface with EPOCH, a real-time satellite control software suite from Kratos. With this integrated solution, operators are able manage a single, multiple or an entire constellation of satellites as well as the breadth and scale of the equipment in the ground station. Compass manages the ACUs, baseband systems and a range of other equipment as well the element management systems located at the facility.

Crystal

• A European media-focused teleport has deployed Crystals NMS to monitor many hundreds of broadcast satellite channels, and to provide training, updates and support. The system monitors and controls thousands of devices, including automatic redundancy switches (which are subject to manual override) for fail-over protection. To achieve a complete view of its systems, the operator has created a custom workflow and ticketing system that manages engineering and operations staff work orders, change orders, assets and purchases. This integrates with both the Crystal system, and its IP networking equipment, which is outside of Crystal and controlled separately using WhatsUp Gold Network monitoring.

Automating with a combination of Crystal and WhatsUp Gold allows the operator to monitor more channels and IP data with fewer people. “Crystal could in theory take over NMS and ticketing and



segment controlled by WhatsUp Gold, which could then report to Crystal NMS, so an operator could view all three using one GUI,” according to the CTO. “That would make a master NMS the sole point of monitoring, from linear broadcast over satellite to OTT, VOD, data services and IP trunking. It would let a single operator oversee network origination transcoding, and transmission.”

• Another provider delivering IP services to Africa and elsewhere over VSAT uses Crystal to monitor teleport equipment (ACUs, TWTA, SSPAs, modems, modulators, and other gear) with SNMP. “The GUI was impressive, giving a breakdown of each earth station, IFL and chains, so you can click to each antenna flow diagram and you have communications access to each device.”

• A U.S.-based operator uses Crystal’s Carrier Monitoring solution, which it recently installed, and is working to attach more devices to the system for M&C. “But this requires drivers to take the MIBS, then understand and process data. There is work to be done for each vendor to integrate. If the vendor does not have a required driver, you must build it and put it in the library,” says the operator’s executive. Crystal reports having developed a broad library of drivers available to its customers.

• For a facility with operations in Europe, Crystal enables remote access so that unmanned and managed earth station sites can be managed by the same system. The teleport uses its remote spectrum analysis systems with web GUI access managed through with the analyzer-agnostic Crystal carrier monitoring system. “That’s a big benefit for us, having an LPT analyzer and to be able to remotely see it without people on site.”

• An operator active in global mobile said they have moved away from Crystal’s spectrum monitoring solution after being one of its first buyers years ago. “We spent a lot of effort back then, using two to three man-months of labor to make their product work the way we wanted it, which helped them make the good product they sell today. But it did not work out of the box, so they worked with us to get all the bugs out.” Making up for that effort, the executive reported that he was charged substantially less than the list price.

Datapath MaxView®

• An operator of multiple teleports combines Crystal’s carrier monitoring with Datapath Maxview for M&C of its RF hardware



systems, providing alarms for any failures or problems at two teleports. The new Maxview system gives end customers monitoring of network status information that they desire, while allowing the teleport to isolate potential faults faster, before customers see them or call.

• On the other side of the globe, an operator uses Datapath Maxview for M&C of its satellite networks and terminals and has integrated the system with its OSS and Business (BSS) automation platforms.

• A U.S.-based teleport had been using the Maxview system in the past but has had upgrade problems. “Following the upgrade, after a week, the entire system became non-functional, so the upgrade was like hav-ing to do a new installation,” says the operator’s technical manager.

Skyline Communications – Dataminer®

• A provider of mobile and aero services, which has previously used Datapath and Crystal systems, has now used Dataminer for the past several years. “We are pretty happy and it is very powerful, but only as good as the resources you put into it. You must have the resources and staff to customize it, improve upon it and maintain it. Some systems are supposed to run out of the box: the reality is you need to work with it enough to be able to make changes on the fly and analyze data to see your performance.”

• An Automatic Trouble Ticket feature needs to be carefully thought out before activation. Two operators using Dataminer’s powerful ticking capabilities observed similar experiences: that the system can very quickly overload operators with unimportant alarms if it is not configured carefully: • “Testing Dataminer, we found the network had a problem and it

bombarded us with hundreds of emails. For everyday stuff it



plays nice, but until you stress test the system, you don’t know what will happen,” says the operator’s tech chief.

• Another operator of a media-focused teleport uses Dataminer for equipment, weather and temperature monitoring and initially had hundreds of non-essential errors and false alarm emails. “Monitoring is a slippery slope. You can monitor many things, but to properly automate you must monitor what is relevant.”

• A media provider using Dataminer found that “they require a subject matter expert to optimize on their system. We now have a person who looks over the systems.”



The Cost-Benefit Analysis for Automating a Process

When you do the cost-benefit analysis for automating a particular process or set of processes, what factors should you consider and what tips the balance toward automation?

Classic reasons for automation are to streamline repetitive, repeatable tasks and processes. The payback can be savings in labor time, headcount, customer refunds and expenses, as well as accuracy that improves quality of service and customer satisfaction. Automating processes also enable volume and scale not possible with manual methods.

The justification for investment in automation systems can be based on a single variable or multiple factors. It can be driven as a strategic business initiative to outpace the competition, or as a “me-to” justification for competitive survival. Operators and vendors shared experiences and cost-benefit considerations for a variety of automation projects. M&C Systems The payback for M&C systems may be hard to quantify since the payback is in performance and mean-time to restoration rather than cost savings, says a teleport executive. Areas worth considering in your cost-benefit analysis may include:

• Elimination of time-consuming manual processes in provisioning, monitoring or reacting to services.

• Refund or credit cost avoidance: If customer refunds are a material part of the business, better monitor and control has the potential to reduce them.

• Quality of Service: automation can enable higher QOS, higher customer retention, and can protect or support increased business.

One software supplier worked with a teleport operator to create a business process plan for streamlining tasks across ten NOCs. The ROI calculations included the cost of managing trouble tickets, alarms, customer calls and field interventions; the labor spent on calculations for SLA reporting; and operational time spent on inventory management. The analysis estimated that the automation project could easily reduce costs by up to 50%.

“M&C systems out of the box can cost from US$25,000 to $300,000, depending on the type,” says a teleport technical executive. “What is the



value of our revenues protected by a system? Does it prevent long outages? Does it improve our restoration time and keep us close to 5 nines? I would rather have systems telling me I have issues than customers calling. That was our decision-making process.”

Another operator holds a similar view: “You have to consider customer satisfaction, QOS and response time to issues. Also, automation allows fewer people to optimally monitor more channels and IP data, so it saves money on labor. But we have concluded that the calculation of ROI or break-even does not matter. Our strength has been extremely fast turnaround and first-class support across the board for our customers.” Trouble Ticketing and IT Support Procedures Manual ticketing processes can take hours, which automation can turn into seconds, while ensuring that trouble tickets are accurate and complete by automating the workflow process. The savings in labor and process improvement are substantial, according to respondents. That is why support functions have been automated across most IT departments. Booking and Asset Management Systems Reducing the time required to book a circuit from New York to Tokyo from an hour to 5 minutes could save US$50 a booking, according to one respondent. Booking errors that require last-minute re-routes can add 5-10% to operating costs. How do those costs compare with the cost of the system and staff resources required to perform tasks manually?

Without a resource scheduling and asset management system, what is the cost of under-utilization of your inventory, be it satellite or ground facilities? Unused transponder capacity is not cheap, and making available bandwidth more visible to potential buyers can increase ROI.

Some teleports are implementing the next level of booking automation with customer self-booking portals, where customers input their own orders according to a prescribed logic and workflow, typically via web-enabled systems. This saves the labor that may have been required around-the-clock to take orders.

Teleports can achieve faster, less error-free provisioning with reduced labor time by automating provisioning, and integrating NMS and asset management capabilities, along with ordering systems, all the way to the end-user. One tech supplier looked at the potential benefits of automating service lifecycle management (spinning up new services, changing services, automating OU sessions for sports news and file transfer) for a teleport provider. The analysis showed that 60-80% cost



savings could be achieved by enabling customer self-provisioning on the teleport’s infrastructure in an automated way.

Flickr Syarifah Riefandania VSAT Cloud

Remote Site Installation and Management “Automating the field install process and integrating it into provisioning and billing and reseller requests drove down our break-even point,” says an executive of a VSAT-centric teleport. “We can now activate hundreds of terminals per day versus 8-10 per day manually, so from a business growth perspective, it was a no-brainer.” The operator’s ISP customers access a TICS (Terminal Information Certification Server), which enables terminal activations remotely. “We integrated the BSS system from multiple locations in Europe and Africa into our Customer Portal. The TICS has APIs into the billing engine and also connects to the NMS. It was easy to justify the integration; when we started VSAT services, we had operators taking 45 minutes to align, crosspol check, and manually coordinate between the NOC operator and field installer. It required higher installer skills, costing more in staffing for site activation.”

The cost-benefit analysis for a remote VSAT site management system can be based upon freeing labor hours, increasing customer satisfaction, and time-savings in response to customer calls, instead of just looking at headcount. A key automation software supplier suggest that you look at the following costs to deploy large numbers of sites:

• How many calls to the NOC per install?



• How much 24/7 support is required remotely and at NOC? • What is the cost to visit and correct a remote site fault due to poor

installation, including travel expenses and labor hours?

The payoff also depends on your scale. For example, one teleport provider found that “investment in a cross-pol and antenna pointing for VSATs could be $100K and higher for our requirements, so we need many users to amortize it. Our Inmarsat and mobile systems are very automated, but since most of our VSAT networks are limited in the number of sites, we handle those manually, for now,” says the teleport operator.

For more information, see the WTA report Understanding and Improving the ROI of VSAT Networks. Value Creation A tech supplier offers a word to the wise on the cost-benefit of automation projects. “It always seems to take longer and cost more than expected, but we also under-estimate the value because you do not see the opportunity cost savings of freeing up staff to do new things, things you never even thought about. There is a huge under-appreciation of the value that automation can create.”



Lessons Learned – The Hard Way “Garbage in, garbage out” is one of the oldest rules in computer processing. What are the lessons operators have learned in automating mission-critical processes? What works and what is hype? Respondents shared their hard-won experience. Identify the Desired Outcomes Ask yourself if you really need the automation system, as obvious as that may seem. Plenty of companies, including large ones, invest in tools and think they will use them efficiently, but wind up with a multi-$100K system that is only used to perform work that could easily be done with spreadsheets. Avoiding this outcome means developing a clear understanding of what will be automated, the manual work that it will eliminate and the manual work that will still be required. The processes to be automated must be sufficiently understood and documented to be turned into code, and the reduction in manual work must be sufficient to justify the time and expense of deployment. User-Friendly Interfaces are Important User-friendliness is not a “soft” benefit – it can spell the difference between a system that minimizes operator errors and one that actively generates them. User-friendliness is even more critical when you will allow customers access to the automation system. A confusing or even unattractive platform will fail to gain traction with users accustomed to watching video on Netflix and shopping on their mobile phones.

“Great software should require little or no user training,” says a tech executive. “No one needs training on Amazon. Well-designed software should guide the user to fix issues. The more complicated to learn and use, the less likely it will be adopted. “ Carefully Map Human Interaction Automation pull tasks out of the labor chain – but human interaction is still required to input information or respond to the system’s output. Any human interaction can create a dead-end point for the system, and these must be carefully thought out, drawing on the expertise of future users. No automation system can anticipate every situation, so there is also a need for systems to permit human intervention in special operational or



financial circumstances. Why and when will a decision to do things manually be made, based on human knowledge of the service, the customer or the company? In live broadcast services, for example, “To restore a service, I don’t want to flip between screens or find checkboxes,” says a technical executive. “I want one physical restoration switch. It could be activated by a mouse click as well, but when you need to restore a service fast, you want a physical button rather than searching for the right window on a screen.”

Map the Logic Behind the Software You need to retain a higher-level understanding of the system’s logic and sustain that understanding in your organization. For example, it is important to understand the actions that should be triggered when an alarm goes off or an outage occurs. Automation should not replace operator awareness.

“Automated monitor and control should give you an awareness of what is going on in the facility and whether or not events are occurring properly,” says a CTO. “But they do not exempt staff from understanding the underlying processes and being ready to intervene manually if needed.”

Ironically, after processes are automated, you may wind up with less access to engineering staff support in an organization. Therefore, individual network operators on staff may need to understand more deeply than before how the systems are being controlled and managed, and what their capabilities are. Understanding and Validating Your Data Converting manual or semi-digital processes into automation puts a premium on data quality. Cleaning and validating your databases is vital –not just for legal, regulatory and security reasons – but for ensuring that automation produces the outcomes you want. Open systems and APIs can share data but not ensure that the data is interpreted properly.



“Assuming your configuration is correct can be a pitfall,” a supplier advises. “It will repay you to make the upfront effort to ensure the quality of inputs.” You also need a thorough understanding of your databases and data sources to automate effectively. One executive gave the example of the detailed knowledge needed to properly automate monitoring of an IRD.

“In monitoring an IRD, data sources could include the receiver’s S/N reporting, a local radiometer measuring atmospheric conditions and a satellite beacon signal,” he notes. “Let’s say that one of the data sources alarms with bad input. That could lead the system to perform the wrong failover action. So, you need logic that validates the data condition against the two other data sources. If the alarm information is not valid, you could then pause and decide what to do, instead of switching receivers due to a bad measurement.” Respondents advise focusing on these aspects of data preparation: Metadata Taxonomy “The first thing you need is a comprehensive and vetted metadata taxonomy, or you won’t know where you are,” says a software executive. “You probably have master data for the automation system scattered in multiple systems, from booking and accounting to systems configurations. Your metadata taxonomy identifies what systems own each part of the data, so that the automation software can talk to the right systems and confirm data choices. Many clients miss that step.”



Nomenclature One of the biggest pitfalls in automating is names. People can use the same words to mean different things. In combining systems and allowing data to be shared among them, step one is to get make sure that every product, endpoint and element of the systems has a unique and unambiguous name. This must be agreed across teams. On the systems side, the nomenclature allows you to write translator software that allows already-installed software to accurately interface with the new one. Taking Out the Garbage For a client portal, the software has to be smart enough to filter out bad data as it is being entered. “It is crucial to properly scrub data upfront before it goes into the system, or you are wasting your time,” says a vendor. “You must clearly understand what data needs to be placed into an order screen and the best way to scrub it.” Preserve Historical Data For applications like monitor & control, the ability to take an historical look at your systems is essential. “We look back at incidents and outages to see what happened first and what followed it,” says an operations executive. “For example, power issues can cascade into other problems. Historical data can show event signatures and help you identify root causes and address them.” Internal Alignment and Resources Organizations can drag out the planning and deployment of an automation system for years when there is no internal alignment. This happens due to fear of job losses, pushback from people who fear they will not be re-trained, internal politics or simple inertia. If you are going to invest in automation and tools, make sure your organization and affected stakeholders are prepared to make it work by addressing their concerns and aligning their incentives early in the process.

All systems are only as good as the internal resources you put around them. It is unwise to expect a vendor to install a process automation system and then walk away. People who will be using the system must have input. “It is really important that systems are fully tested, that staff is comfortable, trained and have the resources to get help if they need it,” an operator CTO emphasizes. Ongoing maintenance, care and feeding come with the territory. You will still need highly skilled staff to deal with bugs, crashes and network blinks. “Do not take it for granted,” advises a technical chief. “Whatever



your technology, you need to apply the basic rules like skilled staff checking and taking measurements correctly. It is easy to become complacent with the technology, but you do so at your own peril. If you are running a system without developers today, you are definitely behind the curve and putting your business at risk.”

“Automation is meant to make things more efficient,” says a CTO, “but success means you can also risk losing expertise and knowledge, as fewer people retain the know-how and understanding of the process being turned over to the automation system. This is a very important fine line to be maintained.”

A software vendor goes so far as to argue that not organizing teams into a “Dev-Ops” style of operation is a pitfall. “People must understand there is a need for continuous integration and continuous delivery in today’s teleports, which have largely become data centers running applications on premises and off premises in the cloud.” Compatibility with Other Systems Automated systems should work well with others. “Given the large number of disparate systems teleports work with for their baseband equipment, it is paramount to review those systems’ abilities to interface with your core automation network,” says a global operator executive. “Creating APIs for each of those systems takes time and money.”

“Vendor applications need to be open, not fixed source code,” says another operator. “We want full API connections between all applications to avoid Garbage In/Out and full B2B API interfaces.” Testing Thorough, even exhaustive testing is essential before you buy and deploy. “You must stress-test systems and play scenarios like when things go really bad,” says a teleport operations and engineering leader. “How will it perform then, and will it flood the system with alarms? Make sure it runs under the most stressful situations and does not overload operators with massive amounts of information they cannot process.”



How to Avoid Deployment Nightmares Teleport executives and tech suppliers offered a number of recommendations for staging the implementation of an automation system, from process to people.

1. Plan It the Right Way

An effective automation plan includes all human interactions, business systems and decisions required. As one contributor put it, “If you cannot document and articulate it clearly, there is a problem.”

Everyone who will have a role in the success of the system needs to be involved at some level in the planning process. The more input from qualified people, the greater the chance of success.

In planning with the team during design, work through scenarios and logic for the most common actions to be taken. This will also reduce training time, because operators understand the logic for an automation event, how it should work, and where to look to fix or recover. Following this approach for example, one vendor separates the logic and code for the part of its system that reads and measures from the part that takes action and executes decisions. This allows operations staff to manually override actions or test scenarios without interrupting real services.

Before proceeding, develop a business case for a simple automation, or a business plan for larger organizational transformation. Have a clear understanding of the desired end-result, having gone through the cost-benefit analysis. Make a clear plan for



each step and every milestone’s success criteria, or there is no point in proceeding.

2. Succeed One Step at a Time

Break the project into manageable bite-sized pieces and prioritize easy wins so teams can buy in early. Some parts of an organization will always need to see value before supporting changes.

Initial steps for automation should be assistive in nature, helping people do their job rather than reducing headcount. Small steps and milestones also let people easily get up to speed on that part of the process, instead of trying to massively train on a large new process.

A software supplier showed one operator how to cut a 17-minute process down to 10 minutes with software to help operators execute a manual task. Once the software managed that task, the project leaders were able to move to new phases that eventually reduced it to 1 minute, and the group embraced automation further.

3. Train for the Right Results

Any significant process change usually means skills need to be upgraded. All operators and suppliers we interviewed agree that it is essential to invest in training and skills with automation projects. Experienced operators advise focusing training spend on admin users. They are the ones who will need to handle software and business changes as your operations evolve, so that you can have less dependence on vendors.

“We have multiple changes on a daily basis, so a skilled staff is absolutely key,” says a CTO. “It is very important to ensure engineers and operators are properly trained. Engineers need to know how to make system changes, and operators need to know where to look for service degradations, so we have invested heavily in training at the operations and engineering level.”

Automation often removes first-level support tasks, increasing the importance of more highly skilled and trained technical support. Better results come from training people rather than replacing people. Seek to harmonize skills and disciplines (IT, RF, engineering) so your team’s skill level increases.

A media services provider explains his organization’s strategy behind training. “Our approach is to take skills from IT and merge them with the reliability and quality of engineers in the broadcast environment, so you have best-in-class operations. Rather than



taking over one from the other, our goals is to harmonize processes: the best in IT practices and best in broadcast practices.”

Using IT Transformation to Fund Re-Training One operator successfully staged a 3-year program that reduced electric bills by over US$1 million by upgrading and transforming its RF and data center and moving from around 100 racks of equipment to a consolidated IT core and cloud-based system managed by automation and OSS. Instead of managing 400 servers, it upgraded and consolidated the IT load to virtual machines onsite, and OSS/BSS in the cloud, gaining major infrastructure opex savings. With the savings budget, the company re-trained its VSAT engineers to run and manage the virtual machines and to develop and architect the OSS/BSS on its services platform. As a result, it needed no new people to deploy its powerful new service platform. “Now the newly trained folks, who work on agile software development with scrum masters and project managers, can move into design and operation environments. The newly trained folks understand the satellite network, and installation of broadcast and VSAT services, but they can also write the story and process to program what was once manual.”

4. Be Agile

Several operators advocate the Agile development process, which can quickly develop and deliver a solution, then rapidly improve it through testing and user feedback. 1

“This is my biggest recommendation,” says a media workflow automation supplier. “It puts software in users’ hands rapidly, knowing you’ll need to change it, but gets you closer to the reality of the job faster. Never think of it as a one-step process like hardware.”

“For a new service you can test-market first, then build automation later when scale is required,” says another automation vendor. “First you work on order capture and billing. When you get an order, you can still do the provisioning manually. Then if demand grows, you can follow with a flexible Agile process to automate provisioning, installation and service assurance.”

Agility requires continuous testing. Just as you tested the software before purchase, it makes sense to continue to have and use a test environment for ongoing product design and testing of upgrades, extensions and integrations, and for training.

1 https://www.amazon.com/Lean-Agile-Software-Development-Achieving-

Enterprise/dp/0321532899

https://www.amazon.com/Lean-Agile-Software-Development-Achieving-Enterprise/dp/0321532899

https://www.amazon.com/Lean-Agile-Software-Development-Achieving-Enterprise/dp/0321532899



What Do Operators Want Next from Automation? Looking to future needs, operators reported interest in more unified management, open interfaces, easier integration, tools for hybrid cloud management, and new technologies like artificial intelligence. Unified NMS Architectures When it comes to operating multiple M&C systems, trying to wrap another M&C system around them becomes very challenging. Operators say they want scale and unified architectures to manage their devices and networks, from SLA and QOS to service management, commissioning, lineup, and cloud service orchestration.

“A single NMS with all systems is where we would like to go,” says the CTO of a network with hundreds of video services. He envisions a system where operators could view the NMS, ticketing and IP network segments (each a different system today) using one GUI. This would allow operators to oversee network origination, transcoding and transmission. The vision calls for doing the same for data and IP trunking: managing, monitoring and controlling multiple beams and channels on satellites with support for DVB S2X up to 500 Mbps services that can leverage HTS satellites.

NMS vendors seem to hear this demand even if perfect compatibility may never be achieved in reality. A leading NMS system says they are working on compatibility, “not only within our own products but also making it very easy to integrate with other products, because we know ground system operators must run our systems alongside other systems. Our intent is to support the concept of getting to ground systems that operate as a single system although made of many different systems.”



Virtualized Environments and Common Platforms Another goal some operators share – and are in various stages of implementing – is to build architectures that help teleports support virtualized software environments to enable better data-center economics and operational efficiencies from the playout to CDN delivery. “We would like to move to virtual environments and common platforms, so we have the ability, for instance, to stand up a VM for a firewall,” says one operator. “This adds a level of agility and scalability that helps keep costs down.” Service Management and IT Orchestration Tools Software suppliers continue to add APIs, allowing easier exchange of data between applications, but more time and work is needed to allow seamless integration of control systems, operators say. Operators would also like to see is more tools to help them leverage openware applications, which save development costs and potentially speed up processes.

NMS supplier roadmaps call for delivering these capabilities. “The biggest ROI from automation is orchestration of infrastructure and services” says one supplier, “and this requires that the automation engine can read and control any component of the infrastructure.”

“Operators should expect to see the number of services they are responsible for grow dramatically,” says an M&C supplier’s executive. “This is happening in cable and broadcast. We are seeing operators building larger networks, where the number of services is exploding. In the next few years, we expect to include more service management, provisioning and monitoring services in our suite.”

“We are investing in orchestration of operational and tech workflows,” adds another NMS supplier. The road map includes creating more efficient ways to integrate with industry software and hardware components, including an open integration tool to make it easy to include any type of product and API into the system, more service resource management and life cycle workflow automation tools, for blending on-premise, public or hybrid cloud services. AI and Machine Learning to Manage Complex Data As operation of hybrid satellite multi-cloud networks grow, the amount of network information outpaces the capacity of people to manage it. Artificial intelligence (AI) or machine learning could help by running many calculations and checking the patterns of equipment and software status. Examples of applications include intelligent dashboards that use AI to



define what dashboard the user requires at any moment and that perform automatic root cause analysis, learning as it goes.

“We have a roadmap and team working on automatic operations, based on deep learning and unsupervised learning,” says a vendor executive, “to make it easier for operations teams to work in more complex environments and more terminals. As software services become too complex for operators, Artificial intelligence can help.” End-to-End Automation Another area on product roadmaps is further integration with back office systems: integrating with IVRS systems, plan maintenance systems, CRM, inventory databases, topology databases, help desk systems via all data exchange (M2M) and associated workflows. The intent is not just for improving technical workflows, but also automating operational processes that can flow into Operational Support Systems (OSS) gateways and Business Support Systems (BSS).

New IP Media Protocols A technical development that could offer big opportunities for improved automation of media services is Networked Media Open Specifications (NMOS) protocols. (https://www.nmos.tv).

NMOS is an IP media networking family of open specifications in development by major media companies and tech suppliers to make media standards easily interoperable with IP standards and protocols.

https://www.nmos.tv/



With IP NMOS, instead of a booking system talking to a network M&C system, this protocol would allow a booking system to talk directly to devices and also monitor the status of circuits, and alarms. “That’s a gigantic step forward, so you can look at nailing a circuit up directly by directly booking the system and monitoring it all the way as a user,” says a software supplier involved in the standards effort for NMOS. By combining NMOS with AI, he sees even more potential: the machine sets up a service, such as an uplink to a satellite with delivering to a control room. “NMOS and AI lets you create a catalog of possibilities and let it choose the best path. A client could go to a portal, and software knows who they are, what they bought, and what circuits are available, so the staff only needs to review, edit and approve instead of provisioning. That impacts headcount as you need fewer people and allows our teleport clients to work with their customers in a much more transparent manner.”



About the Report Automating the Teleport was developed under the direc-tion of WTA’s Research Committee, led by Serge van Herck. Dan Freyer of Adwavez Marketing conducted the research interviews and wrote the report, which was edited by WTA Executive Director Robert Bell.

About the World Teleport Association Since 1985, the World Teleport Association (WTA) has focused on improving the business of satellite communications from the ground up. At the core of its membership are the world's most innovative operators of teleports, from independents to mul-tinationals, niche service providers to global carriers. WTA is dedicated to advocating for the interests of teleport operators in the global telecommunications market and promoting excellence in teleport busi-ness practice, technology and operations. Members benefit from the opportunity to: • Collaborate for Mutual Benefit, from maintaining a level playing field

for competition to implementing management practices that reduce costs.

• Network Within the Sector, to identify business opportunities, stra-tegic partners and market insights.

• Improve Their Global Profile, through WTA-hosted events, listings in WTA’s buyer’s guide and placement in WTA’s publications.

• Raise Their Competitive Game with free access to WTA research, white papers and market studies.

World Teleport Association +1 212-825-0218 [email protected] www.worldteleport.org

van Herck

Freyer

Bell

Date post:	30-May-2020
Category:	Documents
Upload:	others
View:	15 times
Download:	0 times

Automating The Teleport - Kratos Defense/media/... · For Automating the Teleport, WTA invited...

Documents