Simplified OSS / BSS Stack
OSS
BSS
Network and Services
CustomerOrder
Order Mgmt
Provisioning & Activation
Service Data
Monitoring and analysis
Billing and Reporting
Bills and Reports
[Build Slide]
Orchestration is responsible for service provisioning and pushes state to the infrastructure
The “C” in FCAPS
OSS analytics is responsible for collecting data from the infrastructure, monitoring and analysis
The “F_APS” in FCAPS
OSS AnalyticsOrchestration
OSS Analytics is becoming a big data problem!
Engineering effort (time)
Perfo
rman
ce
Big data analytics-based
Small data analysis
What changes?
PMO FMO
Orientation Single domain Cross domain
Realisation Small data, tool driven Big data, data driven
Data aggregation and analysis
Coupled Decoupled
Domain Data Schema Scheme-on-write Schema-on-read
Analysis Prescriptive Prescriptive + Stochastic + ML
Customisation Design time Run time
• Tight coupling of data aggregation/store/analysis
• Multiple analytics pipelines implemented from open source components
• Common design patterns ~75% of effort wasted / duplicated
• Siloes limit the potential of big data analytics and lead to industry divergence
Today’s siloed analytics pipelines
Telemetry
Metrics
Data sources
HDFS
Data store
Spark Streaming
MapR
Data analysis
Hbase
Storm
Kafka
Streamsets
Data aggregation
Kafka
Impala
Query
Outputs
Dashboard & ReportingNiFi
Logs
What is PNDA?PNDA brings together a number of open source technologies to provide a simple, scalable open big data analytics Platform for Network Data Analytics
Linux Foundation Collaborative Project based on the Apache ecosystem
• Simple, scalable open data platform
• Provides a common set of services for developing analytics applications
• Accelerates the process of developing big data analytics applications whilst significantly reducing the TCO
• PNDA provides a platform for convergence of network data analytics
PNDA
PNDAPlugins
ODL
Logstash
OpenBPM
pmacct
XR Telemetry
Real-time
Data D
istribution
FileStore
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
Stream
Batch
Processing
SQL Query
OLAP Cube
Search/Lucene
NoSQL TimeSeries
DataExploration
Metric Visualisation
Event Visualisation PNDA
Mnged App
PNDA Mnged App
UnmngedApp
UnmngedApp
Query Visualisationand Exploration
PNDA Applications
PNDAProducer API
PNDAConsumer API
• Horizontally scalable platform for analytics and data processing applications
• Support for near-real-time stream processing and in-depth batch analysis on massive datasets
• PNDA decouples data aggregation from data analysis
• Consuming applications can be either platform apps developed for PNDA or client apps integrated with PNDA
• Client apps can use one of several structured query interfaces or consume streams directly.
• Leverages best current practise in big data analytics
PNDA
PNDAPlugins
ODL
Logstash
OpenBPM
pmacct
XR Telemetry
Real-time
Data D
istribution
FileStore
Platform Services: Installation, Mgmt, Security, Data Privacy
App Packaging and Mgmt
Stream
Batch
Processing
SQL Query
OLAP Cube
Search/Lucene
NoSQL TimeSeries
DataExploration
Metric Visualisation
Event Visualisation PNDA
Mnged App
PNDA Mnged App
UnmngedApp
UnmngedApp
Query Visualisationand Exploration
PNDA Applications
PNDAProducer API
PNDAConsumer API
Why PNDA?There are a bewildering number of big data technologies out there, so how do you decide what to use?
We've evaluated and chosen the best tools, based on technical capability and community support.
PNDA combines them to streamline the process of developing data processing applications.
Why PNDA?Innovation in the big data space is extremely rapid, but combining multiple technologies into an end-to-end solution can be extremely complex and time-consuming
PNDA removes this complexity and allows you to focus on developing the analytics applications, not on developing the pipeline – significantly reducing the effort required and time-to-value
Architecture
• Needs BGP Speakers with BMP protocol support
• BMP session established between BGP Speakers and OBMP
Architecture
• Logstash required to perform ‘avro’ encoding of BMP data
• BGP App runs as Batch job, running periodically
• OBMP gives us the ability to record the dynamics of the Internet
• PNDA platform enables• ‘Raw’ event recording capability, with horizontal scaling (HDFS)• Run analysis over large data-sets with parallelism• Ask questions of the aggregate data about the Internet• Drill down analysis• Per-prefix• Per-AS• Per AS-Path
What does this give us?
• What can we do with large-scale collection of historical event information?
• Event impact analysis –• Stability• Security• Misconfiguration
• Application of ML/DL to data-set• Pattern-detection and network ‘weather forecasting’
Potential