1

PART3

Data collection methodology and NM

paradigms

2

Mediation Device Functionality: How to Process Data Records Filtering Estimation from Sampling Threshold Monitoring Data Record Correlation and Enrichment Flow De-Duplication Data Record Formatting and Storage

Outline

3

filtering

Two main areas of filtering exist at the mediation device:

Filtering to reduce the volume of the data collection

Filtering for application purposes

4

Filtering to reduce the volume of the data collection

Complex filtering for volume reduction is a mediation device task, because the implementation of process-intensive filters at the network element has a performance impact

Ideally, the collection granularity and filtering functions at the device would allow for configuring exactly the data set that is required

5

Filtering

To reduce the performance impact of metering to a minimum:

Simple filters are implemented at the device, ideally in hardware instead of software operations.

Complex and CPU-intensive operations are realized at the mediation device.

6

Filtering for application purposes

Filtering for application purposes is based on the "divide and conquer" paradigm: collect data records once, but use them as input for multiple applications, such as capacity planning, billing, and security monitoring

7

Filtering example

8

Estimation from Sampling

This section is specific to NetFlow and applies only if NetFlow sampling is configured at the device.

It is an important task, because the child population gathered by sampling must be adjusted for the estimation of the parent population and to deduce an approximation of the volume based on the sampling rate.

9

A NetFlow mediation device estimates the absolute traffic volumes by renormalizing the volume of sampled traffic through multiplication with the meter's sampling frequency

Example: If sampling is applied with a sampling rate of 1:100, the data records need to be multiplied by a factor of 100.

Absolute traffic volume=data records * sampling factor

Estimation from Sampling

10

The example in the section "Filtering at the Network Device" defined three traffic classes: priority, business, and best effort. Priority traffic was fully collected and needs no adjustment. Business traffic was sampled with a rate of 1:100 and therefore needs to be multiplied by a factor of 100. Best-effort traffic had a sampling rate of 1:1000 and needs to be multiplied by 1000.

Estimation from Sampling

11

The sampling factor must be included in the exported data record, because otherwise the

estimation from sampling is incorrect.

Estimation from Sampling

12

Threshold Monitoring

It can be implemented at both the mediation device and the application server level

It is not so relevant where the function is located, but which purpose it serves

13

Threshold Monitoring in application server level

A metering device for a traffic planning application might leave the monitoring of the threshold up to the application server, because they are not critical for planning purposes

14

Threshold Monitoring at mediation device

On the other hand, if metering is applied for security monitoring, a relevant feature is to set thresholds for the received traffic and monitor them in real time, because a reaction to an attack must occur quickly

Exceeded thresholds can identify security issues, such as a denial-of-service (DoS) attack, in which a huge number of very small datagrams flood a network and eventually stop the services in the network

15

Data Aggregation

The concept of aggregation describes the task of reducing the granularity by identifying common criteria (key fields) and combining information from multiple records into a single record.

16

Aggregation concepts

Two different aggregation concepts exist:Aggregation of key fieldsAggregation over time

17

Aggregation of key fields

Aggregation of common criteria is related specifically to accounting records, whereas aggregation over time can be applied to both accounting and performance records

18

Aggregation Example

19

20

Data Record Correlation and Enrichment

Another task at the mediation layer is correlating information from different metering sources to enrich the data records.

Example: Modifying a data record by correlating the record details with DNS information to replace an IP address with a username

21

Correlation benefit

Grouping information from different sources into a common data records is a clear benefit of upper-layer applications, such as billing, which can retrieve enriched data sets instead of very basic sets, which need correlation afterwards.

22

Flow De-Duplication

Duplicate records lead to inaccurate results at the application level; therefore, these duplications need to be eliminated.

23

Flow De-Duplication steps

The following steps are performed: Identify common flow parameters, such as

source and destination address, port numbers, AS number, ToS/DHCP fields, and others

Check the time-stamps Associate the information and eliminate

duplicate flows

24

Data Record Formatting and Storage

Finally, the processed data records are stored in a database and made available to other applications.

Records have to describe usage type details, such as keys and values, where a key links to an index in a database table.

25

Common data format

A common data format definition protects the NMS and OSS applications from the variety of accounting formats that are implemented at the device level

26

Records storage

The location where the records are stored can be a simple flat file system, where a new folder is created for each device and subfolder per interface, with separate text files for each aggregation scheme and interval. Alternatively, the data store can be a complex relational database system that provides sophisticated operations for local and remote access.

27

Record format

The records are implemented as XML schemas with self-defining field attributes, including five major attributes: who, what, where, when, and why Who?— Responsible user ID When?— Time when the usage took place What?— Service description and consumed resources Where?— Source and destination ID Why?— Reason for reporting the event

28

Summery

Figure 2-27 summarizes all mediation device functions in a flow chart. As mentioned, some tasks, such as threshold monitoring and aggregation over time, can be applied at the upper-layer application level instead of the mediation device.

29

Figure 2-27

30

Abbreviations

Meaning

DoS denial-of-service

ToS Terms of Service

AS Autonomous System

DHCP Dynamic Host Configuration Protocol

OSS Operations support systems

XML EXtensible Markup Language