Cisco ACI Latency and Precision Time ProtocolNew and Changed Information 2

Revised: March 18, 2020

New and Changed InformationThe following table provides an overview of the significant changes up to the current release. The table does not provide an exhaustivelist of all changes or of the new features up to this release.

Table 1: New Features and Changed Behavior in Cisco APIC

DescriptionFeatureCisco APIC Release Version

Support for changing the resolutionfactor to 11 which then canmeasureup to 214 milliseconds with anaccuracy of 204ns.

Resolution Factor enhancementRelease 4.0(1x)

This guide was released to providea description of APIC Latency andPrecision Time Protocol.

APIC Latency and PrecisionTime Protocol

Release 3.0(1x)

About Fabric LatencyFabric latency is a troubleshooting tool to monitor the time taken by a packet to traverse from source to destination in the fabric. Itcan be used to measure latency between a combination of endpoints, endpoint groups, external interfaces, and IP addresses. Latencyis measured from the Arrival time in the ingress leaf switch to the Departure time in the egress leaf switch. A prerequisite for fabriclatency measurement is that all the nodes shall be synchronized with uniform time. Precision Time Protocol (PTP) is used for this,due to its' sub-microsecond accuracy, compared to NTP, which has only millisecond precisions. NTP is not sufficient to measurepacket flight times within an ACI fabric, which is in the order of microseconds. Hence, latency feature requires all the nodes in thefabric to be synchronized using PTP.

There are two types of latency measurement:

• Ongoing TEP-to-TEP latency

• On-demand Tenant latency

Ongoing latency or Leaf-to-leaf (TEP to TEP) latency is used to measure latency across Tunnel End Points in leaf switches. It providesthe average and maximum latency, standard deviation, and packet count computed at the destination leaf switch. The latency datacollected for the last 30 seconds as well as the cumulative latency values are provided. The TEP-to-TEP latency measurements areenabled as soon as PTP is turned on in the fabric.

Tenant latency measurements can be configured to troubleshoot issues at the level of individual applications. They can be enabledfor the IP flows matching a specific Flow rule programmed in the Latency TCAM. The flow rules semantics are similar to the currentAtomic counter flow rules.

If latency measurement is configured for a particular IP flow, then the latency measurement simultaneously being done forthis flow’s tunnel, will not account for latency of this flow.

Note

The following flow rules are supported for latency measurement, in addition to atomic counters:

2

• Measure EP to EP latency

• Measure EP to EPG latency

• Measure EP to External IP latency

• Measure EPG to EP latency

• Measure EPG to EPG latency

• Measure EPG to External IP latency

• Measure External IP to EP latency

• Measure External IP to EPG latency

• Measure Any to EP latency

• Measure External IP to External IP latency

• Measure EP to Any latency

Both Atomic counters and Latency measurements can be independently enabled or disabled on the same IP flow rules.Note

Latency data can be measured in two modes; average and histogram. The mode can be specified independently for ongoing latencyas well as for each flow rule in tenant latency policies.

Average Mode

Average mode enables the following measurements.

• Average latency for last 30 seconds

• Standard deviation for last 30 seconds

• Packet count for last 30 second

• Accumulated average latency

• Accumulated Maximum latency

• Accumulated Packet Count

The latency measurement in average mode may slightly differ in the low order multiples, of 0.1 microsecond, compared toan external measurement equipment.

Note

Histogram Mode

Histogrammode enables the visualization of the distribution of packet counts across different latency intervals. There are 16 Histogrambuckets, and each bucket is configured with a measurement interval. Bucket 0's measurement interval is 0 to 5 microseconds, andBucket 1 between 5 to 10 microseconds ending in 80 microseconds for the last bucket. Each of these buckets include a 64 bit counterto measure packets whose latency fell within the bucket’s configured latency interval.

The histogram charts are useful for understanding the latency trends, but may not reflect the exact packet count. For measuring theactual number of packets, atomic counters may be used.

3

The maximum number of TEP-to-TEP latency entries supported is 384. In EX-based TORs, we can have at most 256 flows in averagemode and 64 flows in histogrammode. In FX-based TORS, we can have at most 640 flows in averagemode and 320 flows in histogrammode.

About PTPPrecision Time Protocol (PTP) is a time synchronization protocol defined in IEEE 1588 for nodes distributed across a network. WithPTP, it is possible to synchronize distributed clocks with an accuracy of less than 1 microsecond via Ethernet networks. PTP’s accuracycomes from the hardware support for PTP in the ACI fabric spines and leafs. It allows the protocol to accurately compensate formessage delays and variation across the network.

PTP is a distributed protocol that specifies how real-time PTP clocks in the system synchronize with each other. These clocks areorganized into a master-slave synchronization hierarchy with the grandmaster clock, which is the clock at the top of the hierarchy,determining the reference time for the entire system. Synchronization is achieved by exchanging PTP timing messages, with themembers using the timing information to adjust their clocks to the time of their master in the hierarchy. PTP operates within a logicalscope called a PTP domain.

The PTP process consists of two phases: establishing the master-slave hierarchy and synchronizing the clocks. Within a PTP domain,each port of an ordinary or boundary clock follows this process to determine its state:

• Examines the contents of all received announce messages (issued by ports in the master state).

• Compares the data sets of the foreign master (in the announce message) and the local clock for priority, clock class, accuracy,and so on.

• Determines its own state as either master or slave.

After the master-slave hierarchy has been established, the clocks are synchronized as follows:

• The master sends a synchronization message to the slave and notes the time it was sent.

• The slave receives the synchronization message and notes the time that it was received. For every synchronization message,there is a follow-up message. Hence, the number of sync messages should be equal to the number of follow-up messages.

• The slave sends a delay-request message to the master and notes the time it was sent.

• The master receives the delay-request message and notes the time it was received.

• The master sends a delay-response message to the slave. The number of delay request messages should be equal to the numberof delay response messages.

• The slave uses these timestamps to adjust its clock to the time of its master.

In ACI fabric, when PTP feature is globally enabled in APIC, the software automatically enables PTP on specific interfaces of allthe supported spines and leafs. This auto-configuration ensures that PTP is optimally enabled on all the supported nodes. In theabsence of an external grandmaster clock, one of the spine switch is chosen as the grandmaster. The master spine is given a differentPTP priority as compared to the other spines and leaf switches so that they will act as PTP slaves. This way we ensure that all theleaf switches in the fabric synchronize to the PTP clock of the master spine.

If an external Grandmaster clock is connected to the spines, the spine syncs to the external GM and in turn acts as a master to the leafnodes.

PTP Default Settings

The following table lists the default settings for PTP parameters.

4

DefaultParameters

Boundary clockPTP device type

Two-step clockPTP clock type

0PTP domain

255PTP priority 1 value when advertising the clock

255PTP priority 2 value when advertising the clock

1 log secondPTP announce interval

3 announce intervalsPTP announce timeout

0 log secondsPTP delay-request interval

-2 log secondsPTP sync interval

1PTP VLAN

PTP operates only in boundary clock mode. Cisco recommends deployment of a Grand Master Clock (10 MHz) upstream,with servers containing clocks requiring synchronization connected to the switch.

Note

PTP Verification

PurposeCommand

Displays the PTP status.show ptp brief

Displays the properties of the local clock, including clock identity.show ptp clock

Displays the state of foreign masters known to the PTP process.For each foreign master, the output displays the clock identity,basic clock properties, and whether the clock is being used as agrandmaster.

show ptp clock foreign-masters record interface ethernetslot/port

Displays the last few PTP corrections.show ptp corrections

Displays the PTP packet counters for all interfaces or for aspecified interface.

show ptp counters [all |interface Ethernet slot/port]

Displays the properties of the PTP parent.show ptp parent

Guidelines and Limitations• Latency requires all the nodes in the fabric to be synchronized using Precision Time Protocol (PTP).

• Latency measurement and PTP are supported only on second generation or later ACI switches. The following first generationACI switches are not supported for latency measurement and PTP:

5

• N9K-C9332PQ

• N9K-C9336PQ

• N9K-C9372PX

• N9K-C9372PX-E

• N9K-C9372TX

• N9K-C9372TX-E

• N9K-C9396PX

• N9K-C9396TX

• N9K-C93120TX

• N9K-C93128TX

• N9K-X9736PQ

• WhenACI fabric has mixed generation of switches, the latency can bemeasured only for packets that go through newer generationleaf and spine switches, which support latency measurement with PTP. This implies all spine switches need to be PTP capableswitches because it is difficult to predict which spine forwards which traffic flow.

• In the presence of first generation leaf switches, it is recommended to use external GM and have its connectivity to all the spineswitches. Otherwise, the PTP messages from the grandmaster spine may be blocked by first generation switches before theyreach all the switches. Using an external GM connected to all spine switches ensures that the PTP messages from the externalGM reach all the switches even in the presence of first generation leaf switches.

• External Grandmaster (GM) clock is not mandatory for PTP in a single Pod. If there is no external GM connected to the ACIfabric, one of the spine nodes acts as the GM. This spine switch has a PTP priority1 value as 254. All the other spine switchesand leaf switches in the fabric will synchronize their clock to this Master spine switch clock. If the external GM is connectedlater to the spine switch, it should have a priority value less than 254 for it to act as the GM for the entire fabric.

• External Grandmaster clock is mandatory for PTP in a multipod scenario. In addition, external GM needs to be connected to theIPN such that the Grandmaster clock is the master to the spine switches in different PODs. The spine switches connected to IPNwill act as the boundary clock and all the nodes within the POD will sync their clock this spine switch.

• PTP operates only in boundary clock mode. End-to-end transparent clock and peer-to-peer transparent clock modes are notsupported.

• PTP supports transport over User Datagram Protocol (UDP). Transport over Ethernet is not supported.

• PTP supports multicast communication only; unicast mode is not supported.

• Beginning with release 4.0(1), support is added for changing the resolution factor to 11 which then can measure up to 214milliseconds with an accuracy of 204ns.

Configuring PTP Using the APIC GUIPTP is a system-wide policy and will be enabled on all the supported leafs/spines of all the PODs. When PTP is enabled, ongoingTEP-to-TEP latency measurements get enabled automatically.

6

Procedure

On the menu bar, click System > System Settings. In the Navigation pane, click Precision Time Protocol and in theWork pane under Properties, select Enabled and click Submit.

Configuring Fabric Latency Parameters Using the APIC GUIPTP must be enabled on all the Leaf switches and Spines in the fabric. See Configuring PTP Using the APIC GUI, on page 6.

Procedure

Step 1 On the menu bar, click Tenants > Tenant_name. In the Navigation pane, click Policies > Troubleshoot > AtomicCounter and Latency Policy.

Step 2 Right click on Atomic Counter and Latency Policy to open a policy configuration wizard and perform the followingactions:a) In the Name field, enter a name for the policy.b) In the Features field, check the Latency Statistics box.c) In the Mode field, choose either avg or historgram latency modes to apply to the policy.d) In the Source (IP or CEP) field, enter the identifying information for the traffic source.

The required identifying information differs depending on the type of source (endpoint, endpoint group,external interface, or IP address.

Note

e) In the Destination (IP or CEP) field, enter the identifying information for the traffic destination.

Step 3 Optionally, in the Filters table, click the + icon to specify filtering of the traffic to be counted. In the resulting CreateAtomic Counter Filter dialog box, you can specify filtering by the IP protocol number (TCP=6, for example) and bysource and destination IP port numbers. Click Submit.

Step 4 In the Work pane, click the Operational tab and click the Latency subtab to view the Latency statistics.

Configuring PTP Using the NX-OS CLI

Procedure

Step 1 Enable PTP.

Example:Enable ptp:========apic# configure terminalapic(config)# ptpDisable ptp:========apic# configure terminalapic(config)# no ptp

Step 2 To verify PTP on ACI switches:

Example:

7

leaf1# show ptp briefPTP port status-----------------------Port State------- --------------Eth1/49 Slave

leaf1#leaf1#leaf1# show ptp clockPTP Device Type: Boundary clockClock Identity : 0c:75:bd:ff:fe:03:1d:10Clock Domain: 0Number of PTP ports: 1Priority1 : 255Priority2 : 255Clock Quality:

Class : 248Accuracy : 254Offset (log variance) : 65535

Offset From Master : 32Mean Path Delay : 128Steps removed : 1Local clock time:Thu Jul 27 19:43:42 2017

leaf1#leaf1# show ptp clock foreign-masters record interface ethernet 1/49

P1=Priority1, P2=Priority2, C=Class, A=Accuracy,OSLV=Offset-Scaled-Log-Variance, SR=Steps-RemovedGM=Is grandmaster

--------- ----------------------- --- ---- ---- --- ----- --------Interface Clock-ID P1 P2 C A OSLV SR--------- ----------------------- --- ---- ---- --- ----- --------

Eth1/49 d4:6d:50:ff:fe:e6:4d:3f 254 255 248 254 65535 0 GM

leaf1#leaf1#leaf1# show ptp corrections

PTP past corrections-----------------------------------------------------------------------------------Slave Port SUP Time Correction(ns) MeanPath Delay(ns)---------- ------------------------------- ------------------ ------------------Eth1/49 Thu Jul 27 19:44:11 2017 364281 36 152Eth1/49 Thu Jul 27 19:44:11 2017 114565 16 132Eth1/49 Thu Jul 27 19:44:10 2017 862912 8 132Eth1/49 Thu Jul 27 19:44:10 2017 610823 8 132Eth1/49 Thu Jul 27 19:44:10 2017 359557 16 132Eth1/49 Thu Jul 27 19:44:10 2017 109937 8 132Eth1/49 Thu Jul 27 19:44:09 2017 858113 16 132Eth1/49 Thu Jul 27 19:44:09 2017 606536 16 132Eth1/49 Thu Jul 27 19:44:09 2017 354837 -16 132Eth1/49 Thu Jul 27 19:44:09 2017 104226 24 148Eth1/49 Thu Jul 27 19:44:08 2017 853263 24 148Eth1/49 Thu Jul 27 19:44:08 2017 601780 16 148Eth1/49 Thu Jul 27 19:44:08 2017 349639 -4 148Eth1/49 Thu Jul 27 19:44:08 2017 99970 16 144Eth1/49 Thu Jul 27 19:44:07 2017 848507 0 144Eth1/49 Thu Jul 27 19:44:07 2017 596143 24 144Eth1/49 Thu Jul 27 19:44:07 2017 344808 4 144Eth1/49 Thu Jul 27 19:44:07 2017 93156 -16 140

8

Eth1/49 Thu Jul 27 19:44:06 2017 843263 24 140Eth1/49 Thu Jul 27 19:44:06 2017 590189 8 140leaf1#leaf1#leaf1# show ptp counters all

PTP Packet Counters of Interface Eth1/49:----------------------------------------------------------------Packet Type TX RX---------------- -------------------- --------------------Announce 56 5424Sync 441 43322FollowUp 441 43321Delay Request 7002 0Delay Response 0 7002PDelay Request 0 0PDelay Response 0 0PDelay Followup 0 0Management 0 0----------------------------------------------------------------

leaf1#leaf1#leaf1# show ptp parent

PTP PARENT PROPERTIES

Parent Clock:Parent Clock Identity: d4:6d:50:ff:fe:e6:4d:3fParent Port Number: 258Observed Parent Offset (log variance): N/AObserved Parent Clock Phase Change Rate: N/A

Grandmaster Clock:Grandmaster Clock Identity: d4:6d:50:ff:fe:e6:4d:3fGrandmaster Clock Quality:

Class: 248Accuracy: 254Offset (log variance): 65535Priority1: 254Priority2: 255

leaf1#

Step 3 To verify troubleshooting steps:

Example:apic1# show troubleshoot eptoep session eptoep latency

Source --> DestinationLast Collection(30 seconds)+--------------------+-------------------------------+--------------+| Average (microsec) | Standard Deviation (microsec) | Packet Count |+--------------------+-------------------------------+--------------+| 18 | 24 | 1086 || | | |+--------------------+-------------------------------+--------------+

Cumulative+--------------------+----------------+--------------------+| Average (microsec) | Max (microsec) | Total Packet Count |+--------------------+----------------+--------------------+

9

| 18 | 202 | 6117438 || | | |+--------------------+----------------+--------------------+

Configuring Latency and PTP Using the REST APITo configure the flow policy parameters, follow the same steps for configuring atomic counters in Cisco APIC Troubleshootingguide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/troubleshooting/b_APIC_Troubleshooting/b_APIC_Troubleshooting_chapter_0110.html#id_40942.

Procedure

Step 1 To enable PTP mode:

Example:/api/node/mo/uni/fabric/ptpmode.xml<latencyPtpMode state=”enabled”>

Step 2 Configure an EP to EP policy:

Example:<dbgacEpToEp name="EP_to_EP_Policy" adminSt="enabled" usage=“latency-stats” latencyCollect =“histogram”></dbgacEpToEp>

Step 3 To enable both atomic counter and latency (average mode), here’s the XML

Example:<dbgacEpToEp name="EP_to_EP_Policy" adminSt="enabled" usage=“latency-stats|atomic-counter”latencyCollect = “average”></dbgacEpToEp>

Step 4 To change the collection type for Ongoing-mode from average to histogram.

Example:<latencyOngoingMode userMode=”histogram”>

10

© 2017–2020 Cisco Systems, Inc. All rights reserved.

Europe HeadquartersAsia Pacific HeadquartersAmericas HeadquartersCiscoSystemsInternationalBVAmsterdam,TheNetherlands

CiscoSystems(USA)Pte.Ltd.Singapore

Cisco Systems, Inc.San Jose, CA 95134-1706USA

Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on theCisco Website at www.cisco.com/go/offices.