Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1

Network OS Bullseye code coverage demoVikram Hosakotevhosakot@cisco.com

June 21, 2012

Software Engineer – NOSTG

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

AGENDA• What is Bullseye ?

• Building a Bullseye NXOS image

• Generating Bullseye code-coverage numbers in NXOS

• Post-processing on NXOS Linux build server

• Merging coverage reports

• Viewing coverage-numbers in the Bullseye GUI

• Demo

• Automation ideas, eARMS integration ideas, HA testing like SSO and ISSU

• Bullseye help

• Q&A

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

What is Bullseye ?• A tool to capture coverage of C and C++ code.

• Used to generate code-coverage numbers in NXOS at Cisco.

• What is code coverage ?The ratio of number of lines of code tested to number of lines of code developed. A number less than 100.Example –If LDP is developed with 100 lines of code, and just 50 lines tested, the code coverage is

50 / 100 = 50%

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4

Building a Bullseye NXOS image• Talk to component DEs and find out

• The component name that needs Bullseye instrumentation (e.g. BGP, MPLS, LDP)• The directory structure of code and binaries (e.g. build/x86s/gdb/ldp/sup,

routing-sw/mpls/ldp/nxos/cradle/)• New filenames, function names, and new lines of code developed (e.g. ldp_feature.c,

ldp_cmi.h, ldp_cmi_show_neighbor_handler()). The new lines of code depend on the function name, component, and the feature.

• If code-coverage is needed on supervisor, or line card, or both ?• Access to NXOS Linux build server• The exact RELEASE TAG in the build tree (e.g. REL_5_2_5_S30)

• Use this link to build the Bullseye NXOS image using ACME, gmake, and isanadd CLIshttp://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

Generating Bullseye code-coverage numbers in NXOS• Load the Bullseye NXOS images – both kickstart and system

• Run the tests

• Code coverage file bootflash:code_coverage/<component>.cov is created both on active and standby

• Copy this coverage file from both active and standby to the NXOS Linux build server via FTP / TFTP

• Do post-processing (next slide) and generate coverage-numbers

• In eARMS, submit Bullseye images like regular NXOS images

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6

Post-processing on NXOS Linux build server• Install Bullseye on the NXOS Linux build server by doing

source /auto/andatc/independent/shellrc-files/current/rc/.cshrc.build

setenv PATH "${PATH}:/auto/andatc/linux/bullseye/7.5.41/bin/"

setenv BULLSEYE_VERSION "7.5.41"

module add bullseye ; which covbr

• Use these Bullseye CLIs with the coverage file <component>.cov for post-processing covsrc (coverage by source filename)covdir (coverage by source file directory)covfn (function coverage, whether a function was invoked or not)

• Bullseye CLIs to generate coverage Reports –covhtml or –html option (generate HTML report)-c option to generate output as comma separated values (CSV)

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7

Post-processing on NXOS Linux build serverNXOS-Linux-Build-Server# covsrc -f ldp.cov

Source Function Coverage C/D Coverage

--------------------------------- --------------------- ---------------------

...p/nxos/cradle/ldp_cltmgr_pss.c 21 / 29 = 72% 35 / 94 = 37%

...pls/ldp/nxos/cradle/ldp_main.c 13 / 17 = 76% 71 / 111 = 63%

...s/ldp/nxos/cradle/ldp_cltmgr.c 45 / 57 = 78% 123 / 268 = 45%

...mpls/ldp/nxos/cradle/ldp_mts.c 47 / 57 = 82% 173 / 312 = 55%

...mpls/ldp/nxos/cradle/ldp_cdb.c 21 / 25 = 84% 61 / 129 = 47%

...s/ldp/nxos/cradle/ldp_sysmgr.c 34 / 40 = 85% 84 / 148 = 56%

...mpls/ldp/nxos/cradle/ldp_pss.c 22 / 25 = 88% 61 / 106 = 57%

...p/nxos/cradle/ldp_utils_nxos.c 25 / 26 = 96% 52 / 87 = 59%

...os/cradle/ldp_cltmgr_pss_tlv.c 3 / 3 = 100% 14 / 27 = 51%

.../nxos/cradle/ldp_cdb_pss_tlv.c 3 / 3 = 100% 22 / 36 = 61%

.../ldp/nxos/cradle/ldp_feature.c 11 / 11 = 100% 43 / 68 = 63%

...cmn/include_lcs/ldpx_mem_api.h 1 / 1 = 100% 0 / 0

...nxos/libldpclt/ldpc_cmn_mmsg.h 1 / 1 = 100% 0 / 0

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

Post-processing on NXOS Linux build serverNXOS-Linux-Build-Server# covdir -f ldp.cov

Directory Function Coverage C/D Coverage

--------------------------------- --------------------- ---------------------

...-bxb/Bullseye/include/routing/ 3 / 141 = 2% 14 / 366 = 3%

...g-sw/mpls/ldp/cmn/include_lcs/ 2 / 27 = 7% 0 / 10 = 0%

...ng-sw/mpls/ldp/nxos/libldpclt/ 7 / 12 = 58% 14 / 47 = 29%

...uting-sw/mpls/ldp/nxos/cradle/ 388 / 617 = 62% 1197 / 3349 = 35%

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

Post-processing on NXOS Linux build serverNXOS-Linux-Build-Server# covfn -f ldp.cov

Function Source Line FnCov C/D Coverage

-------- --------------------------------------- ----- ---------------------

ldp...me ...mpls/ldp/nxos/cradle/ldp_cmi.c 3699 1 / 1 8 / 10 = 80%

ldp...it ...mpls/ldp/nxos/cradle/ldp_pss.c 472 1 / 1 8 / 10 = 80%

ldp...rs ...s/ldp/nxos/cradle/ldp_sysmgr.c 465 1 / 1 8 / 10 = 80%

ldp...ng ...mpls/ldp/nxos/cradle/ldp_cdb.c 55 1 / 1 4 / 5 = 80%

ldp...it ...pls/ldp/nxos/cradle/ldp_main.c 201 1 / 1 10 / 12 = 83%

ldp...sz .../nxos/cradle/ldp_cdb_pss_tlv.c 490 1 / 1 5 / 6 = 83%

ldp...ey ...s/ldp/nxos/cradle/ldp_cltmgr.c 295 1 / 1 5 / 6 = 83%

ldp...it ...p/nxos/cradle/ldp_utils_nxos.c 259 1 / 1 5 / 6 = 83%

ldp...et ...mpls/ldp/nxos/cradle/ldp_mts.c 1637 1 / 1 12 / 14 = 85%

ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1850 1 / 1 14 / 16 = 87%

ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1992 1 / 1 23 / 26 = 88%

ldp...ch ...mpls/ldp/nxos/cradle/ldp_mts.c 1351 1 / 1 9 / 10 = 90%

ldp...en ...mpls/ldp/nxos/cradle/ldp_pss.c 32 1 / 1 9 / 10 = 90%

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10

Post-processing on NXOS Linux build serverNXOS-Linux-Build-Server# covfn -f ldp.cov --html > ldp_covfn.html

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11

Post-processing on NXOS Linux build serverNXOS-Linux-Build-Server# covfn -f ldp.cov -c > ldp_covfn_csv

"ldp_base_init","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_main.c",201,1,10,12,83%

"ldp_cdb_pss_union_key_get_tlv_sz","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_cdb_pss_tlv.c",490,1,5,6,83%

"ldp_cm_fsb_lookup_by_clt_key","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_cltmgr.c",295,1,5,6,83%

"ldp_thread_info_init","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_utils_nxos.c",259,1,5,6,83%

"ldp_mts_msg_options_set","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1637,1,12,14,85%

"ldp_mts_sysmgr_thread","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1850,1,14,16,87%

"ldp_mts_ctrl_thread","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1992,1,23,26,88%

"ldp_dispatch","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1351,1,9,10,90%

"ldp_pss_uri_open","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_pss.c",32,1,9,10,90%

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12

Merging coverage reports• Bullseye CLI covmerge is used for merging multiple coverage files (*.cov) from multiple N7K

routers.

• Example to merge coverage files from the active and standby of two N7Kscovmerge -c –f merged_ldp_coverage.cov N7K1_active.cov N7K1_standby.cov N7K2_active.cov N7K2_standby.cov

• If • A function has 4 decision paths (say 1,2,3 and 4), one N7K runs tests to cover paths 1 and 4,

and another N7K runs tests to cover paths 1 and 2,

• then• The merged coverage file from both the N7Ks will show 75 % coverage (1,2,4). Path 3 will be

shown "uncovered"

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13

Viewing coverage-numbers in the Bullseye GUI/auto/andatc/linux/bullseye/7.5.41/bin/CoverageBrowser ldp.cov &

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14

Demo

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15

Automation ideas# Inside package CodeCoverage

proc get_bullseye_coverage_data {\ {Bullseye_Build_Server_IP “101.0.0.102”}\ {FTP_username “rcpuser”}\ {FTP_Password “123rcp!”}\ Component_Name} {# Refer to flowchart on right

USAGE

package require CodeCoverage

get_bullseye_coverage_data “ldp”

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16

eARMS integration ideas

• Read variable in job file –set BULLSEYE_COVERAGE_NEEDED “yes”

ORset BULLSEYE_COVERAGE_NEEDED $env(EARMS_CODE_COVERAGE)

• eARMS enhancement similar to CFLOW (CSCtw78100) -

• In eARMS, submit Bullseye images like regular NXOS images

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17

HA testing like SSO and ISSU• Unlike CFLOW, Bullseye coverage data will NOT get erased and will remain intact through

• Router reload

• Stateful Switchover (SSO)

• In Service Software Upgrade (ISSU)

• In Service Software Downgrade (ISSD)

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18

Bullseye help

• http://www.bullseye.com/help/index.html

• http://www.bullseye.com/

• http://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative

• https://and-sw-web.cisco.com/twiki/bin/view/Main/WebHome

• ACME commands - https://and-sw-web.cisco.com/twiki/bin/view/ACME/ACMECommands

• http://www.bullseye.com/contact.html

• and-sw-build@cisco.com

• Vikram Hosakote (vhosakot@cisco.com)

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19

Bullseye Webex recordings

• Webex recording of Bullseye demo to group.bstoner - https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61685767&rKey=0ec1c04f55cf4a23

• Webex recording of Bullseye demo to nostg-nxos-pi-test and nxos-pi-ft-core-team – https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61860282&rKey=8c23197cd63a9a18

• Webex recording of Bullseye demo to ASG-core and STEP teams – https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61903277&rKey=0a391701dee0c555

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20

Q & A

Thank you.