Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D Slide 1

Desktop Metrics:What Should We Measure?

John McDermon

Group LeaderDCS-2

Departmental Computing Services Division

LA-UR-09-03076

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Three questions What behavior do you want to promote?

What story do you want to tell?

Who do you want to tell this story to?

Slide 2

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Behavior You don’t get what you want, you get what you measure1

• Beware of unintended consequences

3 traditional vectors• Better

— Quality, Accuracy, Efficacy• defects, repeats

• Faster— Timeliness, Efficiency

• response time, resolution time

• Cheaper— Total cost, Cost per unit

• define costs, define units

Where does quantity fall?

Slide 3

1. <http://www.whitehouse.gov/omb/expectmore/Council_for_Excellence_in_Government_ExpectMore_Booklet.pdf>

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Story Purpose• Inform• Explain• Persuade

Timeframe• Past

— “Once upon a time…”

• Present— Real time – dashboard, HUD

• Future— “Gaze into our crystal ball…”

Slide 4

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Audience Customers

Upper management

Internal management

Competitors

Peers

Critics

Others?

Slide 5

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Other things to consider Leading or Lagging

Trends or Spot values

Cost and difficulty of acquiring data• What data are you collecting now?• What are your current systems capable of?• Frequency

Slide 6

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Discussion Example 1:• Faster• Explain about future• Customers

Example 2:• Cheaper• Persuade about past• Upper management

Example 3:• Better• Inform about present• Peers

Slide 7

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

LANL Environment ≈ 12,000 employees• all types, including students

40+ square miles

≈ 30,000 network devices— ≈ 15,000 fingerprint as WIN— ≈ 6,000 other Operating Systems— + printers, switches, etc.

≈ 3,000 standalone systems

Slide 8

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Departmental Computing Services• 5 Groups ≈ 280 staff

— 1 Central Services Group ≈ 30 staff— 4 Field Groups ≈ 250 staff

• Support over 80% of LANL Departmental Computing assets• Includes:

— Standards, Tools & Services— Call center— Electronic Software Distribution (ESD)— Departmental servers and services

• Does not include:— Network or Phones— Enterprise application development or operations

Slide 9

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Volume of work

Slide 10

FY08

120,132

FY09

68,042

FY07

116,805

FY06

112,060

FY05

78,483

FY04

69,230

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Service Level Agreement High – 80% resolved in 4 work hours

Medium – 80% resolved in 3 work days (24 work hrs)

Slide 11

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY08 High (4 hrs)

Slide 12

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY08 Medium (3 days)

Slide 13

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY09 High (4 hrs)

Slide 14

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY09 Medium (3 days)

Slide 15

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Workload Tracking Tickets Created

Tickets Closed

Tickets in Queue

Slide 16

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY07 Ticket Queue

Slide 17

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY08 Ticket Queue

Slide 18

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY09 Ticket Queue

Slide 19

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

FY07 – FY09 Ticket Queue

Slide 20

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Attempt to show age of queue

Slide 21

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Customer Satisfaction Every ticket closed generates a survey email

3 vectors• Promptness

— How satisfied were you with the promptness and efficiency of our desktop support service?

• Accuracy— How satisfied are you that your request was completed accurately without

creating other problems?

• Professionalism— How satisfied are you with the courtesy, knowledge and experience of our

staff?

5 point scale (-2 to +2)• Very Dissatisfied = -2, Neutral = 0, Very Satisfied = 2

Slide 22

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Survey Response Rate

Slide 23

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Survey Scores

Slide 24

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Research: quantifying an environment

Assertion:• Complexity increases cost of support

Examples of complexity • Number of systems• Number of operating systems and versions• Number of applications and versions• Classified (complexity of security plan)• Servers, dual boot, etc.• Method of access (network vs. stand alone)

How to quantify complexity?

Could these be variables in linear equation?

Slide 25

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Harvey Mudd ClinicA MULTI-CRITERIA OPTIMIZATION MODEL FOR TRADE-OFFS BETWEEN SERVICES AND COSTS IN COMPUTER SUPPORT SERVICES

This project shall deliver a decision analysis model that relates the cost of support to the types of services available. It will include a multi-criteria optimization model for making trade-offs between level of service provided and costs of that service. The cost of support will be modeled as a function of actual costs (e.g. salaries, cost of systems, cost of equipment, software, etc) and time costs (actual costs in the form of FTE's required to support a given environment, but also including training and professional development costs). The model will permit comparisons amongst different environments so that choices between technologies and level of support can be made by the users.

Slide 26

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Harvey Mudd Clinic (cont.)

To estimate the number of technicians required at each skill level, we anticipate developing a multi-class queueing model where each "server" (technician) is capable of serving different classes of customers based on their skill set. Statistical analysis will be required to estimate the distribution of the time between service requests of each type (e.g. the arrival rate and inter-arrival time distribution) and the distribution of service times; these would be the inputs to the queueing model. Queueing theory as well as computer simulations will be used to estimate the correct number of technicians at each skill level. This queueing model in turn will provide a simple "rule-of-thumb" metric relating environment "complexity" and number of technicians. This metric will be obtained by fitting a statistical model to the results of the queueing simulation to infer which factors have the greatest impact on number of FTE's required.

Slide 27

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

U N C L A S S I F I E D

Questions?

Contact: John McDermon

jmcdermo@lanl.gov

505-667-7315

Slide 28