DARPADARPA

A Metrics System for Continuous A Metrics System for Continuous Improvement of Design TechnologyImprovement of Design Technology

Andrew B. Kahng and Stefanus MantikAndrew B. Kahng and Stefanus Mantik

12/9/99

2

Motivation: Complexity of the Design ProcessMotivation: Complexity of the Design Process

Ability to make silicon has outpaced ability to design itAbility to make silicon has outpaced ability to design it Complex data, system interactionsComplex data, system interactions SOCSOC

more functionality and customization, in less timemore functionality and customization, in less timedesign at higher levels of abstraction, reuse existing design design at higher levels of abstraction, reuse existing design

componentscomponentscustomized circuitry must be developed predictably, with less riskcustomized circuitry must be developed predictably, with less risk

Key question: “Will the project succeed, i.e., finish on Key question: “Will the project succeed, i.e., finish on schedule and under budget while meeting performance schedule and under budget while meeting performance goals?”goals?”

SOC design requires an organized, optimized design SOC design requires an organized, optimized design processprocess

12/9/99

3

Value of CAD Tools Improvement Not ClearValue of CAD Tools Improvement Not Clear

What is $ value of a “better” scheduler, mapper, placer?What is $ value of a “better” scheduler, mapper, placer?

What is $ value of GUI, usability, …?What is $ value of GUI, usability, …?

What is the right objective?What is the right objective?min wirelength min wirelength routable routablemin literals min literals amenable to layout amenable to layout

Value well-defined only in context of overall design Value well-defined only in context of overall design processprocess

12/9/99

4

What is the Design Process?What is the Design Process?

Not like any “flow/methodology” bubble chartNot like any “flow/methodology” bubble chartbacks of envelopes, budgeting warsbacks of envelopes, budgeting warschanged specs, silent decisions, e-mails, lunch discussionschanged specs, silent decisions, e-mails, lunch discussionsad hoc assignments of people, tools to meet current needsad hoc assignments of people, tools to meet current needsproprietary databases, incompatible scripts/tools, platform-proprietary databases, incompatible scripts/tools, platform-

dependent GUIs, lack of usable standardsdependent GUIs, lack of usable standardsdesign managers operate on intuition, engineers focus on tool design managers operate on intuition, engineers focus on tool

shortcomingsshortcomings Why did it fail?Why did it fail?

““CAD tools”CAD tools”““inexperienced engineers”inexperienced engineers”

Must measure to diagnose, and diagnose to improveMust measure to diagnose, and diagnose to improve

12/9/99

5

What Should be Measured?What Should be Measured?

Many possibilitiesMany possibilitiesrunning a tool with wrong options, wrong subset of standardrunning a tool with wrong options, wrong subset of standardbug in a translator/readerbug in a translator/readerassignment of junior designer to project with multiple clocksassignment of junior designer to project with multiple clocksdifference between 300 MHz and 200 MHz in the specdifference between 300 MHz and 200 MHz in the specchanging an 18-bit adder into a 28-bit adder midstreamchanging an 18-bit adder into a 28-bit adder midstreamdecision to use domino in critical pathsdecision to use domino in critical pathsone group stops attending budget/floorplan meetingsone group stops attending budget/floorplan meetings

Solution: record everything, then mine the dataSolution: record everything, then mine the data

12/9/99

6

Design Process Data CollectionDesign Process Data Collection

What revision of what block was what tool called on?What revision of what block was what tool called on?by whom?by whom?when?when?how many times? With what keystrokes?how many times? With what keystrokes?

What happened within the tool as it ran?What happened within the tool as it ran?what was CPU/memory/solution quality?what was CPU/memory/solution quality?what were the key attributes of the instance?what were the key attributes of the instance?what iterations/branches were made, under what conditions?what iterations/branches were made, under what conditions?

What else was occurring in the project?What else was occurring in the project?e-mails, spec revisions, constraint and netlist changes, …e-mails, spec revisions, constraint and netlist changes, …

Everything is fair game; bound only by server bandwidthEverything is fair game; bound only by server bandwidth

12/9/99

7

Unlimited Range of Possible DiagnosesUnlimited Range of Possible Diagnoses

User performs same operation repeatedly with nearly User performs same operation repeatedly with nearly identical inputsidentical inputs tool is not acting as expectedtool is not acting as expectedsolution quality is poor, and knobs are being twiddledsolution quality is poor, and knobs are being twiddled

Email traffic in a project:Email traffic in a project:

missed deadline, missed revised deadline; people disengaged; missed deadline, missed revised deadline; people disengaged; project failedproject failed

On-line docs always open to particular pageOn-line docs always open to particular pagecommand/option is unclearcommand/option is unclear

12/9/99

8

METRICS System ArchitectureMETRICS System Architecture

Metrics Data Warehouse

Tool Tool Tool

xmitter xmitter xmitter

Data-Mining

Reporting

Inter/Intra-net

Server

JavaApplets

WebBrowsers

Wrapper orembedded

12/9/99

9

METRICS TransmitterMETRICS Transmitter

No functional change to the toolNo functional change to the tooluse API to send the available metricsuse API to send the available metrics

Low overheadLow overheadexample: standard-cell placer usingexample: standard-cell placer using

Metrics API Metrics API < 2% runtime overhead < 2% runtime overhead

even less overhead with bufferingeven less overhead with buffering

Won’t break the tool on transmittal failureWon’t break the tool on transmittal failurechild process handles transmission child process handles transmission

while parent process continues its jobwhile parent process continues its job

initToolRun()

sendMetrics()

sendMetrics()

sendMetrics()

sendMetrics()

12/9/99

10

METRICS TransmitterMETRICS Transmitter

EDATool

Tool wrapper

EDATool

API

JavaServlet Oracle8i

Inter/Intra-net

XML

XML

XML SQL

12/9/99

11

Transmitter ExampleTransmitter Example

/** API Example **//** API Example **/int main(int argc, char * argv[ ] )int main(int argc, char * argv[ ] ){{

......toolID = initToolRun( projectID, flowID );toolID = initToolRun( projectID, flowID );......printf( “Hello World\n” );printf( “Hello World\n” );sendMetric( projectID, flowID, toolID,sendMetric( projectID, flowID, toolID,

““TOOL_NAME”, “Sample” );TOOL_NAME”, “Sample” );sendMetric( projectID, flowID, toolID,sendMetric( projectID, flowID, toolID,

““TOOL_VERSION”, “1.0” );TOOL_VERSION”, “1.0” );......terminateToolRun( projectID, flowID,terminateToolRun( projectID, flowID,

toolID );toolID );return 0;return 0;

}}

## Wrapper example## Wrapper example( $File, $PID, $FID ) = @ARGV;( $File, $PID, $FID ) = @ARGV;$TID = `initToolRun $PID $FID`;$TID = `initToolRun $PID $FID`;open ( IN, “< $File” );open ( IN, “< $File” );while ( <IN> )while ( <IN> ){{

if ( /Begin\s+(\S+)\s+on\s+(\S+.*)/)if ( /Begin\s+(\S+)\s+on\s+(\S+.*)/){{

system “sendMetrics $PID $FID $TID \system “sendMetrics $PID $FID $TID \TOOL_NAME $1”;TOOL_NAME $1”;

system “sendMetrics $PID $FID $TID \system “sendMetrics $PID $FID $TID \START_TIME $2”;START_TIME $2”;

}}......

}}system “terminateToolRun $PID $FID \system “terminateToolRun $PID $FID \

$TID”;$TID”;

12/9/99

12

Example of METRICS XMLExample of METRICS XML

<? xml version=“1.0” ?><METRICSPACKET>

<REQUEST><TYPE> TOOL </TYPE><PROJECTID> 173 </PROJECTID><FLOWID> 9 </FLOWID><PARAMETER> 32 </PARAMETER>

</REQUEST><METRICS>

<PROJECTID> 173 </PROJECTID><FLOWID> 9 </FLOWID><TOOLID> P32 </TOOLID><DATETIME> 93762541300 </DATETIME><NAME> TOOL_NAME </NAME><VALUE> CongestionAnalysis </VALUE>

</METRICS></METRICSPACKET>

12/9/99

13

Current Testbed: A Metricized P&R FlowCurrent Testbed: A Metricized P&R Flow

LEF

DEF

Placed DEF

QP ECO

Legal DEF CongestionMap

WRoute

Capo Placer

Routed DEF

CongestionAnalysis

Incr. WRoute

Final DEF

METRICS

12/9/99

14

METRICS ReportingMETRICS Reporting

Web-based Web-based platform independentplatform independent

accessible from anywhereaccessible from anywhere

Example: correlation plots created on-the-flyExample: correlation plots created on-the-flyunderstand the relation between two metricsunderstand the relation between two metrics

find the importance of certain metrics to the flowfind the importance of certain metrics to the flow

always up-to-datealways up-to-date

12/9/99

15

METRICS ReportingMETRICS Reporting

JavaServlet Oracle8i

Inter/Intra-net

SQL

WEBBrowser

LocalGraphing

Tool(GNUPlot)

dataplot

Request

ReportRequest

Report

Request

Data

3rd PartyGraphing

Tool(Excel,Lotus)

Wrapper

Data

Future implementation

12/9/99

16

Example ReportsExample Reports

Congestion vs WL

# Via vs WL

12/9/99

17

METRICS StandardsMETRICS Standards

Standard metrics naming across toolsStandard metrics naming across toolssame name same name same meaning, independent of tool supplier same meaning, independent of tool supplier

generic metrics and tool-specific metricsgeneric metrics and tool-specific metrics

no more ad hoc, incomparable log filesno more ad hoc, incomparable log files

Standard schema for metrics databaseStandard schema for metrics database

12/9/99

18

Generic and Specific Tool MetricsGeneric and Specific Tool Metrics

tool_name chartool_version chartool_vendor charcompiled_date mm/dd/yyyystart_time hh:mm:ssend_time hh:mm:sstool_user charhost_name charhost_id charcpu_type charos_name charos_version charcpu_time hh:mm:ss

Generic Tool Metrics

num_cells integernum_nets integerlayout_size doublerow_utilization doublewirelength doubleweighted_wl double

num_layers integernum_violations integernum_vias integerwirelength doublewrong-way_wl doublemax_congestion double

Placement Tool Metrics

Routing Tool Metrics

12/9/99

19

Current StatusCurrent Status

Completion of METRICS server with Oracle8i, Completion of METRICS server with Oracle8i,

Servlet, and XML parserServlet, and XML parser

Initial transmittal API in C++Initial transmittal API in C++

METRICS wrapper for Cadence P&R toolsMETRICS wrapper for Cadence P&R tools

Simple reporting scheme for correlationsSimple reporting scheme for correlations

12/9/99

20

Additional InfrastructureAdditional Infrastructure

Industrial standard network discoveryIndustrial standard network discovery Jini, UPNP (Universal Plug & Play), SLP (Salutation)Jini, UPNP (Universal Plug & Play), SLP (Salutation)

SecuritySecurity encryption for XML dataencryption for XML data

SSL (Secure Socket Layer)SSL (Secure Socket Layer)

user id & password authentication (reporting)user id & password authentication (reporting)

registered users (transmitting)registered users (transmitting)

3rd party reporting tool3rd party reporting tool MS Office integration, Crystal report, …MS Office integration, Crystal report, …

Data miningData mining

12/9/99

21

METRICS DemoMETRICS Demo

Transmission of metricsTransmission of metricsAPI inside toolsAPI inside tools

Perl wrapper for log filesPerl wrapper for log files

ReportingReportingcorrelation reportscorrelation reports

progress on current tool run, flow, designprogress on current tool run, flow, design

12/9/99

22

Potential Benefits to Project ManagementPotential Benefits to Project Management

Accurate Resource Prediction At any point in Design CycleAccurate Resource Prediction At any point in Design Cycle up front estimates for People, Time, Technology, EDA Licenses, IP re-use...up front estimates for People, Time, Technology, EDA Licenses, IP re-use... go/no go at earliest pointgo/no go at earliest point

Accurate Project Post-mortemsAccurate Project Post-mortems Everything tracked - tools, flows, users, notesEverything tracked - tools, flows, users, notes Optimize for next Project based on past resultsOptimize for next Project based on past results No “loose”, random data or information left at Project end (log files!!!)No “loose”, random data or information left at Project end (log files!!!)

Management ConsoleManagement Console Web-based, status-at-a-glance of Tools, Designs, Systems at any point in Web-based, status-at-a-glance of Tools, Designs, Systems at any point in

projectproject

No wasted resourcesNo wasted resources prevent out of sync runsprevent out of sync runs no duplication of data or effortno duplication of data or effort

12/9/99

23

Potential Benefits to Tools R&DPotential Benefits to Tools R&D

Methodology for continuous tracking data over entire lifecycle of Methodology for continuous tracking data over entire lifecycle of

instrumented toolsinstrumented tools

More efficient analysis of realistic dataMore efficient analysis of realistic data no need to rely only on extrapolations of tiny artificial “benchmarks”no need to rely only on extrapolations of tiny artificial “benchmarks”

no need to collect source files for test cases, and re-run in houseno need to collect source files for test cases, and re-run in house

Facilitates identification of key design metrics, effects on toolsFacilitates identification of key design metrics, effects on tools standardized vocabulary, schema for design/instance attributesstandardized vocabulary, schema for design/instance attributes

Improves benchmarkingImproves benchmarking apples to apples, and what are the apples in the first placeapples to apples, and what are the apples in the first place

apples to oranges as well, given enough correlation researchapples to oranges as well, given enough correlation research

12/9/99

24

Potential Research Enabled by METRICSPotential Research Enabled by METRICS

Tools:Tools: scope of applicabilityscope of applicability predictabilitypredictability usabilityusability

Designs:Designs: difficulty of design or manufacturingdifficulty of design or manufacturing verifiability, debuggability/probe-abilityverifiability, debuggability/probe-ability likelihood of a bug escapelikelihood of a bug escape $ cost (function of design effort, integratability, migratability, ...)$ cost (function of design effort, integratability, migratability, ...)

Statistical metrics, time-varying metricsStatistical metrics, time-varying metrics What is the appropriate abstraction of manufacturing process for What is the appropriate abstraction of manufacturing process for

design?design? Impact of manufacturing on design productivityImpact of manufacturing on design productivity Inter- and intra-die variationInter- and intra-die variation Topography effectsTopography effects Impact, tradeoffs of newer lithography techniques and materialsImpact, tradeoffs of newer lithography techniques and materials

12/9/99

25

Ongoing WorkOngoing Work

Work with EDA, designer community to establish Work with EDA, designer community to establish standardsstandards tool users: list of metrics needed for design process optimizationtool users: list of metrics needed for design process optimization tool vendors: implementation of the metrics requested with the tool vendors: implementation of the metrics requested with the

standardized namingstandardized naming Improve the transmitterImprove the transmitter

add message bufferingadd message buffering““recovery” system for network / server failurerecovery” system for network / server failure

Extend METRICS system to include project management Extend METRICS system to include project management tools, email communications, etc.tools, email communications, etc.

Additional reports, data miningAdditional reports, data mining