GEMS: A Generic Model Based Source CodeInstrumentation Framework

Pavan Kumar Chittimalli and Vipul ShahTata Consultancy Services, 54 B, Hadapsar Industrial Estate, Pune, India, 411013

Email: {pavan.chittimalli,v.shah}@tcs.com

Abstract—Software Programmers need to monitor and mea-sure dynamic behavior of programs. Program instrumentationtools and techniques have aided profiling, debugging, coverageanalysis, and dynamic program analysis. While several opensource and commercial instrumentation tools are available, thatsupport multitude of techniques and source languages, noneof the tools support a cross section of languages. Moreover,instrumentation tools lack support for systems that have beendeveloped using multiple languages. The output produced byeach tool is different, leading to problems in usage of the sameby other tools to be a challenge.

As an IT service provider, our organization maintains systemsdeveloped using a large number of programming languages. Wedevelop in-house dynamic program analysis and testing tools,that use instrumentation to help with the maintenance activities.To address the availability as well as the compatibility issues, wepropose a novel, generic, model based instrumentation technique.Our technique proposes addition of instrumentation code to aunified programming language model. In this paper, we presentGEMS, a generic model based source code instrumentationframework. Language specific parsers in the framework instan-tiate the unified model. Instrumentation code is added to themodel and language pretty printers unparse the model back toappropriate source language. Though the instrumentation codeis ultimately added to the source code, the experimental resultsprovided in this paper indicate relatively low execution overheadof about 3-6% over other instrumentation tools.

I. INTRODUCTION

Software Developers often need to monitor the dynamicbehavior of a program for analysis, debugging or testing pur-poses. Program instrumentation is an activity which inserts aset of probes in a program to enable information gathering thatwould be useful for these purposes. A probe can be defined asa monitor call to track a particular entity in the program. Theseprobes are placed at different program points. For example,for statement instrumentation, a probe is inserted after eachstatement in the program. While the inserted probes executeat the same time as the program, program instrumentation toolsand techniques need to ensure that this does not have an impacton the output of the program.

1) Source code instrumentation: Performed at source level,this technique faces the problem of the need to supportmultiple language versions and dialects.

2) Byte code instrumentation: Performed at byte code level,this type of instrumentation is for Java and .NET pro-grams. The underlying assumption is that while the lan-guage may evolve, its byte code architecture is unlikelyto change as frequently, making the instrumentation

technique and tool more easily usable. This assumptionwas largely held true for some time for Java, but withJava1.5+ there have been byte code changes.

3) Binary instrumentation: Performed at execution time -dynamically.

Table I lists some of the popular open source and com-mercial instrumentation tools. Almost all of these tools aredeveloped for specific languages. For example, Cobertura is abytecode instrumentation tool that is developed for Java andsupports statement and condition coverage instrumentation.Cobertura has been developed exclusively for Java programsand hence cannot be used for other languages. All the Javainstrumentation tools in Table I deploy a bytecode instrumen-tation strategy, which is easier and faster and more immuneto language changes. The overhead of the instrumentation isalso significantly less. Java is an interpreted language andhence this works well. Languages like C/C++/COBOL arecompiled, and therefore every platform poses a challenge.Instrumentation of such languages is usually at source codelevel. An additional challenge from an instrumentation toolperspective is that each compiler supports its own dialect ofthe language.

As an IT service provider, we maintain a large number ofsystems developed in multitude of languages. We routinelywork with project teams maintaining large legacy systemsdeveloped in languages like COBOL, PLI and RPG as well assystems built using C/C++, Java, VB, Java script, C# amongother languages. To help maintain such applications we havedeveloped a number of dynamic program analysis, programtransformation, testing and test selection tools. One of the chal-lenges that we face, is that there is no single instrumentationtool that supports all or even a large number of languagesthat we deal with. Working with multiple tools on multipleplatforms is a challenge as they not only work at differentlevels and provide different kinds of instrumentation, they alsoproduce different output. Using output from multiple toolsis time and effort intensive and hampers us from providinggeneric solutions, it is also a potential cause of defects. Aninstrumentation tool that can be used for large number oflanguages will go a long way in helping us help our projectteams with the maintenance task

In this paper, we present a novel, generic, model basedinstrumentation technique and tool - GEMS, to address theabove limitations. We propose a technique that instruments ona unified programming language model developed in-house.

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

978-0-7695-4670-4/12 $26.00 © 2012 IEEE

DOI 10.1109/ICST.2012.195

909

Toolname Agitar Bullseye Clover Cobertura CodeWarrior Pin EMMA gcov Insure++ JCover JTest Purify+ TCAT

Language Java C++/C Java, Java C++/C C++/C Java C++/C C++/C Java Java Java, C++.Net .Net C++/C, Java

.Net

Line Y Y Y Y Y Y Y Y Y Y Y Y

Decision Y Y Y Y Y Y Y Y Y

Method Y Y Y Y Y Y Y Y

Class Y Y Y Y

Source Y

ByteCode Y Y Y Y Y Y

Binary Y Y Y Y Y Y

TABLE I: Instrumentation tools and their support for code coverage

The model serves as an intermediate representation of thesource program. Instrumentation code is added to the modeland the model is unparsed back to the source language,providing source level instrumentation. Our technique is notonly independent of the source language but has the flexibilityto support different kinds of instrumentation. We presentresults comparing GEMS with two other instrumentation tools- Pin and gcov. We conclude from empirical results that ourtechnique imposes low execution time overhead.

The main contributions of the paper are:• A technique based on the unified programming language

model• A tool, GEMS, based on our technique which currently

supports C and Java but can potentially support languageslike C++, COBOL, RPG and PL/1.

• A case study performed on industrial and open source(SIR repository[11]) C subjects.

II. RELATED WORK

A. Chawla and A. Orso have proposed a generic instru-mentation framework [6] for custom instrumentation. Theframework, InsECT, lets users collect runtime informationwhere users can specify the types of entities to be monitored,parts of the program to be instrumented, information to be col-lected and processed. This framework was further improved toInsECTJ [7]. Though these frameworks are generic in nature,and provide great deal of flexibility, the implemented toolsInsECT and InsECTJ are bytecode instrumentation tools andhence can be used only for Java programs. GEMS frameworkconsists of multiple language parsers that build an intermediaterepresentation that supports a unified programming languagemodel. The approach provides separation of concerns byproviding a language agnostic method to insert instrumentationprobe to the common intermediate representation, enablingsupport for number of languages.

In contrast to source and bytecode instrumentation tech-niques, the binary instrumentation technique [8], [9] inserts

probe calls after the programs binary executable code isgenerated. In this case, the additional instrumentation codeis inserted at runtime dynamically by patching the applicationexecutable binary in memory [5]. In other case, the additionalinstrumentation code is inserted through rewriting the appli-cation executable prior to execution. However, these low-leveltechniques are very architecture and compiler dependent. Thisrestricts the applicability to the supported set of platforms andcompilers.

K. A. Lindlan and et al. [10] have proposed a generic sourcecode based instrumentation, TAU source code instrumentor.This instrumentation works on source code of any applicationusing a preprocessor before it is passed to the parser to findout the potential instrumentation points. TAU uses ProgramDatabase Toolkit (PDT) exclusively developed to cater theneed for this instrumentor. Currently TAU instrumentor sup-ports C/C++, Fortran languages.

III. UNIFIED PROGRAMMING LANGUAGE MODEL

Typically, a program in any programming language, consistsof declarations, and computations guided by control-flow. Thedeclarations provide details of data being used and its type- this information is captured in a Symbol Table (ST). Thecomputations and control flow are represented by an AbstractSyntax Tree (AST). AST and ST together constitute the inter-mediate representation (IR) for a program. To support multiplelanguages, we define a core IR model that contains languageconstructs common to the languages and a language specificmodel that contains the extensions or differences. IR modelfor each language inherits from the core model. This is themechanism used to construct IR models for various program-ming languages like COBOL, C, PASCAL, C++, RPG, Javaand PL/I. These models have now evolved and are now partof unified OMG ASTM [12]. Further, we define a Union IRmodel that contains all constructs belonging to all languages(i.e. union of all IR models). We term this as the UnifiedProgramming Language Model (UPLM). Language processing

910

Fig. 1: GEMS: Block diagram

tools like program analyzers, program transformers make useof UPLM. The basis for our instrumentation technique is theUnified Programming Language Model. Figure 2 depicts coreAST.

IV. INSTRUMENTATION ON MODEL

We propose an approach to instrumentation, that promotesseparation of concerns by separating language dependentparts like parsing and unparsing of source programs frominstrumentation. A unified program model, as described insection III, is used as intermediate representation of theprogram. Instrumentation probes, as monitor calls are insertedat appropriate program points in the model and the model isunparsed back to source language using a pretty printer. Theprobes are implemented in language specific libraries that arenow compiled along with the source programs. The method isgeneric as language parsers for multiple languages transformthe source program to the unified model and the act of insertingprobes remains the same, irrespective of the source language.

A. Implementation

We call the implementation of our framework - GEMS(GEneric Model based Source code instrumentation). Thedetails of the framework are illustrated in Figure 1. It com-prises of set of language parsers, Probe inserter, Unparserand Program executor. Each language parser transforms thesource program into intermediate representation that is aninstance of UPLM. The tool currently supports statement,condition, branch and method level instrumentation. The tool

Fig. 2: Core Abstract Syntax Tree

911

is flexible and can support additional instrumentation througha specification file that lists the AST node and the instrumen-tation probe. The probe inserter traverses the ST and ASTmodels to insert monitor probes at appropriate locations inthe AST. An unparser module traverses the transformed STand AST models to reproduce the source program, albeit withinstrumentation code.

01 void m2(){02 s2: int i,j;03 s3: for(i=0;i<5;i++){04 s4: for(j=0;j<5;j++){05 s5: if(i==j) break;06 s6: printf("Hello...");07 }08 }09 }

Fig. 3: An Example C Program

A unique identifier is assigned to each program entity. Themonitor probe takes the identifier of the program entity to beinstrumented as parameter. During the probe insertion phase,a file listing all the identifiers along with line numbers isgenerated. The executor module initially loads the identifierfile and invokes the source program to be instrumented. Eachmonitor probe when executed, marks the appropriate programentity as visited and finally a report is generated that providescoverage information.

Fig. 4: IR models for example in Figure 3

GEMS framework currently supports several languageparsers such as C/C++, Java, PLI, COBOL and RPG. We havehowever experimented with only C and Java instrumentation,as we have C and Java unparsers readily available. Extendinginstrumentation support for other languages would be a matter

of creating monitor probe library and customizing the executormodule.

Fig. 5: Partial transformed IR models for example in Figure3

void m2() {int i, j;probe_stmt (s3);for (i = 0; i <= 5; i++) {

probe_cond(s3);probe_stmt(s4);for (j = 0; j <= 5; j++) {

probe_cond(s4);probe_stmt(s5);if (i == j) {

probe_cond(s5);probe_stmt(s5);break;

}probe_stmt(s6);printf(Hello...);

}}

}

Fig. 6: Modified C Program

We explain the process through an example. Example 3 is aC program that has nested for and if statements. The programprints hello 20 times. Figure 4 and Figure 5 depict the IRmodels for the example before and after execution probes havebeen inserted. The un-parsed code for method m2 is shownin figure 6. In section V we compare performance of GEMSwith two other instrumentation tools.

912

V. EMPIRICAL STUDIES

We conducted a study to compare our instrumentationapproach to two existing instrumentation tools that were listedin Table I, namely Pin and gcov. This section describes ourempirical study, and presents the results of the study and ananalysis of it.

A. Empirical Setup

For the object of analysis, we have used 5 subjects fromthe Software-artifact Infrastructure Repository (SIR)1 and 1in-house subject. Tcas is an avionic related program writtenin C and has 173 lines of code, printtokens2 is in the Siemensbenchmark as part of SIR [11] repository. gzip2 is an com-pression tool written in C that has 5680 lines of code. cfeis a C language front-end developed at TRDDC [13]. cfe isdeveloped entirely in C language and has 22000 lines of code.The details of the subjects are provided in Table II.

Name tcas printtokens gzip flex grep cfe

LOC 173 726 5680 10459 10068 22000

Number of 1608 4130 214 525 471 823Test Cases

TABLE II: Details of the subjects used in the experimentalstudies.

For comparison purposes, for all subjects, we carry out thefollowing activities using Pin, gcov and GEMS:

• measure the number of probes inserted for statementcoverage

• measure actual coverage achieved while executing testsuite for cfe

• compute total execution time with GEMS• compute the source code and object code size with the

instrumentation probes

We also execute the test suite for each subject beforeand after adding the instrumentation probes with GEMS. Werecord the output in each case and compare the same to ensurethat there are no losses in the process and the programs aresemantically the same. The measure on number of probesinserted is to verify that we have not missed out any probes.Execution time will indicate the overhead imposed by addingprobes directly in the source and the code size would indicatea measure of overhead from size perspective. To measureexecution time overhead we compare execution time withoutinstrumentation to that with instrumentation.

1Software-artifact Infrastructure Repository (SIR) is availableathttp://sir.unl.edu/portal/index.html.

2Originally created by Tom Ostrand and colleagues at Siemens CorporateResearch

B. Results and Analysis

The output from the subjects, pre and post addition ofinstrumentation probes, for all the test cases matched. One cansafely infer from this that there was no loss of informationin the source code. Table IV presents a comparison of thetotal number of probes inserted by Pin, gcov and GEMS.For example, Pin inserted 5122 probes in gzip as against5131 probes by gcov compared to 5234 probes by GEMSfor statement coverage. For printtokens subject, both Pin andGEMS inserted identical number of probes - 616, whereasgcov inserted 603 probes. Both Pin and gcov instrumentbinaries. GEMS UPLM has a few dummy nodes for compoundstatements as part of the common model and this has resultedin additional probes. The table indicates that the number ofprobes are comparable for statement coverage.

Name Using Pin Using gcov Using GEMS

tcas 127 127 127

printtokens 616 603 616

gzip 5122 5131 5234

flex 9800 9828 9911

grep 9901 9899 9901

cfe 20136 21340 22087

TABLE IV: Comparison of number of instrumentation probes

Name Using Pin Using gcov Using GEMS

Statement 81% 86% 81%

condition 69% 74% 69%

TABLE V: Comparison of coverage information for cfe

Test suite for each subject was executed with instrumenta-tion for statement and condition coverage using all the threetools. Comparison of the computed coverage information ispresented in Table V for cfe. Table V indicates that statementand condition coverage using our instrumentation tool wascomparable to Pin and gcov. So while we insert slightly highernumber of probes, the coverage achieved is comparable.

Next we compared the overhead with respect to executiontime. We measured the time pre and post addition of instru-mentation probes on all subjects. The results are presented inthe Table VI indicate an overhead of 3-6% over Pin and gcov.We also compared the overhead in terms of the size of sourcecode and object code pre and post addition of instrumentationcode by our tool on all subjects. Instrumentation probes wheninserted directly into source code lead to increase in the LOCcount by close to 50%. The LOC count increase correspondsto the number of probes inserted in the code.

913

Name Original Using Pin Using gcov Using GEMS Overheadsource program with GEMS

tcas 4s 4s 4s 4s 0%

printtokens 23s 23.1s 23.3s 23.5s 2.13%

gzip 4 min 32s 4 min 35s 4 min 4 s 4 min 50s 6.20%

flex 6 min 11s 6 min 18s 6 min 30s 6 min 36s 6.31%

grep 6 min 56s 7 min 05s 7 min 15s 7 min 23s 6.09%

cfe 10 min 34s 10 min 41s 11 min 02s 11 min 27s 7.71%

TABLE III: Comparison of execution times

Name On original Using GEMS Overheadsource program (in %)

(LOC) (LOC)

tcas 173 300 42.33

printtokens 726 1342 45.90

gzip 5680 10914 47.96

flex 10459 20360 48.63

grep 10068 19969 49.58

cfe 22000 41023 46.37

TABLE VI: Comparison of the size of the program before andafter instrumentation.

VI. THREATS TO VALIDITY

Our current framework and unified model supports C/C++,Java, PLI, COBOL and RPG. The unified model is designedfor imperative languages and we may not be able to supporta number of languages.

While our approach is suitable for instrumentation, programanalysis and debugging of the modified source code will betedious due to the instrumentation probes in the source code.As the primary motivation to develop GEMS was instrumen-tation, the above limitation was acceptable to us.

VII. CONCLUSION

In this paper, we presented a generic program instrumen-tation technique and tool GEMS, that provides us with anoption to use a single tool/framework to instrument programsdeveloped in multiple languages as against using differenttools for different languages. GEMS works as part of in-house framework comprising of a set of language parsers,a unified programming language model that serves as aintermediate representation and a set of pretty printers. Theinstrumentation tools are used as part of testing tool set. Theapproach is designed to be flexible to accommodate different

types of dynamic analysis and monitoring needs. Empiricalresults indicate that the approach has low execution overhead.Dynamic program analysis and testing tools can now usea single instrumentation tool making their development andmaintenance task easy. Support for applications that havebeen developed using multiple languages is no longer difficultas long as each language used is represented in the unifiedprogramming language model. Since we have been using theunified programming model for several years for programanalysis, though we have validated the approach for C andJava, we believe it will work for other languages as well.

REFERENCES

[1] Eclipse Integrated Developement Environmnet (IDE) for Java Develop-ers. http://eclipse.org/downloads/moreinfo/java.php

[2] Microsoft .NET Environment. www.microsoft.com/net[3] IBM Websphere. www.ibm.com/software/websphere[4] S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott,

G. C. Patton, B. M. Horowitz : Model-Based Testing in Practice. InProceedings of the International Conference on Software Engineering(ICSE), pp. 285-294 (1999).

[5] C. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace,V. J. Reddi, K. Hazelwood Pin: Building customized program analysistools with dynamic instrumentation. In Proceedings of the 2005ACM SIGPLAN conference on Programming Language Design andImplementation (PLDI ’05, pp. 190-200 (2005).

[6] A. Chawla, A. Orso : A Generic Instrumentation Framework forCollecting Dynamic Information. In ACM SIGSOFT ACM SoftwareEngineering Notes, Vol. 29, No. 5, pp. 1-4, September 2004.

[7] A. Seesing, A. Orso : InsECTJ: A Generic Instrumentation Frameworkfor Collecting Dynamic Information within Eclipse. In Proceedings ofthe eclipse Technology eXchange (eTX) Workshop at OOPSLA 2005.

[8] B. Buck, J. K. Hollingsworth : An API for runtime code patching. InJournal of High Performance Computing Applications 14(4), 317329(2000).

[9] L. De Rose, T. Hoover Jr., J. K. Hollingsworth : The dynamic probe classlibrary an infrastructure for developing instrumentation for performancetools”. In Proceedings of the 15th International Parallel & DistributedProcessing Symposium (IPDPS 2001), Washington, DC. IEEE ComputerSociety, Los Alamitos (2001)

[10] K. A. Lindlan, J, Cuny, A. D. Malony, S. Shende, B. Mohr, R. Riven-burgh : A tool framework for static and dynamic analysis of object-oriented software with templates. In Proceedings of SC 2000: HighPerformance Networking and Computing Conference (November 2000)

[11] H. Do, S. Elbaum, and G. Rothermel : Empirical Software Engineering:An International Journal, Volume 10, No. 4, pages 405-435, 2005.

[12] http://www.omg.org/spec/ASTM/1.0/[13] http://www.tcs-trddc.com

914