Lecture 5-6
• Extract data from code repository / source code / run-time data
• Self organizing maps
• SOM in search-based software engineering
Lecture 3-4: Program comprehension
• Program comprehension in software engineering
• Search Based Software Engineering for Program Comprehension
• Case studies – clustering based
Refactoring identification using clustering- overview
How to extract information from source code repositories
Source code repositories (version control, revision control or source control) - Software applications used for the management of changes to software artifacts (source code, html documents, images, etc)
1 Use Command line tools / batch files:
Example for: GIT - https://git-scm.com/
git clone <repo> => copy existing repository from a remote location
git branch -r => print all remote branches
git log => pint all commits
2 Programmatically using APIs:
Example for SVN (Subversion) - https://subversion.apache.org/ Sample framework: https://svnkit.com
repo = SVNRepositoryFactory.create( SVNURL.parseURIDecoded( url ) );
authM = SVNWCUtil.createDefaultAuthenticationManager( name , pwd );
repo.setAuthenticationManager(authM);
entries = repo.getDir( path, -1 , null , (Collection) null );
iterator = entries.iterator( );
while ( iterator.hasNext( ) ) {
SVNDirEntry entry = ( SVNDirEntry ) iterator.next( );
System.out.println(entry.getName( ))
if ( entry.getKind() == SVNNodeKind.DIR ) {
listEntries(repo, entry.getName( ));
}
}
3 Using existing client applications:
Ex: https://bitbucket.org
How to extract information from source code / machine code
From source code to executed programs:
Source: https://www.cs.rit.edu/~ats/cc-2010-2/tutorial/
Source: 1. GCC Marvell Taiwan Intern Wei-Sheng Chou 1
How to extract information from source code / machine code
• Textual search, regular expressions
▪ hard to extract semantic information (usage, inheritance, etc.)
• Source code parser
▪ use parser generators
▪ Ex. ANTLR (http://www.antlr.org/) can generate lexers, parsers, tree parsers,
and combined lexer-parsers.
o Parsers can automatically generate abstract syntax trees which can be
further processed with tree parsers.
o ANTLR provides a notation for specifying lexers, parsers, and tree parsers.
o Grammar exist for: c/c++,java, python, vb, etc.
(https://github.com/antlr/grammars-v4)
• Introspection
▪ obtain information about a class during run-time (methods, fields, parameters,
etc.)
▪ not all languages support this.
▪ Usually a limited set of information (ex. Can not go inside a method)
• instrumentation
▪ adding extra code (usually automatically, macros/instrumenting byte-
code/instrumenting code at the compiler level, etc) to record certain aspects of
the (running) code
• byte code analysis
▪ specialized frameworks to analyze the byte code
▪ useful for languages like java/.net (languages using “Virtual Machines”)
▪ ex. ASM framework - Java byte-code manipulation and analysis framework
▪ get detailed information (method, instruction level)
▪ use the byte code representation (no need for the source code)
Generate or use an existing source code parser
ANTLR - (ANother Tool for Language Recognition)
ANTLR is a powerful parser generator that you can use to read, process, execute, or translate structured text or binary files.
Given a grammar: Hello.g4
grammar Hello; r : 'hello' ID ; // match keyword hello followed by an identifier ID : [a-z]+ ; // match lower-case identifiers WS : [ \t\r\n]+ -> skip ; // skip spaces, tabs, newlines
Generate a parser: antlr4 Hello.g4
Use Existing parsers
org.eclipse.jdt.core.dom.ASTParser
A Java language parser for creating abstract syntax trees (ASTs).
char[] source = ...;
ASTParser parser = ASTParser.newParser(AST.JLS3); // handles JDK 1.0,.., 1.6
parser.setSource(source);
// In order to parse 1.5 code, some compiler options need to be set to 1.5
Map options = JavaCore.getOptions();
JavaCore.setComplianceOptions(JavaCore.VERSION_1_5, options);
parser.setCompilerOptions(options);
CompilationUnit result = (CompilationUnit) parser.createAST(null);
Example: Create basic AST from source string
Inspecting java byte-code using ASM Framework
• http://asm.ow2.org/
• in many cases reflection is not sufficient we may need more detailed data
• ASM can be used to inspect java byte code
• offer access at the instruction level
• using ASM we can perform static analysis on the application, we can extract
dependencies, annotations, etc. using just the byte code (no source code needed)
• the framework use the Visitor design pattern, class byte code is parsed and using
custom visitor classes (class visitor, method visitor, annotation visitor,...)
information can be gathered
Visitor design pattern
ASM sample analyzer
public class ClassInfoCollector implements ClassVisitor { public static int allClasses = 0; public static int nrMethVizited = 0; private MClass currentClass; public void visit(int version, int access, String name, String signature, String superName, String[] interfaces) { if (name.contains("$")) { currentClass = null;// e un inner class return; } allClasses++; currentClass = infC.getCreateMClass(name, superName); } public FieldVisitor visitField(int access, String name, String desc, String signature, Object value) { if (currentClass == null) { return null; } currentClass.addCreateField(name, desc, access); return null; } public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) { nrMethVizited++; if (currentClass == null) { return null; } if (name.equals("<init>")) { // e constructor implicit return null; } if (name.equals("main") && desc.equals("([Ljava/lang/String;)V")) { currentClass = null;// e un main return null; } MMethod meth = currentClass.addCreateMethod(name, desc, access); return new MethodInfoCollector(infC, meth); }
Sample - Java Instrumentation
Package java.lang.instrument Description Provides services that allow Java programming language agents to instrument programs running on the
JVM. The mechanism for instrumentation is modification of the byte-codes of methods.
Key Components of Java Instrumentation
• Agent – is a jar file containing agent and transformer class files.
• Agent Class – A java class file, containing a method named ‘premain’.
• Manifest – manifest.mf file containing the “premain-class” property.
• Transformer – A Java class file implementing the interface ClassFileTransformer.
An agent is started by adding this option to the command-line:
-javaagent:jarpath[=options]
Instrumentation Agent Class
Agent class contains the premain method and that is key in Java insturmentation.
This is similar to the ‘main’ method. This class is loaded by the same system
classloader as it loads the other Java classes. premain method can have the follow-
ing signatures,
1. public static void premain(String agentArgs, Instrumentation inst);
2. public static void premain(String agentArgs);
Instrumentation Transformer
Transformer classes are to be registered with ‘instrumentation’ instance. All those
transformers that are registered with instrumentation instance will be invoked by
the classloader every time a new class is loaded by that.
Java Instrumentation Activity Sequence
Generate assembly / abstract code representation
https://godbolt.org/ analyse assembly generated by different compilers / languages
Use the output generated by the compiler.
Example:
g++ -S foo.cpp => generate assembly for foo.cpp
clang -emit-llvm foo.cpp => generate LLVM IR for foo.cpp
LLVM
LLVM Project (https://llvm.org/) is a collection of modular and reusable compiler and toolchain
technologies capable of supporting both static and dynamic compilation of arbitrary programming
languages.
API to generate instructions in a format called an intermediate representation, or IR. LLVM can then
compile the IR into a standalone binary or perform a JIT (just-in-time) compilation on the code to run
in the context of another program, such as an interpreter for the language.
Can be used for many programming languages as a compiler backend: C, C++, Rust, Swift, Haskell,
Python, Go, Node.js, C#, Fortran
Source: https://community.arm.com/tools/hpc/b/hpc/posts/advancing-scientific-codes-with-arm-fortran-
compiler
Self organizing maps
A self-organizing map (SOM) - self-organizing feature map (SOFM) or Kohonen
map - is a type of artificial neural network that is trained using unsupervised learning to produce a low-
dimensional (typically two-dimensional), discretized representation of the input space of the training
samples, called a map.
Self-organizing maps are different from other artificial neural networks in the sense that they use a
neighbourhood function to preserve the topological properties of the input space.
The SOM algorithm is based on unsupervised, competitive learning. It provides a
topology preserving mapping from the high-dimensional space to map units.
Map units, or neurons, usually form a two-dimensional grid and thus the mapping is a mapping from a
high-dimensional space onto a plane.
topology preserving - groups similar input data vectors on neurons: points that are near each other in
the input space are mapped to nearby map units in the SOM.
The SOM can serve as a clustering tool as well as a tool for visualizing high-dimensional data.
Given a set of input object represented using a vector space model (each object is represented as set of
features) after training the SOM will map each input object to a lower dimensional space. The obtained
mapping can be use for visualization, clustering or to discover hidden patterns in the input data.
SOM Network architecture
A self-organizing map consists of components called nodes or neurons. Associated with each node are a
weight vector of the same dimension as the input data vectors, and a position in the map space
Each neuron on the computational layer is fully connected to all the nodes in the input layer
Computational layer may arranged:
• hexagonal or rectangular grid
• toroidal grid (opposite edges are connected)
Self organizing maps – training
Unsupervised learning - does not need a target output to be specified
usually performed using the Kohonen algorithm.
There are two phases of operation:
• the similarity matching phase
o the distances between the inputs and the weights are computed. Next, the output node
having the minimum distance is chosen and is declared as the “winner” node (best matching
unit)
• the weight adaptation phase.
o the weights from the inputs to the “winner” node are adapted. In addition, a topological
neighbourhood of the winning node is defined and the weights connecting the inputs to the
nodes contained in this topological neighbourhood are also adapted
Training occurs in several steps and over many iterations:
• Each node's weights are initialized.
• A vector is chosen at random from the set of training data and presented to the lattice.
• Every node is examined to calculate which one's weights are most like the input vector. The
winning node is commonly known as the Best Matching Unit (BMU).
• The radius of the neighbourhood of the BMU is now calculated. This is a value that starts large,
typically set to the 'radius' of the lattice, but diminishes each time-step. Any nodes found within
this radius are deemed to be inside the BMU's neighbourhood.
• Each neighbouring node's (the nodes found in step 4) weights are adjusted to make them more
like the input vector. The closer a node is to the BMU, the more its weights get altered.
• Repeat step 2 for N iterations.
Self organizing maps – sample implementation
public void train(int nrIteration, SOMTrainData trData, double startLearnigRate, double startNeighborRadius, SOMTrainingListener l, SOMTrainInputChooser inputChooser) { double lambdaTimeConstant = nrIteration / Math.log(startNeighborRadius); for (int iteration = 0; iteration < nrIteration; iteration++) { // calculez learningRate double learningRate = startLearnigRate * Math.exp(-(double) iteration / lambdaTimeConstant); double neighborRadius = startNeighborRadius * Math.exp(-(double) iteration / lambdaTimeConstant); // alege un numar(corespunzator unui vector de intrare) int[] inputIndexes = inputChooser.getNextInputIndex();
for (int i = 0; i < inputIndexes.length; i++) { int inputIndex = inputIndexes[i]; trainingStep2(iteration, trData.get(inputIndex), learningRate, neighborRadius, l, trData.getLabel(inputIndex)); } } } public void trainingStep2(int curIter, double input[], double learningRate, double neighborRadius, SOMTrainingListener l, Object lbl) { // caut Best maching unit BMU bmu = computeBestMatchingUnit(input, lbl); // actualizez weights de la bmu bmu.getNeuron().adjustWeights(input, learningRate, 1); // obtin lista vecinilor ce pica in radiusul cerut List<NeighborSOMNeuron> neighbors = comptationL.getNeighbors( bmu.getNeuron(), neighborRadius); // actualizez valorile in functie de cat de apropiat e de BMU if (neighbors != null && neighbors.size() > 0) { updateNeighbors(input, learningRate, neighborRadius, neighbors); } } public BMU computeBestMatchingUnit(double[] input) { int nrNeurons = comptationL.getNrNeurons(); SOMNeuron bmuN = comptationL.getNeuron(0); // neuronul castigator double minDist = dist.distance(input, bmuN.getWeights()); for (int i = 1; i < nrNeurons; i++) { SOMNeuron neuron = comptationL.getNeuron(i); double d = dist.distance(input, neuron.getWeights()); if (d < minDist) { // cu cat distanta e mai mica, neuronul e mai aproape de input minDist = d; bmuN = neuron; } } return new BMU(bmuN, minDist, input); }
How to use self-organizing maps
Describe input instances as a list of feature values and train the SOM.
We can visualize the SOM:
• Visualize the underlining neurons (topology) and outline the BMU (best matching unit) for every instance
• use the UMatrix to get more insight into the learned weights
• represent the neurons in 2D/3D according to their associated weights
The U-matrix value of a particular node is the average distance between the node and its closest neighbors. In a rectangular
grid for instance, we might consider the closest 4 or 8 nodes. The U-Matrix method uses the distances between the units in a
SOM as a boundary defining criteria. These distances can be then be displayed as heights giving a U-matrix landscape
Why use SOM:
• SOM is able to discover hidden patterns in the data
• we analyze the final SOM (after training) and similar instances will have there associated BMU close to each
other in the network
• we can group (partition) the instances. On the map we can identify group of related instances (there BMU) by
visual inspection (maybe using the UMatrix)
• Interpretation of the U-matrix: altitudes or the high places on the U-matrix will encode data that are dissimilar
while the data falling in the same valleys will represent input vectors that are similar.
• heuristic (find instances in a close range)
• using clustering algorithm on the neurons
• we can use the SOM as a classifier.
• If we know the class of every instances used in training we can classify new instances based on the BMU.
Compute the BMU for the new instance and select the closest instance from the training data set.
Self organizing maps – sample apps
UMatrix
Self organizing maps – sample apps
Software restructuring using self organizing maps
The software system is analyzed and we obtain: list of classes, methods, attributes and relationships between them
(inheritance, composition, dependency)
Each entity (class, method or class) s_i (1 <= i <= n) from the software system S is characterized by a l - dimensional
vector: (s_i1, s_i2, . . . , s_il), where sij (any j, 1 <= j <= l) is computed as follows:
for a given entity e in S, p(e) is a set of entities from S that are related to e:
• If e in Attr(S) (e is an attribute) then p(e) consists of: the attribute itself, the application class where the attribute is
defined, and all methods from Meth(S) that access the attribute.
• If e in Meth(S) (e is a method) then p(e) consists of: the method itself, the application class where the method is
defined, all attributes from Attr(S) accessed by the method, all the methods from S used by method e, and all
methods from S that overwrite method e.
• If e in Class(S) (e is an application class) then p(e) consists of: the application class itself, all attributes and
methods defined in the class, all interfaces implemented by class e and all classes extended by class e.
Train the SOM using the above vector space model for the entities from the software system.
Use the obtained mapping to group entities from the software system.
Software restructuring using self organizing maps – example
public class Class_A { public static int attributeA2; public static int attributeA1; public static void methodA1() { methodA2(); } public static void methodA2() { attributeA2 = 0; Class_A.attributeA1 = 12; } public static void methodA3() { attributeA2 = 0; methodA1(); methodA2(); } } public class Class_B { private static int attributeB1; private static int attributeB2; public static void methodB1() { attributeB1 = 0; Class_A.methodA1(); } public static void methodB2() { attributeB1 = 0; attributeB2 = 0; Class_A.attributeA1 = 12; } public static void methodB3() { attributeB1 = 0; methodB1(); methodB2(); Class_A.attributeA1 = 12; } public static void methodB4() { attributeB1 = 0; methodB2(); Class_A.attributeA1 = 12; } }
Restructuring using SOM – classA; classB
Software restructuring using self organizing maps – example
UMatrix for JHotDraw
Package restructuring using self-organizing maps – discussion
Given a software system establish a proper package structure for the project
