Metaprogramming techniques in Groovy and Grails

Numan Salatinuman.salati@gmail.comNY Groovy/Grails Meetup,

What makes a language dynamic?

• Dynamic type system• Mutable types• Flexible method dispatch• Evaluate code at runtime (access to the

interpreter)

Key Idea: late binding !

Dynamic features in Java

• dynamic class loading

• dynamic binding – subclass Polymorphism

• runtime annotations

• dynamic proxies and reflection API– mostly read only – dynamic implementation of interfaces

Dynamic Groovy

• Groovy has all this and much more

– Intercept methods/properties

– Create new methods/properties/constructors

– Create classes at runtime

– Runtime mixins (mutable types)

– Evaluate any valid code string

method invocation example

obj.method(arg1, arg2, arg3)

• In Java– single dispatch– invokedynamic bytecode instruction

• In Groovy– multiple dispatch– much more complicated logic but very flexible

method invocation example

What gets called in Java vs. Groovy ?

class Foo{

def print (Object o) { println “println object" }def print (String s) { println “println string" }

}

Object arg = "string"new Foo().print(arg)

Metaprogramming

• Wikipedia definitions:– Programs that write or manipulate other programs– Expose internals of runtime engine to programming code through

API”– Dynamic execution of string expression

• Meta object protocol: Make program semantics – Explicit– Extensible

How much of runtime and compile time structures are exposed?

Groovy MOP

Excellent support for metaprogramming

Compile time• Hook into the Groovy AST during compilation

Runtime• Hook into method dispatching• Dynamically create methods/properties• Mutable types• Execution of code strings

Example

>> groovyc Person.groovy>> javap –public Person

Compiled from "Person.groovy"public class Test extends java.lang.Object implements groovy.lang.GroovyObject {

…..

public groovy.lang.MetaClass getMetaClass(); public void setMetaClass(groovy.lang.MetaClass); public java.lang.Object invokeMethod(java.lang.String, java.lang.Object); public java.lang.Object getProperty(java.lang.String); public void setProperty(java.lang.String, java.lang.Object);

….}

class Person{ def name def sleep() { println "sleeping"}}

GroovyObject

GroovyObject• All Groovy classes implement this interface

• Open the class file in jad decompiler:

public Object invokeMethod(String s, Object obj){ return getMetaClass().invokeMethod(this, s, obj);}

public Object getProperty(String s){ return getMetaClass().getProperty(this, s);}

Default implementation delegates to metaClass

• Compiler assigns a metaClass to every POJO and POGO

GroovyObject

• These methods are the hooks into method dispatch and property access/assignment

• Overriding getProperty() and setProperty we can dynamically add properties and methods– This is exactly what Expando does• Dynamically create classes• Add methods by creating properties that are closures

Expando in Groovyclass SimpleExpando { def propertyMap = [:]

def getProperty(String name) { propertyMap[name]?: null }

void setProperty(String name, Object value) { propertyMap[name] = value; }

def invokeMethod(String name, Object args) { try {

metaClass.invokeMethod(name, args); } catch (GroovyRuntimeException e) { def value = propertyMap[name]; if (value instanceof Closure) { value.setDelegate(this)

value.call(args); } else { throw e } }

}}

s = new SimpleExpando()s.add = {x, y -> x + y}println s.add(19,1)

Why set delegate before invoking?

InvokeMethod

• Overriding invokeMethod in the class– intercepts all non existing method calls

• What if we want to intercept all method calls?– Implement GroovyInterceptable marker interface– Override invokeMethod(String name, args)– Careful about stack overflow!

• use metaClass.invokeMethod inside invokeMethod

• Only non existing methods?– implement methodMissing(String name, args) – higher precedence than invokeMethod if both present

MetaClass

• invokeMethod and methodMissing can be implemented in the class or on the metaClass

• Metaclass defines the dynamic behavior of the object

• Query runtime structure of the class/object– respondsTo– hasProperty– getMethods vs. getMetaMethods– getProperties vs. etMetaProperties

MetaClass• Define new methods and constructors on class using ExpandoMetaClass

Person.metaClass.play = { println "play"}Person.metaClass.eat = { pritnln "eat" }Person.metaClass.code = { println "code"}Person.metaClass.static.read = { println "reading" }Person.metaClass.constructor = {name -> new Person("Sir: " + name) }

Heap Overflow! - use BeanUtils.instantiateClass to instantiate outside of groovy

Person.metaClass { play { println "play"} eat { pritnln "eat" } code { println "code"} 'static' { read { println "reading" } } constructor { name -> BeanUtils.instantiateClass(Person, "Sir: " + name) }}

or using EMC DSL

EMC• Injection works for both POJOs and POGOs

– Integer.randomNum = { … }– String.metaClass = <Roll your own super cool metaClass>

• Add to instance only?

p1 = new Person()p2 = new Person()

p2.metaClass.party = { println "partying"}

p2.party()p1.party() MissingMethodException

• Works for POJOs too

EMC

• new methods are reflected in the subclass hierarchy

• Adding methods to interfaces?– set enableGlobally on EMC to affect all

implementing classes

Summary of method dispatch so far..

• If a method is defined in the metaClass invoke that

• Or else look for hooks in the class or metaClass:– invokeMethod– methodMissing– getProperty– setProperty

Method dispatch flow diagram

Categories

• Injection of methods within a scope

import org.codehaus.groovy.runtime.TimeCategoryuse(TimeCategory) {

println 2.days.from.nowprintln 3.years.from.nowprintln 10.minutes.from.nowprintln 3.weeks.from.now

}

• Injects getDays() and getYears() method defined in TimeCategory into the meta class of Integer objects in the “use” block and for the current thread

Categoriespublic class TimeCategory { ....

• public static DatumDependentDuration getMonths(final Integer self) {• return new DatumDependentDuration(0, self.intValue(), 0, 0, 0, 0, 0);• }

• public static DatumDependentDuration getYears(final Integer self) {• return new DatumDependentDuration(self.intValue(), 0, 0, 0, 0, 0, 0);• }

• public static Duration getWeeks(final Integer self) {• return new Duration(self.intValue() * 7, 0, 0, 0, 0);• }

• public static TimeDuration getHours(final Integer self) {• return new TimeDuration(0, self.intValue(), 0, 0, 0);• }

....

}

• All methods are static

• First argument is the class getting injected

Categories

• Can nest categories– in case of method clash, last one takes precedence

• Can use multiple categories in the “use” clause– same precedence rule as above

• Other built in categories– DomCategory– SerlvetCategory

Categories

• How it work internally:1. Creates new scope2. Adds static methods from category class into thread local stack3. Call closure4. Remove methods from stack

– check out “use” method in “GroovyCategoryStack.java”

• Slower than metaClass injection– scanning static methods– cleanup

Runtime Mixins• Java mixins vs. Groovy mixins

• Inject methods from other types

• Works on classes and interfaces

class Superman { def fly() { println "flying" }}

class Ninja { def fight() { println "fighting" }}

Person.mixin Superman, Ninjap = new Person()

p.sleep()p.fly()p.fight()

• Doesn’t not work on instances

• Global change

• Easier to use than Categories

• For method conflict last mixin takes precedence

Applications

• Dynamic finders• Builders• Custom DSL• Dependency Injection• Method injection• Interceptors

Dynamic finders in Grails

private static addDynamicFinderSupport(GrailsDomainClass dc, ….) { def mc = dc.metaClass

def dynamicMethods = [ … ]

mc.static.methodMissing = {String methodName, args -> def result = null StaticMethodInvocation method = dynamicMethods.find {it.isMethodMatch(methodName)} if (method) { synchronized(this) { mc.static."$methodName" = {List varArgs -> method.invoke(dc.clazz, methodName, varArgs) } } result = method.invoke(dc.clazz, methodName, args) } else { throw new MissingMethodException(methodName, delegate, args) } result } }

Register method on metaclass for faster lookup on subsequent invocations

findAllBy, CountBy, ListOrderBy patterns…

HibernatePluginSupport.groovy

Builders• Easy way to hierarchical/recursive structures like XML, GUI components

builder = new NodeBuilder()root = builder.persons {• person (name: 'obama') {• address(zip: 10016, street: 36)

profession 'president'• }

• person (name: 'joe') {• address(zip: 30312, street: 12)

profession 'vice-president'• }

}

println root.'**'.profession GPath expression

Main Concepts

• Method interception through invokeMethod or methodMissing– intercept method calls and dynamically create a

node

• Closure delegates – make sure all closures are relaying method calls to

the builder

Anatomy of a builderbuilder = new NodeBuilder()root = builder.persons {• person (name: 'obama') {• address(zip: 10016, street: 36)

profession 'president'• }

• person (name: 'joe') {• address(zip: 30312, street: 12)

profession 'vice-president'• }

}

println root.'**'.profession

1. create node person by intercepting g the method call ( e.g builder.persons {…} )

2. Execute the closure but first set the delegate to the builder

3. Recursively do this until all nodes are created

How to write a builder• Extends BuilderSupport

public class NodeBuilder extends BuilderSupport {

• public static NodeBuilder newInstance() {• return new NodeBuilder();• }

• protected void setParent(Object parent, Object child) {• }

• protected Object createNode(Object name) {• return new Node(getCurrentNode(), name, new ArrayList());• }

• protected Object createNode(Object name, Object value) {• return new Node(getCurrentNode(), name, value);• }

• protected Object createNode(Object name, Map attributes) {• return new Node(getCurrentNode(), name, attributes, new ArrayList());• }

• protected Object createNode(Object name, Map attributes, Object value) {• return new Node(getCurrentNode(), name, attributes, value);• }

• protected Node getCurrentNode() {• return (Node) getCurrent();• }

}

BuilderSupport takes care of method interception and setting closure delegates

Some common builders

• Grails- MarkupBuilder- SwingBuilder

• Groovy- ConstrainedPropertyBuilder- BeanBuilder- HibernateCriteriaBuilder

AST Transformation

• Compile time metaprogramming technique

• Example from languages– C Macros (preprocessing)– C++ Templates (compile time instantiation)– Lisp Macros (very powerful)– Java Annotations (mostly code gen)

Basic Idea

• You can manipulate code at many representations – Source (templating), AST, Bytecode (e.g AspectJ), runtime

• Hook into compilation process and manipulate the AST– Higher level abstraction that bytecode

– Different from traditional java annotations where transformations happen outside of the compiler

Groovy compilation process: 100,000 ft view

Source

Tokens

Antlr AST

Groovy AST

Bytecode

Lexical analysis with a scanner

Parsing with ANTLR

Transform ANTL AST To Groovy AST

- Semantic analysis- Canonicalization- Instruction selection- Class generation- Output- Finalization

Compile phases on the AST

Example AST 1 def sum (List lst){ 2 def total = lst.inject(0) { s, i -> s = s + i } 3 println total 4 return total 5 }

println totalExpressionStatement

MethodCallStatement

VariableExpressions ConstantExpression ArgumentListExpression

“this” “println” VariableExpression

“total”

Example ASTdef total = lst.inject(0) { s, i -> s = s + i }

Parameter

ExpressionStatement

DeclarationExpression

VariableExpression

ArgumentListExpressionConstantExpression

ClosureExpression

BlockStatement

ExpressionStatement

BinaryExpression

BinaryExpression

VariableExpression

VariableExpression

VariableExpression

MethodCallExpression

VariableExpression

ConstantExpression

“=”

Parameter

“total”

“lst” “inject”

“=”

“0”

“s” “i”

“s” “i”

“=”“s”

Types of transformations

• Local– Applied to tagged (annotated) elements– Annotation driven– Can only be applied to Semantic Analysis phase or

later

• Global– applied to all classes that are compiled

Local AST Transformation

Steps:1. Create your transformation class by implementing

ASTTransformation interface.• specify compile phase

@GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)

2. Create annotation and link to your transformation class

• @GroovyASTTransformationClass(“full path to your transformation class”)

3. Write client code and annotate your elements (methods, fields, classes etc)

Step 1@GroovyASTTransformation (phase = CompilePhase.SEMANTIC_ANALYSIS)class LoggingASTTransformation implements ASTTransformation{

def void visit(ASTNode[] nodes, SourceUnit sourceUnit)• {

def methodList = sourceUnit.ast?.methods.findAll {MethodNode method -> method.getAnnotations(new ClassNode(WithLogging))

• }

methodList.each {MethodNode method ->• Statement startMessage = createPrintlnAst("Starting $method.name")• Statement endMessage = createPrintlnAst("Ending $method.name")

• Statement code = method.getCode()• List existingStatements = code.getStatements()

existingStatements.add(0, startMessage) existingStatements.add(endMessage)

• }• }

• private Statement createPrintlnAst(String message)• {• return new ExpressionStatement(• new MethodCallExpression(• new VariableExpression("this"),• new ConstantExpression("println"),• new ArgumentListExpression(• new ConstantExpression(message)• )• )• )• }

}

AST through sourceUnit

Creating AST for simple statement. YUCK!

Compile Phase

Expression and statements within the method body

Visitor pattern

Step 2

@Retention(RetentionPolicy.SOURCE)

@Target([ElementType.METHOD])@GroovyASTTransformationClass([“full path to your transformation class” "])public @interface WithLogging {

}

Step 3

@full.path.to.WithLoggingdef sum(List lst){ def total = lst.inject(0) { s, i -> s = s + i } println total

• return total}

Examples• @Immutable

– No mutators– All fields must be private and final– All fields must be included in equals, hashCode and toString computation– class must be final

• @Singleton– lazy flavor– static instance

• Grails– @EntityASTTransformation:

• Injects Id, Version, toString and Associations to grails domain classes

Final Thoughts on AST Transformations

• Cumbersome to write transformation currently

• Future tooling (Groovy 1.7):– AST Browser– AST Builder

Summary of techniques• evaluate(“def add = {x, y -> x + y”)

– Evaluate string as code

• invokeMethod– Intercept all method call (Existing and non existing methods)

• methodMissing– Intercept only non existing methods

• getProperty/setProperty– Intercept property access and assignments

• ExpandoMetaClass– Dynamically add methods, constructors, properties

• Categories– scoped injection

• Runtime Mixins– add methods from other types

• AST Transformations– Transformations on groovy AST

Can be defined on the class itself or on the metaClass

Runtime

Compile time

References1. What’s new in Groovy 1.6: http://www.infoq.com/articles/groovy-1-6

2. Hamlet D’Arcy blog: http://hamletdarcy.blogspot.com

3. Book: “Groovy in Action” by Dierk Koenig with Andrew Glover, Paul King, Guillaume Laforge and Jon Skeetsdsd

4. Various examples on http://groovy.codehaus.org