Plugging into the Java CompilerÉamonn McManus <[email protected]>Christian Gruber <[email protected]>
JavaOne 2014
Speaker introduction: Éamonn McManus
● At Google since 2011○ Gmail servers initially○ App Engine SDK now○ 20% time on Java Core Libraries (Guava)
● Formerly at Sun then Oracle, on the JDK team● Author of two annotation processors
○ AutoValue (based on an idea by Kevin Bourrillion)○ JavaFX Builder generator
JavaOne 2014
Speaker introduction: Christian Gruber
● At Google since 2009○ “Test Mercenary”○ Mobile testing and development infrastructure○ Java Core Librarian
■ focusing on Dependency Injection (guice, dagger), and testing● Formerly at Sun/JavaSoft (via Lighthouse) and Oracle Consulting● Co-author of
○ Dagger 1 and 2, dependency injection using annotation processors○ auto-common, tools for easing annotation processor development
● Created (with other Googlers) the Truth assertion/testing library● Open-source bigot
JavaOne 2014
Overview
● What is an annotation processor?● Example: AutoValue● Example: Dagger● Write your own annotation processor● Q&A
JavaOne 2014
Reminder: Annotations
@SuppressWarnings(“unchecked”)public class Foo { @SafeVarargs public void bar(Optional<String>... args) {...} public void baz(@Nullable String s) {...}}
JavaOne 2014
What is an annotation processor?
● A way to extend the Java compiler● Standardized by JSR 269 in Java 6
○ javax.annotation.processing, javax.lang.model○ supported by javac (JDK) and ecj (Eclipse)
● Analyze Java code being compiled● Maybe introduce new errors and warnings● Maybe generate new Java code
JavaOne 2014
Exam
ple:
AutoValue
JavaOne 2014
Value Types
● A value type is a class where:○ properties never change (immutable)○ instances with the same properties are interchangeable
● Immutability is good!○ easy to reason about○ thread-safe
● Many uses in Java, for example:○ returning more than one value from a method○ combining values for use in a map key or value
■ Map<Tuple3<String, Integer, Country>, Tuple2<Long, Long>> ?
JavaOne 2014
Value types (ideal simplicity)
public class Address { public final String streetAddress; public final int postCode;
public Address( String streetAddress, int postCode) { this.streetAddress = streetAddress; this.postCode = postCode; }}
JavaOne 2014
Accessors and validation
public class Address { private final String streetAddress; private final int postCode;
public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }
public String streetAddress() { return streetAddress; }
public int postCode() { return postCode; }}
JavaOne 2014
equals, hashCode, toStringpublic class Address { private final String streetAddress; private final int postCode;
public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }
public String streetAddress() { return streetAddress; }
public int postCode() { return postCode; }
@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false; } }
@Override public int hashCode() { return Objects.hash(streetAddress, postCode); }
@Override public String toString() { return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}"; }}
JavaOne 2014
equals, hashCode, toStringpublic class Address { private final String streetAddress; private final int postCode;
public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }
public String streetAddress() { return streetAddress; }
public int postCode() { return postCode; }
@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false; } }
@Override public int hashCode() { return Objects.hash(streetAddress, postCode); }
@Override public String toString() { return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}"; }}
postCode
JavaOne 2014
AutoValue to the rescue
@AutoValue public abstract class Address { public abstract String streetAddress(); public abstract int postCode();
public static Address create( String streetAddress, int postCode) { return new AutoValue_Address( streetAddress, postCode); }}
JavaOne 2014
AutoValue generates subclass
class AutoValue_Address extends Address { private final String streetAddress; private final int postCode; AutoValue_Address(String streetAddress, int postCode) { ...check streetAddress not null... ...assign fields... } @Override public String streetAddress() {...} @Override public int postCode() {...} @Override public boolean equals(Object o) {...} @Override public int hashCode() {...} @Override public String toString() {...}}
JavaOne 2014
Foo.java
Bar.java
Address.java
@AutoValue ⇒
@AutoValue
AutoValue_Address.java
Annotation processing (1)
generatecompileAutoValueProcessor
JavaOne 2014
compile No furtherannotations
AutoValue_Address.java
Annotation processing (2)
JavaOne 2014
AutoValue_Address.java
generatecode
Foo.class
Bar.class
Address.class
AutoValue_Address.class
Annotation processing (3)
Address.java
Bar.java
Foo.java
JavaOne 2014
Demo
JavaOne 2014
Exam
ple:
Dagger
JavaOne 2014
Dagger
● Dependency-injection framework using JSR 330● Directed Acyclic Graph... of classes and their dependencies.● Dagger 1.x
○ Created by Googlers, and ex-Googlers at Square■ Christian Gruber, Jesse Wilson, Jake Wharton, and others...
○ Open source with contributions from Square, Google and others○ Source code generation and compile-time analysis
● Dagger 2.x○ 100% compile-time, via annotation processing and generated sources○ originated at Google, with design oversight by Dagger 1 contributors.
■ Concept by Greg Kick and Christian Gruber, impl mainly by Greg Kick
JavaOne 2014
Dependency Injection
● Having a class know how to obtain its collaborators is fragile○ Hard to change implementation○ Hard to unit-test
● Pattern described by Martin Fowler:○ http://www.martinfowler.com/articles/injection.html
● Early examples:○ Spring, PicoContainer, Apache HiveMind/Tapestry
● Later examples:○ Guice, CDI, Dagger
JavaOne 2014
Static dependencies crystallized in constructors
public class MailServer { // ... public MailServer() { this.messageStore = new MessageStoreImpl(); this.userService = new UserServiceImpl(); // ... }}
● Cannot test MailServer without invoking MessageStoreImpl, etc.● Cannot swap in alternate messaging, auth, etc.● Shared collaborators (singletons) end up requiring global statics
JavaOne 2014
Instead, declare the dependencies, and pass them in...
public class MailServer { // ...
public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; // ... }}
JavaOne 2014
Automation from annotation signals - no writing the wiring
public class MailServer { // ...
@Inject public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; // ... }}
JavaOne 2014
Explicit configuration is defined using annotations...
@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl; } // Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn); }}
JavaOne 2014
Access to the graph defined by @Component
@Component public interface Services { MailServer mailServer(); NotificationServer notificationServer();}
● Annotations provide the signals● Graph analysis begins at annotated interfaces● All the “wiring” or “glue” code for the graph is generated● @Component interfaces’ implementation generated with a builder
for configuration● Plain old java code
JavaOne 2014
Explicit configuration is defined using annotations...
@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl; } // Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn); }}
JavaOne 2014
Why Annotation Processors for D-I?
● Guice, Spring, etc. work well, so why processors and code-gen?
● Performance○ Reflection is expensive in some environments, such as Android○ Graph Validation is work - can affect startup times
● Developer productivity:○ Errors at compile-time vs. load time or even later improves velocity○ Generated code less "magical" and can be seen and reasoned about
● Design○ Knowing the structure of the code lets us build cleaner approaches
than the "magic map" of Injector/Container frameworks.
JavaOne 2014
Demo
Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues
JavaOne 2014
Write
Your
Own
JavaOne 2014
What processors can and cannot see
● Processors can see the structure of your code:○ Class names, inheritance, generics○ Method names, parameter types, return types, generics○ Field names, types, compile-time constant values
● Processors cannot see the contents of the code○ Static initialization blocks○ Method bodies○ Initializer expressions
JavaOne 2014
What processors can and cannot do
● Processors can do quite a few things, such as○ generate new Java source code to be compiled○ generate other files (XML, META-INF/services, arbitrary text)○ perform analysis and emit warnings and errors○ associate the errors with specific source elements
● Processors cannot, however○ modify the code of existing classes
■ including source they generate once written○ introduce new fields or methods into existing classes○ introduce new nested classes
JavaOne 2014
"Annotation" processors
● Annotation processors are usually associated with annotations, obviously
● But, you can also write a processor that analyzes all input classes, whether annotated or not○ Return Collections.singleton("*") from getSupportedAnnotationTypes()
JavaOne 2014
Defining an annotation
import java.lang.annotation.*;
@Retention(RetentionPolicy.SOURCE)@Target(ElementType.TYPE)// @Documentedpublic @interface MyAnnotation { String value() default "";}
// @MyAnnotation class Foo {...}// @MyAnnotation("bar") interface Baz {...}// @MyAnnotation(value = "buh") enum Wibble {...}
JavaOne 2014
Outline of a processor (1)
import javax.annotation.processing.*;import javax.lang.model.*;
@AutoService(Processor. class)public class MyProcessor extends AbstractProcessor { @Override public Set<String> getSupportedAnnotationTypes() { return ImmutableSet.of(MyAnnotation. class.getName()); } @Override public SourceVersion getSupportedSourceVersion() { return SourceVersion.latestSupported(); } ...}
JavaOne 2014
Outline of a processor (2)
import javax.annotation.processing.*;import javax.lang.model.*;
@AutoService(Processor. class)public class MyProcessor extends AbstractProcessor { ... @Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class); handle(annotatedElements); return false; }}
JavaOne 2014
Claiming annotations
● An annotation processor can say that it "supports" one or more annotations (getSupportedAnnotationTypes)
● Its process method will be called only for program elements where those annotations appear
● If it returns true, it has "claimed" the annotations and a later processor that also supports them will not be called
● We recommend never claiming an annotation○ You don't know what that later processor is or whether it should run
JavaOne 2014
API tips: Types and Elements
● A lot of useful functionality is contained in javax.lang.model.util.{Types,Elements}
● If you can't figure out how to do something, check these interfaces to see if they hold the solution
● Get an instance of either in your process method or any method it calls, via the inherited processingEnv field:@Override public boolean process(...) { Types typeUtils = processingEnv.getTypeUtils(); Elements elementUtils = processingEnv.getElementUtils();
JavaOne 2014
API Tips: ElementFilter
@Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class);
List<ExecutableElement> annotatedMethods = ElementFilter.methodsIn(annotatedElements); handle(annotatedMethods); return false; }
JavaOne 2014
API tips: TypeMirror and TypeElement
● The distinction between TypeElement and TypeMirror is subtle● In practice, use whichever one the API gives you, or convert to the
other if the method you need is there● TypeMirror → TypeElement:
Types typeUtils = processingEnv.getTypeUtils();TypeElement typeElement = (TypeElement) typeUtils.asElement(typeMirror);
● TypeElement → TypeMirror:TypeMirror typeMirror = typeElement.asType();
orDeclaredType typeMirror = typeUtils.getDeclaredType(typeElement);
JavaOne 2014
API tips: Upstream errors can cause processor failure
● Processed code may be missing literal elements, be missing imports, be in a broken state, etc.○ Very strange results including core javac types in ClassCastExceptions or
NullPointerExceptions deep within the compiler.○ Upstream compile errors are masked by these exceptions.
● Auto-common provides SuperficialValidation.validateElements(...) ○ Simple sanity check for each Element and contents○ If validateElements() returns false, two options:
■ skip processing supplied elements (if done in a loop)■ return from the processor immediately
JavaOne 2014
API tips: Gotchas
● TypeMirror.equals() is not a reliable comparison. Should use:○ Types.isSameType(mirror1, mirror2) (if available)○ MoreTypes.equal(mirror1, mirror2) (if in a static context)
● TypeMirror instanceof checks are generally incorrect○ MoreTypes.asDeclared(typeMirror) converts correctly or throws IAE○ MoreTypes.asArray(...)... etc. - converters for all TypeMirror kinds.
JavaOne 2014
API tips: Google’s auto-common utilities
● MoreTypes and MoreElements○ Wrappers for equivalence○ Static methods for conversion and comparison
● SuperficialValidation○ For simple sanity checking of elements before processing
● AnnotationMirrors and AnnotationValues○ convenience methods and Equivalence wrappers○ static utilities to handle default values
● Types/Elements have useful methods, but are instances. Auto common utilities provides many static utility method equivalents.
Note: AnnotationValues and AnnotationMirrors pending extraction from Dagger 2.
JavaOne 2014
Generating code: JavaWriter
● Build up your class programmatically.● Refer to elements without using “stringly-typed” references.● Write to any Appendable.
JavaWriter javaWriter = JavaWriter.inPackage("some.package");ClassWriter klass = javaWriter.addClass("SomeType");VariableWriter field = klass.addField(String.class, "foo");field.addModifiers(PRIVATE, FINAL);ConstructorWriter constructor = klass.addConstructor();VariableWriter p0 = constructor.addParameter(Key.class, "key");constructor.body() .addSnippet("this.%s = %s.getFormat();", field.name(), p0.name());javaWriter.write(appendable);
JavaOne 2014
Generating code: JavaWriter
● Get clean, organized and readable code out.
package some.package;
import java.security.Key;
class SomeType { private final String foo;
SomeType(Key key) { this.foo = key.getFormat(); }}
Note: JavaWriter 3 is pending review with Square and extraction from Dagger2. JavaWriter 2 is available.
JavaOne 2014
Generating code: JavaWriter
● Pros:○ Programmatic creation of code○ Reduce errors with cross referencing○ Automatic handling of imports and type shortening○ Very useful in loops where each loop touches many parts of the code○ Some built-in validation based on code structure
■ some erroneous things are simply impossible to write● Cons:
○ Code that generates code can look quite different in shape than output
JavaOne 2014
#foreach ($p in $props) @Override ${p.access}${p.type} ${p}() { #if ($p.kind == "ARRAY") #if ($p.nullable) return $p == null ? null : ${p}.clone(); #else return ${p}.clone(); #end #else return $p; #end }#end
Generating code: templates
@Override public int[] postCodes() { return postCodes == null ? null : postCodes.clone(); }
JavaOne 2014
Generating code: templates
● Apache Velocity is a good choice of template engine○ Supported in all major IDEs (even Emacs)○ Directives clearly distinguishable from Java code snippets
● In AutoValue we do a post-processing step to remove superfluous spaces and blank lines, just so the generated code looks nicer
JavaOne 2014
Testing Processors - Unit testing environment
● Unit testing still requires javac environment, for Types/Elements● Compile-testing has a CompilationRule suitable for JUnit4
import com.google.testing.compile.CompilationRule;
@RunWith(JUnit4.class)public class SomeProcessorTest { @Rule public final CompilationRule compilationRule = new CompilationRule();
private Elements elements; private Types types;
@Before public void setUp() { this.elements = compilationRule.getElements(); this.types = compilationRule.getTypes(); }
JavaOne 2014
Testing Processors - Failing integration tests
● Need to test failing compiles without failing the outer build● Compile-testing has assertions based on the Truth library● These assertions can variously:
○ assert about javac runs success or failure○ be configured to run arbitrary Processor instances○ run against source from files or strings○ execute and store results in-memory○ assert about errors with specific message contents and locations○ assert against contents of generated files○ assert against contents of generated java source, comparing AST
JavaOne 2014
Testing Processors - Failing integration test assertions
JavaFileObject file0 = JavaFileObjects.forSourceLines("test.Foo", "package test", "class Foo {"; ... );
JavaFileObject file1 = JavaFileObjects.forResource("expected/SomeFile.java");
assert_().about(JavaSourcesSubject.javaSources()) .that(ImmutableSet.of(file0, file1)) .processedWith(new BlahProcessor()) .failsToCompile() .withErrorContaining"Invalid use of Annotation @Blah") .in(javaFileObject).onLine(7);
JavaOne 2014
Testing Processors - Succeeding integration tests
● For successful compilation, two approaches:○ Use compile-testing to compare against a golden file, or○ Functionally test that the generated code behaves as it should
● Golden file tests:○ Brittle - small changes can require a lot of change in test files/code○ Useful to demonstrate expected output
● Functional tests:○ Exercise more code paths and use-cases with less verbosity
JavaOne 2014
Testing Processors - Overall strategy
● Use Functional tests to cover the bulk of use-cases
● Use CompilationRule to unit-test the internals of your Processor
● Use compile-testing assertions to test expected errors
● Use compile-testing assertions to test a small number of golden expectation files, for illustration of processor output
JavaOne 2014
Summary
JavaOne 2014
Summary
● Annotation processors are a powerful way to plug in to javac● An ecosystem is evolving to ease writing custom processors
○ JavaWriter, auto-common, compile-testing, @AutoService● Several useful annotation processors exist presently
○ AutoValue, Dagger● Lots of advantages to annotation processors
○ Early error-checking○ Performance improvements○ Viewable generated source code
JavaOne 2014
Annotation Processor Resources
● Check out existing annotation processors:○ Google Auto: https://github.com/google/auto○ Dagger 1: https://github.com/square/dagger○ Dagger 2: https://github.com/google/dagger
● Projects that may help:○ JavaWriter: https://github.com/square/javawriter○ CompileTesting: https://github.com/google/compile-testing○ Google Auto common utilities: https://github.
com/google/auto/tree/master/common
JavaOne 2014
Q&A
JavaOne 2014
Manual IDE configuration
● Make a jar file with your processor and all of its dependencies● Ensure it has META-INF/services/javax.annotation.processing.
Processor○ @AutoService(Processor.class) is the easiest way
● NetBeans:○ Project Properties > Libraries > Processor > Add JAR/Folder
● Eclipse:○ Properties > Java Compiler > Annotation Processing
■ Enable Annotation Processing + Factory Path