A new approach to meta programming
Author: Andreas Rumpf
A new approach tometaprogramming
Copyright © 2013 Andreas Rumpf
Talk structure
1.What is Nimrod?
2.Implementation aspects
3."Hello World"
4.Meta programming features
5.Optimizing "Hello World" via term rewriting macros
6.Hoisting via term rewriting macros
7.Summary of Nimrod's meta programming features
Talk structure
Copyright © 2013 Andreas Rumpf
What is Nimrod?
What is Nimrod?
Copyright © 2013 Andreas Rumpf
What is Nimrod?
• A statically typed
var input: TaintedString
What is Nimrod?
Copyright © 2013 Andreas Rumpf
What is Nimrod?
• A statically typed
var input: TaintedString
• systems programming language
var a = cast[int](gch.stackBottom)
What is Nimrod?
Copyright © 2013 Andreas Rumpf
What is Nimrod?
• A statically typed
var input: TaintedString
• systems programming language
var a = cast[int](gch.stackBottom)
• with a clean syntax
iterator from1to2(): int = yield 1; yield 2
What is Nimrod?
Copyright © 2013 Andreas Rumpf
What is Nimrod?
• A statically typed
var input: TaintedString
• systems programming language
var a = cast[int](gch.stackBottom)
• with a clean syntax
iterator from1to2(): int = yield 1; yield 2
• and strong meta programming capabilities.
template `!=`(x, y: expr): expr = not (x == y)
What is Nimrod?
Copyright © 2013 Andreas Rumpf
Implementation aspects
• Nimrod compiles to C; C++ and Objective-C are also supported
• it compiles to JavaScript
• it provides a realtime GC which supports max pause times of 1-2miliseconds which means it is perfectly useful even for demandinggames
• the Nimrod compiler and all of the standard library (including theGC) are written in Nimrod
• whole program dead code elimination: stdlib carefully crafted tomake use of it; for instance parsers do not use (runtime) regularexpression -> re engine not part of the executable
• the GC is also optimized away if you do not use it
Implementation aspects
Copyright © 2013 Andreas Rumpf
Implementation aspects
• our infrastructure (IDE, build farm, build tools, package manager) isalso completely written in Nimrod
Implementation aspects
Copyright © 2013 Andreas Rumpf
Hello world
echo "hello ", "world", 99
Hello world
Copyright © 2013 Andreas Rumpf
Hello world
echo "hello ", "world", 99
is rewritten to:
echo([$"hello ", $"world", $99])
•echo is declared as: proc echo(a: varargs[string, `$`]); $(Nimrod's toString operator) is applied to every argument
• local type converter: only in this context everything is converted to astring via $
• in contrast to C#/Java's solution this requires no dynamic binding
Hello world
Copyright © 2013 Andreas Rumpf
Hello world
echo "hello ", "world", 99
is rewritten to:
echo([$"hello ", $"world", $99])
•echo is declared as: proc echo(a: varargs[string, `$`]); $(Nimrod's toString operator) is applied to every argument
• local type converter: only in this context everything is converted to astring via $
• in contrast to C#/Java's solution this requires no dynamic binding
• it is extensible:
proc `$`(x: MyObject): string = x.svar obj = MyObject(s: "xyz")echo obj # works
Hello world
Copyright © 2013 Andreas Rumpf
Meta programming features
Nimrod's focus is meta programming; macros are used
1.to avoid code duplication / boilerplate:
01 template htmlTag(tag: expr) {.immediate.} =02 proc tag(): string = "<" & astToStr(tag) & ">"0304 htmlTag(br)05 htmlTag(html)0607 echo br()
Produces:
<br>
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
2.for control flow abstraction:
01 template once(body: stmt) =02 var x {.global.} = false03 if not x:04 x = true05 body0607 proc p() =08 once:09 echo "first call of p"10 echo "some call of p"1112 p()13 once:14 echo "new instantiation"15 p()
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
2.for control flow abstraction:
01 template once(body: stmt) =02 var x {.global.} = false03 if not x:04 x = true05 body0607 proc p() =08 once:09 echo "first call of p"10 echo "some call of p"1112 p()13 once:14 echo "new instantiation"15 p()
Produces:
first call of psome call of pnew instantiationsome call of p
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
3.for lazy evaluation:
01 template log(msg: string) =02 if debug:03 echo msg0405 log("x: " & $x & ", y: " & $y)
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
4.to implement DSLs:
01 html mainPage:02 head:03 title "now look at this"04 body:05 ul:06 li "Nimrod is quite capable"0708 echo mainPage()
Produces:
<html> <head><title>now look at this</title></head> <body> <ul> <li>Nimrod is quite capable</li> </ul> </body></html>
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
Implementation:
01 template html(name: expr, matter: stmt) {.immediate.} =02 proc name(): string =03 result = "<html>"04 matter05 result.add("</html>")0607 template nestedTag(tag: expr) {.immediate.} =08 template tag(matter: stmt) {.immediate.} =09 result.add("<" & astToStr(tag) & ">")10 matter11 result.add("</" & astToStr(tag) & ">")1213 template simpleTag(tag: expr) {.immediate.} =14 template tag(matter: expr) {.immediate.} =15 result.add("<$1>$2</$1>" % [astToStr(tag), matter])1617 nestedTag body18 nestedTag head19 nestedTag ul20 simpleTag title21 simpleTag li
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
After macro expansion:
template html(name: expr, matter: stmt) {.immediate.} = proc name(): string = result = "<html>" matter result.add("</html>")
template head(matter: stmt) {.immediate.} = result.add("<" & astToStr(head) & ">") matter result.add("</" & astToStr(head) & ">")
...
template title(matter: expr) {.immediate.} = result.add("<$1>$2</$1>" % [astToStr(title), matter])
template li(matter: expr) {.immediate.} = result.add("<$1>$2</$1>" % [astToStr(li), matter])
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
01 html mainPage:02 head:03 title "now look at this"04 body:05 ul:06 li "Nimrod is quite capable"0708 echo mainPage()
Is translated into:
proc mainPage(): string = result = "<html>" result.add("<head>") result.add("<$1>$2</$1>" % ["title", "now look at this"]) result.add("</head>") result.add("<body>") result.add("<ul>") result.add("<$1>$2</$1>" % ["li", "Nimrod is quite capable"]) result.add("</ul>") result.add("</body>") result.add("</html>")
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
Compile time function evaluation optimizes 'mainPage()' into:
"<html><head><title>now look at this</title></head><body>..."
Meta programming features
Copyright © 2013 Andreas Rumpf
Meta programming features
5.to provide user defined optimizations (via term rewriting macros).
Meta programming features
Copyright © 2013 Andreas Rumpf
Hello world 2.0 (1)
01 type02 MyObject = object03 a, b: int04 s: string05 let obj = MyObject(a: 3, b: 4, s: "abc")06 echo obj # note: calls $ implicitly
Produces (roughly):
( a: 3 b: 4 s: "abc")
How does it work?
•$ for object is in Nimrod's system module (like Haskell's prelude)
Hello world 2.0 (1)
Copyright © 2013 Andreas Rumpf
How $ for object works
01 # in system module (simplified):02 proc `$` [T: object](x: T): string =03 result = "("04 for name, value in fieldPairs(x):05 result.add("$1: $2\n" % [name, $value])06 result.add(")")
Notes:
• [T: object] -- generic type + constraint
• relies on builtin fieldPairs iterator
How $ for object works
Copyright © 2013 Andreas Rumpf
How $ for object works
• Builtin fieldPairs iterator transforms the loop into the following:
result = "("result.add("$1: $2\n" % ["a", $obj.a])result.add("$1: $2\n" % ["b", $obj.b])result.add("$1: $2\n" % ["s", $obj.s])result.add(")")
How $ for object works
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (1)
• Builtin fieldPairs iterator transforms the loop into the following:
result = "("result.add("$1: $2\n" % ["a", $obj.a])result.add("$1: $2\n" % ["b", $obj.b])result.add("$1: $2\n" % ["s", $obj.s])result.add(")")
Desired result:
result = `&`("(a: ", $obj.a, "\nb: ", $obj.b, "\ns: ", $obj.s, "\n)")
Optimizing Hello World (1)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (2)
We need partial evaluation for '%'.
%'s implementation (simplified):
01 proc `%`(f: string, a: openArray[string]): string =02 result = ""03 var i = 004 while i < f.len:05 if f[i] == '$':06 case f[i+1]07 of '1'..'9':08 var j = 009 i += 110 while f[i] in {'0'..'9'}:11 j = j * 10 + ord(f[i]) - ord('0'); i += 112 result.add(a[j-1])13 else:14 invalidFormatString()15 else:16 result.add(f[i]); i += 1
Optimizing Hello World (2)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (3)
01 macro optFormat{`%`(f, a)}(f: string{lit}, a: openArray[string]): expr =02 result = newCall("&")03 var i = 004 while i < f.len:05 if f[i] == '$':06 case f[i+1]07 of '1'..'9':08 var j = 009 i += 110 while f[i] in {'0'..'9'}:11 j = j * 10 + ord(f[i]) - ord('0'); i += 112 result.add(a[j-1])13 else:14 invalidFormatString()15 else:16 result.add(newLit(f[i])); i += 1
Implements this optimization:
"$1: $2\n" % ["s", $obj.s] --> `&`("s", ':', ' ', $obj.s, '\n')
Optimizing Hello World (3)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "("result.add("$1: $2\n" % ["a", $obj.a])result.add("$1: $2\n" % ["b", $obj.b])result.add("$1: $2\n" % ["s", $obj.s])result.add(")")
After partial evaluation:
result = "("result.add(`&`("a", ':', ' ', $obj.a, '\n'))result.add(`&`("b", ':', ' ', $obj.b, '\n'))result.add(`&`("s", ':', ' ', $obj.s, '\n'))result.add(")")
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "("result.add(`&`("a", ':', ' ', $obj.a, '\n'))result.add(`&`("b", ':', ' ', $obj.b, '\n'))result.add(`&`("s", ':', ' ', $obj.s, '\n'))result.add(")")
After constant folding:
result = "("result.add(`&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
After constant folding:
result = "("result.add(`&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
Further optimization via term rewriting templates:
template optAdd1{x = y; x.add(z)}(x, y, z: string) = x = y & z
template optAdd2{x.add(y); x.add(z)}(x, y, z: string) = x.add(y & z)
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "("result.add(`&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')result.add(`&`("s: ", $obj.s, '\n'))result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n') & `&`("s: ",$obj.s, '\n')result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n') & `&`("s: ",$obj.s, '\n')result.add(")")
template optAdd1{x = y; x.add(z)}(x, y, z: string) =
x = y & z
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
Optimizing Hello World (4)
After applying these rules the code is:
result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')) & `&`("s: ", $obj.s, '\n')) & ")"
After constant folding (that the compiler performs for us) it becomes:
result = `&`("(a: ", $obj.a, "\nb: ", $obj.b, "\ns: ", $obj.s, "\n)")
Optimizing Hello World (4)
Copyright © 2013 Andreas Rumpf
DRY means templates
• Implementation of % and optFormat look suspiciously alike.
• Can we avoid the code duplication?
DRY means templates
Copyright © 2013 Andreas Rumpf
DRY means templates
01 template formatImpl(handleChar: expr) =02 var i = 003 while i < f.len:04 if f[i] == '$':05 case f[i+1]06 of '1'..'9':07 var j = 008 i += 109 while f[i] in {'0'..'9'}:10 j = j * 10 + ord(f[i]) - ord('0'); i += 111 result.add(a[j-1])12 else:13 invalidFormatString()14 else:15 result.add(handleChar(f[i])); i += 11617 proc `%`(f: string, a: openArray[string]): string =18 template identity(x: expr): expr = x19 result = ""; formatImpl(identity)2021 macro optFormat{`%`(f, a)}(f: string{lit}, a: openArray[string]): expr =22 result = newCall("&"); formatImpl(newLit)
DRY means templates
Copyright © 2013 Andreas Rumpf
Hoisting
01 proc re(x: string): Regex =02 # wrapper around PCRE for instance0304 template optRe{re(x)}(x: string{lit}): Regex =05 var g {.global.} = re(x)06 g0708 template `=~`(s: string, pattern: Regex): bool =09 when not definedInScope(matches):10 var matches {.inject.}: array[maxSubPatterns, string]11 match(s, pattern, matches)1213 for line in lines("input.txt"):14 if line =~ re"(\w+)=(\w+)":15 echo "key-value pair; key: ", matches[0], " value: ", matches[1]
• Perl performs same optimization; however regexes are built into Perl,but not into Nimrod
• 'inject' breaks hygiene
Hoisting
Copyright © 2013 Andreas Rumpf
Summary of meta programming features
You name it, Nimrod got it (except fexprs ;-):
• compile time function evaluation; including staticRead andstaticExec
• declarative (template) and imperative (macro) AST based macros:both hygienic and dirty
• term rewriting macros; side-effect and alias analysis constraints
• source code filters
• programmable annotation system ("pragmas")
Summary of meta programming features
Copyright © 2013 Andreas Rumpf
Summary of meta programming features
• quasi quoting:
01 macro check(ex: expr): stmt =02 var info = ex.lineInfo03 var expString = ex.toStrLit04 result = quote do:05 if not `ex`:06 echo `info`, ": Check failed: ", `expString`0708 check 1 < 2
Summary of meta programming features
Copyright © 2013 Andreas Rumpf
Thank you
Thank you for listening. We are always looking for contributors:
• http://nimrod-code.org
• http://forum.nimrod-code.org
• https://github.com/Araq/Nimrod
• irc.freenode.net/nimrod
Thank you
Copyright © 2013 Andreas Rumpf