A new approach to meta programming - Nim · •whole program dead code elimination: stdlib...

transcript

A new approach to meta programming

Author: Andreas Rumpf

A new approach tometaprogramming

Talk structure

1.What is Nimrod?

2.Implementation aspects

3."Hello World"

4.Meta programming features

5.Optimizing "Hello World" via term rewriting macros

6.Hoisting via term rewriting macros

7.Summary of Nimrod's meta programming features

Talk structure

What is Nimrod?

• A statically typed

var input: TaintedString

What is Nimrod?

• systems programming language

var a = cast[int](gch.stackBottom)

What is Nimrod?

• with a clean syntax

iterator from1to2(): int = yield 1; yield 2

What is Nimrod?

• with a clean syntax

iterator from1to2(): int = yield 1; yield 2

• and strong meta programming capabilities.

template `!=`(x, y: expr): expr = not (x == y)

What is Nimrod?

Implementation aspects

• Nimrod compiles to C; C++ and Objective-C are also supported

• it compiles to JavaScript

• it provides a realtime GC which supports max pause times of 1-2miliseconds which means it is perfectly useful even for demandinggames

• the Nimrod compiler and all of the standard library (including theGC) are written in Nimrod

• whole program dead code elimination: stdlib carefully crafted tomake use of it; for instance parsers do not use (runtime) regularexpression -> re engine not part of the executable

• the GC is also optimized away if you do not use it

• our infrastructure (IDE, build farm, build tools, package manager) isalso completely written in Nimrod

Hello world

echo "hello ", "world", 99

Hello world

is rewritten to:

echo([$"hello ", $"world", $99])

•echo is declared as: proc echo(a: varargs[string, `$`]); $(Nimrod's toString operator) is applied to every argument

• local type converter: only in this context everything is converted to astring via $

• in contrast to C#/Java's solution this requires no dynamic binding

Hello world

is rewritten to:

echo([$"hello ", $"world", $99])

•echo is declared as: proc echo(a: varargs[string, `$`]); $(Nimrod's toString operator) is applied to every argument

• local type converter: only in this context everything is converted to astring via $

• in contrast to C#/Java's solution this requires no dynamic binding

• it is extensible:

proc `$`(x: MyObject): string = x.svar obj = MyObject(s: "xyz")echo obj # works

Hello world

Meta programming features

Nimrod's focus is meta programming; macros are used

1.to avoid code duplication / boilerplate:

01 template htmlTag(tag: expr) {.immediate.} =02 proc tag(): string = "<" & astToStr(tag) & ">"0304 htmlTag(br)05 htmlTag(html)0607 echo br()

Produces:

2.for control flow abstraction:

01 template once(body: stmt) =02 var x {.global.} = false03 if not x:04 x = true05 body0607 proc p() =08 once:09 echo "first call of p"10 echo "some call of p"1112 p()13 once:14 echo "new instantiation"15 p()

2.for control flow abstraction:

01 template once(body: stmt) =02 var x {.global.} = false03 if not x:04 x = true05 body0607 proc p() =08 once:09 echo "first call of p"10 echo "some call of p"1112 p()13 once:14 echo "new instantiation"15 p()

Produces:

first call of psome call of pnew instantiationsome call of p

3.for lazy evaluation:

01 template log(msg: string) =02 if debug:03 echo msg0405 log("x: " & $x & ", y: " & $y)

4.to implement DSLs:

01 html mainPage:02 head:03 title "now look at this"04 body:05 ul:06 li "Nimrod is quite capable"0708 echo mainPage()

Produces:

<html> <head><title>now look at this</title></head> <body> <ul> <li>Nimrod is quite capable</li> </ul> </body></html>

Implementation:

01 template html(name: expr, matter: stmt) {.immediate.} =02 proc name(): string =03 result = "<html>"04 matter05 result.add("</html>")0607 template nestedTag(tag: expr) {.immediate.} =08 template tag(matter: stmt) {.immediate.} =09 result.add("<" & astToStr(tag) & ">")10 matter11 result.add("</" & astToStr(tag) & ">")1213 template simpleTag(tag: expr) {.immediate.} =14 template tag(matter: expr) {.immediate.} =15 result.add("<$1>$2</$1>" % [astToStr(tag), matter])1617 nestedTag body18 nestedTag head19 nestedTag ul20 simpleTag title21 simpleTag li

After macro expansion:

template html(name: expr, matter: stmt) {.immediate.} = proc name(): string = result = "<html>" matter result.add("</html>")

template head(matter: stmt) {.immediate.} = result.add("<" & astToStr(head) & ">") matter result.add("</" & astToStr(head) & ">")

template title(matter: expr) {.immediate.} = result.add("<$1>$2</$1>" % [astToStr(title), matter])

template li(matter: expr) {.immediate.} = result.add("<$1>$2</$1>" % [astToStr(li), matter])

01 html mainPage:02 head:03 title "now look at this"04 body:05 ul:06 li "Nimrod is quite capable"0708 echo mainPage()

Is translated into:

proc mainPage(): string = result = "<html>" result.add("<head>") result.add("<$1>$2</$1>" % ["title", "now look at this"]) result.add("</head>") result.add("<body>") result.add("<ul>") result.add("<$1>$2</$1>" % ["li", "Nimrod is quite capable"]) result.add("</ul>") result.add("</body>") result.add("</html>")

Compile time function evaluation optimizes 'mainPage()' into:

"<html><head><title>now look at this</title></head><body>..."

5.to provide user defined optimizations (via term rewriting macros).

Hello world 2.0 (1)

01 type02 MyObject = object03 a, b: int04 s: string05 let obj = MyObject(a: 3, b: 4, s: "abc")06 echo obj # note: calls $ implicitly

Produces (roughly):

( a: 3 b: 4 s: "abc")

How does it work?

•$ for object is in Nimrod's system module (like Haskell's prelude)

Hello world 2.0 (1)

How $ for object works

01 # in system module (simplified):02 proc `$` [T: object](x: T): string =03 result = "("04 for name, value in fieldPairs(x):05 result.add("$1: $2\n" % [name, $value])06 result.add(")")

Notes:

• [T: object] -- generic type + constraint

• relies on builtin fieldPairs iterator

• Builtin fieldPairs iterator transforms the loop into the following:

result = "("result.add("$1: $2\n" % ["a", $obj.a])result.add("$1: $2\n" % ["b", $obj.b])result.add("$1: $2\n" % ["s", $obj.s])result.add(")")

Optimizing Hello World (1)

• Builtin fieldPairs iterator transforms the loop into the following:

Desired result:

result = `&`("(a: ", $obj.a, "\nb: ", $obj.b, "\ns: ", $obj.s, "\n)")

We need partial evaluation for '%'.

%'s implementation (simplified):

01 proc `%`(f: string, a: openArray[string]): string =02 result = ""03 var i = 004 while i < f.len:05 if f[i] == '$':06 case f[i+1]07 of '1'..'9':08 var j = 009 i += 110 while f[i] in {'0'..'9'}:11 j = j * 10 + ord(f[i]) - ord('0'); i += 112 result.add(a[j-1])13 else:14 invalidFormatString()15 else:16 result.add(f[i]); i += 1

01 macro optFormat{`%`(f, a)}(f: string{lit}, a: openArray[string]): expr =02 result = newCall("&")03 var i = 004 while i < f.len:05 if f[i] == '$':06 case f[i+1]07 of '1'..'9':08 var j = 009 i += 110 while f[i] in {'0'..'9'}:11 j = j * 10 + ord(f[i]) - ord('0'); i += 112 result.add(a[j-1])13 else:14 invalidFormatString()15 else:16 result.add(newLit(f[i])); i += 1

Implements this optimization:

"$1: $2\n" % ["s", $obj.s] --> `&`("s", ':', ' ', $obj.s, '\n')

After partial evaluation:

result = "("result.add(`&`("a", ':', ' ', $obj.a, '\n'))result.add(`&`("b", ':', ' ', $obj.b, '\n'))result.add(`&`("s", ':', ' ', $obj.s, '\n'))result.add(")")

After constant folding:

result = "("result.add(`&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")

After constant folding:

Further optimization via term rewriting templates:

template optAdd1{x = y; x.add(z)}(x, y, z: string) = x = y & z

template optAdd2{x.add(y); x.add(z)}(x, y, z: string) = x.add(y & z)

template optAdd1{x = y; x.add(z)}(x, y, z: string) =

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n'))result.add(`&`("b: ", $obj.b, '\n'))result.add(`&`("s: ", $obj.s, '\n'))result.add(")")

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')result.add(`&`("s: ", $obj.s, '\n'))result.add(")")

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')result.add(`&`("s: ", $obj.s, '\n'))result.add(")")

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n') & `&`("s: ",$obj.s, '\n')result.add(")")

x = y & z

result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n') & `&`("s: ",$obj.s, '\n')result.add(")")

x = y & z

After applying these rules the code is:

result = "(" & `&`("a: ", $obj.a, '\n')) & `&`("b: ", $obj.b, '\n')) & `&`("s: ", $obj.s, '\n')) & ")"

After constant folding (that the compiler performs for us) it becomes:

result = `&`("(a: ", $obj.a, "\nb: ", $obj.b, "\ns: ", $obj.s, "\n)")

DRY means templates

• Implementation of % and optFormat look suspiciously alike.

• Can we avoid the code duplication?

DRY means templates

01 template formatImpl(handleChar: expr) =02 var i = 003 while i < f.len:04 if f[i] == '$':05 case f[i+1]06 of '1'..'9':07 var j = 008 i += 109 while f[i] in {'0'..'9'}:10 j = j * 10 + ord(f[i]) - ord('0'); i += 111 result.add(a[j-1])12 else:13 invalidFormatString()14 else:15 result.add(handleChar(f[i])); i += 11617 proc `%`(f: string, a: openArray[string]): string =18 template identity(x: expr): expr = x19 result = ""; formatImpl(identity)2021 macro optFormat{`%`(f, a)}(f: string{lit}, a: openArray[string]): expr =22 result = newCall("&"); formatImpl(newLit)

DRY means templates

Hoisting

01 proc re(x: string): Regex =02 # wrapper around PCRE for instance0304 template optRe{re(x)}(x: string{lit}): Regex =05 var g {.global.} = re(x)06 g0708 template `=~`(s: string, pattern: Regex): bool =09 when not definedInScope(matches):10 var matches {.inject.}: array[maxSubPatterns, string]11 match(s, pattern, matches)1213 for line in lines("input.txt"):14 if line =~ re"(\w+)=(\w+)":15 echo "key-value pair; key: ", matches[0], " value: ", matches[1]

• Perl performs same optimization; however regexes are built into Perl,but not into Nimrod

• 'inject' breaks hygiene

Hoisting

Summary of meta programming features

You name it, Nimrod got it (except fexprs ;-):

• compile time function evaluation; including staticRead andstaticExec

• declarative (template) and imperative (macro) AST based macros:both hygienic and dirty

• term rewriting macros; side-effect and alias analysis constraints

• source code filters

• programmable annotation system ("pragmas")

• quasi quoting:

01 macro check(ex: expr): stmt =02 var info = ex.lineInfo03 var expString = ex.toStrLit04 result = quote do:05 if not èx`:06 echo ìnfo`, ": Check failed: ", èxpString`0708 check 1 < 2

Thank you

Thank you for listening. We are always looking for contributors:

• http://nimrod-code.org

• http://forum.nimrod-code.org

• https://github.com/Araq/Nimrod

• irc.freenode.net/nimrod

Thank you

A new approach to meta programming - Nim · •whole program dead code elimination: stdlib...

Documents