Writing DSL with Applicative Functors

David Galichet Freelance functional programmer

!twitter: @dgalichet

Content normalization

• We want to parse heterogeneous data formats (CSV, XML …) and transform them to a pivot format (Scala object in our case)

• The transformation should be described as a DSL

• This DSL must be simple to use, and enable any kinds of data transformations or verifications

Expected DSL format

val reader = ( ! Pick(0).as[String].map(_.capitalize) and ! Pick(1).as[Date].check(_.after(now())) !).reduce(FutureEvent)!!reader("PSUG; 21/08/2014") // returns Success(FutureEvent("PSUG", date))

Inspired by Play2 Json API

Conventions & material

• Code is available on Github : https://github.com/dgalichet/PsugDSLWritingWithApplicative

• Code revision is define on the top of any slides including source code (just checkout the specified tag)

tag: step1

Reading single entry• We have a CSV line and we want to read one column

• We will introduce several abstractions:

• Picker: fetch a data from CSV or XML

• Result: either a Success or a Failure

• Converter: convert value from Picker to Reader

• Reader: container with methods to process its content

tag: step1

Introducing Pickertag: step1

case class Picker(p: String => Result[String]) {! def as[T](implicit c: Converter[T]): Reader[T] = c.convert(p)!}!!object CsvPicker {! def apply[T](i: Int)(implicit separator: Char): Picker = Picker { s: String =>! val elems = s.trim.split(separator)! if (i > 0 && elems.size > i) Success(elems(i).trim)! else Failure(s"No column ${i} for ${s}")! }!}

Introducing Pickertag: step1

case class Picker(p: String => Result[String]) {! def as[T](implicit c: Converter[T]): Reader[T] = c.convert(p)!}!!object CsvPicker {! def apply[T](i: Int)(implicit separator: Char): Picker = Picker { s: String =>! val elems = s.trim.split(separator)! if (i > 0 && elems.size > i) Success(elems(i).trim)! else Failure(s"No column ${i} for ${s}")! }!}

Picker wraps a function from String to Result

The Resulttag: step1

sealed trait Result[+T] case class Success[T](t: T) extends Result[T] case class Failure(error: String) extends Result[Nothing]!

The Convertertag: step1

trait Converter[T] { def convert(p: String => Result[String]): Reader[T] }!!object Converter {! implicit val string2StringConverter = new Converter[String] {! override def convert(p: String => Result[String]) = Reader[String](p)! // See code on Github for more converters!}

The Convertertag: step1

trait Converter[T] { def convert(p: String => Result[String]): Reader[T] }!!!object Converter {! implicit val string2StringConverter = new Converter[String] {! override def convert(p: String => Result[String]) = Reader[String](p)! // See code on Github for more converters!}

Convert the content of the Picker to a Reader

The Readertag: step1

case class Reader[O](p: String => Result[O]) { def apply(s: String): Result[O] = p(s) }

A Reader doesn’t contain a value but a process to transform original data (CSV line or XML) to a Result

Usage sampletag: step1

import Converter._ // import implicit converters!implicit val separator = ‘;’!!CsvPicker(1).as[String].apply("foo;bar") === "bar"

Enhancing the Reader

• The first defines a very simple Reader. We must add a method to combine two instances of Reader

• We will also enhance Failure to store multiple error messages

tag: step2

Enhancing the Readertag: step2

case class Reader[O](p: String => Result[O]) {! def apply(s: String): Result[O] = p(s)!! def and[O2](r2: Reader[O2]): Reader[(O, O2)] = Reader { s: String =>! (p(s), r2.p(s)) match {! case (Success(s1), Success(s2)) => Success((s1, s2))! case (Success(_), Failure(f)) => Failure(f)! case (Failure(f), Success(_)) => Failure(f)! case (Failure(f1), Failure(f2)) => Failure(f1 ++ f2)! }! }! def map[T](f: O => T): Reader[T] = Reader { s: String =>! p(s) match {! case Success(o) => Success(f(o))! case f: Failure => f! }! }! def reduce[T] = map[T] _ // alias for map!}

Enhancing Result typetag: step2

sealed trait Result[+T]!case class Success[T](t: T) extends Result[T]!case class Failure(error: NonEmptyList[String]) extends Result[Nothing]!!object Failure {! def apply(s: String): Failure = Failure(NEL(s))!}!!case class NonEmptyList[T](head: T, tail: List[T]) { def toList = head::tail def ++(l2: NonEmptyList[T]): NonEmptyList[T] = NonEmptyList(head, tail ++ l2.toList) } object NEL { def apply[T](h: T, t: T*) = NonEmptyList(h, t.toList) }

Usage sampletag: step2

implicit val separator = ';' implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy") import Converter.string2StringConverter import Converter.string2DateConverter!!val reader = ( CsvPicker(1).as[String] and CsvPicker(2).as[Date] ).reduce { case (n, d) => FutureEvent(n, d) }!reader("foo;bar;12/10/2014") === Success(FutureEvent("bar", dtFormatter.parse("12/10/2014")))!!case class FutureEvent(name: String, dt: Date)

Usability problemtag: step2

• The use of reduce (or map) method to transform a Reader[(0, 02)] into an instance of Reader[FutureEvent] for example is quite verbose

• This will be even more verbose for instances of Reader[(0, (02, 03))]

• We want the API to automatically bind tuple elements to a constructor as we can encounter in Play2 Json API

Applicative functorstag: step3

• To tackle our problem, we will use Applicative Functors and play2 functional library (and especially FunctionalBuilder)

• This approach is inspired by @sadache (Sadek Drobi) article https://gist.github.com/sadache/3646092

• An Applicative Functor is a Type Class relying on ad-hoc polymorphism to extends a Class with some properties

• Play2 functional library (or Scalaz) provides mechanism to compose Applicatives in a smart way

Applicative functorstag: step3

M is an Applicative Functor if there exists the following methods :

def pure[A](a: A): M[A] def map[A, B](m: M[A], f: A => B): M[B] def apply[A, B](mf: M[A => B], ma: M[A]): M[B]!

with the following Laws : • Identity: apply(pure(identity), ma) === ma where ma is an Applicative M[A] • Homomorphism: apply(pure(f), pure(a)) === pure(f(a)) where f: A =>

B and a an instance of A • Interchange: mf if an instance of M[A => B]

apply(mf, pure(a)) === apply(pure {(g: A => B) => g(a)}, mf)!• Composition: map(ma, f) === apply(pure(f), ma)

Applicative functorstag: step3

trait Applicative[M[_]] { def pure[A](a: A): M[A] def map[A, B](m: M[A], f: A => B): M[B] def apply[A, B](mf: M[A => B], ma: M[A]): M[B] } Applicative is an Higher Kinded type

(parameterized with M that take a single type parameter)

Reader is an Applicativetag: step3

case class Reader[O](p: String => Result[O]) { def apply(s: String): Result[O] = p(s)! def map[T](f: O => T): Reader[T] = Reader { s: String => p(s) match { case Success(o) => Success(f(o)) case f: Failure => f } }!}!object Reader { def map2[O, O1, O2](r1: Reader[O1], r2: Reader[O2])(f: (O1, O2) => O): Reader[O] = Reader { s: String => (r1.p(s), r2.p(s)) match { case (Success(s1), Success(s2)) => Success(f(s1, s2)) case (Success(_), Failure(e)) => Failure(e) case (Failure(e), Success(_)) => Failure(e) case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2) } } …!}

Reader is an Applicativetag: step3

object Reader { … // map2 implicit val readerIsAFunctor: Functor[Reader] = new Functor[Reader] { override def fmap[A, B](m: Reader[A], f: (A) => B) = m.map(f) } implicit val readerIsAnApplicative: Applicative[Reader] = new Applicative[Reader] { override def pure[A](a: A) = Reader { _ => Success(a) } override def apply[A, B](mf: Reader[A => B], ma: Reader[A]) = map2(mf, ma)((f, a) => f(a)) override def map[A, B](m: Reader[A], f: A => B) = m.map(f) } }

Usage sampletag: step3

import Converter.string2StringConverter import Converter.string2DateConverter!import play.api.libs.functional.syntax._ import Reader.readerIsAnApplicative!!implicit val separator = ';' implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!!val reader = ( CsvPicker(1).as[String] and CsvPicker(2).as[Date] )(FutureEvent) // here we use CanBuild2.apply reader("foo;bar;12/10/2014") === Success(FutureEvent("bar", dtFormatter.parse("12/10/2014")))!

Usage sample (errors accumulation)

tag: step3

import Converter.string2StringConverter import Converter.string2DateConverter!import play.api.libs.functional.syntax._ import Reader.readerIsAnApplicative!!implicit val separator = ';' implicit val dtFormatter = new SimpleDateFormat("dd/MM/yyyy")!!val reader = ( CsvPicker(1).as[Int] and CsvPicker(2).as[Date] )((_, _)) reader(List("foo", "not a number", "not a date")) === Failure(NEL(!"Unable to format 'not a number' as Int", !"Unable to format 'not a date' as Date"))!

Benefitstag: step3

• Making Reader an Applicative Functor give ability to combine efficiently instances of Reader

• Due to Applicative properties, we still accumulate errors

• Play2 functional builder give us a clean syntax to define our DSL

Introducing XML Pickertag: step4

case class Picker(p: String => Result[String]) { def as[T](implicit c: Converter[T]): Reader[T] = c.convert(p) }!!object XmlPicker { def apply[T](query: Elem => NodeSeq): Picker = Picker { s: String => try { val xml = XML.loadString(s) Success(query(xml).text) } catch { case e: Exception => Failure(e.getMessage) } } }!

Usage sampletag: step4

import play.api.libs.functional.syntax._ import Reader.readerIsAnApplicative!import Converter._ implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!val xml = """ <company name="Dupont and Co"> <owner> <person firstname="jean" lastname="dupont" birthdate="11/03/1987"/> </owner> </company>""" val r = ( XmlPicker(_ \\ "person" \ "@firstname").as[String] and XmlPicker(_ \\ "person" \ "@lastname").as[String] and XmlPicker(_ \\ "person" \ "@birthdate").as[Date] )(Person)!r(xml) === Success(Person("jean","dupont",dF.parse("11/03/1987"))) case class Person(firstname: String, lastname: String, birthDt: Date)

Implementation problemtag: step4

• The Reader[O] takes a type argument for the output. The input is always a String

• With this implementation, an XML content will be parsed (with XML.load) as many times as we use XmlPicker. This will cause unnecessary overhead

• We will have the same issue (with lower overhead) with our CsvPicker

Introducing Reader[I, 0]tag: step5

To resolve this problem, we will modify Reader to take a type parameter for the input

Introducing Reader[I, 0]tag: step5

case class Reader[I, O](p: I => Result[O]) { def apply(s: I): Result[O] = p(s) def map[T](f: O => T): Reader[I, T] = Reader { s: I => p(s) match { case Success(o) => Success(f(o)) case f: Failure => f } }!}!object Reader { def map2[I, O, O1, O2](r1: Reader[I, O1], r2: Reader[I, O2])(f: (O1, O2) => O): Reader[I, O] = Reader { s: I => (r1.p(s), r2.p(s)) match { case (Success(s1), Success(s2)) => Success(f(s1, s2)) case (Success(_), Failure(e)) => Failure(e) case (Failure(e), Success(_)) => Failure(e) case (Failure(e1), Failure(e2)) => Failure(e1 ++ e2) } }

Introducing Reader[I, 0]tag: step5

object Reader {!implicit def readerIsAFunctor[I] = new Functor[({type λ[A] = Reader[I, A]})#λ] { override def fmap[A, B](m: Reader[I, A], f: (A) => B) = m.map(f) } implicit def readerIsAnApplicative[I] = new Applicative[({type λ[A] = Reader[I, A]})#λ] { override def pure[A](a: A) = Reader { _ => Success(a) } override def apply[A, B](mf: Reader[I, A => B], ma: Reader[I, A]) = map2(mf, ma)((f, a) => f(a)) override def map[A, B](m: Reader[I, A], f: (A) => B) = m.map(f) }

What are Type lambdas ?tag: step5

If we go back to Applicative definition, we can see that it’s an Higher Kinded type (same with Functor) : !trait Applicative[M[_]] { … } // Applicative accept parameter M that take itself any type as parameter!!Our problem is that Reader[I, 0] takes two parameters but Applicative[M[_]] accept types M with only one parameter. We use Type Lambdas to resolve this issue: !new Applicative[({type λ[A] = Reader[I, A]})#λ]!

Go back to Reader[I, 0]tag: step5

object Reader {!implicit def readerIsAFunctor[I] = new Functor[({type λ[A] = Reader[I, A]})#λ] { override def fmap[A, B](m: Reader[I, A], f: A => B) = m.map(f) } implicit def readerIsAnApplicative[I] = new Applicative[({type λ[A] = Reader[I, A]})#λ] { override def pure[A](a: A) = Reader { _ => Success(a) } override def apply[A, B](mf: Reader[I, A => B], ma: Reader[I, A]) = map2(mf, ma)((f, a) => f(a)) override def map[A, B](m: Reader[I, A], f: A => B) = m.map(f) }

Go back to Reader[I, 0]tag: step5

object Reader {!import scala.language.implicitConversions // Here we help the compiler a bit. Thanks @skaalf (Julien Tournay) !// and https://github.com/jto/validation implicit def fcbReads[I] = functionalCanBuildApplicative[({type λ[A] = Reader[I, A]})#λ] implicit def fboReads[I, A](a: Reader[I, A])(implicit fcb: FunctionalCanBuild[({type λ[x] = Reader[I, x]})#λ]) = new FunctionalBuilderOps[({type λ[x] = Reader[I, x]})#λ, A](a)(fcb)

Converter[I, T]tag: step5

trait Converter[I, T] { def convert(p: I => Result[String]): Reader[I, T] }!object Converter { implicit def stringConverter[I] = new Converter[I, String] { override def convert(p: I => Result[String]) = Reader[I, String](p) }!! implicit def dateConverter[I](implicit dtFormat: DateFormat) = new Converter[I, Date] { override def convert(p: I => Result[String]) = Reader[I, Date] { s: I => p(s) match { case Success(dt) => try { ! Success(dtFormat.parse(dt)) } catch { case e: ParseException => Failure(s"...") } case f: Failure => f }}}

Picker[I]tag: step5

case class Picker[I](p: I => Result[String]) { def as[T](implicit c: Converter[I, T]): Reader[I, T] = c.convert(p) } object CsvPicker { def apply[T](i: Int): Picker[List[String]] = Picker { elems: List[String] => if (i > 0 && elems.size > i) Success(elems(i).trim) else Failure(s"No column ${i} found in ${elems.mkString(";")}") }} object XmlPicker { def apply[T](query: Elem => NodeSeq): Picker[Elem] = Picker { elem: Elem => try { Success(query(elem).text) } catch { case e: Exception => Failure(e.getMessage) }}}!

Usage sampletag: step5

import play.api.libs.functional.syntax._ import Reader._!import Converter._!implicit val dF = new SimpleDateFormat("dd/MM/yyyy")!val xml = XML.loadString(""" <company name="Dupont and Co"> <owner> <person firstname="jean" lastname="dupont" birthdate="11/03/1987"/> </owner> </company>""")!!val r = ( XmlPicker(_ \\ "person" \ "@firstname").as[String] and XmlPicker(_ \\ "person" \ "@lastname").as[String] and XmlPicker(_ \\ "person" \ "@birthdate").as[Date] )(Person)!r(xml) === Success(Person("jean", "dupont", dF.parse("11/03/1987")))

Adding combinatorstag: step6

• We now add new abilities to Reader

• We especially want a method to validate content

Adding combinatorstag: step6

case class Reader[I, O](p: I => Result[O]) {!! def flatMap[T](f: O => Reader[I, T]): Reader[I, T] = Reader { s: I => p(s) match { case Success(o) => f(o)(s) case f: Failure => f } } def verify(f: O => Result[O]): Reader[I, O] = flatMap { o: O => Reader( _ => f(o)) }!}

Usage sampletag: step6

val r: Reader[String, String] = Reader { Success(_) } r.verify { x => if (x == "OK") Success(x) else Failure("KO") }("OK") === Success("OK")

Conclusion• We have created a simple and powerful DSL for

processing CSV and XML content

• This DSL give us ability to Pick data, transform and verify it and also accumulate encountered errors

• We have seen that making Reader an instance of the Applicative Functor Type Class add it new capabilities

• Using ad-hoc polymorphism using Type Classes gives us ability to extends Reader without altering it

Follow-up• Type Classes are defined by functions that must be

implemented with regards to their laws (left/right identity …)

• Proving the correctness of Type Class Laws can be a bit tricky we usual approach

• I will introduce the framework ScalaCheck at scala.io 2014, and show how to test them

Follow-up

• In the roadmap that has been announced by @typesafe (http://scala-lang.org/news/roadmap-next), it seems that Scala 2.14 (aka « Don Giovanni ») will clean up lambda types syntax