Rust Closures as Input Parameters

Edit: Update (May 25)

I am learning Rust and, as a beginner, I have sometimes problems achieving some little tasks that would be so easy in other programming languages I know better.

But when I met some Rust developers and they ask me about my difficulties, I often forget about them.

I therefore decided to write about my difficulties in Rust to keep track of them.

So today about closures.

When reading a blog post introducing Rust for Node.js Developers I made the same Todo application.

At the end, the program contains such piece of code:

1
2
3
4
5
6
7
8
9
10
11
fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
  if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
      todo.deleted = true;
  }
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
  if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
      todo.completed = true;
  }
}

The first function marks a Todo as deleted if it can be found by its ID in the vector. The second function marks a Todo as completed, also if it can be found in the vector of Todos.

Some code is duplicated and I decided to refactor the common code in a third function, that would do something on a Todo if found in a vector.

This third function would take a closure as input parameter, like in pseudo-code:

1
2
3
4
5
fn with_todo_id(todos: &mut Vec<Todo>, todo_id: i16, f: <closure - do something on a Todo>) {
    if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
        f(todo);
    }
}

so that the 2 initial functions are simplified like that:

1
2
3
4
5
6
7
fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.deleted = true);
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.completed = true);
}

This closure is a side-effect on a Todo. It should accept a mutable Todo as parameter and return nothing.

One source of documentation for closures as input parameters mentions that there exist 3 kinds of closures:

  • Fn: takes captures by reference (&T)
  • FnMut: takes captures by mutable reference (&mut T)
  • FnOnce: takes captures by value (T)

This is a lot of information for a new developer.

I tried different possibilities, like:

1
2
3
4
5
6
7
8
9
10
11
12
13
fn with_todo_id(todos: &mut Vec<Todo>, todo_id: i16, f: &Fn(&mut Todo)) {
    if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
        f(todo);
    }
}

fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.deleted = true);
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.completed = true);
}

Without any success:

1
2
3
4
5
$ cargo run
   Compiling todo-list v0.1.0 (file:///Users/yannsimon/projects/rust/rust-playground/todo-list)
src/main.rs:27:38: 27:50 error: the type of this value must be known in this context
src/main.rs:27    with_todo_id(todos, todo_id, |todo| todo.deleted = true);
                                                      ^~~~~~~~~~~~

The official documentation was not so much help neither.

I asked for help on #rust-beginners. People on this channel are very helpful and kind. The community of Rust is awesome!

I was proposed 2 solutions. Both work, and I choose that one:

1
2
3
4
5
6
7
8
9
10
11
12
13
fn with_todo_id<P>(todos: &mut Vec<Todo>, todo_id: i16, f: P) where P: Fn(&mut Todo) {
    if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
        f(todo);
    }
}

fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.deleted = true);
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.completed = true);
}

Compared to my previous attempt, the f: &Fn(&mut Todo) is replaced by f: P where P: Fn(&mut Todo).

I still do not completely understand why this works and not the previous version. I was explained Rust can use the reference to the closure… I will continue reading documentation about it…. ;)

If you have any good source for this, please tell me.

In conclusion I still find closure as input parameters quite complex in Rust. I surely need to more understand the theory behind the language to fully understand them.

The Rust community is very helpful, but it may not scale if there are more and more beginners like me.

Update (May 25)

The following tweet from @rustlang provided me the good keywords to search for:

it's about trait objects vs type parameters, which can be tough when you're learning

Trait objects are used for dynamic dispatch, feature found in most OO languages.

With that in mind, I could understand the Rust book about closures.

If I use trait objects, this version works:

1
2
3
4
5
6
7
8
9
10
11
12
13
fn with_todo_id(todos: &mut Vec<Todo>, todo_id: i16, f: &Fn(&mut Todo)) {
    if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
        f(todo);
    }
}

fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, &|todo| todo.deleted = true);
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, &|todo| todo.completed = true);
}

Trait objects force Rust to use dynamic dispatch.

If I use type parameter instead of a trait object:

1
2
3
4
5
6
7
8
9
10
11
12
13
fn with_todo_id<P>(todos: &mut Vec<Todo>, todo_id: i16, f: P) where P: Fn(&mut Todo) {
    if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
        f(todo);
    }
}

fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.deleted = true);
}

fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
    with_todo_id(todos, todo_id, |todo| todo.completed = true);
}

then Rust is able to monomorphize the closure and use static dispatch, and does not need any object for the dyamic dispatch.

Another great example of the zero-cost abstraction possible with Rust!

Migrate a Playframework App From 2.3 to 2.5

Recently I migrated several play applications from the version 2.3 to the version 2.5.

As this experience may interest other people, I decided to write about it.

So let’s start!

§ Migration from 2.3 to 2.4

First, we will migrate the application from the version 2.3 to the version 2.4.

For that, the migration guide will be our reference.

How do I proceed?

1. Update the sbt config to play 2.4

First, I update the SBT configuration:

  • update the play version in project/plugins.sbt:
1
2
-addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.10")
+addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.4.6")
  • if needed, update the sbt version in project/build.properties:
1
sbt.version=0.13.11
  • update the build.sbt:
1
2
-import PlayKeys._
+import play.sbt.PlayImport._

I also had to replace playVersion.value with play.core.PlayVersion.current.

The goal of this first step is to be able to load the application with sbt, even if the application itself does not compile. At this point, Intellij can load the application with a play 2.4 version, and can provide auto-completion.

2. Fix compilation errors

Then, I fix the compilation

Inside sbt, I just trigger the compilation:

1
~compile

and fix all compilation errors. The play team has made a very good job at documenting all steps in the migration guide.

At this point, I just want the application to compile with the minimal number of changes. I do not care if I use deprecated APIs.

3. Run the application

When all compilation errors are fixed, I just make sure that the application can run:

In sbt:

1
~run

When running the application, I sometimes discovered version conflicts between libs (ex: Netty versions) and I know if I can complete the migration or if I have to update a dependency before.

Analysing all dependencies can be difficult. I use the sbt-dependency-graph to have an overview of all dependencies and find overriding versions.

4. Fix the tests

Then I fix all compilation errors in the tests, and then check that the tests are successful.

In sbt:

1
~testQuick

When all the tests are green, the application is migrated. The migration to play 2.4 is now completed, profit from the new version! ;)

5. Remove all usages of deprecated API

Now it is time to clean our application and to fix all code using a deprecated API.

This clean up can be done step by step.

I use the Scala API, and I prefer not to use any runtime dependency injection framework.

I introduced the so-called compile time dependency injection to simply instantiate my controllers in one application loader.

For more info about this topic, please refer to:

This cleanup is necessary before updating to play 2.5, as deprecated API are likely to be removed in the next version.

§ Migration from 2.4 to 2.5

For that, the migration guide will be our reference.

The migrate follow the same steps as the previous migration:

1
2
-addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.4.6")
+addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.2")
1
2
-    Logger.configure(context.environment)
+    LoggerConfigurator(context.environment.classLoader).foreach { _.configure(context.environment) }
1
-routesGenerator := InjectedRoutesGenerator

Trampoline Execution Context With Scala Futures

Disclaimer: I am continually learning, and this post reflects my current understanding of the topic. I may be wrong. Do not believe directly what I write. Test what you need. If you want to provide some precisions or corrections, please contact me on twitter, and I’ll fix this post.

Rúnar showed in a blog post how Scalaz Tasks can perform better than the Scala Futures.

He explains the details very well. If you have not read that post, I recommend it highly.

The main point if that Scala Future adds a context switching for each map or flatMap. With Scalaz Task, we have to describe which tasks need a new thread, the other ones are called on the same thread as the previous computation, avoiding these context switchings.

With Scala Futures, if you want to multiply the result of a Future[Int] by 2, you need an ExecutionContext (EC):

1
2
3
import scala.concurrent.ExecutionContext.global
val futureCount: Future[Long] = futureCountOfUsers()
val result = futureCount.map(i => i * 2)(global)

or with an implicit resolution:

1
2
3
import scala.concurrent.ExecutionContext.Implicits.global
val futureCount: Future[Long] = futureCountOfUsers()
val result = futureCount.map(i => i * 2)

To compute the i => i * 2, the ExecutionContext may use a different thread than the one having the result of the futureCountOfUsers. We observe a context switching between the future and the callback in the map. And the thread executing i => i * 2 can run on a different CPU/core than the one having the result of futureCount, meaning that the CPU cache is missed.

This overhead is not problematic for simple computations. But if we do 100 or 1000 of them, then it can have a significant impact on performances.

And in my opinion, Scala Futures have other downsides.

For example, with the following code:

1
2
3
4
for {
  value1 <- functionThatReturnsFutureValue1
  value2 <- functionThatReturnsFutureValue2
} yield (value1, value2)

functionThatReturnsFutureValue1 and functionThatReturnsFutureValue2 runs sequentially, even if there is no dependency between the two.

On the other hand:

1
2
3
4
5
6
7
val futureValue1 = functionThatReturnsFutureValue1
val futureValue2 = functionThatReturnsFutureValue2

for {
  value1 <- futureValue1
  value2 <- futureValue2
} yield (value1, value2)

computes functionThatReturnsFutureValue1 and functionThatReturnsFutureValue2 in parallel.

It means that Scala Futures do not respect the principe of “referential transparency”. It’s not only a theoretical problem, new users of Scala Futures often have problems with that.

And what I do not like about Scala Future is that we always need an ExecutionContext, even for small non-blocking computations.

For example, instead of writing:

1
2
def multiplyBy2(f: Future[Long]): Future[Long] =
  f.map(_ * 2)

We need either to import a ExecutionContext that is always used, or leave the liberty to the caller of the function and write:

1
2
def multiplyBy2(f: Future[Long])(implicit ec: ExecutionContext): Future[Long] =
  f.map(_ * 2)

My first impression with Scalaz Tasks is that they have a better design than the Scala Futures. But I have not used Scalaz Tasks extensively and cannot say if they have other problems.

But all in all, Scala Futures are here to stay. They are part of the standard API and are used everywhere.

I’m still wondering why the Scala Futures were designed that way. I can only guess, but I think this avoids some common pitfalls:

  • “easy” for new-comers: simply import the default execution context and that’s all
  • safe by default: If a callback takes a long time (blocking IO and expensive computation), this callback will not block other ones. The execution context will be able to use a different thread for other computations.
  • and a design like Scala Tasks works well if all parts of the system are non-blocking and using one thread pool. The reality is more complex. Each driver/http client can use its own thread pool. For example, an asynchronous http client may have its own thread pool because some parts of the networking API in Java is blocking like the standard ns lookup InetAddress.getByName(). Running a computation directly on the thread without forking it will run the computation of the thread pool of the http client. And that can lead to an exhaustion of the thread pool of the http client, and the http client cannot function anymore.

Introducing the trampoline execution context

This performance problem with the standard execution context is not new. The play framework team had this problem, especially with Iteratees that compute everything with a Future and uses callbacks extensively on stream of data. To solve this problem, James Roper introduced the trampoline Execution Context. This trampoline execution context is a great piece of software:

  • it makes sure the callbacks are run on the same thread than the future to avoid context switchings.
  • it does not overflow with recursive callbacks.

To show the benefit of the trampoline execution context, let’s call this function that does not make any sense, but simply calls Future.map n times:

1
2
def range(n: Int)(implicit ec: ExecutionContext): Future[Int] =
  (0 to n).foldLeft[Future[Int]](Future.successful(0)) { case (f, _)  f.map(_ + 1) }

With n = 5 000 000:

  • Scala Futures with standard EC: 0.037 ops/s
  • Scala Futures with trampoline EC: 1.397 ops/s

Should we use the trampoline EC?

When we are confident with execution contexts, I thing we can use this trampoline EC if:

  • the callback is running very fast. For example:
1
2
def multiplyBy2(f: Future[Long]): Future[Long] =
  f.map(_ * 2)(trampolineEC)
  • we never call some blocking IO. This point can be tricky: some scala libs can use some java libs that use InputStream or OutputStream that can block.

If you are unsure, use the standard EC.

If you want to try this yourself, the code is on github.

Review of Essential Slick (for Slick 3.0)

I had the pleasure to review the book “Essential Slick” about Slick 3.

I really enjoyed reading it. For anyone starting a project with Slick, I highly recommend this book, in particular to learn about:

  • how to build queries that can be composed and re-used
  • how to structure the code
  • the patterns, anti-patterns and pitfalls to avoid.

The book goes over a lot of concepts, from simple queries and data-modeling to views, projections, joins and aggregates.

Slick 3 has very powerful concepts that could be hard to manage. The book tries to teach us how to use them.

Some chapters are more demanding, for instance the 2nd chapter introduces in only a few pages concepts like Queries, Actions, Effects, Streaming.

But once the step managed, I had the feeling to have the good abstractions in my mind to build an application based on Slick. And I could use these concepts to solve the exercises.

What I really liked:

  • exercices: I think I can only learn a library when I use it and the exercises in the book are a very good opportunity to practice.
  • the sbt console is so configured that I could copy/paste all examples from the book into the console and play with them.
  • focus on composability and re-usability
  • the book also points out when one abstraction is not optimal for a particular problem and proposes some alternatives.

What I missed:

  • the 3.0 version of Slick is a lot about asynchronity, streaming and I wish the book could dig further in these concepts.

All in all, a very good book!

Play Framework Meetup in Berlin in November 2015

During the play framework meetup in Berlin in November 2015:

for a better workflow between designers and developers

Laura made a very good talk about how to better organize the work between front-end designers and developers.

For that, her team build some tools:

Slides: http://slides.com/lauraluiz/handlebarsplay

Project using this workflow: https://github.com/sphereio/sphere-sunrise

type classes in Scala

I introduced type classes in Scala: http://www.slideshare.net/yann_s/introduction-to-type-classes-in-scala

Step by step, I explained how the Json Reads/Writes/Formats work in the scala API of the play framework.

DI With Play 2.4

During the play framework meetup in Mai 2015, there were 3 talks about Dependency Injection (DI) in Play 2.4:

Goto Conference 2014

My notes from the goto conference Berlin 2014:

Thursday

Opening Keynote: Software Design in the 21st Century - Martin Fowler

We, developers, take responsibility in the code we write.
We cannot simply say: “I implemented that because I was told so”.
Avoid dark patterns.

We are not code monkeys.

Aeron: The Next Generation in Open-Source High-Performance Messaging - Martin Thompson

slides blog

Aeron is a OSI layer 4 Transport for message oriented streams. is simply impressive, achieving a very low latency.
I’d like to see an integration of Aeron in Akka cluster.

similar talk

Writing highly Concurrent Polyglot Applications with Vert.x - Tim Fox

It was a good introduction to Vert.x.
But I was expecting more than an introduction.

Fast Analytics on Big Data - Petr Maj and Tomas Nykodym

slides

Presentation of an ML runtime for bigdata.

Code With Spark API

TODO: have a look at https://github.com/0xdata/h2o-training

Security Architecture for the SmartHome - Jacob Fahrenkrug

slides

A good presentation about the future threats of connected objects and the solution yetu implements.

Graph All The Things!! Graph Database Use Cases That Aren’t Social - Emil Eifrem

slides

very good presentation of Neo4j.
We should use a graph database where it make sense - the search queries are much easier to write / read, and the performance very good.

Party Keynote: Staying Ahead of the Curve - Trisha Gee

slides

Friday

Morning Keynote: Excellence Culture & Humane Keeping of Techies - Prof. Dr. Gunter Dueck

a very good keynote, very funny and true at the same time.

Aerospike: Flash-optimized, High-Performance noSQL database for All - Khosrow Afroozeh

slides

A new database on my radar - impressive.

Docker - A Lot Changed in a Year - Chris Swan

slides

Docker is maturating.
Just do not forget that “containers do not contain” -> no complete security between container and host, and between containers.

The Joys and Perils of Interactive Development - Stuart Sierra

slides

New Concurrency Utilities in Java 8 - Angelika Langer

slides

  • Future API is becoming usable, it is now possible to chain futures and callbacks.
  • new StampedLock is optimized for reading. (personal note: but lock-free algorithms should be preferred, like in Lock-Based vs Lock-Free Concurrent Algorithms)
Adaptive Planning - Beyond User Stories - Gojko Adzic

blog slides

Gojko Adzic really thinks agile. Good presentation of how to make better user stories.
Do not describe a desired behavior but a behavior change.

Asynchronous IO for Play! Applications in Servlet 3.1 Containers With the Play2-war Plugin

A Play application does not need any container and runs directly in production.

However some organizations prefer to run play applications within a servlet container and can use for this the WAR Plugin.

I am very pleased to share that this plugin is now compatible with servlet 3.1 containers. It can now use the new asynchronous IO possibilities.

Let me explain why and when it is important.

Play applications are asynchronous

The Play Framework is build to be totally asynchronous and reactive. It uses no blocking IO. A Play application scales very well, using very few threads.

This reactive consuming or construction of IO stream is designed in play with Iteratees. It will progressively be completed with Akka Streams, the implementation in Akka of the reactive stream project (http://www.reactive-streams.org/)

Servlet containers use blocking IO

On the other hand, servlet containers traditionally use a thread per request. When doing IO (database access, external web request), the thread waits for the IO completion (blocking IO). That’s why servlet containers need a lot of working threads to handle requests in parallel (default 200 threads max for tomcat)

Asynchronous 3.0 servlet

Servlet 3.0 containers introduced the possibility to “suspend” a request. For example, if an application makes an HTTP request to another web service using the WS client, the play2 war plugin is able to suspend the servlet request until the external web service answers. It means that with the same number of servlet threads, a servlet 3.0 container can support more requests in parallel than a servlet 2.x container. It does not need that a thread waits for an HTTP request, this thread can be used for other requests.

Limitations of asynchronous 3.0 servlet

When uploading or downloading a big file, the servlet container is able to stream the data. But between each chunks, the servlet thread is blocking, waiting for the next one. It is not possible to consume one chunk, and later, when we are ready to consume another one, tell the container: “now I am ready, you can send me the next chunk as soon as you receive one from the browser”.

If a browser needs an hour to upload a file with a slow Internet connection, the container needs a thread during an hour, even if the application does not do anything, just waiting for the upload completion.

Play applications deployed as war are limited by the servlet container

A Play application deployed in a war container is limited by the technical possibilites of the servlet API. With a servlet 2.x or 3.0, a Play application does not scale as well as when run natively.

New asynchronous IO in servlet 3.1

The servlet 3.1 specification added asynchronous IO. Based on events, it is now possible to completly handle an upload and a download asynchronously.

Asynchronous IO in WAR Plugin for Play! applications

Since the version 1.2, the play2 war plugin is using this API to provide a complete reactive upload and download.

To use this, simply configure the servlet container version and deploy to a servlet 3.1 server.

When to use this feature?

This feature should be used especially if the application is using big download/upload. The servlet 3.1 will help to scale much better.

During my tests, I could upload and download files from several GB in parallel. The container and the JVM garbage collection could support that without any problems. The memory usage was very low.

Also please test this new version and report issues!

How to build the application to scale better?

To scale as much as possible, the application should not block. It should always use asynchronous IO API like in the WS client.

But in the Java World a lot a librairies are still designed for a one-thread-per-request model and do not provide asynchronous API. It is for example the case for a JDBC driver. In that case, a separate dispatcher should be configured to handle blocking IO. More Information for this can be found in the Play Framework documentation.

Implementation history

The implementation of asynchronous IO in the WAR plugin lasted a few months. The first pull request introduced the asynchronous download, and the second one the asynchronous upload. I’d like to thank James Roper and Viktor Klang for their reviews.

This feature was quite challenging to implement:

  • I had to find a good way to implement the glue between two very different APIs. The servlet 3.1 API is quite imperative and use EventListener and methods with side effects. The Iteratee API is functional and I needed some time to feel at ease with it.

  • The servlet 3.1 specification was recently finalized as I began. The first implementations in some containers contained bugs. Reporting the problems, explaining, convincing other developers took a lot of time and energy.

  • The servlet 3.1 specification is not always explicit. The implementations in the different servlet containers are also different. Finding an implementation that covers all these subtle differences was challenging. The testing infrastructure of the play2-war plugin provides a lot of integration tests with different containers and helped a lot for this.

My firm Leanovate gave me some time to work on that. Having worked two days full time on it helped me a lot. Thanks Leanovate for this!

All in all it was a great experience to contribute to the WAR Plugin, and I hope my work will be useful for others.

Enlarge Your Test Scope

At the beginning at the year, I had the chance to present how to organize a play application with the Cake pattern at Ping Conf. I showed how this pattern enable designing the application as components, how to reduce visibility of specific gateway’s model only to components that need it. One side-effect of the cake pattern is that it allows a dependency injection resolved at compilation time.

In one of my last slides, I warned against abusing a dependency injection mechanism to write too much unit tests. To stay within the time slot, I have not developed so much my concerns about that point.

During the talk, I implemented an application to demonstrate my points. This application consumes two external web services to render HTML pages. It is quite a typical application we can see in an infrastructure build with micro-services.

I’ve now took the time to write a new version of this application I used in the demo. And This new version is not using any unit tests but only some sort of component tests.

Let’s dig into the details how this new version differs from the ones build around the Cake pattern.

Traditional view of unit tests

When building an application, we usually structure the code into layers to separate responsibilities, thus enabling re-use of logic, and avoiding repetition.

In the demo I used for the talk, the application is for example layered into views, controllers, services and gateways. All these layers have access to a common model.

A traditional approach of unit test is to consider one class or function of one layer as a unit to test. The other dependent layers are mocked.

For example, to test the service layers, we use mocks for the gateways, simulating responses from external web services.

This approach works well, but has several downsides:

  • the unit tests prove that one layer is working as expected, but they said nothing about all the layers used together.
  • the unit tests use the internal details of the implementation. Re-factoring the code implies then to change a lot of tests.

By using dependency injection and mocks, it is nowadays very easy to write unit tests. The effect if some applications are tested almost only with unit tests:

Traditional view of component tests

To complement the unit tests, a team can decide to implement some component tests.

For the sample used in the talk, the component is the font end application we are currently implementing.

The most common way to run component tests is to start the tested application. The application is configured not to use the real external web services, but some local mock implementations. The local mock implementations are started as http servers as well, running on different ports.

When the application and all the mocks are started, we can test the application by sending some http requests and by analyzing the responses.

Setting up the test environment with this approach is quite complex. For each external web service, a mock must be implemented as a real local http server. We must start all mock implementations, and then the new configured application. At the end of the tests, we must shutdown all services, even in case of exceptions.

But the main drawback with this approach in my opinion is that running the tests take a lot of time, too much to be integrated in a normal development flow (write code -> compile -> test)

An alternative approach between component and unit tests

To strictly adhere to the definition of component tests, we should treat the tested application as a black box, and simulate all external web services. We saw that this approach is somewhat heavy to use: each external web service must be mock with a real http server.

Starting and running the tests in that configuration take time. Debugging an error can be difficult.

The strategy I used in new version of the demo application (TBA_07) is a little bit different. The idea is still to use a request / response to test the application, but without having to run the application and any external web services.

Implementing that is actually quite easy: each layer declared as dependency an HTTP client (a WSClient in play framework 2.3)

  • The http client is a dependency at the top (controllers' layer):
1
2
3
4
5
6
7
8
package controllers

class Application(ws: WSClient, app: play.api.Application) extends Controller {

  val topVideoService = new TopVideoService(ws, app)
//...

}

(a second “dependency” is the current play application. This approach is very convenient to simulate different configurations)

  • The real implementation of the http client is then “injected” at the last time, when we construct the controller singleton:
1
2
def playCurrent = play.api.Play.current
object Application extends Application(WS.client(playCurrent), playCurrent)
1
2
class ApplicationControllerFixture
  extends Application(MockWS(playerRoute), FakeApplication())
  • The playerRoute simulate the external player web service:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
val playerId = PlayerId(34)

val playerRoute: MockWS.Routes = {
  case ("GET", u) if u == s"http://localhost:9001/players/$playerId" =>
    Action { Ok(Json.parse(playerJson(playerId))) }

  case _ => Action { NotFound }
}

def playerJson(playerId: PlayerId) =
  s"""{
       |  "id": $playerId,
       |  "name": "ze name",
       |  "height": "ze height",
       |  "weight": "ze weight",
       |  "team": "ze team"
       |}
     """.stripMargin

The MockWS.Routes type defines a partial function PartialFunction[(String, String), EssentialAction], making really easy to combine different routes together with orElse:

1
SimulatedVideoBackend.videoRoute orElse SimulatedPlayerBackend.playerRoute
  • and we can test the response by calling the controller with a FakeRequest:
1
2
val result = index.apply(FakeRequest())
status(result) mustEqual OK

The application is constructed as if it was depending from the http client and the current play application.

These tests are not strictly component tests, as we are not testing the real implementation of the http client. The application is not entirely treated as a black box. But most of the code is tested like in production.

Drawbacks of this approach:

  • writing a test is more complicated than testing a little unit of code
  • writing unit test can help avoiding code mixing responsibilities. We do not profit from that.
  • when a test suddenly fails, it is more difficult to find out why.
  • we do not test the complete application stack. For example, the play filters and the real http client is not tested.

Advantages of this approach:

  • a developer must understand how the application works in general to be able to write good tests
  • the application can be re-factored without modifying the tests
  • the user functions are better tested
  • the integration of all layers together is tested
  • we do not need any running server to check all the tests. The tests run very fast.
  • the code is simple (compare the TopVideoService in that version with the one using the Cake pattern)

Experience with that approach

With one team, we are currently testing this approach. The results are quite encouraging: more than 80 % of the statements are covered by tests. We have more than 200 tests running in 10 seconds on my machine.

And I could deeply change the code with almost no impact on the tests. ;)