Yann's blogZola2021-12-21T00:00:00+00:00https://yanns.github.io/atom.xmlTalk about mongo, FS2 and reactive streams2021-12-21T00:00:00+00:002021-12-21T00:00:00+00:00https://yanns.github.io/blog/2021/12/21/fs2-mongo-reactivestreams/<p>At the <a href="https://www.meetup.com/de-DE/Scala-Berlin-Brandenburg/">scala meetup in Berlin</a>, I presented my investigation on how the <a href="https://mongodb.github.io/mongo-java-driver/">mongo java driver</a>, <a href="https://www.reactive-streams.org/">reactive streams</a> and <a href="https://fs2.io/">FS2</a> are working together.</p>
<p>By comparing akka-stream and fs2, I could find out a difference in how many elements are asked to the reactive publisher.
That resulted in much more mongo queries when using FS2.</p>
<ul>
<li><a href="https://www.slideshare.net/yann_s/fs2-mongo-reactivestreams">Slides</a></li>
<li><a href="https://github.com/yanns/mongo-fs2">Source code</a></li>
</ul>
<p>This whole investigation resulted in a <a href="https://github.com/typelevel/fs2/pull/2666">new API in FS2</a>.
With this change, FS2 is now capable of consuming a reactive stream publisher by asking more than one element at a time. When used with the mongo java driver, it results in less queries fetching each more documents.</p>
GraphQL Conf 20212021-10-28T00:00:00+00:002021-10-28T00:00:00+00:00https://yanns.github.io/blog/2021/10/28/graphql-conf/<p>I am in a lucky situation: at commercetools, I've experienced the adoption of GraphQL as a public API, started as an experiment in 2015 until full support in production.</p>
<p>I've already related about this experience at a <a href="/blog/2020/06/11/graphql-meetup/">GraphQL Meetup</a> and I was invited at <a href="https://graphqlconf.org/">GraphQL Conf 2021</a> to do the same.</p>
<p>If you are interested, you can have a look at the <a href="https://www.slideshare.net/yann_s/bringing-a-public-graphql-api-from-the-whiteboard-to-production-250332399">slides</a> or even <a href="https://www.youtube.com/watch?v=FwAITJkozr8&list=PL9GHW1s6-K4ZXvbb_nPdV76KlJslmQaEX&index=16">watch the talk online</a>:</p>
<p><a href="https://www.youtube.com/watch?v=FwAITJkozr8&list=PL9GHW1s6-K4ZXvbb_nPdV76KlJslmQaEX&index=16"><p><img src="/assets/2021-10-28/GraphQL Conf.png" alt="GraphQL Conf"/></p>
</a></p>
Bitten by hash codes on production2021-02-24T00:00:00+00:002021-02-24T00:00:00+00:00https://yanns.github.io/blog/2021/02/24/bitten-by-hash-codes-on-production/<h1 id="hash-codes-can-be-surprising">Hash codes can be surprising</h1>
<p>I'd like to write about a performance issue I found out on production.</p>
<p>This issue is coming from change to a Scala function. That change looked totally innocent at the first place.</p>
<p>The code in question is about fetching JSON object from external resources.
The function accepts <code>ids</code> as input and returns a <code>Map</code> where the value is an <code>Option[JObject]</code> to represent, for each <code>id</code> if a JSON object could be found or not.</p>
<p>I'll try to explain this with some code. I simplified the original code a lot to focus on the important port.</p>
<p>First some code just to fake the data structures used later:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> Id</span><span>(</span><span style="color:#bf616a;">value</span><span>: </span><span style="color:#ebcb8b;">String</span><span>)
</span><span>
</span><span style="color:#65737e;">// just to represent a JSON object
</span><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> JObject</span><span>(</span><span style="color:#bf616a;">fields</span><span>: </span><span style="color:#ebcb8b;">List</span><span>[(</span><span style="color:#ebcb8b;">String</span><span>, </span><span style="color:#b48ead;">Any</span><span>)])
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">loadFromExternalServer</span><span>(</span><span style="color:#bf616a;">id</span><span>: </span><span style="color:#ebcb8b;">Id</span><span>): </span><span style="color:#ebcb8b;">Option</span><span>[</span><span style="color:#ebcb8b;">JObject</span><span>] =
</span><span> </span><span style="color:#b48ead;">if </span><span>(id.value.hashCode() % </span><span style="color:#d08770;">2 </span><span>== </span><span style="color:#d08770;">0</span><span>) </span><span style="color:#65737e;">// simulate finding the external resource or not
</span><span> None
</span><span> </span><span style="color:#b48ead;">else
</span><span> Some(JObject(List("</span><span style="color:#a3be8c;">id</span><span>" -> id.value, "</span><span style="color:#a3be8c;">name</span><span>" -> "</span><span style="color:#a3be8c;">a name</span><span>")))
</span></code></pre>
<h3 id="the-initial-implementation">The initial implementation</h3>
<p>The function was initially implemented like this:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">loadRecords</span><span>(</span><span style="color:#bf616a;">ids</span><span>: </span><span style="color:#ebcb8b;">Seq</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>]): </span><span style="color:#ebcb8b;">Map</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>, </span><span style="color:#ebcb8b;">Option</span><span>[</span><span style="color:#ebcb8b;">JObject</span><span>]] =
</span><span> ids
</span><span> .map(</span><span style="color:#bf616a;">id </span><span style="color:#b48ead;">=> </span><span>(id, loadFromExternalServer(id)))
</span><span> .toMap
</span></code></pre>
<p>If we call this function, we will get the expected <code>Map</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">ids </span><span>= Seq(Id("</span><span style="color:#a3be8c;">id-1</span><span>"), Id("</span><span style="color:#a3be8c;">id-2</span><span>"), Id("</span><span style="color:#a3be8c;">id-3</span><span>"), Id("</span><span style="color:#a3be8c;">id-4</span><span>"), Id("</span><span style="color:#a3be8c;">id-5</span><span>"))
</span><span>
</span><span>loadRecords(ids)
</span><span style="color:#65737e;">// Map(Id(id-1) -> Some(JObject(List((id,id-1), (name,a name)))), Id(id-5) -> Some(JObject(List((id,id-5), (name,a name)))), Id(id-4) -> None, Id(id-3) -> Some(JObject(List((id,id-3), (name,a name)))), Id(id-2) -> None)
</span></code></pre>
<h3 id="the-change">The change</h3>
<p>At some point, one notices that we sometimes are calling <code>loadRecords</code> with duplicated ids. To avoid that, we changed the <code>Seq</code> to a <code>Set</code> to make sure that all ids are unique:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">loadRecords</span><span>(</span><span style="color:#bf616a;">ids</span><span>: </span><span style="color:#ebcb8b;">Set</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>]): </span><span style="color:#ebcb8b;">Map</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>, </span><span style="color:#ebcb8b;">Option</span><span>[</span><span style="color:#ebcb8b;">JObject</span><span>]] =
</span><span> ids
</span><span> .map(</span><span style="color:#bf616a;">id </span><span style="color:#b48ead;">=> </span><span>(id, loadFromExternalServer(id)))
</span><span> .toMap
</span></code></pre>
<p>As you can notice, we only changed <code>ids: Seq[Id]</code> to <code>ids: Set[Id]</code> in the function signature.</p>
<p>The result of calling this function does not change:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">ids </span><span>= Set(Id("</span><span style="color:#a3be8c;">id-1</span><span>"), Id("</span><span style="color:#a3be8c;">id-2</span><span>"), Id("</span><span style="color:#a3be8c;">id-3</span><span>"), Id("</span><span style="color:#a3be8c;">id-4</span><span>"), Id("</span><span style="color:#a3be8c;">id-5</span><span>"))
</span><span>
</span><span>loadRecords(ids)
</span><span style="color:#65737e;">// Map(Id(id-1) -> Some(JObject(List((id,id-1), (name,a name)))), Id(id-5) -> Some(JObject(List((id,id-5), (name,a name)))), Id(id-4) -> None, Id(id-3) -> Some(JObject(List((id,id-3), (name,a name)))), Id(id-2) -> None)
</span></code></pre>
<h3 id="the-impact">The impact</h3>
<p>When looking at the servers on production, I found a strange part in one flame graph:</p>
<p><img src="/assets/2021-02-24/flame-graph-productHash.png" alt="Flame graph showing an important portion of scala/util/hashing/MurmurHash3.productHash"/></p>
<p>The servers were spending a significant amount of time computing the hash codes of JSON objects.</p>
<h3 id="the-explanation">The explanation</h3>
<p>It took me some time to figure out the issue.</p>
<p>Let's focus on this part:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span> ids
</span><span> .map(</span><span style="color:#bf616a;">id </span><span style="color:#b48ead;">=> </span><span>(id, loadFromExternalServer(id)))
</span></code></pre>
<p>This piece of code produced a <code>Seq[(Id, Option[JObject])]</code> before the change.
After the change, it produces a <code>Set[(Id, Option[JObject])]</code>. </p>
<p>For every item inserted, a <code>Set</code> must check for duplicates. It does that with the help of the hash code of each element.</p>
<p>After the change, <code>Set</code> was also asking for the <code>hashCode()</code> of the <code>JObject</code>, computed recursively by traversing the whole JSON object. For large objects, this computation can be costly. (At least, costly enough to make it apparent in a flame graph)</p>
<p>This change went unnoticed because the <code>hashCode()</code> method exists on all objects in Java.</p>
<p>If we had used type classes, we may have noticed this issue before. By assuming that there is no instance of <code>def hashCode(): Int</code> for <code>JObject</code> (which seems fair), the compiler would have fail.</p>
<h3 id="avoiding-intermediate-collections">Avoiding intermediate collections</h3>
<p>To avoid computing intermediate collections, we can also use the <code>iterator()</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">loadRecords</span><span>(</span><span style="color:#bf616a;">ids</span><span>: </span><span style="color:#ebcb8b;">Set</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>]): </span><span style="color:#ebcb8b;">Map</span><span>[</span><span style="color:#ebcb8b;">Id</span><span>, </span><span style="color:#ebcb8b;">Option</span><span>[</span><span style="color:#ebcb8b;">JObject</span><span>]] =
</span><span> ids
</span><span> .iterator
</span><span> .map(</span><span style="color:#bf616a;">id </span><span style="color:#b48ead;">=> </span><span>(id, loadFromExternalServer(id)))
</span><span> .toMap
</span></code></pre>
<p>That way, we avoid instantiating the intermediate <code>Seq</code> or <code>Set</code>.</p>
Sangria & Scala Futures, round 12020-12-24T00:00:00+00:002020-12-24T00:00:00+00:00https://yanns.github.io/blog/2020/12/24/sangria-future-round-1/<h1 id="sangria-and-future">Sangria and <code>Future</code></h1>
<p><a href="https://sangria-graphql.github.io/">Sangria</a> is a Scala library implementing GraphQL on the server side.</p>
<p>To use it, one defines the GraphQL schema by defining types, fields, and, for each field, how to solve it.</p>
<p>When Oleg started this library, he used the Scala <code>Future</code> as main representation for asynchronous computation.</p>
<p>The main entry point is the <code>sangria.execution.Executor</code>. Here is a simplified version:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> Executor</span><span>[</span><span style="color:#ebcb8b;">Ctx</span><span>, </span><span style="color:#ebcb8b;">Root</span><span>](</span><span style="color:#bf616a;">schema</span><span>: </span><span style="color:#ebcb8b;">Schema</span><span>[</span><span style="color:#ebcb8b;">Ctx</span><span>, </span><span style="color:#ebcb8b;">Root</span><span>])(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">executionContext</span><span>: </span><span style="color:#ebcb8b;">ExecutionContext</span><span>) </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">execute</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">queryAst</span><span style="color:#eff1f5;">: ast.</span><span style="color:#ebcb8b;">Document</span><span style="color:#eff1f5;">)(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">scheme</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionScheme</span><span style="color:#eff1f5;">): scheme.</span><span style="color:#ebcb8b;">Result</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Ctx</span><span style="color:#eff1f5;">, marshaller.</span><span style="color:#ebcb8b;">Node</span><span style="color:#eff1f5;">] </span><span>= </span><span style="color:#eff1f5;">{...}
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>where <code>scheme.Result</code> is by default implemented as a <code>Future</code>.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span> </span><span style="color:#b48ead;">implicit object</span><span style="color:#ebcb8b;"> Default </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">ExecutionScheme </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">type </span><span style="color:#eff1f5;">Result[</span><span style="color:#ebcb8b;">Ctx</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">Res</span><span style="color:#eff1f5;">] </span><span>= </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Res</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> }
</span></code></pre>
<p>The resolution of the GraphQL query is delegated to <code>sangria.execution.Resolver</code>, which is using <code>Future</code> internally:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> Resolver</span><span>[</span><span style="color:#ebcb8b;">Ctx</span><span>](</span><span style="color:#bf616a;">queryAst</span><span>: ast.</span><span style="color:#ebcb8b;">Document</span><span>)(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">executionContext</span><span>: </span><span style="color:#ebcb8b;">ExecutionContext</span><span>) </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">processFinalResolve</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">resolve</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Resolve</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[(</span><span style="color:#ebcb8b;">Vector</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">RegisteredError</span><span style="color:#eff1f5;">], marshaller.</span><span style="color:#ebcb8b;">Node</span><span style="color:#eff1f5;">)] </span><span>= </span><span style="color:#eff1f5;">{...}
</span><span style="color:#eff1f5;">}
</span></code></pre>
<h1 id="future-is-not-the-only-option"><code>Future</code> is not the only option</h1>
<p>In the <code>Scala</code> world, <code>Future</code> is not the only option to work with asynchronous computations.</p>
<p>And at the time I'm writing this post, <a href="https://typelevel.org/cats-effect/">cats-effect</a> 3.x is being developed, providing a set of very interesting features and performance improvements.</p>
<p>I've also used Sangria in application using the <a href="https://monix.io/">Monix</a> <code>Task</code> instead of <code>Future</code>. The code integrating Sangria is converting back and forth between <code>Task</code> and <code>Future</code>. It's not a major issue, but still does not feel optimal.</p>
<p>I think that there is value for a library like Sangria to better support alternatives to <code>Future</code>.</p>
<h1 id="supporting-alternatives-to-future">Supporting alternatives to <code>Future</code></h1>
<p>So I'm trying to enhance Sangria to be able to support any alternatives to <code>Future</code>.</p>
<p>By looking at the code, I see some usages of <code>future.map</code>, <code>future.sequence</code>, etc.</p>
<h2 id="abstracting-with-a-functional-library">Abstracting with a functional library</h2>
<p>My first reflex is to use a functional library like <code>cats</code> and to make the code abstract by using type classes like <a href="https://typelevel.org/cats/typeclasses/monad.html"><code>Monad</code></a>, <a href="https://typelevel.org/cats/typeclasses/applicative.html"><code>Applicative</code></a> and so on.</p>
<p>But Oleg wanted the Sangria library to have minimal dependencies. He has not used any functional library.</p>
<p>So I'll respect this decision and try another approach.</p>
<h2 id="abstracting-over-future">Abstracting over <code>Future</code>.</h2>
<p>To make the Sangria code not depending on <code>Future</code> directly, I'm introducing a trait that should abstract <code>Future</code> away:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">trait</span><span style="color:#ebcb8b;"> Effect</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#bf616a;">_</span><span>]] </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">pure</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">a</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">map</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">fa</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">])(</span><span style="color:#bf616a;">f</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A </span><span>=> </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>To stay compatible with <code>Future</code>, I also provide a default implicit implementation for it.
If my experiment goes well, at the end, this should be the only place where Sangria is using <code>Future</code>.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> Effect </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">implicit def </span><span style="color:#8fa1b3;">FutureEffect</span><span style="color:#eff1f5;">(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">ec</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionContext</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">Effect</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">] </span><span>=
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">Effect</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">] {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">override def </span><span style="color:#8fa1b3;">pure</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">a</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">] </span><span>= </span><span style="color:#eff1f5;">Future.successful(a)
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">override def </span><span style="color:#8fa1b3;">map</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">fa</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">])(</span><span style="color:#bf616a;">f</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A </span><span>=> </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">] </span><span>=</span><span style="color:#eff1f5;"> fa.map(f)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>I know it would be better to first have <code>flatMap</code> and <code>unit</code> as <code>map</code> could be expressed in <code>flatMap</code> of <code>unit</code> but I don't care for now. The name is also not optimal.</p>
<p>My goal is first to check if I can quickly make Sangria using <code>Effect</code> instead of <code>Future</code>.
Once Sangria can compile with it, and if the current tests relying on <code>Future</code> pass, I can refine <code>Effect</code>.</p>
<p>So I'm starting using <code>Effect</code>. The compiler is calling me names.</p>
<p>But I am changing code step by step, and this is feeling good. I can remove <code>Future</code> completely from some classes:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>-</span><span style="color:#b48ead;">import</span><span> scala.concurrent.{ExecutionContext, Future}
</span><span>
</span><span>- </span><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> DeferredResult</span><span>(</span><span style="color:#bf616a;">deferred</span><span>: </span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Defer</span><span>]]], </span><span style="color:#bf616a;">futureValue</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#ebcb8b;">Result</span><span>])
</span><span>+ </span><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> DeferredResult</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#bf616a;">_</span><span>]: </span><span style="color:#ebcb8b;">Effect</span><span>](
</span><span>+ </span><span style="color:#bf616a;">deferred</span><span>: </span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Defer</span><span>]]],
</span><span>+ </span><span style="color:#bf616a;">futureValue</span><span>: </span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#ebcb8b;">Result</span><span>])
</span><span> </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">Resolve </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">appendErrors</span><span style="color:#eff1f5;">(
</span><span style="color:#eff1f5;"> </span><span style="color:#bf616a;">path</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionPath</span><span style="color:#eff1f5;">,
</span><span style="color:#eff1f5;"> </span><span style="color:#bf616a;">errors</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Vector</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Throwable</span><span style="color:#eff1f5;">],
</span><span style="color:#eff1f5;">- </span><span style="color:#bf616a;">position</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Option</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">AstLocation</span><span style="color:#eff1f5;">]) </span><span>=
</span><span style="color:#eff1f5;">+ position: </span><span style="color:#ebcb8b;">Option</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">AstLocation</span><span style="color:#eff1f5;">]): </span><span style="color:#ebcb8b;">DeferredResult</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">] </span><span>=
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(errors.nonEmpty)
</span><span style="color:#eff1f5;">- copy(futureValue </span><span>=</span><span style="color:#eff1f5;"> futureValue.map(</span><span style="color:#bf616a;">_</span><span style="color:#eff1f5;">.appendErrors(path, errors, position)))
</span><span style="color:#eff1f5;">+ copy(futureValue </span><span>= </span><span style="color:#eff1f5;">Effect[</span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">]().map(futureValue)(</span><span style="color:#bf616a;">_</span><span style="color:#eff1f5;">.appendErrors(path, errors, position)))
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">else </span><span style="color:#bf616a;">this
</span><span style="color:#eff1f5;"> }
</span></code></pre>
<p>The code is a bit more complex but still manageable:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>- </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">resolveValue</span><span>(
</span><span>+ </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">resolveValue</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#bf616a;">_</span><span>]: </span><span style="color:#ebcb8b;">Effect</span><span>](
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#ebcb8b;">ExecutionPath</span><span>,
</span><span> </span><span style="color:#bf616a;">astFields</span><span>: </span><span style="color:#ebcb8b;">Vector</span><span>[ast.</span><span style="color:#ebcb8b;">Field</span><span>],
</span><span> </span><span style="color:#bf616a;">tpe</span><span>: </span><span style="color:#ebcb8b;">OutputType</span><span>[</span><span style="color:#bf616a;">_</span><span>],
</span><span>@@ -</span><span style="color:#d08770;">1231</span><span>,</span><span style="color:#d08770;">23</span><span> +</span><span style="color:#d08770;">1239</span><span>,</span><span style="color:#d08770;">23</span><span> @@ </span><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> Resolver</span><span>[</span><span style="color:#ebcb8b;">Ctx</span><span>](
</span><span> resolveSimpleListValue(simpleRes, path, optional, astFields.head.location)
</span><span> </span><span style="color:#b48ead;">else </span><span>{
</span><span> </span><span style="color:#65737e;">// this is very hot place, so resorting to mutability to minimize the footprint
</span><span>- </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">deferredBuilder </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">VectorBuilder</span><span>[</span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Defer</span><span>]]]
</span><span>- </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">resultFutures </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">VectorBuilder</span><span>[</span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#ebcb8b;">Result</span><span>]]
</span><span>+ </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">deferredBuilder </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">VectorBuilder</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Defer</span><span>]]]
</span><span>+ </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">resultFutures </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">VectorBuilder</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#ebcb8b;">Result</span><span>]]
</span><span>
</span><span> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">resIt </span><span>= res.iterator
</span><span>
</span><span> </span><span style="color:#b48ead;">while </span><span>(resIt.hasNext)
</span><span> resIt.next() </span><span style="color:#b48ead;">match </span><span>{
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#bf616a;">r</span><span>: </span><span style="color:#ebcb8b;">Result </span><span style="color:#b48ead;">=>
</span><span>- resultFutures += Future.successful(r)
</span><span>- </span><span style="color:#b48ead;">case </span><span style="color:#bf616a;">dr</span><span>: </span><span style="color:#ebcb8b;">DeferredResult </span><span style="color:#b48ead;">=>
</span><span>+ resultFutures += Effect[</span><span style="color:#ebcb8b;">F</span><span>]().pure(r)
</span><span>+ </span><span style="color:#b48ead;">case </span><span style="color:#bf616a;">dr</span><span>: </span><span style="color:#ebcb8b;">DeferredResult</span><span>[</span><span style="color:#ebcb8b;">F</span><span>] </span><span style="color:#b48ead;">=>
</span><span> resultFutures += dr.futureValue
</span><span> deferredBuilder ++= dr.deferred
</span><span> }
</span><span>
</span><span> DeferredResult(
</span><span> deferred = deferredBuilder.result(),
</span><span>- futureValue = Future
</span><span>+ futureValue = Effect[</span><span style="color:#ebcb8b;">F</span><span>]()
</span><span> .sequence(resultFutures.result())
</span><span> .map(resolveSimpleListValue(</span><span style="color:#bf616a;">_</span><span>, path, optional, astFields.head.location))
</span><span> )
</span></code></pre>
<p>As I abstract more and more functions on <code>Future</code>, the <code>Effect</code> abstraction is growing:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">trait</span><span style="color:#ebcb8b;"> Effect</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#bf616a;">_</span><span>]] </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">pure</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">a</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">failed</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">T</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">exception</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Throwable</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">T</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">map</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">fa</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">])(</span><span style="color:#bf616a;">f</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A </span><span>=> </span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">B</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">sequence</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">vector</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Vector</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]]): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Vector</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]]
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">recover</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">U </span><span>>: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">fa</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">])(</span><span style="color:#bf616a;">pf</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">PartialFunction</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Throwable</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">U</span><span style="color:#eff1f5;">]): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">U</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>It's not beautiful but I don't consider this as an issue for now. Once everything compiles, I will take time to refine it and to reduce the surface of this abstraction.</p>
<h2 id="promise-on-the-way"><code>Promise</code> on the way</h2>
<p>But then I encounter some code using <code>Promise</code>, like:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span> </span><span style="color:#b48ead;">case class</span><span style="color:#ebcb8b;"> ChildDeferredContext</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#bf616a;">_</span><span>]: </span><span style="color:#ebcb8b;">Effect</span><span>](</span><span style="color:#bf616a;">promise</span><span>: </span><span style="color:#ebcb8b;">Promise</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">F</span><span>[</span><span style="color:#ebcb8b;">Vector</span><span>[</span><span style="color:#ebcb8b;">Defer</span><span>]]]]) </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">resolveDeferredResult</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">uc</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Ctx</span><span style="color:#eff1f5;">, </span><span style="color:#bf616a;">res</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">DeferredResult</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">]): </span><span style="color:#ebcb8b;">F</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Result</span><span style="color:#eff1f5;">] </span><span>= </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> promise.success(res.deferred)
</span><span style="color:#eff1f5;"> res.futureValue
</span><span style="color:#eff1f5;"> }
</span></code></pre>
<p><code>Promise</code> is a <a href="https://www.scala-lang.org/api/current/scala/concurrent/Promise.html">standard Scala class</a> that allows to build a <code>Future</code> based on side effects.
It's low level, and easy to make mistakes when using it.</p>
<p>I don't know why Oleg has decided to use it. I guess it was for performance optimizations to avoid mapping over <code>Future</code> everywhere. Something to check later.</p>
<p>But it's quite a shock. I was not expecting that.</p>
<p>This makes an abstraction over <code>Future</code> much more complicated now.</p>
<h2 id="what-to-do-now">What to do now</h2>
<p>Encountering <code>Promise</code> is a new challenge.</p>
<p>I see two options from here:</p>
<h3 id="option-1-abstracting-over-promise">option 1 - abstracting over <code>Promise</code></h3>
<p>The first option is to extend the abstraction to also abstract over <code>Promise</code>.</p>
<p>Frankly I doubt this will lead to somewhere. The resulting code might be quite complex. And it might be much more difficult to provide alternative implementations for those abstractions.</p>
<h3 id="option-2-removing-promise">option 2 - removing <code>Promise</code></h3>
<p>The second option is to remove the usage of <code>Promise</code>. For that I first need to understand why it was used.</p>
<p>Removing <code>Promise</code> could lead to more functional code at the end. But I'm afraid it is a big change.</p>
<h1 id="sangria-1-0-yann">Sangria 1 - 0 Yann</h1>
<p>So I've tried to abstract over <code>Future</code> for sangria, but it was more complex than expected.</p>
<p>Time to stop this first round.</p>
<p>If you have some suggestions how to deal with <code>Promise</code>, I'm all hear on <a href="https://twitter.com/simon_yann">Twitter</a> or <a href="https://mastodon.partipirate.org/@yanns">Mastodon</a>.</p>
GraphQL Meetup (London, Berlin, Munich)2020-11-06T00:00:00+00:002020-11-06T00:00:00+00:00https://yanns.github.io/blog/2020/06/11/graphql-meetup/<p>I am in a lucky situation: I can experience how my company is using GraphQL, started as an experiment in 2015 until supporting it a public API.</p>
<p>I was invited at a <a href="https://www.meetup.com/graphql-berlin/events/274229061">GraphQL Meetup</a> to speak about this experience.</p>
<p>If you are interested, you can have a look at the <a href="https://www.slideshare.net/yann_s/bringing-a-public-graphql-api-from-beta-to-production-ready-239122223">slides</a> or even <a href="https://www.youtube.com/watch?v=Y4aP-ryulgQ&t=489s">watch the talk online</a>:</p>
<p><a href="https://www.youtube.com/watch?v=Y4aP-ryulgQ&t=489s"><p><img src="/assets/2020-11-05/GraphQL_meetup_yann_simon.png" alt="Online GraphQL Meetup"/></p>
</a></p>
<p>There was also a <a href="https://www.youtube.com/watch?v=Y4aP-ryulgQ&t=5158s">panel discussion about "GraphQL in Microservices"</a>.</p>
<p><a href="https://www.youtube.com/watch?v=Y4aP-ryulgQ&t=5158s"><p><img src="/assets/2020-11-05/Graphql_meetup_panel_microservices.png" alt="Online GraphQL Meetup"/></p>
</a></p>
Open source and sole maintainers2020-02-11T00:00:00+00:002020-02-11T00:00:00+00:00https://yanns.github.io/blog/2020/02/11/open-source-and-sole-maintainers/<h1 id="open-source-and-sole-maintainers">Open source and sole maintainers</h1>
<p>Last year I was in the very sad situation of <a href="https://github.com/sangria-graphql/sangria/issues/445">losing one colleague and friend, Oleg</a>.</p>
<p>He was the creator and the maintainer of some open source libraries.</p>
<p>Among them the famous <a href="https://sangria-graphql.org/">Sangria</a>, a scala implementation of GraphQL. This library is used by major companies, such as Twitter or the New York Times.</p>
<p>Over the years Oleg has put a lot of time and energy into building and maintaining Sangria. He also gave many talks about it at <a href="https://www.youtube.com/watch?v=ymILgZAdfnA">different conferences</a>.</p>
<p>After his disparition, we, his colleagues, started investigating <a href="https://github.com/sangria-graphql/sangria/issues/446">how to proceed with Sangria in the future</a>.</p>
<p>It become apparent early that the bureaucracy involved will be very painful process. Partly due to the fact that Oleg was the only one who granted himself access to almost all his repositories hosted on Github.</p>
<h2 id="dealing-with-github">Dealing with Github</h2>
<p>The general procedure of dealing with GitHub during this process of transitioning the ownership is very fuzzy. It involves a lot of paperwork. At each subsequent step, we were asked for more documents than during the previous one.</p>
<p>I will spare us all the details as I would like to focus on the current status. By now the process has been ongoing for several months. Sadly, we still do not have transitioned ownership rights of Sangria.</p>
<h2 id="the-temporary-solution">The temporary solution</h2>
<p>As a temporary solution, we <a href="https://github.com/sangria-graphql/sangria/issues/446#issuecomment-546281588">set up another Github organization</a>, and we had to fork all repositories.</p>
<p>For this I would like to thank <a href="https://twitter.com/travisbrown">Travis Brown</a> who jumped in and set up the initial infrastructure.</p>
<h2 id="dealing-with-sonatype">Dealing with Sonatype</h2>
<p>Also Sonatype was very supportive. They were responsive and helped by granting us the <a href="https://issues.sonatype.org/browse/OSSRH-48782">rights to publish the artifacts of Sangria</a>. With these permissions, we were able to <a href="https://github.com/sangria-graphql-org/sangria/releases">release new versions</a>.</p>
<p>Thanks again Sonatype for the support!</p>
<h1 id="why-am-i-telling-you-that">Why am I telling you that</h1>
<p>The point of my writings is not to blame Github! Any company follows processes in place and struggles with cases like this. If it was not Github today, it would have been another company we would be involved with.</p>
<p>What I would however like to emphasize is how complex and time consuming it is to work with open source library when the maintainer has disappeared.</p>
<p>Oleg’s tragic destiny is not the only cause why one cannot or do not want to maintain an open source library anymore. Many other valid reasons exist.</p>
<p>However, if we would have prepared ourselves by setting up a few things while Oleg was still alive, the remaining steps needed would have caused far less problems and friction for all of us.</p>
<h1 id="about-me-being-a-single-point-of-failure">About me being a single point of failure</h1>
<p>Having said all that, <a href="https://github.com/leanovate/play-mockws/issues/66">I looked for another maintainer for a modest library of mine</a> and found <a href="https://github.com/avdv">someone</a> who stepped up for the library by reviewing and merging pull requests.</p>
<p>Additionally, we also made sure that he can publish artifacts. As a result he took care of <a href="https://github.com/leanovate/play-mockws/releases/tag/v2.8.0">publishing the last release</a>.</p>
<p>@Claudio: thank you for jumping in. I am now feeling relieved by not being alone on this project anymore.</p>
<h1 id="avoid-being-a-single-point-of-failure">Avoid being a single point of failure</h1>
<p>My main message here is: try to make sure you are not a single point of failure for your projects. Think of others using your libraries and their expectations and dependencies towards it. Onboard other maintainers early if you can and delegate some of your reponsibilities.</p>
<p>Admittedly administrative work is needed to grant access to a new maintainer and align on the resulting expectations. Frankly, it is not a very motivating task.</p>
<p>Still, please go through it and do it.</p>
<p>It will make everyone’s life much easier.</p>
<hr />
<p><em>Many thanks to <a href="https://twitter.com/tdeekens">Tobi</a> for the review</em></p>
Supporting heterogeneous types in Scala2019-10-18T00:00:00+00:002019-10-18T00:00:00+00:00https://yanns.github.io/blog/2019/10/18/heterogeneous-types-scala/<h1 id="the-objective">The objective</h1>
<p>We want to create a function that takes a list of scala classes as input, and outputs a list of all elements, each element being encoded as json.</p>
<p>For encoding in json, we use <a href="https://circe.github.io/">circe</a>.</p>
<p>This function should be generic and works for any type we can find a <a href="https://circe.github.io/circe/codec.html">circe</a> <code>Encoder</code> for it.</p>
<p>Our first try could be something like the following function:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> io.circe.{Encoder, Json}
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encode</span><span>[</span><span style="color:#ebcb8b;">A</span><span>: </span><span style="color:#ebcb8b;">Encoder</span><span>](</span><span style="color:#bf616a;">l</span><span>: </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">A</span><span>]): </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">Json</span><span>] =
</span><span> l.map(Encoder[</span><span style="color:#ebcb8b;">A</span><span>].apply)
</span></code></pre>
<p>This <code>encode</code> function seems to work:</p>
<ul>
<li>to encode several Strings:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List("</span><span style="color:#a3be8c;">hello</span><span>", "</span><span style="color:#a3be8c;">world</span><span>"))
</span><span style="color:#65737e;">// List[io.circe.Json] = List("hello", "world")
</span></code></pre>
<ul>
<li>to encode several integers:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List(</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">2</span><span>, </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, 2, 3)
</span></code></pre>
<p>But this function cannot be used with a list of heterogeneous types:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List(</span><span style="color:#d08770;">1</span><span>, "</span><span style="color:#a3be8c;">hello</span><span>", </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">/// error: diverging implicit expansion for type io.circe.Encoder[Any]
</span></code></pre>
<p>Indeed the scala compiler first resolves the type parameter <code>A</code> by finding a common type for <code>String</code> and <code>Int</code>. In this scala version, it chooses <code>Any</code>.</p>
<p>Then the scala compiler tries to find an instance of <code>Encoder[Any]</code> and fails to do so.</p>
<iframe height="280px" frameborder="0" style="width: 100%" src="https://embed.scalafiddle.io/embed?sfid=lmeb8C8/1&theme=dark&layout=h74"></iframe>
<h1 id="first-solution-replacing-type-parameters-with-path-dependent-types">First solution: replacing type parameters with path dependent types</h1>
<h2 id="introducing-toencode-with-a-path-dependent-type-to-capture-the-type-of-each-element">Introducing <code>ToEncode</code> with a path dependent type to capture the type of each element</h2>
<p>One solution is to force the scala compiler to capture the <code>Encoder</code> instance for each element, instead of using one generic <code>Encoder</code> for the whole list.</p>
<p>For that, we introduce a trait to capture the value and the <code>Encoder</code> instance:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">trait</span><span style="color:#ebcb8b;"> ToEncode </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">type </span><span style="color:#eff1f5;">Value
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">value</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Value
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encoder</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Encoder</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Value</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>Our <code>encode</code> function does not have any type parameter anymore as the type of each element is already captured in the instance of <code>ToEncode</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encode</span><span>(</span><span style="color:#bf616a;">l</span><span>: </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">ToEncode</span><span>]): </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">Json</span><span>] =
</span><span> l.map(</span><span style="color:#bf616a;">v </span><span style="color:#b48ead;">=></span><span> v.encoder.apply(v.value))
</span></code></pre>
<p>Let's define some helper functions to create <code>ToEncode</code> instances:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">stringToEncode</span><span>(</span><span style="color:#bf616a;">v</span><span>: </span><span style="color:#ebcb8b;">String</span><span>)(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">instance</span><span>: </span><span style="color:#ebcb8b;">Encoder</span><span>[</span><span style="color:#ebcb8b;">String</span><span>]): </span><span style="color:#ebcb8b;">ToEncode </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">ToEncode </span><span>{
</span><span> </span><span style="color:#b48ead;">type </span><span>Value = </span><span style="color:#ebcb8b;">String
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">value </span><span>= v
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encoder </span><span>= instance
</span><span>}
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">intToEncode</span><span>(</span><span style="color:#bf616a;">v</span><span>: </span><span style="color:#b48ead;">Int</span><span>)(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">instance</span><span>: </span><span style="color:#ebcb8b;">Encoder</span><span>[</span><span style="color:#b48ead;">Int</span><span>]): </span><span style="color:#ebcb8b;">ToEncode </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">ToEncode </span><span>{
</span><span> </span><span style="color:#b48ead;">type </span><span>Value = </span><span style="color:#b48ead;">Int
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">value </span><span>= v
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encoder </span><span>= instance
</span><span>}
</span></code></pre>
<p>Now we can use <code>encode</code> with homogeneous or heterogeneous types:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List(stringToEncode("</span><span style="color:#a3be8c;">hello</span><span>"), stringToEncode("</span><span style="color:#a3be8c;">world</span><span>")))
</span><span style="color:#65737e;">// List[io.circe.Json] = List("hello", "world")
</span><span>
</span><span>encode(List(intToEncode(</span><span style="color:#d08770;">1</span><span>), intToEncode(</span><span style="color:#d08770;">2</span><span>), intToEncode(</span><span style="color:#d08770;">3</span><span>)))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, 2, 3)
</span><span>
</span><span>encode(List(intToEncode(</span><span style="color:#d08770;">1</span><span>), stringToEncode("</span><span style="color:#a3be8c;">hello</span><span>"), intToEncode(</span><span style="color:#d08770;">3</span><span>)))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, "hello", 3)
</span></code></pre>
<p>By making our helpers <code>implicit</code>, we can avoid some boilerplate:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List("</span><span style="color:#a3be8c;">hello</span><span>", "</span><span style="color:#a3be8c;">world</span><span>"))
</span><span style="color:#65737e;">// List[io.circe.Json] = List("hello", "world")
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">2</span><span>, </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, 2, 3)
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, "</span><span style="color:#a3be8c;">hello</span><span>", </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, "hello", 3)
</span></code></pre>
<iframe height="625px" frameborder="0" style="width: 100%" src="https://embed.scalafiddle.io/embed?sfid=jfGglKO/1&theme=dark&layout=h74"></iframe>
<h2 id="generic-solution">Generic solution</h2>
<p>Instead of defining helper functions like <code>stringToEncode</code> or <code>intToEncode</code> for each type we need, we can also have a generic solution by building a <code>ToEncode</code> whenever we find an <code>Encoder</code> instance:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">trait</span><span style="color:#ebcb8b;"> ToEncode </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">type </span><span style="color:#eff1f5;">Value
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">value</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Value
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encoder</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Encoder</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">Value</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;">}
</span><span>
</span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> ToEncode </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">implicit def </span><span style="color:#8fa1b3;">fromEncoder</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span>: </span><span style="color:#ebcb8b;">Encoder</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">a</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">ToEncode </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">ToEncode </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">type </span><span style="color:#eff1f5;">Value </span><span>= </span><span style="color:#ebcb8b;">A
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">value </span><span>=</span><span style="color:#eff1f5;"> a
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encoder </span><span>= </span><span style="color:#eff1f5;">Encoder[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">}
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encode</span><span>(</span><span style="color:#bf616a;">l</span><span>: </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">ToEncode</span><span>]): </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">Json</span><span>] =
</span><span> l.map(</span><span style="color:#bf616a;">v </span><span style="color:#b48ead;">=></span><span> v.encoder.apply(v.value))
</span></code></pre>
<p>Now we can use our <code>encode</code> function with any kind of type that has an <code>Encoder</code> instance.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>encode(List("</span><span style="color:#a3be8c;">hello</span><span>", "</span><span style="color:#a3be8c;">world</span><span>"))
</span><span style="color:#65737e;">// List[io.circe.Json] = List("hello", "world")
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">2</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, 2)
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, "</span><span style="color:#a3be8c;">hello</span><span>", </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, "hello", 3)
</span></code></pre>
<iframe height="550px" frameborder="0" style="width: 100%" src="https://embed.scalafiddle.io/embed?sfid=IMTlBrw/2&theme=dark&layout=h74"></iframe>
<h2 id="disadvantage">Disadvantage</h2>
<p>To use this approach, we need one instance of <code>ToEncode</code> for each element in the list.</p>
<p>Calling <code>encode</code> with a list of 500 strings will create 500 instances of <code>ToEncode</code>!</p>
<h1 id="second-solution-using-value-classes-to-encode-each-element">Second solution: using value classes to encode each element</h1>
<p>After great feedback from <a href="https://twitter.com/travisbrown/status/1186062469850112002">Travis</a>, I could get rid of the <code>ToEncode</code> instances.</p>
<p>Let's introduce a value class that will contain the <code>Json</code> instance for each element:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> AsJson</span><span>(</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">json</span><span>: </span><span style="color:#ebcb8b;">Json</span><span>) </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">AnyVal
</span></code></pre>
<p>We use a value class to (hopefully) avoid runtime instances. A version with Dotty could use phantom type.</p>
<p>Instead of encoding each value in the <code>encode</code> function, we encode directly when converting each element to a <code>AsJson</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> AsJson </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">implicit def </span><span style="color:#8fa1b3;">toAsJson</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span>: </span><span style="color:#ebcb8b;">Encoder</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">a</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">AsJson </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">AsJson</span><span style="color:#eff1f5;">(Encoder[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">].apply(a))
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>The <code>encode</code> function has almost nothing to do except forcing the compiler to encode each element:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">encode</span><span>(</span><span style="color:#bf616a;">l</span><span>: </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">AsJson</span><span>]): </span><span style="color:#ebcb8b;">List</span><span>[</span><span style="color:#ebcb8b;">Json</span><span>] =
</span><span> l.map(</span><span style="color:#bf616a;">_</span><span>.json)
</span></code></pre>
<p>By making <code>AsJson.toAsJson</code> visible in the scope, we can now use our <code>encode</code> function with heterogeneous types:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> AsJson.</span><span style="color:#bf616a;">_
</span><span>
</span><span>encode(List("</span><span style="color:#a3be8c;">hello</span><span>", "</span><span style="color:#a3be8c;">world</span><span>"))
</span><span style="color:#65737e;">// List[io.circe.Json] = List("hello", "world")
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">2</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, 2)
</span><span>
</span><span>encode(List(</span><span style="color:#d08770;">1</span><span>, "</span><span style="color:#a3be8c;">hello</span><span>", </span><span style="color:#d08770;">3</span><span>))
</span><span style="color:#65737e;">// List[io.circe.Json] = List(1, "hello", 3)
</span></code></pre>
<iframe height="400px" frameborder="0" style="width: 100%" src="https://embed.scalafiddle.io/embed?sfid=XcTtE5u/1&theme=dark&layout=h74"></iframe>
<h1 id="feedback-welcome">Feedback welcome</h1>
<p>If you know a better solution, please do not hesitate to tell <a href="https://twitter.com/simon_yann/status/1185276156561506304">me</a>.</p>
Rust in 20182018-01-14T00:00:00+00:002018-01-14T00:00:00+00:00https://yanns.github.io/blog/2018/01/14/rust-in-2018/<p>This is my contribution to the <a href="https://blog.rust-lang.org/2018/01/03/new-years-rust-a-call-for-community-blogposts.html">#Rust2018 blog posts</a>.</p>
<h1 id="why-i-started-with-rust">Why I started with Rust</h1>
<p>My daily job is to implement scalable backends in Scala.</p>
<p>I very like Functional Programming and async IO.</p>
<p>But I have some pain point in Scala/JVM and I was very interested to see how Rust could fit here:</p>
<h2 id="fp-in-scala-is-not-gc-friendly">FP in Scala is not GC-friendly</h2>
<p>FP in Scala tends to create a lot of small living objects, putting pressure on the Garbage Collector.</p>
<p>For very hot code, I have to re-implement some FP scala code into imperative style for better performances, and I very dislike that.</p>
<p>Rust on the other hand provide abstractions without overhead (or very little).</p>
<h2 id="java-programs-tend-to-be-very-fat">Java programs tend to be very "fat"</h2>
<p>A Rust program can be statically compiled and directly packaged as a <a href="https://github.com/yanns/zeugnis/blob/master/Dockerfile">from scratch docker container</a>, very lightweight compared to java applications that needs a distribution and a JVM.</p>
<h1 id="rust-in-2017">Rust in 2017</h1>
<p>In 2017, I tried different programs in Rust, from a <a href="https://github.com/sphereio/sphere-hello-api/tree/master/rust">minimal SDK</a> that I tried to move to <a href="https://github.com/sphereio/sphere-hello-api/tree/async_hyper/rust">async IO</a>, to a GUI program. I tried a lot of different HTTP servers.</p>
<p>And I contributed to an <a href="https://github.com/hjr3/weldr">HTTP proxy</a> based on Tokio and Futures.</p>
<h1 id="rust-in-2018">Rust in 2018</h1>
<p>My wishes for Rust in 2018 are mostly based on the pain I had with Rust in 2017:</p>
<h2 id="easier-futures">Easier Futures</h2>
<p>My experience with Futures was... very painful.</p>
<p>I had a lot of strange compiler errors, especially with lifetimes.</p>
<p>Sometimes, to fix then, I just had to move one variable from one closure to the upper one, something that makes no sense from the implement logic point of view.</p>
<p>I have a lot of difficulties with the static lifetime of Futures.</p>
<p>I hope that this will improve. Maybe non lexical lifetimes will help here.</p>
<ul>
<li><code>impl Trait</code> feature</li>
</ul>
<p>And I'll also follow the <a href="https://github.com/alexcrichton/futures-await">async/await coroutines</a>.</p>
<h2 id="a-standard-http-server-for-backends">a "standard" HTTP server for backends</h2>
<p>I'd need a good asynchronous http server ready for production usage, with a good community.
There should exist good libraries that can integrate easily with this "standard" HTTP server, like for logs and metrics.</p>
Rust closures as input parameters2016-05-23T00:00:00+00:002016-05-23T00:00:00+00:00https://yanns.github.io/blog/2016/05/23/rust-closures-as-input-parameters/<p><em>Edit:</em> <a href="https://yanns.github.io/blog/2016/05/23/rust-closures-as-input-parameters/#update-2016-05-25"><em>Update (May 25)</em></a></p>
<p>I am learning <a href="https://www.rust-lang.org/">Rust</a> and, as a beginner, I have sometimes problems achieving some little tasks that would be so easy in other programming languages I know better.</p>
<p>But when I met some Rust developers and they ask me about my difficulties, I often forget about them.</p>
<p>I therefore decided to write about my difficulties in Rust to keep track of them.</p>
<p>So today about closures.</p>
<p>When reading a <a href="http://fredrik.anderzon.se/2016/05/10/rust-for-node-developers-part-1-introduction/">blog post introducing Rust for Node.js Developers</a> I made the same Todo application.</p>
<p>At the end, the program contains such piece of code:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">remove_todo</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(todo) = todos.</span><span style="color:#96b5b4;">iter_mut</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">todo</span><span>| todo.id == todo_id) {
</span><span> todo.deleted = </span><span style="color:#d08770;">true</span><span>;
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">mark_done</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(todo) = todos.</span><span style="color:#96b5b4;">iter_mut</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">todo</span><span>| todo.id == todo_id) {
</span><span> todo.completed = </span><span style="color:#d08770;">true</span><span>;
</span><span> }
</span><span>}
</span></code></pre>
<p>The first function marks a Todo as deleted if it can be found by its ID in the vector.
The second function marks a Todo as completed, also if it can be found in the vector of Todos.</p>
<p>Some code is duplicated and I decided to refactor the common code in a third function, that would do <em>something</em> on a Todo if found in a vector.</p>
<p>This third function would take a closure as input parameter, like in pseudo-code:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">with_todo_id</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>, </span><span style="color:#bf616a;">f</span><span>: <closure </span><span style="background-color:#bf616a;color:#2b303b;">-</span><span> do something on a Todo>) {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(todo) = todos.</span><span style="color:#96b5b4;">iter_mut</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">todo</span><span>| todo.id == todo_id) {
</span><span> </span><span style="color:#96b5b4;">f</span><span>(todo);
</span><span> }
</span><span>}
</span></code></pre>
<p>so that the 2 initial functions are simplified like that:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">remove_todo</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.deleted = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">mark_done</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.completed = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span></code></pre>
<p>This closure is a side-effect on a Todo. It should accept a mutable Todo as parameter and return nothing.</p>
<p>One <a href="http://rustbyexample.com/fn/closures/input_parameters.html">source of documentation for closures as input parameters</a> mentions that there exist 3 kinds of closures:</p>
<ul>
<li>Fn: takes captures by reference (<code>&T</code>)</li>
<li>FnMut: takes captures by mutable reference (<code>&mut T</code>)</li>
<li>FnOnce: takes captures by value (<code>T</code>)</li>
</ul>
<p>This is a lot of information for a new developer.</p>
<p>I tried different possibilities, like:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">with_todo_id</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>, </span><span style="color:#bf616a;">f</span><span>: &Fn(&</span><span style="color:#bf616a;">mut Todo</span><span>)) {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(todo) = todos.</span><span style="color:#96b5b4;">iter_mut</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">todo</span><span>| todo.id == todo_id) {
</span><span> </span><span style="color:#96b5b4;">f</span><span>(todo);
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">remove_todo</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.deleted = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">mark_done</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.completed = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span></code></pre>
<p>Without any success:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">$</span><span> cargo run
</span><span> </span><span style="color:#bf616a;">Compiling</span><span> todo-list v0.1.0 (file:///Users/yannsimon/projects/rust/rust-playground/todo-list)
</span><span style="color:#bf616a;">src/main.rs:27:38:</span><span> 27:50 error: the type of this value must be known in this context
</span><span style="color:#bf616a;">src/main.rs:27</span><span> with_todo_id(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| </span><span style="color:#bf616a;">todo.deleted</span><span> = true);
</span><span> </span><span style="color:#bf616a;">^~~~~~~~~~~~
</span></code></pre>
<p>The <a href="https://doc.rust-lang.org/book/closures.html#taking-closures-as-arguments">official documentation</a> was not so much help neither.</p>
<p>I asked for help on <a href="https://client00.chat.mibbit.com/?server=irc.mozilla.org&channel=%23rust-beginners">#rust-beginners</a>.
People on this channel are <strong>very</strong> helpful and kind. The community of Rust is awesome!</p>
<p>I was proposed 2 solutions. Both work, and I choose that one:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">with_todo_id</span><span><P>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>, </span><span style="color:#bf616a;">f</span><span>: P) </span><span style="color:#b48ead;">where</span><span> P: Fn(&</span><span style="color:#b48ead;">mut</span><span> Todo) {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(todo) = todos.</span><span style="color:#96b5b4;">iter_mut</span><span>().</span><span style="color:#96b5b4;">find</span><span>(|</span><span style="color:#bf616a;">todo</span><span>| todo.id == todo_id) {
</span><span> </span><span style="color:#96b5b4;">f</span><span>(todo);
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">remove_todo</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.deleted = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">mark_done</span><span>(</span><span style="color:#bf616a;">todos</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Vec<Todo>, </span><span style="color:#bf616a;">todo_id</span><span>: </span><span style="color:#b48ead;">i16</span><span>) {
</span><span> </span><span style="color:#96b5b4;">with_todo_id</span><span>(todos, todo_id, |</span><span style="color:#bf616a;">todo</span><span>| todo.completed = </span><span style="color:#d08770;">true</span><span>);
</span><span>}
</span></code></pre>
<p>Compared to my previous attempt, the <code>f: &Fn(&mut Todo)</code> is replaced by <code>f: P where P: Fn(&mut Todo)</code>.</p>
<p>I still do not completely understand why this works and not the previous version. I was explained Rust can use the reference to the closure... I will continue reading documentation about it.... ;)</p>
<p>If you have any good source for this, please <a href="https://twitter.com/simon_yann">tell me</a>.</p>
<p>In conclusion I still find closure as input parameters quite complex in Rust. I surely need to more understand the theory behind the language to fully understand them.</p>
<p>The Rust community is very helpful, but it may not scale if there are more and more beginners like me.</p>
<h4 id="update-2016-05-25"><a href="https://yanns.github.io/blog/2016/05/23/rust-closures-as-input-parameters/#update-2016-05-25"><em>Update (May 25)</em></a></h4>
<p>The following <a href="https://twitter.com/rustlang/status/734946700536774656">tweet</a> from <a href="https://twitter.com/rustlang">@rustlang</a> provided me the good keywords to search for:</p>
<blockquote>
<p>it's about trait objects vs type parameters, which can be tough when you're learning</p>
<p>— <a href="https://twitter.com/rustlang/status/734946700536774656">@rustlang</a></p>
</blockquote>
<p><a href="https://doc.rust-lang.org/book/trait-objects.html">Trait objects</a> are used for <a href="https://en.wikipedia.org/wiki/Dynamic_dispatch">dynamic dispatch</a>, feature found in most OO languages.</p>
<p>With that in mind, I could understand the <a href="https://doc.rust-lang.org/book/closures.html#taking-closures-as-arguments">Rust book about closures</a>.</p>
<p>If I use trait objects, this version works:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>fn with_todo_id(todos: &mut Vec<Todo>, todo_id: i16, f: &Fn(&mut Todo)) {
</span><span> if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
</span><span> f(todo);
</span><span> }
</span><span>}
</span><span>
</span><span>fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
</span><span> with_todo_id(todos, todo_id, &|todo| todo.deleted = true);
</span><span>}
</span><span>
</span><span>fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
</span><span> with_todo_id(todos, todo_id, &|todo| todo.completed = true);
</span><span>}
</span></code></pre>
<p>Trait objects force Rust to use dynamic dispatch.</p>
<p>If I use type parameter instead of a trait object:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>fn with_todo_id<P>(todos: &mut Vec<Todo>, todo_id: i16, f: P) where P: Fn(&mut Todo) {
</span><span> if let Some(todo) = todos.iter_mut().find(|todo| todo.id == todo_id) {
</span><span> f(todo);
</span><span> }
</span><span>}
</span><span>
</span><span>fn remove_todo(todos: &mut Vec<Todo>, todo_id: i16) {
</span><span> with_todo_id(todos, todo_id, |todo| todo.deleted = true);
</span><span>}
</span><span>
</span><span>fn mark_done(todos: &mut Vec<Todo>, todo_id: i16) {
</span><span> with_todo_id(todos, todo_id, |todo| todo.completed = true);
</span><span>}
</span></code></pre>
<p>then Rust is able to monomorphize the closure and use static dispatch, and does not need any object for the dyamic dispatch.</p>
<p>Another great example of the <a href="http://blog.rust-lang.org/2015/05/11/traits.html">zero-cost abstraction</a> possible with Rust!</p>
GraphQL Meetup in Berlin2016-05-19T00:00:00+00:002016-05-19T00:00:00+00:00https://yanns.github.io/blog/2016/05/19/graphql-meetup-in-berlin/<p>Honeypot organized the first meetup about <a href="http://graphql.org/">GraphQL</a> in Berlin and I had the pleasure to be invited to talk about my experience.</p>
<p>My topic was <a href="http://www.slideshare.net/yann_s/performance-optimisation-with-graphql">Performance optimisation with GraphQL</a>.</p>
<p>You can find the <a href="http://blog.honeypot.io/honeypot-tech-meetup-graphql/">slides for the other talks here</a>.</p>
<p><img src="/assets/graphql-meetup-1.jpg" alt="Lots of attendees"/></p>
<p><img src="/assets/graphql-meetup-2.jpg" alt="Performance gain with GraphQL field selection"/></p>
migrate a playframework app from 2.3 to 2.52016-04-26T00:00:00+00:002016-04-26T00:00:00+00:00https://yanns.github.io/blog/2016/04/26/migrate-a-playframework-app-from-2-dot-3-to-2-dot-5/<p>Recently I migrated several <a href="https://www.playframework.com/">play</a> applications from the version 2.3 to the version 2.5.</p>
<p>As this experience may interest other people, I decided to write about it.</p>
<p>So let's start!</p>
<h1 id="migration-from-2-3-to-2-4">Migration from 2.3 to 2.4</h1>
<p>First, we will migrate the application from the version 2.3 to the version 2.4.</p>
<p>For that, the <a href="https://www.playframework.com/documentation/latest/Migration24">migration guide</a> will be our reference.</p>
<p>How do I proceed?</p>
<h3 id="1-update-the-sbt-config-to-play-2-4">1. Update the sbt config to play 2.4</h3>
<p>First, I update the SBT configuration:</p>
<ul>
<li>update the play version in <code>project/plugins.sbt</code>:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>-addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.10")
</span><span>+addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.4.6")
</span></code></pre>
<ul>
<li>if needed, update the sbt version in <code>project/build.properties</code>:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sbt.version=0.13.11
</span></code></pre>
<ul>
<li>update the <code>build.sbt</code>:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>-import PlayKeys._
</span><span>+import play.sbt.PlayImport._
</span></code></pre>
<p>I also had to replace <code>playVersion.value</code> with <code>play.core.PlayVersion.current</code>.</p>
<p>The goal of this first step is to be able to load the application with <code>sbt</code>, even if the application itself does not compile. At this point, Intellij can load the application with a play 2.4 version, and can provide auto-completion.</p>
<h3 id="2-fix-compilation-errors">2. Fix compilation errors</h3>
<p>Then, I fix the compilation</p>
<p>Inside sbt, I just trigger the compilation:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>~compile
</span></code></pre>
<p>and fix all compilation errors. The play team has made a very good job at documenting all steps in the <a href="https://www.playframework.com/documentation/latest/Migration24">migration guide</a>.</p>
<p>At this point, I just want the application to compile with the minimal number of changes. I do not care if I use deprecated APIs.</p>
<h3 id="3-run-the-application">3. Run the application</h3>
<p>When all compilation errors are fixed, I just make sure that the application can run:</p>
<p>In sbt:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>~run
</span></code></pre>
<p>When running the application, I sometimes discovered version conflicts between libs (ex: Netty versions) and I know if I can complete the migration or if I have to update a dependency before.</p>
<p>Analysing all dependencies can be difficult.
I use the <a href="https://github.com/jrudolph/sbt-dependency-graph">sbt-dependency-graph</a> to have an overview of all dependencies and find overriding versions.</p>
<h3 id="4-fix-the-tests">4. Fix the tests</h3>
<p>Then I fix all compilation errors in the tests, and then check that the tests are successful.</p>
<p>In sbt:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>~testQuick
</span></code></pre>
<p>When all the tests are green, the application is migrated.
The migration to play 2.4 is now completed, profit from the new version! ;)</p>
<h3 id="5-remove-all-usages-of-deprecated-api">5. Remove all usages of deprecated API</h3>
<p>Now it is time to clean our application and to fix all code using a deprecated API.</p>
<p>This clean up can be done step by step.</p>
<p>I use the Scala API, and I prefer not to use any runtime dependency injection framework.</p>
<p>I introduced the so-called <a href="https://www.playframework.com/documentation/2.5.x/ScalaCompileTimeDependencyInjection"><em>compile time dependency injection</em></a> to simply instantiate my controllers in one application loader.</p>
<p>For more info about this topic, please refer to:</p>
<ul>
<li><a href="http://de.slideshare.net/yann_s/play-24dimacwire">my talk about it</a> at the <a href="/blog/2015/05/20/di-with-play-2-dot-4/">play meetup</a></li>
<li><a href="http://loicdescotte.github.io/posts/play24-compile-time-di/">this great post from Loïc Descotte</a></li>
</ul>
<p>This cleanup is necessary before updating to play 2.5, as deprecated API are likely to be removed in the next version.</p>
<h1 id="migration-from-2-4-to-2-5">Migration from 2.4 to 2.5</h1>
<p>For that, the <a href="https://www.playframework.com/documentation/latest/Migration25">migration guide</a> will be our reference.</p>
<p>The migrate follow the same steps as <a href="/#migrate-from-2.3-to-2.4">the previous migration</a>:</p>
<ul>
<li>update the sbt config, but to play 2.5:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>-addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.4.6")
</span><span>+addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.2")
</span></code></pre>
<ul>
<li>fix the compilation errors, like in application loader:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>- Logger.configure(context.environment)
</span><span>+ LoggerConfigurator(context.environment.classLoader).foreach { _.configure(context.environment) }
</span></code></pre>
<ul>
<li>run the application</li>
<li>fix the tests</li>
<li>clean up the application<br>
For example, I can remove the <code>InjectedRoutesGenerator</code> as it is now the default.</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>-routesGenerator := InjectedRoutesGenerator
</span></code></pre>
trampoline execution context with scala Futures2016-02-10T00:00:00+00:002016-02-10T00:00:00+00:00https://yanns.github.io/blog/2016/02/10/trampoline-execution-context-with-scala-futures/<p>Disclaimer: I am continually learning, and this post reflects my current understanding of the topic. I may be wrong. Do not believe directly what I write. Test what you need. If you want to provide some precisions or corrections, please contact me on <a href="https://twitter.com/simon_yann">twitter</a>, and I'll fix this post.</p>
<p><a href="https://twitter.com/runarorama">Rúnar</a> showed in a <a href="http://blog.higher-order.com/blog/2015/06/18/easy-performance-wins-with-scalaz/">blog post how Scalaz Tasks can perform better than the Scala Futures</a>.</p>
<p>He explains the details very well. If you have not read that post, I recommend it highly.</p>
<p>The main point if that Scala <code>Future</code> adds a context switching for each <code>map</code> or <code>flatMap</code>.
With Scalaz <code>Task</code>, we have to describe which tasks need a new thread, the other ones are called on the same thread as the previous computation, avoiding these context switchings.</p>
<p>With Scala Futures, if you want to multiply the result of a <code>Future[Int]</code> by 2, you need an <code>ExecutionContext</code> (EC):</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> scala.concurrent.ExecutionContext.global
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">futureCount</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>] = futureCountOfUsers()
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">result </span><span>= futureCount.map(</span><span style="color:#bf616a;">i </span><span style="color:#b48ead;">=></span><span> i * </span><span style="color:#d08770;">2</span><span>)(global)
</span></code></pre>
<p>or with an implicit resolution:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> scala.concurrent.ExecutionContext.Implicits.global
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">futureCount</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>] = futureCountOfUsers()
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">result </span><span>= futureCount.map(</span><span style="color:#bf616a;">i </span><span style="color:#b48ead;">=></span><span> i * </span><span style="color:#d08770;">2</span><span>)
</span><span>
</span></code></pre>
<p>To compute the <code>i => i * 2</code>, the ExecutionContext may use a different thread than the one having the result of the <code>futureCountOfUsers</code>. We observe a context switching between the future and the callback in the <code>map</code>. And the thread executing <code>i => i * 2</code> can run on a different CPU/core than the one having the result of <code>futureCount</code>, meaning that the CPU cache is missed.</p>
<p>This overhead is not problematic for simple computations. But if we do 100 or 1000 of them, then it can have a significant impact on performances.</p>
<p>And in my opinion, Scala Futures have other downsides.</p>
<p>For example, with the following code:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">for </span><span>{
</span><span> </span><span style="color:#bf616a;">value1 </span><span><- functionThatReturnsFutureValue1
</span><span> </span><span style="color:#bf616a;">value2 </span><span><- functionThatReturnsFutureValue2
</span><span>} </span><span style="color:#b48ead;">yield </span><span>(value1, value2)
</span></code></pre>
<p><code>functionThatReturnsFutureValue1</code> and <code>functionThatReturnsFutureValue2</code> runs sequentially, even if there is no dependency between the two.</p>
<p>On the other hand:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">futureValue1 </span><span>= functionThatReturnsFutureValue1
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">futureValue2 </span><span>= functionThatReturnsFutureValue2
</span><span>
</span><span style="color:#b48ead;">for </span><span>{
</span><span> </span><span style="color:#bf616a;">value1 </span><span><- futureValue1
</span><span> </span><span style="color:#bf616a;">value2 </span><span><- futureValue2
</span><span>} </span><span style="color:#b48ead;">yield </span><span>(value1, value2)
</span></code></pre>
<p>computes <code>functionThatReturnsFutureValue1</code> and <code>functionThatReturnsFutureValue2</code> in parallel.</p>
<p>It means that Scala Futures do not respect the principe of <a href="https://en.wikipedia.org/wiki/Referential_transparency">"referential transparency"</a>.
It's not only a theoretical problem, new users of Scala Futures often have problems with that.</p>
<p>And what I do not like about Scala Future is that we always need an ExecutionContext, even for small non-blocking computations.</p>
<p>For example, instead of writing:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">multiplyBy2</span><span>(</span><span style="color:#bf616a;">f</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>]): </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>] =
</span><span> f.map(</span><span style="color:#bf616a;">_</span><span> * </span><span style="color:#d08770;">2</span><span>)
</span></code></pre>
<p>We need either to import a ExecutionContext that is always used, or leave the liberty to the caller of the function and write:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">multiplyBy2</span><span>(</span><span style="color:#bf616a;">f</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>])(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">ec</span><span>: </span><span style="color:#ebcb8b;">ExecutionContext</span><span>): </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>] =
</span><span> f.map(</span><span style="color:#bf616a;">_</span><span> * </span><span style="color:#d08770;">2</span><span>)
</span></code></pre>
<p>My first impression with Scalaz Tasks is that they have a better design than the Scala Futures.
But I have not used Scalaz Tasks extensively and cannot say if they have other problems.</p>
<p>But all in all, Scala Futures are here to stay. They are part of the standard API and are used everywhere.</p>
<p>I'm still wondering why the Scala Futures were designed that way.
I can only guess, but I think this avoids some common pitfalls:</p>
<ul>
<li>"easy" for new-comers: simply import the default execution context and that’s all</li>
<li>safe by default: If a callback takes a long time (blocking IO and expensive computation), this callback will not block other ones. The execution context will be able to use a different thread for other computations.</li>
<li>and a design like Scala Tasks works well if all parts of the system are non-blocking and using one thread pool. The reality is more complex. Each driver/http client can use its own thread pool. For example, an asynchronous http client may have its own thread pool because some parts of the networking API in Java is blocking like the standard ns lookup <code>InetAddress.getByName()</code>. Running a computation directly on the thread without forking it will run the computation of the thread pool of the http client. And that can lead to an exhaustion of the thread pool of the http client, and the http client cannot function anymore.</li>
</ul>
<h3 id="introducing-the-trampoline-execution-context">Introducing the trampoline execution context</h3>
<p>This performance problem with the standard execution context is not new. The play framework team had this problem, especially with Iteratees that compute everything with a Future and uses callbacks extensively on stream of data.
To solve this problem, James Roper introduced the <a href="https://github.com/playframework/playframework/blob/master/framework/src/iteratees/src/main/scala/play/api/libs/iteratee/Execution.scala#L31-L128">trampoline Execution Context</a>.
This trampoline execution context is a great piece of software:</p>
<ul>
<li>it makes sure the callbacks are run on the same thread than the future to avoid context switchings.</li>
<li>it does not overflow with recursive callbacks.</li>
</ul>
<p>To show the benefit of the trampoline execution context, let's call this function that does not make any sense, but simply calls <code>Future.map</code> n times:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">range</span><span>(</span><span style="color:#bf616a;">n</span><span>: </span><span style="color:#b48ead;">Int</span><span>)(</span><span style="color:#b48ead;">implicit </span><span style="color:#bf616a;">ec</span><span>: </span><span style="color:#ebcb8b;">ExecutionContext</span><span>): </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Int</span><span>] =
</span><span> (</span><span style="color:#d08770;">0</span><span> to n).foldLeft[</span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Int</span><span>]](Future.successful(</span><span style="color:#d08770;">0</span><span>)) { </span><span style="color:#b48ead;">case </span><span>(</span><span style="color:#bf616a;">f</span><span>, </span><span style="color:#bf616a;">_</span><span>) </span><span style="color:#b48ead;">⇒</span><span> f.map(</span><span style="color:#bf616a;">_</span><span> + </span><span style="color:#d08770;">1</span><span>) }
</span></code></pre>
<p>With n = 5 000 000:</p>
<ul>
<li>Scala Futures with standard EC: 0.037 ops/s</li>
<li>Scala Futures with trampoline EC: 1.397 ops/s</li>
</ul>
<h3 id="should-we-use-the-trampoline-ec">Should we use the trampoline EC?</h3>
<p>When we are confident with execution contexts, I thing we can use this trampoline EC if:</p>
<ul>
<li>the callback is running very fast. For example:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">multiplyBy2</span><span>(</span><span style="color:#bf616a;">f</span><span>: </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>]): </span><span style="color:#ebcb8b;">Future</span><span>[</span><span style="color:#b48ead;">Long</span><span>] =
</span><span> f.map(</span><span style="color:#bf616a;">_</span><span> * </span><span style="color:#d08770;">2</span><span>)(trampolineEC)
</span></code></pre>
<ul>
<li>we never call some blocking IO. This point can be tricky: some scala libs can use some java libs that use InputStream or OutputStream that can block.</li>
</ul>
<p>If you are unsure, use the standard EC.</p>
<p>If you want to try this yourself, the <a href="https://github.com/yanns/trampoline-EC">code is on github</a>.</p>
review of Essential Slick (for Slick 3.0)2015-12-07T00:00:00+00:002015-12-07T00:00:00+00:00https://yanns.github.io/blog/2015/12/07/review-of-essential-slick/<p>I had the pleasure to review the book "<a href="http://underscore.io/books/essential-slick/">Essential Slick</a>" about <a href="http://slick.typesafe.com/">Slick 3</a>.</p>
<p>I really enjoyed reading it. For anyone starting a project with Slick, I highly recommend this book, in particular to learn about:</p>
<ul>
<li>how to build queries that can be composed and re-used</li>
<li>how to structure the code</li>
<li>the patterns, anti-patterns and pitfalls to avoid.</li>
</ul>
<p>The book goes over a lot of concepts, from simple queries and data-modeling to views, projections, joins and aggregates.</p>
<p>Slick 3 has very powerful concepts that could be hard to manage. The book tries to teach us how to use them. </p>
<p>Some chapters are more demanding, for instance the 2nd chapter introduces in only a few pages concepts like Queries, Actions, Effects, Streaming.</p>
<p>But once the step managed, I had the feeling to have the good abstractions in my mind to build an application based on Slick. And I could use these concepts to solve the exercises.</p>
<p>What I really liked:</p>
<ul>
<li>exercices: I think I can only learn a library when I use it and the exercises in the book are a very good opportunity to practice.</li>
<li>the sbt console is so configured that I could copy/paste all examples from the book into the console and play with them.</li>
<li>focus on composability and re-usability</li>
<li>the book also points out when one abstraction is not optimal for a particular problem and proposes some alternatives.</li>
</ul>
<p>What I missed:</p>
<ul>
<li>the 3.0 version of Slick is a lot about asynchronity, streaming and I wish the book could dig further in these concepts.</li>
</ul>
<p>All in all, a very good book!</p>
play framework meetup in Berlin in November 20152015-11-17T00:00:00+00:002015-11-17T00:00:00+00:00https://yanns.github.io/blog/2015/11/17/introduction-to-type-classes-in-scala/<p>During the <a href="http://www.meetup.com/Play-Berlin-Brandenburg/events/226561633/">play framework meetup in Berlin in November 2015</a>:</p>
<h2 id="for-a-better-workflow-between-designers-and-developers">for a better workflow between designers and developers</h2>
<p><a href="https://twitter.com/lluizesc">Laura</a> made a very good talk about how to better organize the work between front-end designers and developers.</p>
<p>For that, her team build some tools:</p>
<ul>
<li>
<p>the designer uses an assets pipeline in nodejs to build a website. The templates are written with <a href="http://handlebarsjs.com/">handlebars</a>. When the website is ready, a release is made and all assets are packaged as a <a href="http://www.webjars.org/">webjar</a>: https://github.com/lauraluiz/handlebars-webjars-demo</p>
</li>
<li>
<p>the developer uses a play framework application that is capable of using directly the assets released by the designer: https://github.com/lauraluiz/play-handlebars-demo</p>
</li>
</ul>
<p>Slides: <a href="http://slides.com/lauraluiz/handlebarsplay">http://slides.com/lauraluiz/handlebarsplay</a></p>
<p>Project using this workflow: <a href="https://github.com/sphereio/sphere-sunrise">https://github.com/sphereio/sphere-sunrise</a></p>
<h2 id="type-classes-in-scala">type classes in Scala</h2>
<p>I introduced type classes in Scala: <a href="http://www.slideshare.net/yann_s/introduction-to-type-classes-in-scala">http://www.slideshare.net/yann_s/introduction-to-type-classes-in-scala</a></p>
<p>Step by step, I explained how the Json Reads/Writes/Formats work in the <a href="https://www.playframework.com/documentation/2.4.x/ScalaJson">scala API of the play framework</a>.</p>
DI with play 2.42015-05-20T00:00:00+00:002015-05-20T00:00:00+00:00https://yanns.github.io/blog/2015/05/20/di-with-play-2-dot-4/<p>During the <a href="http://www.meetup.com/Play-Berlin-Brandenburg/events/222130013/">play framework meetup in Mai 2015</a>, there were 3 talks about Dependency Injection (DI) in Play 2.4:</p>
<ul>
<li>
<p>from Micheal: runtime DI in Java with <a href="https://github.com/google/guice">Guice</a>, the default framework introduced in play 2.4: <a href="https://github.com/schleichardt/play-2-4-di-talk">https://github.com/schleichardt/play-2-4-di-talk</a></p>
</li>
<li>
<p>from <a href="https://twitter.com/easyangel">Oleg</a>: runtime DI in Scala with <a href="http://scaldi.org/">Scaldi</a>: <a href="http://scaldi.github.io/scaldi-play-2.4.0-presentation/">http://scaldi.github.io/scaldi-play-2.4.0-presentation/</a></p>
</li>
<li>
<p>compile-time DI in Scala with constructor arguments or <a href="https://github.com/adamw/macwire">MacWire</a>: <a href="http://de.slideshare.net/yann_s/play-24dimacwire">http://de.slideshare.net/yann_s/play-24dimacwire</a>.
For that talk, I re-used the TBA application I used at the <a href="/blog/2014/02/17/ping-conf-2014/">Ping Conf</a> last year and added a version using MacWire: https://github.com/yanns/TPA/tree/master/frontend/TBA_macwire</p>
</li>
</ul>
goto conference 20142014-11-10T00:00:00+00:002014-11-10T00:00:00+00:00https://yanns.github.io/blog/2014/11/10/goto-conference-2014/<p>My notes from the <a href="http://gotocon.com/berlin-2014">goto conference Berlin 2014</a>:</p>
<h2 id="thursday">Thursday</h2>
<h5 id="opening-keynote-software-design-in-the-21st-century-martin-fowler"><a href="http://gotocon.com/berlin-2014/presentation/Opening%20Keynote:%20Software%20Design%20in%20the%2021st%20Century">Opening Keynote: Software Design in the 21st Century</a> - <a href="https://twitter.com/martinfowler">Martin Fowler</a></h5>
<p>We, developers, take responsibility in the code we write.<br/>
We cannot simply say: "I implemented that because I was told so".<br/>
Avoid <a href="http://darkpatterns.org/">dark patterns</a>.</p>
<p>We are not code monkeys.</p>
<h5 id="aeron-the-next-generation-in-open-source-high-performance-messaging-martin-thompson"><a href="http://gotocon.com/berlin-2014/presentation/Aeron:%20The%20Next%20Generation%20in%20Open-Source%20High-Performance%20Messaging">Aeron: The Next Generation in Open-Source High-Performance Messaging</a> - <a href="https://twitter.com/mjpt777">Martin Thompson</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/MartinThompson_AeronTheNextGenerationInOpenSourceHighPerformanceMessaging.pdf">slides</a>
<a href="http://mechanical-sympathy.blogspot.com/">blog</a></p>
<p><a href="https://github.com/real-logic/Aeron">Aeron</a> is a OSI layer 4 Transport for message oriented streams.
is simply impressive, achieving a very low latency.<br/>
I'd like to see an integration of Aeron in Akka cluster.</p>
<p><a href="https://www.youtube.com/watch?v=tM4YskS94b0">similar talk</a></p>
<h5 id="writing-highly-concurrent-polyglot-applications-with-vert-x-tim-fox"><a href="http://gotocon.com/berlin-2014/presentation/Writing%20highly%20Concurrent%20Polyglot%20Applications%20with%20Vert.x">Writing highly Concurrent Polyglot Applications with Vert.x</a> - <a href="https://twitter.com/timfox">Tim Fox</a></h5>
<p>It was a good introduction to Vert.x.<br/>
But I was expecting more than an introduction.</p>
<h5 id="fast-analytics-on-big-data-petr-maj-and-tomas-nykodym"><a href="http://gotocon.com/berlin-2014/presentation/Fast%20Analytics%20on%20Big%20Data">Fast Analytics on Big Data</a> - Petr Maj and Tomas Nykodym</h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/PetrMaj_and_TomasNykodym_FastAnalyticsOnBigData.pdf">slides</a></p>
<p>Presentation of an ML runtime for bigdata.</p>
<p><a href="https://github.com/0xdata/h2o">Code</a> <a href="https://github.com/0xdata/h2o-dev">With Spark API</a></p>
<p>TODO: have a look at https://github.com/0xdata/h2o-training</p>
<h5 id="security-architecture-for-the-smarthome-jacob-fahrenkrug"><a href="http://gotocon.com/berlin-2014/presentation/Security%20Architecture%20for%20the%20SmartHome">Security Architecture for the SmartHome</a> - <a href="https://twitter.com/jacobfahrenkrug">Jacob Fahrenkrug</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/JacobFahrenkrug_SecurityArchitectureForTheSmartHome.pdf">slides</a></p>
<p>A good presentation about the future threats of connected objects and the solution <a href="http://www.yetu.com/">yetu</a> implements.</p>
<h5 id="graph-all-the-things-graph-database-use-cases-that-aren-t-social-emil-eifrem"><a href="http://gotocon.com/berlin-2014/presentation/Graph%20All%20The%20Things!!%20Graph%20Database%20Use%20Cases%20That%20Aren't%20Social">Graph All The Things!! Graph Database Use Cases That Aren't Social</a> - <a href="https://twitter.com/emileifrem">Emil Eifrem</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/EmilEifrem_GraphAllTheThingsGraphDatabaseUseCasesThatArentSocial.pdf">slides</a></p>
<p>very good presentation of <a href="http://neo4j.com/">Neo4j</a>.<br/>
We should use a graph database where it make sense - the search queries are much easier to write / read, and the performance very good.</p>
<h5 id="party-keynote-staying-ahead-of-the-curve-trisha-gee"><a href="http://gotocon.com/berlin-2014/presentation/Party%20Keynote:%20Staying%20Ahead%20of%20the%20Curve">Party Keynote: Staying Ahead of the Curve</a> - <a href="https://twitter.com/trisha_gee">Trisha Gee</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/TrishaGee_PartyKeynoteStayingAheadOfTheCurve.pdf">slides</a></p>
<h2 id="friday">Friday</h2>
<h5 id="morning-keynote-excellence-culture-humane-keeping-of-techies-prof-dr-gunter-dueck"><a href="http://gotocon.com/berlin-2014/presentation/Morning%20Keynote:%20Excellence%20Culture%20&%20Humane%20Keeping%20of%20Techies">Morning Keynote: Excellence Culture & Humane Keeping of Techies</a> - <a href="https://twitter.com/wilddueck">Prof. Dr. Gunter Dueck</a></h5>
<p>a very good keynote, very funny and true at the same time.</p>
<h5 id="aerospike-flash-optimized-high-performance-nosql-database-for-all-khosrow-afroozeh"><a href="http://gotocon.com/berlin-2014/presentation/Aerospike:%20Flash-optimized,%20High-Performance%20noSQL%20database%20for%20All">Aerospike: Flash-optimized, High-Performance noSQL database for All</a> - <a href="https://twitter.com/khaf">Khosrow Afroozeh</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/KhosrowAfroozeh_AerospikeFlashOptimizedHighPerformanceNoSQLDatabaseForAll.pdf">slides</a></p>
<p>A new database on my radar - impressive.</p>
<h5 id="docker-a-lot-changed-in-a-year-chris-swan"><a href="http://gotocon.com/berlin-2014/presentation/Docker%20-%20A%20Lot%20Changed%20in%20a%20Year">Docker - A Lot Changed in a Year</a> - <a href="http://twitter.com/cpswan">Chris Swan</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/ChrisSwan_DockerALotChangedInAYear.pdf">slides</a></p>
<p>Docker is maturating.<br/>
Just do not forget that "containers do not contain" -> no complete security between container and host, and between containers.</p>
<h5 id="the-joys-and-perils-of-interactive-development-stuart-sierra"><a href="http://gotocon.com/berlin-2014/presentation/The%20Joys%20and%20Perils%20of%20Interactive%20Development">The Joys and Perils of Interactive Development</a> - <a href="https://twitter.com/stuartsierra">Stuart Sierra</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/protected/StuartSierra_TheJoysAndPerilsOfInteractiveDevelopment.pdf">slides</a></p>
<h5 id="new-concurrency-utilities-in-java-8-angelika-langer"><a href="http://gotocon.com/berlin-2014/presentation/New%20Concurrency%20Utilities%20in%20Java%208">New Concurrency Utilities in Java 8</a> - <a href="https://twitter.com/AngelikaLanger">Angelika Langer</a></h5>
<p><a href="http://gotocon.com/dl/goto-berlin-2014/slides/protected/AngelikaLanger_NewConcurrencyUtilitiesInJava8.pdf">slides</a></p>
<ul>
<li>Future API is becoming usable, it is now possible to chain futures and callbacks.</li>
<li>new StampedLock is optimized for reading. (personal note: but lock-free algorithms should be preferred, like in <a href="http://mechanical-sympathy.blogspot.com/">Lock-Based vs Lock-Free Concurrent Algorithms</a>)</li>
</ul>
<h5 id="adaptive-planning-beyond-user-stories-gojko-adzic"><a href="http://gotocon.com/berlin-2014/presentation/Adaptive%20Planning%20-%20Beyond%20User%20Stories">Adaptive Planning - Beyond User Stories</a> - <a href="https://twitter.com/gojkoadzic">Gojko Adzic</a></h5>
<p><a href="http://gojko.net/">blog</a>
<a href="http://gotocon.com/dl/goto-berlin-2014/slides/GojkoAdzic_AdaptivePlanningBeyondUserStories.pdf">slides</a></p>
<p>Gojko Adzic really thinks agile. Good presentation of how to make better user stories.<br/>
Do not describe a desired behavior but a behavior change.</p>
asynchronous IO for Play! applications in servlet 3.1 containers with the play2-war plugin2014-08-10T00:00:00+00:002014-08-10T00:00:00+00:00https://yanns.github.io/blog/2014/08/10/complete-asynchronous-io-for-play-applications-in-servlet-3-dot-1-containers-with-the-play2-war-plugin/<p>A <a href="http://playframework.com/">Play application</a> does not need any container and <a href="http://playframework.com/documentation/2.3.x/Production">runs directly in production</a>.</p>
<p>However some organizations prefer to run play applications within a servlet container and can use for this the <a href="https://github.com/play2war/play2-war-plugin">WAR Plugin</a>.</p>
<p>I am very pleased to share that this plugin is now compatible with servlet 3.1 containers. It can now use the new asynchronous IO possibilities.</p>
<p>Let me explain why and when it is important.</p>
<h3 id="play-applications-are-asynchronous">Play applications are asynchronous</h3>
<p>The Play Framework is build to be totally asynchronous and reactive. It uses no blocking IO. A Play application scales very well, using very few threads.</p>
<p>This reactive consuming or construction of IO stream is designed in play with <a href="http://www.playframework.com/documentation/2.3.x/Iteratees">Iteratees</a>. It will progressively be completed with <a href="http://typesafe.com/blog/typesafe-announces-akka-streams">Akka Streams</a>, the implementation in Akka of the reactive stream project (http://www.reactive-streams.org/)</p>
<h3 id="servlet-containers-use-blocking-io">Servlet containers use blocking IO</h3>
<p>On the other hand, servlet containers traditionally use a <a href="http://www.slideshare.net/brikis98/the-play-framework-at-linkedin/62">thread per request</a>. When doing IO (database access, external web request), the thread waits for the IO completion (blocking IO). That's why servlet containers need a lot of working threads to handle requests in parallel (<a href="http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html">default 200 threads max for tomcat</a>)</p>
<h4 id="asynchronous-3-0-servlet">Asynchronous 3.0 servlet</h4>
<p>Servlet 3.0 containers introduced the possibility to “suspend” a request.
For example, if an application makes an HTTP request to another web service using the <a href="http://www.playframework.com/documentation/2.3.x/ScalaWS">WS client</a>, the play2 war plugin is able to suspend the servlet request until the external web service answers. It means that with the same number of servlet threads, a servlet 3.0 container can support more requests in parallel than a servlet 2.x container. It does not need that a thread waits for an HTTP request, this thread can be used for other requests.</p>
<h4 id="limitations-of-asynchronous-3-0-servlet">Limitations of asynchronous 3.0 servlet</h4>
<p>When uploading or downloading a big file, the servlet container is able to stream the data. But between each chunks, the servlet thread is blocking, waiting for the next one.
It is not possible to consume one chunk, and later, when we are ready to consume another one, tell the container: “now I am ready, you can send me the next chunk as soon as you receive one from the browser”.</p>
<p>If a browser needs an hour to upload a file with a slow Internet connection, the container needs a thread during an hour, even if the application does not do anything, just waiting for the upload completion.</p>
<h3 id="play-applications-deployed-as-war-are-limited-by-the-servlet-container">Play applications deployed as war are limited by the servlet container</h3>
<p>A Play application deployed in a war container is limited by the technical possibilites of the servlet API. With a servlet 2.x or 3.0, a Play application does not scale as well as when run natively.</p>
<h3 id="new-asynchronous-io-in-servlet-3-1">New asynchronous IO in servlet 3.1</h3>
<p>The <a href="https://jcp.org/en/jsr/detail?id=340">servlet 3.1 specification</a> added asynchronous IO. Based on <a href="http://docs.oracle.com/javase/7/docs/api/java/util/EventListener.html">events</a>, it is now possible to completly handle an <a href="https://javaee-spec.java.net/nonav/javadocs/javax/servlet/ReadListener.html">upload</a> and a <a href="https://javaee-spec.java.net/nonav/javadocs/javax/servlet/WriteListener.html">download</a> asynchronously.</p>
<h3 id="asynchronous-io-in-war-plugin-for-play-applications">Asynchronous IO in WAR Plugin for Play! applications</h3>
<p>Since the <a href="https://github.com/play2war/play2-war-plugin/releases/tag/1.2">version 1.2</a>, the play2 war plugin is using this API to provide a complete reactive upload and download.</p>
<p>To use this, simply <a href="https://github.com/play2war/play2-war-plugin/wiki/Configuration#servlet-31-container-configuration">configure the servlet container version</a> and deploy to a servlet 3.1 server.</p>
<h4 id="when-to-use-this-feature">When to use this feature?</h4>
<p>This feature should be used especially if the application is using big download/upload. The servlet 3.1 will help to scale much better.</p>
<p>During my tests, I could upload and download files from several GB in parallel. The container and the JVM garbage collection could support that without any problems. The memory usage was very low.</p>
<p>Also please test this new version and report issues!</p>
<h4 id="how-to-build-the-application-to-scale-better">How to build the application to scale better?</h4>
<p>To scale as much as possible, the application should not block. It should always use asynchronous IO API like in the WS client.</p>
<p>But in the Java World a lot a librairies are still designed for a one-thread-per-request model and do not provide asynchronous API. It is for example the case for a JDBC driver. In that case, a separate dispatcher should be configured to handle blocking IO. More Information for this can be found in the <a href="http://www.playframework.com/documentation/2.3.x/ThreadPools">Play Framework documentation</a>.</p>
<h3 id="implementation-history">Implementation history</h3>
<p>The implementation of asynchronous IO in the WAR plugin lasted a few months.
The <a href="https://github.com/play2war/play2-war-plugin/pull/204">first pull request</a> introduced the asynchronous download, and <a href="https://github.com/play2war/play2-war-plugin/pull/235">the second one</a> the asynchronous upload.
I'd like to thank <a href="https://twitter.com/jroper">James Roper</a> and <a href="https://twitter.com/viktorklang">Viktor Klang</a> for their reviews.</p>
<p>This feature was quite challenging to implement:</p>
<ul>
<li>
<p>I had to find a good way to implement the glue between two very different APIs. The servlet 3.1 API is quite imperative and use <a href="http://docs.oracle.com/javase/7/docs/api/java/util/EventListener.html">EventListener</a> and methods with side effects. The Iteratee API is functional and I needed some time to feel at ease with it.</p>
</li>
<li>
<p>The servlet 3.1 specification was recently finalized as I began. The first implementations in some containers contained bugs. Reporting the problems, explaining, convincing other developers took a lot of time and energy.</p>
</li>
<li>
<p>The servlet 3.1 specification is not always explicit. The implementations in the different servlet containers are also different. Finding an implementation that covers all these subtle differences was challenging. The <a href="https://play-war.ci.cloudbees.com/job/Play_2_War_Run_integration_tests_-_Play_22x/">testing infrastructure of the play2-war plugin</a> provides a lot of integration tests with different containers and helped a lot for this.</p>
</li>
</ul>
<p>My firm <a href="http://www.leanovate.de/">Leanovate</a> gave me some time to work on that. Having worked two days full time on it helped me a lot. Thanks Leanovate for this!</p>
<p>All in all it was a great experience to contribute to the WAR Plugin, and I hope my work will be useful for others.</p>
enlarge your test scope2014-05-30T00:00:00+00:002014-05-30T00:00:00+00:00https://yanns.github.io/blog/2014/05/30/enlarge-your-test-scope/<p>At the beginning at the year, I had the chance to present <a href="/blog/2014/02/17/ping-conf-2014/">how to organize a play application with the Cake pattern</a> at <a href="http://www.ping-conf.com/">Ping Conf</a>.
I showed how this pattern enable designing the application as components, how to reduce visibility of specific gateway's model only to components that need it. One side-effect of the cake pattern is that it allows a dependency injection resolved at compilation time.</p>
<p>In one of my last slides, I warned against abusing a dependency injection mechanism to write too much unit tests.
To stay within the time slot, I have not developed so much my concerns about that point.</p>
<p><img src="http://image.slidesharecdn.com/play-with-cake-export2-140121150250-phpapp01/95/slide-66-638.jpg" alt="Do not over-use unit testing"/></p>
<p>During the talk, I implemented an application to demonstrate my points. This application consumes two external web services to render HTML pages. It is quite a typical application we can see in an infrastructure build with micro-services.</p>
<p>I've now took the time to write a new version of this application I used in the demo.
<a href="https://github.com/yanns/TPA/tree/master/frontend/TBA_07">And This new version is not using any unit tests but only some sort of component tests.</a></p>
<p>Let's dig into the details how this new version differs from the ones build around the Cake pattern.</p>
<h3 id="traditional-view-of-unit-tests">Traditional view of unit tests</h3>
<p>When building an application, we usually structure the code into layers to separate responsibilities, thus enabling re-use of logic, and avoiding repetition.</p>
<p>In the demo I used for the talk, the application is for example layered into views, controllers, services and gateways. All these layers have access to a common model.</p>
<p><img src="http://image.slidesharecdn.com/play-with-cake-export2-140121150250-phpapp01/95/slide-10-638.jpg" alt="code structured in layers"/></p>
<p>A traditional approach of unit test is to consider one class or function of one layer as a unit to test. The other dependent layers are mocked.</p>
<p>For example, to test the service layers, we use mocks for the gateways, simulating responses from external web services.</p>
<p><img src="http://image.slidesharecdn.com/play-with-cake-export2-140121150250-phpapp01/95/slide-21-638.jpg" alt="Testing with components"/></p>
<p>This approach works well, but has several downsides:</p>
<ul>
<li>the unit tests prove that one layer is working as expected, but they said nothing about all the layers used together.</li>
<li>the unit tests use the internal details of the implementation. Re-factoring the code implies then to change a lot of tests.</li>
</ul>
<p>By using dependency injection and mocks, it is nowadays very easy to write unit tests. The effect if some applications are tested almost only with unit tests:</p>
<p><img src="http://image.slidesharecdn.com/play-with-cake-export2-140121150250-phpapp01/95/slide-65-638.jpg" alt="Test pyramide"/></p>
<h3 id="traditional-view-of-component-tests">Traditional view of component tests</h3>
<p>To complement the unit tests, a team can decide to implement some component tests.</p>
<p>For the sample used in the talk, the component is the font end application we are currently implementing.</p>
<p>The most common way to run component tests is to start the tested application. The application is configured not to use the real external web services, but some local mock implementations. The local mock implementations are started as http servers as well, running on different ports.</p>
<p><img src="http://image.slidesharecdn.com/play-with-cake-export2-140121150250-phpapp01/95/slide-13-638.jpg" alt="Component tests"/></p>
<p>When the application and all the mocks are started, we can test the application by sending some http requests and by analyzing the responses.</p>
<p>Setting up the test environment with this approach is quite complex. For each external web service, a mock must be implemented as a real local http server. We must start all mock implementations, and then the new configured application. At the end of the tests, we must shutdown all services, even in case of exceptions.</p>
<p>But the main drawback with this approach in my opinion is that running the tests take a lot of time, too much to be integrated in a normal development flow (write code -> compile -> test)</p>
<h3 id="an-alternative-approach-between-component-and-unit-tests">An alternative approach between component and unit tests</h3>
<p>To strictly adhere to the definition of component tests, we should treat the tested application as a black box, and simulate all external web services. We saw that this approach is somewhat heavy to use: each external web service must be mock with a real http server.</p>
<p>Starting and running the tests in that configuration take time. Debugging an error can be difficult.</p>
<p>The strategy I used in new version of the demo application (<a href="https://github.com/yanns/TPA/tree/master/frontend/TBA_07">TBA_07</a>) is a little bit different.
The idea is still to use a request / response to test the application, but without having to run the application and any external web services.</p>
<p>Implementing that is actually quite easy: each layer declared as dependency an HTTP client (a <a href="http://www.playframework.com/documentation/2.3.x/api/scala/index.html#play.api.libs.ws.WSClient">WSClient</a> in play framework 2.3)</p>
<ul>
<li>The http client is a dependency at the top (controllers' layer):</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">package </span><span>controllers
</span><span>
</span><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> Application</span><span>(</span><span style="color:#bf616a;">ws</span><span>: </span><span style="color:#ebcb8b;">WSClient</span><span>, </span><span style="color:#bf616a;">app</span><span>: play.api.</span><span style="color:#ebcb8b;">Application</span><span>) </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">Controller </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">topVideoService </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">TopVideoService</span><span style="color:#eff1f5;">(ws, app)
</span><span style="color:#65737e;">//...
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>(a second "dependency" is the current play application. This approach is very convenient to simulate different configurations)</p>
<ul>
<li>The real implementation of the http client is then "injected" at the last time, when we construct the controller singleton:</li>
</ul>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>def playCurrent = play.api.Play.current
</span><span>object Application extends Application(WS.client(playCurrent), playCurrent)
</span></code></pre>
<ul>
<li>To test the application, we then simply have to instantiate the controller with an <a href="https://github.com/yanns/TPA/blob/master/frontend/TBA_07/test/httpclient/MockWS.scala">alternative implementation of the http client capable of simulating external web services</a>:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> ApplicationControllerFixture
</span><span> </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">Application</span><span>(MockWS(playerRoute), FakeApplication())
</span></code></pre>
<ul>
<li>The <code>playerRoute</code> simulate the external player web service:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">playerId </span><span>= PlayerId(</span><span style="color:#d08770;">34</span><span>)
</span><span>
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">playerRoute</span><span>: </span><span style="color:#ebcb8b;">MockWS</span><span>.</span><span style="color:#ebcb8b;">Routes </span><span>= {
</span><span> </span><span style="color:#b48ead;">case </span><span>("</span><span style="color:#a3be8c;">GET</span><span>", </span><span style="color:#bf616a;">u</span><span>) </span><span style="color:#b48ead;">if</span><span> u == </span><span style="color:#96b5b4;">s</span><span>"</span><span style="color:#a3be8c;">http://localhost:9001/players/</span><span style="color:#bf616a;">$playerId</span><span>" </span><span style="color:#b48ead;">=>
</span><span> Action { Ok(Json.parse(playerJson(playerId))) }
</span><span>
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#bf616a;">_ </span><span style="color:#b48ead;">=> </span><span>Action { NotFound }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">playerJson</span><span>(</span><span style="color:#bf616a;">playerId</span><span>: </span><span style="color:#ebcb8b;">PlayerId</span><span>) =
</span><span> </span><span style="color:#96b5b4;">s</span><span>"""</span><span style="color:#a3be8c;">{
</span><span style="color:#a3be8c;"> | "id": </span><span style="color:#bf616a;">$playerId</span><span style="color:#a3be8c;">,
</span><span style="color:#a3be8c;"> | "name": "ze name",
</span><span style="color:#a3be8c;"> | "height": "ze height",
</span><span style="color:#a3be8c;"> | "weight": "ze weight",
</span><span style="color:#a3be8c;"> | "team": "ze team"
</span><span style="color:#a3be8c;"> |}
</span><span style="color:#a3be8c;"> </span><span>""".stripMargin
</span></code></pre>
<p>The <code>MockWS.Routes</code> type defines a partial function <code>PartialFunction[(String, String), EssentialAction]</code>, making really easy to combine different routes together with <code>orElse</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>SimulatedVideoBackend.videoRoute orElse SimulatedPlayerBackend.playerRoute
</span></code></pre>
<ul>
<li>and we can test the response by calling the controller with a <a href="http://www.playframework.com/documentation/2.3.x/api/scala/index.html#play.api.test.FakeRequest"><code>FakeRequest</code></a>:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">result </span><span>= index.apply(FakeRequest())
</span><span>status(result) mustEqual OK
</span></code></pre>
<p>The application is constructed as if it was depending from the http client and the current play application.</p>
<p>These tests are not strictly component tests, as we are not testing the real implementation of the http client.
The application is not entirely treated as a black box. But most of the code is tested like in production.</p>
<h4 id="drawbacks-of-this-approach">Drawbacks of this approach:</h4>
<ul>
<li>writing a test is more complicated than testing a little unit of code</li>
<li>writing unit test can help avoiding code mixing responsibilities. We do not profit from that.</li>
<li>when a test suddenly fails, it is more difficult to find out why.</li>
<li>we do not test the complete application stack. For example, the <a href="http://www.playframework.com/documentation/2.3.x/ScalaHttpFilters">play filters</a> and the real http client is not tested.</li>
</ul>
<h4 id="advantages-of-this-approach">Advantages of this approach:</h4>
<ul>
<li>a developer must understand how the application works in general to be able to write good tests</li>
<li>the application can be re-factored without modifying the tests</li>
<li>the user functions are better tested</li>
<li>the integration of all layers together is tested</li>
<li>we do not need any running server to check all the tests. The tests run very fast.</li>
<li>the code is simple (compare the <a href="https://github.com/yanns/TPA/blob/master/frontend/TBA_07/app/services/TopVideoService.scala">TopVideoService in that version</a> with the <a href="https://github.com/yanns/TPA/blob/master/frontend/TBA_05_final/app/services/TopVideoServiceComp.scala">one using the Cake pattern</a>)</li>
</ul>
<h3 id="experience-with-that-approach">Experience with that approach</h3>
<p>With one team, we are currently testing this approach. The results are quite encouraging: more than 80 % of the statements are covered by tests. We have more than 200 tests running in 10 seconds on my machine.</p>
<p>And I could deeply change the code with almost no impact on the tests. ;)</p>
SLF4J Mapped Diagnostic Context (MDC) with play framework2014-05-04T00:00:00+00:002014-05-04T00:00:00+00:00https://yanns.github.io/blog/2014/05/04/slf4j-mapped-diagnostic-context-mdc-with-play-framework/<p>I'd like the share with this post one solution I found to use a Mapped Diagnostic Context (MDC) in an asynchronous environment like the play framework.</p>
<h2 id="edit-september-2014">Edit (September 2014)</h2>
<p>Based on <a href="https://github.com/jroper/thread-local-context-propagation/">one implementation from James Roper</a>, I added one solution based on <a href="http://doc.akka.io/docs/akka/current/scala/dispatchers.html">Akka Dispatcher</a>.</p>
<h2 id="tl-dr">tl;dr</h2>
<p>This post provides two solution to propagate the MDC context in an asynchronous Play application:</p>
<ul>
<li>using a custom Akka <code>Dispatcher</code>. This solution needs minimal change to a current application.</li>
<li>using a custom <code>ExecutionContext</code> that propagates the MDC from the caller's thread to the callee's one. A custom <code>ActionBuilder</code> is necessary as well to completely use this custom <code>ExectionContext</code>.</li>
</ul>
<h2 id="the-mapped-diagnostic-context-mdc">The Mapped Diagnostic Context (MDC)</h2>
<p>The play framework uses for logging <a href="http://logback.qos.ch/">Logback</a> behind <a href="http://www.slf4j.org/">SLF4J ("Simple Logging Facade for Java")</a>.<br/>
This library provides a convenient feature: the <a href="http://logback.qos.ch/manual/mdc.html">Mapped Diagnostic Context (MDC)</a>.
This context can be used to store values that can be displayed in every Logging statement.<br/>
For example, if we want to display the current user ID:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> org.slf4j.MDC
</span><span>
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">id </span><span>= currentUser.id
</span><span>MDC.put("</span><span style="color:#a3be8c;">X-UserId</span><span>", currentUser.id)
</span><span>
</span><span style="color:#b48ead;">try </span><span>{
</span><span> </span><span style="color:#65737e;">// the block of code that uses the Logger
</span><span> </span><span style="color:#65737e;">// for example:
</span><span> play.api.Logger.info("</span><span style="color:#a3be8c;">test</span><span>")
</span><span>} </span><span style="color:#b48ead;">finally </span><span>{
</span><span> </span><span style="color:#65737e;">// clean up the MDC
</span><span> MDC.remove("</span><span style="color:#a3be8c;">X-UserId</span><span>")
</span><span>}
</span></code></pre>
<p>(This code could be in a <a href="https://www.playframework.com/documentation/latest/ScalaHttpFilters">filter</a>, run for each request)</p>
<p>Logback must be configured to display the <code>X-UserId</code> value:</p>
<pre data-lang="xml" style="background-color:#2b303b;color:#c0c5ce;" class="language-xml "><code class="language-xml" data-lang="xml"><span><</span><span style="color:#bf616a;">appender </span><span style="color:#d08770;">name</span><span>="</span><span style="color:#a3be8c;">stdout</span><span>" </span><span style="color:#d08770;">class</span><span>="</span><span style="color:#a3be8c;">ch.qos.logback.core.ConsoleAppender</span><span>">
</span><span> <</span><span style="color:#bf616a;">encoder</span><span>>
</span><span> <</span><span style="color:#bf616a;">pattern</span><span>>%d{HH:mm:ss.SSS} %coloredLevel %logger{35} %mdc{X-UserId:--} - %msg%n%rootException</</span><span style="color:#bf616a;">pattern</span><span>>
</span><span> </</span><span style="color:#bf616a;">encoder</span><span>>
</span><span></</span><span style="color:#bf616a;">appender</span><span>>
</span></code></pre>
<p>In the log file, the MDC value for <code>X-UserId</code> is now printed out.</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>10:50:54.773 [info] application jean.leloup - test
</span></code></pre>
<h2 id="limitation-of-the-default-implementation-of-the-mdc">Limitation of the default implementation of the MDC</h2>
<p>To record the values in the MDC, Logback uses a <code>ThreadLocal</code> variable.
This strategy works when one thread is used for one request, like in servlet container before the 3.1 specification.</p>
<p>Play framework, on the other hand, is <a href="http://www.playframework.com/documentation/2.2.x/ScalaAsync">asynchronous</a>. The processing of a request is composed of several function calls, and each call can be run on a different thread. ("Don't call me, I'll call you")</p>
<p>The implementation of the MDC with a <code>ThreadLocal</code> cannot work with this non-blocking asynchronous threading model.</p>
<h2 id="first-solution-with-akka-dispatcher">First solution with Akka Dispatcher</h2>
<h4 id="defining-a-custom-akka-dispatcher">Defining a custom Akka dispatcher</h4>
<p>Play dispatchs the jobs on different threads with a <a href="https://www.playframework.com/documentation/latest/ThreadPools">thread pool</a>. The Play default thread pool is an <a href="http://doc.akka.io/docs/akka/current/scala/dispatchers.html">Akka dispatcher</a>.</p>
<p>To use the MDC, we provide a custom Akka <code>Dispatcher</code> that propagates the MDC from the caller's thread to the callee's one.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">package </span><span>monitoring
</span><span>
</span><span style="color:#b48ead;">import</span><span> java.util.concurrent.TimeUnit
</span><span>
</span><span style="color:#b48ead;">import</span><span> akka.dispatch.</span><span style="color:#bf616a;">_
</span><span style="color:#b48ead;">import</span><span> com.typesafe.config.Config
</span><span style="color:#b48ead;">import</span><span> org.slf4j.MDC
</span><span>
</span><span style="color:#b48ead;">import</span><span> scala.concurrent.ExecutionContext
</span><span style="color:#b48ead;">import</span><span> scala.concurrent.duration.{Duration, FiniteDuration}
</span><span>
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * Configurator for a MDC propagating dispatcher.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * To use it, configure play like this:
</span><span style="color:#65737e;"> * {{{
</span><span style="color:#65737e;"> * play {
</span><span style="color:#65737e;"> * akka {
</span><span style="color:#65737e;"> * actor {
</span><span style="color:#65737e;"> * default-dispatcher = {
</span><span style="color:#65737e;"> * type = "monitoring.MDCPropagatingDispatcherConfigurator"
</span><span style="color:#65737e;"> * }
</span><span style="color:#65737e;"> * }
</span><span style="color:#65737e;"> * }
</span><span style="color:#65737e;"> * }
</span><span style="color:#65737e;"> * }}}
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * Credits to James Roper for the [[https://github.com/jroper/thread-local-context-propagation/ initial implementation]]
</span><span style="color:#65737e;"> */
</span><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> MDCPropagatingDispatcherConfigurator</span><span>(</span><span style="color:#bf616a;">config</span><span>: </span><span style="color:#ebcb8b;">Config</span><span>, </span><span style="color:#bf616a;">prerequisites</span><span>: </span><span style="color:#ebcb8b;">DispatcherPrerequisites</span><span>)
</span><span> </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">MessageDispatcherConfigurator</span><span>(config, prerequisites) </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">private val </span><span style="color:#bf616a;">instance </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">MDCPropagatingDispatcher</span><span style="color:#eff1f5;">(
</span><span style="color:#eff1f5;"> </span><span style="color:#bf616a;">this</span><span style="color:#eff1f5;">,
</span><span style="color:#eff1f5;"> config.getString(</span><span>"</span><span style="color:#a3be8c;">id</span><span>"</span><span style="color:#eff1f5;">),
</span><span style="color:#eff1f5;"> config.getInt(</span><span>"</span><span style="color:#a3be8c;">throughput</span><span>"</span><span style="color:#eff1f5;">),
</span><span style="color:#eff1f5;"> FiniteDuration(config.getDuration(</span><span>"</span><span style="color:#a3be8c;">throughput-deadline-time</span><span>"</span><span style="color:#eff1f5;">, TimeUnit.NANOSECONDS), TimeUnit.NANOSECONDS),
</span><span style="color:#eff1f5;"> configureExecutor(),
</span><span style="color:#eff1f5;"> FiniteDuration(config.getDuration(</span><span>"</span><span style="color:#a3be8c;">shutdown-timeout</span><span>"</span><span style="color:#eff1f5;">, TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS))
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">override def </span><span style="color:#8fa1b3;">dispatcher</span><span style="color:#eff1f5;">(): </span><span style="color:#ebcb8b;">MessageDispatcher </span><span>=</span><span style="color:#eff1f5;"> instance
</span><span style="color:#eff1f5;">}
</span><span>
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * A MDC propagating dispatcher.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * This dispatcher propagates the MDC current request context if it's set when it's executed.
</span><span style="color:#65737e;"> */
</span><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> MDCPropagatingDispatcher</span><span>(</span><span style="color:#bf616a;">_configurator</span><span>: </span><span style="color:#ebcb8b;">MessageDispatcherConfigurator</span><span>,
</span><span> </span><span style="color:#bf616a;">id</span><span>: </span><span style="color:#ebcb8b;">String</span><span>,
</span><span> </span><span style="color:#bf616a;">throughput</span><span>: </span><span style="color:#b48ead;">Int</span><span>,
</span><span> </span><span style="color:#bf616a;">throughputDeadlineTime</span><span>: </span><span style="color:#ebcb8b;">Duration</span><span>,
</span><span> </span><span style="color:#bf616a;">executorServiceFactoryProvider</span><span>: </span><span style="color:#ebcb8b;">ExecutorServiceFactoryProvider</span><span>,
</span><span> </span><span style="color:#bf616a;">shutdownTimeout</span><span>: </span><span style="color:#ebcb8b;">FiniteDuration</span><span>)
</span><span> </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">Dispatcher</span><span>(_configurator, id, throughput, throughputDeadlineTime, executorServiceFactoryProvider, shutdownTimeout ) </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#bf616a;">self </span><span style="color:#b48ead;">=>
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">override def </span><span style="color:#8fa1b3;">prepare</span><span style="color:#eff1f5;">(): </span><span style="color:#ebcb8b;">ExecutionContext </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">ExecutionContext </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">// capture the MDC
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">mdcContext </span><span>= </span><span style="color:#eff1f5;">MDC.getCopyOfContextMap
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">execute</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">r</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Runnable</span><span style="color:#eff1f5;">) </span><span>=</span><span style="color:#eff1f5;"> self.execute(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">Runnable </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">run</span><span style="color:#eff1f5;">() </span><span>= </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">// backup the callee MDC context
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">oldMDCContext </span><span>= </span><span style="color:#eff1f5;">MDC.getCopyOfContextMap
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">// Run the runnable with the captured context
</span><span style="color:#eff1f5;"> setContextMap(mdcContext)
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">try </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> r.run()
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">finally </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">// restore the callee MDC context
</span><span style="color:#eff1f5;"> setContextMap(oldMDCContext)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> })
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">reportFailure</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">t</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Throwable</span><span style="color:#eff1f5;">) </span><span>=</span><span style="color:#eff1f5;"> self.reportFailure(t)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">private</span><span style="color:#eff1f5;">[</span><span style="color:#bf616a;">this</span><span style="color:#eff1f5;">] </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">setContextMap</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">context</span><span style="color:#eff1f5;">: java.util.</span><span style="color:#ebcb8b;">Map</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">String</span><span style="color:#eff1f5;">, </span><span style="color:#ebcb8b;">String</span><span style="color:#eff1f5;">]) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(context </span><span>== </span><span style="color:#d08770;">null</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> MDC.clear()
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">else </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> MDC.setContextMap(context)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;">}
</span></code></pre>
<h4 id="using-a-custom-akka-dispatcher-everywhere">Using a custom Akka dispatcher everywhere:</h4>
<p>To use this custom Akka dispatcher everywhere, we just have to configure it:</p>
<pre data-lang="bash application.conf" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash application.conf "><code class="language-bash application.conf" data-lang="bash application.conf"><span>play {
</span><span> akka {
</span><span> actor {
</span><span> default-dispatcher = {
</span><span> type = "monitoring.MDCPropagatingDispatcherConfigurator"
</span><span> }
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>and that's all! ;)</p>
<p>The MDC context is propagated when we use the play default <a href="https://www.playframework.com/documentation/2.3.x/api/scala/index.html#play.api.libs.concurrent.Execution$"><code>ExecutionContext</code></a>.</p>
<h4 id="optimization">Optimization</h4>
<p>So that this approach works in dev mode, simply make a library (jar) of this custom Akka dispatcher and add this as dependency in your play application.</p>
<h2 id="second-solution-with-a-custom-execution-context">Second solution with a custom execution context</h2>
<h4 id="defining-a-custom-execution-context">Defining a custom execution context</h4>
<p>The dispatching of the jobs on different threads in done with an <code>ExecutionContext</code>. Each <code>ExecutionContext</code> manages a <a href="https://www.playframework.com/documentation/latest/ThreadPools">thread pool</a>.</p>
<p>To use the MDC, we just have to use a custom <code>ExecutionContext</code> that propagates the MDC from the caller's thread to the callee's one.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> org.slf4j.MDC
</span><span style="color:#b48ead;">import</span><span> scala.concurrent.{ExecutionContextExecutor, ExecutionContext}
</span><span>
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * slf4j provides a MDC [[http://logback.qos.ch/manual/mdc.html Mapped Diagnostic Context]]
</span><span style="color:#65737e;"> * based on a [[ThreadLocal]]. In an asynchronous environment, the callbacks can be called
</span><span style="color:#65737e;"> * in another thread, where the local thread variable does not exist anymore.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * This execution context fixes this problem:
</span><span style="color:#65737e;"> * it propagates the MDC from the caller's thread to the callee's one.
</span><span style="color:#65737e;"> */
</span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> MDCHttpExecutionContext </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * Create an MDCHttpExecutionContext with values from the current thread.
</span><span style="color:#65737e;"> */
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">fromThread</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">delegate</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionContext</span><span style="color:#eff1f5;">): </span><span style="color:#ebcb8b;">ExecutionContextExecutor </span><span>=
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">MDCHttpExecutionContext</span><span style="color:#eff1f5;">(MDC.getCopyOfContextMap, delegate)
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;">}
</span><span>
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * Manages execution to ensure that the given MDC context are set correctly
</span><span style="color:#65737e;"> * in the current thread. Actual execution is performed by a delegate ExecutionContext.
</span><span style="color:#65737e;"> */
</span><span style="color:#b48ead;">class</span><span style="color:#ebcb8b;"> MDCHttpExecutionContext</span><span>(</span><span style="color:#bf616a;">mdcContext</span><span>: java.util.</span><span style="color:#ebcb8b;">Map</span><span>[</span><span style="color:#bf616a;">_</span><span>, </span><span style="color:#bf616a;">_</span><span>], </span><span style="color:#bf616a;">delegate</span><span>: </span><span style="color:#ebcb8b;">ExecutionContext</span><span>) </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">ExecutionContextExecutor </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">execute</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">runnable</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Runnable</span><span style="color:#eff1f5;">) </span><span>=</span><span style="color:#eff1f5;"> delegate.execute(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">Runnable </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">run</span><span style="color:#eff1f5;">() {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">oldMDCContext </span><span>= </span><span style="color:#eff1f5;">MDC.getCopyOfContextMap
</span><span style="color:#eff1f5;"> setContextMap(mdcContext)
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">try </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> runnable.run()
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">finally </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> setContextMap(oldMDCContext)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> })
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">private</span><span style="color:#eff1f5;">[</span><span style="color:#bf616a;">this</span><span style="color:#eff1f5;">] </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">setContextMap</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">context</span><span style="color:#eff1f5;">: java.util.</span><span style="color:#ebcb8b;">Map</span><span style="color:#eff1f5;">[</span><span style="color:#bf616a;">_</span><span style="color:#eff1f5;">, </span><span style="color:#bf616a;">_</span><span style="color:#eff1f5;">]) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(context </span><span>== </span><span style="color:#d08770;">null</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> MDC.clear()
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">else </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> MDC.setContextMap(context)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">reportFailure</span><span style="color:#eff1f5;">(</span><span style="color:#bf616a;">t</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Throwable</span><span style="color:#eff1f5;">) </span><span>=</span><span style="color:#eff1f5;"> delegate.reportFailure(t)
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>Then we can define the default ExecutionContext in our application:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">package </span><span>concurrent
</span><span>
</span><span style="color:#b48ead;">import</span><span> scala.concurrent.ExecutionContext
</span><span>
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * The standard [[play.api.libs.concurrent.Execution.defaultContext]] loses the MDC context.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * This custom [[ExecutionContext]] propagates the MDC context, so that the request
</span><span style="color:#65737e;"> * and the correlation IDs can be logged.
</span><span style="color:#65737e;"> */
</span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> Execution </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> Implicits </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">implicit def </span><span style="color:#8fa1b3;">defaultContext</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionContext </span><span>= </span><span style="color:#eff1f5;">Execution.defaultContext
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">defaultContext</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionContext </span><span>= </span><span style="color:#eff1f5;">MDCHttpExecutionContext.fromThread(play.api.libs.concurrent.Execution.defaultContext)
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>Now we will use the <code>concurrent.Execution.defaultContext</code> instead of the one from play (<code>play.api.libs.concurrent.Execution.defaultContext</code>)</p>
<h4 id="using-a-custom-execution-context-everywhere">Using a custom execution context everywhere</h4>
<p>Using a custom execution context is sometimes as easy as replacing
<code>import play.api.libs.concurrent.Execution.Implicits._</code> with <code>import concurrent.Execution.Implicits._</code></p>
<p>The default <a href="https://www.playframework.com/documentation/latest/ScalaActions"><code>Action</code></a> uses the default <code>play.api.libs.concurrent.Execution.defaultContext</code>.
We must define a custom <code>ActionBuilder</code> that uses our new <code>ExecutionContext</code>:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">package </span><span>controllers
</span><span>
</span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> Action </span><span style="color:#b48ead;">extends </span><span style="color:#a3be8c;">ActionBuilder</span><span>[</span><span style="color:#ebcb8b;">Request</span><span>] </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">invokeBlock</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">](</span><span style="color:#bf616a;">request</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Request</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">], </span><span style="color:#bf616a;">block</span><span style="color:#eff1f5;">: (</span><span style="color:#ebcb8b;">Request</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">A</span><span style="color:#eff1f5;">]) </span><span>=> </span><span style="color:#ebcb8b;">Future</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">SimpleResult</span><span style="color:#eff1f5;">]) </span><span>= </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> block(request)
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * The standard [[play.api.mvc.Action]] loses the MDC context.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * This action builder sets the [[ExecutionContext]] so that the
</span><span style="color:#65737e;"> * MDC context is propagated.
</span><span style="color:#65737e;"> * With this custom [[ExecutionContext]], the request and the correlation IDs
</span><span style="color:#65737e;"> * can be logged.
</span><span style="color:#65737e;"> */
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">override def </span><span style="color:#8fa1b3;">executionContext</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">ExecutionContext </span><span>= </span><span style="color:#eff1f5;">Execution.defaultContext
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>Instead of using of <code>play.api.mvc.Action</code>, we just have to use the newly defined <code>controllers.Action</code>.</p>
<p>With each of these customizations, we are now able to use the Mapped Diagnostic Context (MDC) with asynchronous actions written in Scala.</p>
Server side rendering for JavaScript ReactJS framework2014-03-15T00:00:00+00:002014-03-15T00:00:00+00:00https://yanns.github.io/blog/2014/03/15/server-side-rendering-for-javascript-reactjs-framework/<h2 id="flicker-effect-with-javascript-applications">Flicker effect with JavaScript applications</h2>
<p>A lot of web applications are nowadays build with a JavaScript framework, rendering the HTML in the browser (client side).
There are a few reasons for this, like:</p>
<ul>
<li>avoiding server-browser round-trips to modify one HTML element</li>
<li>it is easier to keep the server side stateless if you maintain the state in the browser</li>
<li>the server can expose a public REST API for partners. And your own JavaScript application can use this API, encouraging <a href="http://en.wikipedia.org/wiki/Eating_your_own_dog_food">eating our own dog food</a></li>
</ul>
<p>Building a client side JavaScript application is not always easy for teams used to server side code, and a few frameworks can help there, like <a href="http://angularjs.org/">AngularJS</a>, <a href="http://emberjs.com/">Ember</a> or <a href="http://facebook.github.io/react/">React</a></p>
<p>We will look at an <a href="http://play-react.herokuapp.com/clientSide">example with React</a></p>
<p>To display the HTML, a few steps are needed:</p>
<ol>
<li>
<p>The browser loads HTML, CSS and JavaScript.<br>
It displays the HTML delivered directly by the server.<br>
<img src="/assets/2014-03-15/server.png" alt="The browser shows the HTML coming from the server" /></p>
<p>With AngularJS, if <a href="http://docs.angularjs.org/guide/expression">inline expression</a> are used, the user can see the following for a few milliseconds:<br/>
hello {{firstname}}<br/>
before AngularJS replaces this expression with its computed value.</p>
</li>
<li>
<p>The JavaScript framework manipulates the DOM and the user can then see the application.
<img src="/assets/2014-03-15/server_and_client.png" alt="The JavaScript application has changed the DOM" /></p>
</li>
<li>
<p>If the application needs to display some data from the server, it must first request it with Ajax. The data is displayed only after being received by the browser.
<img src="/assets/2014-03-15/server_and_client_and_data.png" alt="The JavaScript application has received data and changed the DOM accordingly" /></p>
</li>
</ol>
<p>(to make the <a href="http://play-react.herokuapp.com/clientSide">flicker</a> more visible, I introduced a latency of 500 ms to simulate a slow backend)</p>
<p>The user experience is not optimal. The application flickers at each step, as the DOM is changed several times in a few seconds.</p>
<h2 id="avoiding-the-flicker-effect">Avoiding the flicker effect</h2>
<h3 id="on-the-client-side">On the client side</h3>
<p>In the browser, we can mitigate the flicker effect.
Some applications show a spinner as long as the page is not ready to be shown.
The not-yet-completed DOM is hidden before being shown in one final step.</p>
<p>For example, AngularJS provides the <a href="http://docs.angularjs.org/api/ng/directive/ngCloak">ng-cloak directive</a>.
With this directive, AngularJS can hide the HTML as long as it is not ready.</p>
<h3 id="welcome-back-to-server-side-rendering">Welcome back to server side rendering</h3>
<p>Instead of rendering everything in the browser, it is also possible to first render the page on the server side, serve it when ready, and when updating it on the client side when necessary.</p>
<p>(Please notice that this technic allows the HTML to be indexed for search engines that do not execute the JavaScript.)</p>
<p>From example, React can render a UI component without any browser with <a href="http://facebook.github.io/react/docs/top-level-api.html#react.rendercomponenttostring">React.renderComponentToString</a>.</p>
<p>With this function, the complete page can be prepared on the server side, send under this form to the browser that can directly display the ready application. On the client side, the same JavaScript code can dynamically manipulate this DOM as a normal client side application.</p>
<p>The <a href="https://github.com/mhart/react-server-example">React server rendering example</a> demonstrates how to use React's server rendering capabilities. Rendering a JavaScript application on the server side is technically possible because the JavaScript is executed by <a href="http://nodejs.org/">Node.js</a>.</p>
<h3 id="and-what-about-the-jvm">And what about the JVM?</h3>
<p>If you are not using NodeJS, but the Java Virtual Machine (JVM), you might be disappointed at this time.
Pre-render a JavaScript application is only possible with Node.js?</p>
<p>In Java, there are a few projects that can save us:</p>
<ul>
<li>
<p><a href="https://github.com/apigee/trireme">trireme</a> provides a Node.js API and can run node.js scripts inside Java. It uses Rhino, the current JavaScript implementation for the JVM. (With Java 8, let's see if trireme will use the new JavaScript implementation, Nashorn, or whether Nashorn will implement the node.js API itself.)</p>
</li>
<li>
<p><a href="https://github.com/typesafehub/js-engine">js-engine</a> provides <a href="http://akka.io/">Akka Actors</a> to execute JavaScript code with trireme or with node.js</p>
</li>
</ul>
<p>As a proof of concept, I implemented a little play application that uses these projects to pre-render a React component on the server side.</p>
<p>The JavaScript is loaded:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">serverside </span><span>= Play.getFile("</span><span style="color:#a3be8c;">public/javascripts/serverside.js</span><span>")
</span></code></pre>
<p>An actor is created for a JavaScript engine (trireme or node.js)</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">engine </span><span>= Akka.system.actorOf(jsEngine, </span><span style="color:#96b5b4;">s</span><span>"</span><span style="color:#a3be8c;">engine-${</span><span>request.id</span><span style="color:#a3be8c;">}</span><span>")
</span></code></pre>
<p>We receive the data from the database:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>data <- initialData
</span></code></pre>
<p>and let the JavaScript code execute with that data as parameter</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>result <- (engine ? Engine.ExecuteJs(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">File</span><span>(serverside.toURI), List(data))).mapTo[</span><span style="color:#ebcb8b;">JsExecutionResult</span><span>]
</span></code></pre>
<p>The result is send to the browser</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>Ok(views.html.index(Html(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">String</span><span>(result.output.toArray, "</span><span style="color:#a3be8c;">UTF-8</span><span>"))))
</span></code></pre>
<p>complete controller code:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span> </span><span style="color:#65737e;">// with js-engine
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">serverSideTrireme </span><span>= serverSideWithJsEngine(Trireme.props())
</span><span>
</span><span> </span><span style="color:#65737e;">// with node
</span><span> </span><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">serverSideNode </span><span>= serverSideWithJsEngine(Node.props())
</span><span>
</span><span> </span><span style="color:#b48ead;">private def </span><span style="color:#8fa1b3;">serverSideWithJsEngine</span><span>(</span><span style="color:#bf616a;">jsEngine</span><span>: </span><span style="color:#ebcb8b;">Props</span><span>) = Action.async { </span><span style="color:#bf616a;">request </span><span style="color:#b48ead;">=>
</span><span> </span><span style="color:#b48ead;">import</span><span> akka.pattern.ask
</span><span> </span><span style="color:#b48ead;">import</span><span> scala.concurrent.duration.</span><span style="color:#bf616a;">_
</span><span>
</span><span> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">serverside </span><span>= Play.getFile("</span><span style="color:#a3be8c;">public/javascripts/serverside.js</span><span>")
</span><span> </span><span style="color:#b48ead;">implicit val </span><span style="color:#bf616a;">timeout </span><span>= Timeout(</span><span style="color:#d08770;">5</span><span>.seconds)
</span><span> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">engine </span><span>= Akka.system.actorOf(jsEngine, </span><span style="color:#96b5b4;">s</span><span>"</span><span style="color:#a3be8c;">engine-${</span><span>request.id</span><span style="color:#a3be8c;">}</span><span>")
</span><span>
</span><span> </span><span style="color:#b48ead;">for </span><span>{
</span><span> </span><span style="color:#bf616a;">data </span><span><- initialData
</span><span> </span><span style="color:#bf616a;">result </span><span><- (engine ? Engine.ExecuteJs(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">File</span><span>(serverside.toURI), List(data))).mapTo[</span><span style="color:#ebcb8b;">JsExecutionResult</span><span>]
</span><span> } </span><span style="color:#b48ead;">yield </span><span>{
</span><span> Ok(views.html.index(Html(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">String</span><span>(result.output.toArray, "</span><span style="color:#a3be8c;">UTF-8</span><span>"))))
</span><span> }
</span><span> }
</span></code></pre>
<p>The code <code>serverside.js</code> uses the <a href="http://nodejs.org/api/modules.html#modules_module_require_id">node.js modules API</a> to render our main component (CommentBox).</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#b48ead;">var </span><span style="color:#bf616a;">React </span><span>= </span><span style="color:#96b5b4;">require</span><span>('</span><span style="color:#a3be8c;">./react</span><span>'),
</span><span> </span><span style="color:#bf616a;">CommentBox </span><span>= </span><span style="color:#96b5b4;">require</span><span>('</span><span style="color:#a3be8c;">./CommentBox</span><span>');
</span></code></pre>
<p>It then loads the data given as first parameter in the controller</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#65737e;">// take data from parameters
</span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">data </span><span>= JSON.</span><span style="color:#96b5b4;">parse</span><span>(process.argv[</span><span style="color:#d08770;">2</span><span>]);
</span></code></pre>
<p>It renders the CommentBox component to a String and output it to console.log so that the Scala controller can receive the result with <code>result.output.toArray</code></p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#ebcb8b;">console</span><span>.</span><span style="color:#96b5b4;">log</span><span>(</span><span style="color:#bf616a;">React</span><span>.</span><span style="color:#8fa1b3;">renderComponentToString</span><span>(</span><span style="color:#8fa1b3;">CommentBox</span><span>(</span><span style="color:#bf616a;">backend</span><span>)({data: </span><span style="color:#bf616a;">data</span><span>, onServerSide: </span><span style="color:#d08770;">true</span><span>})));
</span></code></pre>
<p>Complete code:</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#b48ead;">var </span><span style="color:#bf616a;">React </span><span>= </span><span style="color:#96b5b4;">require</span><span>('</span><span style="color:#a3be8c;">./react</span><span>'),
</span><span> </span><span style="color:#bf616a;">CommentBox </span><span>= </span><span style="color:#96b5b4;">require</span><span>('</span><span style="color:#a3be8c;">./CommentBox</span><span>');
</span><span>
</span><span style="color:#65737e;">// take data from parameters
</span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">data </span><span>= JSON.</span><span style="color:#96b5b4;">parse</span><span>(process.argv[</span><span style="color:#d08770;">2</span><span>]);
</span><span>
</span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">backend </span><span>= {
</span><span> </span><span style="color:#8fa1b3;">loadCommentsFromServer</span><span>: </span><span style="color:#b48ead;">function</span><span>(</span><span style="color:#bf616a;">settings</span><span>) {
</span><span> },
</span><span> </span><span style="color:#8fa1b3;">handleCommentSubmit</span><span>: </span><span style="color:#b48ead;">function</span><span>(</span><span style="color:#bf616a;">settings</span><span>) {
</span><span> }
</span><span>};
</span><span>
</span><span style="color:#ebcb8b;">console</span><span>.</span><span style="color:#96b5b4;">log</span><span>(</span><span style="color:#bf616a;">React</span><span>.</span><span style="color:#8fa1b3;">renderComponentToString</span><span>(</span><span style="color:#8fa1b3;">CommentBox</span><span>(</span><span style="color:#bf616a;">backend</span><span>)({data: </span><span style="color:#bf616a;">data</span><span>, onServerSide: </span><span style="color:#d08770;">true</span><span>})));
</span></code></pre>
<p><a href="http://play-react.herokuapp.com/serverSide">This page</a> does not flicker anymore compared to the <a href="http://play-react.herokuapp.com/clientSide">first version</a>.</p>
<h3 id="drawback-with-server-side-rendering">Drawback with server side rendering</h3>
<p>The drawback with pre-rendering the page on the server side is that we have to wait to have all the data before sending the page.
In the <a href="http://play-react.herokuapp.com/serverSide">sample application</a>, I introduced a latency when requesting the data to simulate a slow database.</p>
<p>The browser must also wait long before getting any HTML. The following diagram shows that the application (deployed on Heroku) needs more than one second to deliver the page!
<img src="/assets/2014-03-15/wait_for_server.png" alt="The browser is waiting for the server" /></p>
<h3 id="can-we-optimize-more">Can we optimize more?</h3>
<p>We can optimize this version by sending the first bytes of the HTML page before having any data.
When the data is there, we can send the rest of the page.</p>
<p>With <a href="http://play-react.herokuapp.com/serverSideStream">that variant</a>, we can include the CSS and part of the JavaScript in the <HEAD> section of the HTML page.
The browser receives this information very quickly and can begin downloading these assets.
The server lets the connection open and when the rest of the page is ready, it is send to the browser.</p>
<p><img src="/assets/2014-03-15/browser_loads_assets.png" alt="The browser can load the CSS and JavaScript" /></p>
<p>To implement this, I used the Facebook’s BigPipe concept as presented in the <a href="http://de.slideshare.net/brikis98/composable-and-streamable-play-apps">talk “Building composable, streaming, testable Play apps” from Yevgeniy Brikman</a></p>
<p>It is not a "Silver Bullet" as we are still waiting for the data before displaying it to the user (that makes sense).
But the browser can load the stylesheets, the JavaScripts very quickly, leading to a more responsive page.</p>
<h2 id="integrate-play-and-trireme">Integrate Play and trireme</h2>
<p>To resolve node.js modules, trireme needs to access the JavaScripts directly on the file system.
But Play Framework package all the public assets in the jar, making the JavaScript assets not available with <a href="http://www.playframework.com/documentation/2.2.x/api/scala/index.html#play.api.Play$">Play.getFile</a></p>
<p>It would be easier if trireme would use a <a href="http://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html">FileSystem</a> object, but this API is only available from Java 7.</p>
<p>To wordaround this, I configured the <a href="https://github.com/sbt/sbt-native-packager">SBT Universal plugin</a> to deploy the public assets to the file system:</p>
<ul>
<li>in build.sbt:</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>PublicOnFileSystem.settings
</span></code></pre>
<ul>
<li>project/PublicOnFileSystem.scala</li>
</ul>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">import</span><span> sbt.</span><span style="color:#bf616a;">_
</span><span style="color:#b48ead;">import</span><span> sbt.Keys.</span><span style="color:#bf616a;">_
</span><span style="color:#b48ead;">import</span><span> play.Keys.playAssetsDirectories
</span><span style="color:#b48ead;">import</span><span> com.typesafe.sbt.SbtNativePackager.</span><span style="color:#bf616a;">_
</span><span>
</span><span style="color:#b48ead;">object</span><span style="color:#ebcb8b;"> PublicOnFileSystem </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">settings </span><span>= </span><span style="color:#eff1f5;">Seq(
</span><span style="color:#eff1f5;"> mappings in Universal <++= playAssetsDirectories map { </span><span style="color:#bf616a;">directories</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">Seq</span><span style="color:#eff1f5;">[</span><span style="color:#ebcb8b;">File</span><span style="color:#eff1f5;">] </span><span style="color:#b48ead;">=>
</span><span style="color:#eff1f5;"> directories.flatMap { </span><span style="color:#bf616a;">dir</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">File </span><span style="color:#b48ead;">=>
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">directoryLen </span><span>=</span><span style="color:#eff1f5;"> dir.getCanonicalPath.length
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">pathFinder </span><span>=</span><span style="color:#eff1f5;"> dir ** </span><span>"</span><span style="color:#a3be8c;">*</span><span>"
</span><span style="color:#eff1f5;"> pathFinder.get map {
</span><span style="color:#eff1f5;"> </span><span style="color:#bf616a;">publicFile</span><span style="color:#eff1f5;">: </span><span style="color:#ebcb8b;">File </span><span style="color:#b48ead;">=>
</span><span style="color:#eff1f5;"> publicFile -> (</span><span>"</span><span style="color:#a3be8c;">public/</span><span>"</span><span style="color:#eff1f5;"> + publicFile.getCanonicalPath.substring(directoryLen))
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> )
</span><span style="color:#eff1f5;">}
</span></code></pre>
<h2 id="conclusion">Conclusion</h2>
<p>I personaly think that we will more and more use JavaScript even on the server side.
Projects like <a href="http://vertx.io/">Vert.x</a> are interesting because they support this from the beginning.
With Play Framework on the JVM, there is currently <a href="http://openjdk.java.net/projects/nashorn/">a</a> <a href="https://github.com/typesafehub/webdriver">lot</a> <a href="https://github.com/typesafehub/js-engine">of</a> <a href="https://github.com/sbt/sbt-web">effort</a> <a href="http://www.webjars.org/">put</a> to support that.</p>
<p>This proof of concept shows that it is already possible to achieve that.
And I guess it will be even easier in the future.</p>
<p>If you need more information, the <a href="https://github.com/yanns/play-react">code is available on github</a>.</p>
ping conf 20142014-02-17T00:00:00+00:002014-02-17T00:00:00+00:00https://yanns.github.io/blog/2014/02/17/ping-conf-2014/<p>In January, I had the pleasure to participate and give a talk at the <a href="http://www.ping-conf.com/">ping conf</a>.</p>
<p>Ping conf was the first world wide community conference about the <a href="http://www.playframework.com/">Play Framework!</a>.</p>
<p>This conference was very well organized. I'd like to thank all the organizers one more time for this.
It was a good opportunity to meet people I only knew online from the mailing list or on twitter.</p>
<p>The talks were very interesting. Some notices:</p>
<h4 id="javascript-functionality-coming-to-play-2-3-from-christopher-hunt">"Javascript functionality coming to Play 2.3" from <a href="https://twitter.com/huntchr">Christopher Hunt</a></h4>
<p>Typesafe take the JavaScript build chain very seriously and the new <a href="https://github.com/sbt/sbt-web">sbt-web</a> offers a lot of new functionalities.
IMO, one of the most important is the possibility to use NodeJS as JavaScript engine. The performance of the JavaScript build pipeline in Play is similar or even better than the ones based on NodeJS (like Grunt or Gulp)<br/>
<a href="http://huntc.github.io/sbt-web-presentation/#/step-1">slides</a></p>
<h4 id="typesafing-your-blobs-with-scala-from-julien-tournay-and-pascal-voitot">"Typesafing your blobs with Scala" from <a href="https://twitter.com/skaalf">Julien Tournay</a> and <a href="https://twitter.com/mandubian/">Pascal Voitot</a></h4>
<p>Julien and Pascal show us the new <a href="https://github.com/jto/Play20/tree/new_validation_api/documentation/manual/scalaGuide/main/validation">validation API</a>, unifying the Form and JSON validation.
This new API should appear in Play 2.3.<br/>
<a href="http://jto.github.io/articles/play_new_validation_api/">More info</a><br/>
<a href="https://docs.google.com/presentation/d/1bc4437zIO3dUD0cYoSFDbNjrSErY3soURfE5QUErbgw/pub?start=false&loop=false&delayms=3000#slide=id.g11c889a6e_23">slides</a></p>
<h4 id="writing-a-reactive-web-app-with-scala-js-and-reactjs-from-matthias-nehlsen">"Writing a reactive web app with Scala.js and ReactJS" from <a href="https://twitter.com/matthiasnehlsen">Matthias Nehlsen</a></h4>
<p>Matthias, famous for his <a href="http://matthiasnehlsen.com/">blog</a> and his <a href="http://birdwatch.matthiasnehlsen.com/">realtime reactive tweet analysing application</a> talked about the <a href="http://facebook.github.io/react/">ReactJS UI library</a> combined with <a href="http://www.scala-js.org/">ScalaJS</a>.
These two tools can be well combined. ReactJS like immutable data structure, that ScalaJS can provide.</p>
<h4 id="play2-and-redis-when-simplicity-meets-productivity-from-nicolas-martignole">"Play2 and Redis : when simplicity meets productivity" from <a href="https://twitter.com/nmartignole">Nicolas Martignole</a></h4>
<p>Nicolas talked about his experience with Redis and Play. This combination is looking very simple and performant.</p>
<h4 id="building-composable-streaming-testable-play-apps-from-yevgeniy-brikman">"Building composable, streaming, testable Play apps" from <a href="https://twitter.com/brikis98">Yevgeniy Brikman</a></h4>
<p>The "Jim" from LinkedIn talked about how to compose web pages together, using the <a href="https://www.facebook.com/note.php?note_id=389414033919">Facebook's BigPipe concept</a>.
Very interesting way to decompose a web site into small web components.<br/>
<a href="http://de.slideshare.net/brikis98/composable-and-streamable-play-apps">slides</a></p>
<h4 id="play-is-for-performance-from-james-roper-tech-lead-for-play-framework">"Play is for Performance" from <a href="https://twitter.com/jroper">James Roper</a>, tech lead for Play! framework</h4>
<p>Very interesting talk about how to optimize (or not) an asynchronous Play! application.<br/>
<a href="https://github.com/jroper/play-is-for-performance">slides as Play! application</a> (simply check it out and start the Play! application, it is very impressive)</p>
<h4 id="making-the-case-for-play-from-adam-evans-and-asher-glynn-bbc">"Making the case for Play" from <a href="https://twitter.com/ajevans85">Adam Evans</a> and <a href="https://twitter.com/asherglynn">Asher Glynn</a>, BBC</h4>
<p>Adam and Asher talked about pushing change within an organization like the BBC, how they introduced the Play Framework!<br/>
<a href="http://pt.slideshare.net/ajevans/making-the-30191542">slides</a></p>
<h4 id="grant-klopper-software-engineer-at-the-guardian"><a href="https://twitter.com/grantklopper">Grant Klopper</a>, Software engineer at The Guardian.</h4>
<p>Grant made a change during the talk and push it into production, very impressive.</p>
<h4 id="tobias-neef"><a href="https://twitter.com/tobnee">Tobias Neef</a></h4>
<p>Tobias talk about action composition and filter in Play, when to use the first ones and when to use the others.</p>
<h4 id="async-reacting-instead-of-waiting-for-better-times-from-johan-andren">"Async: Reacting instead of waiting for better times" from <a href="https://twitter.com/apnylle">Johan Andrén</a></h4>
<p>Johan talk about asynchronous programming and how it works with Play. A very good introduction!<br/>
<a href="http://de.slideshare.net/johanandren/async-react-dont-wait-ping-conf">slides</a></p>
<hr/>
<p>My talk was about "Structure your Play application with the cake pattern (and test it)"</p>
<p>I talked about how to organize a Play! application written in Scala into components with the cake pattern.
The main goal of these components is to encaspulate and expose services only to others components, and to declare dependencies if needed.</p>
<p>As a side effect, the cake pattern allows to inject dependencies at compile time.
A particularity of this "dependency injection" mechanism is that it does not need any container like Spring or Guice at runtime.</p>
<p><a href="http://www.ustream.tv/recorded/42775808">Video of my talk "Structure your Play application with the cake pattern (and test it)"</a><br/>
<a href="http://de.slideshare.net/yann_s/play-withcake-export2">slides</a></p>
<p>I had great feedbacks, like these ones:</p>
<blockquote>
<p>Leanovate Software Engineer Yann Simon gives the best presentation on the cake pattern I’ve ever seen.</p>
<p>— <a href="http://tech.gilt.com/post/74733533436/a-few-things-about-ping-a-play-framework-conference">Gilt</a></p>
</blockquote><blockquote>
<p>it was great that Yann pointed out the disadvantages as well every step along the way and made it clear how far it’s worth going in various cases.</p>
<p>— <a href="http://csaba.palfi.me/ping-conf-day-1/">Csaba Palfi</a></p>
</blockquote><blockquote>
<p>Currently talking, is astounding the audience with live baking an application, slam dunk videos feat. Sulley - of Monsters fame - and hand drawn presentation slides!</p>
<p>— <a href="http://pingconference.kinja.com/yann-simon-1502530900">kinja</a></p>
</blockquote><blockquote>
<p>All this was presented using funny monster videos and a French-German accent.</p>
<p>— <a href="http://www.mariussoutier.com/blog/2014/01/20/ping-conference-play-edition-day-1/">Marius Soutier</a></p>
</blockquote><blockquote>
<p>the infamous Totoro-defense! <a href="https://twitter.com/search?q=%23pingconf&src=hash">#pingconf</a> <img src="/assets/2014-02-17/BeGeO5rIQAEYVSd.jpg"></p>
<p>— <a href="https://twitter.com/_serif/statuses/423793818857930752">serif</a></p>
</blockquote>
<p>Thanks everybody!</p>
<hr/>
<p>Some people saw that I was using my phone to controll the slideshow during the talk and asked me how I did that.
I was simply using <a href="https://www.libreoffice.org/">LibreOffice Impress</a> for my slides. I controlled the slideshow with <a href="https://play.google.com/store/apps/details?id=org.libreoffice.impressremote&hl=en">Impress Remote app</a> installed on my phone.
It works very well. The app displays the duration of your talk, the current, previous and next slide.</p>
DevOpsDays Berlin 20132013-05-08T00:00:00+00:002013-05-08T00:00:00+00:00https://yanns.github.io/blog/2013/05/08/devopsdays-berlin-2013/<p>direct from DevOpsDay Berlin 2013, my notices:</p>
<p><a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/program/">Program</a>
<a target="_blank" href="http://new.livestream.com/accounts/4051563">Videos</a></p>
<h2 id="day1"><a href="https://yanns.github.io/blog/2013/05/08/devopsdays-berlin-2013/#day1">Day #1</a></h2>
<p>From the <a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/proposals/DevOps3.0/">presentation from Immobilienscout</a> - <i>Marcel Wolf, <a target="_blank" href="https://twitter.com/felixsperling">Felix Sperling</a></i></p>
<ul>
<li>Dev AM rotation: exchange between Dev and Ops</li>
<li>Self service VM</li>
<li>unique configuration server, accessible to anyone</li>
</ul>
<p><a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/proposals/DevOps3.0/">DevTools team at Etsy</a> - <a target="_blank" href="https://twitter.com/mrtazz">Daniel Schauenberg</a></p>
<ul>
<li><a target="_blank" href="http://fr.slideshare.net/mrtazz/devtools-at-etsy">Slides</a></li>
<li><a target="_blank" href="https://github.com/etsy/deployinator">deployinator</a></li>
<li>Each new employee should deploy on her first day.</li>
<li><a target="_blank" href="http://lxc.sourceforge.net/">LXC container</a> for tests</li>
<li>Statistics with <a target="_blank" href="https://github.com/etsy/statsd/">statsd</a>, <a target="_blank" href="https://github.com/etsy/logster">logster</a> and <a target="_blank" href="http://graphite.wikidot.com/">graphite</a></li>
<li>Log streamer with <a target="_blank" href="https://github.com/etsy/supergrep">supergrep</a></li>
<li><a target="_blank" href="https://github.com/etsy">Open Source projects</a></li>
</ul>
<h2 id="day2"><a href="https://yanns.github.io/blog/2013/05/08/devopsdays-berlin-2013/#day2">Day #2</a></h2>
<p><a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/proposals/How%20the%20QA%20team%20got%20Prezi%20ready%20for%20DevOps/">How the QA team got Prezi ready for DevOps</a> - <a target="_blank" href="https://twitter.com/pneumark">Peter Neumark</a></p>
<ul>
<li><a target="_blank" href="http://prezi.com/qmrekeeqqvyf/how-the-qa-team-got-prezi-ready-for-devops/">presentation</a></li>
<li>Like the <a target="_blank" href="https://twitter.com/simon_yann/status/339462592315682816/photo/1">error handling</a>: "When blame inevitably arises, the most senior people in the room should repeat this mantra: if a mistake happens, shame on us for making it so easy to make that mistake". Very similar to risk management culture.</li>
</ul>
<p><a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/proposals/Podularity%20FTW/">Podularity FTW!</a> - <a target="_blank" href="https://twitter.com/tlossen">Tim Lossen</a></p>
<ul>
<li><a target="_blank" href="http://fr.slideshare.net/tim.lossen.de/podularity-ftw">slides</a></li>
<li>team = autonomous cell (even technological stack, product...)</li>
<li>The organization is a supercell, bindings autonomous cells together.</li>
<li><a target="_blank" href="https://twitter.com/simon_yann/status/339461227552047104/photo/1">Lunch roulette</a></li>
<li>Interesting question from the audience about the business continuity: if each team can choose its technological stack, is not it a problem then the team change, and when the new members do not know the new stack he is working with? Tim answered that it was indeed a problem the organization though of, but in practice, it never happened.<br/>I like this approach that I could try to summarize like this: do not spend your time trying to avoid problems, but solve real problem that exist.</li>
</ul>
<p><a target="_blank" href="http://www.devopsdays.org/events/2013-berlin/proposals/How%20we%20built%20and%20deployed%20the%20Honshu%20way/">Island Life: How we built and deployed the Honshū way</a> - Wes Mason</p>
<ul>
<li>from one monolith app to several islands like components</li>
</ul>
<h2 id="other-notices"><a href="#other-notices">Other notices:</a></h2>
<ul>
<li><a target="_blank" href="http://hubot.github.com/">robot for chat room</a> to post more information</li>
<li>secrets are hard to deploy in a secure way</li>
<li>distributed file system: <a target="_blank" href="http://ceph.com/">ceph</a><br/>more info: <a target="_blank" href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.110.4574&rep=rep1&type=pdf">Ceph: A scalable, high-performance distributed file system (2006)</a>
</li>
<li><a target="_blank" href="http://www.docker.io/">doker.io</a> to manage linux containers. The demo was impressive, deploying one version and then another one in a few minutes.</li>
<li>Discussion with <a target="_blank" href="http://www.gutefrage.net/">gutefrage.net</a> who use Scala / <a target="_blank" href="https://github.com/twitter/finagle">Finagle</a> with <a target="_blank" href="http://thrift.apache.org/">Thrift</a>.<br>Storage of statistics with <a target="_blank" href="http://opentsdb.net/">OpenTSDB</a></li>
<li><a target="_blank" href="http://zeroturnaround.com/software/liverebel/">LiveRebel</a>: ZeroTurnaround made a presentation of LiveRebel<br/>
LiveRebel contains some versions of the application.<br/>
These versions can be uploaded, manually or automatically (maven plugin, command line tool...)<br/><br/>
Then LiveRebel can deploy a specific version on production (or staging...)<br/>
For this, an agent is running on each server.<br/><br/>
LiveRebel can deploy to one server, check it with some configured smoke test.<br/>
If the test is successful, the server is activated on the cluster, and the deployment process continues with the next server.<br/><br/>
Liquibase is used to deploy a version to a database.</li>
</ul>
Handling data streams with Play2 and Server-Send Events2012-08-12T00:00:00+00:002012-08-12T00:00:00+00:00https://yanns.github.io/blog/2012/08/12/handling-data-streams-with-play2-and-server-send-events/<h2 id="handling-data-streams">Handling data streams</h2>
<p>As the version 2 of Play! Framework was published, I was very interested in its new capabilities to handle data streams reactively.<br/>
As a technical proof of concept, I wrote a parser that works with chunks of data instead of loading the whole content in memory.
My source was a file containing the geographical coordinates of Wikipedia articles.<br/>
(This file is the result of an experience of Triposo, showing how Wikipedia has spread over the planet since the start of the Wikipedia project. Do not forget to watch the other labs from Triposo, they are great!)<br/></p>
<p>Play2 architecture is based on event, and gives us some tools to work with streams of data:</p>
<ul>
<li><a target="_blank" href="http://www.playframework.com/documentation/2.2.x/Enumerators">Enumerators</a> produce chunk of data</li>
<li><a target="_blank" href="http://www.playframework.com/documentation/2.2.x/Enumeratees">Enumeratees</a> transform these chunks</li>
<li><a target="_blank" href="http://www.playframework.com/documentation/2.2.x/Iteratees">Iteratees</a> consumes these chunks</li>
</ul>
<p>(For more information, you can read:</p>
<ul>
<li><a target="_blank" href="https://gist.github.com/sadache/3072893">Is socket.push(bytes) all you need to program Realtime Web apps?</a> from <a href="https://twitter.com/Sadache">Sadek Drobi, CTO Zenexity</a>.</li>
<li><a target="_blank" href="http://greweb.me/2012/08/zound-a-playframework-2-audio-streaming-experiment-using-iteratees/">Zound, a PlayFramework 2 audio streaming experiment using Iteratees</a> from <a target="_blank" href="https://twitter.com/greweb">Gaetan Renaudeau</a></li>
<li>If you understand french, you can read <a target="_blank" href="http://www.touilleur-express.fr/2012/08/05/realtime-web-application-un-exemple-avec-play2/">Realtime Web Application, un exemple avec Play2</a> from <a target="_blank" href="https://twitter.com/nmartignole">Nicolas Martignole</a>)
</li>
</ul>
<p>The production of data is an Enumerator, sending line after line of the input file:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">def </span><span style="color:#8fa1b3;">lineEnumerator</span><span>(</span><span style="color:#bf616a;">source</span><span>: </span><span style="color:#ebcb8b;">Source</span><span>) : </span><span style="color:#ebcb8b;">Enumerator</span><span>[</span><span style="color:#ebcb8b;">String</span><span>] = {
</span><span> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">lines </span><span>= source.getLines()
</span><span>
</span><span> Enumerator.fromCallback1[</span><span style="color:#ebcb8b;">String</span><span>] ( </span><span style="color:#bf616a;">_ </span><span style="color:#b48ead;">=> </span><span>{
</span><span> </span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">line </span><span>= </span><span style="color:#b48ead;">if </span><span>(lines.hasNext) {
</span><span> Some(lines.next())
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> None
</span><span> }
</span><span> Future.successful(line)
</span><span> }, source.close)
</span><span>}
</span></code></pre>
<p>With an Enumeratee, each line can be possibly parsed into a Coordinate class:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">lineParser</span><span>: </span><span style="color:#ebcb8b;">Enumeratee</span><span>[</span><span style="color:#ebcb8b;">String</span><span>, </span><span style="color:#ebcb8b;">Option</span><span>[</span><span style="color:#ebcb8b;">Coordinate</span><span>]] = Enumeratee.map[</span><span style="color:#ebcb8b;">String</span><span>] { </span><span style="color:#bf616a;">line </span><span style="color:#b48ead;">=>
</span><span> line.split("</span><span style="color:#96b5b4;">\t</span><span>") </span><span style="color:#b48ead;">match </span><span>{
</span><span> </span><span style="color:#b48ead;">case </span><span>Array(</span><span style="color:#bf616a;">_</span><span>, IsDouble(</span><span style="color:#bf616a;">latitude</span><span>), IsDouble(</span><span style="color:#bf616a;">longitude</span><span>)) </span><span style="color:#b48ead;">=> </span><span>Some(Coordinate(latitude, longitude))
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#bf616a;">_ </span><span style="color:#b48ead;">=> </span><span>None
</span><span> }
</span><span>}
</span></code></pre>
<p>The Enumerator can be composed with an Enumeratee with:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">source </span><span>= scala.io.Source.fromFile(Play.getExistingFile("</span><span style="color:#a3be8c;">conf/coosbyid.txt</span><span>").get)
</span><span>lineEnumerator(source) &> lineParser
</span></code></pre>
<p>I know, the method's name "&>" can make some of you go away. Please stay! This sign is like the pipe in bash. It is very easy to understand:</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>lineEnumerator(source) &> lineParser
</span></code></pre>
<p>is the same as (removing infix notation)</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>lineEnumerator(source).&>(lineParser)
</span></code></pre>
<p>which is the same as (method alias)</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span>lineEnumerator(source).through(lineParser)
</span></code></pre>
<p>Use the last form if the first one is not your taste... :)</p>
<p>With a last Enumeratee to produce JSON, I can send the stream directly to the browser with Server Send Events.</p>
<pre data-lang="scala" style="background-color:#2b303b;color:#c0c5ce;" class="language-scala "><code class="language-scala" data-lang="scala"><span style="color:#b48ead;">val </span><span style="color:#bf616a;">source </span><span>= scala.io.Source.fromFile(Play.getExistingFile("</span><span style="color:#a3be8c;">conf/coosbyid.txt</span><span>").get)
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">jsonStream </span><span>= lineEnumerator(source) &> lineParser &> validCoordinate &> asJson
</span><span style="color:#b48ead;">val </span><span style="color:#bf616a;">eventDataStream </span><span>= jsonStream &> EventSource()
</span><span>Ok.chunked(eventDataStream).as("</span><span style="color:#a3be8c;">text/event-stream</span><span>")
</span></code></pre>
<p>What to notice:</p>
<p>Only chunks of data are in memory. The whole content of the source file is never loaded completely.</p>
<p>Each step of the process is isolated in an Enumertor or Enumeratee, making it very easy to modify, to re-use, to combine in a different way.</p>
<p>The Enumerator is reading a file, but you can imagine it could read data from a web service, of from a database.</p>
<h2 id="server-send-events">Server-Send Events</h2>
<p>When we want to send events in "real time" to the browser, what technologies are available?</p>
<ul>
<li>polling: the browser pools the server every x milliseconds to check if there is a new message. This method is not very efficient, because a lot of requests are necessary to give the illusion to update the application in real time.</li>
<li>long-polling (or <a target="_blank" href="http://en.wikipedia.org/wiki/Comet_(programming)">Comet</a>): the browser opens a connection to the server (for example in a iframe), and the server keeps the connection opened. When the server wants to push data to the client, it sends this data with the opened connection. The client receives the data, and opens a connection again for further messages. With this method, the browser is always showing that it is waiting for data. This technology does not scale on threaded system, as each opened connection uses a thread. In JEE environment, we need an asynchronous servlet 3.1 not to make the server exploding.</li>
<li><a target="_blank" href="http://dev.w3.org/html5/eventsource/">Server-Send Events (SSE)</a> are quite similar to Comet. The main difference is that the browser manages this connection. For example, it opens the connection again if it falls.</li>
<li><a target="_blank" href="http://dev.w3.org/html5/websockets/">WebSockets</a> provide a bi-directional, full-duplex communications channels. It is a different protocol than HTTP.</li>
</ul>
<p>I choose to use Server-Send Events instead of WebSockets because of the following reasons:</p>
<ul>
<li>I've already played with WebSockets and wanted to try something new.</li>
<li>WebSockets are great and can communicate in both directions. But this technology is a new protocol, sometimes difficult to integrate in an existing infrastructure (Proxy, Load-Balancer, Firewall...) Server-Send Events, on the other hand, use the HTTP protocol. The PaaS <a target="_blank" href="https://www.heroku.com/">Heroku</a> does not support WebSockets yet, but support SSE. When pushing data from the server to clients is all what you need, SSE can be what is the most appropriate and is <a target="_blank" href="http://caniuse.com/#search=eventsource">well supported</a> (except in IE for the moment)</li>
</ul>
<h2 id="server-send-events-api">Server-Send Events API</h2>
<p>The Javascript API is very simple:</p>
<pre data-lang="javascript" style="background-color:#2b303b;color:#c0c5ce;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#bf616a;">feed </span><span>= new EventSource('</span><span style="color:#a3be8c;">/stream</span><span>');
</span><span>
</span><span style="color:#65737e;">// receive message
</span><span style="color:#bf616a;">feed</span><span>.</span><span style="color:#96b5b4;">addEventListener</span><span>('</span><span style="color:#a3be8c;">message</span><span>', </span><span style="color:#b48ead;">function</span><span>(</span><span style="color:#bf616a;">e</span><span>) {
</span><span> </span><span style="color:#b48ead;">var </span><span style="color:#bf616a;">data </span><span>= JSON.</span><span style="color:#96b5b4;">parse</span><span>(</span><span style="color:#bf616a;">e</span><span>.data);
</span><span> </span><span style="color:#65737e;">// do something with data
</span><span>}, </span><span style="color:#d08770;">false</span><span>);
</span></code></pre>
<h2 id="visualizing-the-results">Visualizing the results</h2>
<p>As the stream is sending coordinates, my first attempt was to display them on a earth in 3D. For this, I used three.js, which was very simple. The first results were promising, but sadly, the browser could not display so much information in 3D. I had to found an alternative.<br/>
My second attempt was to display these coordinates on a 2D canvas, and that worked well, although less impressive that a 3D map... :)<br/>
You can see the result on Heroku: http://wiki-growth.herokuapp.com/<br/>
The code source is available on github: https://github.com/yanns/play2-wiki-growth-sse<br/>
You can run it by yourself with Play and let heroku sleeping.</p>