Zilverline tech blog

Multiple instances in Terraform: count versus foreach

2019-12-06T00:00:00+00:00

A typical use case when ops-ing servers is to rotate servers: Create a new server and remove the old server. Since our servers are managed by Terraform and Ansible this should be an easy exercise. If you know what you are doing…

Introduction

For an existing web-server we needed to update some token with an external api before a specific date. Since we did not want to do an in-place update, since this can break the current live environment, we wanted to add a web-server connect with the external api with a new token and then phase-out the old server.

Rotating servers is a typical use case and fully supported by Terraform if you use the correct stanza. With the help of the Terraform user forum we were able to figure out the correct way for our use case.

Setup

The simplified version of the web-server terraform module:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
 ami = "abc"
 key_name = "ben"
 tags = {
  Name = var.server_name
  ServerRole = "web"
  Env = var.environment
 }
}

Then we can create a single instance per environment as follows:

In terraform/acceptance/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "acceptance"
  server_name = "web01.acceptance.server.nl"
}

When you want to create multiple instances of the same module you can use count_and since 0.12.6 you can also use foreach.

Using count

Since we have some specific values per instance we need to refactor our setup for using count as follows:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
  ami = "abc"
  key_name = "ben"
  count = length(var.server_names)
  private_ip = var.ip_addresses[count.index] 
  tags = {
    Name = var.server_names[count.index]
    ServerRole = "web"
    Env = var.environment
  }
}

Then we can create a single instance as follows:

In terraform/acceptance/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "acceptance"
  server_name = ["web01.acceptance.server.nl"]
  ip_addresses = ["1.2.3.4"]
}

Now this already felt a bit strange to create to lists that are connected by their index.

To create multiple instances we can then simply add a server_name to server_names

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  server_names = ["web01.acceptance.server.nl", "web02.acceptance.server.nl"]
  ip_addresses = ["1.2.3.4", "1.2.3.5"]
}

When you run terraform plan it will propose to create a new server called module.web-server.aws_instance.web_server[1]

So far so good. Now after provisioning the new server we can delete the old server by just removing the server from the server_names list. At least that is what we thought…

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  server_names = ["web02.acceptance.server.nl"]
}

After running terraform plan terraform suggest to destroy one server and re-create (meaning destroy and create another server). As explained in the forum post by Martin Atkins:

The behavior you are describing is a typical result for using count against a list because it causes Terraform to identify the individual instances by the indexes into the list, and so removing an item will assign new indices to all items appearing after it.

This can be “solved” by fiddling around with the terraform state but that is quite tedious and error-prone. So this kind of ruled out using count for rotating servers described in our use case.

Using foreach

The foreach option is much more suitable for our use case. This is because of the fact that Terraform will no longer identify each instance by the index in the list but by a named id.

So we have to refactor the web-server module to use foreach:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
  foreach = var.host_to_ip
  ami = "abc"
  key_name = "ben"
  private_ip = each.value 
  tags = {
    Name = each.key
    ServerRole = "web"
    Env = var.environment
  }
}

To create a single instance using foreach we need to pass in a map from host to ip addresses.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web01.acceptance.server.nl" = "1.2.3.4"
  }
}

When running terraform plan it still wants to destroy the existing instance since its id does not match: module.web-server.aws_instance.web_server versus module.web-server.aws_instance.web_server["web01.acceptance.server.nl"]. In order to overcome this problem we need to import the existing aws instance with the new name and remote the old one only once:

# Import instance under the new name
terraform import module.web-server.aws_instance.web_server[\"web01.acceptance.server.nl\"] YOUR_ID_IN_AWS
# Remove the instance from the current terraform state
terraform state rm module.web-server.aws_instance.web_server

Now when running terraform plan terraform reports no changes! So we are good to go to add a new server and remove the old one.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web01.acceptance.server.nl" = "1.2.3.4"
    "web02.acceptance.server.nl" = "1.2.3.5"
  }
}

Running terraform apply will create web02.acceptance.server.nl. After provisioning, and connecting it to our external api, we can remove the old server by just removing it from the terraform/acceptance/main.tf file and running terraform apply again.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web02.acceptance.server.nl" = "1.2.3.5"
  }
}

Summary

In Terraform 0.12.6 foreach was introduced. This perfectly matches the rotate servers use case and saves a lot of fiddling around with the Terraform state. When ever you define modules that can have multiple instances I suggest you you foreach right from the start so you never have to import and change the state manually.

Terraform 0.12, Ansible and AWS setup

2019-11-22T00:00:00+00:00

Our AWS setup is automated using Terraform and Ansible. We recently upgraded to the next major release Terraform 0.12 and needed to refactor our setup in order to support this new major version. This post is a write up of how to setup Terraform 0.12, AWS and Ansible using dynamic inventory.

Introduction

Before the upgrade our setup was as follows:

Ansible
Terraform 0.11
terraform-inventory

Terraform-inventory was used in order to use the terraform state as dynamic inventory for Ansible. Unfortunately the terraform-inventory library has multiple issues with the latest terraform release. Therefor we needed to figure out how to setup the dynamic inventory again.

Upgrading Terraform

There is a pretty good upgrade guide from Terraform describing which steps to take in order to upgrade to the latest 0.12 version. Since we needed to deal with multiple terraform version we started using Terraform version manager, inspired by rbenv, to easily manage terraform multiple versions. In the end following the upgrade guide did not give us many problems, other than fixing some deprecation warnings.

Terraform setup

We have Terraform and Ansible code in the same git repository. The terraform structure is basically as follows:

# Modules
terraform/modules/web-server

# Config per environment
terraform/acceptance/main.tf
terraform/acceptance/inventory/
terraform/production/main.tf
terraform/production/inventory/

The simplified version of the web-server terraform module:

resource "aws_instance" "web_server" {
 ami = "abc"
 key_name = "ben"
 tags = {
  Name = var.server_name
  ServerRole = "web"
  Env = var.environment
 }
}

Then we can create instances per environment as follows:

In terraform/acceptance/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "acceptance"
  server_name = "web01.acceptance.server.nl"
}

In terraform/production/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "production"
  server_name = "web01.production.server.nl"
}

Ansible setup

Next we want to connect the Terraform setup with our Ansible playbooks in order to provision the servers. The Ansible setup is roughly:

# Ansible setup
group_vars/app-web.yml
group_vars/app-web-acceptance.yml
group_vars/app-web-production.yml
app-web.yml

The app-web.yml

---
- hosts: app_web

# rest omitted

The app-web.yml is the playbook, the group_vars/app-web.yml contains shared variables which can be overridden by the more specific group_vars/app-web-***.yml. This is not something that works out of the box when using dynamic inventory.

In order to have dynamic inventory we choose to use the aws_ec2 inventory source plugin.

To use the aws_ec2 plugin you need to enable the plugins in your ansible.cfg file:

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

Then create a file ending with aws_ec2.yml in your inventory directory. In our case in terraform/acceptance/inventory/

The acceptance.aws.ec2.yml:

plugin: aws_ec2
strict: True
regions:
  - eu-central-1
boto_profile: my-aws-profile-corresponding-to-aws-credentials
keyed_groups:
  - key: tags.ServerRole
    prefix: app
filters:
  tag:Env: acceptance
  vpc-id: vpc-123456
compose:
  ansible_host: private_ip_address

hostnames:
  - tag:Name
  - private-ip-address

The boto_profile points to the name of your aws-profile typically defined in your ~/.aws/credentials file.

The keyed_groups will create groups to be used by Ansible to target multiple hosts at once. In this case it will create a group called app_web.

With filters you can control which servers you want to exist in this inventory file. In our case we create an inventory file per environment which correspond to a vpc-id (Virtual Private Cloud in AWS) and, to be super sure, a tag called Env with the value acceptance.

Finally the compose.ansible_host ensures that Ansible will use the private_ip_address to connect to the server. So you have to be connected to the VPC to be able to provision the servers.

To test our setup run ansible-inventory -i terraform/acceptance/inventory/acceptance.aws_ec2.yml --graph. This displays the hosts per group created by the aws_ec2 plugin:

@all:
  |--@aws_ec2:
  |  |--web01.acceptance.server.nl
  |--@app_web:
  |  |--web01.acceptance.server.nl
  |--@ungrouped:

As expected this created the group defined in keyed_groups called app_web. This of course corresponds to the host group we defined in our Ansible playbook app-web.yml: hosts: app_web.

This however will not load our group_vars files and hierarchy in order to be able to overwrite environment specific variables as mentioned earlier.

Luckily Ansible allows us to use multiple host files, so you can combine static and dynamic inventory. You can specify them each individually like: ansible-inventory -i terraform/acceptance/inventory/acceptance.aws_ec2.yml -i terraform/acceptance/inventory/hosts --graph or just point to a directory: ansible-inventory -i terraform/acceptance/inventory/ --graph.

Please note that the files are read in alphabetical order so to ensure this setup always works as expected it is wise to prefix them for instance with 1.acceptance.aws_ec2.yml and 2.hosts

In terraform/acceptance/inventory/2.hosts

# This will ensure that group_vars/app-web-acceptance.yml 
# overrides variables defined in the group_vars/app-web.yml
[app-web:children]
app-web-acceptance

# This will connect the Ansible world with the aws_ec2 group
# returned from aws, the underscored one.
[app-web-acceptance:children]
app_web

Now when running ansible-inventory -i terraform/acceptance/inventory --graph it will create all the groups with the correct hierarchy we want:

@all:
  |--@aws_ec2:
  |  |--web01.acceptance.server.nl
  |--@app-web:
  |  |--@app-web-acceptance:
  |  |  |--@app_web:
  |  |  |  |--web01.acceptance.server.nl

Final structure

To sum up the final directory structure looks like:

# Terraform
terraform/modules/web-server
terraform/acceptance/main.tf
terraform/acceptance/inventory/1.acceptance.aws_ec2.yml
terraform/acceptance/inventory/2.hosts
terraform/production/main.tf
terraform/production/inventory/2.production.aws_ec2.yml
terraform/production/inventory/2.hosts
# Ansible
group_vars/app-web.yml
group_vars/app-web-acceptance.yml
group_vars/app-web-production.yml
app-web.yml

Event Sourcing: Invariants spanning multiple aggregates

2017-04-21T00:00:00+00:00

This post should mostly stand on its own, but if you find yourself getting lost, you can refer to the post I wrote two weeks ago for some background: Event Sourcing in Elixir.

Introduction

When you are designing your aggregate boundaries in an event sourced system, you are usually thinking about how large or small you want to make your aggregates.

What is an aggregate? In simple terms, it’s a single stack of events, usually referred to by a universal unique identifier (UUID).

Let’s think for a moment about a system in which there is only a single aggregate. In this system, all state is represented by or computed from a single stack of events. Data consistency is never a concern, since all invariants are by definition encapsulated in this one aggregate. A drawback is that all write operations to the system will have to be done consecutively. So one trade-off between large and small aggregates (putting aside the usual concerns that larger aggregates will be more complex) is between having invariants encapsulated in a single aggregate, and enabling concurrent writes.

For this reason, we usually search for ways to break up our system into smaller aggregates.

This begs the question, however, what if you want to have smaller aggregates, but that results in an invariant that will be split across two aggregates?

Example: A Model Bank

Now is a good time to introduce a concrete example, so let’s do that.

Let’s say we want to model a bank. This simplified model bank will allow the following functionality: you can create a bank account, and transfer money from one account to another.

There are a number of invariants that we want to enforce in our model bank.

You cannot withdraw more money from an account than there is money present in the account.
In a money transfer, an equal amount of money must be withdrawn as deposited.

There are two approaches available to us in this scenario:

All bank accounts are grouped in one aggregate.
Each bank account has its own aggregate.

Approach #1 has the benefit that both invariants can be encapsulated in a single aggregate, but all transactions within the system will have to be done consecutively (which will severely limit the eventual throughput of the system).

Approach #2 solves the problem of concurrent updates, but introduces the problem that invariant #2 can not be encapsulated in a single aggregate.

Let’s assume for the purposes of this blog post that approach #1 is out of the question because it would limit the throughput of the system too much.

Enforcing an invariant across aggregate boundaries

The challenge will be to enforce invariant #2 across aggregate boundaries. The most obvious solution is to simply use a transaction of sorts to ensure that both halves of the transaction either complete or fail.

defmodule BankAccountCommandHandler do
  alias Model.BankAccount

  def execute(command = %BankAccount.TransferAmount{}) do
    reference = UUID.uuid4()

    Aggregate.Repository.with_aggregate(BankAccount, command.to_account, fn(to_state) ->
      :ok = Aggregate.Repository.with_aggregate(BankAccount, command.from_account, fn(from_state) ->
        from_state |> BankAccount.withdraw(reference, command.amount)
      end)

      to_state |> BankAccount.deposit(reference, command.amount)
    end)
  end
end

Let’s break this apart line by line. First, we create a unique reference ID for the transaction. The function given to Aggregate.Repository.with_aggregate/3 will be run by the process representing command.to_account. From that process, we will again call Aggregate.Repository.with_aggregate/3 to do the actual work of the transaction. The process for command.to_account will wait for the process representing command.from_account to finish before it continues, guaranteeing that we only deposit the amount if it has already been withdrawn. There is no invariant being checked in BankAccount.deposit/3 so we can safely complete the transaction this way.

Is it safe? Imagine someone pulls the plug on your server right before you deposit the amount to complete the transaction. You should be able to cross-reference every deposit with every withdrawal (with reference_id) to find incomplete transactions and reverse them. It’s not perfect, but it will work.

But there is a drawback to this approach. What if two people try to transfer money at exactly the same time? Let’s imagine one transaction from A to B, and another from B to A. Let’s name our two requests Y and Z. Request Y is going to start by locking process A. Request Z is going to lock process B. Then when request Y tries to lock process B, it has to wait, up to 5s (or whatever timeout you set), because process B is already locked by request Z. It’s a deadlock, and one or both of our transfers will fail!

To illustrate this, I ran 50 transactions, 4 at a time, on 2 bank accounts, and checked the results. It was pretty ugly; of the 50 requested transfers, only 18 of them succeeded. The system was also only able to achieve a throughput of 1.24 requests per second. Increasing the concurrency only results in a higher percentage of requests failing.

A Better Solution

Instead of trying to make the transfer atomic, we can attempt to coordinate the transfer between the two aggregates.

defmodule BankAccountCommandHandler do
  alias Model.BankAccount

  def execute(command = %BankAccount.TransferAmount{}) do
    reference = UUID.uuid4()

    :ok = Aggregate.Repository.with_aggregate(BankAccount, command.from_account, fn(state) ->
      state |> BankAccount.reserve(reference, command.amount)
    end)

    :ok = Aggregate.Repository.with_aggregate(BankAccount, command.to_account, fn(state) ->
      state |> BankAccount.deposit(reference, command.amount)
    end)

    :ok = Aggregate.Repository.with_aggregate(BankAccount, command.from_account, fn(state) ->
      state
      |> BankAccount.cancel_reservation(reference)
      |> BankAccount.withdraw(reference, command.amount)
    end)
  end
end

Using this approach, we first reserve the amount in the account from which we are withdrawing. Then we deposit it in the second bank account. Finally, we cancel the reservation and withdraw the amount for real.

There are more events in this version, but the benefit is that only one aggregate is ever locked at a time, meaning we can avoid deadlocks entirely.

Is it safe? At any point in this process, if someone pulls the plug on our server, we will be able to examine the situation and determine whether a transfer has been completed. Better yet, we don’t have to examine all past transfers, only the “reservations” from each account.

Once again I ran 50 transactions, 4 at a time, on 2 bank accounts. The results: all 50 transactions succeeded, with a throughput of 48 requests per second.

This solution also has the benefit that it’s easier to read and understand, despite the extra events.

I was wondering what would happen if I cranked up the number of concurrent requests. So I ran it again with 800 transfers, 100 at a time. Results: all 800 transactions succeeded, throughput of 90 requests per second.

Final Thoughts

It should be pretty obvious that the second solution can deal with simultaneous requests in a much better way. Whether this is a useful pattern to apply in your own code will depend on whether you expect to see simultaneous requests in your application. I can tell you that this is not a problem for Jortt, where usually only one person per organization is doing bookkeeping at a time.

On the one hand, it’s never bad to be prepared for the worst, but on the other hand, if the naive approach works 99.999% of the time, maybe it’s good enough?

But it’s always good to know that there are clean solutions that work under high load for enforcing invariants across aggregate root boundaries.

React + Redux: our pursuit of performance (part 1)

2017-04-14T00:00:00+00:00

While having dabbled with the critically acclaimed React + Redux combo for the past year, we’ve had our fair share of performance challenges. While React is pretty darn fast out of the box, there comes a time that you’ll run into a brick wall, performance wise. This is a series of posts for those who are in the same boat as we were and will serve as a guide to row this boat safely to shore. We will be sharing our experiences and providing solutions.

Improving performance starts with measuring the current performance. Then apply tweaks and measure again to see if the performance actually improves. This is where React Performance Tools come in handy. It gives you insight in how long it takes to mount & render React components, and even show you ‘wasted’ time, where component renders stayed the same, so the DOM wasn’t touched. There’s even a Chrome Extension! We’ll be using the latter since it’s simple to setup and use.

In this post we’ll show you the steps we’ve taken and the lessons we’ve learned profiling a React app and improving it’s performance. An example app to help visualize the challenges/solutions we cover in this post is available on Github.

A basic example

Consider the following example. An app that displays a list of buttons that highlight when activated.

index.js

import React from 'react';
import { render } from 'react-dom';
import { Provider } from 'react-redux'
import { createStore } from 'redux'
import reducer from './reducer'
import App from './App';

const store = createStore(reducer);

render(
  <Provider store={store}>
    <App />
  </Provider>,
  document.getElementById('root')
);

App.js

import React, { Component } from 'react';
import { connect } from 'react-redux';

import { turnOn, turnOff } from './reducer'
import Button from './Button';

const range = [...Array(5).keys()];

class App extends Component {
  handleButtonClick(index) {
    if (this.isButtonActive(index)) {
      this.props.dispatch(turnOff(index));
    } else {
      this.props.dispatch(turnOn(index));
    }
  }

  isButtonActive(index) {
    return this.props.activeIndices.indexOf(index) !== -1;
  }

  render() {
    return (
      <div>
        {
          range.map(index => {
            return (
              <Button
                key={index}
                index={index}
                onClick={this.handleButtonClick.bind(this)}
                active={this.isButtonActive(index)}
              />
            );
          })
        }
      </div>
    );
  }
}

const mapStateToProps = (state) => {
  return {
    activeIndices: state
  };
}

export default connect(mapStateToProps)(App);

Button.js

export default class Button extends PureComponent {
  render() {
    const { active, index, onClick } = this.props;

    return (
      <button className={active ? 'active' : ''} onClick={() => onClick(index)}>click me</button>
    );
  }
}

reducer.js

const initialState = [];

export default (state = initialState, action = {}) => {
  switch (action.type) {
    case 'ON':
      return state.concat(action.itemIndex);
    case 'OFF':
      const newState = state.slice();
      const index = newState.indexOf(action.itemIndex);
      newState.splice(index, 1);
      return newState;
    default:
      return state;
  }
}

export const turnOn = (index) => {
  return { type: 'ON', itemIndex: index };
}

export const turnOff = (index) => {
  return { type: 'OFF', itemIndex: index };
}

Measure, measure, measure

Install React Perf Chrome extension and add the following to make it work.

App.js (with React Perf)

import React, { Component } from 'react';
...
import Perf from 'react-addons-perf';

export default class App extends Component {
  componentDidMount() {
    window.Perf = Perf;
  }
  ...

}

...

You should now be able to measure your React app’s performance using the React Perf extension. It’s available in Chrome’s DevTools. Let’s see what it displays for our example app after we click one of our buttons:

At this point you’ve probably noticed that we have some wasted renders in our app. Let’s see what we can do about this.

Meet `shouldComponentUpdate()`

The shouldComponentUpdate() lifecycle method allows you to tell your component when to re-render. By default it returns true, meaning it will always re-render a component when (new) props are received. Let’s think for a moment which prop changes we should consider to determine when to re-render one of our buttons. A simple implementation could be:

export default class Button extends Component {
  shouldComponentUpdate(prevProps) {
    return prevProps.active !== this.props.active;
  }

  ...
}

Let’s see what React Perf shows us:

No more wasted Button renders! That’s what we wanted to achieve. Or is it?

While ‘shouldComponentUpdate’ could result in a performance boost, please be careful with implementing it, as slow implementations may have even worse performance impacts than a fast wasted render.

Besides, you need to carefully think about which components should implement this method (if at all) to get the best results. Especially given the fact that child components won’t get updated as well.

Lastly, implementing shouldComponentUpdate in many different components would become cumbersome and difficult to maintain. So let’s look at some alternatives.

Meet `React.PureComponent`

Thus-far our components have extended React.Component. React offers a different class to inherit from: React.PureComponent. To quote it’s documentation:

React.PureComponent is exactly like React.Component but implements shouldComponentUpdate() with a shallow prop and state comparison.

Let’s remove our custom shouldComponentUpdate and extend React.PureComponent.

import React, { PureComponent } from 'react';

export default class Button extends PureComponent {
  ...
}

And let’s see what React Perf tells us:

Darn it! Our wasted renders are back. Lets see what our <Button /> props contain:

index
active
onClick

The onClick function prop is the culprit here. It’s passed to the <Button /> component like so:

<Button
  key={index}
  index={index}
  onClick={this.handleButtonClick.bind(this)}
  active={this.isButtonActive(index)}
/>

Now let’s see what the documentation of the Function.prototype.bind() function states:

The bind() method creates a new function that, when called, has its this keyword set to the provided value, with a given sequence of arguments preceding any provided when the new function is called.

Aha! A new function is created. Every time our <Button /> is rendered. That newly created function is not equal (!==) to the function created in the previous render, resulting in a wasted render.

Luckily there are several ways of setting the ‘this’ context correctly.

Meet `React.createClass`

Although React.createClass is deprecated, it’s worth mentioning for the sake of familiarity. It will automatically bind this to all functions, but you shouldn’t use it.

const App = React.createClass({
  handleButtonClick() {
    // correct 'this' context
  },

  render() {
    return <Button onClick={this.handleButtonClick}>Submit</Button>;
  }
}

Use constructor binding

Another way to set this correctly is by moving the .bind(this) into the class’ constructor. This way the function is only recreated once.

class App extends Component {
  constructor(props) {
    super(props);
    this.handleButtonClick = this.handleButtonClick.bind(this);
  }

  handleButtonClick() {
    // correct 'this' context
  }

  render() {
    return (
      ...
        <Button onClick={this.handleButtonClick}>Submit</Button>
      ...
    );
  }
}

Meet `@autobind`

This is the solution we chose. This autobind decorator will take care of the this binding for you. Without the hassle of having to bind them in the constructor or having to resort to the old React.createClass syntax.

class App extends Component {
  @autobind
  handleButtonClick() {
    // correct 'this' context
  }

  render() {
    return (
      ...
        <Button onClick={this.handleButtonClick}>Submit</Button>
      ...
    );
  }
}

All of these 3 solutions fix our wasted renders.

Final thoughts

Building web apps with React has been a pleasant experience for us thus-far. But as with many technologies there are common pitfalls that can slow down your development as well as your apps performance significantly. We’ve shown you how you can measure your apps performance and gave you some tools to eliminate wasted renders.

This is not where our journey ends. We’ve come along several other challenges and will discuss them in future blog posts.

Event Sourcing in Elixir

2017-04-07T00:00:00+00:00

In this post we will explore how we can use some of Elixir’s more interesting features to implement event sourcing. OTP is the star of this show, enabling us to do some very nice tricks that would be practically impossible with Ruby. OTP stands for Open Telecom Platform, and is Erlang’s answer to the problem of concurrency. OTP relies on immutability of data to provide some very nice abstractions that make working with it very easy.

I will assume that the reader is familiar with OTP, Elixir syntax, and Event Sourcing. If not, I would recommend at least skimming the following links:

Elixir Getting Started - Pattern Matching

What is OTP?

Elixir Getting Started - Agent

Greg Young - CQRS and Event Sourcing - Code on the Beach 2014

General approach

If you are building an event-sourced system in Ruby, the usual way you would process a command is like this:

read all events from an aggregate into memory (possibly use of snapshotting will limit this)
build up internal aggregate state
evaluate write request based on internal state (apply invariants)
write new events to disk
process events (update the view model)

With OTP, we can reduce steps 1 and 2 (which are by far the most expensive parts of the process for large aggregates), reducing the entire operation from O(N) to O(1) for an aggregate that has already been loaded into memory.

This approach also eliminates the “optimistic locking” that you usually see in the DB layer in event sourced solutions. With optimistic locking you generally accept that some requests will fail if they are in a race against one other. With this approach, they will all succeed (unless there is application logic that prevents that).

This is possible if we build our system with the following guidelines:

aggregate state is encapsulated in a process, one process per aggregate
all computation related to a particular aggregate occurs inside that aggregate’s process
aggregate repository maintains a cache of aggregate processes and controls access to them

Event Store

Let’s dive right into the code.

I’ve chosen Redis to back my event store for demonstration purposes but it should be trivial to write an event store using any other storage system. An event store is on its most basic level just a key-value store, with every key pointing to a stack of events.

The EventStore stores events in the event store, and notifies listeners via EventHandler (this will come back later). It has only two functions. commit will add events to a particular aggregate, identified by its uuid. load simply loads events from a particular aggregate.

#lib/event_store.ex
defmodule EventStore do
  @event_store RedisEventStore

  def commit(uuid, events) when is_list(events) do
    @event_store.commit(uuid, (events |> Enum.map(&EventSerializer.serialize/1)))

    notify(uuid, events)
  end
  def commit(uuid, event), do: commit(uuid, [event])

  def load(uuid), do: @event_store.load(uuid) |> Enum.map(&EventSerializer.deserialize/1)

  defp notify(_uuid, []), do: :ok
  defp notify(uuid, [event | events]) do
    EventHandler.handle({uuid, event})
    notify(uuid, events)
  end
end

We also need a serializer to convert our events from structs to strings and back again.

#lib/event_store.ex
defmodule EventSerializer do
  def serialize(event) do
    event |> Map.from_struct |> Map.merge(%{__struct__: to_string(event.__struct__)}) |> Poison.encode!
  end

  def deserialize(serialized_event) do
    serialized_event |> Poison.decode!(keys: :atoms) |> decode()
  end

  def decode(event_map) do
    new_event = event_map |> Enum.reduce(%{}, fn ({key, val}, acc) -> Map.put(acc, key, val) end)
    [Map.fetch!(new_event, :__struct__)] |> Module.concat() |> struct(new_event)
  end
end

Finally, the bit of code that writes and reads from Redis. If you want to implement an event store using another back-end, this is the code you’ll have to adapt.

#lib/event_store.ex
defmodule RedisEventStore do
  def commit(_uuid, []), do: nil
  def commit(uuid, [event|events]) do
    {:ok, _} = Redix.command(:redix, ["RPUSH", uuid, event])
    commit(uuid, events)
  end

  def load(uuid) do
    {:ok, result} = Redix.command(:redix, ~w(LRANGE #{uuid} 0 -1))
    result
  end
end

Note that we are not making use of OTP here. As you will see in a moment, we will isolate each aggregate into its own process, which will provide us with the consistency guarantee that we need.

We have made use of the pattern matching that Elixir affords us to deal with lists of events, and check responses of function calls.

Event Handlers

The event handler is unremarkable. It has only one function, handle, which dispatches an event to multiple handlers. Again we are making use of pattern matching to help us process the list of handlers.

# lib/event_handler.ex
defmodule EventHandler do
  @handlers [CounterEventHandler]

  def handle(_, []), do: :ok
  def handle({uuid, event}, [handler | handlers] \\ @handlers) do
    :ok = handler.handle(event)
    handle({uuid, event}, handlers)
  end
end

The event handler for a simple counter might look like this:

# lib/counter.ex
defmodule CounterEventHandler do
  alias Model.Counter

  def handle(%Counter.Created{aggregate_id: aggregate_id}) do
    %Eslixir.Counter{aggregate_id: aggregate_id, count: 0}
    |> Eslixir.Repo.insert

    :ok
  end

  def handle(%Counter.Incremented{aggregate_id: aggregate_id, count: count}) do
    import Ecto.Query, only: [from: 2, update: 2, where: 2]

    from(c in Eslixir.Counter, where: c.aggregate_id == ^aggregate_id)
    |> Eslixir.Repo.update_all(set: [count: count])

    :ok
  end

  def handle(_), do: :ok
end

Note the last function definition: def handle(_), do: :ok. This is necessary to prevent Elixir from complaining that a particular event handler doesn’t have a function definition for a particular event. So we simply return :ok for all events not otherwise handled by a particular event handler.

Aggregate

The aggregate is the first part of our system that will be built using OTP. It’s important for our strategy to ensure that there is only ever one process running per aggregate. In order to do this, we are making use of Registry, which ensures that there is only one process alive per model / aggregate_id combination. This eliminates race conditions and guarantees data consistency.

#lib/aggregate_root.ex
defmodule Aggregate do
  use GenServer

  def with_aggregate(model, aggregate_id, function) do
    :ok = start_link(model, aggregate_id)
    via_tuple(model, aggregate_id) |> GenServer.call(function)
  end

  def start_link(model, aggregate_id) do
    name = via_tuple(model, aggregate_id)

    case GenServer.start_link(__MODULE__, {model, aggregate_id}, name: name) do
      {:ok, _} ->
        GenServer.cast(name, :finish_init)
        :ok
      {:error, {:already_started, _}} ->
        :ok
    end
  end

  defp via_tuple(model, aggregate_id) do
    {:via, Registry, {:aggregate_repository, {model, aggregate_id}}}
  end

  # callbacks
  #
  def init({model, aggregate_id}) do
    {:ok, {model, %{aggregate_id: aggregate_id}}}
  end

  def handle_cast(:finish_init, {model, state}) do
    events = if state.aggregate_id, do: EventStore.load(state.aggregate_id), else: []

    {:noreply, {model, state_from_events(model, events) || nil}}
  end

  def handle_call(function, _from, {model, state}) do
    case function.({state, []}) do
      {_, {:error, reason}} ->
        {:reply, {:error, reason}, {model, state}}
      {new_state, new_events} ->
        EventStore.commit(new_state.aggregate_id, Enum.reverse(new_events))
        {:reply, :ok, {model, new_state}}
    end
  end

  defp state_from_events(model, events) do
    events
    |> Enum.reduce(nil, fn(event, state) -> model.next_state(event, state) end)
  end
end

The aggregate instance is also a GenServer, capturing the state of the aggregate, and ensuring data consistency by ensuring that all writes for a particular aggregate get executed in a single process. This eliminates race conditions present in typical CRUD applications (which may have multiple “read -> compute new state -> write” processes computing at once).

In start_link, we send an asynchronous message to the aggregate to tell it to finish its own initialization. The aggregate must read its events from the event store and build up its state before it is ready to do any computation. start_link is always run in the Aggregate.Repository process, but we want EventStore.load to execute in the Aggregate.Instance process. This will reduce the amount of computation done in Aggregate.Repository, improving its throughput. Aggregate.Repository is a bottleneck in the system, since every write request will be using it.

Because we are writing this in a functional language, appending to lists is an expensive O(N) operation, while prepending to lists is O(1). This is due to immutability and the structure of linked lists. So while it may seem logical to append new events, we will actually prepend them and then reverse the order later with Enum.reverse.

If the model returns {:error, reason}, then the state reverts to the old state (or more accurately is simply not updated), and nothing is written to the event store. Only when the result of function is a valid new state, does the state get updated and the new events get written to the event store.

In order to have as many aggregates loaded in memory as possible it makes sense to limit the amount of state that will be tracked to only that which is needed to perform business logic. Data that is not used in invariants should not be tracked in the State struct of an aggregate.

The function Aggregate.with_aggregate/3 is worth mentioning. It takes a model, aggregate id, and function as arguments. It calls GenServer.call with the via tuple (which routes the call through Registry) and function as arguments. This sends the function to the aggregate process, which then executes it. This approach eliminates race conditions and guarantees that our data will always be consistent.

Of course, if you run this in a production environment, sooner or later you are going to run out of memory as your system loads more and more aggregates into memory. Setting an upper limit on the number of processes would be a good idea. There are many strategies possible when deciding which processes to kill when you reach the upper limit, but that’s far outside of the scope of this post.

Aggregate Model

#lib/counter.ex
defmodule Model.Counter do
  defmodule State, do: defstruct [:aggregate_id, :count]

  defmodule Create, do: defstruct [:aggregate_id]
  defmodule Increment, do: defstruct [:aggregate_id]

  defmodule Created, do: defstruct [:aggregate_id]
  defmodule Incremented, do: defstruct [:aggregate_id, :count]

  def create({state, new_events}, aggregate_id) do
    if(state) do
      {state, {:error, "Already created"}}
    else
      {state, new_events} |> apply_event(%Created{aggregate_id: aggregate_id})
    end
  end

  def increment({state, new_events}) do
    {state, new_events} |> apply_event(%Incremented{aggregate_id: state.aggregate_id, count: state.count + 1})
  end

  def next_state(%Created{aggregate_id: aggregate_id}, _), do: %State{aggregate_id: aggregate_id, count: 0}
  def next_state(%Incremented{aggregate_id: aggregate_id, count: count}, state), do: %State{state | count: count}

  defp apply_event({state, {:error, reason}}, _), do: {state, {:error, reason}}
  defp apply_event({state, new_events}, event) do
    { next_state(event, state), [event | new_events] }
  end
end

This model is defining a simple counter. Every model has a State struct, that’s what’s going to be held in memory in the Aggregate.Instance process. There are also some commands defined here (Create, Increment) and some events (Created, Incremented).

Model.Counter.create/2 applies Created, but only if the counter has not already been created.

Model.Counter.increment/2 applied Incremented, and increases the count by 1.

Model.Counter.next_state is used to translate events to state. This is used both when the aggregate is being loaded from the event store, and when a new event is applied to the aggregate. This way the state is always kept up to date.

Model.Counter.apply_event/3 simply coordinates setting the new state and prepending the new event.

Command Service

The command service will handle dispatching a command to the right command handler.

#lib/command_service.ex
defprotocol CommandService do
  def execute(command)
end

CommandService is defined using a protocol (since we only want a command to be processed once). Each command will have its own implementation of the CommandService protocol.

#lib/counter.ex
defimpl CommandService, for: Model.Counter.Create do
  alias Model.Counter
  def execute(command) do
    Aggregate.with_aggregate(Counter, command.aggregate_id, fn(state) ->
      state |> Counter.create(command.aggregate_id)
    end)
  end
end

defimpl CommandService, for: Model.Counter.Increment do
  alias Model.Counter
  def execute(command) do
    Aggregate.with_aggregate(Counter, command.aggregate_id, fn(state) ->
      state |> Counter.increment
    end)
  end
end

One execute function can also describe multiple instructions. These are chainable, so the following code is also possible:

def execute(%Counter.Increment2{}) do
  Aggregate.with_aggregate(Counter, aggregate_id, fn(state) ->
    state
    |> Counter.increment
    |> Counter.increment
  end)
end

Conclusion

With Elixir we can easily implement a robust event sourcing system. Thanks to OTP, we can maintain a single, lightweight process per aggregate, that is accessible to the entire Elixir runtime (even transparently across multiple servers). We can avoid the complexity of snapshotting, and increase throughput by requiring fewer database accesses (and in some cases – if an invariant is violated – none), increasing throughput. Perhaps more importantly, because our data is immutable, this can be done without locking, and concurrently across however many processes or servers we would like.

Edited 19 May 2017: Use Registry instead of homemade Aggregate.Repository and implement CommandService as a protocol.

Rearranging and hiding items in a UITableView

2017-03-17T00:00:00+00:00

Creating a TableView just like Apple’s own Mail App where you rearrange and hide Mail Folders is pretty straigthforward, once you know how to do it.

After searching blogs, iOS documentation and Stack Overflow I finally found the (a?) proper way. In this blog I share my findings. I will build a table that can be rearranged and where you can select rows (to hide them).

Part 1 - Rearranging items

Create a Simple TableView first

Create a (Swift) XCode project first with one UITableViewController. Create a UITableViewController inside a UINavigationController and connect it to InterfaceBuilder. Or check out MultiSelectTableView on Github. There are tags for each step in the repo. Run the App and verify everything is in order.

Let’s add some content.

class MultiSelectTableViewController: UITableViewController {

  let toDos = ["Go biking", "Italian Groceries", "Fix heating", "Install dimmer", "Pick up books"]

  override func viewDidLoad() {
    super.viewDidLoad()
  }

  override func numberOfSections(in tableView: UITableView) -> Int {
    return 1
  }

  override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
    return toDos.count
  }

  override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
    let cell = tableView.dequeueReusableCell(withIdentifier: "cell", for: indexPath)

    cell.textLabel?.text = self.toDos[indexPath.row]

    return cell
  }
}

Make the table editable

With just one line of code you can make this work, no need to use InterfaceBuilder. Add the following line to your viewDidLoad:

self.navigationItem.rightBarButtonItem = self.editButtonItem

Hmm great, we can put the table in edit mode. But I don’t want to delete items. Just reorder them. Let’s fix this:

override func tableView(_ tableView: UITableView, editingStyleForRowAt indexPath: IndexPath) -> UITableViewCellEditingStyle {
  return UITableViewCellEditingStyle.none
}

Reorder items in the table

In order to reorder the items you only need to override two methods:

override func tableView(_ tableView: UITableView, canMoveRowAt indexPath: IndexPath) -> Bool {
  return true
}

override func tableView(_ tableView: UITableView, moveRowAt sourceIndexPath: IndexPath, to destinationIndexPath: IndexPath) {
}

Now in edit mode you can rearrange your Todo list:

Store the order of the items

We can now reorder the items, but we need to persist them between views. We can use UserDefaults to do that:

let toDoStore = UserDefaults(suiteName: "MultiSelectTable")
var toDos = [String]()

func initializeToDoList() {
  if let defaults = self.toDoStore?.array(forKey: "toDos") as? [String] {
    self.toDos = defaults
  } else {
    self.toDos = ["Go biking", "Italian Groceries", "Fix heating", "Install dimmer", "Pick up books"]
    self.toDoStore?.set(self.toDos, forKey: "toDos")
  }
}

And initialize in viewDidLoad for example. The only thing left to do is to store the changes when we move items around:

override func tableView(_ tableView: UITableView, moveRowAt sourceIndexPath: IndexPath, to destinationIndexPath: IndexPath) {
  let todo = self.toDos.remove(at: sourceIndexPath.row)
  self.toDos.insert(todo, at: destinationIndexPath.row)
  self.toDoStore?.set(self.toDos, forKey: "toDos")
}

Part 2 - Hiding items

Make items selectable

This is easy again, just add:

self.tableView.allowsMultipleSelectionDuringEditing = true

to your viewDidLoad.

But no items are selected, and by default we want to select everything. We need to tell our cell to do so:

override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
  let cell = tableView.dequeueReusableCell(withIdentifier: "cell", for: indexPath)

  cell.textLabel?.text = self.toDos[indexPath.row]
  if self.isEditing {
    self.tableView.selectRow(at: indexPath, animated: false, scrollPosition: UITableViewScrollPosition.none)
  }

  return cell
}

Almost there. We need to tell the table to redraw itself once it goes into editing mode. Let’s override the default behavior:

override func setEditing(_ editing: Bool, animated: Bool) {
  super.setEditing(editing, animated: animated)
  tableView.setEditing(editing, animated: true)

  self.tableView.reloadData()
}

And get rid of the ugly background color while we’re at it:

    let bgColorView = UIView()
    bgColorView.backgroundColor = UIColor.white
    cell.selectedBackgroundView = bgColorView

Store the selection

Every time we select or deselect an item, we need to store this. Let’s just store all items we want to hide.

var hiddenToDos = Set<String>()

First deal with (de)selection of items:

override func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
  self.hiddenToDos.remove(self.toDos[indexPath.row])
}

override func tableView(_ tableView: UITableView, didDeselectRowAt indexPath: IndexPath) {
  self.hiddenToDos.insert(self.toDos[indexPath.row])
}

And hide them when we’re done editing

Unless editing, we need to hide the items from the tableView, else we need to show all items with their selection:

override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
  var nrOfRows = self.toDos.count
  if !self.isEditing {
    nrOfRows -= self.hiddenToDos.count
  }
  return nrOfRows
}

override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
  let cell = tableView.dequeueReusableCell(withIdentifier: "cell", for: indexPath)

  if self.isEditing {
    cell.textLabel?.text = self.toDos[indexPath.row]
    if !self.hiddenToDos.contains(self.toDos[indexPath.row]) {
      self.tableView.selectRow(at: indexPath, animated: false, scrollPosition: UITableViewScrollPosition.none)
    }
    let bgColorView = UIView()
    bgColorView.backgroundColor = UIColor.white
    cell.selectedBackgroundView = bgColorView
  } else {
    let todosWithHiddenItemsSkipped = self.toDos.filter { todo in
      !self.hiddenToDos.contains(todo)
    }
    cell.textLabel?.text = todosWithHiddenItemsSkipped[indexPath.row]
  }

  return cell
}

At last persist the hidden items

At initialization add:

if let hiddenToDoFromStore = self.toDoStore?.array(forKey: "hidden") as? [String] {
  self.hiddenToDos = Set<String>(hiddenToDoFromStore)
}

And when we change selection update the store:

self.toDoStore?.set(Array(self.hiddenToDos), forKey: "hidden")

And we’re done!

Elixir-y goodness from front to back with Elixirscript

2017-03-17T00:00:00+00:00

Elixir is an amazing language for the backend. Erlangs BEAM VM gives it a battle tested, highly reliable, functional base. Pattern matching is a joy and OTP offers tried and true answers to concurrency, scalability and uptime challenges. Mix in Elixirs pleasant ruby-like syntax, the lovely |> pipeline operator and zero-cost abstractions with hygienic macros and the result is a cocktail I could drink all day, every day.

Isn’t it a shame we have to fall back to the less sophisticated taste of JavaScript when dealing with the frontend? That’s like pouring a dram of Laphroaig An Cuan Mor single malt whisky, then blending it with coke. The blasphemy!

Enter Elixirscript

Thanks to bryanjos’s brainchild Elixirscript you won’t have to. Save for some (current) limitations it lets you write Elixir on the frontend to your heart’s content. While it’s still very much in development you can already do quite a lot with it.

Boilerplate

If you want to get up and running quickly the elixirscript-project-boilerplate will help you on your way. It sets you up with babel and webpack so you can start writing your code instantly.

Phoenix integration

For a full stack experience the elixir_script package integrates elixirscript with the phoenix framework. This hooks in to Phoenix’s brunch configuration to give you automatic recompilation and live reloading in the browser. To configure it we need to make sure the following is present:

in mix.exs

def project do
  [
    app: :my_first_app,
    # Other stuff...
    compilers: [:phoenix, :gettext, :elixir_script] ++ Mix.compilers,
    elixir_script: [
      # These paths will change when phoenix 1.4 is released
      input: "web/static/elixirscript",
      output: "web/static/js/build"
    ]
  ]
end

#...

defp deps do
  [
    # Other dependencies...
    {:elixir_script, "~> 0.26.1"}
  ]
en

in config/dev.exs

config :expoker, Expoker.Web.Endpoint,
  #...
  watchers: [
    node: [
      "node_modules/brunch/bin/brunch", "watch", "--stdin", cd: Path.expand("../assets", __DIR__)
    ],
    mix: ["elixirscript.watch"]
  ]

Elixirscript in practice

Easy interoperability

Calling javascript from elixirscript couldn’t be easier. Logging to console can be done with :console.log("Hello"). This is similar to the way erlang interop works. Using dependencies is also a breeze. Just npm install (or, preferably, yarn add) your dependencies and define them in your mix.exs (or pass them in the command line):

[input: "lib/frontend",
 output: "assets/js/build",
 js_modules: [
   {React, "react"},
   {Socket, "js/socket"}]]]

Reusing code

Elixirscript transpiles your regular .ex files. This allows you to reuse your code between frontend and backend. Do be aware not to use elixir dependencies in shared files. Also left = right matching isn’t quite there yet so you might need to make some small modifications.

A simple usecase for this is validating forms in your frontend before sending them to the server:

defmodule President do
  defstruct name: nil

  def valid?(president) do
    president.name != "Donald Trump"
  end
end

bernie = %President{name: "Bernie Sanders"}

President.valid?(bernie) === true

What would be awesome is when you have an event-sourced system reusing event handlers in the frontend to optimistically apply state while the backend is still processing. But that’s a story for another time!

React views

There is some discussion going on about how to integrate JSX with elixirscript but it’s not there yet. In the meantime it’s trivial to implement a nice DSL using the power of macros as we can see in elixirscript_react (Note that it uses a slightly outdated version of elixirscript). Implementing a DSL for React as r.ex allows you to write react components like this:

def view(model) do
  div do
    form do

      input [type: "text", placeholder: "Name", onChange: &update(&1, :name), value: model.name]
      input [type: "email", placeholder: "Email", onChange: &update(&1, :email), value: model.email]
    end
  end

See the full example in app.ex. Those familiar with Elm will notice some similarities with The Elm Architecture which feels more natural in a functional language than React’s way of doing things.

Looking ahead

I’m looking forward to Elixirscript hitting a higher level of feature parity and stability. As Elixir hits the mainstream lets see if it will get more attention as well. Until now it has mostly been the work of bryanjos. Do checkout the elixirscript organization on Github. There’s the elixirscript repository, the tailored repo implementing elixir style patternmatching in javascript, elixir-estree implementing the AST from the EStree spec and some loaders and example projects.

I’m planning on playing around with elixirscript some more and hope to be able to contribute. Directions I’m interested in:

Targeting WebAssembly
ReactNative + Elixirscript
Code reuse (CQRS/ES)

Building an OAuth2 provider in Node.js

2017-03-17T00:00:00+00:00

Last week I needed a Node.js web application to provide OAuth 2 access tokens under the authorization_code grant. I choose to use the node-oauth2-server, mainly because it is reasonably popular, does not require passport and it looks well-written.

Getting the integration to fully work was a bit of a struggle, since the library is somewhere between version 2.x and 3.x. Or otherwise stated, the library is version 3.x and the documentation is version 2.x. So I decided to write down what I needed to know, it might just help someone else.

In case you’re not familiar with OAuth 2.0, it’s a framework that enabled third-parties to obtain (limited) access to an HTTP service. It’s well described in RFC 6749.

Setup

The HTTP service is built with the Express web framework. There’s the express-oauth-server adapter to integrate node-oauth-server in an Express app.

The examples in this article are based on Express, though you should be able to follow along when using a different web framework.

First we must initialize the OAuth server. Don’t worry about the Model for now, I’ll get back to it in a bit.

const OAuthServer = require("express-oauth-server");
const oauth = new OAuthServer({ model: Model });

Then we mount the authorize and token endpoints, both on POST. A GETrequest on authorize should present your user (resource owner) with a form to authorize or deny the third-party. Note that you must ensure that all OAuth query parameters on the GET request (client_id, redirect_uri, et al) are also available on the POST request.

app.get("/oauth/authorize", function(req, res) {
    // render an authorization form
});
app.post("/oauth/authorize", oauth.authorize());
app.post("/oauth/token", oauth.token());

Then we can require a valid OAuth token on secure areas.

app.use("/secure", oauth.authenticate());

Model

The toughest part was providing the correct model to the OAuthServer. The documentation nor the examples of this model were apt, so mostly I reverse engineered this from the source.

For the authorization_code and refresh_token grants the model must implement the functions listed below.

Function	Signature
`getAccessToken`	`(accessToken): Token`
`getAuthorizationCode`	`(authorizationCode): AuthorizationCode`
`getClient`	`(clientId, ?clientSecret): Client`
`getRefreshToken`	`(refreshToken): Token`
`revokeAuthorizationCode`	`(AuthorizationCode): Boolean`
`revokeToken`	`(Token): Boolean`
`saveAuthorizationCode`	`(AuthorizationCode, Client, User): AuthorizationCode`
`saveToken`	`(Token, Client, User): Token`
`validateScope`	`(scope): Boolean`

getClient gets the clientSecret only when the client authenticates (i.e. when requesting or refreshing a token). When the clientSecret is provided getClient must validate it.

Note that any function may return its result wrapped in a Promise.

The data structures used by these functions are:

Token: {
    accessToken: String,
    accessTokenExpiresAt: Date,
    refreshToken: String,
    refreshTokenExpiresAt: Date,
    user: User
}

AuthorizationCode: {
    code: String,
    scope: String,
    expiresAt: Date,
    redirectUri: String,
    client: Client,
    user: User
}

Client: {
    id: String, // client_id
    clientSecret: String,
    grants: [ String ],
    redirectUris: [ String ]
}

User: {
    id: String
}

By default the generated tokens are the SHA1 hash of 256 random bytes. In case you want different tokens, you can implement generateAccessToken, generateAuthorizationCode, and generateRefreshToken in the model (with signature (): String).

Mapping custom sessions to users

In case the regular authentication flow or your web application does not use bearer tokens (e.g. when using session cookies), you need to implement your own handler to get the current user when POSTing to authorize. This is a function handle that maps a Request to a User (see above for its definition).

For example, load the User from your database by the sessionid:

function loadCurrentUser(req) {
    return db.getUserBySessionId(req.session.sessionid);
}

Then pass it in the authenticateHandler:

app.post("/oauth/authorize", oauth.authorize({
    authenticateHandler: {
        handle: loadCurrentUser
    }
}));

Final thoughts

node-oauth2-server is quite stable and while working my way through I didn’t run into any bugs I couldn’t fix by changing my model. Its assertions help a long way with debugging.

The lack of accurate documentation is made up for by its well-structured source code. You just have to read the source, Luke :)

Share code between mobile and web with react-native-web

2017-03-17T00:00:00+00:00

Nowadays applications often have to support multiple platforms. Besides a web-application, there is also a need for mobile applications on multiple platforms. Therefore a developer requires knowledge of webdevelopment, iOS development, Android development and probably a lot of time. This means projects often require multiple teams of developers, working on the same application for different platforms with multiple codebases. Wouldn’t it be great to have a single shared codebase that would work on both mobile and web?

React is a component-based Javascript library to create interactive user interfaces in the DOM. React Native lets you build native mobile apps with Javascript in a similar way to React. In React Native it is already possible to have a single codebase which is able to create both native apps for iOS and Android. So instead of building an app in both Swift and Java, a developer can write React code. A technique a webdeveloper is often familiar with nowadays. React Native is already used in production for Facebook, Instagram, Airbnb apps and many others.

So, imagine you are using React Native and also creating a web application with React. You will probably encounter a lot of duplication between those applications. React Native uses <View> and <Text> components, while React just uses <div> and <span> elements. This is where the React Native Web library comes in handy. This will translate your React Native components to HTML elements which will render in your browser. So, simply said a <View> component will become a <div> and a <Text> element will become a <span> element.

Setup

Below is an example of a simple webpack config suitable for React Native Web. The most important is the last part (react-native: react-native-web). An alias is created for React Native, so that all the React Native components are delegated to React Native Web.

const path = require('path');
const webpack = require('webpack');

module.exports = {
  devServer: {
    contentBase: path.join(__dirname, 'src')
  },
  entry: [
    path.join(__dirname, '../index.web.js')
  ],
  module: {
    loaders: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        loaders: [
          'react-hot',
          'babel-loader?cacheDirectory=true'
        ]
      },
      {
        test: /\.(gif|jpe?g|png|svg)$/,
        loader: 'url-loader',
        query: { name: '[name].[hash:16].[ext]' }
      }
    ]
  },
  output: {
    filename: 'bundle.js'
  },
  plugins: [
    new webpack.DefinePlugin({
      'process.env.NODE_ENV': JSON.stringify(process.env.NODE_ENV || 'development')
    }),
    new webpack.optimize.DedupePlugin(),
    new webpack.optimize.OccurenceOrderPlugin()
  ],
  resolve: {
    alias: {
      'react-native': 'react-native-web'
    }
  }
};

For more detailed instructions visit the React Native Web Github repository.

Generated DOM

React Native Web generates your DOM based on your React Native components.

As you can see React Native Web generates a class for every styling property. This will not result in a concise, clean and uncluttered DOM. Luckily the React developer console will give you a better overview.

Animations

React Native Web also supports some animations provided by React Native. So even your animations will still work in your browser. Nice!

Missing modules and views

React Native Web is still in an initial phase. This means not all the modules and views from React Native are released yet for the web counterpart. It is not even possible to implement all the components for the web. Simply because some functionality is absent for the web. But there are some workarounds for this problem. Like in React Native you can guard the use of some code by using Platform.OS which equals 'web' for React Native Web. For complete components you are able to create a separate file with the same component name. So imagine you have a component called Counter.js you can generate a file called Counter.web.js. When importing Counter the correct file will automatically be imported based on the platform extension.

Conclusion

React Native Web is interesting as it is possible to share a lot of code between your platforms. However, it will never be completely shareable between platforms and the need of custom components will still exist. Luckily this is fairly easy to achieve. When you like a clean DOM this is probably not a suitable solution, because the DOM generated by React Native Web is very polluted. React Native Web is certainly something to keep an eye on, but it seems too early to use it for most projects.

Handling large result sets from postgresql in active-record

2017-01-05T00:00:00+00:00

There are several ways for handling large result sets from the database in your application. Each way has its own limitations and pros and cons. All of them have the advantage of reducing memory footprint. In this blogpost we will discuss three ways and see how they compare.

For this blog we will generate a set of test data containing a couple of Orders with some LineItems. For demonstration purposes we generate a small test data set, but it basically behaves the same for thousands of rows.

(1..5).each do |i|
  order = Order.create!(description: "Order #{i}")
  (1..3).each do |j|
    LineItem.create!(order: order, description: "Line item #{j} for order #{i}")
  end
end

Batches.find_each

Amongst similar methods active record by default provides find_each. This means you won’t need any other dependencies when handling large result sets! Downside is you can’t define a custom order since it relies on the id column for querying the batches.

Order.find_each(batch_size: 2) do |order|
  puts order.description
  order.line_items.each {|line_item| nil; }
end

Results in the following queries (I cleaned the log a bit for readability):

SQL: SELECT  "orders".* FROM "orders"  ORDER BY "orders"."id" ASC LIMIT 2
>>> order 2210 Order 1
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2210]]
>>> order 2211 Order 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2211]]
SQL: SELECT  "orders".* FROM "orders" WHERE ("orders"."id" > 2211)  ORDER BY "orders"."id" ASC LIMIT 2
>>> order 2212 Order 3
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2212]]
>>> order 2213 Order 4
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2213]]
SQL: SELECT  "orders".* FROM "orders" WHERE ("orders"."id" > 2213)  ORDER BY "orders"."id" ASC LIMIT 2
>>> order 2214 Order 5
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2214]]

The first query active record does is SELECT "orders".* FROM "orders" ORDER BY "orders"."id" ASC LIMIT 2. As stated above active record uses the id to create batches of 2 using > in the next batch of 2: SELECT "orders".* FROM "orders" WHERE ("orders"."id" > 2152) ORDER BY "orders"."id" ASC LIMIT 2. It uses LIMIT to only retrieve the batch_size number of orders.
As stated, the downside of using id is you can’t define the order of the query. If you add an order clause it will log a message saying Scoped order and limit are ignored, it's forced to be batch order and batch size and the order is ignored.
Another problem using this approach is that it creates a n+1 problem since for each order it will do a query to the database to fetch the child relations. With real large result sets this will flood your database. Luckily this is easily fixed by fetching the child relations eagerly using includes:

Order.includes(:line_items).find_each(batch_size: 2) do |order|
  puts order.description
  order.line_items.each {|line_item| nil; }
end

Results in the following queries (without n+1 problem):

SQL: SELECT  "orders".* FROM "orders"  ORDER BY "orders"."id" ASC LIMIT 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2210, 2211)
>>> order 2210 Order 1
>>> order 2211 Order 2
SQL: SELECT  "orders".* FROM "orders" WHERE ("orders"."id" > 2211)  ORDER BY "orders"."id" ASC LIMIT 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2212, 2213)
>>> order 2212 Order 3
>>> order 2213 Order 4
SQL: SELECT  "orders".* FROM "orders" WHERE ("orders"."id" > 2213)  ORDER BY "orders"."id" ASC LIMIT 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2214)
>>> order 2214 Order 5

Conclusion

No custom ordering possible
Has support for includes

Gem postgresql_cursor

The postgresql_cursor is a nice gem that brings real postgresql cursors to active record. In order to use just require the gem and it will add several methods to you models for iterating over result sets. In this example we use each_instance:

Order.each_instance(block_size: 2) do |order|
  puts ">>> order #{order.id} #{order.description}"
  order.line_items.each {|line_item| nil; }
end

Results in the following statements:

SQL: BEGIN
SQL: declare cursor_b1d5a08826e84f2693ce31dfd5b746e8 cursor for SELECT "orders".* FROM "orders"
SQL: fetch 2 from cursor_b1d5a08826e84f2693ce31dfd5b746e8
>>> order 2225 Order 1
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2225]]
>>> order 2226 Order 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2226]]
SQL: fetch 2 from cursor_b1d5a08826e84f2693ce31dfd5b746e8
>>> order 2227 Order 3
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2227]]
>>> order 2228 Order 4
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2228]]
SQL: fetch 2 from cursor_b1d5a08826e84f2693ce31dfd5b746e8
>>> order 2229 Order 5
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2229]]
SQL: fetch 2 from cursor_b1d5a08826e84f2693ce31dfd5b746e8
SQL: close cursor_b1d5a08826e84f2693ce31dfd5b746e8
SQL: COMMIT

Here we see that the postgresql_cursor creates a postgresql cursor on the fly. Also a transaction is started since the cursor is created without the WITH HOLD directive. This means that all resources are cleaned after the transaction completes automatically. See the postgresql documentation on declare for more information. Using the postgresql_cursor we can now add custom ordering to our query like:

Order.order('description desc').each_instance(block_size: 2) do |order|
  puts ">>> order #{order.id} #{order.description}"
  order.line_items.each {|line_item| nil; }
end

Results in the following statements:

SQL: BEGIN
SQL: declare cursor_0bca06d6e41a436e8b96f8dc765cfa38 cursor for SELECT "orders".* FROM "orders"  ORDER BY description desc
SQL: fetch 2 from cursor_0bca06d6e41a436e8b96f8dc765cfa38
>>> order 2259 Order 5
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2259]]
>>> order 2258 Order 4
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2258]]
SQL: fetch 2 from cursor_0bca06d6e41a436e8b96f8dc765cfa38
>>> order 2257 Order 3
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2257]]
>>> order 2256 Order 2
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2256]]
SQL: fetch 2 from cursor_0bca06d6e41a436e8b96f8dc765cfa38
>>> order 2255 Order 1
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" = $1  [["order_id", 2255]]
SQL: fetch 2 from cursor_0bca06d6e41a436e8b96f8dc765cfa38
SQL: close cursor_0bca06d6e41a436e8b96f8dc765cfa38
SQL: COMMIT

Conclusion

Support for custom ordering
No support for includes which results in n+1 problem when traversing child relations

A custom postgresql cursor

Since I recently needed both requirements (support for n+1 and ordering) I combined the two approaches and created a cursor to fit my needs.

Cursor.new(Order.order('description desc').includes(:line_items), fetch_size: BATCH_SIZE).find_each do |order|
  puts ">>> order #{order.id} #{order.description}"
  order.line_items.each {|line_item| nil; }
end

Without going into the internals this cursor basically wraps the query that is passed in and creates a postgresql cursor based on the query requirements. To be able to do this it inspects the active record internals and constructs the queries. It preloads the relations by using the ActiveRecord::Associations::Preloader the same way active record uses it.

This results in the following statements:

SQL: BEGIN
SQL: declare mycursor3cf9b668771e4ef988a7c989680b104c cursor for SELECT "orders".* FROM "orders"  ORDER BY description desc
SQL: fetch 2 in mycursor3cf9b668771e4ef988a7c989680b104c
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2259, 2258)
>>> order 2259 Order 5
>>> order 2258 Order 4
SQL: fetch 2 in mycursor3cf9b668771e4ef988a7c989680b104c
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2257, 2256)
>>> order 2257 Order 3
>>> order 2256 Order 2
SQL: fetch 2 in mycursor3cf9b668771e4ef988a7c989680b104c
SQL: SELECT "line_items".* FROM "line_items" WHERE "line_items"."order_id" IN (2255)
>>> order 2255 Order 1
SQL: fetch 2 in mycursor3cf9b668771e4ef988a7c989680b104c
SQL: close mycursor3cf9b668771e4ef988a7c989680b104c
SQL: COMMIT

Conclusion

Support for custom ordering
Has support for includes
Limited query methods available

Zilverline tech blog

Multiple instances in Terraform: count versus foreach

Introduction

Setup

Using count

Using foreach

Summary

Terraform 0.12, Ansible and AWS setup

Introduction

Upgrading Terraform

Terraform setup

Ansible setup

Final structure

Event Sourcing: Invariants spanning multiple aggregates

Introduction

Example: A Model Bank

Enforcing an invariant across aggregate boundaries

A Better Solution

Final Thoughts

React + Redux: our pursuit of performance (part 1)

A basic example

index.js

App.js

Button.js

reducer.js

Measure, measure, measure

App.js (with React Perf)

Meet shouldComponentUpdate()

Meet React.PureComponent

Meet React.createClass

Use constructor binding

Meet @autobind

Final thoughts

Event Sourcing in Elixir

General approach

Event Store

Event Handlers

Aggregate

Aggregate Model

Command Service

Conclusion

Rearranging and hiding items in a UITableView

Part 1 - Rearranging items

Create a Simple TableView first

Make the table editable

Reorder items in the table

Store the order of the items

Part 2 - Hiding items

Make items selectable

Store the selection

And hide them when we’re done editing

At last persist the hidden items

Elixir-y goodness from front to back with Elixirscript

Enter Elixirscript

Boilerplate

Phoenix integration

Elixirscript in practice

Easy interoperability

Reusing code

React views

Looking ahead

Building an OAuth2 provider in Node.js

Setup

Model

Mapping custom sessions to users

Final thoughts

Share code between mobile and web with react-native-web

Setup

Generated DOM

Animations

Missing modules and views

Conclusion

Handling large result sets from postgresql in active-record

Batches.find_each

Gem postgresql_cursor

A custom postgresql cursor

Meet `shouldComponentUpdate()`

Meet `React.PureComponent`

Meet `React.createClass`

Meet `@autobind`