Trifork Blog

Banner Elixir

Category ‘From The Trenches’

Collecting data from a private LoRaWAN sensor network into Elastic

May 20th, 2016 by
(http://blog.trifork.com/2016/05/20/collecting-data-from-a-private-lorawan-sensor-network-into-elastic/)

Introduction to LoRaWAN and ELK

Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc … ?

LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that can connect to one network and the relatively big range that can be covered with one LoRa router. One gateway can coordinate around 20’000 nodes in a range of 10–30km. It’s a very flexible protocol and allows the developers build various types of network architectures according to the demand of the client. The general description of the LoRaWAN protocol together with a small tutorial are available in my previous post.

What is the ELK stack, and why use it with LoRaWAN?

In the figure above, you can see a simplified model of what a typical LoRaWAN network looks like.
As you can see, the data from the LoRa endpoints, has to go through several devices before it reaches the back end application. Nowadays there are a lot of tools that would allow us to gather and manipulate the data. A very good solution is the ELK stack which consists of Elasticsearch, Logstash and Kibana; these three tools allow to gather, store and analyze big amounts of data. More information and details can be found on the official website: https://www.elastic.co/.

Read the rest of this entry »

From The Trenches: LoRa, LoRaWAN tutorial with the LoRaBee

March 4th, 2016 by
(http://blog.trifork.com/2016/03/04/from-the-trenches-lora-lorawan-with-the-lorabee/)

Once in a while you stumble across an interesting technology which is an enabler of new solutions, and new business cases. At Trifork Eindhoven we believe LoRa / LoRaWan might be such a technology. We are currently working on several Internet of Things solutions which could directly benefit from long distance communication, with a low energy footprint.

In this document I will describe the things that I learnt while working with the LoRaBee development kit. I will also provide some basic tutorials on how to set it up, how to start a connection and how to connect it with different platforms.
The hardware that I am using:

  • SODAQ Mbili with Atmel 1284p Microcontroller x 2
  • SODAQ LoRaBee which contains the Microchip RN2483 Module x 2
  • A set of peripherals for data acquisition ( sensors, buttons, LEDs, serial interfaces… )

Quick intro into LoRa

According to Wikipedia:

LoRaWAN (Long Range Wide Area Network) is a low power wireless networking protocol designed for low-cost secure two-way communication in the Internet of Things (IoT). LoRaWANs use of sub-GHz ISM bands also means the network can penetrate the core of large structures and subsurface deployments within a range of 2 km.[1]

The technology utilized in a LoRaWAN network is designed to connect low-cost, battery-operated sensors over long distances in harsh environments that were previously too challenging or cost prohibitive to connect. With its penetration capability, a LoRa gateway deployed on a building or tower can connect to sensors more than 10 miles away or to water meters deployed underground or in basements. The LoRaWAN protocol offers unique and unequaled benefits in terms of bi-directionality, security, mobility and accurate localization that are not addressed by other LPWAN technologies. These benefits will enable the diverse use cases and business models in deployments of LPWAN IoT networks globally.”

In the following picture, you can see the typical LoRaWAN network:

As you can see, all three classes are required for a fully working LoRaWAN network. In the tutorial, we used two LoRa end node devices for a P2P communication; however that is a simple radio transmit/receive between the LoRa modules and not the LoRaWAN protocol.

Read the rest of this entry »

Spring-AMQP and payload validation: some notes from the trenches

February 29th, 2016 by
(http://blog.trifork.com/2016/02/29/spring-amqp-payload-validation/)

It’s been a while since I’ve written one of our from-the-trenches blogs: that’s mostly because I’ve been very busy in those trenches developing systems for our customers.

This week I completed a Spring Boot-based microservice which is responsible for interacting with some 3rd party SOAP service: its own clients communicate with it by sending request message over RabbitMQ, and the service then sends back a response to a response queue after handling the SOAP response message.

Of course I used Spring-AMQP to build this service. Spring-AMQP supports a nice annotation-based listener programming model, based on Spring’s generic Message support.
That allows writing listener methods like this:

@RabbitListener(queues = REQUEST_QUEUE) 
public DeclarationResponse submitDeclaration(DeclarationRequest request) { 
  // handle the request and return a response 
}

The request parameter here is the result of converting the incoming AMQP message using a Spring-AMQP MessageConverter, after which it is considered to be the payload of the message (even when headers are used in the conversion as well).

The request messages that the clients send have some required fields: without those fields, the service can’t make the SOAP calls. While reading the RabbitListener JavaDoc I noticed that Spring-AMQP allows you to apply validation to message payload parameters by annotating it. When using this, you also have to add the @Payload annotation (which is optional without validation if your method doesn’t have any other arguments), so the result looks like this:

@RabbitListener(queues = REQUEST_QUEUE)
public DeclarationResponse submitDeclaration(@Valid @Payload DeclarationRequest request) { … }

By the way, Spring’s own @Validated (even as a meta-annotation) and in fact every annotation whose name starts with “Valid” are supported for this purpose as well.

Now we can add some JSR-303 Bean Validation annotations to the fields in our DeclarationRequest, like @NotNull, to express our validation constraints.

Read the rest of this entry »

City-wide crowd management in Amsterdam

November 10th, 2015 by
(http://blog.trifork.com/2015/11/10/city-wide-crowd-management-in-amsterdam/)

As most residents and visitors of Amsterdam know, every year more people are visiting Amsterdam, city wide events like GayPride, Koningsdag and MuseumNacht are getting bigger and more frequent, putting more strain on the city’s infrastructure and all people living in the city center.

That’s why this November 7th, Amsterdam Marketing organized the Museumn8 hackathon to allow developers to come up with creative and innovative solutions for improving improving mobility, navigation and crowd management in the city. Twenty teams eventually participated.

Trifork (Rienk Prinsen, Marleine van Kampen, Marijn van Zelst) and weCity (David Kat, Luc Deliance) teamed up and joined the hackathon to give their take on solving this problem. Their solution:

city-live-logo

By transforming the advertisement billboards of JCDecaux into large information screens displaying real-time information, visitors can get informed about activities and interesting places in the vicinity of the billboard. They receive live crowd information, travel times to and queue lengths at museums and even recommendations where to go next. Read the rest of this entry »

Large organizations are just not set up to be agile

September 21st, 2015 by
(http://blog.trifork.com/2015/09/21/large-organizations-are-just-not-set-up-to-be-agile/)

There is something going wrong in the organization I am currently part of an onsite project, yet I can’t get my head around the specifics.

This very large governmental organization is growing agile initiatives all over the IT department. Development teams are having fun talking to their customers and are trying to create the best solutions to the needs of their stakeholders, yet everything agile about Agile halts when a shippable increment is ready to begin its journey through the rest of the organization.

A few weeks back I received an email from the team that is tasked with the becoming agile of our organization, asking me to fill out a form so I could show how SCRUM we were. I know it was meant to let us see for ourselves how agile we were, but to me it felt as if we were proving our agility through a checklist.

I know I checked every box there was, so according to the person sending it we were 100% Scrum. I just knew we weren’t agile, so there must be a flaw in checking of the Scrum checklist to see if you are agile.

Read the rest of this entry »

A wrinkle in time

June 30th, 2015 by
(http://blog.trifork.com/2015/06/30/a-wrinkle-in-time/)

Leap second

I clearly remember the morning of Sunday July 1, 2012, almost three years ago. I was at church, actually, when I got a call from one of our clients: “The website doesn’t seem to be working.” All I could check at that point was that, indeed, the website was not responding. So I called our sysadmin, who found that even SSH-ing into the machine running the site was taking much longer than usual. Finally, restarting everything solved the problem, but we were still unsure about what had happened.

As we found out later, it was related to the leap second, a phenomenon I had not even heard of until then. Read the rest of this entry »

Axon from the trenches: how to keep your code compatible with legacy events and Sagas

June 8th, 2015 by
(http://blog.trifork.com/2015/06/08/axon-from-the-trenches-how-to-keep-your-code-compatible-with-legacy-events-and-sagas/)

Imagine you’re using Axon to run an event sourcing application. Your production event store might contain millions of events, created in various versions of the application. Wouldn’t it be nice to know for sure that the latest version of your application plays nicely with all your production events and Sagas, including those from previous versions? Well, you can check for that, and it is fairly easy. Read the rest of this entry »

Documenting your REST APIs

May 8th, 2015 by
(http://blog.trifork.com/2015/05/08/documenting-your-rest-apis/)

Whenever you deliver some API that is to be consumed by another party, you will get the inevitable question of providing documentation. Probably every developer’s least favorite task.

In Java there is javadoc, but that doesn’t cut it if you are delivering a Web Service API. In that realm we already know WSDL for SOAP based Web Services. Then again, every developer seems to prefer REST based Web Services these days and those are not WSDL based… So what then? That is a question with multiple answers. In the last few years there have been three different open-source projects that have tried to give an answer to this: swagger, RAML and API BluePrint. Of those Swagger has been around the longest and arguably gained the largest following.

Based on the completely subjective criteria ‘it needs to support Java’, ‘what about Spring MVC?’ and ‘can you deliver it to the customer by Monday?’ I decided to take a stab at documenting an existing API using Swagger. It is written in something that can run on a JVM, has Spring MVC support (via third party libraries) and seemed to be the easiest to set up based on their examples and various other online resources.

Read the rest of this entry »

Dealing with NodeNotAvailableExceptions in Elasticsearch

April 8th, 2015 by
(http://blog.trifork.com/2015/04/08/dealing-with-nodenotavailableexceptions-in-elasticsearch/)

tl;dr

Elasticsearch provides distributed search with minimal setup and configuration. Now the nice thing about it is that, most of the time, you don’t need to be particularly concerned about how it does what it does. You give it some parameters – “I want 3 nodes”, “I want 3 shards”, “I want every shard to be replicated so it’s on at least two nodes”, and Elasticsearch figures out how to move stuff around so you get the situation you asked for. If a node becomes unreachable, Elasticsearch tries to keep things going, and when the lost node appears and rejoins, the administration is updated so everything is hunky-dory again.

The problem is when things don’t work the way you expect…

Computer says “no node available”

Read the rest of this entry »

Developing .NET software on Linux with Mono

February 19th, 2015 by
(http://blog.trifork.com/2015/02/19/developing-dotnet-software-on-linux-with-mono/)

The motivation

The obvious question here is why would you want to develop .NET software on Linux or for Linux? At the risk of sounding like throwing buzzwords around, I will say it is because Linux dominates the cloud completely. Many cloud-related technologies such as Docker, Mesos, and others build on Linux as a base. Sure, it is possible to run Windows in the cloud one way or another, but it is really hard to match the flexibility of Linux, especially when running more than just a few instances.

Quite recently Microsoft announced open-sourcing of .NET Core paving new grounds for a truly cross-platform .NET implementation. It has already been possible to run a lot of .NET software on Linux and OSX for quite some time on an independent .NET implementation called Mono, and now Microsoft is saying that they will work with the Mono project on a common code base that will eventually become the .NET core. In fact, Microsoft has been close to Xamarin, a company behind Mono, for a while now, so this step is not that surprising.

But how usable is Mono right now? That is what I set out to find out in my little experiment. Read the rest of this entry »