Trifork Blog

Axon Framework, DDD, Microservices

Smart energy consumption insights with Elasticsearch and Machine Learning

15:07 on August 21st, 2017 by
No Comments

At home we have a Youless device which can be used to measure energy consumption. You have to mount it to your energy meter so it can monitor energy consumption. The device then provides energy consumption data via a RESTful api. We can use this api to index energy consumption data into Elasticsearch every minute and then gather energy consumption insights by using Kibana and X-Pack Machine Learning.

The goal of this blog is to give a practical guide how to set up and understand X-Pack Machine Learning, so you can use it in your own projects! After completing this guide, you will have the following up and running:

  • A Complete data pre-processing and ingestion pipeline, based on:
    • Elasticsearch 5.4.0 with ingest node;
    • Httpbeat 3.0.0.
  • An energy consumption dashboard with visualizations, based on:
    • Kibana 5.4.0.
  • Smart energy consumption insights with anomaly detection, based on:
    • Elasticsearch X-Pack Machine Learning.

The following diagram gives an architectural overview of how all components are related to each other:

Read the rest of this entry »

Posted in: Docker | Elasticsearch | Machine Learning

Heterogeneous microservices

13:52 on July 13th, 2017 by
No Comments

Heterogeneous microservices

Microservices architecture is increasingly popular nowadays. One of the promises is flexibility and easier working in larger organizations by reducing the amount of communication and coordination between teams. The thinking is, teams have their own service(s) and don’t depend on other teams, meaning they can work independently, therebty reducing coordination efforts.

Especially with multiple teams and multiple services per team, this can mean there are quite a few services with quite different usage. Different teams can have different technology preferences, for example because they are more familiar with the one or the other. Similar different usage can mean quite different requirements, which might be easier to fulfill with the one or the other technology.

The question i’m going to discuss in this blog post, how free or constrained should technology choices be in such an environment?

Read the rest of this entry »

Posted in: Custom Development

Interview with Sam Newman, author of Building Microservices

13:15 on July 6th, 2017 by
1 Comment

After living in Australia for the last five years, the Londoner, and author of the well-received Building Microservices, has returned home to focus on his business as an independent consultant. We caught up via Skype to discuss his upcoming visit to Amsterdam and the tech trends that he is keeping his eye on.

Reading time: Less than 5 minutes

Read the rest of this entry »

Tags: , , , , , , , , , , ,

Posted in: Knowledge | Newsletter | Training

How to send your Spring Batch Job log messages to a separate file

10:09 on April 14th, 2017 by
1 Comment

In one of my current projects we’re developing a web application which also has a couple of dozen batch jobs that perform all sort of tasks at particular times. These jobs produce quite a bit of logging output when they’re run, which is important to see what has happened during a job exactly. What we noticed however, is that the batch logging would make it hard to quickly spot the other logging performed by the application while also running a batch job. In addition to that, it wasn’t always clear in the context of what job a log statement was issued.
To address these issues I came up with a simple solution based on Logback Filters, which I’ll describe in this blog.

Logback Appenders

We’re using Logback as a logging framework. Logback defines the concept of appenders: appenders are responsible for handling the actual log messages emitted by the loggers in the application by writing them to the console, to a file, to a socket, etc.
Many applications define one or more appenders and them simply list them all as part of their root logger section in the logback.xml configuration file:

<configuration scan="true">

  <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>logstash-server</destination>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>log/server.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>log/server.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %mdc %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>
  <root level="info">
    <appender-ref ref="LOGSTASH"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

This setup will send all log messages to both of the configured appenders. Read the rest of this entry »

Posted in: DevOps | From The Trenches | Java

Machine Learning: Predicting house prices

11:06 on February 16th, 2017 by
No Comments

Recently i have followed an online course on machine learning to understand the current hype better. As with any subject though, only practice makes perfect, so i was looking to apply this new knowledge.

While looking to sell my house i found that would be a nice opportunity: Check if the prices a real estate agents estimates are in line with what the data suggests.

Linear regression algorithm should be a nice algorithm here, this algorithm will try to find the best linear prediction (y = a + bx1 + cx2 ; y = prediction, x1,x2 = variables). So for example this algorithm can estimate a price per square meter floor space or price per square meter of garden. For a more detailed explanation, check out the wikipedia page.

In the Netherlands funda is the main website for selling your house, so i have started by collecting some data, i used data on the 50 houses closest to my house. I’ve excluded apartments to try and limit data to properties similar to my house. For each house i collected the advertised price, usable floor space, lot size, number of (bed)rooms, type of house (row-house, corner-house, or detached) and year of construction (..-1930, 1931-1940, 1941-1950, 1950-1960, etc). These are the (easily available) variables i expected would influence house price the most. Type of house is a categorical variable, to use that in regression I modeled them as several binary (0/1) variables.

As preparation, i checked for relations between the variables using correlation. This showed me that much of the collected data does not seem to affect price: Only the floor space, lot size and number of rooms showed a significant correlation with house price.

For the regression analysis I only used the variables that had a significant correlation. Variables without correlation would not produce meaningful results anyway.

Read the rest of this entry »

Posted in: Custom Development | General | Machine Learning

Simulating an Elasticsearch Ingest Node pipeline

9:50 on February 2nd, 2017 by
No Comments

Indexing document into your cluster can be done in a couple of ways:

  • using Logstash to read your source and send documents to your cluster;
  • using Filebeat to read a log file, send documents to Kafka, let Logstash connect to Kafka and transform the log event and then send those documents to your cluster;
  • using curl and the Bulk API to index a pre-formatted file;
  • using the Java Transport Client from within a custom application;
  • and many more…

Before version 5 however there where only two ways to transform your source data to the document you wanted to index. Using Logstash filters, or you had to do it yourself.

In Elasticsearch 5 the concept of the Ingest Node has been introduced. Just a node in your cluster like any other but with the ability to create a pipeline of processors that can modify incoming documents. The most frequently used Logstash filters have been implemented as processors.

For me, the best part of pipelines is that you can simulate them. Especially in Console, simulating your pipelines makes creating them very fast; the feedback loop on testing your pipeline is very short. Making using pipelines a very convenient way to index data.

Read the rest of this entry »

Tags: , , , , ,

Posted in: Custom Development | Elasticsearch

Public Elasticsearch clusters are being held ransom

14:02 on January 18th, 2017 by
No Comments

Last week several news sites and researchers reported that Elasticsearch clusters that are connected to the internet without proper security are being held ransom.

You can use shodan.io to search for Elasticsearch clusters: https://www.shodan.io/search?query=port%3A9200+json&language=en.

The first hit is actually a cluster that is ‘infected’:

There are some secured clusters as well:

But the default ‘root’ account with username “elastic” and password “changeme” (docs) will grant access. So not much security here… But at least your data is still there. For now.

Please do not connect your cluster to the internet without securing. Use X-Pack Security for authentication and authorization.

Elastic Cloud could also be something for you. Security in Elastic Cloud is default.

Tags:

Posted in: Elasticsearch

Handling a massive amount of product variations with Elasticsearch

14:59 on December 22nd, 2016 by
5 Comments

In this blog we will review different techniques for modelling data structures in Elasticsearch. A project case is used to describe our approach on handling a small sized product data set with a large sized related product variations data set. Furthermore we will show how certain modelling decisions resulted in a 1000 factor query performance gain!

The flat world

Elasticsearch is a great product if you want to index and search through a large number of documents. Functionality like term and range queries, full-text search and aggregations on large data sets are very fast and powerful. But Elasticsearch prefers to treat the world as if it were flat. This means that an index is a flat collection of documents. Furthermore, when searching, a single document should contain all of the information that is required to decide whether it matches the search request.

In practice, however, domains often are not flat and contain a number of entities which are related to each other. These can be difficult to model in Elasticsearch in such a way that the following conditions are met:

  • Multiple entities can be aggregated from a single query;
  • Query performance is stable with low response times;
  • Large numbers of documents can easily be mutated or removed.

The project case

This blog is based on a project case. In the project, two data sets were used. The data sets have the following characteristics:

  • Products:
    • Number of documents: ~ 75000;
    • Document characteristics: A product contains a set of fields which contains the primary information of a product;
    • Mutation frequency: Updates on product attributes can occur fairly often (e.g. every 15 minutes).
  • Product variations:
    • Number of documents: ~ 500 million;
    • Document characteristics: A product variation consists of a set of additional attributes which contain extra information on top of the corresponding product. The number of product variations per product varies a lot, and can go up to 50000;
    • Mutation frequency: During the day, there is a continuous stream of updates and new product variations.

Read the rest of this entry »

Posted in: Elasticsearch

Service Discovery using Consul & Spring Cloud

10:08 on December 14th, 2016 by
11 Comments

Introduction

In one of our customer projects we are heavily using Spring Boot in combination with other Spring projects for our microservices.

One of the more complex parts of microservices, especially when you are using them as fine-grained as meant to be, will be the fact that you need to setup and maintain the connections between all those services. In Spring you are typically doing this using some way of externalized configuration like property files. But even then, it can become quite a challenge when you need to connect with for instance 20 other microservices.

To make it even more complex you definitely want, especially in cloud based solutions, something like scalability. Actually, this should be accomplished by running just another instance of your microservice. They are self-contained, so they just need some basic configuration like setting the port-number. But then you also need something to load-balance the different microservices serving the same purpose. And to be honest: I don’t care about the location and port! I just want a service which offers me a certain contract. And at runtime, when needed, I want them to behave differently depending on configuration.

So what we are actually looking for is a solution which provides an easy way to do service discovery and even better, can act as a load-balancer and even better, can provide my services with their configuration.

This is where Consul.io comes to the rescue. According to their website Consul is a solution which makes service discovery and service configuration easy and is distributed, highly available and datacenter-aware.

So let’s discover how Consul plays nicely with Spring Boot!

Read the rest of this entry »

Posted in: Custom Development | Spring

Writing less code

11:10 on November 23rd, 2016 by
1 Comment

Have you had that feeling that you have to write too much code to build simple functionality? Some things just feel repetitive, they feel you should be not have to write them yourself, instead a framework should make your life easier.

Recently I’ve been building a project in Java/Spring, and after some time I started wondering about alternatives and how to build the same functionality with less code.

There is lots of alternative frameworks and multiple ways of building rest endpoints in Java/Spring.

  • Building the controller/service/dao layers manually in Spring ; https://spring.io/guides/tutorials/bookmarks/
  • Using spring-data-rest to export your spring-data repositories ; https://spring.io/guides/gs/accessing-data-rest/
  • Groovy/grails RestfulController ; https://examples.javacodegeeks.com/jvm-languages/groovy/grails/grails-rest-example/
  • Python/django django-rest-framework ; http://www.django-rest-framework.org/tutorial/6-viewsets-and-routers/
  • etc

Examples

Below some abbreviated examples of how a simple rest endpoint looks for each approach. To actually run the examples, you’ll need check out the tutorials mentioned earlier. My goal here is a quick comparison of how you do things in each framework.

Read the rest of this entry »

Posted in: Custom Development | Groovy | Java | Spring

« Older Entries