Every long running project accrues technical debt. It may be that the requirements today have evolved in a different direction from what was foreseen when the project was designed, or it may be that difficult infrastructure tasks have been put off in favor of new functionality. From time to time, you need to refactor your […]
Kibana Histogram on Day of Week
I keep track of my daily commutes to and from the office. One thing I want to know is how the different days of the week are affecting my travel duration. But when indexing all my commutes into Elasticsearch, I can not (out-of-the-box) create a histogram on the day of the week. My first visualization […]
Smart energy consumption insights with Elasticsearch and Machine Learning
At home we have a Youless device which can be used to measure energy consumption. You have to mount it to your energy meter so it can monitor energy consumption. The device then provides energy consumption data via a RESTful api. We can use this api to index energy consumption data into Elasticsearch every minute […]
Simulating an Elasticsearch Ingest Node pipeline
Indexing document into your cluster can be done in a couple of ways: using Logstash to read your source and send documents to your cluster; using Filebeat to read a log file, send documents to Kafka, let Logstash connect to Kafka and transform the log event and then send those documents to your cluster; using […]
Public Elasticsearch clusters are being held ransom
Last week several news sites and researchers reported that Elasticsearch clusters that are connected to the internet without proper security are being held ransom. You can use shodan.io to search for Elasticsearch clusters: https://www.shodan.io/search?query=port%3A9200+json&language=en. The first hit is actually a cluster that is ‘infected’: There are some secured clusters as well: But the default ‘root’ account with […]
Handling a massive amount of product variations with Elasticsearch
In this blog we will review different techniques for modelling data structures in Elasticsearch. A project case is used to describe our approach on handling a small sized product data set with a large sized related product variations data set. Furthermore we will show how certain modelling decisions resulted in a 1000 factor query performance gain! […]
Collecting data from a private LoRaWAN sensor network into Elastic
Introduction to LoRaWAN and ELK Why LoRaWAN, and what makes it different from other types of low power consumption, high range wireless protocols like ZigBee, Z-Wave, etc … ? LoRa is a wireless modulation for long-range, low-power, low-data-rate applications developed by Semtech. The main features of this technology are the big amount of devices that […]
Elastic{ON} 2016
Last week a colleague and I attended Elastic{ON} in San Francisco. The venue at Pier 48 gave a nice view on (among others) the Oakland Bay Bridge. Almost 2000 Elastic fanatics converged to listen to and talk about everything in the Elastic Stack. I have been to a lot of sessions. I think the two […]
Dealing with NodeNotAvailableExceptions in Elasticsearch
tl;dr Elasticsearch provides distributed search with minimal setup and configuration. Now the nice thing about it is that, most of the time, you don’t need to be particularly concerned about how it does what it does. You give it some parameters – “I want 3 nodes”, “I want 3 shards”, “I want every shard to […]
Shield your Kibana dashboards
You work with sensitive data in Elasticsearch indices that you do not want everyone to see in their Kibana dashboards. Like a hospital with patient names. You could give each department their own Elasticsearch cluster in order to prevent all departments to see the patient’s names, for example. But wouldn’t it be great if there […]