Trifork Blog

Category ‘Enterprise Search’

Dealing with NodeNotAvailableExceptions in Elasticsearch

April 8th, 2015 by
(http://blog.trifork.com/2015/04/08/dealing-with-nodenotavailableexceptions-in-elasticsearch/)

tl;dr

Elasticsearch provides distributed search with minimal setup and configuration. Now the nice thing about it is that, most of the time, you don’t need to be particularly concerned about how it does what it does. You give it some parameters – “I want 3 nodes”, “I want 3 shards”, “I want every shard to be replicated so it’s on at least two nodes”, and Elasticsearch figures out how to move stuff around so you get the situation you asked for. If a node becomes unreachable, Elasticsearch tries to keep things going, and when the lost node appears and rejoins, the administration is updated so everything is hunky-dory again.

The problem is when things don’t work the way you expect…

Computer says “no node available”

Read the rest of this entry »

Shield your Kibana dashboards

March 5th, 2015 by
(http://blog.trifork.com/2015/03/05/shield-your-kibana-dashboards/)

You work with sensitive data in Elasticsearch indices that you do not want everyone to see in their Kibana dashboards. Like a hospital with patient names. You could give each department their own Elasticsearch cluster in order to prevent all departments to see the patient’s names, for example.

But wouldn’t it be great if there was only one Elasticsearch cluster and every departments could manage their own Kibana dashboards? And still have the security in place to prevent leaking of private data?

With Elasticsearch Shield, you can create a configurable layer of security on top of your Elasticsearch cluster. In this article, we will explore a small example setup with Shield and Kibana.

Read the rest of this entry »

ANWB Big data Proof of Concept

February 9th, 2015 by
(http://blog.trifork.com/2015/02/09/anwb-big-data-proof-of-concept/)

At the ANWB people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW) provides an enormous database of both real-time and historic traffic data.

This data comes from many different sources and is available as open data. Wouldn’t it be great if the ANWB could use this open data to provide more accurate traffic information, either in real-time or as a prediction for a certain period? In a proof of concept we have collected and analysed the real-time traffic information to calculate the traffic intensity on the roads using elasticsearch. We also used weather information to see if the weather has influence on the need of roadside assistance.

Read the rest of this entry »

Creating an advanced Kibana dashboard using a script

May 20th, 2014 by
(http://blog.trifork.com/2014/05/20/advanced-kibana-dashboard/)

Logo van Kibana

Some time ago, Kibana joined the elasticsearch family. A lot of good things have come out of it. These days Kibana is becoming more advanced. But with more users also come more demands. One of those demands is more advanced dashboards than can be clicked together in the very nice GUI. We want to be able to customize dashboards, prepare dashboards to be used by others.

In this blogpost I am going to show you some of the options you have to create a more advanced dashboard. I use an index I have created based on my iTunes library. We are going to create a dashboard showing information about artists, albums and we show how to use parameters through the url.

Read the rest of this entry »

Elasticsearch, Spring MVC & Sencha Touch 2 in the Cloud – Part 2

May 6th, 2014 by
(http://blog.trifork.com/2014/05/06/elasticsearch-spring-mvc-sencha-touch-2-in-the-cloud-part-2/)

logo-senchaThis is the second part of my blog on how to develop an application using Elasticsearch, Spring MVC and Sencha Touch 2. In my previous blog post part 1 I showed and explained which technologies I used to accomplish the connection between the frontend and backend. In addition I presented the steps to connect a database service (Elasticsearch) with a Spring MVC service. Part 2 will continue the development, in particular the connection between Sencha Touch 2 and the Spring MVC projects. Finally, I will show how to deploy the developed application into the cloud.

Read the rest of this entry »

Elasticsearch & Spring MVC & Sencha Touch 2 in the Cloud – Part 1

April 15th, 2014 by
(http://blog.trifork.com/2014/04/15/elasticsearch-spring-mvc-sencha-touch-2-in-the-cloud-part-1/)

elasticsearch-logo

 

Introduction

Welcome to my third blog entry. In this one, I want to show how to connect three different technologies (database, REST service, and a mobile framework) and deploy them into the cloud. Those three technologies are:

  • Elasticsearch (Database)
  • Spring MVC (REST service)
  • Sencha Touch 2 (Client side framework)

First, I want to give a short introduction of the three technologies. I am going to start with Elasticsearch, followed by Spring MVC and in the end Sencha Touch 2. In the second part, I will explain the services that I used. The first service is called searchly and is a database service. The second service is cloudbees and is used as cloud host. In the end a conclusion is given. Part 2 of this blog will cover a demo with the steps to connect everything with each other. I wish you fun reading this entry and if there are any questions don’t hesitate to drop a comment (Of course you can drop also a comment if you just like it).

Read the rest of this entry »

Server-side clustering of geo-points on a map using Elasticsearch – continued

March 26th, 2014 by
(http://blog.trifork.com/2014/03/26/server-side-clustering-of-geo-points-on-a-map-using-elasticsearch-continued/)

In a previous post I described a problem of data visualization and a possible solution provided by a plugin of elasticsearch. I noticed that elasticsearch might one day evolve to make the plugin unnecessary. That day seems to have come: starting from version 1.0.0, elasticsearch includes Aggregations, a new API for data mining. In this post I’ll show you how to use aggregations to reproduce the functionality of the plugin.

Read the rest of this entry »

Evaluating elasticsearch and marvel on the raspberry pi

February 8th, 2014 by
(http://blog.trifork.com/2014/02/08/evaluating-elasticsearch-and-marvel-on-the-raspberry-pi/)

IMG 0208

The past years I have been working with search solutions, mostly elasticsearch. During this time a bought myself a raspberry pi and installed java and elasticsearch on it. Then I put it in the closet and it did not come out anymore. Than a few weeks a go the guys from elasticsearch released marvel. Marvel is a monitoring tool for your elasticsearch cluster. Suddenly I realized what the problem with the raspberry pi is. It is not fun to have just one. Therefore I decided to buy two more and create an elasticsearch cluster. With this cluster I can do experiments. The first experiment is evaluating marvel.

In this blog post I will show some of the concepts of marvel. To make this possible I will also explain the steps I had to take to install elasticsearch on my raspberry pi cluster.

Read the rest of this entry »

Using logstash, elasticsearch and Kibana to monitor your video card – a tutorial

January 28th, 2014 by
(http://blog.trifork.com/2014/01/28/using-logstash-elasticsearch-and-kibana-to-monitor-your-video-card-a-tutorial/)

A few weeks ago my colleague Jettro wrote a blog post about an interesting real-life use case for Kibana: using it to graph meta-data of the photos you took. Given that photography is not a hobby of mine I decided to find a use-case for Kibana using something closer to my heart: gaming.

This Christmas I treated myself to a new computer. The toughest decision I had to make was regarding the video card. In the end I went with a reference AMD R9 290, notoriously known for its noisiness. Because I’m really interested in seeing how the card performs while gaming, I decided to spent some time on my other hobby, programming, in order to come up with a video card monitoring solution based on logstash, elasticsearch & Kibana. Overkill? Probably. Fun? Definitely.

I believe it’s also a very nice introduction on how to set up a fully working setup of logstash – elasticsearch – Kibana. Because of the “Windowsy” nature of gaming, some of the commands listed are the Windows version. The Unix folk should have no problems translating these as everything is kept very simple.

Read the rest of this entry »

elasticsearch – how many shards?

January 7th, 2014 by
(http://blog.trifork.com/2014/01/07/elasticsearch-how-many-shards/)

We’ve all been there – you’re provisioning for an elasticsearch index and one of the first questions that comes to mind is “how many shards should I create my index with?”. In my previous posts on the subject, I wrote about how to find the maximum shard size for elasticsearch. Although informative, the results of the tests also raised a new question: would more shards on a single elasticsearch node increase performance? In this blog post I’m going to try to show the performance consequences of different choices for the number of shards.

Read the rest of this entry »