Trifork Blog

Posts Tagged ‘elasticsearch’

Simulating an Elasticsearch Ingest Node pipeline

February 2nd, 2017 by
(http://blog.trifork.com/2017/02/02/elasticsearch-ingest-node/)

Indexing document into your cluster can be done in a couple of ways:

  • using Logstash to read your source and send documents to your cluster;
  • using Filebeat to read a log file, send documents to Kafka, let Logstash connect to Kafka and transform the log event and then send those documents to your cluster;
  • using curl and the Bulk API to index a pre-formatted file;
  • using the Java Transport Client from within a custom application;
  • and many more…

Before version 5 however there where only two ways to transform your source data to the document you wanted to index. Using Logstash filters, or you had to do it yourself.

In Elasticsearch 5 the concept of the Ingest Node has been introduced. Just a node in your cluster like any other but with the ability to create a pipeline of processors that can modify incoming documents. The most frequently used Logstash filters have been implemented as processors.

For me, the best part of pipelines is that you can simulate them. Especially in Console, simulating your pipelines makes creating them very fast; the feedback loop on testing your pipeline is very short. Making using pipelines a very convenient way to index data.

Read the rest of this entry »

Public Elasticsearch clusters are being held ransom

January 18th, 2017 by
(http://blog.trifork.com/2017/01/18/public-elasticsearch-clusters-are-being-held-ransom/)

Last week several news sites and researchers reported that Elasticsearch clusters that are connected to the internet without proper security are being held ransom.

You can use shodan.io to search for Elasticsearch clusters: https://www.shodan.io/search?query=port%3A9200+json&language=en.

The first hit is actually a cluster that is ‘infected’:

There are some secured clusters as well:

But the default ‘root’ account with username “elastic” and password “changeme” (docs) will grant access. So not much security here… But at least your data is still there. For now.

Please do not connect your cluster to the internet without securing. Use X-Pack Security for authentication and authorization.

Elastic Cloud could also be something for you. Security in Elastic Cloud is default.

Elastic{ON} 2016

February 20th, 2016 by
(http://blog.trifork.com/2016/02/20/elasticon-2016/)

Elastic{ON} 2016 - ViewLast week a colleague and I attended Elastic{ON} in San Francisco. The venue at Pier 48 gave a nice view on (among others) the Oakland Bay Bridge. Almost 2000 Elastic fanatics converged to listen to and talk about everything in the Elastic Stack.

I have been to a lot of sessions. I think the two most important things that I will take home are “5.0” and “graphs”.

5.0

The next version of the Elastic Stack will be 5.0. This means that all main Elastic products (Elasticsearch, Logstash, Kibana and Beats) are having the same version number in all following release bonanzas. This will be easier for all customers and clients.

I mentioned the Elastic Stack. This is a little rebranding of the ELK Stack plus Beats. More rebranding is the renaming of the Elastic as a Service solution Found to Elastic Cloud. I think those are simple but good changes.

Also Elastic created the concept of packs to combine extensions. Most notably the X-Pack will all the monitoring, alerting and security (and more) goodies wrapped together.

More about 5.0 on the Elastic blog.

Graphs

Elastic{ON} 2016 - GraphThe other main take-away are the graph capabilities (Graph API) that will be added to Elasticsearch (through the X-Pack). It is still in an early phase but it looks awesome! It looks very easy to use and it is very fast. The UI is written as a Kibana plugin.

Actually there will be some more Kibana plugins. Managing users and roles via the Security API, for example.

Talks

Off course there were a lot of talks. Common subjects were security and recommendation. Graphs could play an important role there!

Some talks were cool user stories of companies that implemented (parts of) the Elastic Stack. Other talks dove deep into the different Elastic products. Some of those turned out to be a little out of my league. For example the math behind the new default BM25 scoring algorithm.

The talks will be put online in the next couple of weeks. So be sure to check them out! Maybe I will see you next year!

GOTO Amsterdam: the Elasticsearch track

May 12th, 2015 by
(http://blog.trifork.com/2015/05/12/goto-amsterdam-the-elasticsearch-track/)
More and more enterprises start using the data they gather. Besides the Big data and Data science, there is a 3rd taste: search. Since January of this year, 1 search technology (stack) came up on top and since then is the number 1 choice when it comes to search: elastic search.
 ElasticSearch-2015

Read the rest of this entry »

Shield your Kibana dashboards

March 5th, 2015 by
(http://blog.trifork.com/2015/03/05/shield-your-kibana-dashboards/)

You work with sensitive data in Elasticsearch indices that you do not want everyone to see in their Kibana dashboards. Like a hospital with patient names. You could give each department their own Elasticsearch cluster in order to prevent all departments to see the patient’s names, for example.

But wouldn’t it be great if there was only one Elasticsearch cluster and every departments could manage their own Kibana dashboards? And still have the security in place to prevent leaking of private data?

With Elasticsearch Shield, you can create a configurable layer of security on top of your Elasticsearch cluster. In this article, we will explore a small example setup with Shield and Kibana.

Read the rest of this entry »

ANWB Big data Proof of Concept

February 9th, 2015 by
(http://blog.trifork.com/2015/02/09/anwb-big-data-proof-of-concept/)

At the ANWB people are constantly trying to improve the services they provide. One of these services is to provide traffic information. In the Netherlands the National Data Warehouse for Traffic Information (NDW) provides an enormous database of both real-time and historic traffic data.

This data comes from many different sources and is available as open data. Wouldn’t it be great if the ANWB could use this open data to provide more accurate traffic information, either in real-time or as a prediction for a certain period? In a proof of concept we have collected and analysed the real-time traffic information to calculate the traffic intensity on the roads using elasticsearch. We also used weather information to see if the weather has influence on the need of roadside assistance.

Read the rest of this entry »

Creating an advanced Kibana dashboard using a script

May 20th, 2014 by
(http://blog.trifork.com/2014/05/20/advanced-kibana-dashboard/)

Logo van Kibana

Some time ago, Kibana joined the elasticsearch family. A lot of good things have come out of it. These days Kibana is becoming more advanced. But with more users also come more demands. One of those demands is more advanced dashboards than can be clicked together in the very nice GUI. We want to be able to customize dashboards, prepare dashboards to be used by others.

In this blogpost I am going to show you some of the options you have to create a more advanced dashboard. I use an index I have created based on my iTunes library. We are going to create a dashboard showing information about artists, albums and we show how to use parameters through the url.

Read the rest of this entry »

Using logstash, elasticsearch and Kibana to monitor your video card – a tutorial

January 28th, 2014 by
(http://blog.trifork.com/2014/01/28/using-logstash-elasticsearch-and-kibana-to-monitor-your-video-card-a-tutorial/)

A few weeks ago my colleague Jettro wrote a blog post about an interesting real-life use case for Kibana: using it to graph meta-data of the photos you took. Given that photography is not a hobby of mine I decided to find a use-case for Kibana using something closer to my heart: gaming.

This Christmas I treated myself to a new computer. The toughest decision I had to make was regarding the video card. In the end I went with a reference AMD R9 290, notoriously known for its noisiness. Because I’m really interested in seeing how the card performs while gaming, I decided to spent some time on my other hobby, programming, in order to come up with a video card monitoring solution based on logstash, elasticsearch & Kibana. Overkill? Probably. Fun? Definitely.

I believe it’s also a very nice introduction on how to set up a fully working setup of logstash – elasticsearch – Kibana. Because of the “Windowsy” nature of gaming, some of the commands listed are the Windows version. The Unix folk should have no problems translating these as everything is kept very simple.

Read the rest of this entry »

GOTO Academy – Training & Courses

January 15th, 2014 by
(http://blog.trifork.com/2014/01/15/goto-academy-training-courses/)
The GOTO Academy is organising regular training sessions around professional software development and architecture.
We are currently running a New Year promo: 20% discount on the iOS trainings.
Take advantage of it! Use the voucher code: TRIFORK2014 to book your seat before end of January 2014. Register here, select the training (iOS beginner or advanced) and type the voucher code in the comment section.

Goodbye SearchWorkings.org

December 3rd, 2013 by
(http://blog.trifork.com/2013/12/03/goodbye-searchworkings-org/)

searchworkings_logoIn 2011 we launched SearchWorkings.org, a community website that aimed to bring search professionals together, mostly around open source search technologies like Apache Lucene and Apache Solr. At the time, the number of resources providing high value content around those technologies was limited. Therefore, we created the searchworkings portal, providing blog entries, white papers and a forum. Next to JTeam’s own search experts (Simon Willnauer, Uri Boness, Martijn van Groningen, Chris Male, Luca Cavanna and Frank Scholten), we also managed to get several external contributors onboard (Isabel Drost, Chris Mattmann, Mike McCandless, Uwe Schindler, Marc Sturlese, Anne Veling, Dawid Weiss and Karl Wright).

Read the rest of this entry »