Trifork Blog

Axon Framework, DDD, Microservices

Creating an advanced Kibana dashboard using a script

May 20th, 2014 by
|

Logo van Kibana

Some time ago, Kibana joined the elasticsearch family. A lot of good things have come out of it. These days Kibana is becoming more advanced. But with more users also come more demands. One of those demands is more advanced dashboards than can be clicked together in the very nice GUI. We want to be able to customize dashboards, prepare dashboards to be used by others.

In this blogpost I am going to show you some of the options you have to create a more advanced dashboard. I use an index I have created based on my iTunes library. We are going to create a dashboard showing information about artists, albums and we show how to use parameters through the url.

Quick intro if you don’t know kibana

Kibana is a tool developed to create nice graphs based on logs send to elasticsearch by logstash. It makes use of the excellent facetted queries as provided by elasticsearch to create tables, histograms, pie charts and even maps with geo points. Kibana is a purely javascript based, so client side, application connecting to the rest interface of elasticsearch. Using the interface you can create lots of different charts presenting data coming out of elasticsearch. the charts are constructed using the forms provided by Kibana.

The structure of a dashboard can also be saved in elasticsearch. You can share your dashboards by url, but we need more. We want to prepare dashboards to be used by others. Preventing them from destroying the dashboard or doing queries that are to heavy on our cluster. Another thing we might want is to change the dashboard based on some specific parameters. The solution is to use dashboard templates or scripted dashboards.

How does it work?

A Kibana dashboard is just a json document. You can store these documents in elasticsearch to keep them for later. In this blog we want to take another approach. There are two other mechanisms to prepare dashboards. One of them is to create a template. This is a json document based on a specific schema. The document you need to create is stored in the kibana directory:

[kibana_home]/app/dashboards

Save a template as a .json document. Each json document that is a dashboard needs to conform to the schema as discussed in the next section.

The schema

Each dashboard can consist of the following items:

Services – Can be reused between the different panels. At the moment these can be filters and queries. These can be found in the kibana interface at the top of the screen.

Screenshot showing queries and fitlers

Row – The object that contains all our rows with panels. This is the object were we add the panels to our screen.

Panel – Kibana comes with a number of different panels that can all be added to your dashboard. Examples are table, histogram, terms, text, map and the even better map.

Index – Configure some index specific properties like the default index and the index pattern.

Other items that I am not going to discuss are: Pulldowns, nav, loader. Check the references for more information about the schema and it’s options.

Exporting existing dashboards as a template

If you are the kind of person (like me) that does not want to start from scratch you can copy one of the examples in the kibana folder. Especially the logstash.json sample is a good start to see what is possible. Another way to start from an existing dashboard is to export a dashboard that you have created using the interface. On the top right menu you can find the disk representing save. Click it and go to the advanced part and chose Export schema. Than you get a json document that you can use as the basis for your template. I personally use this to get a good start for panels I want to add to my dashboard.

Screen Shot showing the export of a schema

In the next section you’ll find out how to look at these dashboards that you store within the kibana website.

Look at an example for logstash

Nice to have these cool templates in Kibana, but how do we acces these new dashboards. This is not very hard, you just need to know two things:

    1. For a template you end the url with: #/dashboard/file/ and the file name of the dashboard. In my case I access the logstash sample dashboard template with the following url.
http://localhost/kibana/#/dashboard/file/logstash.json
    1. When creating a script, like in the next section, the url is different. Instead of file we use script and instead of the extension json, we use js.
http://localhost/kibana/#/dashboard/script/logstash.js

The example dashboard

I mentioned in the introduction that I am using an index with my music as a sample index. The index contain all songs as documents. So presenting all songs from an albums means an aggregation over the album field. We can also find all albums for certain artist. In the sample dashboard we show a panel with the artists with the most songs. Next to that a panel with the songs decided by genre in a pie chart. Can you spot an error in my index using the pie chart? Write a comment if you think you know what I mean. Of course when clicking on a genre, a filter is added and the table with artists is changed. At the bottom we have a table with all the songs.

That is all nice, but this is easy to do with the user interface. Now we want to provided the option to pass the name of an artist or a genre to the dashboard. Than we do not need the specific panel, so we replace it with some text. Now we also add a third panel to the top row showing the top songs that belong to the chosen genre or artist. Before showing the results, let us dive into some code.

The first part is some initialization of the dashboard. Most of the comments are coming from the logstash sample as provided by elasticsearch. We set the index to be mymusic and the query to the match all query.

'use strict';

var dashboard, queries

// All url parameters are available via the ARGS object
var ARGS;

// Intialize a skeleton with nothing but a rows array and service object
dashboard = {
  rows : [],
  services : {}
};

dashboard.title = 'Show information about my music';
dashboard.failover = false;
dashboard.index = {
  default: 'mymusic',
  interval: 'none'
};

queries = {
  0: {
    query: '*',
    id: 0,
  }
};

// Now populate the query service with our objects
dashboard.services.query = {
  list : queries,
  ids : _.map(_.keys(queries),function(v){return parseInt(v,10);})
};

The last few lines might be a bit tricky. But all queries are added using an id. We also need to add an array with the same ids.

Next step is to extract some parameters from the url.

Parameters that we provide

You can provide the following parameters:

    • genre – Name of the genre to filter by.
    • artist – name of the artist to filter by.

Using these parameters we decide whether to add and enable predefined filters as services.

dashboard.services.filter = {
  list: {
    0:{
      "type": "terms",
      "field": "genre",
      "value": ARGS.genre,
      "mandate": "must",
      "active": !_.isUndefined(ARGS.genre),
      "alias": "",
      "id": 0
    },
    1:{
      "type": "terms",
      "field": "artist",
      "value": ARGS.artist,
      "mandate": "must",
      "active": !_.isUndefined(ARGS.artist),
      "alias": "",
      "id": 1
    }

  },
  ids: [0,1]
};

As you can see, we add 2 terms filters and enable them if the parameters ARGS.genre and ARGS.artist are available. ARGS is a special object to acces parameters or arguments passed using the url.

The next step is to add some rows and panels to the dashboard.

Add some rows and panels

We have two rows in our dashboard. The second row is just a table with some predefined columns. All properties should explain themselves.

dashboard.rows[1].panels = [
  {
    "error": false,
    "span": 12,
    "editable": true,
    "type": "table",
    "loadingEditor": false,
    "size": 100,
    "pages": 5,
    "offset": 0,
    "sort": [
      "album.raw",
      "asc"
    ],
    "overflow": "min-height",
    "fields": [
      "artist",
      "album",
      "name",
      "genre"
    ],
    "highlight": [],
    "sortable": true,
    "header": true,
    "paging": true,
    "field_list": false,
    "all_fields": false,
    "trimFactor": 300,
    "localTime": false,
    "timeField": "@timestamp",
    "spyable": false,
    "queries": {
      "mode": "all",
      "ids": [
        0
      ]
    },
    "style": {
      "font-size": "9pt"
    },
    "normTimes": true,
    "title": "the songs"
  }
];

Now we zoom in on the first row. This row is more complicated. It had 3 panels. But the first two panels can change from a terms panel to a text panel if the artist or the genre is provided in the url. Next to that if you provide the artist, we add a third panel with a terms panel containing the albums. The following code block shows added the panels to a row.

dashboard.rows[0].panels = [
];

if(_.isUndefined(ARGS.artist)) {
  dashboard.rows[0].panels.push({
    title: 'Artist',
    type: 'terms',
    field: "artist.raw",
    chart: 'table',
    span: 4,
    spyable: false
  });
} else {
  dashboard.rows[0].panels.push({
    title: 'Choosen artist',
    type: 'text',
    mode: 'markdown',
    content: 'You have provided the artist with name:*'+ARGS.artist+'*',
    span: 4
  });
}

if(_.isUndefined(ARGS.genre)) {
  dashboard.rows[0].panels.push({
    title: 'Genre',
    type: 'terms',
    field: "genre",
    chart: 'pie',
    span: 4,
    spyable: false
  });
} else {
  dashboard.rows[0].panels.push({
    title: 'Choosen genre',
    type: 'text',
    mode: 'markdown',
    content: 'You have provided the genre:*'+ARGS.genre+'*',
    span: 4
  });
}

if (!_.isUndefined(ARGS.artist)) {
  dashboard.rows[0].panels.push({
    title: 'Album',
    type: 'terms',
    field: "album.raw",
    chart: 'table',
    span: 4,
    spyable: false
  });
}

Check how we use the arguments provided by ARGS. Adding a panel means adding an object to the dashboard.rows[0].panels array. The final thing I want to mention is the usage of markdown for the text panels.

Look at our result

First the top row without adding parameters.

http://localhost/kibana/#/dashboard/script/mymusic.js

Screen Shot Dashboard without parameters

Next the dashboard when providing a parameter for artist

http://localhost/kibana/#/dashboard/script/mymusic.js?artist=amos

Screen Shot Dashboard with parameters

References

16 Responses

  1. July 16, 2014 at 15:54 by guyo

    I think this is a greate script.But I am new to develop script and I need more tutorials.Please, can you help me?
    I am looking forward.

  2. August 5, 2014 at 23:35 by mars

    Why is it that some have quotations such as “field_list”: false while other don’t have quotations such as title: ‘Genre’??

  3. August 28, 2014 at 02:58 by Tim Ward

    Any idea why nginx would keep giving access forbidden when I create the script directory under /usr/share/nginx/app/dashboards/script/ ?

    • September 2, 2014 at 15:47 by Jettro Coenradie

      No sorry, I do not really have experience with nginx

  4. September 1, 2014 at 20:23 by guillaume

    hello, nice one, thanks,

    Do you know, per any chance if it is possible to set a query like (without curl..)
    curl -XGET ‘http://url?pretty’ -d ‘{
    “aggregations”: {
    “user_types”: {
    “terms”:{
    “field”:”userType”
    },
    “aggregations”:{
    “type_count”: {
    “cardinality”:{
    “field”:”userId”
    }
    }
    }
    }
    }
    },
    “size”: 0
    }’

    in an dashboard JS ? Thanks for your lights ๐Ÿ™‚

    • September 2, 2014 at 15:45 by Jettro Coenradie

      As far as I know you cannot use aggregations yet in Kibana.

  5. September 26, 2014 at 17:07 by Sk

    @Tim: As I understand, you want to access your custom script dashboard. If so, keep your script file (.js) directly under app/dashboards and access that http://localhost/kibana/#/dashboard/script/.js

  6. November 18, 2014 at 12:42 by Anoop.P.A

    Nice article. If a link to the data was given it would have been nice!

  7. December 26, 2014 at 09:32 by thejas

    Hi, i have shipped 3 different types of logs , in graphical representation i see all 3 logs but am how am not able to see the fields of all 3 logs … I see fields of only one log(1st shipped log). how to check the fields of other logs . Please need immediate answer

  8. January 12, 2015 at 16:56 by diego

    Hi! very good example, anyone know how to get a URL directly to a panel, i will like to embedded just a panel, or a dashboard (without topbar) in another page.

    Thanks!
    diego

  9. January 20, 2015 at 13:46 by binit Kumar

    I want a dashboard with log message and timestamp at a time in kibana 4 version ,help me plz thanks in advance ๐Ÿ™‚

  10. March 17, 2015 at 06:58 by Bijoy

    Does Kibana has Geo Charts?

  11. April 14, 2015 at 09:16 by vijay

    is their any way to change the top menu bar in kibana to my customized menubar and delete the edit and save options,,

  12. April 17, 2015 at 18:18 by Derek

    Correct me if i’m wrong, but this tutoriel only applies to Kibana 3 and not Kibana 4. Is there a way to code a custom dashboard in Kibana 4 ? I’m trying to get multicolumn agregation results in tables and navigable widgets.

  13. April 30, 2015 at 15:38 by Kriek Jooste

    To make something like this work, I found that I had to add a ‘return dashboard’ at the end, and had to remove the use strict.

  14. June 3, 2015 at 23:50 by MD

    This article is great but it looks like it will work for Kibana 3 only. Can you update the article for Kibana 4?