Menu
Amazon Elasticsearch Service
Developer Guide (API Version 2015-01-01)

Kibana and Logstash

This chapter describes some considerations for using Kibana and Logstash with Amazon Elasticsearch Service.

Kibana

Kibana is a popular open source visualization tool designed to work with Elasticsearch. Amazon ES provides an installation of Kibana with every Amazon ES domain. You can find a link to Kibana on your domain dashboard on the Amazon ES console. The URL is https://domain.region.es.amazonaws.com/_plugin/kibana/. Queries using this default Kibana installation have a 60-second timeout.

The following sections address some common Kibana use cases:

Controlling Access to Kibana

Kibana does not natively support IAM users and roles, but Amazon ES offers several solutions for controlling access to Kibana:

Domain Configuration Access Control Options
Public access
VPC access

Using a Proxy to Access Amazon ES from Kibana

Note

This process is only applicable if your domain uses public access and you don't want to use Amazon Cognito Authentication for Kibana. See Controlling Access to Kibana.

Because Kibana is a JavaScript application, requests originate from the user's IP address. IP-based access control might be impractical due to the sheer number of IP addresses you would need to whitelist in order for each user to have access to Kibana. One workaround is to place a proxy server between Kibana and Amazon ES. Then you can add an IP-based access policy that allows requests from only one IP address, the proxy's. The following diagram shows this configuration.

  1. This is your Amazon ES domain. IAM provides authorized access to this domain. An additional, IP-based access policy provides access to the proxy server.

  2. This is the proxy server, running on an Amazon EC2 instance.

  3. Other applications can use the Signature Version 4 signing process to send authenticated requests to Amazon ES.

  4. Kibana clients connect to your Amazon ES domain through the proxy.

To enable this sort of configuration, you need a resource-based policy that specifies roles and IP addresses. Here's a sample policy:

{ "Version": "2012-10-17", "Statement": [{ "Resource": "arn:aws:es:us-west-2:111111111111:domain/my-domain/*", "Principal": { "AWS": "arn:aws:iam::111111111111:role/allowedrole1" }, "Action": [ "es:ESHttpGet" ], "Effect": "Allow" }, { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Condition": { "IpAddress": { "aws:SourceIp": [ "123.456.789.123" ] } }, "Resource": "arn:aws:es:us-west-2:111111111111:domain/my-domain/*" } ] }

We recommend that you configure the EC2 instance running the proxy server with an Elastic IP address. This way, you can replace the instance when necessary and still attach the same public IP address to it. To learn more, see Elastic IP Addresses in the Amazon EC2 User Guide for Linux Instances.

If you use a proxy server and Amazon Cognito Authentication for Kibana, you might need to add settings for Kibana and Amazon Cognito to avoid redirect_mismatch errors. See the following nginx.conf example:

server { listen 443; location /login { proxy_pass https://$cognito_host/login; proxy_cookie_domain $cognito_host $proxy_host; proxy_redirect https://$kibana_host https://$proxy_host; } location / { proxy_pass https://$kibana_host; proxy_redirect https://$cognito_host https://proxy_host; proxy_cookie_domain $kibana_host $proxy_host; proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; } } $cognito_host=your-cognito-domain-name.auth.us-west-2.amazoncognito.com $kibana_host=search-your-es-domain.us-west-2.es.amazonaws.com $proxy_host=your-proxy-server.us-west-2.compute.amazonaws.com

Configuring Kibana to Use a WMS Map Server

Due to licensing restrictions, the default installation of Kibana on Amazon ES domains that use Elasticsearch 5.x or greater does not include a map server for tile map visualizations. Use the following procedure to configure Kibana to use a Web Map Service (WMS) map server.

To configure Kibana to use a WMS map server:

  1. Open Kibana. You can find a link to Kibana in the domain summary at https://console.aws.amazon.com/es/.

  2. Choose Management.

  3. Choose Advanced Settings.

  4. Locate visualization:tileMap:WMSdefaults, and then choose the edit button to modify the default value.

  5. Change enabled to true and url to the URL of a valid WMS map server.

  6. (Optional) Locate visualization:tileMap:WMSdefaults, and then choose the edit button to modify the default value.

  7. (Optional) Change "layers": "0" to a comma-separated list of map layers that you want to display. Layers vary by map service. The default value of 0 is often appropriate.

  8. Choose the save button.

To apply the new default value to visualizations, you might need to reload Kibana.

Note

Map services often have licensing fees or restrictions. You are responsible for all such considerations on any map server that you specify. You might find the map services from the U.S. Geological Survey useful for testing.

Connecting a Local Kibana Server to Amazon ES

If you have invested significant time into configuring your own Kibana instance, you can use it instead of (or in addition to) the default Kibana instance that Amazon ES provides.

To connect a local Kibana server to Amazon ES:

  • Make the following changes to config/kibana.yml:

    kibana_index: ".kibana-5" elasticsearch_url: "http://elasticsearch_domain_endpoint:80"

You must use the http prefix and explicitly specify port 80.

Loading Bulk Data with the Logstash Plugin

Logstash provides a convenient way to use the bulk API to upload data into your Amazon ES domain with the S3 plugin. The service also supports all other standard Logstash input plugins that are provided by Elasticsearch. Amazon ES also supports two Logstash output plugins: the standard Elasticsearch plugin and the logstash-output-amazon-es plugin, which signs and exports Logstash events to Amazon ES.

You must install your own local instance of Logstash and make the following changes in the Logstash configuration file to enable interaction with Amazon ES.

Configuration Field Input | Output Plugin Description
bucket Input Specifies the Amazon S3 bucket containing the data that you want to load into an Amazon ES domain.
region Input Specifies the AWS Region where the Amazon S3 bucket resides.
hosts Output Specifies the service endpoint for the target Amazon ES domain.
ssl Output Specifies whether to use SSL to connect to Amazon ES.

This example configures Logstash to do the following:

  • Point the output plugin to an Amazon ES endpoint

  • Point to the input plugin to the wikipedia-stats-log bucket in S3

  • Use SSL to connect to Amazon ES

input{ s3 { bucket => "wikipedia-stats-log" access_key_id => "lizards" secret_access_key => "lollipops" region => "us-east-1" } } output{ elasticsearch { hosts => "search-logs-demo0-cpxczkdpi4bkb4c44g3csyln5a.us-east-1.es.example.com" ssl => true } }

Note

The service request in the preceding example must be signed. For more information about signing requests, see Signing Amazon ES Requests. Use the logstash-output-amazon-es output plugin to sign and export Logstash events to Amazon ES. For instructions, see the plugin README.