Using the Prometheus data source
Prometheus settings
| Name | Description |
|---|---|
Name
|
The data source name. This is how you see the data source in panels and queries. |
Default
|
Default data source means that it will be pre-selected for new panels. |
Url
|
The URL of your Prometheus server; for
example, https://prometheus.example.org:9090.
|
Access
|
Server (default) = URL must be accessible from the Grafana backend/server. |
Basic Auth
|
Enable basic authentication to the Prometheus data source. |
User
|
User name for basic authentication. |
Password
|
Password for basic authentication. |
Scrape interval
|
Set this to the typical scrape and evaluation interval configured in Prometheus. Defaults to 15s. |
Disable metrics lookup
|
Checking this option will disable the metrics chooser and metric/label support in the query field’s autocomplete. This helps if you have performance issues with bigger Prometheus instances. |
Custom Query Parameters
|
Add custom parameters to the Prometheus query URL. For
example timeout, partial_response,
dedup, or
max_source_resolution. Multiple parameters
should be concatenated together with an
"&". |
Prometheus query editor
The following sections provide information and options for Prometheus query editor in the dashboard and in Explore.
Query editor in dashboards
Open a graph in edit mode by choosing the title and then choosing Edit (or by pressing e key while pausing on the panel).
| Name | Description |
|---|---|
Query expression
|
For more information about Prometheus query
expressions, see the Prometheus documentation |
Legend format
|
Controls the name of the time series, using name or
pattern. For example {{hostname}} is
replaced with the label value for the label
hostname. |
Min step
|
An additional lower limit for the step parameter of Prometheus range
queries$__interval
and $__rate_interval variables. The limit
is absolute and not modified by the
Resolution setting. |
Resolution
|
1/1 sets both the $__interval
variable and the step parameter of Prometheus range
queries1/2 only retrieves a data
point for every other pixel, and 1/10
retrieves one data point per 10 pixels. Note that both
Min time interval and
Min step limit the final value
of $__interval and step.
|
Metric lookup
|
Search for metric names in this input field. |
Format as
|
Switch between Table, Time
series, or Heatmap.
Table works only in the table panel.
Heatmap is suitable for displaying
metrics of the histogram type on a heatmap panel. It
converts cumulative histograms to regular ones and sorts
series by the bucket bound. |
Instant
|
Perform an "instant" query, to return only the latest value that Prometheus has scraped for the requested time series. Instant queries return results much faster than normal range queries. Use them to look up label sets. |
Min time interval
|
This value multiplied by the denominator from the
Resolution setting sets a lower
limit to both the $__interval variable and
the step parameter of Prometheus range
queries |
Note
Amazon Managed Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.
Instant queries in dashboards
The Prometheus data source allows you to run instant queries, which query only the latest value. You can visualize the results in a table panel to see all available labels of a time series.
Instant query results are made up of only one data point per
series. They can be shown in the graph panel with the help of series
overrides. To show them in the graph as a latest value point, add a
series override and select Points > true. To show a
horizontal line across the whole graph, add a series override and
select Transform > constant For more information
about series overrides, see Series overrides.
Query editor in Explore
| Name | Description |
|---|---|
Query expression
|
For more information about Prometheus query
expression, see the Prometheus documentation |
Step
|
Step parameter of Prometheus range
queriess (seconds). |
Query type
|
Range, Instant, or
Both. When running Range query, the result of
the query is displayed in graph and table. Instant query
returns only the latest value that Prometheus has
scraped for the requested time series and it is
displayed in the table. When Both is selected, both instant query and
range query is run. Result of range query is displayed
in graph and the result of instant query is displayed in
the table. |
Metrics browser
The metrics browser allows you to quickly find metrics and select relevant labels to build basic queries. When you open the browser you will see all available metrics and labels. If supported by your Prometheus instance, each metric will show its HELP and TYPE as a tooltip.
When you select a metric, the browser narrows down the available labels to show only the ones applicable to the metric. You can then select one or more labels for which the available label values are shown in lists in the bottom section. Select one or more values for each label to tighten your query scope.
Note
If you do not remember a metric name to start with, you can also select a few labels first, to narrow down the list and then find relevant label values.
All lists in the metrics browser have a search field above them to quickly filter for metrics or labels that match a certain string. The values section only has one search field. Its filtering applies to all labels to help you find values across labels once they have been selected, for example, among your labels app, job, job_name only one might with the value you are looking for.
Once you are satisfied with your query, click “Use query” to run the query. The Use as rate query button adds a rate(...)[$__interval] around your query to help write queries for counter metrics. The “Validate selector” button will check with Prometheus how many time series are available for that selector.
Limitations
The metrics browser has a hard limit of 10,000 labels (keys) and 50,000 label values (including metric names). If your Prometheus instance returns more results, the browser will continue functioning. However, the result sets will be cut off above those maximum limits.
Templating
Instead of hardcoding things such as server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. You can use these dropdown boxes to change the data being displayed in your dashboard.
For more information about templating and template variables, see Templates and variables.
Query variable
Variable of the type Query allows you to query Prometheus for a list of metrics, labels, or label values. The Prometheus data source plugin provides the following functions you can use in the Query input field.
| Name | Description |
|---|---|
label_names()
|
Returns a list of label names. |
label_values(label)
|
Returns a list of label values for the
label in every metric. |
label_values(metric, label)
|
Returns a list of label values for the
label in the specified metric. |
metrics(metric)
|
Returns a list of metrics matching the specified
metric regex. |
query_result(query)
|
Returns a list of Prometheus query result for the
query. |
For information about what metric names,
label names and label
values are, see the Prometheus documentation
Using interval and range variables
Note
Support for $__range, $__range_s,
and $__range_ms are available only from Grafana
v5.3.
You can use some global variables in query variables:
$__interval, $__interval_ms,
$__range, $__range_s, and
$__range_ms. For more information, see Global variables.
These can be convenient to use with the query_result
function when you must filter variable queries because the
label_values function doesn’t support queries.
To get the correct instances when changing the time range on the
dashboard, make sure to set the variable’s refresh
trigger to be On Time Range Change.
The following code example shows how to populate a variable with the busiest five request instances based on average QPS over the time range shown in the dashboard.
Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance))) Regex: /"([^"]+)"/
The following code example shows how to populate a variable with
the instances having a certain state over the time range shown in
the dashboard, using $__range_s.
Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>) Regex:
Using
$__rate_interval variable
The $__rate_interval variable is meant to be used in the
rate function. It is defined as max( $__interval +
Scrape interval, 4 * Scrape
interval). Scrape interval is the
Min step setting (AKA query_interval, a setting per PromQL query), if
any is set, and otherwise the Scrape interval as
set in the Prometheus data source (but ignoring any Min interval setting
in the panel, because the latter is modified by the resolution setting).
Using variables in queries
There are two syntaxes:
-
$<varname>Example: rate(http_requests_total{job=~"$job"}[5m]) -
[[varname]]Example: rate(http_requests_total{job=~"[[job]]"}[5m])
Why two ways? The first syntax is easier to read and write but does
not allow you to use a variable in the middle of a word. When the
Multi-value or Include all
value options are enabled, Grafana converts the labels
from plaintext to a regex compatible string. Which means you have to use
=~ instead of =.
Annotations
You can use annotations to overlay rich event information on top of graphs. You add annotation queries using the Dashboard menu or Annotations view. For more information, see Annotations.
Prometheus supports two ways to query annotations.
-
A regular metric query
-
A Prometheus query for pending and firing alerts. For more information, see Inspecting alerts during runtime
).
The step option is useful to limit the number of events returned from your query.
