cw-infra-apps/grafana/charts/feature-annotation-autodiscovery

feature-annotation-autodiscovery

Gathers metrics automatically based on Kubernetes Pod and Service annotations

The annotation-based autodiscovery feature adds scrape targets based on Kubernetes annotations.

How it works

With this feature enabled, any Kubernetes Pods or Services with the k8s.grafana.com/scrape annotation set to true will be automatically discovered and scraped by the collector.

You can use several other annotations to customize the behavior of the scrape configuration, such as:

• k8s.grafana.com/job: The value to use for the job label.
• k8s.grafana.com/instance: The value to use for the instance label.
• k8s.grafana.com/metrics.path: The path to scrape for metrics. Defaults to /metrics.
• k8s.grafana.com/metrics.portNumber: The port on the Pod or Service to scrape for metrics. This is used to target a specific port by its number, rather than all ports.
• k8s.grafana.com/metrics.portName: The named port on the Pod or Service to scrape for metrics. This is used to target a specific port by its name, rather than all ports.
• k8s.grafana.com/metrics.scheme: The scheme to use when scraping metrics. Defaults to http.
• k8s.grafana.com/metrics.scrapeInterval: The scrape interval to use when scraping metrics. Defaults to 60s.

Testing

This chart contains unit tests to verify the generated configuration. The hidden value deployAsConfigMap will render the generated configuration into a ConfigMap object. While this ConfigMap is not used during regular operation, you can use it to show the outcome of a given values file.

The unit tests use this ConfigMap to create an object with the configuration that can be asserted against. To run the tests, use helm test.

Be sure perform actual integration testing in a live environment in the main k8s-monitoring chart.

Maintainers

Name	Email	Url
petewall	pete.wall@grafana.com

Source Code

• https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-annotation-autodiscovery

Values

Annotations

Key	Type	Default	Description
annotations.instance	string	"k8s.grafana.com/instance"	Annotation for overriding the instance label
annotations.job	string	"k8s.grafana.com/job"	Annotation for overriding the job label
annotations.metricsContainer	string	"k8s.grafana.com/metrics.container"	Annotation for selecting the specific container to scrape.
annotations.metricsParam	string	"k8s.grafana.com/metrics.param"	Annotation for setting __param_<key> parameters when scraping. Example: k8s.grafana.com/metrics.param_key: "value".
annotations.metricsPath	string	"k8s.grafana.com/metrics.path"	Annotation for setting or overriding the metrics path. If not set, it defaults to /metrics
annotations.metricsPortName	string	"k8s.grafana.com/metrics.portName"	Annotation for setting the metrics port by name.
annotations.metricsPortNumber	string	"k8s.grafana.com/metrics.portNumber"	Annotation for setting the metrics port by number.
annotations.metricsScheme	string	"k8s.grafana.com/metrics.scheme"	Annotation for setting the metrics scheme, default: http.
annotations.metricsScrapeInterval	string	"k8s.grafana.com/metrics.scrapeInterval"	Annotation for overriding the scrape interval for this service or pod. Value should be a duration like "15s, 1m". Overrides metrics.autoDiscover.scrapeInterval
annotations.metricsScrapeTimeout	string	"k8s.grafana.com/metrics.scrapeTimeout"	Annotation for overriding the scrape timeout for this service or pod. Value should be a duration like "15s, 1m". Overrides metrics.autoDiscover.scrapeTimeout
annotations.scrape	string	"k8s.grafana.com/scrape"	Annotation for enabling scraping for this service or pod. Value should be either "true" or "false"

Scrape Settings

Key	Type	Default	Description
bearerToken	object	{"enabled":true,"token":"/var/run/secrets/kubernetes.io/serviceaccount/token"}	Sets bearer_token_file line in the prometheus.scrape annotation_autodiscovery.
scrapeInterval	string	60s	How frequently to scrape metrics from discovered pods and services. Only used if the k8s.grafana.com/metrics.scrapeInterval annotation is not set. Overrides global.scrapeInterval
scrapeTimeout	string	10s	The scrape timeout for discovered pods and services. Only used if the k8s.grafana.com/metrics.scrapeTimeout annotation is not set. Overrides global.scrapeTimeout

Discovery Settings

Key	Type	Default	Description
excludeNamespaces	list	[]	The list of namespaces to exclude from autodiscovery.
extraDiscoveryRules	string	""	Rule blocks to be added to the discovery.relabel component for discovered pods and services. These relabeling rules are applied pre-scrape against the targets from service discovery. Before the scrape, any remaining target labels that start with __ (i.e. __meta_kubernetes*) are dropped. (docs)
labelSelectors	object	{}	Filter the list of discovered pods and services by labels. Example: labelSelectors: { 'app': 'myapp' } will only discover pods and services with the label app=myapp. Example: labelSelectors: { 'app': ['myapp', 'myotherapp'] } will only discover pods and services with the label app=myapp or app=myotherapp.
namespaces	list	[]	The list of namespaces to include in autodiscovery. If empty, all namespaces are included.

Metric Processing Settings

Key	Type	Default	Description
extraMetricProcessingRules	string	""	Rule blocks to be added to the prometheus.relabel component for discovered pods and services. These relabeling rules are applied post-scrape against the metrics returned from the scraped target, no __meta* labels are present. (docs)
maxCacheSize	string	nil	Sets the max_cache_size for cadvisor prometheus.relabel component. This should be at least 2x-5x your largest scrape target or samples appended rate. (docs) Overrides global.maxCacheSize
metricsTuning.excludeMetrics	list	[]	Metrics to drop. Can use regular expressions.
metricsTuning.includeMetrics	list	[]	Metrics to keep. Can use regular expressions.

General settings

Key	Type	Default	Description
fullnameOverride	string	""	Full name override
nameOverride	string	""	Name override

Global Settings

Key	Type	Default	Description
global.maxCacheSize	int	100000	Sets the max_cache_size for every prometheus.relabel component. (docs) This should be at least 2x-5x your largest scrape target or samples appended rate.
global.scrapeInterval	string	"60s"	How frequently to scrape metrics.
global.scrapeTimeout	string	"10s"	The scrape timeout for discovered pods and services.

Pod Discovery Settings

Key	Type	Default	Description
pods.enabled	bool	true	Enable discovering Pods with annotations.
pods.labelSelectors	object	{}	Filter the list of discovered Pods by labels. Example: labelSelectors: { 'app': 'myapp' } will only discover Pods with the label app=myapp. Example: labelSelectors: { 'app': ['myapp', 'myotherapp'] } will only discover Pods with the label app=myapp or app=myotherapp.

Pod Metric Processing Settings

Key	Type	Default	Description
pods.labels	object	{}	Add labels to metrics from discovered Pods. Runs during discovery, so _meta labels are available. See the documentation for the full list of meta labels.
pods.staticLabels	object	{}	Metric labels to set with static data for discovered Pods.
pods.staticLabelsFrom	object	{}	Static labels to set on metrics from discovered Pods, not quoted so it can reference config components.

Services

Key	Type	Default	Description
services.enabled	bool	true	Enable discovering Services with annotations.

Service Discovery Settings

Key	Type	Default	Description
services.labelSelectors	object	{}	Filter the list of discovered Services by labels. Example: labelSelectors: { 'app': 'myapp' } will only discover Services with the label app=myapp. Example: labelSelectors: { 'app': ['myapp', 'myotherapp'] } will only discover Services with the label app=myapp or app=myotherapp.

Service Metric Processing Settings

Key	Type	Default	Description
services.labels	object	{}	Add labels to metrics from discovered Services. Run during discovery, so _meta labels are available. See the documentation for the full list of meta labels.
services.staticLabels	object	{}	Metric labels to set with static data for discovered Services.
services.staticLabelsFrom	object	{}	Static labels to set on metrics from discovered Services, not quoted so it can reference config components.