Deleting a Time Series

Video Lecture

Description

There may be a time when you want to delete data from the Prometheus TSDB database.

Data will be automatically deleted after the storage retention time has passed. By default it is 15 days.

If you want to delete specific data earlier, then you are able.

You need to enable the admin api in Prometheus before you can.

sudo nano /etc/default/prometheus

Add --web.enable-admin-api to the ARGS="" variable. eg,

ARGS="--web.enable-admin-api"

Restart Prometheus and check status

sudo service prometheus restart
sudo service prometheus status

You can now make calls to the admin api.

In my example I want to delete all time series for the instance="sbcode.net:9100"

So I run the delete_series api endpoint providing the value to match. eg,

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="sbcode.net:9100"}'

When I re execute the Prometheus query, the time series I wanted deleted no longer exists.

You can have more complicated match queries, eg,

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=a_bad_metric&match[]={region="mistake"}'

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_exporter:memory:percent'

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_filesystem_free_percent'

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_memory_MemFree_percent'

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS'

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS_FOR_STATE'

After deleting you should disable the admin api again,

sudo nano /etc/default/prometheus

Remove --web.enable-admin-api from the ARGS variable. eg,

ARGS=""

Restart Prometheus and check status

sudo service prometheus restart
sudo service prometheus status

Useful Links

Delete Series