Deleting a Time Series
Video Lecture
Description
There may be a time when you want to delete data from the Prometheus TSDB database.
Data will be automatically deleted after the storage retention time has passed. By default it is 15 days.
If you want to delete specific data earlier, then you are able.
You need to enable the admin api in Prometheus before you can.
sudo nano /etc/default/prometheus
Add --web.enable-admin-api
to the ARGS=""
variable. eg,
ARGS="--web.enable-admin-api"
Restart Prometheus and check status
sudo service prometheus restart sudo service prometheus status
You can now make calls to the admin api.
In my example I want to delete all time series for the instance="sbcode.net:9100"
So I run the delete_series api endpoint providing the value to match. eg,
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="sbcode.net:9100"}'
When I re execute the Prometheus query, the time series I wanted deleted no longer exists.
You can have more complicated match queries, eg,
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=a_bad_metric&match[]={region="mistake"}' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_exporter:memory:percent' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_filesystem_free_percent' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_memory_MemFree_percent' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=ALERTS_FOR_STATE'
After deleting you should disable the admin api again,
sudo nano /etc/default/prometheus
Remove --web.enable-admin-api
from the ARGS
variable. eg,
ARGS=""
Restart Prometheus and check status
sudo service prometheus restart sudo service prometheus status