Control the index data retention in Splunk - Purge and Claim disk space

Written By askMLabs on Sunday, August 2, 2020 | 12:03 PM

This article is about index data retention of Splunk. Splunk is a too that is used for searching, monitoring, and analyzing machine-generated big data via a Web-style interface.

If you keep indexing your data, all the indexed data will be stored in index. It will keep increasing your disk space. At some point, you need to think of data retention to save the disk space. The default value of data retention is "188697600" seconds(Apprx 6 Yrs). Keeping the historical data depends on the use case. If you need 6 Yrs worth of historical data, you can keep the settings same and estimate the disk space requirements appropriately.

In our specific case, I don't need 6 years worth of indexed data. Becase I know the specific use case of our data, I can decide the retention of this data. Lets assume, I have to set the retention of the indexed data to 30days.

30days = 30*24*60*60 seconds = 2592000 seconds

The configuration file where we can set default retention that applies to all the indexes in the Splunk is /opt/splunk/etc/system/default/indexes.conf. And the configuration parameter that controls the retention period is "frozenTimePeriodInSecs".

Following steps should help you setting this parameter.....

Check the disk space....( My splunk indexes are using /opt/splunk for storing indexed data).

[root@askmlabs-splunk01 ~]# df -h /opt/splunk

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/data-data

493G 416G 52G 90% /opt/splunk

[root@askmlabs-splunk01 ~]#

Modify the parameter frozenTimePeriodInSecs in file /opt/splunk/etc/system/default/indexes.conf

NOTE : There are multiple places that this parameter is specified in the indexes.conf file. You need to modify the parameter under the section named "index specific defaults".

[root@askmlabs-splunk01 default]# diff indexes.conf indexes.conf_bak
42c42
< frozenTimePeriodInSecs = 2592000
---
> frozenTimePeriodInSecs = 188697600
[root@askmlabs-splunk01 default]#

Now restart the Splunk instance to take the value effective....

[root@askmlabs-splunk01 default]# /opt/splunk/bin/splunk restart

Check if the disk space has been reduced by changing the retention....

[root@askmlabs-splunk01 default]# df -h /opt/splunk
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/data-data
493G 224G 244G 48% /opt/splunk
[root@askmlabs-splunk01 default]#

Conclusion :

Splunk indexed data retention can be controlled using the parameter frozenTimePeriodInSecs in the configuration file /opt/splunk/etc/system/default/indexes.conf.

Hope this information helps you. Please post your questions in the comments section.

Thanks

SRI

Share this article :

askMLabs