User Tools

Site Tools


Navigation Menu

Flight-Control

<
Previous mounth
03/29/2024
>
Next mounth
SMTWFTS
13
24
24
25
25
26
26
27
27
28
28
29
29
30
1431010203040506
1507080910111213
1614151617181920
1721222324252627









Hot Projects

SEEDStack

SEEDStack - Open 3D printable seed/sprouting systemDIY Food Hacking

UCSSPM

UCSSPM - Unified Clear-Sky Solar Prediction ModelOpen Solar Power

picoReflow

picoReflow - DIY PID Reflow Oven Controller based on RaspberryPiDIY Reflow Soldering

PiGI

PiGI - DIY Geiger Counter based on RaspberryPiRasPi Geiger Counter

DIY ARA-2000

Active Wideband Receiver Antenna for SDR - ARA-2000Wideband Antenna

DSpace

DSPace - Map everythingMap everything!

Mission-Tags

This is an old revision of the document!


Elasticsearch Garbage Collector

If you're using Elasticsearch you'll sooner or later have to deal with diskspace issues. The setup I currently have to manage gathers 200 to 300 million docs per 24 hours and a solution was needed to always guarantee enough free diskspace so that Elasticsearch wouldn't fail.

The following bash script is the last incarnation of this quest to have an automatic “ringbuffer”:

#!/bin/bash

  LOCKFILE=/var/run/egc.lock

# Check if Lockfile exists

  if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
    echo "EGC process already running"
    exit 1
  fi

# Make sure the Lockfile is removed 
# when we exit and then claim it

  trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT

# Create Lockfile

  echo $$ > ${LOCKFILE}

# Always keep a minimum of 30GB free in logdata
# by sacrificing oldest index (ringbuffer)

  DF=$(/bin/df /dev/md0 | sed '1d' | awk '{print $4}')

  if [ ${DF} -le 30000000 ]; then
    INDEX=$(/bin/ls -1t /logdata/elasticsearch/nodes/0/indices/ | tail -1)
    curl -XDELETE "http://localhost:9200/${INDEX}"
  fi

# Check & clean elasticsearch logs 
# if disk usage is > 10GB

  DU=$(/usr/bin/du /var/log/elasticsearch/ | awk '{print $1}')

  if [ ${DU} -ge 10000000 ]; then
    rm /var/log/elasticsearch/elasticsearch.log.20*
  fi

# Remove Lockfile

  rm -f ${LOCKFILE}

  exit 0

Make sure to check/modify the script to reflect your particular setup: It's very likely that your paths and device names are different.

It runs every 10 minutes (as a cron job) and checks the available space on the device where Elasticsearch stores its indices. In this example /logdata is mounted on /dev/md0. If md0 has less than 30gb of free diskspace it automagically finds the oldest Elasticsearch index and drops it via Elasticsearch's REST API.

A simple locking mechanism will prevent multiple running instances in case of timing issues. All you need is curl for it to work and it will increase your storage efficiency so that you can always have as much past data available as your storage allows without the risk of full disk issues or manual maintaining.

Discussion

Really good post

chrono, 2014/02/20 15:06

The script has been updated to only find logstash-* indexes, otherwise it could tamper with river or marvel/kibana indexes.

Leo, 2023/11/29 07:05

If you are interested in your favorite actors' heights and detailed information, please take a look at celeb height wiki database.

OKBet, 2023/12/14 02:20

It's nice to see your writing, which is exactly what I need, it's very detailed 15% First Deposit Bonus ng OKBet

colorgame, 2024/01/29 03:28

Salamat sa pagbabahagi ng iyong kaalaman. Ito'y naging inspirasyon sa akin. Sana'y masilip mo ang aking website:

christmas jewelry, 2024/01/29 03:52

I deeply appreciate your article, it provided much-needed guidance. Please consider visiting my website:

25fox, 2024/02/12 23:40

Buy a driver's license Hello, welcome to the world's largest online driver's license organization. We sell authentic and registered driving licenses and we have several driving schools with which we collaborate.

https://origineelrijbewijskopen.com/

https://origineelrijbewijskopen.com/2023/11/10/rijbewijs-kopen-telegram/

https://origineelrijbewijskopen.com/2023/10/24/rijbewijs-kopen-nederland/ https://origineelrijbewijskopen.com/2023/11/01/rijbewijs-kopen-belgie/

https://origineelrijbewijskopen.com/2023/11/07/rijbewijs-a-kopen/

https://origineelrijbewijskopen.com/2023/11/07/rijbewijs-kopen-prijs/

https://origineelrijbewijskopen.com/2023/11/07/rijbewijs-te-koop-belgie/

https://origineelrijbewijskopen.com/2023/11/10/belgisch-rijbewijs-kopen/

https://origineelrijbewijskopen.com/2023/11/10/rijbewijs-kopen-duitsland/ https://origineelrijbewijskopen.com/2023/11/10/rijbewijs-examens-kopen/

https://origineelrijbewijskopen.com/2023/11/10/rijbewijs-kopen-hongarije/

What a nice post! I'm so happy to read this. What you wrote was very helpful to me. Thank you. Actually, I run a site similar to yours. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.

https://cf-cc-qualitylab.com/

https://cf-cc-qualitylab.com/product/american-dollars-for-sale/

https://cf-cc-qualitylab.com/product/buy-counterfeit-aud-online/

https://cf-cc-qualitylab.com/product/buy-fake-british-pounds-online/

https://cf-cc-qualitylab.com/product/buy-fake-euro-banknotes-online/

https://cf-cc-qualitylab.com/product/buy-new-zealand-dollars-online/ https://cf-cc-qualitylab.com/product/buy-undetected-canadian-dollars/

It is incredibly average to see the best inconspicuous components presented in a basic and seeing way Thank you. Actually, I run a site similar to yours. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.

https://cf-cc-qualitylab.com/product/paypal-flip/

https://cf-cc-qualitylab.com/product/western-union-flip/

https://cf-cc-qualitylab.com/product/cash-app-flip/ https://cf-cc-qualitylab.com/product/western-union-flip-sale/

https://cf-cc-qualitylab.com/product/legit-paypal-flip/

https://cf-cc-qualitylab.com/product/legit-cash-app-flip/ https://cf-cc-qualitylab.com/product/buy-cloned-cards/

https://cf-cc-qualitylab.com/product/buy-credit-card-dumps-online/

https://cf-cc-qualitylab.com/product/cloned-credit-cards-for-sale/ https://cf-cc-qualitylab.com/product/legit-paypal-money-transfer/

https://cf-cc-qualitylab.com/product/legit-paypal-money-transfer/

https://cf-cc-qualitylab.com/product/western-union-money-flip/

https://cf-cc-qualitylab.com/product/hacked-cash-app/

Such an especially significant article. To a great degree charming to examine this article.I should need to thank you for the undertakings you had made for creating this astonishing article. https://cf-cc-qualitylab.com/product/automated-money-developer-machines/

https://cf-cc-qualitylab.com/product/high-quality-automatic-solution/ https://cf-cc-qualitylab.com/product/ssd-chemical-solution-packaging-materials/

https://cf-cc-qualitylab.com/product/vector-paste-ssd-solution/ https://cf-cc-qualitylab.com/product/humine-activation-powder/

https://cf-cc-qualitylab.com/product/99-99-pure-gallium-metal-alloy-for-mercury-replacement/ https://cf-cc-qualitylab.com/product/automated-money-developer-machines/

I am another client of this site so here I saw different articles and posts posted by this site,I inquisitive more enthusiasm for some of them trust you will give more data on this points in your next articles.

https://cf-cc-qualitylab.com/2024/02/01/western-union-uberweisen-umdrehen/ https://cf-cc-qualitylab.com/2024/02/01/kaufen-sie-western-union-umdrehen/

https://cf-cc-qualitylab.com/product/automated-money-developer-machines/

https://cf-cc-qualitylab.com/2024/02/01/koop-western-union-omzet/

https://cf-cc-qualitylab.com/2024/01/31/acquistare-lanciare-western-union/

https://cf-cc-qualitylab.com/2024/01/31/contanti-ribaltabili-della-western-union/

https://cf-cc-qualitylab.com/2024/01/30/volteo-de-dinero-de-western-union/

https://cf-cc-qualitylab.com/2024/01/30/comprar-voltear-de-western-union/

https://cf-cc-qualitylab.com/2024/01/28/transfert-dargent-western-union/

https://cf-cc-qualitylab.com/2024/01/28/retournement-de-western-union/

https://cf-cc-qualitylab.com/2024/01/27/online-kredietkaart-dumps-kopen/

https://cf-cc-qualitylab.com/2024/01/27/gekloonde-kredietkaarten-te-koop/

https://cf-cc-qualitylab.com/2024/01/27/gekloonde-kaarten-kopen/ https://cf-cc-qualitylab.com/2024/01/27/kaufen-sie-kreditkarten-dumps-online/

https://cf-cc-qualitylab.com/2024/01/27/geklonte-karten-dumps-zum-verkauf/

All things considered I read it yesterday yet I had a few musings about it and today I needed to peruse it again in light of the fact that it is extremely elegantly composed.

https://cf-cc-qualitylab.com/2024/01/27/kaufen-sie-geklonte-karten/

https://cf-cc-qualitylab.com/2024/01/27/acquistare-scarico-di-carte-di-credito-in-linea/ https://cf-cc-qualitylab.com/2024/01/27/carte-di-credito-clonate-in-vendita/ https://cf-cc-qualitylab.com/2024/01/27/carte-clonate-in-vendita/ https://cf-cc-qualitylab.com/2024/01/26/comprar-volcados-de-tarjetas-de-credito/

https://cf-cc-qualitylab.com/2024/01/26/tarjetas-de-credito-clonadas-a-la-venta/ https://cf-cc-qualitylab.com/2024/01/25/comprar-tarjetas-clonadas/ https://cf-cc-qualitylab.com/2024/01/25/acheter-des-dumps-de-cartes-de-credit-en-ligne/

https://cf-cc-qualitylab.com/2024/01/25/cartes-de-credit-clonees-a-vendre/

https://cf-cc-qualitylab.com/2024/01/25/cartes-clonees-a-vendre/ https://cf-cc-qualitylab.com/2024/01/23/requisitengeld-zum-verkauf/ https://cf-cc-qualitylab.com/2024/01/23/falschgeld-kaufen/

https://cf-cc-qualitylab.com/2024/01/23/gefalschter-euro-zum-verkauf/

https://cf-cc-qualitylab.com/2024/01/21/koop-valse-polymeerbiljetten/ https://cf-cc-qualitylab.com/2024/01/21/koop-valse-bankbiljetten/

 Actually, I run a site similar to yours. If you have time, could you visit my site? Please leave your comments after reading what I wrote. If you do so, I will actively reflect your opinion. I think it will be a great help to run my site. Have a good day.

https://cf-cc-qualitylab.com/2024/01/21/koop-valse-euro-bankbiljetten/

https://cf-cc-qualitylab.com/2024/01/21/comprar-dinero-falso-en-linea/

https://cf-cc-qualitylab.com/2024/01/21/comprar-dinero-falso/

https://cf-cc-qualitylab.com/2024/01/21/billetes-de-euro-a-la-venta/

https://cf-cc-qualitylab.com/2024/01/17/soldi-realistici-in-vendita/

https://cf-cc-qualitylab.com/2024/01/17/acquistare-denaro-falso/ https://cf-cc-qualitylab.com/2024/01/17/banconote-in-euro-in-vendita/ https://cf-cc-qualitylab.com/2024/01/17/argent-accessoire-a-vendre/ https://cf-cc-qualitylab.com/2024/01/17/acheter-des-billets-indetectables/ https://cf-cc-qualitylab.com/2024/01/17/faux-euros-a-vendre-en-ligne/

https://cf-cc-qualitylab.com/2024/01/09/buy-euros-online-cyprus/

https://cf-cc-qualitylab.com/2024/01/09/fake-polymer-notes-for-sale/

https://cf-cc-qualitylab.com/2024/01/09/counterfeit-money-for-sale/

https://cf-cc-qualitylab.com/2024/01/09/buy-fake-dollar-bills/

https://cf-cc-qualitylab.com/2024/01/09/buy-prop-money-canada/

https://cf-cc-qualitylab.com/2024/01/09/buy-fake-polymer-notes/

Enter your comment. Wiki syntax is allowed:
W R S F N