Script for backing up to Google Nearline storage

Recently I’ve been working on a mechanism to mirror a dataset from a local filesystem to Google Nearline cloud storage in an encrypted format. The costs are really the compelling factor. While I could buy some hard drives and put them at another location (called the colo-buddy system!) it just made sense to not have to deal with the logistics, maintenance, power, network, etc… Of course being me, it’s a PHP cli script, because well – me. If you’re looking…

Read More

Purging old indexes from elasticsearch with logstash

Currently doing some work on logstash and found myself wanting to delete indexes over a certain age. The following PHP script gets the job done. I use php as my primary command line scripting language so use or port as interested. #!/usr/bin/php <?php date_default_timezone_set(“America/Chicago”); $index_name = “logstash-“; $purge_age = 300; $elastic = “http://elastic.domain.com:9200”; $data = file_get_contents(“${elastic}/_cat/indices/${index_name}*”); foreach(split(“\n”, $data) as $line) { $split = preg_split(“/\s+/”, $line); if(count($split) > 1) { $date = preg_replace(“/${index_name}|\./”, “”, $split[2]); $split[] = floor((time() – strtotime($date)) /…

Read More

Filtering output on linux with color

Last night I found myself desperate to read the output of tail but looking for specific content. (Basically tailing a log file and needing to have the fields pop out at me.) There’s no easy colorize command in linux so I did some digging, found something similar and modified it. (This is based on Kenny Moen’s blog entry about baretail on unix.) Anyway, the following perl script will allow any content to be highlighted using a simple regex and a…

Read More

Linux raid hard drive serial numbers and temperatures

Thought I’d share this little script. Basically what I needed to do was build a way to track what hard drives were connected to my linux raid arrays along with their serial numbers and temperatures. This script requires hdparm and hddtemp which are both readily available on most distros. #!/bin/bash # set LANG for encoding celcius symbol, UTF-8 screws it up LANG=en_US IFS=$’\n’ for file in /dev/md[0-9]*; do VAL=`mdadm -D $file`; for line in $VAL; do if [[ $line =~…

Read More