Monitoring ELK dataset with Ruby

As a longtimeĀ elasticsearch/logstash user,Ā one of the things I’ve always planned to do was setup basic rules-based monitoring of elasticsearch. I’ve broken ground on the work to do so and am sharing this simple framework for anyone who would like to expand upon it. All of this was put together over 2 days having never used ruby before so this should not be taken as an opportunity to assess the quality of the code. Functionality and security were the first considerations.…

Read More

Continuing the saga with Google Nearline backups

My last write up about backing up to Google nearline with a script had a flaw I pointed out which was the single threaded nature and the slow process to backup a large dataset. Well wait no longer. (If you were waiting.) I’ve added pthread support to the script and an update is attached. You will need to add pthread support to your PHP installation. This will also require ZTS support. Note that a lot of bundled PHP copies do…

Read More

Script for backing up to Google Nearline storage

Recently I’ve been working on a mechanism to mirror a dataset from a local filesystem to Google Nearline cloud storage in an encrypted format. The costs are really the compelling factor. While I could buy some hard drives and put them at another location (called the colo-buddy system!) it just made sense to not have to deal with the logistics, maintenance, power, network, etc… Of course being me, it’s a PHP cli script, because well – me. If you’re looking…

Read More

Purging old indexes from elasticsearch with logstash

Currently doing some work on logstash and found myself wanting to delete indexes over a certain age. The following PHP script gets the job done. I use php as my primary command line scripting language so use or port as interested.

 

Read More

Water still over the bridge

Nearly 8 years to the day and here I am talking about net neutrality again. This is a conversation that refuses to end because in the natural order companies want to maximize their profits and find new revenue streams. However when they want to find those streams without introducing new services or making existing services better it is always opposite the values of the consumer. And in this case they want to muscle it in. They’ve never had such a…

Read More

New Digs

Not all VPS providers are equal. That is to say that you get what you pay for and while I’m paying more than I was, I want to take a minute to make a recommendation for a company that has gone out of their way to help me allocate a new home for many of my web services. In this case I’m talking about Point North Networks / Kickass VPS. You see I have been a customer of burst.net for…

Read More

Filtering output on linux with color

Last night I found myself desperate to read the output of tail but looking for specific content. (Basically tailing a log file and needing to have the fields pop out at me.) There’s no easy colorize command in linux so I did some digging, found something similar and modified it. (This is based on Kenny Moen’s blog entry about baretail on unix.) Anyway, the following perl script will allow any content to be highlighted using a simple regex and a…

Read More

Redundancy with IPtables/netfilter – VRRP

Ever try to setup iptables in a redundant configuration? That’s exactly what I recently set out to accomplish and after a few bumps, figured out a nice clean way to accomplish it. Granted, I don’t yet have a way to do state failover but for my environment that wasn’t a concern. That being said, this was accomplished using iptables and keepalived. I did not use ipvs. I just built two iptables systems and installed keepalived. vrrp is done on the…

Read More

Linux raid hard drive serial numbers and temperatures

Thought I’d share this little script. Basically what I needed to do was build a way to track what hard drives were connected to my linux raid arrays along with their serial numbers and temperatures. This script requires hdparm and hddtemp which are both readily available on most distros. #!/bin/bash # set LANG for encoding celcius symbol, UTF-8 screws it up LANG=en_US IFS=$’\n’ for file in /dev/md[0-9]*; do VAL=mdadm -D $file; for line in $VAL; do if [[ $line =~…

Read More

Should I run Vista?

As a long running advocate of Linux, I know I’m stepping out of my usual terrain but I think it has to be said by a competent computer user: Yes, you can run Vista. Don’t get me wrong, I still have both desktop and server installations of Linux and love it dearly. I also have a desktop that runs Vista. In fact I took it a step further and run 64-bit Vista and other than Quicken 2008 which gets shut…

Read More