22 May

Water still over the bridge

Nearly 8 years to the day and here I am talking about net neutrality again. This is a conversation that refuses to end because in the natural order companies want to maximize their profits and find new revenue streams. However when they want to find those streams without introducing new services or making existing services better it is always opposite the values of the consumer. And in this case they want to muscle it in.

They’ve never had such a great opportunity with the new FCC chair being a former lobbyist on their behalf. The courts recently said that the FCC did not have the power to implement rules the way they had previously done to support net neutrality so now that they have an opportunity and requirement to rewrite those rules, it appears they will look at ways to do so that support the big internet service providers goals of prioritizing the internet in their favor and increasing costs to end users. And make no mistake that this will absolutely increase costs.

After all, it’s not like our internet in the US is not already overpriced. Take a look at the chart below which is a 2013 review of the cost of internet adjusted in price based upon individual buying power (a true scale of economic cost) and you will quickly see that while our level of service is on par with many other developed countries, our costs are significantly higher. This means that consumers in America pay more for internet. This is not because fiber in the US is more expensive than it is in those other countries. It is because we’ve been at this longer and there is limited competition in this space. Not only are costs not controlled by labelling these corporations as public utilities but also many standard consumer protections like those guarding against monopolies are not applied.


Source: The cost of connectivity, 2013. New America Foundation

Businesses argue that they should have the right to charge however they want for their services. However they never acknowledge that the businesses they are going after (Google, Netflix, etc) are also consumers of internet service. They pay very expensive bills for very large pipes to the internet.

The real issue is the fact that people use the internet more now than they used to and these businesses which have so far profited extensively on high prices for basic services also appear to have short changed everyone by installing the smallest level of service they could get away with. Now that consumers are actually using the service they paid for, these companies are having to invest heavily to upgrade that infrastructure at a cost to the profits they have pocketed over the years.

I truly believe that in the end we have come to a head and that it is time to either introduce legislation that makes competition easy to create in the internet/telecommunications space or appropriately label these organizations public utilities.


17 May

New Digs

Not all VPS providers are equal. That is to say that you get what you pay for and while I’m paying more than I was, I want to take a minute to make a recommendation for a company that has gone out of their way to help me allocate a new home for many of my web services. In this case I’m talking about Point North Networks / Kickass VPS.

You see I have been a customer of burst.net for quite some time and frankly it’s only because I gave them every opportunity to address some serious issues in their performance. I monitor my VPS closely looking at uptime, utilization, processes and response content and the number of outages I experienced at burst.net were severe. Calls to support were often greeted by someone who sounded 20 years old, dazed and confused. It would generally go like this:

Me: Yes I have a VPS that is not responding. A traceroute is not completing to the first hop at your network.
Them: Oh. Uh ok. We’ll uh, have someone look at that.
Me: Ok… can you have someone contact me and let me know the ETA?
Them: Uh, sure.
Me: Ok. Thanks.
Them: Ok. Bye.

Now that I’ve transitioned, the difference is night and day. If I open a ticket someone looks into it while I’m on the phone. If I’m on chat, they get me a case number and follow-up. I had a case where a strange circumstance after a patch caused a kernel panic and the owner connected up at 2:00am and fixed it. It’s a small company but everyone I’ve worked with has been directly invested in keeping things running.

So Point North Networks gets my geek stamp of approval and kudos on the record. Good job, guys.

03 Jun

Filtering output on linux with color

Last night I found myself desperate to read the output of tail but looking for specific content. (Basically tailing a log file and needing to have the fields pop out at me.)

There’s no easy colorize command in linux so I did some digging, found something similar and modified it. (This is based on Kenny Moen’s blog entry about baretail on unix.) Anyway, the following perl script will allow any content to be highlighted using a simple regex and a pipe.

For example, to highlight an entire line wherever named is listed:
tail -f /var/log/messages | colorize –highlight “blue:.*named.*”

Or to highlight the incoming interface and outgoing interface on a firewall log with red for the inside and green for the outside:
tail -f /var/log/messages | colorize –highlight “red:w+=eth0″ –highlight “green:w+=eth1″



07 Apr

Redundancy with IPtables/netfilter – VRRP

Ever try to setup iptables in a redundant configuration? That’s exactly what I recently set out to accomplish and after a few bumps, figured out a nice clean way to accomplish it. Granted, I don’t yet have a way to do state failover but for my environment that wasn’t a concern. That being said, this was accomplished using iptables and keepalived. I did not use ipvs. I just built two iptables systems and installed keepalived. vrrp is done on the inside interface. Pretty easy other than a few gotchas. Here are some issues I ran into and how I got over them.

  • Limited IP addresses on the outside interface
    To get over this issue, I installed both firewalls without an outside IP. I turned all used addresses into virtual interfaces in the config. This way the firewall is not addressable if it’s not active. Until it claims an address it stays quiet.
  • Keepalived uses gratuitous ARP
    This was unfortunate. I use comcast business for my connection and their router does not support gratuitous ARP. It would keep the old MAC address after a failover. There are ways to get around this using aliased interfaces (look up ip link add link on google) but my way to get around this was much simpler. My firewalls are virtual machines and VMware lets you adjust MAC addresses. Therefore I gave the same MAC address to both machine’s external interfaces. To the internet it will look no different than business as usual.
  • Routes are not configured on failover
    Because I do not have an IP bound upon failover, the result is that the IP fails over but the routing table is wrong. To get over this I used a notify script as part of the config. You just put a line like line “notify_master /opt/script/masterrouteupdate” in your vrrp_instance configuration directive for the internal interface and in that file, I put:
    /sbin/route del -net default gw
    /sbin/route add -net netmask eth2
    /sbin/route add -net default gw
    Of course you’ll want to modify to fit your needs! (Some IPs modified to protect the innocent.) Note that you see I have it deleting a default rule – this is because I have two scripts, a master update and a backup update. The backup adds an internal route to the other firewall so when it reverts and pulls back inside it acts just like an inside host – mangeable, updatable, all that fun.
  • Services bound externally don’t pickup
    Some services running on the machine that bind to the external interface did not pickup on the IP change. For those, I had to add kill -HUP to the script to reread their configs and the machine setup. Problem solved.

And thus, my config is born:

global_defs {
notification_email {
smtp_connect_timeout 60
router_id stigr

vrrp_instance VI_INT {
interface eth0
state MASTER
virtual_router_id 151
priority 101
virtual_ipaddress { dev eth4 dev eth5 dev eth5

  notify_master /path/to/master_active
notify_backup /path/to/backup_active

The end configuration is a very fast failover that allows me to manage both systems internally but save on resources externally. It’s not flawless as I said – no state failover – but the end result is admirable.

Note: Some may wonder why I didn’t use the virtual_routes directive. Well honestly, because I found it to not work and it didn’t give me the flexibility to have an alternate backup route – something I needed for system updates.

16 Dec

Linux raid hard drive serial numbers and temperatures

Thought I’d share this little script. Basically what I needed to do was build a way to track what hard drives were connected to my linux raid arrays along with their serial numbers and temperatures. This script requires hdparm and hddtemp which are both readily available on most distros.


# set LANG for encoding celcius symbol, UTF-8 screws it up
for file in /dev/md[0-9]*; do
VAL=mdadm -D $file;
for line in $VAL; do
if [[ $line =~ \/dev\/(h|s)d[a-z][0-9] ]] ; then
DISK=echo $line | awk '{ print $7 }'
TEMP=hddtemp $DISK | awk -F': ' '{ print $3 }'
SERIAL=hdparm -I $DISK | grep "Serial Number" | awk '{ print $3 }'
echo $line | sed “s;$DISK;$SPECLINE;”
echo $line

for disk in echo $VAL | grep '^[ ]*[0-9]' | awk '{ print $7 }'; do
echo $disk;

— sample output —

Version : 1.2
Creation Time : Tue Nov 30 15:12:06 2010
Raid Level : raid5
Array Size : 5860552704 (5589.06 GiB 6001.21 GB)
Used Dev Size : 976758784 (931.51 GiB 1000.20 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Update Time : Fri Dec 17 01:58:35 2010
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
Name : skyn:bigstore  (local to host skyn)
UUID : 77df4227:2c4c30e8:93438ed9:49939563
Events : 60568
Number   Major   Minor   RaidDevice State
0       8       97        0      active sync   /dev/sdg1 WD-WMATV8578493 36°C
1       8      145        1      active sync   /dev/sdj1 WD-WMATV7948948 34°C
2       8      209        2      active sync   /dev/sdn1 WD-WMATV7901255 36°C
3       8       81        3      active sync   /dev/sdf1 WD-WMATV7875948 37°C
4       8      161        4      active sync   /dev/sdk1 WD-WMATV8049056 37°C
5       8       33        5      active sync   /dev/sdc1 WD-WMATV8516087 34°C
7       8        1        6      active sync   /dev/sda1 WD-WMATV7450752 34°C

24 Jun

Should I run Vista?

As a long running advocate of Linux, I know I’m stepping out of my usual terrain but I think it has to be said by a competent computer user: Yes, you can run Vista.

Don’t get me wrong, I still have both desktop and server installations of Linux and love it dearly. I also have a desktop that runs Vista. In fact I took it a step further and run 64-bit Vista and other than Quicken 2008 which gets shut down by DEP occasionally (Quicken refuses to look into it, by the way, thanks Intuit) the system is running great.

Compatibility? You’re probably unlikely to run into a complication. Unless your application is 6-10 years old it’s most likely going to work even if it wasn’t designed for Vista. Guaranteed? No, but the odds are in your favor. I ran into only one compatibility issue and that is Sony who refuses to support 64-bit with their Personal Voice Recorder (an ICD-P520.) I dropped $100 for it only a few months ago. Sony, you’re a bunch of jerks.

That said, it had some stability issues when it first came out but Microsoft resolved those after about 3 or 4 months. It’s been pretty good for me since then. YMMV but I’d say go for it.

08 Apr

Building a liquid cooled system

The latest addition to the ManGeek’s computing power is a new PC. The old laptop, while quite capable, was just not satisfactory for geekdom. Anyone who’s seen a ham radio operator today knows what happens when geeks get out of touch with the times. Thus I decided to push the limits on a reasonable budget and build a liquid cooled system.

Picture a radiator on your car and you have the same concept for a PC. The radiator mounts on the back and pumps coolant through tubes that are hooked up to critical components. (Processor, video card, north/south bridge…) It requires a bit of savvy and time. I needed a couple hours to do all the metal work and you have to think carefully about where each component will be installed and where plumbing will run. (Remember that the side cover has a lip on the back for the screws? Yeah. I had to get creative since I didn’t.) Still after all is said and done, the system is quite quiet and the CPU runs at about 62 degrees fahrenheit.

The final specs? Intel q6600, nvidia 8800GTS 640MB, 4GB of RAM and everything on SATA. Nearly everything in Vista scores a 5.9 on the satisfaction index and it’s peppy as can be. I’m running Vista 64bit as well which has actually been reasonably easy to work with. (Though I have a Logitech G15 keyboard hooked up to it and had to update the driver which was causing a blue screen.) Total cost including two 22″ 2ms LCD screens? About $1750. Similarly built through an OEM? About $5k. Not a bad deal if I do say so myself.

06 Dec

EVD – The Chinese Job

I just read an article on MSNBC.com that discusses with some FUD, a move by the Chinese state and manufacturing industries to attempt to corner the media market and claim control over the licensing of media technologies.

Now of course while I see the FUD (fear, uncertainty and doubt) in the article since it fails to identify how they will satisfy the requirements of the rest of the world, I certainly see it being something in line with the Chinese government’s practices in the past.

I fail to see why it is better to adopt a pattern of isolationism and “my way or the highway” tactics. Manufacturing remains one of their only strengths and if they won’t build it, I’m sure we can find someone else who will. The price may be higher but you have to play ball evenly or you won’t be invited to the game.

I see no value in EVD. Nor in handing over control of media technology to the Chinese government.

Thank you for reading this post which would be likely to get me jailed in China.

02 Oct

The Internet, Privacy and your Kids!

If there’s ever been a touchy subject to deal with parents, it’s the raising of their children. Unfortunately in some cases it can be even worse if the parents know their own deficiencies but choose instead of facing them to let poor behaviors continue. (It’s not hard to be an Ostrich with your head in the sand.)

…on that note, has anyone ever seen an Ostrich with its head in the sand? But I digress.

I’m going to do my best to avoid dwelling on my own thoughts about parenting. It would be particularly hypocritical since I am not actually a parent but I try my best to not ignore the things I see in life. With that in mind:

  • How well your kids trust you will reflect in if they resemble the Beav or your friendly local CIA agent.
  • Do not underestimate their intelligence simply because they can not properly form a sentence. There is a distinct difference between intelligence and knowledge.
  • Do not fool yourself into thinking you can stop them from doing something absolutely stupid. In fact they intend to. They just don’t realize it yet.

So rather than locking them in their room and taking away their keyboard, I’d push people towards these do’s and don’ts:


  • Know what your kid is doing and viewing!
    Now understand, teenage boys… well, we are what we are. I do however know of one recently who was big into anime. His parents had no idea what it was and hadn’t looked into the subject. For those who aren’t in the know, much anime is paramount to porn and a lot of it gets pretty violent. Your child is going to seek out porn but you’d better talk to them before their friends at school do.
  • Recognize when their computer use dwells on anti-social behavior.
    Some kids are not happy. Some of them are going to frequent very dark corners of the Internet and understand when I tell you that their are very dark corners. (I don’t intend however to fall back on my first point.) I want to illustrate that sometimes this can have a very detrimental affect on their lifestyle and behavior. Those who don’t ignore the warning signs can divert problems at school or potentially tragedy.
  • Do promote their interest in computers.
    Draconian controls are something to fight against. It would seem that desire is in our genes. With this in mind, you should recognize the ways they use their computers and the information available on the Internet when it’s valuable to their future.


  • Do not leave them with an Internet connection and a license to kill.
    If you don’t know what’s going on over the Internet connection you pay for every month then you are delinquent in your responsibilities. You need to monitor this connection. (I’ll address this more below.)
  • Do not avoid opportunities to ask others who have been around.
    If you see something or hear something you don’t understand, research it. Talk to friends or perhaps teachers. Do searches on Wikipedia and try to keep as up-to-date as you can. In this digital age this is not easy but raising kids never is.
  • Don’t show your cards!
    If you learn something and you tell them how you learned it, they will figure out how to avoid letting you learn anything in the future. Once again, this goes back to my second point under the list of Do’s.
  • Try to avoid being draconian.
    Learning and innovation comes when people are allowed to think free and express themselves. With this in mind, if you are dealing with an issue don’t lock the keyboard in the closet. They’ll just borrow one from a friend and hide it under their mattress. The more responsibility you give them, the less they’ll try to hide the things they do.

With all this in mind (and once again I do apologize for the potential hypocrisy of my very subjective comments) I want to give some actual advice on technical things you can do to take the upper hand.

Avoid anything that runs on the computer they use.

This includes key loggers, web control products, internet filters, IM loggers or any other type of local control. It’s a waste of time. They are going to find a way around it. The ways are easy. They know them. For that reason alone, here’s how you get around them (and how they WILL get around them):

  • Remove hardware key loggers. If they lock onto the case, get a USB keyboard and hook it up to an uncontrolled port.
  • Download a liveCD operating system from the Internet and boot off that CD. This will eliminate all operating system controls on the computer itself and leave no trace of what was done while under the liveCD OS.
  • Software-based control programs can be disabled in most cases provided you have local administrative access. And even if you don’t, I know most home PCs aren’t patched in time. Wait until a vulnerability is posted, crack the local machine and bust the control software. Install a backup admin user and use it only when necessary.

So parents, are you ready for the game of cat and mouse? If you’re going to trust anything on the computer they use then that’s the game you will be playing! You’re probably not up to the task so don’t subject yourself to it!

Do not use the same computer as them.

I understand it can be cost prohibitive but do what you can to have your own computer (preferably a laptop) and do not let them use it! Why do I recommend a laptop? It’s harder to put a keylogger on a laptop without the person knowing it. (Of course by this I mean you use the keyboard on the laptop and not an external keyboard which would eliminate this strength.)

Also make sure you use strong passwords and you keep them in a safe place! (Preferably your head!) They will likely guess or figure out your password if it’s easy. Also DO NOT GIVE THEM YOUR PASSWORDS! Under no circumstances does anyone else need your password. I don’t tell anyone my passwords… parents, girlfriends… anyone. Don’t make exceptions and your life will be a lot easier.

Talk to your ISP about restricting access to your network connection.

This is mostly applicable to those who have high-speed connections (DSL or cable for example.) Understand that any controls you put into place can be easily by-passed when they (will) plug your line into their own computer or another piece of hardware borrowed from a friend. At that point they own the connection. Game over man, game over.

Hopefully your ISP will be helpful and set it up so that your modem or router cannot be disconnected and still allow the line to work.

Setup network monitoring and gateway control

This often means installing a piece of hardware which will cost money. Still you’re talking about a one time charge to help you keep aware of what’s going on with your Internet connection. I recommend not ever telling your kids what you’re doing. Unless they run afoul of filters the rest should be transparent. Once again this goes back on not showing your cards but using a gateway or network monitoring tool, you can see into the details of what they are doing. This is not a free chance to tromp on their privacy and break their trust but it is good information. Like I keep saying, our government doesn’t do unlawful wiretaps to prosecute people. They do it so that no one gets hurt. They’re not going to slap a subpoena in your face and you should act th

29 Sep

What will a man do for a terabyte?

Some day a brilliant geneticist will discover a gene present in all masculine people that makes us want to do things bigger than before. That very gene drives us to the verge of insanity for goals that may not ever actually make our lives better. I think it also has something to do with why most redneck deaths begin with the words “Hey man, check this out!”

Still, in my quest to satisfy this genetic craving (oh yes, it’s all the genes fault believe me!) I’ve undertaken the challenge… picked up the gauntlet… to build a fast affordable 1.5 TB (terabyte) RAID array. Since I work with graphics and am a file whore, I will actually put this to use although I expect it to carry me for about 6-8 years. Christening it was a particular challenge. Many names were recommended by friends because afterall, such a machine deserves a title worthy of a king. Well after much thought I came to my conclusion. Since this system represented almost limitless capabilities for storing knowledge and history, I had to respectfully name it after one of my favorite literary characters: Albus Dumbledore.

So back to the technology: In this quest I’ve learned a few things. One is that my trusty an faithful linux can actually let me down! (I’ll explain.) Another is that it takes a long time to initialize a 1.5TB RAID. Are you as shocked as I was? Indeed:

Personalities : [raid5] md0 : active raid5 sdg1[7] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
1465175424 blocks level 5, 64k chunk, algorithm 2 [7/6] [UUUUUU_]
[=>...................] recovery = 5.1% (12510368/244195904) finish=353.9min speed=10906K/sec
unused devices:

So while we patiently wait; the hardware for those *-philes:

  • 550W Power Supply
  • New full tower chassis (with some case mods to support the hardware better and a number of spliced power connectors since a standard power supply only comes with so many)
  • 7 new Western Digital 250GB SATAII drives
  • 6 SATA power converters (since the PS only had one)
  • 1 very old Maxtor 14.4GB hard drive for a boot devices
  • 2 Promise SATA300-TX4 PCI adapters (a bear under linux at first, but we’ll tackle this) which lucky for me come with 4 SATA cables each
  • A trusty (for the previous owner) Tyan Tiger-133 Dual-PIII 800 Motherboard, offered to the cause by said previous owner and long-time buddy for an amazingly low price of $50 (with 768MB of RAM!)

Now first thing I have to say is that I am quite annoyed with the Linux difficulties I ran into but I understand how they happened. The short is that somewhere along the line (after 2.6.11) support was dropped for this Tyan motherboard. (Inadvertently by a bright penny of a devevloper I’m sure.) I’ll discuss the issus below. So as it is, I have to stick with 2.6.11 for now. This limits my distribution options because while I could do Gentoo and downgrade the kernel, I just don’t want to take the time to jump through the hoops. So the result? Fedora Core 4. Hey, after these many years (my first distro was Slackware on 1.0.27) a distro is a distro is a distro.

So what’s the problem with the Tyan you ask? Well basically it doesn’t detect the PCI bus. I am not kidding, lspci outputs nothing. No PCI devices are detected. This will occur with any kernel between 2.6.15 and 2.6.18.

[jbly@albus jbly]$ lspci
[jbly@albus jbly]$

Now, if you were to run 2.6.12 you would actually get PCI bus enumeration. However your network card won’t work. Believe me, strange as it may soundd I tried two different network cards, multiple cables, switch ports and ultimately concluded it was something in the voodoo of the kernel/driver. Still, 2.6.11 works beautifully.

So, now that we have that issue tackled here comes what I expected to be the real challenge: the promise controllers. Linux support is said to be minimal but I think that is novice users talking. Compiling a kernel is not that hard for me (as I said, I just hate how long it takes) so I downloaded the source tar ball from Promise. It came through with shining results. That is, with the exception of a momentarily cryptic error message:

ulsata2: Unknown symbol scsi_remove_host
ulsata2: Unknown symbol scsi_unregister
ulsata2: Unknown symbol scsi_register
ulsata2: Unknown symbol scsi_scan_host
ulsata2: Unknown symbol scsi_add_host

Ahh poor foolish me who compiled SCSI support as a module. “modprobe scsi_mod” and I was back on track. dmesg filled with a line of disks and I was off to create my 7 type fd partitions. In truth I gloss over that but it took some serious brain power to think through the situation and this is ultimately the whole reason I post this! I pray perhaps I can save one other the peril that is configuring this driver. Oh and btw, if you compile it on kernels later than 2.6.13, you should go into pdc-ulstata2.c and remove the line about scsi_set_device() because it’s no longer needed apparently.

And the final number? 1,442,184,636 1K blocks.

Would anyone argue that the most lovely thing about this is that it has actually gone over df’s ability to do pretty formatting? I believe that’s a success! So raise your glasses.. here’s to the next 6-8 years!

(Well, at least when it finishes bringing up the parity disk… 11.2% now)