Cheerful Curmudgeon

I have learned to change lead into gold, and back again, for real! Well, sort of. Thus far, it only works within the virtual world of programming languages like Python. Here is the recipe. (If you are not into geek-speak, skip to the bottom where I natter on about reading ebooks on an iPad.)

class Lead(object):
    def prestoChango(self):
        self.__class__ = Gold

class Gold(object):
    def prestoChango(self):
        self.__class__ = Lead

pb = Lead()
print pb             # prints Lead
pb.prestoChango()
print pb             # prints Gold
pb.prestoChango()
print pb             # prints Lead

Voila! First it’s lead. Then it’s gold. Then it’s lead again.

I understand why you might want to do something like this but, at least within the projects that I work on, it would obfuscate the program too much for my liking.

I learned this from reading the Python Coookbook ebook on my iPad, which has been thoroughly enjoyable. I like the iBooks app more than I expected to. I can highlight portions of the book, without actually trashing the pages. I can scribble notes next to my highlights. I can easily browse a list of the sections that I highlighted/noted. Perhaps even more useful, I can select a word or phrase from the text of the book and instantly search either Google or Wikipedia for it.

The Python programming language has become my first choice for most tasks over the last year or so. The more I use it, the more I find to like about it. I just stumbled across generators in a way that made them make sense to me and it is so cool that I want to share it with you. A generator can make a program immensely more readable by separating the task of producing (or generating) data from the task of processing the data.

This will make more sense with an example: print an alphabetized list of all the usernames for a Linux system. On a computer running Linux, the file /etc/passwd contains information about all of the users. Here is the file for my laptop:

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13:proxy:/bin:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bin/sh
list:x:38:38:Mailing List Manager:/var/list:/bin/sh
irc:x:39:39:ircd:/var/run/ircd:/bin/sh
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
libuuid:x:100:101::/var/lib/libuuid:/bin/sh
syslog:x:101:102::/home/syslog:/bin/false
klog:x:102:103::/home/klog:/bin/false
hplip:x:103:7:HPLIP system user,,,:/var/run/hplip:/bin/false
avahi-autoipd:x:104:110:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/bin/false
gdm:x:105:111:Gnome Display Manager:/var/lib/gdm:/bin/false
saned:x:106:113::/home/saned:/bin/false
pulse:x:107:114:PulseAudio daemon,,,:/var/run/pulse:/bin/false
messagebus:x:108:117::/var/run/dbus:/bin/false
polkituser:x:109:118:PolicyKit,,,:/var/run/PolicyKit:/bin/false
avahi:x:110:119:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/bin/false
haldaemon:x:111:120:Hardware abstraction layer,,,:/var/run/hald:/bin/false
art:x:1000:1000:Art Zemon,,,:/home/art:/bin/bash
postfix:x:112:124::/var/spool/postfix:/bin/false
candy:x:1001:1002:Candy Zemon,,,:/home/candy:/bin/bash
sshd:x:113:65534::/var/run/sshd:/usr/sbin/nologin
mediatomb:x:114:126:MediaTomb Server,,,:/var/lib/mediatomb:/usr/sbin/nologin
couchdb:x:115:116:CouchDB Administrator,,,:/var/lib/couchdb:/bin/bash
speech-dispatcher:x:116:29:Speech Dispatcher,,,:/var/run/speech-dispatcher:/bin/sh
kernoops:x:117:65534:Kernel Oops Tracking Daemon,,,:/:/bin/false
usbmux:x:118:46:usbmux daemon,,,:/home/usbmux:/bin/false
festival:x:119:29::/home/festival:/bin/false
rtkit:x:120:128:RealtimeKit,,,:/proc:/bin/false

Since the username is the first “word” on each line, up to the first colon, most of that file is drek and can be ignored. So given that file of stuff, the program breaks down into these tasks:

1. Open the file /etc/passwd.
2. Read every line from the file and get the username, the first word, off of each line.
3. Construct a list of all the usernames.
4. Sort the list.
5. Print the results.

My first attempt at such a program would have been something like this:

namelist = []
passwd = open('/etc/passwd')
for line in passwd:
    username, drek = line.split(':', 1)
    namelist.append(username)
passwd.close()
namelist.sort()
for name in namelist:
    print name

This little Python program does what I just described, producing this output:

art
avahi
avahi-autoipd
backup
bin
candy
couchdb
daemon
festival
games
gdm
gnats
haldaemon
hplip
irc
kernoops
klog
libuuid
list
lp
mail
man
mediatomb
messagebus
news
nobody
polkituser
postfix
proxy
pulse
root
rtkit
saned
speech-dispatcher
sshd
sync
sys
syslog
usbmux
uucp
www-data

The ugliness is that the for-loop does two things which are unrelated to each other: It finds the usernames within the /etc/passwd file and it constructs a list of the usernames. Why does a piece of a program which finds usernames care what happens to the usernames after they have been found? Why does a piece of a program which constructs a list of usernames need to care where the names came from? This is an artificially contrived example, so each of these pieces is very simple, but it is generally A Good Thing if each piece of a program does exactly one task. This makes everything easier: design, coding, testing, and debugging.

By using a generator, we can pry these two tasks apart and the program becomes easier to understand:

def usernames():
    passwd = open('/etc/passwd')
    for line in passwd:
        username, drek = line.split(':', 1)
        yield username
    passwd.close()

namelist = []
for name in usernames():
    namelist.append(name)
namelist.sort()
for name in namelist:
    print name

The generator at the top does just one thing: it produces usernames, one at a time. Python takes care of all the complexities. We can simply use the generator wherever we need a list of usernames. On first use, the /etc/passwd file is opened. Then each line is read, the username split off the beginning of the line, and the username yielded up to whatever other part of the program needs it. When the file has been completely processed, it is closed.

The second part of the program has become an easy-to-read loop: for name in usernames() This loop processes each name. We can understand that without being distracted by the details of processing the /etc/passwd file. Sweet.

[Update: I particularly enjoy programming because there is always something new to be learned. I have updated the following example, shortening it by one line while simultaneously making it easier to understand.]

Of course, Python offers more shortcuts and we can make the program more concise. Try this flavor:

def usernames():
    passwd = open('/etc/passwd')
    for line in passwd:
        username, drek = line.split(':', 1)
        yield username
    passwd.close()

print '\n'.join(sorted(usernames()))

Reading from the inside to the outside: usernames() produces the list of usernames. sorted(...) produces an alphabetized list of usernames. '\n'.join(...) takes the alphabetized list of names and joins them together into a string, one name per line, which is ready to be printed.

I hope that this has not been too deep a peek into the machinations of a programmer’s mind.

I just upgraded the instrument approach plates that I use when flying IFR in my airplane. My new iPad with ForeFlight Mobile HD replaces my Sony PRS-505 with ReaderPlates that I have been using since December 2008. The Sony and ReaderPlates replaced paper that I had been using since 1987. This is a very good thing and it demonstrates just how rapidly technology is improving.

When I am in instrument meteorological conditions (IMC), which is a government approved way of saying “in the clouds,” and need to land my plane, I use a detailed map called an approach plate. The approach plate tells me exactly where to fly the plane both horizontally and vertically so that I get safely to the runway. It’s a lot like playing a complex video game, which I find both exhilarating and highly satisfying.

Paper worked well for years because it was pretty much the only game in town. I was very happy to switch to the Sony PRS-505 ebook reader, though, because it meant that I could stop recycling a 3″ stack of paper every 28 days. Yes, Virginia, you read that right; the government updates the instrument approach plates every 28 days, 13 times per year. Switching to an electronic format saved me quite a few dollars and was kinder to the environment.

The biggest disadvantage of the Sony PRS-505 is its small screen. As you can see in this photo, it is significantly smaller than the paper. I can get an overall view of the approach but have to press a key to zoom in and make the text large enough to read. The iPad fixes  this problem with a sufficiently large screen to display the approach plate 100% the size of the original.

ILS Approach Plate on Sony PRS-505, paper, and iPad (click to enlarge)

The transition from paper to Sony PRS-505 to Apple iPad in just 18 months is remarkable. The new screen is plenty bright enough to read in direct sunlight and, surprisingly, even higher contrast than the newsprint. There are no moving parts to fail.

I had one concern about switch to the iPad from the PRS-505: would the touch sensitive screen allow me to accidentally “lose” my approach plate at a critical moment of the flight? I don’t think so. When displaying an approach plate, the only part of the screen which can make the plate vanish or change is the “close” button at the top, left corner.

I will have more on ForeFlight and more on the iPad soon. Before posting more about ForeFlight, I want to actually fly with it. Before posting more about the iPad, I have to stop playing with it and write about it.

Facebook added the proverbial last straw with its latest privacy faux pas. It has demonstrated, yet again, that in pursuing it’s goal of selling advertising, Facebook places very little importance on our personal privacy. Remember that, while Facebook ostensibly is a web site designed to help people connect with like-minded people, in fact Facebook is a business which derives it’s revenues from other businesses, not from it’s subscribers. In plain English: Unless you are paying big dollars to Facebook, you are not Facebook’s primary audience.

Don’t believe me? I just spent half an hour tightening up my Facebook privacy settings; it was a bewildering maze of pages and checkboxes and pop-up windows. I thought maybe I was just dim, that it couldn’t be as hard as it seemed to be. But no; it really is that hard. The New York Times counted the words and discovered that Facebook’s privacy policy is longer than the US constitution!

The new opt-out settings certainly are complex. Facebook users who hope to make their personal information private should be prepared to spend a lot of time pressing a lot of buttons. To opt out of full disclosure of most information, it is necessary to click through more than 50 privacy buttons, which then require choosing among a total of more than 170 options.

Users must decide if they want only friends, friends of friends, everyone on Facebook, or a customized list of people to see things like their birthdays or their most recent photos. To keep information as private as possible, users must select “only friends” or “only me” from the pull-down options for all the choices in the privacy settings, and must uncheck boxes that say information will be shared across the Web.

The last straw was discovering a page which allowed my personal information to be shared with third-parties (advertisers and other businesses) when my friends do stuff, not because of my own actions. Here is the page, after I turned everything off; all of the boxes had been checked when I first came to the page.

Just one example: I am perfectly happy allowing my friends to know my birthday but I was angry to discover that, when a friend of mine “visits a Facebook Platform application or website,” my birthday was revealed to the business running that “application or website.” That’s just not right; I did not give my permission for this. I do not want it to happen. Facebook added this “feature” and began giving out this information without asking me.

In response to that discovery, I have done a couple of things. First, I took the time to go through every Facebook privacy page and tighten up the settings. My friends can still see stuff about me. The friends of my friends can also see some stuff about me. To the extent possible, I have blocked business’ abilities to obtain my data. Second, I have removed all of the data which I do not want publicly shared. Since I cannot trust Facebook to keep it private, I no longer store those data in my Facebook profile.

If you are reading this on Facebook, you should know that Facebook is posting a copy of my original article. I actually wrote this on my own blog at www.CheerfulCurmudgeon.com and I invite you to visit the site directly. Facebook does not copy everything from the blog and you are missing good stuff by staying in Facebook and not coming over to the actual website.

I choose to control access to my data, sharing it only with the people that I trust. Facebook has proven, time and again, to be a very untrustworthy arbiter of our data.

Sometimes (often) XKCD hits the nail squarely on the head.

May this passover find you free from all bondage and from anything which keeps you from thoroughly enjoying life. L’chaim (to life)!

I got a frantic email from a friend this week. One of his subcontractors went crazy and trashed several of my friend’s clients’ web sites as well as my friend’s own business site. The police have been involved but much damage has already been done. I wish I had been hosting my friend’s sites. Had I been, I could have recovered everything from backups. As it is, all I could do was sit by and fume, wishing that his hosting company had had something to offer him in the way of assistance.

I know my friend’s pain. In the 30 years that I have been doing system administration, there have been numerous times when my own bacon has been saved by backups. I have been struck by the dread BUOD error (Bad User On Device) in which a glitch sitting between the chair and the keyboard has made the computer do all kinds of hideous deeds. The worst, early in my career, idled a team of a dozen programmers for three days. Why three full days? You guessed it: no backups. At the other end of the spectrum, a member of my team recently trashed a critical configuration file on one of our servers. This, however, resulted in no downtime; we simply grabbed a copy from the backups and continued on our merry ways.

If you accidentally delete a file from your web site (or, in my friend’s case, all of the files), can you recover it? Does your hosting company provide backups and, if so, can  you recover files from their backup? In many cases, hosting companies’ backups are only for their use in cases of disk drive failure.

My company offers one (excellent, in my opinion) solution, Nest Egg Backup for Web Servers. There are many other alternatives. Do choose and implement one. When you go comparison shopping, ask the key question: How long are the backups retained? If only for one night, that means that your window of opportunity is extremely limited. If you delete a file at 10:00pm and wait until 8:00am to try to get it back, you are out of luck. You should have at least three days of retention, preferably more, preferably a lot more. Thirty days can give you a nice warm, fuzzy feeling of safety and security.

Lesson of the day: Back up your hosting accounts! And be sure to include everything (email folders, MySQL databases, PHP config files, etc. etc. etc.) The day disaster strikes is a day too late to start backing stuff up.

Back when I administered VAXen running BSD UNIX at FileNet, “just” 25 years ago or so, we didn’t have a high speed internet connection to use in transferring files between computers. I used a bank of Racal-Vadic 2400 baud modems to run UUCP and shuffle email and usenet articles around. With five modems in the bank, felix the VAX 750 grew to be a modest UUCP hub in SoCal. Today we measure our internet connections in megabits or Mbps, millions of bits per second, instead of baud. I am typing this on a cable modem connection that just achieved 11.4 Mbps downloading data and 1.7 Mbps uploading data. By comparison, felix the VAX had about 0.0024 Mbps of bandwidth, and downloading did not go any faster than uploading.

10.5" magnetic tape

We used to say, never underestimate the bandwidth of a station wagon full of magnetic tapes. With just 0.0024 Mbps of bandwidth available, it was completely impractical to transfer large files across “the net.” Instead, we would write the files to 10.5″ reels of magnetic tapes and drive them to their destination by car.

DEC RM03 disk drive

It would take several tapes, and several hours, to “back up” a single 67 MB, washing machine sized, RM03 disk drive. Once the  tapes were written, we would bundle them into  the nearest car and take a road trip across town to where they needed to be. Then would begin the (usually slower) process of reading the tapes into the new computer. It was way faster to move data by “station wagon” than pretty much anything else.

Flash forward to 2010. On Tuesday, I had 62,000 MB of files on a computer in a datacenter in Houston that had to be moved to a new computer in a datacenter on the east coast. I live in the middle of the country (St. Louis) and do not have ready access to either datacenter. Through the miracles of the internet, I logged into the Houston computer and typed one command:

rsync -az /backup/htn/ root@newmachine.com:/backup/htn

About eight hours later, with no intervention from me, all of the files had been replicated onto the new machine. I had just moved 1,000 times as much data as one of felix’s entire disk drives in a fraction of the time and with virtually no effort.

Yup… life is good.

I am so happy. Netflix will finally stream movies to the Sony Playstation PS3. We have a Roku player, too, but it will be more convenient to have all of our entertainment stuff in one box. And the PS3 ought to produce a better picture, too.

My deepest appreciation to ThinkGeek for making me feel old first thing in the morning. They have recreated the “classic student slide rule” and are hawking them for a pittance of what this paragon of useful technology is worth.

ThinkGeek Slide Rule

The sad thing is… not only do I remember slide rules, I depended on them to get my physics homework done in high school. Not only do I remember slide rules, those of us who used them were faster than the folks with Bowmar Brain calculators.

I did succumb to the glitter and glitz of calculators and computers, becoming an RPN bigot as any newly minted geek was in the 1970s. (Good thing, too. Have you ever tried to blog on a slide rule?)

Slide rules re-entered my life in 1986 when I took flying lessons. The E6B flight computer is little more than a circular slide rule with  special scales for doing some temperature calculations and a special rotating, clear window on the back for solving wind triangle problems. Spock recognized the utility of the E6B, too, using it on the bridge of the Enterprise in Who Mourns for Adonis.

Spock using an E6B Flight Computer

I ended up with two slide rules, my own and my grandfather’s (and my E6B flight computer). My slide rule evaporated years ago but my grandfather’s is still in the basement. And you know what, it still works… no batteries required.

My (Grandfather's) Slide Rule