Linux Software RAID and GRUB – Recovering From a Failure

August 18th, 2006

A couple of weeks ago, I had the bright idea to move an internal server here at Voyager from my office into a data room. I issued the customary sudo shutdown now and proceeded to move the box.

I was dismayed not to see it boot right back up afterwards. Ouch! I had specifically configured it with software RAID because the hard drives in the old spare box were a bit dodgy. Turns out that was a good idea, since one of the drives had failed (apparently the one that had the GRUB bootloader appropriately loaded on it).

I was faced with two 20 Gig HDDs, only one of which worked exactly right, and a computer which failed to boot off the remaining HDD. A quick trip to uBid and $70 later, I had two 60 Gig drives ready to use (20 Gigs are darn near impossible to find). I knew enough about partitions and whatnot to get this much done:

– Got a bootable rescue CD with a good set of utils (PLD Linux) downloaded and burned (it’s good to have one of these handy, rather than trying to burn them as-needed — see below under “Tricky stuff” for my unfortunate experiences with that).

– Trial-and-errored the two old HDDs to find which one was failing. Removed the bad one and replaced with New HDD #1.

– Used cfdisk, the curses-based fdisk, to read the exact size and type of the good RAID partition from Old HDD. Used that information to create an identical physical partition at the beginning of New HDD #1, including the same (Linux Software RAID) partition type.

– Used dd, the bit-for-bit copier, to copy the verbatim entire partition from the Old HDD’s main partition, /dev/hda1, to the New HDD #1’s identically situated partition, /dev/hdc1, both of which were unmounted at the time.

– Swapped out the Old HDD with New #2, and repeated the last couple steps to make a new partition on New #2 and copy New #1’s first partition to it.

– Used mdadm --assemble to put the two identical RAID partitions — /dev/hda1 and /dev/hdc1 — back together into a RAID array and let it re-sync them until mdadm reported them to be in good health.

– Used GRUB to re-install the MBR on both HDDs. This was a damn sight harder than it sounds (see below).

All in all, it was a far cry from sliding a replacement hot-swap SCSI into a nice hardware-based array — but at $70, a fraction of the cost, though hardly timely (my use of this server is as a document archive, web proxy, cron-job runner, and general workhorse for background processing and speculative projects for automated information-gathering tasks — none of which are mission-critical for us at Voyager).

Tricky stuff:

– Windows XP apparently doesn’t come with ANY ability to burn ISOs. WTF, Microsoft? Operating system ISOs are just about the only legal thing I have ever wanted to burn to a CD, and that’s the one thing you won’t support? (Well, duh, really.)

– The latest Knoppix (5.0?) just plain barfed. It may have been the speed at which the dodgy ISO burning software I downloaded burned it (errors?). In any case, burned about an hour of my life trying different “nousb” and similar switches to no avail.

– PLD Linux‘s rescue disk was small and booted perfectly (though I took care to burn it at a low speed).

– BLACK MAGICK: When booting from the rescue disk, to get mdadm to appropriately deal with the raid, there weren’t any md devices in /dev on which I could mount the RAID. I needed a couple of times to create the node /dev/md0 by issuing the commands: 

mknod /dev/md0 b 9 0

Which, as I understand it, is “make node /dev/md0, type block, numerical type #9 (the magic number for RAID?), and the 0th such block.” Then, since mdadm refused to automatically find and mount the drives for /dev/md0, I had to find the UUID for the RAID volume thus: 

mdadm --examine --scan

And then copy the UUID (thanks, GNU Screen!) into the command: 

mdadm /dev/md0 --assemble --uuid=<WHATEVER>

– Getting GRUB installed on the hard drives was, in the end, easier than I thought but was rocky due to the complexity of the issues involved and me not understanding them fully.

If you search for “software raid grub” you’ll find a number of web pages that more or less get you there with what you need to know.

For me to get GRUB to work, I did the following.

– First, I had the /dev/md0 partition (the “RAID partition”) holding my / (root) partition, with NO separate /boot partition. That means I had to make each of /dev/hda1 and /dev/hdc1 (the “RAID constituents”) bootable. Much of what you read follows the old advice of having a separate / and /boot, which I did not have.

– Second, I had to boot from the rescue CD, get the RAID partition assembled, mount it, and chroot into the mount point of the RAID partition. Like: 

mknod /dev/md0 b 9 0 mdadm /dev/md0 --assemble --uuid=<WHATEVER> mkdir /tmp/md0 mount /dev/md0 /tmp/md0 cd /tmp/md0 chroot .

Note that since the partition names and numbers were the same on my New #1 and #2 drives as they were on the old ones (hd[ac]1), there’s no problem with the old legacy /etc/mdadm/mdadm.conf and it can tell the kernel how to assemble and mount the RAID partition (important for below).

– Then, once chroot‘ed into the RAID partition, I ran run-time GRUB (“run-time GRUB” being when you run grub as root on an already booted machine for purposes of installing stuff or whatnot; this is opposed to “boot-time GRUB” which looks pretty damn similar but is what is run off of the master boot record — MBR — of a bootable partition onto which GRUB has been installed) off of there, which used the existing Ubuntu menu.lst file. For some reason, that file ended up binary and corrupted. Therefore, I had to scrap it and come up with a new one. Here’s the meat of my new menu.lst: 

# Boot automatically after 15 secs. timeout 15  # By default, boot the first entry. default 0  # Fallback to the second entry. fallback 1  # For booting Linux title  Linux root (hd0,0) kernel /vmlinuz root=/dev/md0 initrd /initrd.img  # For booting Linux title  Linux root (hd1,0) kernel /vmlinuz root=/dev/md0 initrd /initrd.img

– Using that menu.lst, the commands I entered in the GRUB shell were as follows: 

device (hd0) /dev/hda root (hd0,0) setup (hd0) device (hd0) /dev/hdc root (hd0,0) setup (hd0)

The rationale behind this is that the first three lines install the current menu.lst (that is, whichever one it finds in /boot/grub/menu.lst) onto the MBR of /dev/hda, the first bootable HDD, and the second three lines install onto the MBR of /dev/hdc, the second HDD, but fake out the installation there of GRUB to act as though it’s on the first, bootable hdd (hd0).

Do you get it? After chrooting, I fired up run-time GRUB, which automatically looks in its current boot/grub for menu.lst. I told it to put MBRs on both /dev/hda and /dev/hdc to make boot-time GRUB behave as specified in the menu.lst. The menu.lst lines say “use hd0,0 (e.g. hda1) as the root directory, find the kernel (vmlinuz) and initrd there, and once the kernel get loaded, tell it to use /dev/md0 as the real root directory, which it can do because it reads /etc/mdadm/mdadm.conf or /etc/mdadm.conf to figure out its md devices.

What puzzled me at first was, "how does the kernel get loaded when the root is /dev/md0 and you obviously must run mdadm in order to assemble and mount /dev/md0?" The answer is that when you do the installation commands listed above, it tells the boot-time GRUB to act as though (hd0,0) (AKA /dev/hda1 or /dev/hdc1, depending on whether the BIOS points to hda or hdc for booting) is its root directory. So, boot-time GRUB, all the way up through the loading of the kernel, treats /dev/hda1 (or hdc1) as its root, and only at the stage where the kernel is loaded enough to check mdadm.conf and run mdadm does it then do a little "chroot" of its own. If I've got this completely wrong, please [rlucas@tercent.com email me] and tell me I'm a bonehead (and include a link to your, more informed, writeup).

There's an elegance to the whole Linux software RAID thing, but it took a darn long time to comprehend.

Posted in bugfix | No Comments »

Earnings Calls in a Trippy Vortex

August 10th, 2006

Today, I called up a news release on finance.yahoo.com for a fairly dodgy publicly traded company. Although I expected to find a transcript of the earnings call, or perhaps a brief table summarizing the expected and actual earnings numbers, I was given a link to the full audio of the call.

When I clicked the link, a full-screen window of Windows Media Player popped open and started playing the earnings call, with a huge, trippy, psychedelic vortex being rendered in the middle of the screen!

(As it happens, a mind-warping black hole of profits is the most appropriate metaphor for what this earnings call was all about.)

Thanks for a bit of inspired weirdness, Microsoft. It was certainly more enjoyable than discovering that your flagship email client can’t “archive” two folders at once.

Posted in vc | No Comments »

PayPal Continues to Suck

August 2nd, 2006

[PayPal http://www.paypal.com/] has always sucked. This is well accepted on the Internet. Witness: http://www.google.com/search?q=paypal+sucks … Results 1 – 100 of about 1,680,000 for paypal sucks. Here are the specific deficiencies I have most often heard cited: * Freezing of funds for arbitrarily long times with no explanation. * Bizzarely intricate information requests. * User unfriendliness to payors. In my case, I went to eBay to purchase some computer hardware. When I won the auction, I went to pay by credit card in the way most readily presented — which was PayPal. Due to some transactions I did the better part of a decade ago (they still had my address from 1999-2000), I was unable to log in to PayPal (an eBay Company). (Incidentally, eBay usernames and passwords have different validation requirements than those for PayPal, e.g. 8 char passwords.) Then, once I got PayPal to send me a “forgot password” link, it told me to /telephone them in Nebraska!/ Bear in mind that this is in the course of attempting to /give my money to a guy on eBay/, and now “an eBay Company” is my biggest obstacle to so doing. The gentleman in Nebraska was kind, but ultimately frustrating — telling me that a “code 31” means that they had to close the old account, but the presence of a stray two dollars in the account meant it would take 72 hours before I could sign up with a new account. After which point, of course, the seller on eBay would have written me off as a deadbeat. The rep suggests that I open a new email account. Just for PayPal. Just for this transaction. *Who is on the buy side here?!* I am a pretty stingy guy — but when I do actually lay down the greenbacks, I just want a modicum of respect. To PayPal and eBay, I say: congratulations. I now will choose *any* method over PayPal whenever possible.

Posted in rants | No Comments »

Everyone Is Here in the Future

August 2nd, 2006

I just came across what I am willing to call some of the most interesting and (personally) relevant cultural commentary I’ve seen in a while. No talk of Hezbollah, nothing on Bush or Britney — rather, art intended to make people living a software-mediated lifestyle stop and think a moment: (Warnings: Flash required; starts playing immediately with audio; possibly epilepsy-triggering.) http://redhanded.hobix.com/cult/everyoneIsHereInTheFuture.html http://redhanded.hobix.com/cult/everyoneIsHereInTheFuture2.html For those of us in the tech world — especially the parts of the tech world that occasionally sees comrades being struck down by angel funding coming out of the sky — this is worth seeing and thinking about. But not too hard. After all, you’ve got to get a business model and write some new dual-core turbo buttons for your mail merge.

Posted in culture | No Comments »

Broken Quoting of Spaces in Table Names in Ruby’s ActiveRecord

July 21st, 2006

Two-part posting.

1. The hot-shit developer boys at http://dev.rubyonrails.org apparently use Python (Trac) for their bug-tracking system, and for extra chuckles, it’s broken.

From http://dev.rubyonrails.org/newticket#preview

(removed a Python stack trace that came down to a NOT NULL constraint violation in the underlying database — the error message was messing up the blog formatting software.)

Not the most helpful when I’m trying to post a bug report!

2. So, I’m posting by bug report below. Essentially, having spaces in your table names throws a major monkey wrench in the “convention over configuration” mantra of Ruby on Rails.

I might fix this, if I can find the time (the lack thereof being why, in the first place, Rails seemed so appealing). If so, I will post a patch.

—

Legacy apps being built to SQL Server databases may find spaces in table names. These can be addressed superficially by a: 

set_table_name '"Spacey Table"'

or 

set_table_name '[Spacey Table]'

This approach makes parent classes behave properly when directly interpolating table_name into a string, avoiding such errors as: 

Invalid object name 'Company'.: SELECT count(*) AS count_all FROM Spacey Table

HOWEVER, pre-escaping the table names in this way breaks the SQLServer ConnectionAdapter’s ability to get info out of SQL Server, as in sqlserver_adapter.rb line 246 (line breaks added): 

sql = "SELECT COLUMN_NAME as ColName, COLUMN_DEFAULT as DefaultValue,  DATA_TYPE as ColType, IS_NULLABLE As IsNullable,  COL_LENGTH('#{table_name}', COLUMN_NAME) as Length,  COLUMNPROPERTY(OBJECT_ID('#{table_name}'), COLUMN_NAME, 'IsIdentity')  as IsIdentity, NUMERIC_SCALE as Scale  FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '#{table_name}'"

As you can see, here it will try to match ‘Spacey Table’ = ‘[Spacey Table]’ for the brackets case (or ‘”Spacey Table”‘ for double quotes).

Also, get_table_name(sql) will have trouble with this.

To make this work without breaking encapsulation will probably require using an escaped table_name and then selectively unescaping it for the SQL Server-specific uses.

All in all, MSFT’s behavior is fairly satanic on this; see below for a link describing syntax and escaping. Note that the only backwards-compatible solution is to SET QUOTED_IDENTIFIER ON and then use “This Table”.”This Column” notation, in order not to run up against problems with SQL Server < 6.5.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/acdata/ac_8_con_03_89rn.asp

Posted in bugfix | No Comments »

Password Policies Must Be Disclosed!

July 20th, 2006

Most all web applications require some sort of username and password to login. However, you never know upon signing up how the application (site, service, whatever) will treat your password. – Option 1: The app does what all good applications, having been entrusted with your password, ought to do: it passes it through a one-way hash function (ideally with a random salt), and stores /only/ that hash code. There is *no* mathematically feasible way to retrieve your password, given only the hash code (although in some cases dictionary attacks work). Hence, if you forget your password, they have to send you a new one, but at no time does a second human have a chance to see your chosen password. – Option 2: The app is brain-dead, and stores your password in plain-text, (or in a symmetrically encrypted form where the decryption key is programmatically available to the app). These boneheads become recognizable when, upon using the “forgot password” functionality, they /send back your original chosen password in unencrypted email!/ (Dear reader, it should hardly be necessary to describe why this is a problem, but consider that in our wireless age, every unencrypted communication may as well be considered public knowledge.) There are exceptions, of course, where option 2 is quite reasonable; the venerable mailing list manager Majordomo tells you up front, “/Do not use a valuable password as it will occasionally be emailed back to you in cleartext/,” which tells you right up front what the situation is. Yet, any number of ostensibly professionally-run web apps — /many of whom anticipate being the lucky recipient of my credit card number to consummate a transaction/ — cannot be bothered either to properly protect my password or to inform me about their practices. *Therefore, I propose:* a “Password Statement”, as an adjust to or embedded within a site’s privacy and security policies, that describes how they plan on treating your shared secret (password). This statement should be summarized in a one-sentence line next to the password box on the registration dialog, with a link to more information (much as has become the /de facto/ standard for a statement about spam policies). Finally, I propose a voluntary “seal” program whereby a TRUSTe- or Verisign secure site-like seal is made available, subject to a benign entity’s copyright and trademark protection, to both signify a site’s password policy at a glance and to serve as a link to the password statement.

Posted in Uncategorized | No Comments »

VC Annoyance: Term Sheets Excluded from Closing Book

July 17th, 2006

The Closing Book of a financing is the definitive collection of all documents relied upon in conducting a financing. It will generally have a “snapshot” of critical documents before and just after the financing, such as the original Articles of Incorporation dated one day, and the “amended and restarted” articles dated the next. Other agreements such as Investor Rights Agreements and Stock Purchase Agreements are included as well.

However, much to the chagrin of analysts like myself who often have to go back to the closing book for reference purposes, the actual term sheet often is not included in the closing book! This is most frustrating, since as a practical matter it is usually more relevant to discuss “term sheet to term sheet” when comparing terms or negotiating a subsequent round, rather than referring to definitive legal documentation (I estimate that the ratio of words and pages in a term sheet to those in the definitive docs based upon the term sheet is around 1:250).

I believe this is because attorneys are loathe to include anything in the closing book that might give even a thin entering wedge of challenging the definitive docs. Specifically, if a term sheet is included, a party might contend that the term sheet was the “parent” of the documents that followed, and should therefore inform the interpretation or construction of the definitive docs.

This may be a legitimate concern, but damn it, lawyers, couldn’t you just stamp: “non-binding, subject to the definitive documentation” in red all over the term sheet and stick it in the closing book??

Posted in vc | No Comments »

Simple Patterns

July 14th, 2006 
       Mutt understands the following simple patterns:         ~A           all messages        ~b EXPR      messages which contain EXPR in the message body        ~B EXPR      messages which contain EXPR in the whole message        ~c EXPR      messages carbon-copied to EXPR        ~C EXPR      message is either to: or cc: EXPR        ~d MIN-MAX   messages with "date-sent" in a Date range        ~D           deleted messages        ~e EXPR      message which contains EXPR in the "Sender" field        ~E           expired messages        ~f EXPR      messages originating from EXPR        ~F           flagged messages        ~g           PGP signed messages        ~G           PGP encrypted messages        ~h EXPR      messages which contain EXPR in the message header        ~H EXPR      messages with spam tags matching EXPR        ~i EXPR      message which match EXPR in the "Message-ID" field        ~k           message contains PGP key material        ~l           message is addressed to a known mailing list        ~L EXPR      message is either originated or received by EXPR        ~m MIN-MAX   message in the range MIN to MAX        ~n MIN-MAX   messages with a score in the range MIN to MAX        ~N           new messages        ~O           old messages        ~p           message is addressed to you (consults $alternates)        ~P           message is from you (consults $alternates)        ~Q           messages which have been replied to        ~r MIN-MAX   messages with "date-received" in a Date range        ~R           read messages        ~s EXPR      messages having EXPR in the "Subject" field.        ~S           superseded messages        ~t EXPR      messages addressed to EXPR        ~T           tagged messages        ~u           message is addressed to a subscribed mailing list        ~U           unread messages        ~v           message is part of a collapsed thread.        ~V           cryptographically verified messages        ~x EXPR      messages which contain EXPR in the "References" field        ~y EXPR      messages which contain EXPR in the "X-Label" field        ~z MIN-MAX   messages with a size in the range MIN to MAX        ~=           duplicated messages (see $duplicate_threads)        ~$           unreferenced message (requries threaded view)         In the above, EXPR is a regular expression.         With the ~m, ~n, and ~z operators, you can also specify ranges in the        forms <MAX, >MIN, MIN-, and -MAX.     Matching dates        The ~d and ~r operators are used to match date ranges, which are        interpreted to be given in your local time zone.         A  date is of the form DD[/MM[/[cc]YY]], that is, a two-digit date,        optionally followed by a two-digit month, optionally followed by a year        specifications.  Omitted fields default to the current month and year.  (source: man (5) muttrc)

Posted in cheatsheets | No Comments »

Movielink is Sorry.

July 13th, 2006

In speaking with folks in the P2P and content delivery space, the name Movielink pops up fairly frequently. I’m not a great follower of the cinema, but when I do want to see a film at home, something within me is repulsed at the idea of letting Blockbuster have another shot at extorting a “purchase” out of me when I have a perfectly good 6 Mbps downstream link to my home.

Also, I no longer have a roommate with 3/4 of a terabyte of movies on a file share in the apartment.

So I recently checked out Movielink’s website. I was unimpressed with how they chose to start our market conversation. Here is literally the very first thing they decided to say to me:

Sorry, but in order to enjoy the Movielink service you must use Internet Explorer 5.0 or higher, which supports certain technologies we utilize for downloading movies. Click here to get the latest version of Internet Explorer. We do not support Mozilla or Netscape. We apologize for any inconvenience this may cause.

(from http://www.movielink.com/store/web/error/siteentry/browserError.jsp)

Yes, Movielink, you are sorry. A sad, sorry sack of software that refuses to use standards, and cultivates customer contempt.

The business side of me is dumbfounded that a comparatively well-funded (backed by a consortium of major movie studios) and ostensibly tech-heavy company has nobody at the helm when it comes to customer experience. (Earth to Movielink execs: even if the specialized DRM you want people to use is incompatible with my browser, a semi-skilled salesman takes that first “no” and uses it to begin the conversation to convert me.)

The tech side of me wants to propose a new RFC for promulgation through the Internet Society: services that require proprietary, non-standard software MUST NOT listen on ports 80 (http) and 443 (https).

Posted in rants | No Comments »

Feedback Rant Disclaimer: I’m Trying to Be Constructive

July 11th, 2006

In the past several weeks I have had the opportunity to alpha-test, or beta-test, or other-Greek-letter test an abnormally large number of sofware products and web sites.

Not coincidentally, I have also had occasion to write a number of “user experience horror story” rants directed into various types of feedback forms, support@whatever.cxm addresses, and the like.

If you are the recipient of one such rants, I am writing this disclaimer for you to let you know a bit of where I’m coming from, in the hopes that you will take my feedback in the constructive spirit in which it was intended.

I love technology and I want to love your software.

I work for a software VC, and I’m an occasional contributor to various open-source (and proprietary) software projects. I install new software all the time, and only in a tiny fraction of a minority do I have any direct or indirect financial interest. So, I wrote my feedback rant mostly to try and be helpful; it’s part of my personal ethic of fighting entropy.

I managed a web app company for a few years, and I know some pitfalls to avoid.

I am by no means a guru, but if I rant at you about something specific to your technology, you might want to listen up. There’s a chance it’s a problem I’ve already fixed, or paid people to fix (or both, repeatedly, in some sad cases), and I want to save you that pain.

Other users will not be as nice.

Sometimes I state my case forcefully to get your attention. Other users will typically not be so nice — they will silently leave, or worse, save their vitriol for public fora (or worst yet, for private ones that you cannot see).

The frustrated user is the one to whom you owe your attentions.

Consider that if a user gets to the point where he is frustrated with your software and actively sending you feedback about that frustration, he has already demonstrated that he is patient (he was not frustrated at first) and persistent (if he were not, he would not have become frustrated but would merely have left). Do not turn a deaf ear! That same motivation for continuing past the point of frustration may be a motivation to spend lots of money with you, to contribute to your project, or otherwise to reach a mutual benefit — but only if you listen and act!

In closing, let me say that I’ve read a lot of user feedback, and not all constructive. My feedback rant is intended in a spirit of constructive criticism, and I ask that you receive it thus.

Posted in rants | No Comments »

« Older Entries
Newer Entries »