rlucas.net: The Next Generation

There are a lot of problems in software that aren’t solved well in a ubiquitous product (think PIMs, Personal Information Managers; they all suck royally despite everybody’s best efforts, and the OSAF Chandler project has taken years trying to redesign the very concept, with little to show for it to date). But there are precious few problems that haven’t been solved at all.

In fact, a ton of things that are being held up these days as “innovations” are rehashes of old concepts from the 90s, or the 80s, or sometimes even the 70s. Today this came into sharp focus when I saw this bit from a circa 1999 document on the Semantic Web by the Right Reverend Tim Berners-Lee. Here he asks, then answers, a question:

<blockquote> Surely all first-order or higher-order predicate caluculus based systems (such as KIF) have failed historically to have wide impact?

The same was true of hypertext systems between 1970 and 1990, ie before the Web. Indeed, the same objection was raised to the Web, and the same reasons apply for pressing on with the dream. </blockquote>

Then, while searching for some theory on append-only databases (such as would be used in a revision-control system), I came across this 1994 piece on “collaborative filtering.” That report in turn points to earlier work on “Information Tapestry” from the early 1990s.

So: 1970-1990, hypertext exists and is studied in CS departments. 1995, Netscape IPO. Early 1990s, collaborative filtering exists and is studied in research labs. 2006-2007, rise of Digg and Delicious.

I think there’s a very strong case to be made that VCs should stop looking for “innovation,” per se, and start looking for 10-20 year old CS masters’ theses that touch on an emerging market space…

Seattle-based Open Source startup Sourcelabs put together the Swik.net wiki a year or two ago. They seeded it with some links of moderate usefulness, and for a brief time, it was a decent, if hit-or-miss, way to find information about an open source project or tool that you were considering using.

No more. Not only are most of the pages I’m finding on Swik these days simply a one-link screen-scrape to the actually interesting page (which often ranks higher in Google alone, anyhow), but Swik has committed the cardinal sin of infovoria: playing audio that automatically starts on page load.

(They do this by means of an auto-starting video advertisement that spams up the top bit of the page. Not quite as egregious as a MIDI object, but every bit as annoying.)

Unbelievable. I thought we’d gotten past this with the turn of the millennium. But everything old, it seems, is new again. Swik, however, now gets the same mental category as “About.com,” namely, ad-filled, nearly unusable results not to click when they appear in a web search.

I’ve been using an “el cheapo,” $15 / month, 64M RAM Debian GNU/Linux VPS from Spry for a couple weeks now. Some things to note:

• You get X amount of RAM with “burstable” extra ram, but you’re not going to get any from the burst. Everyone else is using it.
• There is NO SWAP as far as I can tell. When you hit 63.9M of RAM, processes start segfaulting and blowing up.
• free and top will lie to you. ps -aux seems to tell the truth.
• You really need to check /proc/user_beancounters as root to get the real number of (4k) pages of memory used, and the number of faults, if you’re curious.

Too early for a verdict. But don’t count on being able to run the same stuff on a 64M Spry VPS as you would on a 64M box with a half-gig of swap (grinding its way through but eventually getting the job done).

Many of these problems arise because RDBMSs are designed typically with the conceit of being the sources of truth within the organization (e.g. invoice #1234 exists because the database says so), but are often used to reflect external truths about the world, which are input messily and which themselves are often shifting or subject to change in ways not contemplated by the schema.

I. The entity deduplication problem.

Alice is entering information about companies, and creates a record #1001 for “Acme Widgets.” She then adds a bunch of information, including relating other tables to Company #1001. For example, she may put in a press clipping about Acme Widgets, which gets linked by a link table to Company.id==1001.

Then, Bob comes along entering information about companies, and creates a record #1234 for “Acme Widget Corporation.” He adds stuff, and includes a different press clipping, which gets linked to Company.id==1234.

Later, Charlie arrives, and notices that there are two records starting with “Acme.” He investigates a bit, and discovers that “Acme Widget Corporation” is the full legal name of the company that is familiarly called “Acme Widgets.”

How can Charlie cause the database only to reflect a single Acme which is associated correctly with both press clippings?

II. The undo problem.

I do some stuff. Then I change some stuff. Then later, I want to see how it was before I changed it. Tough luck.

III. Bitemporality

Let’s include financials on our companies! Acme revenues are $12 M. (wait a year.) Now Acme revenues are $15 M. But wait: we now need a couple of slots for revenues. Oh, ok, a revenue report is associated with a year. Acme 2006 revenues were $12 M, 2007 revenues were $15 M. Wait. I was wrong. Acme 2006 revenues were $9 M in reality. OK, update that. Now the boss calls up and wants to know who it was that we told the wrong 2006 number to. But we can’t, since we don’t know what we thought we knew when we thought we knew it.

(In fairness, bitemporality is somewhat easier than the other ones; you just double-time-stamp everything. But it’s still a pain in the ass.)

IV. The incomplete multi-table entity problem

This one is not really an RDBMS problem so much as a Web applicaiton architecture problem.

Doug wants to create a record in our database — let’s say, a “publication.” The publication must have an “editor” (n:1) and at least one “author” (n:m, m >= 1). Doug gets to inserting the publication, but is stopped because it has a null editor_id, violating the editor constraint. D’oh! OK, we can work around this: good databases permit deferring of constraints until the end of a transaction. But in order to make that work with your application, you now must tightly couple the transactionality between the RDBMS and the server part of the Web application.

Let’s try again with Doug. He inserts the publication, no sweat, and now can insert an editor. Then, he adds some other stuff, like some authors. (Maybe Doug is using an AJAX-y Web front-end that lets him add new authors on the fly without going to a new “screen,” or maybe he has to navigate between modal “screens” to do this.) Because he’s added multiple different entities (a publication, an editor, some authors), he gets to feeling that what he’s done is already written to the database. He leaves without hitting save. Do’oh! Two things have now happened: first, the entire graph of all the entities he’s added is in limbo, and second, the server part of the Web app is holding open a (possibly scarce) connection to the RDBMS (which may or may not, depending on how fancy the example gets, be holding locks with a SELECT FOR UPDATE…).

The first sub-problem — the limbo — seems not that bad, because Doug is used to losing all his data when he forgets to hit “save” on a desktop application, so he’s fairly well trained and will avoid this. But if the program is “DRY” (Don’t Repeat Yourself), and especially if it had modal “screens,” Doug probably hit “save” or “update” or “submit” several times on interfaces that look like the normal (non-multi-table-entity graph) ones in order to add his multiple required entities (editors and authors). Therefore, Doug has a not unreasonable expectation that he’s done his part for saving, based on the fact that when he edits or creates an author entity in isolation, he uses that same workflow and it saves OK.

The second sub-problem — the RDBMS connection — is sort of more pernicious, because now Doug isn’t just losing (or jeopardizing) data he though he’d saved, but because he’s potentially contributing to a denial or degradation of service for all users. This is sort of an artifact of the way that DB connection pooling evolved from the mid-90s to today. Traditionally, DB connections have been scarce and costly to set up (network IO in addition to whatever socket / semaphore / locks / whatever had to be written to disk). Therefore, the widespread practice evolved in Web application design to use connection pooling — where some subsystem in the server layer makes a bunch of connections to the database, and then does fast handoffs of DB connections to application requests, freeing them up when each request is done. That way, you can run, say, 300 near-simultaneous requests with, say, 30 database connections.

Of course, if you’re pooling database connections, you pretty much have to run your entire transaction, succeed or fail, within the space of one application request (hopefully a sub-500ms affair), since transactions bind to database connections, but connections don’t bind to end-clients (stateless HTTP again). You can try and bind the DB connection to a particular client’s cookie- or token-identified session (like Seaside does, I think), but then you lose out on the ostensible benefits of pooling — now you need 300 RDBMS connections for your 300 clients, and your DB machine is choked. What’s more, if Doug leaves his transaction open, and you don’t fairly quickly time it out and kill it, you could end up needing more than 300 DB connections for your 300 clients, because you also have 300 old “Dougs” who’ve left stale transactions open — and now your DB machine is crashed.

I will update this post periodically with other relational database problems, and, I hope, solutions.

Last night, I was typing happily along on my MacBook Pro, unplugged and on battery power with about 40% indicated remaining, when bam, the power went right out like a light. Curse words. Flipped the thing over, and got no LED action from the battery charge indicators. Plugged it in and it would boot, but pull the plug and it goes down hard again.

I had just bought the thing in December 2006, refurbished direct from Apple and with a 3 year Applecare extended warranty.

Well, I hopped on the bike and made it to the Apple store in the peculiar ring of hell known as “University Village” (an upscale retail cesspool here in Seattle. Aside #1: Never go there. Aside #2: When you must, ride a motorcycle or bicycle, because parking is impossible, and the sheer Schadenfreude of watching the SUVs joust and jockey while you glide to within a few meters of your destination almost makes the place bearable).

The Apple storeites had me belly up to the Genius Bar (their open-air repair facility). I handed the MacBook over with a brief description of the problem. As the guy was keying in something (serial number?) he started to regale me with an explanation of the normal sleep procedure and how the battery is supposed to work. “Oh shit,” I thought, “here comes runaround city.”

The guy leaves, I get ready to get comfortable and either wait it out or raise sufficient hell to get what I need.

To my surprise and delight, when the “Genius” returned, he had a box in his hand just slightly bigger than the batter in question. Snip, snip, out comes the battery, and into my laptop it goes. Hands it over without a word and gets to printing me a receipt. Holy shit — he just silently and competently fixed my problem, without me paying a dime or sitting through any bullshit (other than the complimentary, brief lecture on hibernate vs. power down while he looked up the inventory for the replacement part)!

Too many other tech repair facilities have long algorithms they have to go through, debugging procedures, etc. — all of which ostensibly save money on parts but waste uncounted hours of techs’ and customers’ time. Apple, despite really dropping the ball when my wife’s laptop started crashing intermittently eleven-and-a-half months into owning it, really came through on this one.

Pros: This is exactly how customers should be treated. Fix it. Sign for it. Send ’em on their way. Go Apple!

Cons: This probably means that the Genius bar sees quite a few cases of MacBook Pro battery-sudden-death.

I read last week in the Durham (NC) Herald-Sun this AP story, which unfortunately I now find linked only at the ghastly FOX News:

Today’s college students are more narcissistic and self-centered than their predecessors, according to a comprehensive new study by five psychologists who worry that the trend could be harmful to personal relationships and American society.

Of course, we can hardly see this without warning bells that it is a salvo in the ever-more-boring-by-the-day Culture Wars. But interpreted another way — freed from its pettily polemical underpinnings — it is yet more evidence that the Myspace values have taken over from the old-school ones that persons of so venerable an age as my own (27) may yet hold. Self-promotion is no longer vice, but virtue. Narcissism, manifested through incessant working on of one’s own profile and pictures, is not the domain of a few egotist introverts but is every barista’s and coed’s pastime.

Reflecting on this shift during a week in Key West reading some of Papa’s value-challenging works on the Lost Generation was interesting. (I will not say enlightening, as the mind-addling effects of the State of Florida tend to preclude enlightenment while in the Sunshine State.)

I spent some time recently playing around on the post-acquisition YouTube. It seems that Google’s legal department has really cracked down on copyrighted material; about one in six links I surfed from within YouTube were broken (removed due to terms of service violations or such).

But in surfing the remaining stuff — most all amateur productions like guitar showoffs or beatboxing flautists — a realization came over me that has renewed my hope for humanity. This is an odd thing to say about grainy, jerky videos filmed in peoples’ garages and living rooms, I realize.

What it was in particular, though, that gave me this sense of hope, is understanding that people — non-professional people, amateurs in the both the “lover” and in the “unpaid” senses — have an incredible desire to express themselves through random-ass creativity, which they will do if only they have an excuse. YouTube is the excuse and the motivation.

Kids who have a crazy idea, say, to play Pachelbel on the electric guitar at breakneck speed, or to lay down a beatbox while blowing a jazz flute tune — a crazy idea that begs to be shared with other people just for the sheer joy of sharing — now have an outlet, a stage big or small upon which to showcase the finished product. Since a lot of this stuff is just too wacky and micro-scale to find outlets in traditional performance or social venues (you could never sell a ticket to see some guy spit and raspberry into his flute for 4 minutes, and it would take quite the rude host to foist such a display on one’s party guests, for example), in a pre-YouTube world one can imagine some number of would-be beatboxing flautists who simply never would bother to practice and innovate, since they’d have nobody with whom to share their micro-triumph.

Now, with YouTube available, one has a real, credible (if nonremunerative) way to get some social props for honing any weird, nonsaleable, but potentially interesting or amusing skill. This, I hope, will spur all sorts of creative activity, the practice of which I hold inherently to be better than mere consumption or participation in externally-created pastimes.

I realize that this is sort of a reformulation of “Long Tail.” But what I want to focus on is not on the existing producers of the long end of the tail, but rather on the incentive this provides for converting mere consumers into long tail producers, and the moral improvement this results in for those ex-consumers. And this is how YouTube has renewed my hopes for our race.

I was checking out the new Judy’s Book in an attempt to find a good local accountant. Well, it’s terrible. But of all the ways (incomplete, too many ads, too cluttered, unhelpful categorization, navigational and search deficiencies) in which it’s terrible, there is one which stands out:

Judy’s Book sells out its users by trading our attention and RSI affliction for extra page views.

What I mean by this is that there is no way to look at an object / entity (in this case, a local business that is reviewed), and see at a glance all the relevant information and reviews. You have to click through to a new page in order to read more than the first several words of each review (of which there may be many for each reviewee). This makes it impossible to quickly scan and get a feel for the business you’re looking at. Maybe the use testers at Judy’s Book are slow readers or in some other way deficient, but any modern infovore is well-adapted to scanning and snarfing. The difference is like ordering a Porterhouse and getting a pile of steak bites.

They must be getting CPM advertising rates, because the only rational reason for pissing off and insulting your users like this is to jack up your page view metrics. (As an example of how much people hate this, consider that it’s seen as bad netiquette to post links to the “online” version of a news story when there is a single page “printable” version available for non-moronic reading.)

So, I’m now on a jihad against websites that conceal information unnecessarily behind “read more” or similar links in order to drive additional page views. Don’t do it. Or else I will get whiney (and get carpal tunnel syndrome from clicking your damn “read more” links).

I was slow to adopt a lot of new social networking technologies. I read blogs for several years (OK, like, three) before getting into “active” blogging (fairly frequent, first-person writing, unlike the mostly-error-message blog posts I put up at my Harvard Law blog.

I was super-fast, however, to adopt Linux, email, instant messaging, online banking, and even “traditional” Web-page-havin’. Why the delta? (Maybe I’m getting old.)

I think it’s got to do with values. My values were about:

• the importance of one-on-one and small-group relationships,
• the value of one’s words, and hence the advantage of controlling and using care with one’s communications,
• measuring twice and cutting once,
• privacy
• etc.

There were also some values that I was aware of (but can’t claim personally to have really made much of):

• modesty,
• non-intrusiveness on others,
• etc.

These values privileged one-to-one communications media (email and IM), and one-to-many, “broadcast” model, write-then-publish media (like print publications, papers, and old-school “resume-ware” personal Web sites). If you believed this stuff, you made a few friends but really good ones. You were loathe to publish things that might look stupid. You didn’t want folks to find out who you knew and who knew you, necessarily, as that could be strategic info. And (of course) you wanted to toot your own horn, but without seeming to be doing so yourself.

But social networking rejects all that. If you hold on to all those old values, you’ll get left behind. In social media, the values are:

• promiscuity,
• self-promotion,
• frequency and speed of communications and replies,
• saying some crap just to be the one talking (this was big at Harvard, come to think of it),
• exhibitionism, or at least sharing one’s bad habits with the world, a la Hugh Foskett,
• etc.

So, we have a generation being raised on social media for their IT experience. What little television they will be watching in 5 years will all be “reality” shows, mixed in with “mash-up” or “remix” TV a la VH1 (where celebrity becomes its own vehicle and its own end, perpetuating itself for the sake of itself, kind of like how degrees in certain subjects qualify the holder only to teach that subject). In this world, having old-ass values will pretty much be limited to accountants, lawyers, and maybe some fussy academics (but not the ones with glossy popularized books or lecture tours).

This is and should be somewhat disturbing. Every day, I ride in an 80-year-old elevator. While I’m sure most of its moving parts have been replaced at least a couple of times over its service life, the mechanism itself still functions as the original engineer created it. I’m frankly rather glad that he was raised with old, crusty values rather than the bright, shiny new values with which we’re going to be replacing them.

I wish this were a flash in the pan. But things like webcams, podcasts, blogs, tumblogs, Twitter, and Dodgeball, not to mention Myspace and the rest, all point to these new shiny values rising to ascendancy.

So, look: if you want to survive and thrive in the brave new world of social media, you gotta jettison those old values that got you into Stanford or through your MBA or into the partner track at Sequoia. You have to embrace the ADD and the confusion and the nakedness.

(Or, you have to figure out how to use your old-school values to operate at an abstraction level above that of the twittering throng…)

I spent some time this week with some friends from the Portland Perl Mongers (PDX.pm) at 2007’s first meeting, devoted to Web app frameworks, games, and Martinis, in no particular order. Just as Seattle is blessed with a vibrant Ruby community (the Ruby Brigade meets up on Capitol Hill), Portland has a truly exceptional Perl community. At any given PDX.pm meeting, you’re likely to see any number of O’Reilly authors, high level Perl monks, and authors of CPAN modules you use regularly.

(Whether or not you know that you use them, the Perl language and its Comprehensive Perl Archive Network, or CPAN, are the behind-the-scenes heavy lifters of a great deal of the dynamic Web. The “modules” on CPAN are rarely applications themselves, but are underlying code libraries or components that enable user applications.)

Many of these luminary or guru type attendees at the PDX.pm, therefore, are effectively vendors and colleagues to those of us who have written Perl software using CPAN modules — whether or not we’ve ever met. And yet, apart from the occasional pint of beer, or royalties on an O’Reilly book or two, most of us will never pay (at least not directly) these Perl gurus for their work. This got me thinking, and, as I get to below, also got me somewhat concerned.

Open Source Software (OSS), including Perl itself and the modules on CPAN, is much loved in the business world due, in part, to its compelling price tag. Some OSS, like the Linux kernel, is developed with a great deal of assistance from commercial entities, who are often motivated to increase compatibility with their product offerings (as in driver development) or to serve some internal functionality requirement. But a dirty secret — one whose propagation has been discouraged by the promoters of OSS as part of their mostly laudable efforts to gain for it the same credibility enjoyed by commercial software — is that a vast amount of OSS is in fact written and / or maintained by the “volunteer” developers of lore. These folks are, I must emphasize, not amateur or “unprofessional” — indeed, the reason that many OSS developers / maintainers undertake the task is for professional recognition or advancement — it is simply that they have no (or only intermittent) direct economic ties to the software they maintain. Therefore, in many cases, you might have a software module whose author maintains it “out of the goodness of his heart.” Without the financial lever, users of the module cannot compel him to resume development if it falls off of his list of priorities.

(Note that I do not condemn OSS on this account; from the user’s perspective, the benefit of having free and unfettered access to the source code, in combination with the gratis price, should more than make up for the incremental uncertainty added by the reduced extortibility of the author.)

What is gained for the author and maintainer of, for example, CPAN modules, is a measure of respect and credibility among his peers. The merry band of geeks at PDX.pm, for example, treats certain of its guru alumni like conquering heroes when they return to give a talk. And I have no doubt that such a status gains one an entree when seeking consulting contracts or employment. But it’s not guaranteed, and just as the user has no leverage over the author, the author has no leverage over his users by which to demand a quid pro quo.

A disturbing consequence of this economic model came up explicitly in the PDX.pm post-meeting chatter at the Lucky Lab, however. There exist major contributors to software projects relied upon and used by thousands (if not millions; and in the case of Perl and its indirect users, it would not be much hyperbole to say a billion) of people — which OSS contributors essentially live in poverty.

Why is this, more so than other forms of poverty, especially disturbing? Well, for starters, these people are clearly creating economic value, but are martyrs to a system that doesn’t pay them back.

That open source is economically valuable may not be obvious to an Econ 101 student or the less thoughtful MBA, but even a nontechnical executive can look at the market cap of e.g. Red Hat ($4.2 B as of January 2007) and consider their ability to get there with only 1,100 employees (that’s $3.8 M per employee, comparable to $4.3 M for e.g. Microsoft, who is nearly 70x larger and is of course the global market leader and therefore presumably can get away with less SG&A per developer, and much more than $1.6 M for Oracle). If that doesn’t convince you, consider the replacement cost of many open-source products with their proprietary counterparts, and even applying a devil’s advocate discount, we find that the OSS developers are creating something worth money — yet, they often capture little or none of that for themselves.

So, although a brilliant developer on his own (or in loose collaboration with a few like minds across the world) can produce some useful thing that we believe to be worth money, that same individual has quite a challenge before him to monetize that work. And in fact, the rewards he seeks — recognition and social credit among peers — are best gained in precisely the circumstances that (are conventionally believed to) preclude monetization. By that, I mean that being a great software guru these days is best achieved by having others read and admire your code — which is done by open-sourcing it — and which in turn (notionally) stops you from extracting a fee from consumers.

Therefore, an injustice is done here because the rewards of good work are distributed not merely inequitably but in fact not at all to those who perform the work.

Another, and more practical consequence, is that there are any number of important parts of any sophisticated open-source software stack that are cared for by persons in limited or sometimes precarious personal situations. I mean cases like developers without health insurance, developers who share rooms with dodgy roommates, developers who could be staring down tax liens. You may point out that there are many more and poorer folks than OSS developers to worry about; of course this is true. But OSS developers are providing the raw materials from which innumerable startups (and incumbents!) are driving the highest-growth parts of our economy and building substantial empires.

There are very real risks for users of OSS that a dependency for some major project could fall victim to the personal hardships of a developer, if, for example, he is made to seek other work to support himself or if he falls ill without insurance. In the biggest cases (say, Google), the firm running the dependent project could simply take over the code — but in the case of a startup building on an OSS stack, the danger looms rather larger.

I have no grand plan to remedy this. And I don’t think that the (terminally broken anyhow) music copyright regime gives us much guidance. While in music it is the peripheral producers who languish unpaid and the stars who get big dollars, in the case of OSS developers, often the marginal producers are comfortably paid corporate contributors while the true “rock stars” of OSS go broke.

And although a large part of this problem comes from the effective waiver of copyright inherent in OSS, closed source software is not the answer. Closed source software (at least that which is meant to be used by developers or geeks) sucks too much.

Probably the closest I have to a recommendation on this is that users of OSS find and reward the “stars” of projects they depend heavily upon. And, having said that, I am off to send money to Bram’s Ugandan orphan
s (for the incomparable vim) and beer and pizza to the GNU Screen guys.