rlucas.net: The Next Generation

If you are finding that some browsers are talking to your new Apache/OpenSSL install,
while some are pulling a total blank (looks like a connection refused
or server not found), and you are getting this error:

…
then heed the warning.  You are likely using the DSA server
certificate that comes with some fresh installs.  Check your cert
directories:

Do you see that your server.crt (or whatever your httpd.conf defines as
your cert) and your server.key (or whatever is your key) are symbolic
links to the default “snakeoil” certs?

Ok, then you might have better luck in using the RSA versions, which play nice with more browsers:

(Remembering that with Apache, when playing with SSL stuff, do a full stop and start upon making changes — a HUP won't cut it)



As per all recommendations, do away with the snakeoil stuff ASAP and certainly before putting anything up on a public network.



CAVEAT: Do not use this advice for production.  This advice should
only be used for your own dev or testing, in order to get a fresh
install at least nominally working.  If you want real SSL and
can't figure it out, pay someone, because your security is worth it.

It appears that Lotus Notes occasionally sends malformed base64
encoded attachments.  I received a message from a Notes user that
was a forwarded message; the old message got hamburgered.

Others?  Appears so.

http://www.linux.ucla.edu/pipermail/linux/2002-February/006367.html

We especially suffer this with our RT installation.  No particular word on a fix at present.

If you are a reasonable Perlista, the first thing you will do when you
have to do some modest but non-trivial munging of data locked up in XML
is to use XML::Simple.  The API is nearly perfect (absent the lack
of some defaults that could be more helpfully set for strictness) for
purposes of comprehensibility and transparency.

However, if you prototype on a small document, and then try to use your
code on a much bigger XML document, you will find the drawback:
tree-building is costly, and you may spend the vast majority of your
program's time parsing in the document.  One handy solution is to
preprocess your XML — just run XML::Simple's XMLin sub, and use
Data::Dumper to spit out the structure that results to a file. 
When you want to use it, you can simply “eval” it, for it defines a
native Perl structure, and you can use the remainder of your code
unchanged.  This resulted for me in a 2x – 10x speedup for certain
documents and certain sizes.

However — now imagine that you have some real torture-test data — 10
MB, heavily nested monstrosities of XML.  The Dumper output of the
parsed tree is now working on 100 MB!  Slurping this in and
evaling it is now the real problem.

Here's an idea: rather than slurping and evaling, try inlining it at
the compilation stage.  That's right — make use of Perl's much
more efficient way of slurping and evaling a filehandle with a pipe:

cat preprocessed_xml.dd myscript.pl | perl

It's somewhat unorthodox, but entirely functional.  Combined with
judicious use of gzip, this could be a very efficient way to get
little-changing XML documents into perl quickly — often very important
when doing dev work for which numerous iterations are required and for
which a minutes-long parse stage would adversely affect progress.

Update: It occurred to me that
using Storable or a Cache::* module might be faster yet.  At this
point, my work proceeds with tolerable speed using Data::Dumper, plus I
like using Dumper so that I can edit the output structures by hand if
need be.  But perhaps you should try those modules if you need
even better performance, or cringe at the hackishness of catenating
files piped to perl.

I use Microsoft Internet Explorer version 6 (IE6) on Windows 2000 when
I have to (much better to use Mozilla or Opera in my opinion; even some
of the Microsoft guys are now eschewing IE for security reasons). 
I tested a document that passed some pretty strict validation and was
showing up fine in Mozilla, under IE6.  The title appeared, but
the body was blank.  What?!

It turns out that the problem was independent of quirks mode on/off
(google for quirks mode if you don't know).  It was dependent upon
two “<script language='javascript' src='blah' />” tags in the
<head> section.  By changing the <script /> to
<script></script> (explicit closing tags), the body
reappeared OK.

“I have created a computer virus and released it over the Internet.  It is named “EULAVirus” version 1.0.

“The virus takes the following actions, besides replicating
itself.  It seeds a pseudorandom number generator with a number
based upon the machine's unique characteristics, such that the PRNG
sequence will always be the same for the same machine.  Then,
during a dialog box, wizard, browser window, or other interactive
session (“dialog”) when certain key words and / or pixel combinations
are detected, it takes a “fingerprint” of the dialog based upon certain
characteristics, and uses the PRNG to determine whether to act on that
dialog.  The manner in which this is done ensures that for a given
dialog on a given machine, the same action will always be taken. 
If a dialog box is acted upon, the virus will cause all text to be
scrolled through, and an approval button to be “clicked” (it does so by
interacting with the operating system at a lower level).  This all
takes places nearly instantly, so that any human watching the computer
perform this will be unable to perceive what has occurred, beyond
perhaps a brief flash of the dialog on a slow computer.

“I have deleted all traces of the virus and any of its documentation
from all computers I control, but not before propagating it out to the
Internet.  It is spreading rapidly but it is exceedingly stealthy,
and it is engineered to avoid detection at all costs.  In order to
prevent its detection, I will not say which operating systems it runs
on, nor will I identify specific vectors of transmission.”

Now: can a EULA (end-user license agreement) ever again be considered legally binding?

Scenario: you are using DBD::Pg to interface with your database (perhaps directly through DBI, or through an abstraction layer like Class::DBI or DBIx::ContextualFetch) when you get an odd result:

DBD::Pg::st execute failed: ERROR: parser: parse error at or near [your string, or the part of your string that doesn't begin with leading digits] at …

or

DBD::Pg::db selectrow_array failed: ERROR: Attribute “yourstring” not found at …

If you look at the PostgreSQL query log, you'll see that “yourstring” was not properly quoted as a literal in the SQL delivered to the parser.

Since you've either been relying upon your abstraction layer or personally doing the Right Thing and binding your values with the “WHERE thing=? AND otherthing=?” syntax, you're quite confused — this should all be quoted.

The problem is that Perl has flagged that scalar as a numeric value, possibly because you used a numeric operator on it (like > or == instead of gt or eq). The solution is to upgrade to DBD::Pg 1.32 or to explicitly stringify your string as “$yourstring”.

Below is the bug filed with CPAN.
—
Mac OS X 10.2, Perl 5.6.0, DBD::Pg 1.22, DBI 1.45

Bind values appear to rely upon Perl's automatic numeric/string scalar conversion in order to determine whether or not to quote.

This bug was discussed on
http://aspn.activestate.com/ASPN/Mail/Message/perl-DBI-dev/1607287

my $dbh = DBI->connect(…); #connect to Postgres; no errors with SQLite
my $scalar = “abc”;
warn “scalar is greater than zero (and now considered numeric)” if $scalar > 0;
warn “dbh->quote(scalar) works ok: ” . $dbh->quote($scalar);
warn “but bind values do not:” . $dbh->selectrow_array(
“SELECT 1 WHERE 1=?”,
undef,
($scalar)
);

Using a numeric operator on the scalar makes Perl auto-convert it to a number; this is interpreted by the magic in Pg.xs as rendering the scalar ineligible for quoting.

One solution is to bind those variables that must be text but might have been numberified with “$varname”, thereby stringifying them in the eyes of Perl.

In Class::DBI, there appears to be a problem with the Postgresql driver
and certain kinds of relationships being defined.  It shows up as
a “can't bind a reference” error in the DBIx::Recordset code for
Postgres.

It persists for me with Class::DBI 0.96 and DBIx::Recordset 0.26.

The effect is seen for me when I retrieve a Class::DBI object id # 56
with some has_a relationships defined, and then try immediately 
to delete it.  It bombs out with

Can't delete 56: Can't bind a reference at blah/blah/blah/DBIx:/Recordset blah blah


I can work around this by 1. removing the has_a relationships, or 2.
stringifying the object before deleting it (found this inadvertantly in
writing some debug statements to explore this; very vexing that reading
a property has such a nonorthogonal effect, but I'll hush since I
haven't the time to become a CDBI developer myself).

CVSPermissions is a set of scripts that are called by CVS upon invoking
certain operations, such as commit (wisely, CVS has hooks for just this
purpose).  The scripts check an access control list, and
selectively permit operations based on username.  Unfortunately,
while the scripts come pretty elegantly close to “the simplest thing
that could possibly work,” they use grep without considering its
propensity for matching substrings within a string, so user “lou” will
match the ACL entry for “alouicious.”  The solution is adding ^
and $ around the grep regexes, which I have done in the attached
tarball.

I haven't heard back from Vivek Venugopalan, the author of
CVSPermissions, about the bug.  So, I am providing CVSPermissions
v 0.3-rlucas-1 with my patches.  GNU GPL applies and AS-IS; I have
tested only on GNU/Linux with CVS 1.12.9 and GNU grep 2.5.1.

http://cyber.law.harvard.edu/blogs/gems/rlucas/cvspermissions0.3rlucas1.tar.gz

Happily, this is an open source success story.  Boy meets slightly
broken but otherwise perfect software, boy fixes software, software
helps boy do work, boy gives software back to the world.  It
brings tears to my eyes.

Update 2005-12-01: Vivek has emailed back and incorporated the changes; get the latest version at: http://www.sanchivi.com/cm/cvspermissions/

This posting has three parts:

I. Why video lottery is a bad thing for the state to run
    in which I describe philosophical and practical problems with the status quo.
II. What the state doesn't tell about the mechanics of video lottery
    in which I describe the key statistical
metrics of how deleterious the game itself is, which metrics are not
published by the state.
III. My proposed scheme for remedying the situation.
    in which I indulge myself with presumed legislative fiat.

It will in turn be followed (preceded, for those of you reading in LIFO
order on the main page) by specific information hinted at in section II.

I. Why video lottery is a bad thing for the state to run.

Video poker (video lottery) as run in Oregon is deplorable for a number of reasons:

1. It uses a state-enforced monopoly to dole out “free money” to the private sector at the whim of the lottery administrators.
    A. No innovations, goods, or services are required
of lottery “retailers” — the machine is in effect an artificially
scarce rent-producing box.
    B. Unlike in a public-private collaboration such as
a utility, where the public (through the state) gives rights-of-way and
monopoly in exchange for the pulic good of universal power and
communications accessibility, the private portion of the video poker
cartel (bars) brings nothing to the table except access to drunk people
with cash.
    C. Giving free rents to qualifying and willing bars
in this manner dramatically disorts the economics of bar ownership,
subsidizing poor management of properties (it is not unusual that a bar should “break even”
on honest revenues, and make the annual profit from video lottery) and
penalizing bars whose morals or decorum make video lottery unacceptable.

2. It figuratively “addicts” the state budget to its revenues, giving itself institutional inertia.
    A. Video lottery sales comprise the lion's share of
lottery sales ($530 M / $895 M in FY 04), making the video lottery the
sine qua non of the entire operation.  (The portion of sales from
video lottery will rise dramatically as “line games” — slot machines
— are added.)
    B. The net transfers to the state in excess of $364
M  make for 3% of the state's budget.  This does not include
an effective “slush” fund of carry-forward earnings.  Legislators
must account for this gap if they take anti-lottery measures,
effectively hobbling forward-thinking lawmakers.  Job and
entitlement cuts that would follow budget cuts due to lottery rollback
give the lottery allies in the powerful public-employee unions.
    C. The lottery uses its earnings, before paying the
state its dividend, to pay for aggressive public-relations campaigns to
ensure its institutional lifespan.  If Republicans are afraid of
having to “starve” state-perpetuated “beasts,” this one is
truly fearsome.

3. It literally addicts gamblers, and constitutes a massive regressive wealth-transfer system.
    A. The video lottery is faster and more addictive
than paper lottery tickets; in fact, Oregon video lottery now
officially includes “line games,” or slot machines.  For more on
the addictiveness of the video lottery in South Carolina, see
http://slate.msn.com/id/36673/
    B. By reducing the need for a higher or more
progressive income tax, the video lottery subsidizes the highest-income
Oregonians.  (Not that we need to be paying particularly higher
income taxes, mind you, but there you have it.)

4. WORST OF ALL: Video lottery is the least transparently-treated part of the whole lottery scheme.
    A. The lottery commission emphasizes the paper
ticket games on its web site and its PR campaigns.  In fact, the
“ticket” lottery sales (excluding keno, multistate, and sports) amount
to less than 22% of sales — the vast majority comes from video lottery.
    B. The lottery publishes frequency charts and odds
of winning for the various ticket games, but publishes little about the
video lottery machines.
    C. Nowhere are the rules, EV / hold, and frequency audits of the video poker machines published. 

On the flip side, however, there are lots of things to like about the video lottery:

1. It gives money to the state budget.
[2. Some folks like to play it for fun.]

How can any principled person like the video lottery?  Religious
folks should hate it because it's gambling.  Liberals should hate it because it's regressive in
its redistribution and because some people have fun with it. 
Conservatives should hate it because it “feeds the beast” and teaches
fiscal irresponsibility, and because some people have fun with it. 
Libertarians should hate it because it's a state-violence-enforced
monopoly in what should be a competitive market and therefore people
aren't free to have fun with it on their own terms.  Humanists
should hate it for its fostering of antisocial behavior and its
reduction of human potential to negative-sum lever-pressing
domapinergic repetition.

II. What the state doesn't tell about the mechanics of video lottery.

What do I mean when I talk about “hold” or “EV?”  These are
measurements of the “edge” in a game (hold being the house edge), or,
from the player's perspective, the Expectation Value, given as portion
of amount wagered to be received back on each trial on average (note
that EV is typically stated less the initial unit bet, e.g. starting at
0, giving a positive or negative figure, -.05 instead of .95.  I
prefer EV given with 1.00 instead of 0 as a starting place, for ease in
the types of calculations I prefer.  If you are using the normal
method, simply subtract 1 from the EV numbers I use).

A quick primer on EV: for every $1 you bet, how much will you make
back, on average?  With a perfect coin-toss, it's $1 in, $1 out —
50/50 odds, or a 1.00 EV.  If the coin were biased to heads by 1%,
then your $1 bet on tails would win $0 [or “lose”] 50.5% of the time,
and win $2 only 49.5% of the time, giving you a 0.99 EV.  EV under
1 is considered “negative,” while the rare chance for a positive EV bet
is one in which the EV is greater than 1, also known as “getting the
best of it.”  Representative EVs include: roulette (American)
.947, blackjack ~.98, craps (pass line) .985, coin toss, 1.00, a year
in a savings account, ~1.01, a year
in the US equity markets, ~ 1.07.  Remember, EV is calculated with
each and every bet, like compound interest, so a .99 EV game done for
100 trials should net you back only 0.37 of your money (though in
reality variance messes up that neat figure).  Contrast that
with 100 years in the stock market at 7% [1.07 EV], which should give
you 867 times your money back, less taxes.  The total amount bet *
the number of bets is “action;” if you bet $5 100 times, you put $500
in action through, even though you may only have ever had $25 to play
with.  The “hold” is1-EV, or the percentage of each bet that the
house gets back; because it's calculated on each trial, the house
expects to earn the “hold” times the “action.”  Got it?

Poker players, stat jockeys, and others with a sense of EV will appreciate this chart from the Oregon Lottery's
annual report FY 2004:

Sales: The Lottery had the highest sales
years ever in each of the following categories:
Total Lottery Sales:                   $895.18 million

Video Lottery Sales:                 $530.97 million

Total Traditional Sales:              $362.30 million
Keno Sales:                             
$116.48 million

Powerball Sales:                      
$  45.97
million

Sports Action Sales:                 $  10.00 million

At the bottom is Sports Action — a game that is technically beatable
(positive EV) if you are an expert handicapper.  Then comes
Powerball, a game that is rarely, but occasionally, positive EV because
it is progressive (if nobody wins, the jackpot can theoretically get
large enough to give a positive return).  Then comes Keno and
“traditional” — the stuff that most people think of when “lottery” is
said — all negative EV games but transparently so.  But almost
all the money, in reality, comes from video lottery — video poker (and
soon slots) — about which almost no information can be found.

Note that video poker can be positive EV — in a competitive market
like Las Vegas, where 1. operators have an incentive to reduce the hold
(increase the EV) and 2. progressive jackpots exist that, when unwon,
can boost the EV to positive.

Why, gentle reader, am I spouting off about the corrupt lottery system?

The answer is that I recently visited a local watering-hole with some
friends, and lined in the cramped ante-bar area were some video lottery
machines (no boon to those of us smooshed together, waiting in line to
get a drink).  My friend A. pointed to the screen of the nearest
one, which read

[Machine Configuration]      [Game Configuration]
[Reports]              
             [etc]

…instead of the usual blinky-beepy come-ons.  Curious, we
pressed the reports button, and found the machine happy to give us
printouts of the game holds, results, revenues, etc.  For your
edification, I will be posting this information shortly.  However,
in sum, please be aware of the following:

– The EV on most all Oregon video lottery games is 0.90, or put another
way, the “hold” is 10%.  This is a truly outrageous rate that
makes casino roulette (~5.3%) blackjack (~1-2%) and craps (~1.5%) look
like great deals.

– The EV on “flush fever” is 0.94, or the hold is 6%.  This is
very bad, but a hell of a lot better than others (like “jacks or
better”).

– The “double up” game, offered after each win, is even-money (EV 1.00).

(Keep in mind that all games played against a “house” with
near-infinite resources subject the player to “gambler's ruin,” a
situation in which the player underperfoms his EV because short-term
variances wipe out his bankroll.  This boost the effective hold
over the theoretical hold, in most cases.)

Why is this significant?  After all, everyone knows that the
lottery is a sucker bet, right?  Well, perhaps, but there are
sucker bets, and then there are sucker bets.  You are wiped out
exponentially faster with such a huge edge as 10%.  Consider a
brisk but realistic pace of 360 video poker hands per hour — the least
that may be wagered is 25 cents.  The total action per hour is $90
— at 1% hold, the player loses less than a buck.  But with Oregon
video lottery, the least he will expect to lose is $9 per hour — more
than the state's minimum wage.  Consider, too, that the average
bet is larger, and that the swings of the game put him at risk of
gambler's ruin.

The difference between a sucker bet, and a sucker bet, is huge. 

III. My proposed scheme for remedying the situation.

Here is what should be done:

[As a general premise: 0. “Indian” casinos should be replaced with a
brand-new way to take land and manpower and make jobs; I call these
special programs Indian “factories.”  The Indian factories can
make “goods” that are exported at a “profit.”  This will give them
jobs and tribal tax revenues.]

[UPDATE: The Wall Street Journal recently covered a development where
it appears others share the above premise — there is now a trend for
Indian tribes to use reservation land to build legitimate industry,
like concrete plants.]

1. The state gives up video lottery.
2. To combat grey market video lottery, to prevent folks driving out of
state to play other states' terminals, and to ensure proper income
reporting for taxation, the state forms a video lottery audit board.
3. The audit board licenses establishments to have a certain number of
machines present, and licenses machine owners to run any number of
privately-owned machines with publicly-audited software.  Each
machine's hold percentage may be set and changed by its owner.
4. Each machine must be conspicuously labelled with its hold percentage / EV (see II. above) and most recent audit.
5. The audit board charges fees sufficient to fill its budget. 
Overage is rebated to license-holders, preventing the state from having
an interest in promoting video lottery.
6. Machine-owner licensees contract at arm's length, at-will, with
establishment-licensees as to machine placement.  This creates a
healthy, adversarial competition for the customer's dollar between the
booze and the slot machine.
7. The state makes its money off of income taxes.

The ends my plan serves are:
1. Elimination of state dependence upon, promotion of, or interest in, people losing more money at the video lottery.
2. Reduction in the “hold” and profitability of video machines by market forces, thereby:
3. Reducing incentives for “Indian” casinos by taking pricing power away from them on video poker-type games.
4. Reducing incentives for poorly managed bars by lowering the amount of the received subsidy from the video lottery.
5. Maximizing  happiness by promoting more play per unit wager for
lovers of video lottery and eliminating video lottery from some bars
for whom the lowered margins do not justify keeping the games.

Executive Summary:

(Some numbers have been rounded to fit annual time frames.)

This machine had ten normal games, of which all had a 10% hold, except
one with 6% (“Flush Fever”) and one with 8% (“Oregon Gold”).  The
“draw high” game is 0% hold (no house edge).

Contrast this with ~5.3% for roulette, ~1.5% for craps, and ~1-2% for blackjack.

By far the most money was played on “Flush Fever,” the game with
the lowest hold.  This is probably because the difference between
a 10% hold and a 6% hold is so dramatic, that even without labeling, a
player can detect it readily.  About 45% of the action (money
played) was at this game. “Jacks or Better” got about 12% of the
action.  With 8% and 7% respectively, were “Deuces Wild” and
“Oregon Gold” (the 8% hold game). 

Since there are ten games, and the #1 and close #4 games are the least
and second-least holding games, we can surmise that people play more at the lower hold games.

Previously, I had speculated that the least you would expect a player to lose
under a 10% hold, playing one 25-cent game every ten seconds (brisk but
not blazing) would be $9 an hour. In fact, nearly all games recorded an
average bet of 75 cents or more.  That means that the hourly loss
rate would be at least $27 an hour, an hourly rate equating to a full-time salary of
$54,000 a year.

In fact, the overall hold percentage is reduced by two facts: 1.
players preferentially play the less-biased games, and 2. an
even-money, “double or nothing” bet with no house edge is offered to
winning hands.  The overall theoretical hold for the machine as
played should have been 6.37%, though it lagged slightly with only 5.8%
actually held.

Of the cash that had, over three years, been fed into the machine, more
than half is denominated in $20s (the largest).  About $267,000 in
bills had been put into the machine.  About $184,000 in winning
tickets had been printed.  Although on each coup the player might
expect 90% back, for every buck actually put in the machine, only 69
cents come back out.

The machine in question had about $1.4 million in action put through it
over 3 years (recall, action is calculated each bet, so it will be many
multiples of cash drop).  The total hold was about $83,000, about $2300 a month (one machine), or
about 5.8% of action (the “draw” game counts toward action, but cuts
down on the hold percentage since it has no inherent advantage).

How much play did this box get?  With an 83 cent average bet, and
476,400 dollars put through a year, that's about 574,000 bets per
year.  That's about 1752 a day, or 131 an hour through a 12-hour
day.   This sanity-checks my 360/hour estimate — 1/3 of the
time in rapid play seems sane.

The bar had five video lottery machines (the legal limit), but only one
of them was kind enough to tell us its financial history.  To
situate it, it's a youngish, dive-y 20s and 30s bar, with pool tables, the kind
of place where a cuba libre costs less than 4 bucks and they don't call
it a “cuba libre.”  In those terms, the patrons of that bar could
have had another 21,000 cubas libres over the last three years instead
of playing video poker.  This is not “el primo” territory for
video poker, though I would guess they do OK by video poker standards.

To do some quick math:
5 machines * 89000/year/machine = $445000 / year / bar drop
5 machines * 27667/year/machine = $138335 / year / bar hold
Retailer commission (average, per Oregon Lottery) 28% = $38733 / year / bar hold

To get $38733 annually, risk free, at 2% interest, you'd need
nearly a cool $2 million in the bank.  What does the retailer
stake for this?  About 25 square feet per machine, including chair
space.  With five machines, that's 125 square feet generating
$38733, or $310 / sq ft / year.  Today in Portland, Oregon's
largest city, you would be hard presesed to find Class A office space
renting for more than $30 / sq ft / year.  So the rental rate that
the Lottery is paying dive bars is only a 1000% premium over that for a
suite in Portland's toniest skyscraper.

To bring it back to earth, let's sanity check all of this against the known figures:

circa 2100 retailers * $138335 / year / retailer = $290 M hold overall

This is in sanity-range with the lottery's published $530 M figure
(there tend to be a few top-performers in the video lottery that skew
the results to the high end).

This could be an interesting case study for anyone looking at the
recently-again-in-the-news issue of Oregon's video lottery. 
Unfortunately, nobody is talking about how we can mitigate the harms;
instead, everybody just wants to wring more money out of the program.