rlucas.net: The Next Generation Rotating Header Image

bugfix

Django auto_now types wreck get_or_create functionality

I recently had occasion to lazily use the Django “get_or_create” convenience method to instantiate some database ORM records. This worked fine the first time through, and I didn’t have to write the tedious “query, check, then insert” rigamarole. So far so good. These were new records so the “or_create” part was operative.

Then, while actually testing the “get_” part by running over the same input dataset, I noticed it was nearly as slow as the INSERTs, even though it should have been doing indexed SELECTs over a modest, in-memory table. Odd. I checked the debug output from Django and discovered that get_or_create was invoking UPDATE calls.

The only field it was updating was:

updated_at = models.DateTimeField(auto_now=True)

Thanks, Django. You just created a tautological truth. It was updated at that datetime because … you updated it at that datetime.

Interestingly, its sister field did NOT get updated:

created_at = models.DateTimeField(auto_now_add=True, editable=False)

This field, true to its args, was not editable, even by whatever evil gnome was going around editing the updated_at field.

Recommendation: if you want actual timestamps in your database, why don’t you use a real database and put a trigger on it? ORM-layer nonsense is a surefire way to have everything FUBAR as soon as someone drops to raw SQL for performance or clarity.

Don’t Bother with S3FS for Mounting Amazon S3 on Mac OS X (2014)

As of January 2014, it’s not worth bothering with the “s3fs” software for mounting Amazon S3 buckets on your local filesystem.

The idea of s3fs is simple and great: use FUSE (Filesystem in Userspace) to “mount” the S3 bucket the same way you’d mount, say, an NFS drive or a partition of a disk. Manipulate the files, let s3fs sync it in the background. Sure, you lose some reliability, but we’ve had NFS and SMB and all kinds of somewhat-latent-over-an-unreliable-link-but-mostly-with-filesystem-semantics software for decades now, right?

Well, forget it. s3fs as of January 2014, used on Mac OS X and against an existing set of buckets, is so utterly unreliable as to be useless.

First, s3fs cannot “see” existing folders. This is because folders are a bit of hack on S3 and weren’t done in a standardized, documented way when s3fs was first written. However, since then, at least two other ways of creating folders on S3 have gained currency: an older, deprecated way with Firefox plugin S3Fox, and a newer defacto standard way with Amazon’s own management dashboard/browser for S3. Whatever the historical reason, you can’t see the existing folders.

Second, although from mailing list posts, theoretically you can *create* an s3fs folder with the same name as your existing folder, and then its contents will magically become visible, empirically something rather different happens. A mkdir on the s3fs mount leads to the creation of a mangled *regular* file on the S3 dashboard. Now you have two “folders,” each of which is unusable as a folder on the other (S3 dashboard, or s3fs) system. Argh.

Finally, you might say, ok, fine, this will just make me use flat-level, non-folder-nested choices about my S3 architecture. (Leave aside for the moment that the very reason you want to use S3 is probably exactly so that you can have lots and lots and lots and lots (like 10^8+) files in a way that will cripple any reasonable filesystem tools that see them all in one “directory.”) However, even that doesn’t work reliably, as s3fs demonstrated today when it went into “write-only mode” such that I could create files locally that would show up on S3 but that subsequently would disappear from my local filesystem. WTF?!?

The unfortunate answer is: S3 is not a filesystem, and it was created by people who are smarter than you, and who have very craftily calculated that if you are forced to weave in the S3 API and its limitations into your application code, you will have a damn hard time ripping it out of your infrastructure, and so they are going to have you do just that. They do not want it to be used as a filesystem, and so guess what: you are *not* going to use it as a filesystem. Not gonna happen.

Say what you will, but our hometown heroes here in Seattle are no dummies. Embrace, extend, extinguish, baby. Not just for OS companies anymore…

(Yes, I know that s3fs is not an Amazon project. But it appears to be the community’s best attempt to put filesystem semantics around S3, and that attempt has been rejected by AWS’s immune system.)

MySQL silently ignores aggregate query errors by default

In a SQL query, if you use aggregate functions (min, max, count, sum, etc.) and mix them with non-aggregate columns, you have to indicate how to “group” things. Otherwise, the output is not predictable.

MySQL by default will just ignore these problems and make up something. This can make bugs in complex queries hard to track down (and it virtually guarantees that a novice or dullard will slip some errors into such queries eventually).

You can fix this with:

SET SQL_MODE=(SELECT CONCAT(@@sql_mode,’,ONLY_FULL_GROUP_BY’));

(That is, you want the “ONLY_FULL_GROUP_BY” option set. The SET above can be run in the mysql> prompt and affects that session only; thinking DBAs should strongly consider enforcing this as a server option.)

I am too tired and busy to give in to the temptation to unleash a rant about MySQL here, but PLEEEEZ. It’s the year 2013 and this is still an issue??

Hat tip Michael McLaughlin: http://blog.mclaughlinsoftware.com/2010/03/10/mysql-standard-group-by/

MySQL docs on this “extension” http://dev.mysql.com/doc/refman/5.1/en/group-by-extensions.html

Force a reference to System.Core in Visual Studio 2010

There are reasons why you might need to add a reference to “System.Core” to your Microsoft Visual Studio project. (For example, if you wish to compile/build both inside the IDE and from the command-line with MSBuild.exe.)

However, if you try to do this through the IDE, it will barf at you: “A reference to ‘System.Core’ could not be added. This component is already automatically referenced by the build system.”

Alas, that’s a big fucking lie. It’s referenced by the IDE when it invokes the build system, but not by MSBuild itself. So sometimes, you indeed must add such a reference, but you can’t do it from IDE-land. So close VS2010 and fire up vim. Add a line to the .csproj file in question, telling it to:

<Reference Include="System.Core" />

Hat tip to Ashby at StackOverflow:
http://stackoverflow.com/questions/1302488/the-type-or-namespace-name-linq-does-not-exist-in-the-namespace-system-data/4331322#4331322

I got to this point when recompiling something for .NET 3.5 that had originally been coded/built for .NET 4.0; it wouldn’t import the LINQ namespace without being told to import System.Linq, but then couldn’t find it without the System.Core reference being made explicit. Arrgh.

Why is Port 21 apparently open on my firewall?

Scenario: You set up a server somewhere on the public Internet.  You lock down its ports to the minimal subset you need using firewall(s).  Yet, somehow and for some strange reason, nmap reports that port 21 (FTP) is open on your server!  Sure enough, you do a “telnet myhost.cxm 21” and it connects!  Shit-damn, what’s going on??

Don’t bang your head against your iptables or pf or PIX or ASA config.  First, check to make sure that the environment you’re checking from behaves right.  Do a “telnet google.com 21” and see if it connects.

Some NAT setups in offices apparently try to do some stateful inspection of outbound active FTP in order to rewrite the addresses/ports involved, and these can intercept outbound requests on port 21, making it seem like any host is picking up on that port.

Try nmap’ing or telnet’ing from an outside host directly connected to the public internet.  And make sure that your subsequent security scans/checks come from such a host.

secedit for setting security policy in windows server 2008 r2 server core

Concepts:

There is a policy running on the system. There may be one or more databases in .sdb files which are files representing a possible policy that could be run. These are stored in c:\windows\security\database\. One may “export” a policy out to a configuration .ini file (the docs for secedit say .inf, but it is clearly the venerable .ini format), which policy comes either from a database .sdb file, or from the current running system policy (if no policy is specified when running secedit.exe on the command line).  One may edit this configuration .ini file (the docs ambiguously call it a “security template” as well, but the command line options all say “cfg”).  You then create a new security database .sdb file with the “import” syntax.  Contrary to a lot of stuff on the Web, you don’t need to put it into some particular magic database (but see below for path gotchas) like the original secedit.sdb; put it in a new one.  Once you have a new, valid, legit database .sdb file, you only then use “configure” to apply the database file to the current system.

Lots of gotchas here.

Sometimes when secedit.exe fails, it is silent, like a good UNIX program, but it will return an %ERRORLEVEL% so check that or you will be bamboozled.  It is noisy when it succeeds and sometimes even noisier when it fails (except when it’s silent).

Secedit silently failed in many confusing ways if either the security database .sdb or the configuration .ini was located on my z: drive, which in this case was a VMware shared folder on a Mac OS X system.  Move stuff to a C: temp dir, then clean up afterwards, because hey, writing xcopy lines in batch files is fun.

The configuration .ini files are in full on, utf-16 format.  Two fucking bytes per character.  Nice.

If you try to create a brand-new configuration .ini file without reference to anything, you do not get a listing of the default settings, but rather an unhelpful, nearly-empty file that informs you that Description=Default Security Settings. (Windows Server).

If you try to look up the values for the various sections, you just plain can’t get them anywhere.  If you go to this horrible javascript monstrosity of a reference site and click Technical Reference (for Windows Server Group Policy), you can get a giagantic, unstructured spreadsheet that has relatively lengthy (but non-technical) prose about the various settings.  But it won’t tell you which sections they go under in the .ini file, nor the requirements for each field, some of which are registry settings and some of which are not.  If you want to see something that was apparently done as a class project by a charity course taught for retarded non-native English speaking remedial computer science students by the generous and distinguished engineers of Microsoft (who are clearly working on better and more interesting problems than, oh, say, making sure the OS’s core security API is sensical and internally consistent), you can look at this horrifying jumble of shittiness, which can’t search for shit but mollifies you with funny Engrish when it fails (I am not kidding: “We probably hit search limit.  Try to redefine your search string.”) and as a bonus demonstrates the sloth of Azure serving AJAX (because, you know, actually putting the documents into an html page where anyone’s browser “find” function could speedily search for it would lack the pedagogical value to the retard-children-programmers).

You can maybe stay sane if you learn that the registry values are prepended by an integer and a comma, where that int seems to specify the data type of the reg value (4=integer, 7=text, 1=some other kind of text).

Specifically in the [Event Audit] section, there are values that are not registry values.  They are ints that appear to be bitmask math fanciness.  (Remember setting options on visual basic windows back in the ’90s, where you got to add up powers of 2? ).  It so happens that they all have two bits, the first one being “log successes” and the second being “log failures.”  So 0 is neither, 1 is successes, 2 is failures, 3 is both.  But this isn’t, as far as I can tell, anywhere on MSFT’s site and it’s sure as fuck not in the giant unstructured spreadsheet reference.

Much of the configuration .ini file can be omitted (so you can just overlay the parts you want).  But you MUST include the [Version] and [Unicode] sections or it will barf.  Use secedit /validate to check it.  However, “validate” does not mean that it will actually round-trip and work right; it doesn’t check the security identifiers in the [Privilege Rights] section so it’s quite possible to have a valid cock-up (see round-trip gotcha below).

Biggest one: secedit CANNOT ROUND-TRIP. The security policy “export” may will (and does for me) produce an output with entries in the [Privilege Rights] section that refer to “Classic .NET AppPool” among others.  If you try to import and configure with this, you’ll get “No mapping between account names and security IDs was done” in the error log.  Turns out you have to manually fix this by adding “IIS AppPool\” before the names of these AppPool entities.  (Hat tip)  If you want to actually find out whether that, or some other hackery, fixes it to something that can be mapped to an SID, find yourself the PSTools download and test the name with PSGetSID.exe.  Awesome.

The “configure” option only really needs a .sdb database specified by /db.  If you give it an additional /cfg parameter, it will muddy up the .sdb with the contents of the specified config ini.  There is no benefit to using this, ever, other than skipping a step that could result in you keeping a sanity-preserving intermediate state backup.

The “overwrite” option doesn’t do what you think it does.  Especially with “configure.”  Just don’t use it, unless you are planning on destroying what is in your .sdb file(s).  The .ini configuration already wins in a tie.

The “configure” option is NOT ATOMIC, and it will happily set your system’s security policy partially to be what was in the file you indicated, and partially not (for example, with the broken round-tripping of IIS AppPool names).  There’s no way to find out whether or not the configuration will succeed, except to “suck it and see.”  And once it does partially, non-atomically make a goulash out of the then-current and database-specified settings, there’s no way to tell what succeeded or failed, except by reading the log, which is formatted in an unparseable mess.  Fantastic.

Web.config and App.config file gotchas

If you try to use idiomatic .NET, and you have even modest configurability architecture requirements, you will almost certainly want to use the *.config system (App.config or Web.config). According to old hands at Win32 programming, this is quite a step forward from *.ini files or registry manipulation. Perhaps so.

However, the *.config regime is extraordinarily fragile and surprise-prone once you start trying to do more than just add name/value pairs to the <appSettings> section. The following are some gotchas that I hope you can avoid if you have to deal with this.

.EXE assembies get App.config, and Web DLLs get Web.config, but non-Web DLLs (e.g. tests) get App.config.

.EXE assemblies look for filename.exe.config, which is in App.config format.  Normally, DLLs do NOT get a config file; rather, they inherit / acquire whatever config is in place in their runtime environment.  But there are two important exceptions.  Web services / sites get built as DLLs.  Their execution environment (presumably IIS or the dev server) looks for Web.config and its format.  Test projects (of the MS type that Visual Studio 2010 makes by default) get built as DLLs, as well, but they get run by (mstest? vstudio?) an execution environment that looks for an App.config file.

So, to sum:

  • .EXE => App.config
  • .DLL Web project => Web.config, via its server runtime
  • .DLL Test project => App.config, via the test / IDE runtime
  • .DLL other => none, inherits runtime environment

Sections such as <appSettings> can be externalized into other files, but there are two subtlely different and incompatible ways to do so.

Specifically, you can add a “file” or “configSource” attribute to your appSettings section.  If you use “file,” that file will be read and will override default values that are set in that section in the .config main file.  If you use “configSource,” however, you must not set any values in your .config main file, and instead must entirely scope out that section (and that section alone, save for the XML declaration) in the file whose name you specify.

Frustratingly, “file” and “configSource” have different rules for what may be included (relative / absolute paths, parent directories, etc.).  Especially restrictive are the rules for Web.config, I believe, though I don’t have them straight.  Effectively what this means is that if you have several Web projects that require a shared configuration section, you cannot put your customSection.config in a parent directory and have your projects pull it in (thereby keeping a Single Point of Truth); rather, you have to propagate multiple copies out to all of the sub-Projects (ick).

For more on this: re: configSource, re: file from MSDN.

Web.config settings are mostly inherited from machine.config down through a hierarchy, but confusingly stop being inherited at the sub-directory level in IIS.

Sometimes, or at least most of the time by default, Web.config settings for a given directory are merged with those of parent directories, and are merged with machine-level config as well.  This can lead to somewhat unfortunate results if you have an app in a subdirectory of another app with divergent configruation requirements.  This fellow seems to have figured out how to resolve this.

Be alert, though, because not all settings *do* propagate properly.  First, a parent Web.config can indicate that its settings should not be inherited.  Second, collection settings are merged together, not replaced, by child specifications.  Third, some settings, just seem stubbornly not to propagate (see this MSDN article which suggests that “anonymousIdentification” does not propagate because it is a secret never-properly-set default magical element).  Finally, the above-quoted MSDN article raises the good point that Web.config only applies to ASP.NET stuff, and that there is an entirely different regime for static content and plain old ASP files.  So watch yourself, there, Tex.

iTunes “Sound Enhancer” Considered Harmful

I have now on two occasions, with two separate, quiet background, vocal-heavy songs, noticed significant and highly distracting audio artifacts introduced by the default “Sound Enhancer” on iTunes for Mac (specifically, iTunes 9.2.1 (5) on Mac OS X 10.6.4, on a Quad-Core Mac Pro with an embarrassingly large amount of RAM).  The two songs were “You make me feel so young” sung by Frank Sinatra (from “Songs for Swingin’ Lovers”) and “Do I love you?” sung by Peggy Lee (from “Beauty and the Beat!”).

Both of these problems occurred with CD-ripped highish-bitrate audio tracks (MP3 at 160kbps and AAC at 256kbps/VBR, respsectively).  I originally thought the problem was that iTunes was using a faulty MP3 decoder until I determined that the problem was for AAC as well.  What finally sealed it was using QuickTime Player to listen to the same file and discovering the noise artifact had disappeared.

Compare these two, which I digitally captured using Audio Hijack (trial version; buy it if you like it!):

Version with iTunes “Sound Enhancer” enabled: http://rlucas.net/audio/do_i_love_you-itunes-sound-enhancer.aiff

Version played through Quicktime (no “Enhancer”): http://rlucas.net/audio/do_i_love_you-quicktime.aiff

(P.S., rightsholders don’t even think about flexing your DMCA at me.  These are 20-second audio quality demonstrations for nonprofit, educational and research purposes, and have no negative impact on market value of the works.  You are on notice that any DMCA abuses will be met with bad-faith treble-damage vengeance.)

Notice, on the second “Do I?” that the iTunes version has a big burst of static on the “I,” while the Quicktime version does not.  The artifact on “..so young” is similar, on the “You and I” at 1:57, but I don’t have the time to capture and post that as well.

The morals of this story:

  • TURN OFF “Sound Enhancer” in preferences.  Just do it.  Not worth it.
  • Upgrade your headphones.  Never noticed this until I moved up to some better hardware.  It gets lost in the Apple earbuds.
  • Don’t trust that because a device/program/product has a big following that it does the right thing, at least from a quality perspective.  (Yes, I know, this should be self-evident to anyone who’s been awake since, say, the Industrial Revolution, or since the invention of mass brewing, but give me a break; I’ve been in the grips of Apple fanboydom since 2003 or so.)
  • Trust your ears, debug your equipment, and don’t put up with shit.  I didn’t believe for a good long time that the problem existed at all, much less that it could have been upstream from my headphones, until I debugged it.  Digital audio holds a promise for mankind, damn it, and you’ve got to make it live up to its potential.

RubyGems assorted errors, mutating APIs, and fixes

If you’re trying to use a newer (e.g. 1.3.6) version of the ruby “gem” system to install ruby packages (like rails, rake, etc.) on a legacy system with an older (v. 1.9.0) ruby installed, you might find yourself running into problems like this:

$ gem install rake

ERROR: While executing gem ... (ArgumentError)
illegal access mode rb:ascii-8bit

If you run with debugging flags, you might get a slightly more informative stack trace:

$ gem --debug install rake
Exception `NameError' at /usr/local/lib/site_ruby/1.9/rubygems/command_manager.rb:163 - uninitialized constant
Gem::Commands::InstallCommand
Exception `Gem::LoadError' at /usr/local/lib/site_ruby/1.9/rubygems.rb:778 - Could not find RubyGem test-unit (>= 0)

Exception `Gem::LoadError' at /usr/local/lib/site_ruby/1.9/rubygems.rb:778 - Could not find RubyGem sources (>
0.0.1)

Exception `ArgumentError' at /usr/local/lib/site_ruby/1.9/rubygems/format.rb:50 - illegal access mode rb:ascii-8bit
ERROR: While executing gem ... (ArgumentError)
illegal access mode rb:ascii-8bit
/usr/local/lib/site_ruby/1.9/rubygems/format.rb:50:in `initialize'
/usr/local/lib/site_ruby/1.9/rubygems/format.rb:50:in `Gem::Format#from_file_by_path'
/usr/local/lib/site_ruby/1.9/rubygems/installer.rb:118:in `initialize'
/usr/local/lib/site_ruby/1.9/rubygems/dependency_installer.rb:257:in `Gem::DependencyInstaller#install'
/usr/local/lib/site_ruby/1.9/rubygems/dependency_installer.rb:240:in `Gem::DependencyInstaller#install'
/usr/local/lib/site_ruby/1.9/rubygems/commands/install_command.rb:119:in `execute'
/usr/local/lib/site_ruby/1.9/rubygems/commands/install_command.rb:116:in `execute'
/usr/local/lib/site_ruby/1.9/rubygems/command.rb:258:in `Gem::Command#invoke'
/usr/local/lib/site_ruby/1.9/rubygems/command_manager.rb:134:in `process_args'
/usr/local/lib/site_ruby/1.9/rubygems/command_manager.rb:104:in `Gem::CommandManager#run'
/usr/local/lib/site_ruby/1.9/rubygems/gem_runner.rb:58:in `Gem::GemRunner#run'
/usr/bin/gem:21

Ignore the first couple exceptions and focus on the ArgumentError right before the stack trace. What you’re seeing there is a use of a syntax for defining the binary read encoding mode for reading in a file, but it’s a syntax that didn’t make it into ruby core until ~ version 1.9.1.

However, the relevant part of rubygems/format.rb that checks for RUBY_VERSION to determine what syntax to use simply checks for RUBY_VERSION > ‘1.9’.

If you patch that to check for RUBY_VERSION > ‘1.9.0’ you’ll make some progress, but you’ll get stuck again with a similar error:

/usr/local/lib/site_ruby/1.9/rubygems/source_index.rb:91:in `IO#read': can't convert Hash into Integer (TypeError)

Although this one looks quite different, it’s the same effect at play: rubygems/source_index.rb checks for RUBY_VERSION < '1.9', when it really ought to check for RUBY_VERSION < '1.9.1' when it uses a 1.9.1+ specific API (specifying the encoding in IO#read). I've added bugs at RubyForge: [#28154] Gem.binary_mode version test for Ruby 1.9 sets invalid rb:ascii-8bit mode and [#28155] source_index.rb uses 1.9.1 IO#read API under RUBY_VERSION 1.9.0; other 1.9.0 issues Hot-patching may be required if you find yourself needing to get gem 1.3.6 working under Ruby 1.9.0. (For example, this is the Ruby 1.9 that comes packaged with Debian 4.0.) If so, it should be safe to make the changes I've indicated above. I'm hesitating to provide a patch file as I am not certain that I've got this 100% right; YMMV. Thank goodness for open source. That said, WTF is a core API item like IO#read doing changing between point versions, without it being loudly obvious in the docs??

Concentrating in History and Science at Harvard

This post was originally at my Harvard Law / Berkman Center blog, but has since been removed. Though it is certainly outdated with respect to the current practices and personalities of the History of Science department, it still may hold some value for prospective students. I originally wrote it as advice to my younger self, and I think it might be particularly useful to undergraduates who share some biases, conceits, and preferences with that past self.

What you should know before undertaking an undergraduate concentration in History and Science at Harvard

Thu, May 6, 2004; by Randall Lucas.

Draft, 06 May 2004

Synopsis

Having completed an A.B. in History and Science, I wish I’d known the following before beginning:

1.You will not receive a thorough background in the history of science before being assigned work presupposing such knowledge. You must undertake to educate yourself with a foundational knowledge if you wish fully to understand and gain from the coursework.

2.You will be in a class with a large contingent of pre-med students who bring a potentially undesirable ethic to the department. You should carefully consider whether you want these people to be your peers for three years.

3.The influence of identity politics and philosophies hostile to science will be significant and may emphasize criticism and polemic. You should understand the “Science Wars” and their implications before you choose the concentration.

4.The department’s faculty is brilliant, but the ideology and personality of individual persons with whom you work will inordinately influence your learning, enjoyment, and grades. You must research the individuals within the department with whom you consider working.

5.MIT is available to you and has a science studies program with excellent faculty. You should seriously consider the MIT offerings for adding breadth and diversity to your coursework.

6.The History of Science Department is in the Science Center, the most dismal place known to man. You must convince your colleagues to flee that decrepit building at every chance.

Introduction

I chose to concentrate in History and Science as an undergraduate (class of 2001) because of my interest in both fields of inquiry and in their intersection. Despite an intense interest, however, I became fundamentally unsatisfied with and alienated by my experience in the department, and sought out other areas in which to focus my energy, causing my grades and level of achievement in my concentration to suffer. Looking back on reasons for this dissatisfaction, and seeking to help others avoid it, I have identified the caveats that follow.

The following points will likely be useful mainly to that minority of students who share certain of my preferences; there are certainly many people who enjoy and thrive on the status quo. However, if the numbered synopsis resonates with you, the entire article is likely worth reading.

While the intended audience of this paper is students considering, or struggling with, a concentration in History and Science, I hope it may be useful to those seeking to reform the department, or indeed, undergraduate education in general. Such readers should note that I take full personal responsibility for my past performance within the department while remaining critical of the defects I describe below.

Discussion

1. You will not receive a thorough background in the philosophy of science, nor in the historiography of science, nor in the actual history of science. Although a sophomore tutorial includes selected works relevant to all three of those disciplines, it provides a solid grounding in none of them. The remainder of the curriculum tends toward rather specific inquiries into the specialty fields of faculty members.

The History of Science Department (HSD) at Harvard is a meeting place of a number of overlapping subject areas (“related disciplines”) that might at another school be called Science Studies. The natural focus is on the history of science, but both critiques of how the history of science has been written in the past (the historiography of science) and studies relevant to the nature of scientific knowledge and its place in the world of knowledge (the philosophy of science) go on alongside and mixed in with the historical studies. These three areas are largely interrelated in practice, and some synthesis of these is usually expected in students’ work.

However, most courses offered in the HSD analyze a relatively narrow piece of history that a faculty member has staked out, organized typically along the lines of geography, time, and / or scientific discipline. These courses are excellent for “drilling-down” into the subject matter given, can be intellectually satisfying in their depth, and often make points of departure for senior theses. However, with regard to the three related disciplines (history, historiography, and philosophy), these drill-down courses offer only a “case study” approach and do not provide any systematic way to cover the related disciplines.

The sophomore tutorial (as I observed it in 1999) seems nominally to be addressed towards this problem of laying the foundation in the related disciplines. However, it suffers from an even worse defect in this mission than does a drill-down course: in effect, the tutorial comprises a number of micro-case studies, thereby sacrificing the systematicity of a survey, yet covers each so superficially as to lose the benefits of in-depth study. The sophomore tutorial is centered around a series of reading selections in all three disciplines, typically with three or so selections (often, but not always, related) to be assigned and discussed in a week. Many of these are excerpts or single examples of primary sources (e.g., Copernicus, Sarton, Feyerabend, in history, historiography, and philosophy, respectively) and are presented as exercises in “close reading.” Such a series of assignments, without an underlying framework and system, is a recipe for a patchwork understanding.

The problem is that close reading and criticism, which are the focus of the later drill-down courses where it is appropriate to focus on a narrow field of inquiry, is an inappropriate starting point. The tutorial fails to give a general and comprehensive background in what, overall, Copernicus, Sarton, or Feyerabend did and said, and how that relates to the other major figures in those fields. What is desperately needed by the underclassman is a solid foundation in the disciplines themselves, not in “close reading” or criticizing the foundational knowledge. How can one be expected to say anything of worth about a closely read page on the philosophy of science, when the sum of one’s instruction in the philosophy of science has been three or four such closely read articles? How can a primary source reading from an ancient astronomer be helpful when one has not mastered at least the narrative history of that astronomer, his forebears and successors, and his work’s relation to that which went before and after?

Only a remarkably arrogant blend of sophistry and ignorance permits the sophomore who has only the tutorial as his guide to speak or write critically on the contentious ideas of, for example, Feyerabend, yet that is exactly what we ask him to do. Harvard undergraduates may be famous for their arrogance and sophistry, but we need not countenance ignorance.

Another problem with prematurely engaging in close, critical readings of primary sources is that, like in many academic fields, there is a lot of cultural context to the practice of History of Science. To produce work that engages the scholarship in the field surrounding it, one must first learn and apply the conventions and tradecraft of History of Science practitioners. Ideally, this, too would be formally taught in a systematic manner, but if it must be absorbed implicitly rather than explicitly, it can be gained effectively only from secondary sources, not primary sources.

I do not claim that my (professedly inadequate) History of Science education qualifies me to prescribe a detailed syllabus, but I can propose a very general prototype that would at least be superior to the current state of affairs. I propose that the sophomore tutorial be replaced with two semesters: first, a roughly chronological survey of science from antiquity to mid-20th century, and second, a full semester on the historiography and philosophy of science, covering Greek thought as a basis and beginning seriously with the renaissance, up through the “Science Wars” of the 90s (see below). The historical survey will not aim to criticize or challenge the “received history” — precisely because the “received history” must first be presented and known before it can thoughtfully be challenged. The second semester of historiography and philosophy will begin the questioning process in a systematic manner, asking the foundational questions such as: What is science, anyhow? How is it different from other forms of knowledge? Who has heretofore written the history of science, and with what biases, and what criticisms have been directed at the historical record? In both of these courses, focus will be away from the performance of actual historical research or philosophical criticism, and directed instead toward a broad, systematic understanding of the received history, the major schools of thought, and the historiographic issues at hand. The student will have two more years to engage these ideas in the performance of research and criticism, but will be much better prepared to do so.

(The inclusion of “Greek thought as a basis” is not meant to exclude the possibility of augmenting or replacing it with other times’ or cultures’ contributions. However, I reject categorically the suggestion that no foundation is proper because it may be biased. In other words, should it be determined that Arab science is the best starting place from which to build a foundational understanding for undergraduate history of science education, then make it so — but such a foundation must be built.)

This idea — that there is at least a general “canon” that one ought to know and that schools should try to cover systematically — has been out of favor for many years. President Summers is demanding of the faculty a reexamination of this issue, which reexamination is in order and indeed overdue.

In the event that you decide to concentrate in History and Science, and the department has not yet adopted measures similar to those I describe above, I highly recommend that you take measures to broaden and systematize your history and philosophy of science education before you enter the sophomore tutorial. Examine reading lists of science studies courses from other universities, acquire some of their recommended general texts, and read these prior to your sophomore year. If you can manage to read a good selection of secondary sources that aim to survey the field prior to your sophomore tutorial, you will be much better able to understand and put to use the ?close reading? materials that the remainder of your course of study in the HSD will comprise.

2. You will be in a class with a large contingent of pre-med students who are seeking a way to combine their pre-med requirements with a humanities degree. These people will bring a potentially undesirable ethic to the department that it likely would not otherwise have.

The tenor of your HSD experience will be substantially influenced by the pre-med ethic. The following are attitudes characteristic of this ethic, and you may find them undesirable:

– Grades are of utmost importance, and certainly are of more importance than developing deep understanding or doing good work.
– Challenging of professors, instructors, or received ideas is virtually unknown (note that I exclude instructor-assigned “challenges” to straw-man notions of received ideas).
– Candor in discussions is unknown, replaced instead by the extremes of “flexing” or meek acquiescence.

If you are pre-med, you may want to reconsider surrounding yourself with so many folks on a similar path (remember, if your plan works out, you’ll be spending many, many years with those people anyhow). Furthermore, athletes should take special note that the HSD is not one of the traditionally athlete-friendly departments, and your colleagues and instructors may be nonplussed at the notion of rescheduling for practices or meets.

3. The influence of identity politics (gender, race, class) will be substantial, as will the influence of philosophies that count science as epistemologically non-exceptional. One result is that criticism and polemic will be prominent and may be emphasized over mastery.

You will likely be expected to include substantial criticism of gender, race, class, and similar issues in some of your work in the department, both implicitly by the nature of the coursework and the interests of your instructors, and explicitly by your coursework. To be sure: often, gender, race, and class considerations can be meaningful, helpful, and interesting to a thorough analysis. However, focusing on these issues tends to emphasize the polemical, and is often orthogonal to the main inquiry. If your interest is in the history of science or philosophy of science, rather than historical criticism or philosophical criticism of and through science, you may not appreciate that shift. If you are working with people (instructors, or classmates, in the case of discussion sections) who are particularly interested in this type of work, you might find the emphasis placed on these issues to be frustrating.

By way of introduction to the second part of this caveat (what I call scientific non-exceptionalism), let me explain very briefly and very generally a phenomenon known as “Science Wars.” Over the better part of the twentieth century, an intellectual current that might be described as a backlash to the notion of the triumphant march of technological progress came into fashion among academics, especially on the left. Work in this vein tends to be critical of scientific authority as an instrument of power, and tends to share methods and language with Marxism, feminism, and literary criticism. Let us call this side “group A.” Group A, along with raising legitimate concerns, tended to attack a straw-man version of “Science” due to their generally not understanding it very rigorously. At the same time, a “group B” took to what it imagined was a defense of science. Predictably, group B made some good points but largely argued right past group A, attacking a straw-man version of “anti-Science” because group B didn’t really understand the specialized critical language of group A. Group A imagined that group B was all about the Manhattan Project, eugenics, and Star Wars, while group B imagined that group A was in favor of astrology, having polio, and replacing Newton with a carved wooden fetish. Things got sillier and sillier until the mid nineties when a physicist named Sokal duped some second-stringer academics into publishing a goofy hoax about how physics was just a matter of subjective interpretation in an attempt to discredit group A. This radicalized folks, who either got angry about Sokal’s deceit or saw it as a debunking of group A.

“Science Wars” continues to this day, although it seems to be somewhat toned down from its peak. The main point of contention between the belligerents is whether science is exceptional; that is, whether science provides a way of knowing or a route to knowledge that is, like revelation is to the faithful, privileged and specially trustworthy. (Full disclosure: as the reader may have surmised, I believe in a limited scientific exceptionalism, but not without noting that, to paraphrase Derrida on philosophy, science appropriates for itself the dialog or process that defines it [e.g., if a supernatural phenomenon such as ESP is observed and documented and measured, it will come to be considered no longer “supernatural” but a scientifically explained phenomenon].)

While it is necessary not to conflate historical with modern science while working on a purely historical topic, the question of exceptionalism will arise in any consideration of modern or historical philosophy of science. In the HSD, if your work relates to the place of science in an epistemology, you will likely be expected to hedge your arguments with reference to the non-exceptionalist ideas. Especially for those of you who engage seriously both with the philosophical question of the place of science and with your science coursework, you may find this expectation troublesome.

Like the presence of the pre-meds, this is more a cultural issue than a structural one, so these are less recommendation for specific changes than advance warnings to students about cultural issues to expect. In fact, I would not recommend any specific changes which would require coercion of persons on these dug-in and contentious topics. However, prospective students should be alert that the “Science Wars” are still smoldering, and that the environment that creates may be unpleasant.

4. The department’s faculty is packed with some of the most brilliant minds you can imagine, but you won’t pick the right courses (that is, the right people) based on the descriptions in Courses of Instruction. Therefore, you must research the individual persons with whom you wish to work.

This point in fact probably extends to all of Harvard. Alas, you are cursed with riches: the HSD has some truly brilliant and fascinating minds at work in its faculty, but you can only work with a few of them in your allotted time. What’s worse, the conventional course selection method is to read a paragraph, perhaps also read the CUE guide, shop a course (in the HSD, often one weekly meeting) and then commit to spend a whole semester on it. This is very risky, and you need to mitigate this risk by researching the individual style, interests, and ideology of the prospective instructors.

The HSD courses are intensely dependent on the personality and interests of the faculty instructors. It is very important for your enjoyment and for what you will ultimately take away from the concentration that you very carefully select the faculty with whom you work. I recommend reading as much as possible of the works of, and about the ideology of, any faculty member whose course you intend on taking.

By an instructor’s ideology, I mean his more or less comprehensive system of ideas about the topic, which can include the critical and polemical ideas I discuss above, but also includes his value judgments on what is interesting (and what is deathly boring) and what is good quality. The paragraph in Courses of Instruction will but hint at this ideology, and your first shopping period section will reveal little more. Your departmental advisors, such as they are, may be helpful for this — I found the departmental advisors of my day to be more of functionaries for processing undergraduates’ papers en masse than legitimately useful. You really must read about your instructors, including their relevant published work on the topic at hand, in order to decide if you want to work with them.

Unlike much of the rest of your education, this will not be a science course, and mental horsepower alone will not see you through: there are no objective exams, and you must be aligned with the ideology and personality of the professor in order that his instruction should prove to you satisfactory and that your work should prove likewise to him.

Although carrying out this recommendation takes time, consider: you are investing a lot in this degree, and you must do your due diligence before investing your precious time in an instructor. Better to lose a few hours in preliminary research than to lose a semester.

5. MIT is available to you and has a science studies program with excellent faculty.

The Science, Technology, and Society program at MIT is a mere two T stops away by train, but a world apart in terms of certain attitudes. If you find yourself dissatisfied with the HSD, or if you are having any hesitations about limiting your courses to those offered in the department, at the very least examine the MIT offerings. Although I discovered them late in my undergraduate career, the MIT courses were a wonderfully refreshing change of attitude.

6. The History of Science Department is in the Science Center, the most dismal place known to man.

Unfinished concrete, exposed ducts and wiring, cheap carpets and furniture, and radical, discomforting swings in temperature will be the hallmarks of your physical experience in the building housing the department. Happily, you are not require to spend much time there, and reasonable measures can be taken to mitigate the time that you must spend there. If your instructors or classmates propose to hold meetings in the Science Center, make every effort to dissuade them. There is plenty of decent real estate on and around campus, and unpleasant physical surroundings are not conducive to good learning.

Conclusion

I understand that this document jumps from broad theoretical issues to narrow practical ones and therefore makes for a poor plan of action for reform of the department. Such a plan is not my objective. Of utmost importance is enabling good choices by undergraduates, and offering concrete strategies for dealing with existing deficiencies.

If any of these points resonate with you, I urge you to consider my recommendations, and to contribute any ideas or strategies you have for dealing with the problems of the department, by way of publishing your own suggestions or by commenting on an interactive electronic version of this document.