programming is terrible—lessons learned from a life wasted

Come here and work on hard problems… except the ones on our doorstep.

I’m quite lucky. In March I took the time to take a genuine break from my real life, and escape to San Francisco to celebrate my thirtieth orbit around the sun. It was my first time outside of Europe, as well as my first time on American soil.

What follows is a very personal recollection of my culture shock. I wrote it for a much smaller audience, but friends have encouraged me to be a little bit more public about my experience. It isn’t really so much about programming, but one idiot’s view from the epicentre of the startup bubble.

I normally shy away from writing about politics, as I’ve witnessed what normally happens when programmers engage in debate. Besides, if using a computer doesn’t qualify you to espouse about people, what does? After all, technology is social before it is technical.

Last night, someone told me “In California, there isn’t a conflict between being a Capitalist, and a Liberal”, with a wry grin on his face.

It is decidedly weird here. i’m staying in Castro, which is like travelling into the future. Not free from violence or persecution but a far safer and tolerant place than many I’ve encountered. Then off to downtown, stepping over the homeless, weaving between the street corner schizophrenics. After a while, you’ll encounter a faceless industrial building emblazoned with an all too familiar logo.

Inside, once you pass the checkpoint, free food, free beer on taps, somewhere between a coffeeshop and a hackerspace, a bunch of rich people on macbooks with the appropriate stickers. Then back outside to the street to watch people die on your way to a microbrewery. A long drawn out argument about scala as you avoid eye contact with the rest of the world.

The companies here are more than just playgrounds, they’re enclaves. Many people here don’t socialise outside of their work, and when they do, it’s ex-coworkers. As a first time visitor i’m surprised at how isolated many of the people here are. in return for building a social space, the companies enable workers to pour their life into their work, with little time outside of it, beyond sleeping.

I’ve been here for nineteen days now, and it’s still shocking: the disparity between rich and poor. Thing is, those in poorer situations flock here, because they can get healthcare, support, and help, but other times it just feels like a passive aggressive fuck-you-got-mine. if you don’t tip, it isn’t so much a snub, it’s saying “i don’t think you deserve healthcare”. Alternatively for those with healthcare provided, it locks them into their job.

The cost of living is always increasing, and the flashy money from silicon valley is accelerating the gentrification of SF. rent-control is a last ditch effort to prevent those who grew up here from being displaced, but in return prevents them from being able to move within the city.

The dissonance here is enabling: come here, earn money, live in our playground, and don’t mind the poor, they’re better off here than many places in America. At least it’s not so cold that people will die sleeping rough.

I’m not sure if sf is pushing me to radicalism or conformism. It’s a tempting bubble—a hedonistic lifestyle where you can relive your early twenties, assuming you can live with the implicit death penalty for the poor and disadvantaged.

Now I’m back safely in my Scottish rut, I can’t say I’ve escaped the gaping void between rich and poor. It just isn’t so obvious on my doorstep.

A lifetime of terrible code

Although many entries posted here are far from optimistic, and often glib criticism dressed up in technical jargon, they don’t always come from a position of arrogance. It is not that my code is devoid of mistakes, but understanding the mistakes of others has given me a perspective I wouldn’t be able to achieve in one lifetime of terrible code.

I’m yet to make mistakes worth talking about, but still I feel I should be a little more honest about the terrible code I’ve written.

My earliest adventures with BBC Basic, OPL16 and JavaScript were spent reimplementing primitives. Although I was flushed with pride the first time I derived a sort algorithm, in hindsight I don’t think an exponential runtime was worthy of such hubris.

I then moved on to writing terrible perl, often cgi-bin scripts. It wasn’t until I came back to perl years later that I truly understood why I kept seeing HASH0x1729 in my output. I still wrote code for my amusement rather than clarity, leading to such gems as:

my $spoon=fork();

Later, university drilled Java into me, and I was one of the lucky few not to ward off object orientation with judicious use of static. They did fail to shake out stupid variable names or tricks. I once used an XOR swap to wind up my tutor, and the punishment was explaining it to the rest of the class—he sat astonished as I spent the next hour explaining that one line of code. The moral: Giving me an audience didn’t make me any less of a smart-arse.

As I moved from one language to another, the hardest thing was not to pick up the new ideas but to abandon the old ones. My first Python program had all the hallmarks of a Java refugee. Not decomposing my program into objects but simply wrapping up global variables inside of them. My second python program wasn’t too friendly either.

def color(self):
    raise Exception("american")

I’ve even had the honour of having my first piece of ruby mocked on twitter. The errant snippet below can be replaced by send():

response = obj.method(methodname).call(*args)

One of the kinder responses caught me in the act —”That looks like a python programmers first ruby program”. No matter how old I get, my code is still embarrassing. Especially the production code i’ve committed (both in the criminal sense and the version control sense).

# ugh
task = task.task.task

This particular line was eventually refactored away after being written in haste in a life-or-death moment for a project, and rationalised as artefact of laziness or sleep deprivation. The admission of disgust in the comment may not have explained why the code looked bad, but my coworker knew I already felt bad writing it. I was saved by a public shaming by those three letters.

I don’t imagine this will ever change. The moment I stop being embarrassed by my code is likely the moment I stop learning new ideas. Or perhaps the moment I gain engineering discipline, but that’s a long way off.

The biggest change for me as I’ve grown as a programmer has not been how terrible my code looks, it’s knowing when i’m writing it. At least now I know when I’m up to no good, and can weigh up the tradeoffs between beautiful and working code.

HTTP as Imagined versus HTTP as Found

(source: LiveJournal Engineering, frumper 15 on TR Forums)

On the left is a pristine network diagram. On the right is how most networks look in practice. This isn’t to bemoan the death of cable lacing, but an illustration of what Richard Cook calls ‘Systems as Imagined’ versus ‘Systems as found’ (stolen from his excellent talk, How Complex Systems Fail).

With any complex system there is always a conflict between intent and implementation, and with protocols there is a always a mismatch between specification and reality. If you make a specification too small, implementation specific behaviour creeps in, and if you make it too long, no-one has a chance of implementing it correctly. A good specification has to strike a balance between prescriptive to descriptive, explaining how things should work, but begrudgingly admitting how existing implementations behave. HTTP is no exception.

From the outset, HTTP seems like a very simple protocol. You make a request, you get a response back. ‘HTTP as Imagined’ is rather straight forward:

GET / HTTP/1.1<CRLF>
Host: www.example.org<CRLF>
<CRLF>


200 OK<CRLF>
Content-Type: text/plain<CRLF>
Content-Length: 5<CRLF>
<CRLF>
hello

Although RFC 2616 officially defines HTTP, HTTP is also defined by how popular browsers and web servers behave. The RFC is over a decade old, so these behaviours are often discovered through spelunking source code, or nightmare debugging sessions. A robust implementation needs to handle the obscure edge cases of the standard, and the mind boggling way in which others have implemented HTTP. For example:

• Headers can span multiple lines.
• Line terminators are meant to be CRLF, but code should accept a solitary LF.
• Blank lines can appear before the first line of a request.
• Response start lines may only have a code, not a phrase.
• Deflate, gzip are used interchangeably.
• Get messages can have a body, but not every server knows this.
• Some servers will let you get away with the full url in the request line.
• You can’t accurately parse a http response without knowing the method used.
• The length of a response body is indicated by a mixture of the response code, the Transfer-Encoding header, Content-Length header, Connection header (and the request method).

A simple request and response may not be so simple in the wild. ‘HTTP as Found’ may require sick-bags.


GET http://www.example.org/ HTTP/1.1<LF>
Host:<LF>
 www.example.org<CRLF>
Content-Length:0<CRLF>
<LF>


100 Continue<CRLF>
<CRLF>
200<CRLF>
Transfer-Encoding: chunked<CRLF>
<CRLF>
5<CRLF>
hello
0<CRLF>
Content-Type: text/plain<CRLF>
<CRLF>


If this looks bad for HTTP/1,1, at least it isn’t HTTP/0.9. This prehistoric version of HTTP is simpler to parse, in the sense that there are no headers or start line in the response, just the content. Despite being as old as the web, there are situations where a modern robust HTTP server will return such a decrepit response.

GET /a_very_long_url............................ HTTP/1.1<CRLF>
Host: www.example.org<CRLF>
<CRLF>


hello

If you ask for an enormous URL, some servers will only process the first thousand characters of the request, without seeing the HTTP/1.1 at the end. In particular, NGINX will assume it is a HTTP/0.9 request, and then strip the header from the response. Robust browsers will fail to parse the response as HTTP/1 or above, assume it’s HTTP/0.9, and render the entirety of the response as HTML.

(If this disgusts you, don’t look at character set detection)

Thankfully, the newest draft of HTTP captures much the folk knowledge needed to write robust implementations. You may feel dirty, but at least your code works.