70cities blog

I had yesterday off work (mostly anyway) and so spent most of the day alternating between boring-but-necessary chores and catching up on shows I'd bookmarked to watch/listen to. Bliss. In the evening, I headed out for a beer - ironically with people from work, although none of whom I actually work with, so that's OK. It's a big company, that can happen and I rather like it.

On the way to the pub I get suddenly redirected by the invitation to impromptu birthday drinks in town. I'm game, so I head to Covent Garden rather than Shepherd's Bush. It's not a hard choice to make.

I end up in a trendy basement lounge/bar arrangement - it's happy hour, and half full of local suits and fashionistas taking advantage of the half-price cocktails. The music's just one long anonymous boom-chk-boom affair, accurately blared out at the level at which it's impossible to have a reasonable conversation with anyone. I know a fair few people there who I've not seen in ages, and with whom I'd appreciate the chance to catch up with, but there's no chance of that. I sup a mojito, which took half an hour to procure and about three minutes to drink.

T, who's 8 months pregnant, shows up with her SO, who I've not met before. She introduces me as 'the smartest guy I know'. I offer a rebuttal, offering that I'm perhaps the tallest guy she knows. The half-joke gets lost in the din.

Hotpanted girls - I don't know if there's a name for them, or what their job titles could possibly be - float around and proffer test tubes full of God-knows-what. It's too loud to find out, and anyway my carefree vodka jelly days are very much over.

H and L show up, and we attempt to catch up. It's nice to see everyone but it's just not happening. Having made sure I've seen everyone for a hello, I slip quietly out and head home.

I'm hungry, so I scoff the tuna salad in the fridge, and am subsequently taken over by the spirit of Peter Rabbit. Conked out on the sofa in a rum/lettuce coma, the cat sits on my head. Zzzzz.

Well, a seven month hiatus - not bad going, really. 2008 has been the most hideously busy, stupidly packed year yet, and it's not over.

Things I've been up to:

• Designed and built a highly-parallelisable log processing engine, based on Map/Reduce principles
  
• Sweated blood in the process
  
• Been to York for a stag weekend, and ended up spending two nights in a dreadful 80's-themed nightclub
  
• Bought a new suit that doesn't make me look or feel like Tony Blair
  
• Went to Newcastle and wrote off our car
  
• Went to Scarborough for a wedding
  
• Bought a new car (shiny!)
  
• Spent a fantastic week in the North York Moors
  
• Stayed in a posh apartment complex in Leeds
  
• Struggled with being afflicted with recurring chronic sinusitus
  
• Drank a lot of tea/coffee
  
• Bought an iPhone
  
• Obtained new eyesight prescription and face furniture, which allows me to drive at night at last
  
• Waved sister-in-law and boyf off to live in San Francisco
  

So, not much really.

I've been playing with Greasemonkey a bit lately as I wanted to refresh my Javascript/DOM foo. I needed something to build, so I searched in my big red plastic bucket of irritations and knocked up a user script that lets you hide Twitter tweets from a definable list of users without actually unfollowing them - so you can selectively ignore people for periods of time without them being any the wiser - thus avoiding any potential awkwardness. Social networks are delicate things - they're made up of humans, after all.

(This was inspired during SXSWi, when half my Twitter follow list went and I didn't, and the endless noise about it got quite wearing - but I didn't want to unfollow anyone as my Twitter universe is generally quite lovely and interesting).

http://userscripts.org/scripts/show/25365

In common with everyone else with an internet connection this week, I heart Muxtape a lot. Simple, no-nonsense, works. Here's mine, although I've no idea how often I'll ever update it. Needs RSS to track changes, perhaps? Mmm. Then I could plumb it into the uberfeed and whatnot.

In other news, since VD Day passed and I shut down all the Amazon EC2 stuff, I've been mostly working, going to the Isle of Wight, drinking beer, eating nice food, hacking around with a potentially-quite-useful Twitter hack using Greasemonkey and not having my hair cut...

After many hours of toil, Meg and myself opened the doors to this year's By My Anti-Valentine site a couple of weeks ago. Redesigned, revamped, relaunched with the hope that it won't fall over under the strain of the traffic it generates on Valentine's Day itself, as it has always done previously.

Given that my involvement in the project is one of technical architect, developer and sysadmin (Meg is the ideas, UX and design brains of the effort), it's probably worth me explaining a bit about the technology that drives the site and the problems that we came across last year and have tried to solve (or at least mitigate) this year.

By 2007, Meg had been running the VD cards site seasonally for six years, driven by a variety of successive flat-file systems and ultimately a generic PHP-based freeware card sending application (Sendcard, I think) that just about did the job but frequently caused her server to go belly-up once the traffic to the site reached its inevitable peak on the 14th February.

The problem was that the "choose card-send email to recipient" workflow was completely synchronous and linear within the application, so the emails were dispatched to the outgoing email server within the same web-spawned PHP Apache process. Naturally, once a few hundred people at a time were hitting the 'Send Card' button, the host's SMTP service would run out available connections and stop responding. Also, PHP 4's rather, ahem, unpredictable DB connection handling under load would generally explode along similar lines. Cue lots of server errors and general mayhem as the whole thing fell apart.

So, when I stepped in last year and volunteered to make the problems go away with a bespoke web app written in Perl to suit Meg's desired functionality, I made a few sensible design decisions. Over the years I've had to deal with my fair share of scaling problems and integration issues, so I built in a few tricks of the trade.

As it turned out, the site worked nicely for the few weeks before the 14th of February.. until it didn't. As it turned out, the traffic the site got was so enourmous that even moving it to its own dedicated, beefy server (kudos to Pair, Meg's hosting people, for being extraordinarily quick to do this) couldn't make it cope. Even extensive caching couldn't save the server from falling over.

So for 2008, whilst reconsidering our options, I decided to give Amazon EC2 - their Xen virtualisation cloud - a whirl. I'd heard plenty about it and had tinkered with virtualisation at work, but never gone the whole hog, and the cynic in me didn't buy the spiel that spinning up working application instances was as easy as a web service call.. but yes, by gum, it works a treat. Initial setup is a bit fiddly, but tools like the EC2 Firefox UI and S3Fox are a godsend when you're trying to do stuff quickly.

The site runs on a group of replicated server instances running a heavily customised version of Ubuntu 7.10 Gutsy, all identical with the exception of one (which I call the 'master', though not a Doctor Who homage) that also runs MySQL and the card-sending backend app. All the other instances connect to the master database when creating a new card or retrieving one for display. This is a pretty naïve architecture, but it works as the percentage of card senders against the overall site traffic is pretty small (less than 7% of visitors choose to send a card). Therefore MySQL isn't doing very much at all, and I didn't need to spin up a separate MySQL server instance to cope, which reduced the overall cost and was one less server instance to manage.
The rest of the application stack is fairly standard - Apache 2, FastCGI, Perl.

So what happened on the day itself? Well, we had a *tiny* bit of a wobble at around 13:00 GMT, when the east coast of America woke up (55% of users come from the USA), but I coped by spinning up another server instance and offloading some of the remaining site furniture onto Amazon S3 with some 302 redirects - which made a huge difference. Once the traffic started to drop off on the 15th, I was able to decommission three instances and balance the load across the remaining two, which coped admirably.

Paul's Top N Web Scalability Tips

• Never maintain state on your web server. The moment you have to keep a user on a specific application instance for their state to be persisted, you lose all hope of ever scaling the sharp end horizontally. Keep state in your database (or on a shared filesystem if you're a masochist).



• Decouple the web interface from backend systems. You don't send emails straight from a web-spawned process. You just don't. In fact, you shouldn't connect to anything other than a database server. Email servers are strange, unpredictable beasts and they generally need to be handled with care. Don't expose one to a process pipeline that you can't explicitly control (namely, a public web server!).
  
  So, when a card instance is 'created' by a user on the VD site, an email object is created and stashed into a database table that effectively acts as a queue. This gets polled by a cron job at regular intervals and the emails get despatched in series. This means the email server is only ever under a constant load, which can be managed depending on what sort of performance you need.



• Cache everything you can cache. If a page never changes, save it out to a cache directory on your filesystem and have the web server serve that instead of building a dynamic page on every hit. Thankfully the VD site only really needs to invoke the application when someone sends a card or retrieves one from a link they've been sent, so I took advantage of that and had easy cache-building methods that I could call to generate flat HTML pages. ('Proper' setups have caching loadbalancers like ZXTMs in front of their servers that can do this transparently, but of course I don't have that luxury and was trying to KISS). Which all leads me on to...



• mod_rewrite is your friend. I can't rave about mod_rewrite enough. It's easy to get scared by URL rewriting if you're not comfortable with regexes and the general HTTP request handling mechanism in Apache, but it's actually incredibly simple to use with a bit of messing around and most modern web frameworks rely on some form of rewrite engine. The ability to abstract your URL scheme from the physical filesystem brings endless possibilities.
  
  The VD site uses it not only to map its tidy URLs to the application endpoint (what us old-schoolers still call a CGI script), but to rewrite certain URLs to serve flat files from the cache directory but keeping the external URL the same, so turning it off has no effect on the end user. Also, with the flick of a hash character in the global .htaccess file, I can have the all the bandwidth-heavy images served from a CDN - Amazon S3 in this case - rather than the app servers.



• Always use INSERT DELAYED when you don't need the row back - Class::DBI, the Perl ORM engine the VD site uses, isn't very good at this, so I had to roll my own methods here, but it was worthwhile - you save a few vital queries when you're stashing your state away and trying to get back to completing the user's request as quickly as possible.
  
  In retrospect I wouldn't have used Class::DBI in the first place, but that's another story..

Phew! That's it over for another year, then. There's a whole bunch of other stuff I could go into (the joys of dealing with Spamcop..), but that's the story thus far. I'm chuffed at having got through a Valentine's Day without having to watch a single server helplessly grind to a halt under the weight of the traffic burst that a site like BMAV inevitably generates, and incredibly impressed that Amazon EC2 makes virtualisation accessible and affordable for projects like this. EC2 FTW!

Curry Hell man dies

Tragic news. I had the honour of working closely with Mr Latif back in the day when I was a jack-of-all-trades web monkey, and maintained the Rupali and Curry Hell websites. I can honestly say I've never met a man more dedicated to the twin pursuits of selling incredibly hot curry to pissed-up Geordies, and honest-to-goodness self-publicity.

A man so generous, he thought nothing of ordering me a free cab back to the office when a lunch-hour visit turned into a three-hour curry/booze fest - whereupon I fell asleep in the sofa in reception and got woken up by the marketing director of the coincidentally recently-deceased Pennine Windows, who was coming in expecting to update his web site.

He'll be sorely missed. Particularly by anyone who had a lamb phaal at his restaurant in recent days.

RIP Mr Latif. Off to his very own Curry Heaven, festooned with party hats and Cobra beer.

I barely ever play computer games. Consoles, whathaveyou - I've just never had much of an interest in any of it. Back in the 80's, when I was growing up and spent hours slaving over a hot ZX Spectrum, I was far more interested in learning how to write stuff and make it do interesting things than spending hours killing aliens or whathaveyou.

The games I liked then, and still do to this day, are the really simple platform/puzzle type ones. Lunar Jetman, Spellbound, Jet Set Willy, Technician Ted, the Dizzy games, Starquake. My later Atari ST, primarily a games machine much like the Amiga, spend most of its life playing simple platformers. When Driller came out and pioneered the immersive 3D worlds that everyone's used to now, I got bored pretty quickly. Later, when I had a PC, stuff like Rise of the Triads, and Sin gave way to pretty brain-dead stuff like Carmageddon. And even then only briefly. I get bored with games so easily.

In fact, in those rare times I find myself in a gamey mood, I don't reach for my non-existent PS2 or Wii and fire up World of Timewasting or whatever - I use a Spectrum emulator and play Starquake, as I have been for over 20 years (I've only finished it twice - I find the gameplay fun, not the end goal). I just don't get games, really. I just don't care.

Maybe it's the same reason I barely read any fiction or watch any films? Just writing that down makes me sound like some uncultured idiot, I know - but the truth is I get enough from current affairs, technology, comedy and music to keep me occupied for a lifetime and I sort of marvel at those who find the time to glue themselves to an Xbox 360 for hours on end or watch tons of films/boxsets. I can only conclude that I got short-changed somewhere along the line and everyone else in the world gets a thirty-hour day.