Three years ago, I wrote about how improvements in technology keep allowing us to punt on sharding the Basecamp database. This is still true, only more so now.
We’ve grown enormously over the last three years but RAM keeps getting cheaper and FusionIO SSD’s keep getting faster. If anything, it seems like recent advances in SSD technology are accelerating and it’s ever more unlikely that we’ll need to shard Basecamp.
Basecamp remains a perfect candidate for sharding. Isolated accounts, no sharing between them. Yet the cost in increased complexity is constant while the cost of throwing hardware at the problem keeps dropping.
It’s like how some old people have a hard time dealing with inflation and “they want how much for a gallon of milk these days?”. Technologists who grew up when RAM cost $1,000 per megabyte can have a hard time dealing with the luxury of RAM being virtually free (we just bought about a terabyte worth of RAM for a Basecamp Next caching system that cost just around $12,000).
The progress of technology is throwing an ever greater number of optimizations into the “premature evil” bucket never to be seen again.
David wrote this on Jan 27 2012 There are 30 comments.
Anonymous Coward 27 Jan 12
“You kids today, with your hip-hop music and saggy-baggy pants, are lazy. Why, in my day, I had to code the entire internet in 2K of RAM . That’s the way it was and we liked it.”
I much prefer the 21st century.
Paul D 27 Jan 12
It truly is amazing that Michael Moore has not only made the greatest movies ever made, but has single-handedly slved the sharding problem!
Bryan Larsen 27 Jan 12
“Why, in my day, I had to code the entire internet in 2K of RAM . That’s the way it was and we liked it.”
We liked it because it was fun. The coolest project I ever did was a 17-function 40-bit precise motor controller with 4K of ROM and 63 bytes of RAM .
Jason 27 Jan 12
So if you know your investment would pay off, why wouldn’t you do it? You wouldn’t immediately earn your investment back, but you’d amplify your gains while riding the wave of technology.
The Spelling Brigade 27 Jan 12
One ‘r’ in terabyte.
terabyte 27 Jan 12
terabyte
Rudiger 27 Jan 12
That’s Dr. Moore.
Eric 27 Jan 12
The only thing that gives me caution is that your resistance to sharding/distributing keeps putting larger and larger eggs in one basket.
Carl Mercier 27 Jan 12
It would be interesting if you could write about your whole infrastructure, servers, how it all works together, etc.
DHH 27 Jan 12
Eric, you don’t do stuff like this without having backup database servers ready. That would be terrible. So we have hot backups standing ready in case of failure.
Tom 27 Jan 12
Just out of interest, can you share the current size of the Basecamp production database?
Karthik Hariharan 27 Jan 12
Too many of the startups today are pushing their databases to cloud servers, not realizing how much performance they are missing out on. A little hardware goes a long way with a database.
It’d be nice if there were more hosts that seamlessly married cloud/VPS for web servers with dedicated hardware for I/O intensive servers like Database servers. A lot of them claim to support this, but simply aren’t priced right for bootstrapped startups.
Duff OMelia 27 Jan 12
It is pretty excellent that hardware advances have allowed you to punt on the complexity of sharding.
The question I have is around how to easily run database migrations when there’s a ton of data. So if you’re using mysql and you have 20 million rows in your messages table, adding or renaming columns without significant downtime isn’t very cinchy.
I’m thinking that one of the reasons that schema-less eventual consistency db’s like Riak have a bright future is that it makes zero-downtime deployment easier even when there’s a ton of data.
If you’re using mysql, then perhaps effectively using the new ActiveRecord::Base.store in Rails 3.2 will help.
Thanks for the post – I’d love to hear your thoughts on how to keep our mysql schemas malleable over time.
Douglas F Shearer 27 Jan 12
A few years ago your post here mentioned you purchased some Schooner MySQL appliances. Still going with those, or have commodity servers proved to be better in the long run?
Eric 27 Jan 12
But it’s not just the database that can cause trouble. Storage capacity keeps growing, but IOPS /capacity is not and neither are communications links ( internal or external ). Hot standby’s are great, but are no substitute for a full backup which gets harder and harder to squeeze out. Hybrid approaches take the load off of predictable workloads, but can fall over for random workloads.
I’ll presume you have your bases covered, but you are really the exception not the rule. With these problems. With advances in sharded DB tech ( choices and simplicity ) the last few years the “cost” of switching is coming down, not staying the same. At some point it very well be the right choice even if Dr Moore has your mainframe covered.
Joe Ruby 27 Jan 12
“Michael Moore has not only made the worst movies ever made”
FIFY .
Troy 27 Jan 12
Do those beefy servers with super fast SSD ’s make schema changes a non issue as well? How long does it take to alter one of basecamp’s larger tables? I’ve always wondered this, hopefully one day I’ll have an app big enough to find out for myself, but until then hopefully those that are paving the way before us can share. Thanks alot!
Taylor 27 Jan 12
@Troy,
They definitely make schema changes much much faster. It varies depending on load / size of the table obviously. We usually try to make sure as much of the table is cached in memory as possible—again that helps.
Taylor 27 Jan 12
@Doug
Schooner no longer makes / supports their appliances. They were just “custom” IBM servers anyways.
We still use Schooner for one of our databases. Although they’ve developed some great stuff, we’ve run into some problems using their Synchronous replication. We are eagerly awaiting engineering improvements.
Anonymous Coward 27 Jan 12
@Troy I’m another happy FusionIO customer, when I do schema changes we’re now CPU bound while the IO subsystem is more or less idling (MySQL is single threaded on schema changes).
Dan 27 Jan 12
It’s not just sharding. When I was in college just ten years ago and cutting my teeth on C++, grades and a good portion of class time emphasized the importance of memory management. Nowadays with the emergence of garbage collection and cheap memory, this only applies really to mobile applications. Even normal forms, of which I am a fan simply because of the cleanliness of the code, really aren’t as important as they used to be.
I’ll always bet on hardware outpacing software. Well written software that takes into account hardware performance seems to be important only in the short term to get the earliest versions of hardware out the door and usable. But over time, the hardware blows past the software and the design considerations that were once vital are just legacy bullshit (generally speaking).
EH 27 Jan 12
Duff: Smoothing features in database technologies are nice, but downtime should be no taboo when you want to do things right.
Joe Van Dyk 27 Jan 12
Duff—if you use postgresql, you can add a column or rename a column in a 20,000,000 row table in a few milliseconds. (assuming you don’t have to insert new data, of course)
Scott 27 Jan 12
That’s DR. Moore, you hooligans.
Kyle Drake 27 Jan 12
I did a presentation on a game I made at Geoloqi that talks about the Fusion IO drives and why I think local hardware with SSD is the best solution for persistent data stores right now: http://www.slideshare.net/KyleDrake/building-mapattack
Kyle Drake 27 Jan 12
Sorry, it didn’t link: http://www.slideshare.net/KyleDrake/building-mapattack
The one warning I want to provide though is that not all SSDs are created equal: Make sure you get one that writes its cache to the disk on power failure, or you’re going to be in a world of hurt.
Douglas Lee 31 Jan 12
A lot of systems which were built when RAM and SSD were orders of magnitudes more expensive than they are today could be profitably “un-sharded” to simplify their design and cut maintenance/operations costs.
Merle 31 Jan 12
Good design means you’re not tiptoeing on the highwire.
Ed 01 Feb 12
Yes, and they have Memory Cube coming in two to three years time. ( OK may be four to five years ) that will increase the capacity by ten times!!!!
We are possibly looking at a 1U Server with 1TB RAM and Superfast SSD built in!
Yousaf 02 Feb 12
Twitter does a good job at database sharding.
This discussion is closed.