We’re constantly working on new features, improvements, and technical upgrades for Basecamp. Many of these changes need to be experienced in the wild to guide their evolution. We need to live with them in our daily use of Basecamp to see whether what seemed like a good idea is actually a good idea.
To this end, we run six different beta servers that all point to the same production database. We’ve found it impossible to accurately evaluate a feature unless it’s being used in anger with real data that actually matters. Evaluating changes against a staging server that’s running an old copy of the database just doesn’t cut it.
The way we book a beta server for use with a certain branch is simple: It’s just in the title of the BCX Campfire room.
When you’re done with a beta server, you change the title to “available”. When you need a beta server, you change it to the name of the running branch.
To select a given beta server, we’ve added a drop-down to all 37signals’ staff accounts within Basecamp itself.
You just pick the server you want to run on and it’ll route you to the beta environment running the branch you’re looking to try out.
Long-term exposure to upcoming features is a great way to get them just right. But it’s also a great way to realize that what you thought was a great idea just wasn’t good enough to ship. We’ve killed many features and changes to Basecamp after living with them for a few weeks. Often times, the most important features are the ones you don’t ship.
David wrote this on Sep 11 2012 There are 22 comments.
Justin Reese 11 Sep 12
“used in anger”
Love this.
Dr Nic Williams 11 Sep 12
Are all DB migrations pushed into master so that DB schema changes on the shared DB are common? Any other specific commits pushed to master early?
Tadas 11 Sep 12
I tried changing subdomain to the ones visible in the picture – they all work. Does it mean everyone can try out beta branches? :)
Josh Hamilton 11 Sep 12
Is each beta server entirely isolated such that it runs its own caches, workers, background processes and whatnot, or do you find that it’s enough to just go with the app servers?
Taylor 11 Sep 12
@Drnic
All beta envs and production share the same prod database. We use a production db in staging to test migrations a few times, then we roll them out in production.
JohnO 11 Sep 12
How do you handle situations where the database needs to change for a new feature?
One of the stickiest parts I’m encountering more and more with using lots of feature branches is the fact that my migration system doesn’t know a thing about source-control branches :/
Taylor 11 Sep 12
@JohnQ
We test them in staging, then we push them to beta / production. In the case of those migrations usually production can run fine with the changes in place.
Ed James 11 Sep 12
We have something very similar, although not quite as sophisticated. We have a single Beta environment which uses our live db, with 6 production servers running our app. So far this has been very useful, although we do find it tricky to test payments (using Spreedly callbacks) and background processes (which can only have a single instance of the process running across all servers) in our Beta environment.
How do you guys approach these problems in your setup?
Arik Jones 11 Sep 12
Any plans to offer a customer-facing beta versions for those interested in the risk? (ala Gmail Labs)
Brian Bravo 11 Sep 12
Thank you for your insightful and brief post. I love how you folks at 37Signals work.
Anonymous 11 Sep 12
This is going to be unpopular.
Dan 12 Sep 12
Does just one person hit the beta server they are “testing” against or do you switch it up so that you each use each other’s features/additions for feedback?
kstcable 12 Sep 12
Maybe is not very popular but thanks your post.
Aurélien 12 Sep 12
Interesting!
The purpose of the testing environment is to protect users & data from incorrect behavior of the new code. Examples: - Data are available to anyone due to malfunction. - Error leading to erase a project when the user wants to delete a file in the project. - Drop of performances on the database for everyone due to a new query.
How do you prevent such events without the dedicated database? Do you raise the level of test before adding the modification? Do you raise the level of monitoring and your capacity of recovering (standby database, automated backout solution etc…) ?
Thanks.
Humidity Meter 12 Sep 12
Thanks for sharing such a useful post with all the readers.
Kim 12 Sep 12
@37signals
This scares the heck out of me.
So production data is in your DEV & TEST environments? Environments which typically have little controls over who has access to it and can view such data.
Jonas 12 Sep 12
@Kim
DEV & TEST is, at least in my view, very much pre-beta stages.
DHH 12 Sep 12
Kim, our beta environments have exactly the same access controls as production does. You can’t see any data you’re not allowed to see because you won’t be able to log into those accounts without knowing user name and password - just like on production.
Bryan 12 Sep 12
Hey Kim,
I think the best way to put it is instead of immediately moving things that are finished, tested and ready to go into production, they publish to a beta server for a period of time.
So, all pending DB changes have been applied as they would for any production roll out, but the UI changes/enhancements are not there yet. This way they can preview how the new UI changes feel before rolling out the front end.
The only time an issue may arise is if a table column was removed that the current production business layer/UI relied on, which I doubt happens very often. 37s is pretty savvy and probably has a way to deal with that as well.
It is a great idea, one that I would like to utilize myself.
Kim 12 Sep 12
@DHH
But don’t 37signals employees have access directly to the database and can view customer data from directly within the database.
That’s what scares me
Calm voice of reason 12 Sep 12
Kim,
Uh, um, yeah…. boy I don’t even know where to start.
Bruno Miranda 13 Sep 12
You guys actually use campfire in the browser? I love the service but without Pyro or Flint it’s pretty easily forgotten behind other tabs.
This discussion is closed.