This is Signal vs. Noise, a weblog by 37signals about design, business, experience, simplicity, the web, culture, and more. Established 1999 in Chicago. Follow us on Twitter for more information on our products.

Allan on 37S website. Big baller! :)

A difference of 1.5% sounds like it could be statistical noise.

I second Alex’s opinion. Website visitors are a chaotic crowd, and 1.5% out of a few thousand might just be random.

(But please don’t take my comment as implying that all testing is pointless and/or meaningless.)

Also, in support of the results – placing the preferred action (“sign up!”) to the right makes a lot of sense, as the visitor is more likely to finish “reading” there and do what he’s told (so to say)

Matt, thanks for the mention.

@Alex and Joel, google website optimizer tells you when the results are valid. It’s way more than a few thousand visitors.

hey allan,

maybe DHH will follow you on twitter after this ;)

—iM

@allan, I just presumed you’d say (aka boast about) “tens/hundreds of thousands” if it were so… :)

Anyway, really cool results.

Putting the text on the left and button on the right feels better. But this “improvement” doesn’t really sound statistically significant. Who knows, maybe, those who took the tour are better customers?

@pwb, I don’t think it’s statistically insignificant. Do the math:

According to compete, they get about 15k uniques per month. Let’s lowball it and say it’s 10k/mo, because compete generally overestimates.

If you consider 2% better conversions on 10,000 people all on their base plan of $12/mo, that’s 200 people x $12, which = $2400/mo.

Let’s make an assumption that their traffic growth is 0 and they keep getting 10k visits/month forever. Using that, let’s figure out the extra money they’ll make from June 1, 2009 to June 1, 2010 (roughly):

$2400×12 (june extra subscribers) + $2400×11 (july extra subscribers) + $2400×10 (august extra subcribers) + ... + $2400×1 (may 2010 extra subscribers)

drum roll

= $187,200

Would you like to make an extra $187,200 in a calendar year? This obviously makes assumptions that everyone who signs up sticks around for a year, etc, etc, but I think the general point is clear.

@Roger: You’re not showing whether it’s statistically significant, you’re just figuring out how much more money they would make if the numbers are meaningful.

If you flip a coin 20 times, you might get heads 55% and tails 45%. That doesn’t mean, however, that if you kept flipping you’d still get those same numbers. In fact, you’d tend towards 50/50, assuming you were flipping a fair coin.

They haven’t provided enough data to show whether or not it’s statistically significant.

Correction: They haven’t provided enough information in this blog post. Following the link gives you more data, and it’s been shown in the comments by the requisite analysis that the results are not statistically significant.

@Roger, I agree 1.5% makes a huge difference in turnover, however the conversion is into free trial accounts, which have a conversion rate into paying plans.

Only a percentage of free trials become paying customers in the end.

Also I think their uniques include their existing customer base, so it will not be all new traffic.

Still, the reasoning is valid for all web applications, optimizing sales through testing is one of the most profitable activities for any company selling on the web.

Cheers, Marko

Cool! When I comment on this site the ’’’”Post this Comment” or “Preview”’’’ button and link are placed opposite to what it should have been. Is this intentional? :)

@Tathagata :)

@elan I think the statistical significance (i.e. volume) of their data is outside of the scope of this commentary. What I was arguing is that “2% matters”...not that 2% will always be 2% :)

@Marko Completely agree – I was definitely making some ultra broad assumptions. Let’s say that I’m 90% off…it’s still $18 grand…which would still be a nice paycheck for words on one side of a box vs. the other!

I knew I’d probably get picked at for my comment, but I was just running some broad, assumptive numbers to argue that 2% matters! As mentioned above, if we’re talking about a 5 minute change of putting words on one side of a box vs. the other, I’ll take 2% any day, for any volume. If we’re talking about a 20-week redesign of the system to achieve the 2%, it probably wouldn’t make sense (i.e. you’d need some serious volume to make that pay off).

Does it bother you guys that they kind of bit the whole 37signals design style?

@DR I Quickly looked at the comp and then assumed it was 37signals, I then looked around at the 37signals site for less accounting. It was not until i read your post that i realized it was a different company.