Let me first frame our starting position, to lay the ground for this pragmatic case example of learning by doing, with the case example of our newsletter adventure, and trial and error in a broader context.
We have now got to a point that we have most bigger features built to our core and we have switched more of our focus to updating and iterating many of our existing features, that we had built early on. Along that we have also started evaluating several of our core tools and services that we use.
As a base for server capacity, we are naturally running on Amazon AWS servers and that experience has been great (excluding one hiccup). For a big company, they truly are the innovators in the server hosting space. It’s great to have seen them bring their customer experience focus and mentality from their consumer side, also to their cloud services. With this field, we have no need to change.
But our system for messaging needed upgrading to a more robust solution and we had been eyeing SendGrid for quite some time. At the same time we had another tool related to messaging, namely MailChimp that we were using for our newsletter.
The ultimate solution we wanted to find was to be able to combine these two. A while ago Amazon had also enabled a messaging service and we evaluated that. But we still had that newsletter issue to solve. Looking at it from the cost perspective, one option would have been to run our own newsletter software on our servers, but we really did not want to get yet another software we would need to maintain, as clearly running newsletter software is not our core business.
We also felt that the Amazon email service being new, was not ready enough for us to jump in with, so we started to look deeper into Sendgrid. Also we were thinking, “Well I guess we can run some open source newsletter tool on our servers and let’s see in a year from now if we then can find something better. But let’s switch to SendGrid for messaging anyway.” We also looked at MailChimp if they would also offer sending system emails, but no such luck.
So we opened the trial account and to our surprise there was this little add on called the newsletter now in SendGrid. It may well be that it had been there already for a while, but we really did not notice in any of their communication until we had signed up for the trial account and looked a bit deeper.
At this point we were starting to get really happy with the potential, as for the same cost of just sending out the newsletters with MailChimp, we would actually be able to get a top of the line messaging system with hosted newsletter software. All this for the cost of just the newsletter on MailChimp. Now anyone in biz knows how good that feels (when you care about not just spending money, but using it wisely).
Doing the Actual Switch
It took us a few weeks to get all the details of the switch done on the technical side, but it really wasn’t that hard. Also SendGrid had several other usable features for our needs, as they clearly have been doing this stuff for a while. From the newsletter side we did not really expect much, as we can understand that it’s not part of their core offering, also considering how little they promote it. Our needs were not that demanding anyway, especially as it seemed to have all the basic features that we had also in MailChimp. Possibly MailChimp had some advanced features as well, but we were not really using those.
The basic moving of verified email accounts and so on went smoothly, and building the first newsletter was after the initial learning curve what you would expect. But then we found this one feature for A/B testing the newsletter variations. Being tech nerds, we immediately started testing that and man, is it a cool feature or what!
In all simplicity it allows you to create several variations of your newsletter, this being with variations in the subject line, content, links and what not. Obviously the idea is not to have different content and core messages (although you could if you wanted to), but just to do variations on how to communicate the same things and observe the impact. Then the system sends the first batch with different variations to a specified sample representing the population, and then measures the open rates or what you had selected to track. Once all is said and done, it automatically sends the ‘winning variation’ according to your specification. This was amazing, that it really can be that easy and effective.
The First A/B Test Newsletter
We decided to start light and just do variations of the subject line, that clearly impacts the open rate. Here are the results, as well as the variations used.
As you can see the difference between the highest and the lowest is roughly 40 % in the open rate. Imagine what that means in customer communication, keeping in mind that all people have opted in to hear what you have to say and if instead of 15% you can reach 60% of them?
This big of a difference has a really big impact on anyone. Making sure this is done so easily that there is no reason not to utilize this systematic, data driven method is fantastic. Everyone should be doing this!
We can already see that we will be doing more testing in the future, and most likely also going deeper into testing with content testing as well. We can see this will also help us learn along the way, what ways of communicating our messages are the most appealing and effective. This information we can then use in all of our other communication channels, that being messages in our application, PowerPoints, email footers and the list goes on.
As a side note, for those of you who are just starting to build your venture and do not have thousands of people to send newsletters to, in order to properly A/B test. You can do that on your website using different tools (like Google Analytics) and if you are even earlier along, anyone can do it with Google Adwords. If you’re thinking of what is the best name for the service or the best tagline, you can spend $100 (or use one of those free Google certificates) to get the exact answer, so you can stop speculating.
After discovering this feature, it may be that the same is possible with MailChimp. So regardless of what software or service you are using, make sure to check that out. We will stick with SendGrid for now, as we were finally able to get an “all in one” package for our messaging needs for the price of just the newsletter service with MailChimp.
Join Grow Venture Community
Other posts you may enjoy
About the author
Valto is a co-founder of Grow VC, an entrepreneur who has started several companies across Europe and the US in the web and mobile fields. Forward thinking and always seeking the next big web and mobile success, as an entrepreneur, investor and advisor. Follow Valto on Grow VC, Twitter, LinkedIn, Google+
This entry was posted on Thursday, December 15th, 2011 at 2:36 pm and is filed under Business Education. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
Add New Comment
- Is Crowdfunding Safe Enough For Investors? You Bet
- Updating the Grow VC front page
- Grow VC’s Jouko Ahvenainen voted Top 100 Digital Influencers
- Is Your Website 2013 Ready? Mobile Sites are Taking Off
- Infocomm Investments backed startup DS3 acquired by Gemalto
- Join us for Crowdfunding Investors & Workshop in Boston
- Public-Private Funds for Startups – The India Story
- The Rise of Mobile Banking
- ‘Where the Money Meets the Talent’ Event Recap – Who Said What