Optimizely: A Culture of Experimentation
Let’s talk about some killer new features Optimizely unveiled at Opticon 2017 – where creativity and science collide. #opticon2017.
We’re going to cover five exciting new features Optimizely announced.
- Support for dynamic websites and apps
- Metrics builder
- Program management and reporting for teams
- Feature management
- Stats accelerator
But first, let’s set the stage…
Our attention spans aren’t what they used to be. Digital distractions and demands upon our time have never been greater.
Experimentation is how the best companies win in the marketplace. From a business standpoint, that means you’ll have to work smarter to compete and thrive.
Testing and iterating should be the first or second step when discussing new ideas. Not an afterthought based on declining revenue. Creating a culture of experimentation was a central theme of Opticon 2017. Proactively delighting your customers, outshining the competition and keeping stakeholders happy can all be within your grasp.
Want further background into how important testing will become? Read “The surprising power of online experiments” Harvard Business Review September 2017. It’s super geeky.
“Experimentation is the next big thing in business management” – Jay Larson, CEO Optimizely
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” – Jeff Bezos, CEO and Founder Amazon
So in a nutshell, we’re gonna have to get past HiPPO and shake thing up.
This is where a tool like Optimizely comes into play. With over 1 million experiments run on this platform and 21% of the Fortune 100 companies in the U.S. as customers, they’re the market leader. A handful of game changing features were unveiled that may radically improve the way your organization brings products to market.
So let’s jump in, starting with the simple stuff and layer on complexity as we go.
1. Support for dynamic websites and single page apps.
With single page websites or apps, you can never be sure which dynamic content will load for a shopper’s second time around. This creates a big problem for the experimenter. Even a simple headline test can require several development collaborations to make it happen.
Now that’s a thing of the past! Optimizely has announced the industry’s first out of the box support for dynamic websites and single page apps regardless of the framework used. Yep, even React and Angular.
For a real world example, let’s explore Rocksbox who wanted to reconsider their brand’s value proposition. To accomplish this, a simple question was proposed.
Did women really want to rent jewelry or did they want a better way to buy jewelry?
Using Optimizely’s visual editor (ensuring brand styles were on point and pixel perfect), three headline variations were tested against the control to see how shoppers reacted.
Part of the experiment setup was defining success. Leading us to Optimizely’s second new feature, the Metrics Builder.
2. Metrics builder
If you want to reach your destination, you’ve gotta have a roadmap! Business is no different.
Setting business objectives tied to KPIs is important. Establishing digital methods of measuring, analyzing and reporting these KPI’s is even more important.
Most important of all, is determining what to do with the results! Meaning, next steps. Optimizely’s new metrics builder helps you accomplish this like never before. In days past, success metrics within Optimizely were more binary.
Did the user click or not?
Did overall revenue increase?
Today, things are different.
Let’s pretend you run an ecommerce business and want to measure experiment impact on things like AOV and quantity of items added to cart. Or perhaps you’re a SaaS company that wants to optimize user flows and LTV.
Metric builder has you covered.
So, which headline performed best for Rocksbox?
All variations beat the control indicating users wanted a try before you buy experience, not just jewelry rentals.
A simple question about what do shoppers really want, posed by Erin Miller, VP of Marketing lead to Rocksbox creating a culture of experimentation that’s at the foundation of everything they do today.
Over 6 months of testing Rocksbox changed their pricing model from $19/mo rental service to $21/mo try before you buy allowing shoppers to apply rental credits towards purchases. That’s the power of experimentation and iterating based on shopper feedback and actual data. After securing a piece of rental jewelry, purchase conversions jumped from 20% to 50% after the changes. Impressive!
Generating, managing and sharing experiment ideas is perhaps the most critical step towards effective testing. The importance of communication between teams can’t be underestimated.
3. Program Management
Imagine a new suite of solutions that will help your company manage experimentation programs throughout the entire lifecycle from ideation to reporting. Meaning forget about copy/pasting from a variety of sources just to get things in motion. That’s where Optimizely’s new teams functionality shines.
Let’s cover four of the stages in more detail.
Ideation: Democratize ideation across your organization.
Collaboration: Bring teams together to collaborate on experiments.
Knowledge Share: Document and share learning and detailed analysis.
Program Reporting: Track success with program level reporting.
Not only is this about improving your testing velocity, it’s about empowering team members to contribute and gain peer approval that motivates successful outcomes. Traditionally only the core testing team are involved in A/B split testing, leaving out the wider organization to contribute ideas and feedback. This can stifle creativity and be a missed opportunity for ‘in plain sight’ ideas that often go unnoticed.
Ideation: So many tools, so little time. Excel, Basecamp, post it notes, Trello and the list goes on.
Forget all of that!
Now you can capture, organize and prioritize ideas through each stage. Of course, full details can be uploaded for anything that supports the idea. Wireframes, analytics screenshots, you name it. By prioritizing your team’s idea backlog, you’ll be able enter the experimentation phase as quickly as possible. That’s important because the more experiments you run, the quicker you’ll learn how to improve.
Your team will be able to score ideas by potential to impact key business metrics on a 4 point scale. Everyone’s input get’s taken into consideration.
Collaboration and Oversight: Your entire organization will be able to get the answer to “what’s being tested now and where?” All steps within the experimentation process can now be shared on the Optimizely platform.
Knowledge capture and reporting
Ideas are great, but what about results reporting? Glad you asked…
Concluded experiments can contain more information than ever before including commenting and collaboration the entire team can see.
At a high level, program reporting reveals qualitative and quantitative information about your experimentation program.
Answering questions like:
“How much are we testing?”
“Which team member’s ideas have generated the most winning tests?”
“What’s our test success rate?”
You’ll see both manually selected winners along with stat’s engine winners.
4. Optimizely X Full Stack
Enable product and engineering teams to go beyond experimentation and control the roll out of new features.
Traditionally, the process to release an app update could take weeks or months.
Design, Build, Launch, Pray.
Optimizely introduced three important tools that will help teams rapidly iterate and mitigate the risks of a bad launch.
Feature Flag – Launch new features silently and enable when you’re ready.
Feature Rollouts – Minimize risk with staged rollouts and immediate rollbacks.
Experiments – Iterate to perfection by optimizing each variable.
Now let’s look at a real world example of how this plays out…
A product manager at the New York Times wanted to see how introducing competition against friends within their crossword puzzle app impacted engagement.
They say competition sparks innovation, so let’s see how NYT introduced this concept into their crossword puzzle app.
Negative messaging was tested against the control.
To accomplish this, NYT pushed an update to the app store that served the new variation to only 25% of users. Meaning only 1 in 4 users would see the new experience.
Positive vs Negative. Hmm, which one will win?
Based on Twitter feedback, it seems like app users were less than impressed! Users felt demoralized and were not having their time of their lives. Whoops!
Now Instead of scrambling to fix the issues over days/weeks and redeploy you can simply rollback to 0% of traffic being served this variation. Don’t throw away features, iterate and improve!
Case in point. Instead of giving up, a new variation was tested. It’s not what you say, it’s how you say it. Consider this new variation below.
Did app users enjoy this experience better? You betcha!
5. Stats Accelerator
Before we jump into our final Optimizely feature recap, here’s why you’ll want to listen. You’ll be able to run more experiments in less time and determine successful tests 3-4X faster. Sounds like a win-win situation to me!
Intrigued? Thought you would be, so here’s what’s new.
Smarter traffic splitting (using machine learning) enables companies to accelerate experimentation and achieve statistical significance up to 300% faster.
Optimizely’s new traffic distribution mode will come in two flavors.
- Accelerate Impact – Perfect for testing headlines or sales tests to find a single winner quickly. This will auto-bucket more users into the best performing variation.
- Accelerate Learning – Helps you understand ALL winners and losers across your entire experiment through machine learning or shall we say (AI) Artificial Intelligence.
Making the shift towards creating a culture of experimentation within your brand will grow your revenue. Admittedly, it’s a bit complicated and takes a fair amount of technical and strategic savvy to accomplish. Here at Copious, we’re fanatical about helping leading brands showcase the stellar customer experience they’ve always dreamed about. It’s what we’re passionate about, so get in touch!