Case Study: Behavior Data Driving Smarter A/B Testing
After a successful migration from Magento to Shopify, and a busy spring season ahead, we took action on data to drive a 15% increase in revenue per visitor
After helping a client that sells baseball and softball sporting goods successfully migrate from Magento to Shopify we went to work tagging the site to help understand the nuances of user behavior.
Step 1: Implement Behavior Analytics Tracking
We used Google Tag Manager (see our GTM tutorial for Shopify) to fire Google Analytics custom event hits including tracking such as:
- Element visibility for key design elements
- Click tracking on events such as product tabs, filters etc
- Custom and calculated metrics
We also implemented Hotjar to provide us with qualitative data to round out our onsite behavior analysis (heatmaps and mouse movement) .
After launch we expected to see some of the new features on the site driving conversions while others wouldn’t pan out like we had hoped. We just weren’t sure which ones though!
Step 2: Use Elevar to Analyze Data
Once we a had a good amount of quantitative and qualitative behavior data to analyze it was time to start looking for conversion optimization opportunities.
We started with Elevar that automated the analysis of thousands of Google Analytics data points for us. We needed insights to start implementing out our a/b testing strategy.
Here is a process we used for one of our tests:
Elevar flagged the product page add to cart ratio and in particular the best selling category of products performed significantly worse than others.
Add to carts is a key area in the purchase funnel (captain obvious speaking here). You work so hard and spent a lot of $$ to get users to the product page. Getting them to take action and add to cart is one step closer to conversion and can unlock retargeting opportunities for those that don’t ultimately convert.
Even a small % increase in add to carts results in big revenue gains.
So – we then looked at the behavior of users on product pages within this category segment.
- How did product detail tab interactions (like clicking to read reviews or viewing size details) affect add to carts and transactions?
- Was the content within these product detail tabs relevant to the category?
We found that product tab interactions like free shipping, price matching, and size details led to higher KPIs. Other tabs weren’t being used and more importantly weren’t driving an improvement in our add to cart metric.
Additionally many products in this category were seasonal releases and didn’t have any reviews. We’ve seen with other clients that this can be a negative confidence builder.
So we are now armed with actionable data to put together our hypothesis and a/b test.
Step 3: Implementing Split Tests
We had our data and were ready to implement this test with Google Optimize.
Our changes were:
- Hiding the reviews detail tab
- Hiding team sales detail tab
- Updated the data within the size guide tab to include a video & custom sizing table that was currently used on a standalone CMS page.
- Removed unnecessary size data that weren’t relevant to these products.
We felt if the user could focus more on the free shipping, price matching, and improved sizing details that it would lead to an improved experience and decision making.
The results for this test?
+15.73% increase in revenue per session
When annualized out over 52 weeks this is a $200,00+ increase in revenue!
One test doesn’t equal ultimate success though. We’re continuing to push for more changes that drive incremental improvements while also learning from our tests that don’t turn out how we hope they do. With every test you’re always learning more about your users.
Interested in our analytics or optimization services? Use the live chat to send me a note or contact us here.