Headless Shopify: Lessons Learned Building with Gatsby, Part 2

Watch our webinar replay with Gatsby where we go through the entire stack and process

—-

If you haven’t read part 1 of our Headless Shopify experience, then you can do so here.

This is part 2 written by our fearless technical leader, Thomas Slade.

How it all began

Brad and I worked together for years previously at an agency. We launched approximately 50 Magento and Shopify sites between us, resulting in many late nights. Through these efforts,  we’ve always discussed the future of e-commerce, which has lead to us speaking about the latest browser vendor features.

Last year, I took on the daunting task of prototyping a headless Magento. I used ObjectionJS, which acted as a graphql wrapper around Magento’s EAV Database. After evaluating a few different frameworks, I chose Gatsby. Soon after, I had a working static e-commerce site. Of course, it wasn’t pretty, but it was blazing-fast. I sent it to Brad and he was immediately blown away by the speed and near-native feel of the site. You might be thinking, how can a static site power a dynamic shopping experience? What about tiered pricing, discounts, the cart, or a wishlist? Don’t worry, I’ll get to that later in this article.

Fast forward a few months: we started to pitch the idea of a statically generated site as part of a Shopify Plus replatform to one of our clients, StriVectin. Their site had performance issues and needed to be modernized.

I’m going to outline the following in this article:

• Technical Goals of the Project
• Decision-making Process for the Stack
• Top 10 Lessons Learned

Technical goals of the project

The goals of the StriVectin project included many stakeholders, including the client, our engineering team, and on-site experts. We decided on the following three key technical goals for the project:

1. Great UX (User Experience)
2. Great DX (Developer Experience)
3. Static Payloads
4. Migrating away from Magento

These goals helped us choose our technology stack, development process, and deployment strategy.

Great UX

Speed and performance

Your site’s speed is correlated to its conversion rate. It’s important to have intent around every single byte delivered to the client. This includes, but is not limited to:

• Onsite assets like images, HTML, CSS, JavaScript, and fonts
• Offsite assets, which are usually marketing pixels and scripts

Perceived performance vs actual performance

Lighthouse provides a good baseline for performance, but it doesn’t browse the site. It instead measures the performance of a single request. There is an opportunity to capitalize on browsing behavior so we can predict what your users will do next.

Preloading links in the viewport is a modern technique for helping with perceived performance. Requests are preloaded in the background as the user scrolls and links come into the viewport. When a user decides to click a preloaded link, the content is immediately displayed. There is also another benefit to this approach. Why keep sending your users the same content over and over as they browse the website? For example, the header and footer should only ever load a single time when visiting a website. The user should only ever load new assets.

Clean design and UX

Is your design process segmented from you? Is there friction after the design process? The designers should align with your technical goals without sacrificing the user experience.

Great DX

Are you happy with the tools that you use daily?

Our favorite tools

• React: Isolated, declarative components
• TypeScript: Typesafe code, helpful editor autocomplete
• GraphQL: Expressive data query language, intuitive relations, GraphiQL
• Hot Reloading: Immediate feedback when developing

Static Payloads

Reduced server costs

Deploying static payloads allows us to put our code on a CDN (Content Delivery Network). This allows the code to be replicated across zones and get to your users faster. I’ve worked with clients paying $100k/month in server costs for “managed hosting”. A static site can be hosted on a CDN for free or for a few dollars a day.

Security

How do you eliminate a hacker from getting into your servers? Remove the server! Just like the good ol’ days of editing HTML with Dreamweaver and FTPing to GoDaddy. We wanted to deploy the entire website as a single package that included all static assets needed to run the site.

Energy

Why should a backend process a request and give the same response over and over again? Why should a client have to download the same response over and over? We wanted a solution that would make things much more energy-efficient for our user’s devices and for the CDN.

Migrating from Magento

End of life

Magento announced the end of support in 2020 for EE v1.X. The Strivectin codebase has been around for 7+ years. During that time, it passed through many hands, making it unstable and unmaintainable. Who wants to work on a codebase with 20,000 files?

Security

Magento-land has exposed us to many of the vulnerabilities that exist in the real world. Usually, site hacks come from one of the following: 

• Not keeping security patches up-to-date
• Not using web application firewalls
• Not upgrading software
• General lack of secure practices (like shared passwords)

Intuitive content modeling

The content management system (CMS) in Magento is very limited. After a few years, the CMS blocks and pages in Magento turn into a confusing mess of references. We didn’t want Strivectin’s content team to ever edit HTML again. We wanted the process of editing and publishing content to feel natural.

The Stack

Gatsby

Gatsby is a web framework that bundles so many great things. It is fast, provides a pleasant developer experience, and has a thriving community! The Gatsby team is obsessed with performance so that we don’t have to be. Out of the box, it provides:

• Server-Side Rendering at build time with React Hydration
• Link Preloading via Intersection Observer
• Content Mesh with GraphQL
• Image Optimization: Webp support, Sharp for resizing and optimizing
• Lazy Loading
• Hot Reloading

Gatsby also has a robust plugin ecosystem that contains source and transformer plugins. The source plugins integrate with third parties like Shopify or Prismic. The transformer plugins transform data of one type into another. These plugins allow for complex pipelines to create the “content mesh”.

Shopify for checkout and account management

We needed a managed backend to handle the transactional aspects of the site. This includes:

• Offsite User Account Management
• Offsite Checkouts

Shopify Plus’ Storefront API allowed us to handle these transactions.

Storybook

How many times have you had to browse a site and perform multitudes of actions to show a specific state? Storybook provides an interface for viewing components in isolation in many different states. A component is anything that can be extracted out of a design, like a Carousel or Product Card. 

The development team identifies these components, creates the code to power them, and adds stories. These stories are viewable by the team in the Storybook site. Once approved, the components in Storybook are used on the live site!

Storybook also has many add-ons that allow for even more user-friendly stories. 

We used Knobs so that we could toggle data that powers the component. We used Actions so that we could receive updates about state changes. For example, imagine a dropdown input. What if the dropdown is in an error state? What if the dropdown is in a disabled state? What would it look like if the label changed? How do we know when the select is changed?

We also used the Viewport add-on so that we could demo functionality in many different device viewports.

Historically you would receive a report on a project’s progress via page-level delivery. The project plan will show something like:

• Week 1: Homepage
• Week 2: Product page
• Week 3: Product listing page 
• Etc.

This enables many things to fall through the cracks. For example, what data is powering those pages and how is it getting into the system? Are all the third parties included on the page? This results in a lengthy QA and UAT process since things only get 80% completed.

The progress is much more granular when using Storybook. Components that span pages can be reviewed in an isolated context (e.g.,  a Hero Banner used on the Homepage and Product Listing Pages). This allows for consistent component vocabulary and data definition up-front so that the development team can focus on the task at hand. We were halfway through the project before we started modeling any content!

Prismic

Prismic is an API first CMS. The admin interface allows for defining “types”. The “types” are a lot like a simplified version of a database table schema. These “types” are comprised of fields which may be text, rich text, slugs, color pickers, relationships, groups of fields, etc. A “document” is an instance of a type (e.g.,  my type is “Product” and my document is “Blue Dress”).

Gatsby has a source plugin for Prismic that makes all Prismic documents part of the “Content Mesh”.

Other useful features include:

Integration fields

Prismic offers an “Integration Field” that connects with Shopify. This allows for associating documents with Shopify products and basically eliminates the need for fetching data directly from Shopify.

At the time this article is being written, integration fields are currently in beta.

Slices

Slices are reusable content blocks. They allow for content groups to exist across different content types. Storybook and Prismic make a great match since your components define the data you need. After that, all you need to do is to define the schema in Prismic and hook the component up to the document.

Webhooks

Webhooks allow for triggering builds after content updates.

Google Cloud

Our Elevar applications are hosted on Google Cloud, so we wanted to stay within the platform. We were able to defer this decision for quite a long time since we didn’t need a CDN immediately. We decided to use Firebase because of its simplicity and cloud function integration.

The more complex piece of the puzzle is the deployment strategy. We decided to use Google’s Cloud Build for building Gatsby. Our build process has many checks that prevents builds from shipping if they fail. Additionally, we introduced cloud functions for auditing the site with lighthouse and triggering builds.

Through the above efforts, we confidently ship code to production multiple times per day!

The Plan

Deployment Strategy

Top 10 Lessons Learned

Looking back on the project, there were a few areas where we made assumptions and encountered roadblocks. We would like to share those pain points with you so you can learn from what we encountered and make better decisions.

1. Shopify apps without a shopify theme

Most of the Shopify applications that interact with the frontend were not guaranteed to work with our project. This was primarily because many of the apps rely on Shopify’s theming engine or the App proxy. Since we weren’t using most of Shopify’s theme engine, this wouldn’t work.

Many App vendors have started to expose formal APIs, like Recharge, which we used for subscription products. Unfortunately, most Shopify apps rely on the theme.

If your client relies on many Shopify apps that use the theme, then it may not be worth implementing a headless Shopify build. It is still possible and encouraged but will require a rethink of the tooling to accomplish the business goal.

2. Third party + site speed

Third-party code is a critical aspect of any e-commerce site since it can make up the majority of the inbound conversions. Some of the third parties are not just pixels or event tracking but are also responsible for rendering content. The rendering is usually simple, but the libraries and boilerplate used are expensive and repeated for each third party. 

We decided to make a point to understand the cost of each and every marketing script that we added to the codebase. We removed all third party scripts to get a baseline in lighthouse with simulated fast 3G throttling and 4x CPU slowdown. Again, we were purely looking at performance. We then compared the baseline against the codebase with only the added marketing script. The results were stunning! They wouldn’t be as exaggerated on a slow site, but when you are using Gatsby, it shows when running lighthouse.

Adding many third-party scripts can result in a “death by a thousand paper-cuts” situation, where a site has jQuery loaded 5 different times. Many third parties pride themselves on the simplicity of the installation, but this comes at a cost to our end users. Instead of making a custom solution that is performant, they usually go with a solution that will work for most of their clients. Third parties will have to change in order to keep up with the performant, modern web. Shout out to Third Party Web for actively collecting this data!

Keep track of your client’s marketing scripts. Educate them on the actual costs associated with each third party. For example, what is the conversion boost associated with the third party? If it is +0.25%, but the slowness introduced by the third party brings the conversion rate down -0.5%, then the net conversion change is -0.25%. A/B test your third-party scripts to accurately get these numbers.

3. Prismic content validation

Prismic does not offer any validation out of the box. We chose not to have component-level data validation as part of the Gatsby build process. Because of this, our team was required to validate and prevent rendering of invalid content. This caused significant bloat in our code since every single slice and document needed to have pass/fail logic.

Discuss optional fields during the design phase of the project. Also discuss the validation of the components with your clients.

4. Prismic integration fields

Prismic’s integration field joins most of the Shopify data needed for rendering, but leaves out one critical piece of content for creating a checkout. The variant’s global UID is required to create a checkout via the Storefront API. We had to query Shopify in order to get this data, and have brought this up with the Prismic team to investigate.

5. Global product context vs. queries everywhere

Strivectin has a relatively small catalog. We didn’t want to have queries in every single place that a product was rendered (e.g., the wishlist or the cart). We also wanted autocomplete, search, and product filtering to be instantaneous and handled by the client entirely. The dream was to have a truly static site!

React’s context API allows data to be shared to child components without prop drilling. We utilized a global product context that stored minimal product details. Again, our client’s catalog size was small, which enabled us to use this approach.

If your client has a small catalog, this may make things significantly easier. If your client has a large catalog, there is a potential for performance issues due to the number of products correlating with the context size.

6. Shopify theme for non-Gatsby pages

We had to theme certain aspects of Shopify where transactional processes occur like Checkout and Account Management. 

It is possible to handle account creation, login, and checkout purely using the API, but this approach has two major downsides:

1. If you are using some Shopify apps that rely on the Shopify storefront to show data (like Swell does), then it likely makes more sense to keep the account management side of things on the Shopify storefront.
2. Adding account management (and maybe checkout) to the Gatsby side of things introduces personal information, such as names and addresses (and even card numbers if you decide to implement checkout completely custom), which we would rather leave to Shopify to handle and secure.

This forced us to recreate a simplified footer and header in Shopify.

If you rely on parts of the Shopify theme, you’ll need to adjust the theme to match the Gatsby site. Keep in mind header components like cart and wishlist count because you’ll need a cookie to keep these in sync.

7. Bleeding edge tech

Using new software is risky. Thankfully, our dependencies were very active. Part of our development process is keeping our dependencies up-to-date. Renovate bot is a tool that can create Github PRs with updated dependencies on a schedule. This kept our dependencies in-line, though we did run into a few issues. 

We enjoyed using the React hooks API exclusively, but toward the end of the project we came across a few hydration issues…thanks for fixing these, Joe!

Evaluate each and every dependency of your project. Older, inactive dependencies should not be used. Keep things up-to-date so that you have the latest bug fixes, features, and potential performance improvements.

8. Link resolution

Our SEO goals involved custom URLs. Prismic offers a UID field that fit our needs perfectly, but some of our content types were in subpaths (e.g., our blog articles). To accommodate, we had to create a link resolver for Prismic rendering and our component rendering.

We also used gatsby-plugin-catch-links to utilize Gatsby links when anchor elements were injected via Rich text fields.

Understand the SEO requirements of a project up-front. Some clients rely on long-tail urls. Prismic doesn’t allow for subpaths in the urls, so this will need to be programmatic.

9. Storybook and Gatsby Image

For components that had images in Storybook, we needed a way to load Gatsby images. Gatsby images are added via GraphQL fragments when used for real data, but we couldn’t use Gatby’s GraphQL content mesh in our Storybook files. We decided to create a custom Webpack loader for Storybook that allowed us to specify static image files with parameters for customizing, much like GraphQL.

If you are interested in using Gatsby image in Storybook, comment below!

10. No templates, more work (or less)

We started this project without any templates. We had to write search, autocomplete, and layered navigation from scratch, and couldn’t rely on out-of-the-box e-commerce behavior. This makes for a lot of work! For search and autocomplete, we utilized fuzzaldrin. Layered navigation is simple with a global product context.

But there is a benefit to starting from scratch. How many sites have we seen squished into a mold that doesn’t fit, only to end up using 10-15% of the features of the platform? This makes for a bad developer experience and ultimately slows down the site. Below are the normalized code stats from the Magento project vs the Gatsby site:

Not all teams or clients are prepared for the amount of work that goes into starting from scratch.  It can be rewarding, but painful if a “missed requirement” or “wrong assumption” comes out toward the end of the project.

Conclusion

Our site base doesn’t suffer from code rot. Our code and features have intent and purpose!

This makes for easier maintenance and painless deployments. And as you can see below, we deploy to production many times per day:

As browser vendors continue to evolve, we’ll be able to keep up. Our front end is agile and decoupled from our content and transactions, as they should be.

The web is rapidly evolving. The historical way we’ve been accustomed to approach eCommerce builds and platforms continues to change. We’ve all seen this first hand with Shopify’s dominating growth for not only small businesses but enterprise direct to consumer brands that are migrating from monolith platforms. 

Will we use Gatsby, Prismic, and Shopify again?

Yes! Strivectin and Elevar teams are both delighted with the results. The launch was effortless.

Felicity Sissener, the CDO at Strivectin commented:

This has also been the smoothest cut-over of my entire career- your team and work are top notch.

I hope this guide and learning lessons was helpful for your next endeavor.

Interested in learning more about Headless? Let us know!

• Shopify