Overhauling error architecture

Problem

• Shopify Shipping, Shopify’s internal shipping solution, enables merchants to send orders to their customers by helping them configure and purchase shipping labels. Using third-party API connections, Shopify Shipping processes and standardizes rules from 20+ shipping carriers, streamlining requirements for shipping addresses, package weight and dimensions, and customs information.

• After a few months on the team, I noticed that new feature releases were not impacting our metrics as expected: we continued to get mixed feedback about usability from marketing surveys, user research, and customer service reports. In order to understand the problem further, I championed an investigation into our information and technical architecture.

• It became clear that merchants were not able to recover from errors they encountered when using Shopify Shipping. It also became clear that, internally, we had no visibility into the error codes we were receiving from shipping carriers or how these codes they were being translated to our UI.

• Error management was a black box, the result of having no standardized mapping of shipping carrier error cases. The experience was creating friction in a high-stress part of the merchant fulfillment journey, delaying product development, and having a negative impact on label purchases, new merchant adoption, and churn.

• My goal was to identify and quantify the systemic issues with the Shopify Shipping error architecture and iteratively improve upon them. This would be crucial for establishing a strong product foundation, allowing us to innovate quickly, scale for the future, and build trust with new and existing merchants.

My role

Phase 1

• Conducted a usability evaluation of our shipping label purchase flows using Jakob Nielsen’s 10 Usability Heuristics, uncovering that the biggest violations were #5 “Error Prevention” and #9 “Help users recognize, diagnose, and recover from errors”. These findings led to the creation of my product team’s first UX Errors project, focused on improving user experience through iterative fixes to highest impact UX debt. The project resulted in a 4% increase in merchant retention and a 67% increase in new merchant conversion.

• Established first content and design principles for managing errors in Shopify Shipping, based on success of UX Errors project. I established the need for product guardrails (instead of giving merchants total freedom to take any action) and defined the foundation of an “actionable” error message: explain the problem, the impact, and how to recover (when possible).

Phase 2

• Recruited engineering and data science to investigate and prove the need for further investment in error messaging, when it became clear that merchants were still struggling to complete shipping label checkouts. I championed this project outside my product teams’ core priorities, establishing goals, success metrics, a road map, and an implementation plan for our scrappy group of three.

• Led the implementation of monorail events to track which error codes were firing when, which error messages where connected to each code, which shipping carrier they were associated with, and in which language(s) they were being displayed. This gave the team a holistic view of carrier errors for the first time since Shopify Shipping was built.

• Categorized errors into three groups: Shopify errors, carrier errors, and purchase errors, and quantified the impact each had on our success metrics. Established that the biggest offenders were carrier errors, those preventing shipping rates from being displayed to merchants (and ultimately blocking checkout).

• Conceived and crafted a mental model shift for displaying carrier errors in our UI, reframing from carrier-specific errors to carrier-specific information. This new approach would significantly reduce noise in the UI, and ensure that concise, accurate, and actionable information was displayed to merchants every time. Presented concept to senior leadership, earning resounding buy-in for prioritizing project on 2023 roadmap.

• Designed wireframes and content rules, then collaborated with product design to polish final screens for integrating the new mental model into the product. We defined new information architecture and design guidelines that complimented existing best practices from the Polaris style guide.

Phase 3

• Began proposal for standardizing the next set of errors: purchase errors, those that appear after a merchant clicks “purchase” and the purchase fails. This involved navigating complex limitations based on all the shipping carrier API connections.

• Crafted new content for top 100 negatively-impactful purchase error messages, focusing on creating best practices for writing highly technical error messages which had no obvious solutions.

• Uncovered that purchase error messages for European merchants were only displayed in English as a result of the team deprioritizing translations during the original error implementation. Advocated for the importance of high-quality translation for all error messages, pressuring engineering to establish a scalable way to “save” translations in our system.

Outcomes

• Uncovered fundamental problems with the architecture used to power our product. Proved that current error handling impacted conversion rate by over 30%, resulting in lost label billings and incurring support escalations. Using data to size the issue and propose a scalable solution, I successfully convinced leadership to prioritize fixing errors, a huge win for merchants and the business.

• Established new best practices for crafting error messages, as well as implementing product tracking and reporting. I showcased that error handling has a holistic impact across all Shopify Shipping features and isn’t something that can be ignored.

• Created a meaningful shift in product-wide strategy: a higher focus would be placed on product quality, instead of just speed and impact. This would avoid incurring massive UX and technical debt in the future, and put more responsibility on craft leaders to understand how their work impacts product. As a result, product reprioritized our roadmap mid-year, adding more time for addressing tech debt and adding UX polish on all projects.

• Levelled-up my colleagues: the error overhaul work resulted in my engineering counterpart getting promoted to Staff Engineer. I also shared my process and approach with a content design colleague who reused it to tackle similar problems on a customer-facing checkout.

Next project >>