Design & UX Quality
Impact
The discovery+ streaming service debuted on January 4, 2021 after two years of development, including 6 months of rigorous QA testing by a globally-distributed Product, Design, and Engineering organization.
I was responsible for Design & UX Quality for the entire consumer-facing experience at launch and created a program that was vital in delivering discovery+ on time, reaching a level of initial product quality, usability, and customer experience that far exceeded industry and customer expectations.
1500+ customer-facing product and design issues identified and resolved
Released on more devices than any other streaming service at launch
Largest-ever content library with 2,500+ shows and 55,000+ episodes
Problems
Discovery+ was the company’s top priority for 2020, but due to the Covid-19 pandemic we were forced to leave our offices and work from the safety of our homes. Leadership had planned to hire 75 additional designers and product managers to test and launch discovery+, but due to the pandemic we had to freeze all hiring. This was actually a blessing in disguise as our Leadership was forced to innovate and created the company’s first exchange program know as “Project Thunder”, borrowing top Product and Design talent from all corners of the company to launch discovery+.
I was nominated to join the Design team as a UX Director and shortly after starting my new role, I escalated the need for a substantial Design & UX QA program and was given full responsibility to stand it up and lead it.
Considerations
The team was a large start-up and hadn’t launched anything together
We were in the throes of a global pandemic, 100% remote, and globally-distributed across 8 timezones
We only had 9 months to launch and had no QA process
Program Goals
Define how the organization would capture visual and experience defects
Establish QA workflows between Product, Design, and Engineering
Ensure UI, UX, and content parity across all platfoms
Bring a new streaming service to market with a world-class experience
Role
Responsible for Design & UX Quality across all platforms, I built and directed the entire program, bringing a wealth of experience in leadership, process, culture, and design quality after years of leading design for some of the company’s most successful products at HGTV and Food Network.
Created training documentation including principles and best practices for QA, checklists for smoke testing builds, and guidelines for capturing defect tickets to avoid misinterpretation and rework
Defined and iterated on the end-to-end Design & UX QA process in coordination with Engineering
Organized staff into platform-based squads with defined roles and responsibilities
Conceived squad ways of working including team rituals and workflows
Conducted training sessions across all squads
Facilitated weekly squad check-ins to coach, solve problems, ensure team health, and refine workflows as needed to maximize testing and development velocity
Led group QA sessions to reach UI, UX, and content parity across all platforms
Actively tested on Web, iOS, Roku, fireTV, and tvOS, and served as UX test lead for HWA (Xbox and Samsung)
Reported to leadership, stakeholders, and partners on progress
Accomplishments
Partners
I’m so thankful for my partners, their teams and all the amazing work that went into making this program a success.
Jenny
Jason
Product
Matt
PMO
UI Design
Mike
Device QA
Yoon
Global QA
Activities & Timeline
| Planning | Kickoff & Training | Squad QA / Testing | Group Parity Sessions | Launch Support |
| Q2 2020 | Q2 2020 | Q3 - Q4 2020 | Q4 2020 | Q4 2020 |
|
|
|
|
|
Squads
Web
Subscription & Onboarding
Core Experience
I organized the squads based on experience and platform proficiency. Each squad was comprised of a Product, UX, and UI lead, plus additional testers where possible.
Mobile
Android
iOS / iPadOS
Connected TV
Fire TV
HWA (Samsung & Xbox)
Roku
tvOS
Product Lead: prioritization, coordinating with QA & platform leads, and general testing
UX Lead: testing functionality, behaviors, and the overall customer experience
UI Lead: testing brand and visual application of the design system
Responsibilities
Test Principles
For cross-platform quality and parity, we needed each squad to test builds consistently, so I provided guiding principles that kept us aligned throughout the project.
Clear Handoffs
When possible, Design should review deliverables and artifacts with Engineering and QA to provide clarity on design intent, systems, and naming conventions to avoid interpretation or rework.
Defend the Customer
We must be OK with compromise, but respectfully defend what’s best for the customer, always.
Document Well
It is difficult for QA and Engineering to reproduce our issues so we must concisely articulate the current behavior vs expected behavior, all the steps needed to reproduce an issue, provide screenshots and/or video, and link to design deliverables when possible. Otherwise, our issue may get deprioritized or removed altogether.
Demo & Discuss
Be proactive and conduct weekly demos with Engineering and QA to give early feedback on dev progress instead of capturing a large amount of tickets at the end of sprints.
Parity for All
Most customers don’t have the latest and greatest devices, so we need to test across a bevy of devices, especially budget devices to ensure product parity and a smooth customer experience for all.
Smoke Test Often
Sometimes fixing one issue creates another. Its inevitable, so be sure to smoke test our most important customer journeys and use cases after each end-to-end build.
Test the Journey
Don’t just test individual pages or elements… always test how customers got there.
Rituals
Mid-Sprint QA
As soon as a ticket was flagged as “Ready for Product” in JIRA, we tested the issue and passed or failed it accordingly. Tickets were a mix of new functionality and fixes to previously captured issues. Product could often verify a fix on their own, but would pull in designers to help confirm as needed.
Weekly Bug Scrubs
Product, Design, and Engineering QA squad leads met weekly to confirm, prioritize, and dedupe all open issues found that week. Some squads could do this via slack, but other squads with significant timezone differences preferred a weekly session.
Weekly Demos
Squads conducted group demos on Fridays to review both nightly and end-to-end builds (every 2 weeks) to discuss and capture glaring issues so Engineering had the chance to make fixes during that same sprint. These sessions were critical to our success and made for a lot of fun.
End-to-End QA
Engineering would release a full, end-to-end build on every platform at the end of every two week sprint and squads would conduct both individual and group QA using our Design QA checklist to ensure critical user journeys and functionality weren’t inadvertently affected.
Group QA
Once a month on Fridays, the Connected TV squads conducted a group QA session to discuss progress, inconsistencies, and issues across platforms. While the Web and Mobile teams did this on an ad-hoc basis, this was a constant for the CTV teams.
Content QA
A few months before launch, all squads participated in a series of group QA sessions to capture content and/or data inconsistencies across all templates and platforms.
Workflows
An early stumble led to a major breakthrough
We originally tested builds at the end of every two week sprint. However, we quickly learned this wouldn’t work because by the time our issues were prioritized the following sprint was already at point capacity, so Engineering couldn’t take on any issues we found until the sprint after and we wouldn’t see progress for upwards of 4-6 weeks! Fortunately, we corrected this after only a few sprints by creating the “mid-sprint” QA workflow.
Allowed us to give early and frequent feedback in the middle of a two week sprint so Engineering could fix issues more quickly. Dev velocity, build quality, and morale immediately increased.
Mid-Sprint QA Workflow
We conducted a thorough smoke-test on end-to-end builds every two weeks and this effort became far less exhaustive over time as most of our issues were found and corrected mid-sprint.
End-to-End QA Workflow
Conclusion
This program was vital in delivering discovery+ on time, reaching a level of initial product quality, usability, and customer experience that far exceeded industry and customer expectations. This was the largest and most complex project I’ve led to date and I’m grateful to have had the opportunity.
I’m passionate about building products that customers love to use, as well as building design teams, processes, and a culture where people can do their best work.
We were in the throes of a global pandemic, 100% remote, and globally-distributed across 8 timezones
We only had 9 months to launch and had no QA process
I devised and led a program that captured and resolved over 1500 customer-facing defects across more platforms that any other streamer at launch
Summary
Press
I was promoted to Head of UX for discovery+
Discovery+ scaled to over 25 million paying subscribers in 11 countries and 13 languages in under two years
Discovery leadership leveraged the success of discovery+ to acquire Warner Media (HBO, CNN, etc) in the largest media merger in the last 20 years and created Warner Bros. Discovery