Is Your Minimum Viable Product (MVP) Biased?
It’s a terrific idea: ship out a software product with the fewest bells and whistles needed to perform a task. The early real-world feedback on this minimum viable product (MVP) will enable refinements and an even better product sooner than traditional software development. MVP was popularized as part of “Lean Startup” companies and was quickly embraced as a component of Agile software development. But the MVP process embeds troubling equity pitfalls.
That’s because many MVP builders don’t apply an equity lens, but Abt is taking steps to create more equitable initial releases and final products by gathering feedback from diverse audiences. The shift will provide products and their tangible benefits more quickly to those who may need them most.
Inequities in MVP
In the rush to get a product out quickly and show quick wins, the MVP approach often results in products that are viable for some users but not all. Here are some examples of inequities that may seep into MVPs:
- Releasing an English version of a solution first with the intention of following up with other languages later
- Performing sufficiently well in high bandwidth locations, but crashing, timing out, or simply taking an unacceptably long-time for those who can’t afford high-speed internet or lack access to it, such as residents of rural areas or lower-bandwidth countries
- Creating iPhone applications before Android applications, which will overrepresent younger Americans. Older Americans and those outside the U.S. are much more likely to be Android users
- Optimizing for desktop browsers when working in Africa, where less than a quarter of the population uses full scale computers.
- Collecting only binary data options on gender and not allowing people to update gender fields
- Building functionality that works for fully abled people while intending to build in accessibility features for people with disabilities in a subsequent release. One example: not creating alternate text for icons to make an application viable for those with vision impairments
- Releasing a speech recognition program that does poorly when trying to decipher accents or vernacular that minority populations use
- Deploying facial recognition or biometric features that work well for middle-aged, white men but not for older, non-white, and non-male (including females and non-binary) users. See Broussard, M (2023) and Buolamwini, J (2018).
In all of these instances, minimum viable turns out to mean viable only for the privileged few and not viable for everyone else. An inequitable MVP is a problem in itself, but it also could end up in inequitable outcomes at the later stages of product development or create problems in the overall project the technology supports. For example, releasing an English-only version early to recruit people for a study may result in skewing results to underrepresent non-English speakers, especially with a shorter recruitment timeline. And while developers may plan to address later the variety of MVP known shortcomings in the list, since funding for future releases isn’t always guaranteed, “later” may never happen. The temporary delay thus becomes a permanent exclusion.
In its original conception, the MVP is supposed to be an experiment. Releasing MVPs opens the product to feedback from early adopters.1 The usage patterns and subjective preferences of early adopters weigh heavily in the product roadmap. This is better than releasing a product with no real-world feedback at all.
But early adopters don’t represent a valid cross section of the general population. According to the late Everett Rogers, a communications and sociology professor who coined the term early adopters, they represent only about 13.5% of the general population. In 2016, the Pew Research Center—using data Abt collected-- found that 28% of the U.S. population described themselves as having a strong preference for early adoption. Unsurprisingly, this preference is highest among younger people, men, and those with higher incomes.
Early adopters are not always from more advantaged groups, however. A Nielsen study in 2016 found that African-Americans are more likely to exhibit “early adopter” behavior than the total population. Two more recent Pew studies of early adopters of ChatGPT validate higher representation among younger, highly educated, higher income, and male sub-groups. Among those who have heard of ChatGPT, Black, Hispanic, and Asian sub-groups were more likely than non-Hispanic whites to have used it.
Still, unequal representation among the early adopters persists, and developers tailor initial releases to their preferences to achieve initial success. The iterative software development lifecycle becomes a vicious cycle as the preferences of the early adopters get amplified and their feedback gets incorporated into the next version. Making the product better becomes equated with making the product match the preferences of early adopters.
The problem is compounded if the application’s target audience is underrepresented among early adopters. For example, diabetes is more prevalent among people 65 and older, people with lower incomes, Native Americans, and Alaska Natives. If an application to serve diabetes patients draws on the standard group of early adopters for feedback, developers would receive precious little feedback from the target group.
The equity risks of the MVP approach are of great concern in the public sector because they jeopardize achieving a government agency’s mission and mandate to serve an entire constituency. As digital government becomes increasingly prevalent—with more government services being delivered online--the influence of high-tech culture is also becoming more prevalent, including the MVP approach. The incentive to listen to technology experts will be strong as customer satisfaction with U.S. federal services still lags behind many for-profit services, according to a McKinsey study. The same study found that non-whites and low-income individuals use government services more frequently than higher income whites and are less satisfied with the services. If the government doesn’t address the equity issue, attempts to use the MVP approach could unintentionally exacerbate existing inequities in government services, to say nothing of further reducing customer satisfaction.
What Can We Do?
One mitigation strategy is hiring a diverse development and test team since teams often believe a product is ready when it works for them. The testers should include members of the demographics of the target user groups, ensuring that those with lived experience provide feedback on the product. Another approach is to incorporate expected user profiles with an equity lens into the definition of what it means to be viable. The product must be viable for each user sub-group to be considered an MVP. A human-centered design (HCD) approach, which focuses on users’ wants, needs, and preferences in every phase of the development process, is critical. So is journey mapping, which visualizes how different personas navigate the system to achieve their goals.
A diverse business analysis team that can translate the results of the HCD and journey mapping exercises into concrete requirements and test cases is also essential. The development roadmap and testing plans should include definitions of viability for each of the solution’s user groups.
Minimum viable is thus not defined with respect to a generic user, which defaults to young or middle-aged, white, male, abled, cisgender, high income, American, urban, and English-speaking. Instead the context for minimum viable is each key user demographic. For example, acceptance criteria for a minimum viable facial recognition product should be defined not as successfully matching 95% of the population overall, but as 95% of men, 95% of women, 95% of African Americans, and 95% of whites.
At Abt, we are also experimenting with using our Configuration Control Board (CCB) to enforce equity goals. The purpose of a CCB is to ensure that all key elements of a product (including security testing, communication, implementation, rollback if needed, etc.) are in place prior to any release. The CCB members include leaders from each of these key disciplines, each of whom reviews the overall change from their specialty area.
Most of the areas the CCB governs are not minimally viable. A product will work (at least for a little while) without backups in place or without enforcing strict security. The CCB already redefines the concept of minimally viable to ensure compliance with certain baseline standards of good practice. We are now including equity among these standards.
Development teams that seek to release new products must demonstrate that they have identified all key target users and that the product is viable for all of those groups. They do so by completing a form that requires them to specify target audiences. The form lists out equity issues that may arise with respect to supporting those groups. The development team then indicates how the product and testing process addresses the concern.
While the work of building the solution equitably begins in the requirements and design phase, the CCB serves as enforcer, since the CCB will not approve the release until the equity criteria are met. Enlisting an equity expert for the board as either a voting or consulting member to review the equity criteria also is a potentially valuable approach.
Another way to address the imbalanced representation in the feedback loop from early adopters as a group is to learn from our survey research colleagues and think of the early adopters as a sample group. Both private and public sector organizations releasing MVPs could collect data on early adopters, then weight the feedback from individuals based on their characteristics of interest.
Neeva, a search engine company that Snowflake recently purchased, did this successfully. The company created a waiting list to get early access to its product and found that more than 75% of initial sign-ups were male. When collecting feedback from users, the company weighted the responses to equalize male and female feedback. This enabled Neeva to build its roadmap to support product features—such as a family subscription plan—that were more heavily favored by women than men.
Disaggregating feedback based on key demographic variables can also help to offset skewed results from an unrepresentative early adopter group. Applying sampling, weighting, and disaggregation approaches to product feedback can be applied to public-sector systems and is a great example of how combining expertise in equity, research methods, and digital work can produce a superior result. If the MVPs are indeed experiments, then sound experimental methods are essential.
Final Thoughts
When people hear the acronym MVP, they usually immediately think of “most valuable player.” Software developers bemusedly inform them that the acronym refers to minimum viable product. Unfortunately, in this case, the joke is on the software teams, since too often the process of defining minimal requirements unintentionally and inequitably anoints certain types of players as more valuable than others. The software team misconstrues itself as the most valuable of all. But with proper awareness and procedures to address inequities, the minimum viable product can become a most valuable product—and viable for all.
1 Early Adopters is the second of five groups of adopters described in Everett M Rogers 1962 book Diffusion of Innovation. The full list includes “Innovators,” “Early Adopters,“ Early Majority, ”Late Majority,” and “Laggards.”
