How to Create a Scalable Product Validation Testing Process

The average salary of a full-stack developer in the US is more than $100,000 per year. To build new features into your product, you need multiple developers. Add to that the investment that goes into infrastructure, research, technology stack, testing, maintenance, and then marketing, and you are looking at hundreds of thousands of dollars’ worth of effort that goes into building a new feature. You want to protect that investment by building features that have a greater chance of being successful.

‍Continual product validation helps you make sure you are building the right features into your product by collecting feedback from your users at all stages of the product lifecycle. You earn stakeholder trust, attract investors, and keep your team focused, all the while reaffirming your belief in your idea. By building a scalable product validation testing process, you can use the same process for new features and other products in the future. Here’s how:

Validate the Severity of the Problem

Your product or new product feature aims to solve a problem. At this stage, you need to gauge the severity of the problem. If it’s not a big enough problem for your target market, it may not be worth pursuing. Similarly, if your user base already has a solution to the problem, they may never adopt your feature, no matter how great it is.

Back in 2013, Facebook launched an app called Facebook Home. It replaced the regular home screen of Android users with a Facebook home screen. The app didn’t work out, and they had to shut it down the same year. The reason: they overestimated the people’s need to see Facebook updates. Facebook users already had the Facebook app, and they could also access Facebook through their browsers. The home screen feed was a solution to a problem that didn’t exist. Facebook would have benefitted from validating the severity of the problem.

To see if the problem you have set out to solve is big enough, plan a survey targeting your user base. Aim for a statistically valid sample size. In a user base of 100,000, you’ll need about 400 participants. Use closed-ended questions designed around the problem. To come up with the questions, first, have a brainstorming session with your team and write down all of your assumptions about the problem. For example, if your idea is to build a new feature that helps your users do their taxes, your assumptions about the severity of the problem can be:

• People spend too much time doing their taxes
• People struggle doing their taxes and have to hire tax preparers/advisors
• People are willing to use a software product that fetches data from their bank and eCommerce accounts

Some of the questions designed around these assumptions can be:

• Do you spend 5–10 hours or more doing taxes?
• Do you hire a tax advisor to file your tax returns?
• Would you like to do it yourself if the process was easier and took less time?
• Would you be interested in an app that would do it for you?

Since these are closed-ended questions, you can validate your idea before investing any resources into it. If 90% of your participants said yes to all questions, you know you are onto something. If, for example, more than 50% said yes to all questions, you are close, but your idea needs to be tweaked and refined. Maybe people don’t spend that much time doing their taxes, or maybe they are perfectly happy with tax advisors unless this new feature does their taxes in just a couple of clicks.

This technique scales with every new feature you want to add to your product. Every time you want to add a feature, go back to the drawing board and validate the severity of the “problem” the new feature aims to solve.

While we recommend using closed-ended questions for this stage, you can also go for open-ended questions, Likert scale questions, and so on. But collecting the data is only part of the validation process. Make sure you know how to interpret data from different types of questions, so you don’t find yourself in a situation like Facebook Home.    

Validate the Size of the Market

Validating market size will help you determine whether it’s worth it for you to invest time in development. It is very easy to fall in love with your idea, validate it on a surface level, and start building, only to find out there isn’t a big enough market for your product to be successful. Developing a new feature without validating is like adding a meat department to your grocery store in a predominantly vegan neighborhood. Yes, there are a few people there that’d regularly buy from you, but not enough for that initiative to be profitable.

To validate market size for your new feature, first determine a target market. This can be the exact same target market that uses your product, or you can be branching out to attract more people with the help of a new feature. You can do this based on job titles, interests, age, location, and other criteria related to the problem. Let’s say you have an app that helps gym managers/owners manage their gyms, and you want to introduce a new feature that helps them post new media to social networking sites. Your initial target market would be gym managers and owners that currently use the app, as well as any gym managers located in your state. So, you will depend on user status, job title, and location as the three variables to define your target market. The other variables, such as age and gender, may not be as important.

Once you have defined your target market, look for data from reliable sources in terms of the number of potential customers. You can find the total number of gyms in your local area from Google maps. You can also use Yelp and BBB.org to find gyms in your state. You can look up gym managers and owners from online fitness forums (such as r/Fitness on Reddit). For current users, you will rely on your own database.

The next step is to determine price. Sometimes, you want to add a new feature for free just to keep up with the competition. On other occasions, you want to charge the users to gain access to the new feature. In either case, you have to consider product price to calculate market size.

If you plan to give the new feature to all users for free, use your current product price in the following equation:

Market size = (Total number of potential buyers) × (product price)

If you plan to charge the users for the new feature, calculate the new product price by taking cost of development and upkeep into account. You also have to consider profit margins and market demand. If there are similar products out there, you can look at their pricing for comparison and see if you want to start with a lower price to increase market penetration.

Once you have determined a price, you can calculate the market size using the following equation:

Market size = (Total number of potential buyers) × (product price)

Let’s say there are 2,500 gyms in your state. If each were to buy your product at a $1,200 one-time fee, your market size would be $3,000,000 (2,500 × 1,200). If your new feature costs $25 extra, your market size would increase by $62,500 (2,500 x $25).

To scale this aspect of the validation process, use it every time you want to introduce a new feature or advertise the product to a new market. Going back to the example of the gym management app, every time you want to introduce a new feature, calculate the market size again and see if the number of potential buyers and the new product price make it worth developing the new feature.