Product discovery: shoppers' unsolved problem

My cofounder David (a Computational Neuroscience PhD) and I have spent many years researching what we often refer to as ‘the product discovery problem’. This is a well known problem in e-commerce and by far the most ‘manual’ part of the path-to-purchase for online shoppers.

Product discovery is the process shoppers go through when they’re looking for something, but they don’t know exactly what that something is yet. This is very common for taste driven purchases such as in Fashion/Apparel and Furniture. People have an outcome in mind e.g. ‘a new outfit for a friend’s wedding’, or ‘a bedside table that fits my bedroom’s decor’, but they don’t have the keywords that pinpoint exactly what that item is yet. 

David and I initially attempted to solve this problem while working on AI and data science projects at large retail platforms, but quickly realized it could not be solved by a retailer. Since then, we’ve spent hundreds of hours watching people toil with this problem, and a whole lot longer than that designing, prototyping, building and testing solutions to it.

Why does the ‘discovery problem’ exist?

Why does it take so long to find the perfect new pair of shoes, outfit, piece of furniture or artwork for your wall?

The answer is simple, and obvious when you think about it. More than 99% of what you’re looking at online was created with someone else’s tastes in mind.

There are over 7 billion people in the world, we all have different preferences and hundreds of thousands of retailers create hundreds of millions of new products each year to serve those different tastes. It is up to you to sift through those options to find what’s relevant.

Nordstrom alone lists over 30,000 different shoes on their site, so it’s no wonder most of the shoes you’re scrolling through aren’t to your liking!

When is product discovery a ‘problem’ for shoppers?

Let’s start with when product discovery is NOT a problem:

• When you know exactly what you are looking for (e.g. you’re buying the same white shirt you bought last time, or someone has recommended you a specific book or game)

• When buying functional commodity products like a stapler, phone charger, batteries etc.

• When buying low-cost items, where making the ‘best choice’ is not a big concern

In all the above cases ‘Search’ as we know it today does a good job. Most items you find on Amazon fall into this category.

The product discovery problem rears its head when:

• The shopper’s individual tastes and preferences are an important part of the decision (fashion, homeware and art are leading examples)

• The cost is significant and the ‘loss’ from making a poor choice is therefore high (e.g. an expensive handbag, jacket, dining table, couch)

• Buying for others (i.e. a gift where you care about the receiver)

Simply put, when the outcome of the purchase matters a lot, people want to get it right. But given there is an almost infinite amount of choice and those options are fragmented across thousands of potential retailers, getting to the ‘right’ item for your specific taste and preferences is an almost impossible task using the current tools available (Search → Filter → Scroll).

As a result, the average shopper spends 79 days of research before making a major purchase and the equivalent of 11 days a year ‘digital window shopping’. 

If you are not a taste driven shopper who cares about finding the ‘right’ outfit to wear or the ‘right’ side table for your apartment, those numbers may come as a surprise. You may even fall into the common misconception among people that don’t do a lot of shopping that ‘people enjoy this process’. After years in this domain, I have never met a shopper who after hours of manually scrolling through pages and pages of options, described the process as enjoyable. They simply care a lot about the choices they make and given the tools currently available to them, it’s like finding a needle in a haystack.

How do consumers currently deal with this problem?

Currently it is up to the shopper to manually sift through the enormous potential universe of products to try to uncover the most relevant items. In most countries, where choice is now almost unlimited, this is practically impossible. Rarely therefore does anyone actually make the ‘best’ decision.

Shoppers’ main online tools to tackle this task are ‘search’ and ‘filter’.

Search is great for certain purchases (as explained above) but it’s a very blunt tool if you don’t know exactly what you are looking for. Even when you do, if you’re looking for a non-functional item like a dress, it is almost impossible to distill your taste into ‘keywords’.

So what do people do instead? Well, they find websites and brands that are more aligned to their ‘taste’. This in itself is a discovery journey as there are now tens of thousands of stores for every different product category.

By finding a store that aligns with their taste, shoppers can narrow down from millions of options to thousands. This is the value of ‘curation’. But now they are jumping from site to site, finding a few potential items here and there. They probably have several tabs open at this point (I’ve seen plenty of shoppers well into the 100’s of tabs).

It’s around this point that ‘choice fatigue’ starts to kick in. Choice or decision fatigue is known as one of the ‘unspoken’ problems of online shopping. It often goes unspoken because ‘choice’ is supposed to be a good thing. This is the paradox of choice. Most consumers globally say there is too much choice online (src). The reality is they just don’t have smart enough tools to navigate it yet, so have to settle for ‘good enough’ far too often.

Managing infinite choice

Given how vast and fragmented e-commerce is, when researching what to buy, people need a way to manage and keep track of all the things that catch their eye along their path to purchase. The discovery journey is a messy one; it can be cross device, include independent stores, aggregation platforms, marketplaces, comparison sites, social media etc. It has become so messy in fact that Google Consumer Insights now terms this the “messy middle” of the purchase journey.

The most common approach is to make a list. 43% of consumers use bookmarks, shopping carts or shortlist the options in some other way such as spreadsheets or pieces of paper.

I witnessed a lot of this in my research into product discovery. Seeing how poor these list making tools were was partly what led Moonsift to build what is referred by our users as ‘one wishlist for every store on the planet’ - which is a bit like Pinterest but specifically designed for shopping: https://www.moonsift.com/wishlist

Building this tool and watching people use it opened our eyes even further to the dynamics underlying the discovery problem. Our community of only a few thousand shoppers already browse over 30,000 different retail platforms and are browsing more than 10,000,000 products a month. These items are then kept neatly organized by our users into different collections - nearly 100,000 different collections have been made to date.

Why hasn’t this problem been solved?

I mentioned at the start that this is a “well known problem”, so why hasn’t it been solved?

Indeed many different people at different times have attempted to solve this problem. In fact, even DeepMind attempted to solve it before being acquired by Google in as early as 2014. But while it may appear to be an easy problem for AI to solve, there are many technical, business and data related reasons why consumers’ product discovery problem remains unsolved.

Fear not though, this may all be about to change. We are hard at work on this problem and the solution has never felt closer.

Learn more about the world's first shopping copilot for the entire internet.