The Secret to Making Search Tuning Easy and Safe for Maximum Results

Or a lot better.

I have exactly zero clients who are satisfied with item findability on their site, and yet 95 or more out of a hundred haven’t done any meaningful search tuning in the last year.

The fact is that regular search tuning is the single most important maintenance activity a business team can do to drive conversions and average order value.

So why do so few otherwise savvy operations do it? The answer, as far as I can tell, is that it can be difficult, and potentially risky, if it’s not handled properly.

The difficulty can stem from three main fronts:

1. Search-result quality is 100 per cent subjective and there is no one person or algorithm that can score it effectively.
2. The Whack-A-Mole Effect: Fixing one burning issue can cause three other unforeseen issues to pop up in different corners of a product catalog.
3. The process is sometimes highly politicized by different category managers, many of whom do not understand search as well as others, and all of whom are jostling for specific fixes, biases or placement.

The potential risk is all derived from the perception the e-commerce site is doing pretty well right now: If it ain’t broke, why fix it?

But is it really performing well? Take an honest look at your key performance metrics for the answer.

I propose something new I like to call Richmond’s Secret of Search Tuning: perform a thorough search-tuning exercise on your site every six months or after every meaningful catalog change, whichever is more frequent.

It sounds like a lot when I say a week or two out of every 24 should be spent tuning your search, but it sounds totally reasonable when I say you should spend four to eight per cent of your effort on search tuning. So let’s go with that.

Here’s how you tackle the complexities of the tuning process.

To manage scoring result quality, there’s a quick, dirty, easy road and a shining, golden panacea road.

The dirty approach is to make a list of your 100 (or 1,000 or whatever) most important search terms. Hint: this is an even mix of the most common terms and your most expensive AdWords buys. Then, have everybody on your team, including your boss, take 30 minutes to power through the list of proposed search terms and rate each other’s suggestions on a scale of one to five. Then average it all out.

The shiny approach is to implement an A/B or multivariate testing approach on your site, pit a few configurations against each other, and pick the winner. Regardless of what you might have heard, this involves a whole lot of work and technical complexity and it’s best handled by professional search tuners like the team at RealDecoy. The MVT approach will get you as close as possible to an ideal configuration.

Earlier I mentioned the Whack-A-Mole Effect. Overcoming it is straightforward but there is no silver bullet. You have to iterate over your changes. Score the site, point out problem areas to your search tech team and consult with them, make a couple of changes, score it again. Then repeat the process as needed. You’ll start to get really good at intuiting what’s getting better and what’s getting worse, and will probably come up with a bunch of other UX enhancements that never would have occurred to you otherwise.

I also mentioned early that you’d potentially need to navigate some internal politics. I could write a whole other blog about it, but the simplest approach is to be fair. If you’re scoring based on 100 terms and you have five departments, then everyone gets 20. Or divide it up by revenue share, volume or whatever makes sense for your company.

Similarly, average out the scores from each iteration and look at the trend. For example, if a change solves everything for Electronics but hammers Outdoors, you’ve got a problem.

Follow this protocol and search tuning becomes easier and safer.

Remember when you read the first paragraph above and were nodding your head? Generally, nobody is happy with search. As long as what you put into production scores better than the control, you have nothing to fear.

There is one caveat: the reason I stated six months instead of “as often as possible” is because users get accustomed to your search’s behavior. If you’re constantly tweaking and fiddling, it gets decreasingly predictable and will frustrate people. Sure, sometimes a VP calls at midnight because he or she can’t find something they “know” you sell, and you have to make a quick change. Just make sure to keep a master list of all the quick fixes you’ve done since the last search tuning effort so you can keep it all in line.

So stop hating on your site and its myriad failings! Just set aside four per cent of your effort for tuning, just like you do for exercise, hair care, a retirement fund or your yard. Hmm. Yeah, I don’t spend four per cent of my time on any of those things either. Maybe it’s time for a tune up…