Reengineering data search for retail advantage

Speed has always mattered in retail – across supply chains, store operations and customer engagement. Today, this extends to how quickly businesses can turn data into a strategic asset.

Interpreting and translating data signals efficiently and rapidly into helpful business and customer outcomes is key - done well can add to the bottom line. So far, so well understood.

Since the early days of analysing and interpreting ‘big data’ the playing field has become more evenly contested and finding a point of difference has grown increasingly difficult. Yet when faced with processing vast data sets, using a popular commercial database isn’t always going to cut it for customers who expect speedy answers to live queries at reasonable cost.

A unique problem in customer analytics, which is often overlooked, is that you need to process at the individual customer level, tracking behaviour over a sustained period without any form grouping or aggregating the data. We are not interested in product sales totals, which can easily be aggregated, as much as we are with the associated products in a customer’s baskets and the trends overtime. Which groups of products have a higher associated basket value, for example? Consequently, we routinely update the basket history and shopping behaviour of over 2.3 bn individuals.

Benefits of a more targeted approach

Employing a large commercial search engine such as BigQuery is to use a highly capable generalist designed to serve many industries and patterns. This generality introduces inefficiencies, or prohibitive costs, when applied to very large-scale retail customer analytics.

The alternative is to use purpose built, highly specialised databases produced exclusively for retail analytical workloads designed to minimise the computational work required.

In traditional cloud platforms, performance is often improved by allocating more compute. As data volumes and query complexity grow however, cost scales accordingly. Bespoke databases achieve performance gains structurally by reducing the amount of work required and using low level data storage techniques to answer each query while delivering:

• Orders-of-magnitude faster execution for retail analytics with predictable performance
• Benchmarked at 100 x faster and 178 x cheaper than Google BigQuery

This is lightning quick, delivering unmatched performance and cost efficiency.

Architecting for accuracy, speed and scale

The key is instead of creating a data schema and model which suits the constraints of the database engine, create a database engine and storage to suit retail data and the associated analytical queries.

And while AI is being applied for many useful purposes across retail, if you take a query like purchases by segment for example, you don't want generative AI making up a plausible answer based on colossal datasets. You've got to be forensically correct in the answer you give and not to expose the underlying raw data to GenAI models that are limited by context space and associated cost.

Speeding up insight in this way is of no consequential use however if it isn't supported by the right platforms that connect data science and technology to serve customers, to empower retailers and to drive growth. Platforms and capacity that can flex and scale with the client journey are what is needed. In retail the nature of the processing is moving away from large batch processing jobs creating predefined reports and insights to more real time interrogation of the data sets be this by a conversational interface or by more conventional means. Regardless of the UI the user expects and demands immediate results.

So, speed, cost, and accuracy remain the watchwords that should govern decision making when considering which partners to work with in this critically important area.