Data has always been the driver of successful retailing whether that was in the head of the store owner or delivered from a vast data warehouse. New technologies mean that there is more data being created and stored - and competitive advantage will be gained from the analysis of that data

Good Data Strategy is at the heart of an efficient supply chain, effective store and e-store operations and an engaging customer relationship. In recent years retailers competed on analytics, ironing out cost in the supply chain, negotiating with vendors and creating loyalty programs and developing efficient decision support systems and vast data warehouses. The next five years will see a proliferation of sources of that data and applications to process it. The acquisition, organization and analysis of mountains of data that dwarf what came before.

What Data is Available

Today's retailers are blessed with Sales transaction data from stores and every other channel, fully attributed item data, supply chain data that traces the physical items progress from Manufacturer to the Customer, customer data to include profile, preference and loyalty, store data, workforce data, call center data, site and usage statistics. Walmart's Retail Link pioneered sharing the vast tracts of data with supply chain partners - providing visibility that drove down supply-chain costs for all parties.

Dr Andreas Wiegend's course on the Social Data Revolution provides a definition: Social data refers to the process where individuals generate and share data: Data about themselves, data about their relationships with others, data about products and services, data about the world. Text based shared content has been around for a while on Yelp and Angie's List but the sharing of data is expanding beyond reviews: Sites like Oyster and Foodspotting provide a more truthful crowd-sourced view of hotels and restaurants through instantly uploaded photos; TurntableFM and LastFM allow users to publish their playlists; and Facebook Places, Pachube and Nimbits take this a step further, building out an internet of things

As retailers really begin to understand their customers - in smaller and smaller segments down to the individual, this social data will increase in significance.

Join and Analyze

Joining up disparate data to deliver unique benefits will be the challenge as soon as the datasets become available and usable. My belief is that the ongoing inability to settle on data standards will prevail and so we should not expect unifying global schemas. However, the semantic web will allow data to be described such that with some effort data sources can be mapped sufficiently well to allow automated applications to act on such data.

Services that join up and make sense of disparate or messy data will emerge to prepare data for analysis. Denodo's Data Virtualization and Data Federation platform and First Retail's Semantic ETL are examples of such services. Once the data is in a somewhat structured format, there are plenty of analysis platforms and consultancies who will deliver a solution - Cloudera, thinkbig and Pete Warden's Data Science Toolkit offer non-specific solutions and companies like EYC, Dunnhumby, Demandtec and Revionics tackle specifically retail problems such as loyalty, pricing and marketing mix.

Retailers will need to emerge from analysis of their data silos to joining up the data and understanding the effects of one part of their business on another. For example, price sensitivity applied at the segment level should be used to drive promotions - this will only be achieved when customer profile and transition data can be crossed with pricing and promotion histories.

A New Order

It's easy to say that retail business decisions should be made based on accurate and up-to-date information from around the business, from competitors, vendors and consumers, but that is not really the problem at hand. The biggest problem is one of information logistics - getting that data to the right decision maker at the right time.

A few solutions are emerging - many of them in-house built on Business Intelligence and Analytics platforms such as SaS Retail or the Retail ERP platforms as offered by companies like Oracle. Ecommera's Intelligent Trader presents a set of out of box dashboards that allow the multi-channel retailer to consolidate and deliver information.

The challenge for retailers is to evolve their processes, information systems and performance measures to be more information-driven. This will require strong direction from CIO's and CTO's, strategic investment in technology and some tactical quick wins to demonstrate the huge potential benefits.

Demanding a new Analytic Infrastructure

I believe that most retail businesses can be summarized as four concurrent but asynchronous processes - that equate to four 'lifecycles':

1. Customer Lifecycle - from the time a customer discovers retailer to the time they say 'remove me from your database'
2. Customer Order Lifecycle - from the moment the order is placed, thru' purchase, delivery and return or refund
3. Replenishment Order Lifecycle - from the moment a Vendor order is placed through the time it is available in the warehouse or store
4. Product Lifecycle - from the time a product is announced to its end of life and disposition

I have a notion of four rivers of information flowing around the business - each river providing the most granular detail around the four life-cycles - and the information being available to and from each business process - a 'retail information grid'. So when a new process is defined and systems designed, an explicit specification of what data will feed into and out of the grid should be made available.

The vision is that I can query the grid to find a list of customers that match a specific profile (e.g. customers that looked for a duvet priced over $100 in February 2011); the location and status of a customer order (e.g. all orders with an Oregon ship to address being shipped Fedex that are more than one day late); inbound order (e.g. which departments have orders due for delivery at stores before August 10th); or product (e.g. show me all products that had a dip in sales greater than 10% between January 1st and January 15th).

Forecast Emergence in Mainstream: 2016 - This is not simple, it is a journey that retailers must embark upon now to ensure that they are able to make competitive business decisions.

Previous post in this series of five: Payments - The Re-Statement of Currency

Main post of the series: Retail Trends - Main

Things I read to get here

Data

• Big data: The next frontier for innovation, competition and productivity McKinsey May 2011
• Building with big data Schumpeter in the Economist, May 2011
• Data Mining and E-Business: The Social Data Revolution Dr Andreas Weigend at Stanford University (this is a wiki, so it will change)
• Competitive Advantage in Retail Through Analytics: Developing Insights, Creating Value Information Management 2006
• 11 Big-Data Analytics Predictions for 2011 TDWI March 2011
• The Social Data Revolutions Andreas Weigend in the Harvard Business Review, 2009
• First Camera, Then Fork New York Times, April 2010