Analytics Made Simple

Analytics Made Simple

PG Madhavan, Ph.D.

Chief Algorist, Syzen Analytics, Inc.

Seattle, WA, USA

Brief bio: PG developed his expertise as an EECS Professor, Computational Neuroscience researcher, Bell labs MTS, Microsoft Architect and multiple startup leader. He has over 100 publications & platform presentations to Sales, Marketing, Product, Standards and Research groups as well as 12 issued US patents. Major Contributions:

·       Computational Neuroscience of Hippocampal Place Cell phenomenon related to the subject matter of 2014 Nobel Prize in Medicine.

·       Random Field Theory estimation methods, relationship to systems theory and industry applications.

·       Systems Analytics bringing model-based methods into current analytics practice.

·       Four startups with two as Founder.

 

Four hard topics in Analytics are explained in plain English in this article:

1.       Machine Learning.

2.       Why is Predictive Analytics important to business?

3.       Prediction – the other dismal science?

4.       Future of Analytics.

Machine Learning in plain English

If someone asks you, “What is ML?”, what will be your conceptual, non-technical answer?

Mine is . . . ML is “cluster”, “classify” and “convert”. I use these words in their English language sense and not as techniques. What do I mean by that?

Cluster: Structure in the data is information – find the structure.

Classify: Transform structure into a Mathematical form.

Convert: Convert into insight/ action.

Do this by Learning – meaning, use the ability to generalize from experience.

This captures the essence of ML for me. From my experience, I find that –

·       Convert: best done by a “paired” (Data Scientist + Domain Expert) combo.

·       Classify: there is a grab bag of tools and techniques that the Data Scientist can exploit on one’s own. You can see my attempt at unifying this bag of tricks here – “Unifying Machine Learning to create breakthrough perspectives”.

·       Cluster: I am not referring to specific clustering *algorithms* here. This step is where the Data Scientist works to sense, identify and extract structure or patterns or features in the data which are the bearers of information!

“Cluster” is the hardest part – data do not tell you where it hides the structure. Finding patterns is an “art” where inspiration, skill, experience, knowledge of inter-related theories, etc. play a major part. In a current algorithm work that I am doing, it turned out (after *months* of slicing and dicing the data) that rendering data into “phasors” (or complex variables) revealed the structure hidden in the data “by itself”!

If you are able to get at the most descriptive and discriminatory features at the “Cluster” stage, the rest of the steps will just fall into place (almost) and provide the best robust solution! If not, you may succeed but you will work many times harder to Classify and Covnert and end up with non-optimal answers.

It must be clear that my comments apply only to the first time development of an algorithm for a new business problem; once an end-to-end algorithm is in place, of course, the Cluster-Classify-Covnert steps can be automated for repeated application to similar data sets. But for the first-time ML algorithm solution development, automation cannot replace art!

Why is Predictive Analytics important to business?

A prerequisite for performance at a high level in business is the ability to understand and manage complexity. Complex systems to be managed properly requires a ton of data at the right time. BIG Data provide us the data we need; to put these data to work in order to take us to the high levels of complexity required while still managing it, we have to anticipate what is about to happen and react when it happens in a closed loop manner. Predictive Analytics will allow us to push our “system” to the edge (without “falling over”) in a managed fashion. This is why businesses embrace Predictive Analytics - to manage businesses at a high level of performance at the edge of complexity overload.

Prediction – the other dismal science?

An insightful person once said, “Prediction is like driving your car forward by looking only at the rearview mirror!”. If the road is dead-straight, you are good . . . UNLESS there is a stalled vehicle ahead in the middle of the road.

We should consider short-term and long-term prediction separately. Long-term prediction is nearly a lost cause. In the 80’s and 90’s, chaos and complexity theorists showed us that things can spin out of control even when we have perfect past and present information (predicting weather beyond 3 weeks is a major challenge, if not impossible). Even earlier, stochastic process theory told us that “non-stationarity” where statistics evolve (slowly or fast) can render longer term predictions unreliable.

If the underlying systems do not evolve quickly or suddenly, there is some hope. Causal systems (in Systems Theory, it means that no future information of any kind is available in the current state of the system), where “the car is driven forward strictly by using the rearview mirror”, outcomes are predictable in the sense that, as long as the “road is straight” or “curves only gently”, we can be somewhat confident in predicting a few steps ahead. This may be quite useful in some Data Science applications (such as in Fintech).

Another type of prediction involves not the actual path of future events (or the “state space trajectories” in the parlance) but the occurrence of a “black swan” or an “X-event” (for an elegant in-depth discussion, see John Casti, “X-Events: Complexity Overload and the Collapse of Everything’, 2013). For that matter, ANY unwanted event can be good to know about in advance – consider unwanted destructive vibrations (called “chatter”) in machine tools, as an example; early warning may be possible and very useful in saving expensive work pieces (“Instantaneous Scale of Fluctuation Using Kalman-TFD and Applications in Machine Tool Monitoring”). We find that sometimes the underlying system does undergo some pre-event changes (such as approach “complexity overload”, “state-space volume inflation”, “increase in degrees of freedom”, etc.) which may be detectable and trackable. However, there is NO escaping False Positives (and associated wastage of resources preparing for the event that does not come) or False Negatives (and be blind-sided when we are told it is not going to happen).

At Syzen Analytics, Inc., we use an explicit systems theory approach to Analytics. In our SYSTEMS Analytics formulation (“Future of Analytics – a definitive Roadmap”), the parameters of the system and its variation over time are tracked adaptively in real-time which tells us how long into the future we can predict safely – if the parameters evolve slowly or cyclically, we have higher confidence in our predictive analytics solutions.

Wanting to know the future has always been a human preoccupation – we see that you cannot truly know the future but in some cases, predictions to some extent are possible . . . surrounded by many caveats; more of “excuses” than definitive answers. Sounds a lot like a dismal science!

Future of Analytics – Spatio-temporal data

As businesses push to higher levels of performance, higher fidelity models are going to be necessary to produce more accurate and hence valuable predictions and recommendations for business operations.

ALL data are spatio-temporal! At the simplest to more complex levels -

·       Data can be considered isolated at the simplest level – a “snap shot”.

·       Then we realize that data exist in a “social” network with mutual interactions.

·       In reality, data exist in *embedded* forms in “influence” networks of one type or the other which are distributed in time and space – a “video”!

Spatial extent of data (distance) can be folded into time if we assume a certain information diffusion speed. Graph-theoretic methods do not account for time dimension. For accurate analysis, no escaping Dynamics over Time; meaning the use of differential (or difference) equations . . . and Systems Theory!

Systems Theory + Analytics = “SYSTEMS Analytics”! A few example business applications are shown above. As you can see, it spans most of the current Analytics use cases and many more promising ones when network graphs and spatio-temporal nature of data are fully incorporated in the coming years – basic theories and some algorithms are already in hand. For specific technologies, see –

·       “SYSTEMS Analytics – the future of Analytics today!"

·       For a full 30-minute discourse, Youtube video on “Future of Analytics – a definitive roadmap”

From the simple explanation of ML, the power and limitations of prediction and the promising Analytics technology roadmap ahead, it is clear that Data Science is indeed a rich area to mine that can create even bigger impact on business performance in the coming years.

PG Madhavan

www.linkedin.com/in/pgmad

 

Unifying Machine Learning to create breakthrough perspectives

Machine Learning – a unifying perspective & new paths

PG Madhavan, Ph.D.

Chairman, Syzen Analytics, Inc., Seattle, WA, USA

pgmad@syzenanalytics.com

Dr. PG Madhavan is the Founder of Syzen Analytics, Inc. He developed his expertise in Analytics as an EECS Professor, Computational Neuroscience researcher, Bell Labs MTS, Microsoft Architect and startup CEO. PG has been involved in four startups with two as Founder.

Major Original Contributions:

·       Computational Neuroscience of Hippocampal Place Cell phenomenon related to the subject matter of 2014 Nobel Prize in Medicine.

·       Random Field Theory estimation methods, relationship to systems theory and industry applications.

·       Early Bluetooth, Wi-Fi, 2.5G/EDGE and Ultra-wideband wireless technology standards and products.

·       Currently developing Systems Analytics bringing model-based methods into current Analytics practice.

PG has 12 issued US patents and over 100 publications & platform presentations to Sales, Marketing, Product, Industry Standards and Research groups. More at www.linkedin.com/in/pgmad

Pedro Domingos in his new book, “The Master Algorithm”, has done us a huge favor. As is true of any emerging technology field, Machine Learning (ML) is a “bag of tricks” today; it takes a while for a unifying framework to emerge. Then, one can see various aspects of ML as special cases of a general theory rather than a grab-bag of tools and techniques.

Pedro has taken a great early step to such unification. He has collected all major ML initiatives into a taxonomy that makes sense; five schools of thought: the evolutionaries, connectionists, symbolists, Bayesians, and analogizers. I believe this does not go far enough in the unification of ML thought however . . .

From the early days of “ML”, I see Pattern Recognition and Classification as a better unifying perspective. In particular, the classic textbook of Duda & Hart, “Pattern Classification & Scene Analysis”, published in 1973 is my starting point!

Duda & Hart’s approach in simple terms is as follows. Given labelled samples, obtain a class description consisting of either a distance metric (Euclidean, intra-class, etc.) or a probability density function and then derive a decision rule (Maximum A-posteriori Probability, Bayes, etc.) from the description. The decision rule specifies a decision boundary in feature space among classes.

Alternatively, decision surface can be derived directly from labelled samples which is then called a “Discriminant Function”, perceptron being an example. Then, most if not all current ML techniques can be seen as dueling methods to derive Discriminant Functions!

Discriminant Functions can be linear or nonlinear (neural network with back-propagation, deep learning, support vector machines, kernel PCA, etc.) and outputs can be binary, integer or real valued. Various learning algorithms can be seen as belonging to the family of iterative/ recursive/ adaptive learning algorithms (Least Mean Square being a great old standby!) that update the parameters of the Discriminant Function as new data arrive.

In the discussion above, features were considered as “static” and not context-sensitive (for identifying a word within a sentence as an example). Context-sensitivity or Dynamics can be added to improve classification by incorporating Markov models (or Hidden Markov Models for tractable computations). Markov model is a special case of State Space Models which are well-studied in Systems Theory.

Setting aside Supervised Classification when labelled samples (or “desired signals”) are available, what can we do when there is no supervision? This is the realm of much harder Unsupervised Learning, which is very useful in transforming basic features into more and more meaningful ones. One usually brings in some overall desirable property to guide unsupervised learning. From the domain of “blind processing” (Radar signal processing, for example), Mutual Information among classes can be minimized as a learning process in the belief that the “best” classification happens when the classes have least overlapping information (better “efficiency” in representation).

Instead of entropy-related quantities that are hard to estimate, it is likely that Scale of Fluctuation which is related to “order” and “state space volume” may be a quantity to optimize for a new unsupervised learning process. (For more information on Scale of Fluctuation, refer to my papers, “Instantaneous Scale of Fluctuation Using Kalman-TFD and Applications in Machine Tool Monitoring”, 1997 & “Kalman Filtering and time-frequency distribution of random signals”, 1996).

In all of the existing ML bags of tricks, we are still staying at the surface level! We are modeling the attributes or data DIRECTLY. What if we went one level deeper? Model the SYSTEM that generates the data! Syzen Analytics, Inc., takes such an explicit approach in what we call “SYSTEMS” Analytics” which has already demonstrated significant value in business applications.

In Syzen’s retail commerce application, our Systems Analytics approach hypothesizes that there is a system, either explicit or implicit, behind the scenes generating customer purchase behaviors and purchase propensities. This ‘one-level-deeper system model parameters’ can be more effective for pattern recognition and classification purposes instead of the data that the model generates! There is a long history of model parameters providing better estimates (in power spectrum analysis, for example). Scale of Fluctuation mentioned earlier seems to have another desirable property of quantifying “coupling” among deeper-level model parameters.

Context-sensitivity dynamics is a very good avenue to exploit. The dynamics could be over any independent variable (time always comes to mind first but it is only one of the possibilities). As I noted in my recent blog (“SYSTEMS Analytics – the next big thing in Big Data & Analytics”), “Extensions to Systems Analytics in the future will be inspired by the insight that in reality, data exist in *embedded* forms in preference and influence networks which are distributed in time and space” AND other independent dimensions (shopper preference, for example).

Let me pull all of the notions discussed so far into a diagram.

Once the patterns have been recognized and classes identified, the resulting classes can be used for all sorts of applications such as Recommendation Engine, Language Translation, Fraud Detection and many others. The approach I outline above allows you to take a unified approach till the application development stage. In doing so, the unified approach also points out new paths ahead for ML!

Some readers would have noticed an undertow of dichotomies while reading this “opinion piece”: Theoretic vs Heuristic; Formal vs Ad hoc; Mathematics vs AI; Electrical Engineering vs Computer Science academic departmental affiliations! I am firmly in the former camps. However, as an engineer, I am personally happy to start with heuristic solutions but quickly put them on firm mathematical foundations before “gotchas” and unintended consequences of ad hoc methods catch up with me. 

The unification of ML proposed here opens up a multilane highway – join the journey and create more breakthroughs with us or on your own!

In this blog, I have not provided many references – web search will get you most; Pedro Domingos’ “The Master Algorithm” book is an excellent source of ML-related literature. For the newer and less familiar work, please contact me directly.

“IA not AI” in Retail Commerce – Enhanced Tanpin Kanri

Highlights: Enhanced Tanpin Kanri is a specific example of Intelligence Augmentation. Store staff's local knowledge and engagement with their shoppers cannot be replaced; but Big Data & Analytics can provide a significant leg-up in their difficult job of hypothesis-generation by providing data-driven predictions that they can safely rely on and improve incrementally. In a highly competitive low-margin business such as fast moving consumer goods retail, pioneering use of IA in their operations will determine the winners.

Dr. PG Madhavan is the Founder and Chairman of Syzen Analytics, Inc. He developed his expertise in analytics as an EECS Professor, Computational Neuroscience researcher, Bell Labs MTS, Microsoft Architect and startup CEO. He has been involved in four startups with two as Founder. He has over 100 publications & platform presentations to Sales, Marketing, Product, Industry Standards and Research groups and 12 issued US patents. He conceived and leads the development of SYSTEMS Analytics and is continually engaged hands-on in the development of advanced Analytics algorithms.

More at www.linkedin.com/in/pgmad

Artificial Intelligence or “AI” is the technology of the future; it has been so for the past 50 years . . . and it continues to be today! Intelligence Augmentation or “IA” has been around for as long. IA as a paradigm for value creation by computers was demonstrated by Douglas Engelbart during his 1968 “Mother of all Demos” (and in his 1962 report, “Augmenting Human Intellect: A Conceptual Framework”). While Engelbart’s working demo had to do with the mouse, networking, hypertext, etc. (way before their day-to-day use), IA has increased in scope massively in the last 5 plus years. Now, Big Data & Analytics can truly augment human intelligence beyond anything Engelbart could have imagined.

IA is contrasted with Artificial Intelligence which in its early days was the realm of theorem proving, chess playing, expert systems and neural networks. AI has made large strides but its full scope is yet to be realized. IA on the other hand can be put to significant use and benefit today. Such is the story of Enhanced Tanpin Kanri in retail commerce.

As a keen observer of the retail ecosystem’s demand chain portion for the past 3 years or so, I am struck by the inefficiencies in large portions of Retail. Shoppers finding what they want on the shelf is considered a BIG problem in the industry (the so called “OOS problem” or Out-Of-Stock problem) – to the tune of $170 Billion per year! The following diagram illustrates the portion of Retail ecosystem on which we will focus in this blog.

The left half represents the current dominant model. “Push” model drives manufacturer’s FMCG (fast moving consumer good from “brands” such as Procter & Gamble, Kraft, Unilever and others) into the supply chain with store shelves as the final destination. As retailers move to the right half of the picture, they tend to be more agile and mature in their practices and are seeking a competitive edge through differentiation – they find that by focusing their store operations to satisfy the LOCAL customer via making available what she prefers on the store shelves, they can win. The poster-child of this revolution is 7-Eleven, the ubiquitous store at virtually every street corner around the globe!

7-Eleven’s use of the “Pull” model since early 2000’s has been very successful as captured in a case study at Harvard Business School in 2011 (HBS Case Study: 9-506-002 REV: FEBRUARY 23, 2011). Quoting from this study, “Toshifumi Suzuki, Chairman and CEO of Seven and I Holdings Co., was widely credited as the mastermind behind Seven-Eleven Japan’s rise” and goes on to say that ‘Suzuki’s emphasis on fresh merchandise, innovative inventory management techniques, and numerous technological improvements guided Seven-Eleven Japan’s rapid growth. At the core of these lay Tanpin Kanri, Suzuki’s signature management framework’.

Proof is in the pudding – “Tanpin Kanri has yielded merchandising decisions that has decreased inventory levels, while increasing margins and daily store sales” since the 2000’s, the HBS case study points out. So what exactly is Tanpin Kanri?

Tanpin Kanri or "management by single product," is an approach to merchandising pioneered by 7-Eleven in Japan that considers demand on a store-by-store and product-by-product basis. Essentially, it empowers store-level retail clerks to tweak suggested assortments and order quantities based on their own educated hypotheses . . . 

You can tell that Tanpin Kanri lies well to the right in the Retail Ecosystem diagram above. To call out some features:

·       PULL model

·       For a buyers’ market

·       Focus on How to satisfy customer

·       Item planning and supply driven by retailer and customer

·       Symbolic of Consumer Initiative

I believe that it is simply a matter of time before Tanpin Kanri variants dominate the Retail demand chain model. As shoppers clamor even more for their preferences to be made available, Retailers will evolve incrementally.

So, where does the PULL Model provide most bang for the buck today?

Where ever Product Density (number of products to be stocked per unit area) is high, “customer pull” will help prioritize what products to stock. Today’s Tanpin Kanri at 7-Eleven Japan accomplishes “customer pull” incorporation via super-diligent store staff manually making the choices.

Here are the “hypothesis testing” steps that 7-Eleven store staff goes through in operationalizing Tanpin Kanri. Based on frequent interactions and personal relationships with the shoppers at a store, the staff generates hypotheses of the shoppers’ needs, wants and dislikes. Based on such information, store staff formulates hypotheses of what to carry (or not) on their store shelves (“merchandising”). Sales during the following days and weeks allow them to ascertain if their hypotheses should be rejected or not; this continuous iteration goes on over time to “track” shopper preferences. Clearly, Tanpin Kanri methodology has been highly successful for 7-Eleven according to HBS case study.

IA based on Big Data & Analytics can play a major role in Tanpin Kanri. In any scientific methodology, coming up with meaningful hypotheses is the HARD part! In Tanpin Kanri case, it is the personal relationships, diligence and intelligence of the store staff that help generate the hypotheses. This is the super-important human value-add that no AI can fully replace! However, we can AUGMENT the hypothesis-driven Tanpin Kanri with Data-Driven precursor that enhances staff intelligence by providing them with predictions that they can build on to formulate their hypotheses.

IA happens in the data-driven precursor step. 7-Eleven has vast amounts of transaction and customer data in their data warehouse. They can be “data-mined” to find shopper preferences at a particular store which can form the basis of what to carry and how much on the store shelf. The data-driven predictions then become the “foundation” on which the store staff adds their own “deltas” based on the shopper quirks that they have surmised through their all-important personal relationships.

Syzen Analytics, Inc. has accomplished IA integration using Machine Learning and a new development in Analytics called “SystemsAnalytics”. A dumb “prediction” for SKU shares is same sales as last year (see the multi-colored bar chart in the middle of the picture below) – in other words, historical sales is the “information-neutral” prediction. But surely, we can do better than that with Systems Analytics.

Syzen is able to provide SKU-by-SKU, store-by-store and week-by-week predictions using typical T-Log data that every Retailer has in its data archives. A typical prediction of Syzen’s ROG-0 SaaS product for a typical SKU at a particular store looks like this.

·       The purple uneven “picket fence” is the weekly predictions – lowest bar chart. This is obtained by combining different “masks”.

·       The papaya-colored mask is the new and most significant one. The values are predicted based on appropriate past intervals of T-Log data digested via Systems Analytics and updated adaptively.

·       The numbered masks in the middle accounts for things that the store manager knows that will happen next year such as a local festival or a rock concert in the nearby park.

·       The MANUAL part of Tanpin Kanri now only involves the store staff simply making daily small adjustments to the SKU “facings” based on local shopper “gossip” to the purple bar chart!

Convenience stores are drawn to the Enhanced Tanpin Kanri method because of the maturity of operations they already possess. With more agile supply chains and the desire to differentiate their stores in response to their local clientele, Syzen finds a lot of enthusiasm among “high-density” Retailers for our predictive solution that makes Tanpin Kanri more scalable due to lesser dependency on super-diligent store staff. Advances in Systems Analytics and other quantitative methods will refine products such as Syzen’s ROG-0 SaaS in the future to sharpen shopper-preference based product assortment predictions.

Enhanced Tanpin Kanri is a specific example of Intelligence Augmentation. Store staff's knowledge of local happenings and engagement with their store shoppers cannot be replaced; but Big Data & Analytics can provide a significant leg-up in their difficult job of hypothesis-generation by providing data-driven predictions that they can safely rely on and improve incrementally. In a highly competitive low-margin business such as fast moving consumer goods retail, pioneering use of IA in their operations will determine the winners.

More on ROG-0 SaaS: http://advancingretail.org/product/syzen-rog-0-saas/

Syzen website: www.syzenanalytics.com

“Tanpin Kanri” is the next “Kaizen”!

“Tanpin Kanri” is the next “Kaizen”!

Does this spell the death of Retail Category Management as we know it?!

Highlights: Tanpin Kanri will be the next big business process/ philosophy export out of Japan after Toyota’s “kaizen”. Why? Tanpin Kanri has yielded merchandising decisions that has decreased inventory levels, while increasing margins and daily store sales”, says an HBS case study. Syzen is finding a lot of enthusiasm among “high product density” Retailers for our predictive analytics SaaS solution that makes Tanpin Kanri scalable.

Dr. PG Madhavan is the Founder and Chairman of Syzen Analytics, Inc. He developed his expertise in analytics as an EECS Professor, Computational Neuroscience researcher, Bell Labs MTS, Microsoft Architect and startup CEO. He has been involved in four startups with two as Founder. He has over 100 publications & platform presentations to Sales, Marketing, Product, Industry Standards and Research groups and 12 issued US patents. He conceived and leads the development of SYSTEMS Analytics and is continually engaged hands-on in the development of advanced Analytics algorithms.

More at www.linkedin.com/in/pgmad

As a keen observer of the retail ecosystem’s demand chain portion for the past 3 years or so, I am struck by the inefficiencies in large portions of Retail. Shoppers finding what they want on the shelf is considered a BIG problem in the industry (the so called “OOS problem” or Out-Of-Stock problem) – to the tune of $170 Billion per year! There are many culprits – let us consider the systemic ones.

The following diagram illustrates the portion of Retail ecosystem on which we will focus in this blog.

The left half represents the current dominant model. “Push” model drives manufacturer’s FMCG (fast moving consumer good from “brands” such as Procter & Gamble, Kraft, Unilever and others) into the supply chain with store shelves as the final destination. As retailers move to the right half of the picture, they tend to be more agile and mature in their practices and are seeking a competitive edge through differentiation – they find that by focusing their store operations to satisfy the LOCAL customer via making available what she prefers on the store shelves, they can win. The poster-child of this revolution is 7-Eleven, the ubiquitous store at virtually every street corner around the globe!

7-Eleven’s use of the “Pull” model since early 2000’s has been very successful as captured in a case study at Harvard Business School in 2011 (HBS Case Study: 9-506-002 REV: FEBRUARY 23, 2011). Quoting from this study, “Toshifumi Suzuki, Chairman and CEO of Seven and I Holdings Co., was widely credited as the mastermind behind Seven-Eleven Japan’s rise” and goes on to say that ‘Suzuki’s emphasis on fresh merchandise, innovative inventory management techniques, and numerous technological improvements guided Seven-Eleven Japan’s rapid growth. At the core of these lay Tanpin Kanri, Suzuki’s signature management framework’.

Proof is in the pudding – “Tanpin Kanri has yielded merchandising decisions that has decreased inventory levels, while increasing margins and daily store sales” since the 2000’s, the HBS case study points out. So what exactly is Tanpin Kanri?

Tanpin Kanri or "management by single product," is an approach to merchandising pioneered by 7-Eleven in Japan that considers demand on a store-by-store and product-by-product basis. Essentially, it empowers store-level retail clerks to tweak suggested assortments and order quantities based on their own educated hypotheses . . . 

You can tell that Tanpin Kanri lies well to the right in the Retail Ecosystem diagram above. To call out some features:

·       PULL model

·       For a buyers’ market

·       Focus on How to satisfy customer

·       Item planning and supply driven by retailer and customer

·       Symbolic of Consumer Initiative

Looking at this list, you may be flabbergasted if I told you that this is NOT how all of Retail works! You may be wondering with all the emphasis on satisfying the customer and ours being a “buyers’ market” in virtually every case, why would any model other than varying shades of Tanpin Kanri exist in the decade of 2010?! That is exactly why I claim that there is an “inevitability” about Tanpin Kanri. And I conclude that Tanpin Kanri will be the next big business process/ philosophy export out of Japan after Toyota’s “kaizen”.

Push model exists virtually everywhere in US grocery retail, super and hyper markets, smaller store chains. The method of “Category Management” referring to a joint-decision process by CPG (consumer product goods) manufacturers and retailers determine what will be on the store shelves for the next 6 or 12 months. Often, all the stores in a chain (some as large as 3000 stores) will have the same “planogram” (a visual representation of product SKUs on the shelves) irrespective of the local clientele! Obviously simple to implement, why has Category Management not evolved to reflect customer preferences explicitly.

A FederalTrade Commission study may hint at the underlying issue; to quote, “The retailer and supplier also typically discuss funds – slotting, promotional, co-op advertising, or other introductory allowances or discounts – some of which would lower the retailer’s per unit purchase cost for an initial period of time”. As is obvious, this exchange of funds can short-change the customer since they are not directly part of the equation! CPG’s are hyper-wealthy compared to razor-thin margin Retailers – that the CPGs can improve the Retailer’s bottom-line directly with these funds will have a distorting effect on commerce due to the joint-decision process of Category Management.

I believe that it is simply a matter of time before Tanpin Kanri variants dominate the Retail demand chain model. If another FTC study finds the exchange of funds between CPGs and Retailers as collaboration of an "unsavory" nature, the transition to Tanpin Kanri will be rapid. If not, as shoppers clamor even more for their preferences to be made available, Retailers will evolve incrementally kicking and screaming – ultimately, we the shoppers pay them more money than CPGs!

So, where does Tanpin Kanri provide most bang for the buck today?

Where ever Product Density (number of products to be stocked per unit area) is high, “customer pull” will help prioritize what products to stock. Today’s Tanpin Kanri at 7-Eleven Japan accomplishes “customer pull” incorporation via super-diligent store staff manually making the choices. Surely, in these days of Big Data and Analytics, there must be a way to provide a helping hand to the store staff to predict “customer pull” or as we call it “Shopper Preferences” . . .

Syzen Analytics, Inc. has accomplished exactly that using Machine Learning and a new development in Analytics called “SystemsAnalytics”. Syzen is able to provide SKU-by-SKU, store-by-store and week-by-week predictions using typical T-Log data that every Retailer has in its data archives. A typical prediction of Syzen’s ROG-0 SaaS product for a typical SKU at a particular store looks like this.

·       The purple uneven “picket fence” is the weekly predictions. This is obtained by combining different “masks”.

·       The papaya-colored mask is the new and most significant one. The values are predicted based on appropriate past intervals of T-Log data digested via Systems Analytics and updated adaptively.

·       The numbered masks in the middle accounts for things that the store manager knows that will happen next year such as a local festival or a rock concert in the nearby park.

·       The MANUAL part of Tanpin Kanri now only involves the store staff simply making daily small adjustments to the SKU “facings” based on local shopper “gossip”!

As we discussed, many retailers are NOT ready to embrace periodic refinement to reflect shopper-preferences, especially if it involves foregoing lucrative fees from CPGs. However, the “old timers” can take advantage of the outputs above for old-fashioned assortment optimization – average the height of the purple figure and use the average as the SKU volume that they order every week; our result is store specific but they can average each SKU over all stores in the chain and generate one Planogram. Of course, they will be throwing out a lot of value (and revenue) by not being responsive to customer preferences on a store-by-store and SKU-by-SKU basis!

Convenience stores are drawn to this “predictive” Tanpin Kanri method because of the maturity of operations they already possess. With more agile supply chains and the desire to differentiate their store in response to their local clientele, Syzen is finding a lot of enthusiasm among “high-density” Retailers for our predictive solution that makes Tanpin Kanri more scalable due to lesser dependency on super-diligent store staff. Advances in Systems Analytics and other quantitative methods will refine products such as Syzen’s ROG-0 SaaS in the future to sharpen shopper-preference based product assortment predictions.

More on ROG-0 SaaS: http://advancingretail.org/product/syzen-rog-0-saas/

Syzen website: www.syzenanalytics.com

X-Event Marketing

Dr. PG Madhavan developed his expertise in Data Analytics as an EECS Professor, Computational Neuroscience researcher, Bell Labs MTS, Microsoft Architect and startup CEO. Overall, he has extensive experience of 20+ years in leadership roles at major corporations such as Microsoft, Lucent, AT&T and Rockwell and startups, Zaplah Corp (Founder and CEO), Global Logic, Solavei and SymphonyEYC He is continually engaged hands-on in the development of advanced Analytics algorithms and all aspects of innovation (12 issued US patents with deep interest in adaptive systems and social networks). More at www.linkedin.com/in/pgmad

This blog is a coming together of a bunch of my past blogs and my reading of a recent book by John Casti called “X-Events”.

PG Blog: “What does ‘Emergent Properties in Network Dynamics’ have to do with Shopping?”; http://pgmadblog.blogspot.com/2012/10/what-does-emergent-properties-in.html?m=1

PG Blog: “Network Dynamics & Coupling: Shannon’s Reverie Reprised . . . “ http://pgmadblog.blogspot.com/2012/11/network-dynamics-coupling-shannons.html?m=1

Book by John Casti, "X-Events: The Collapse of Everything";

http://www.amazon.com/X-Events-Collapse-Everything-John-Casti/dp/B00BJEP6B4

Reading X-Events book triggered ideas from some of my past academic work (http://www.jininnovation.com/SoF.PDF) combined with my present startup, “Syzen Analytics” (http://www.SyzenAnalytics.com/), leads me to suggest an interesting way to create “pocket” X-events in Retail Commerce via X-Event Marketing or “XM”!

Let us start at the beginning . . .

1.     Shannon understood the importance of rare events in conveying information. He sought a mathematical formulation to capture this important property of our brains and came up with the definition of “entropy”. As I had made clear in the original blog, this is a fictitious reverie I made up to give one the guts to turn other valuable physical/ practical insights into mathematical forms!

2.     Similarly, from my past research in Neuroscience, I had some strong impressions that have refused to go away despite the passage of decades. One is the activity that precedes the most significant X-Event in each of our lives, Death! Unreplicated and unpublished observations in our neurophysiology labs had shown that one of the surprising events that happen before a mouse with brain in-dwelling electrodes dies is the all-out firing of “complex spike cells” (which when the mouse is alive and performing “place cell” tasks are painfully difficult find and record). In other words, the “natural state” of the brain seems to be excitatory and the work of a healthy brain seems to be to assert control and coordination by keeping the *inhibitory tone* high so that all hell does not break loose (by cells firing away with no coordination or control). I speculate a regime-transition of the sort shown below in death: the right-most picture is indicative of brain cells firing away in an uncontrolled fashion (just before a major “X-Event”, death or epileptic seizure) with the left-most indicative of a properly functioning brain (“nicely” coupled). The next story will illuminate the idea of the middle picture where the “system is uncoupled” with the system poised for all its energy to pile into a single “mode” and create a major suppression of inhibition in the brain.

3.     Following “Shannon’s approach”, how do I make these intuitions mathematical? Let us look at the right-most picture. As I outlined in my “Shannon’s Reverie Reprised” blog, everything is firing away and the potential field will show up equally across the scalp, much like a single “wavefront” that reaches all the regions of the scalp simultaneously. Imagine you are at a beach looking out to the sea and gentle waves are rolling in – let us say in parallel to the beach. If you are standing knee-deep in the water and look in a direction parallel to the beachfront (i.e., up or down the beach), the spatial frequency in your “look direction” (or the frequency of “corrugation”) is 0 cycles/meter! Much like the ocean waves, the single wavefront of activity is nearly “constant” across the distributed neocortex and its relevant strength can be thought of as the power at zero frequency. I happen to know that power at zero frequency is called “Scale of Fluctuation” or “θ” in random field theory. From the previous work of Eric Vanmarcke (“Random Fields”, 1983, with an updated edition in 2010), θ is defined below. I refer you to my blog and academic paper mentioned at the outset for the gory details!

 

Is Theta as powerful and useful as Shannon’s entropy . . . time will tell. As you see in my academic publication, Theta does have some intriguing properties in the case of 2^nd order linear time-invariant systems which may indeed prove useful in the future! Adding results from my simulation and data analysis of machine tool chatter, my summary observation is that:

Large value of Theta can be good or bad. It indicates “Coupling”: *good* coupling as a nicely functioning brain (or lathe) or a *bad* coupling as in death (or chatter)! The key here is that low value of theta can be a “predictor” of impending X-Event!

As you can imagine, if our speculation above holds up, we may be able to predict X-Events (or Taleb’s “black swans”) such as 2008 Great Recession or Tohoku earthquake and eventual tsunami. In the case of Tohoku, a few extra hours of warning may have helped Fukushima engineers reach a consensus and move the power generators to a higher location (thus avoiding the nuclear disaster).

XM:

In this blog on X-Event Marketing (“XM”), my purpose is different. I want to *create* desirable, “pocket” X-Events to beneficial ends!

One of the preconditions of “interesting” dynamics in systems is sufficient “complexity”. This can be taken to mean multiple actors with high degrees-of-freedom, dense interconnections with linear and non-linear coupling and in general, hard to analyze! One such natural system is the “shopper network”.

A modern-day shopper is enmeshed in an ever-varying network of preferences and influences (from friends and family) and embedded in a network distributed in time and space – just the type of complex systems where “interesting” dynamics can arise. We want to create desirable “pocket” X-Events in the shopper network with the purpose of increasing business value for the shopper or the seller or both.

This is the so-called “Purchase Funnel” that an individual shopper goes through when executing a product transaction. The marketing, merchandizing and offer activities are devised to impact the shopper at predetermined points of importance in the shopper’s decision making process. For example, a well-conceived offer delivered via shopper’s mobile phone at the point of “Desire” may precipitate a Purchase “Action” – a win for the seller (in this case both the retailer and the manufacturer of that product; for example, Walmart and Procter & Gamble, respectively).

This was one shopper. Now, you have to consider the shopper being embedded in a network. At this point, it may be useful for you to review my earlier blog, “Social Network Theory” http://pgmadblog.blogspot.com/2013/08/social-network-theory.html, where Barabasi’s work in Network Graphs and the following terms are discussed:

1. Clusters (of friends who “hang out”).

2. Weak links (to your long-forgotten high school classmates).

3. Hubs (politicians and others with massive number of contacts).

Clusters and hubs are some examples of the influence on an individual shopper. There are many such shoppers, all interconnected, when studying the dynamics with a view to creating a “pocket” X-Event. Even this does not complete the complexity picture – the shoppers are distributed in space (you are in NYC and your buddy from Seattle tweets about a cool product that he bought) and time (good and bad product reviews reaches you at various times and not in a synchronized fashion).

That the Shopper Network is sufficiently complex to engender “interesting” dynamics must be beyond a doubt by now! The picture above may be a reasonable representation of the Shopper Network (WITHOUT the time element captured – we will need a video showing the undulations in the ”heat map”). The “heat” shown as ‘yellow’ spots indicate the total dollars spent each day on a particular product on a particular day, aggregated to a county for all of USA. Clearly, this will vary from day-to-day.

How do we capture the dynamics of the Shopper Network picture shown above in a tractable manner so that we can understand the dynamics and then track the effect of any manipulations we do in the network so that we can close the loop to adaptively adjust our manipulations to achieve a desired effect?

This is the essential question of XM. Of course, there are other important questions such as the type and timing of manipulations (Offers to selected customers? A country-wide branding effort? Is it a new product or an existing one? And so on). As a systems engineer, I will find a partner who is more qualified than I am in answering the marketing questions; I will focus on Systems Analytics tools to create a quantitative execution infrastructure to deploy marketeers’ ingenuity!

Engineers love scalars! I suspect it is because having a single control variable is easier to manage than many. Probability theory is replete with reduction to scalars – I am talking about mean, variance, correlation coefficient, Entropy, Theta, . . . The full underlying information is available in the joint probability density functions but they are a bear to handle! So, we reduce it to a scalar – clearly, in the process, we have thrown away MUCH information but what is left is what we plan to use; so our justification for selecting and using a scalar for a particular engineering task is very *operational*.

We will use the scalar, Theta, to answer Shopper Network analysis and control question I asked a few paragraphs earlier.

Scenario: New Product Introduction

Procter & Gamble is going to introduce a new detergent and wants to create a ground-swell of purchase interest (our “pocket” X-Event).

·         P&G introduces the product quietly and collects the heat-map data from T-log data of their partner Retailers.

·         Sysan (our startup) processes the heat-map data: total dollar spent each day on the new detergent, aggregated to a county for all of USA.

·         Sysan calculates Theta (a single number) as the baseline measure.

·         P&G Marketing has developed 2 major tools for the launch of the new detergent: (1) nation-wide TV ad campaign and (2) a 10% cash discount for the first 1 million customers in US.

·         P&G runs a 1-week TV ad campaign and while collecting T-log data.

·         Sysan monitors Theta for the week.

·         P&G suspends the TV ad campaign and starts the cash discount offer.

·         Sysan monitors Theta and finds that Theta is entering a “Decoupled” regime with low values of Theta.

·         Sysan advises P&G to resume the TV ad campaign in an effort to trigger “bad” coupling among various counties of US.

·        This pushes Theta into a high-value, the coupling between counties get tight and a “pocket” X-Event occurs where there is a ground-swell of purchases of the new detergent country-wide!

NOTE that in this fictitious example, P&G would have pretty much done the various marketing campaigns that I speculate here on their own. The KEY point is that P&G does NOT have a scalar (or any) measure to get immediate feedback regarding their efforts, combined for the whole country. The ability to quantify “closed-loop” intervention is the value of our humble scalar, Theta, in X-Event Marketing!

Clearly, what manipulations are best applied to the network and when is an open question. Utilizing Theta, Sysan is developing methods using “massive simulation methods” (for background, please see: John Casti, “Would-Be Worlds: How Simulation is Changing the Frontiers of Science”) that can predict the effect of various interactions at various instances of time and points in space, thus providing guidance for the most effective use of vast sums of money required for ad campaigns and discount offers.

Stay tuned for our initial results . . .

PG