IMB 621

Kiran R, Doctoral Student, Indian Institute of Management Lucknow, Arunabha Mukhopadhyay, Associate Professor, Indian Institute of

Management Lucknow, and U. Dinesh Kumar, Professor of DSIS, Indian Institute of Management Bangalore prepared this case for class

discussion. This case is not intended to serve as an endorsement, source of primary data, or to show effective or inefficient handling of decision

or business processes.

Copyright © 2017 by the Indian Institute of Management Bangalore. No part of the publication may be reproduced or transmitted in any form or

by any means – electronic, mechanical, photocopying, recording, or otherwise (including internet) – without the permission of Indian Institute of

Management Bangalore.

MACHINE LEARNING ALGORITHMS TO DRIVE CRM

IN THE ONLINE E-COMMERCE SITE AT VMWARE

KIRAN R, ARUNABHA MUKHOPADHYAY AND U DINESH KUMAR

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 2 of 16

On February 25, 2016, in the VMWare (VMW) office in Silicon Valley, next to Stanford University, in

the sprawling 100+ acre green campus in Palo Alto, California, winter had just ended and it was warm

weather, as great as it possibly could be in February. In his office cabin in building Hilltop E, Michael

Butler, the global head of the store business of VMW was in discussion with Parag Girish Chitalia, the

global leader for advanced analytics and data sciences. Michael and Parag were discussing how to drive

more revenues from Workstation business in the VMW store. The VMW store was the online portal of

VMW (store.vmware.com), where end-customers could purchase certain products of VMW such as

Fusion and Workstation online. The store was similar to any e-commerce site with a home page, category

pages, and product detail pages, add to cart pages, checkout page and a confirmation of order page.

Fusion helped end-customers and businesses run Windows on top of Mac machines, whereas Workstation

helped customers run Mac on top of Windows machines. Since many customers would like to have both

Windows and Mac operating systems on their computers, VMW store received many visitors to its

website. Data on customer’s usage of VMW store is collected to understand consumer behavior. With

rich behavioral data of the VMW website, Michael Butler was keen to see how the data sciences and

analytics team could be leveraged to drive further Workstation sales as it was a key product in the

competitive business environment.

ABOUT VMWare

VMware (VMW) has been a Palo Alto headquartered software company that reported USD 6.57 billion in

2015, up 9% from 2014. VMW has been one of the most profitable software companies in history with

GAAP net income of approximately USD 1 billion in 2015. Cash flows were healthy as well with free

cash of USD 1.56 billion generated in 2015 (Exhibit 1). Founded in 1998 by Stanford Professors Diane

Greene and Mendel Rosenblum, the company was headed by Pat Gelsinger in 2016 and had more than

18,000 employees worldwide. VMW has been the industry leader in virtualization business with more

than 80% market share. Virtualization is about using software to virtualize hardware – for example, the

same central processing unit (CPU) can be shared by multiple users using the VMW software.

Virtualization brings about great savings in costs to IT departments of companies and VMW has been the

industry leader by a distance in this space with market share several times that of its nearest competitors.

VMW garnered its revenues from three streams namely software defined data center (vSphere – for

computing virtualization, NSX – for software defined networking & security, VSAN – for storage

virtualization), end-user computing (Airwatch – for mobile computing, Horizon – enterprise desktop,

Fusion, Workstation), and cloud (Private cloud vCloud Air).

Michael Butler was in charge of the store (Exhibit 13) business powered by Fusion and Workstation

products. Parag had joined VMW in 2014 to set up the advanced analytics and data sciences team called

Analytics Community of Excellence under the Information Innovation Center/Enterprise Information

Management organization. The team comprised data scientists and analysts hired from premier institutes

in India such as the Indian Institute of Technology (IITs) and Indian Institute of Management (IIMs) and

from around the world such as Georgia Tech and Stanford. Ravi Kondapalli was the lead data scientist in

the data sciences innovations team powering Parag’s team. Ravi, a NIT Warangal grad with double

Masters from Georgia Tech and IIM Bangalore had more than 15 years of experience in the industry.

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 3 of 16

Driving Higher Workstation Revenues from the Store

The primary objective of the meeting between Parag and Michael was to discuss how Parag’s newly

formed data sciences group could assist in increasing store revenues with focus on key products starting

with Workstation. Michael started the meeting by saying:

Workstation forms the bulk of the purchases for our online store/e-commerce business for

which we have both individual consumers and businesses as our customers. Growing

revenues this year will be a challenge as there is no new version of Workstation planned.

In a software business, renewals via upgrade to a latest software version form a major

portion of the revenue and this year will be a challenge. I would like to understand how

we can leverage data sciences and advanced analytics to target new workstation

customers, up-sell to existing customers, cross-sell to customers that do not have

Workstation.

Parag shared some macro-level data on Workstation sales that Ravi, his lead data scientist in the data

sciences innovations team, had compiled. Workstation revenues had doubled in the last 8 years

(Exhibit 2) and formed a significant portion of the Overall Store Bookings (Exhibit 3). Different

versions of Workstation had been launched over the years. Workstation 6 was launched in 2007 and the

latest versions of the Workstation product were Workstation 12 and Workstation 12 Player. Significant

portion of VMW Workstation customers upgraded to higher versions of Workstation. In Exhibit 4, each

cell xij in the table denotes the number of customers that upgraded from Workstation version in the row i

to the Workstation version in the column j. There was an opportunity in the sense that a large number of

the customer base had not yet upgraded to the latest versions of Workstation. The store was visited by

approximately 7 million visitors annually of whom approximately 2 million viewed some page related to

Workstation products. However, only around 1.6 million visitors out of the 7 million were identifiable

with an e-mail id (Exhibit 5). The visitor data contained rich clickstream/digital data that was housed in a

Hadoop big data environment that the analytics team leveraged continually for their analysis. Apart from

this visitor behavior, all previous purchases (if any) by the e-mail ids were stored in the Greenplum data

warehouse. Greenplum is a massively parallel database and owned by Pivotal that has proven to be better

than Teradata, Oracle, and other data warehouses. Online–offline integration for the de-anonymized

visitors was possible with “e-mail id” as the common inter-linking key.

Parag had driven the following key points to lay the ground for a discussion on analytics engagement.

 Workstation was going to be an important driver of the overall store revenues.

 There was untapped opportunity in the form of the old Workstation customers that had not yet

upgraded to the latest version of Workstation, presenting an opportunity for up-sell.

 There were a large number of visitors to the online store that included those that had bought other

store products presenting an opportunity for cross-sell.

 The data sciences and analytics team had access to rich sets of information about the customers

and also the potential customers including their digital footprint (online) and their purchase

history (offline).

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 4 of 16

Being a sales leader, Michael liked the key points. He got straight to the point:

We definitely have a great Customer Relationship Management (CRM) opportunity here

in the form of up-sell, cross-sell and targeting. These present multiple challenges that I

and my management team will go into in detail. For example: While I can drive

incremental sales with coupons, I would want to give the coupons only to those customers

that are most likely to buy and not indiscriminately to all.

Can your team provide me with the list of the email ids most likely to purchase our latest

products Workstation 12 or Workstation 12 Player in the next 3 months so that my team

can target these email ids?

Parag immediately proposed a propensity model as a quick win. A propensity model rank ordered e-mail

ids or customers in their decreasing order of likelihood to purchase. His advanced analytics team powered

by the data sciences innovations team had delivered great results in the past by the usage of these models.

This propensity model could leverage the online and offline data for the e-mail ids and rank order them

using machine learning techniques.

Michael said:

That’s awesome Parag! I only believe things that cause an increase in my sales! If you

can create such a list, I will be happy to execute via one of the digital marketing channels

(email with coupon, re-targeting on other websites, social targeting) or by targeting on

our website.

I will believe your list only when the cash machine rings up Workstation sales and when I

can measure the upside scientifically.

Michael was a technology geek and would only believe things once they were scientifically proven. Parag

said he would get back to Michael with a propensity scored list within a couple of weeks. The presence of

Ravi in the team gave Parag the confidence to suggest two weeks.

PROPENSITY MODEL DEVELOPMENT

Through e-mail, Parag briefed Ravi, the lead data scientist to be ready with what it would take to build a

propensity model and also to brainstorm on what should be the overall data sciences plan that was to be

presented to Michael. They had a detailed telephonic conversation the next day.

Ravi set the baseline for the discussion:

This is an example of a binary classification problem, where the visitor either buys or

does not buy Workstation. The target variable will be if a visitor who visited the site buys

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 5 of 16

Workstation in the next few months. The value of that target can be either 0 or 1, making

it a classical binary classification problem.

Ravi went through a deck which highlighted the following challenges.

 What should be the entity on which we should build a propensity model? As Exhibit 5 shows,

only about 1.6 million out of about 7 million visitors had an e-mail id.

 We should decide on the sampling strategy, Should we use random sampling, time-based

sampling or stratified random sampling?

 What data sciences and machine learning techniques should we try out in this instance?

 What cross-validation or training-validation technique should we use in order to have an estimate

of how the model would perform in the real world?

Ravi’s recommendations to Parag were as follows:

 Given a quick win, we should model on e-mail id level for the first cut. Longer term, we have to

think of analytical approaches to target those without an e-mail id.

 There is only one right way to perform cross-validation. In this instance, we should do time-based

cross-validation. In this method, we simulate the real world by aggregating data to a period and

then predicting for the next period.

o For example: Say we need to predict who will buy during April–June 2016. In this instance:

 For training, we could aggregate data up to September 2015 and predict the Workstation

buyers during October–December 2015.

 For validation, we could aggregate data up to December 2015 and compare the

predictions against actual Workstation buyers during January–March 2016.

 For scoring, we could aggregate the data up to March 2016.

 We could try any 2-class classifier such as Naïve Bayes, Logistic Regression, Decision Tree, or

machine learning algorithms such as Random Forest, Gradient Boosting, etc. We could compare

the lift curves of different models to see which one would work best.

 We could use the lift numbers on the validation set to obtain an estimate of the real world.

Ravi further explained the time-based cross-validation using the following conceptual diagram.

Ravi said he could build the model in a couple of weeks.

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 6 of 16

DATA DESCRIPTION

In order to build a detailed propensity model, Ravi collected data from 2008 to 2016. A stratified sample

of 100,000 de-anonymized customers was used (provided in a separate spreadsheet). He aggregated data

at an e-mail id level to come up with a set of features across online and offline (Exhibit 6), which could

be used for model building. Sample training data is shown in Exhibit 7 with the variable names in

Exhibit 8.

DATA ANALYSIS

To understand which features were important, Ravi’s team examined odds ratios of the target variable

against each of the features. Odds ratio is explained in Exhibit 9. The key findings are shown in

Exhibit 10. Odds ratio greater than 1 indicates that the feature is favorable towards purchase and odds

ratio less than 1 indicates the opposite. A higher odds ratio would indicate a higher degree of favorability.

OBJECTIVES

The final objective was to leverage data sciences and analytics for targeting, up-sell and cross-sell to

customers in the online store, thereby increasing customer value. The immediate need was a propensity to

buy a model that could result in the set of top customers that Michael and team should target.

At this point, Ravi had the following questions in mind.

 What feature selection techniques could he use?

 If he were to use the standard techniques – logistic regression or decision tree and any one

advanced technique (random forest or neural network or support vector machine or gradient

boosting…), how would the lift curves appear?

 Based on the lift curve, how should he communicate the potential opportunity from the model to

Michael?

 Could there be incremental lift or other approaches that he could adopt – for example, clustering

before classification?

Having built several propensity models at VMW, Ravi knew that sales teams liked Whitebox models.

Whitebox models are models whose workings can be explained to the sales teams. For example:

Customer X is more likely to upgrade if the support for the older version is coming to an end OR if a

compelling newer version is being launched. Sales leaders are not comfortable with just getting a list that

works. They also want to know why the list worked. The question on Ravi’s mind was also how best to

explain the characteristics of a Workstation buyer to the business.

At the same point, Parag had a further list of questions to discuss with Ravi once the model was fully

built.

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 7 of 16

 How should Parag and Ravi arrive at the number of e-mail ids that Michael should send?

o Remember the e-mails were to be sent with a coupon. Sending too many could impact the

margins.

o Should this list be different for different marketing channels?

 How do we interpret the results for business decision making?

 While lift is an analytics or internal validation measure, what marketing intervention should he

suggest to Michael so that there can be a scientific measurement of the return on investment to

the store business from the exercise?

o Can we conduct some form of Control–Test experiment to quantify the upside? If yes, how

should the experiment be set up?

Parag was also thinking about how he should set up an executive deck to summarize the results and

measurement plan to Michael. At the same time, he was wondering about the overall value proposition

that he could drive for the VMW store using analytics and data sciences.

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 8 of 16

Exhibit 1

VMW Financials

Source: VMW Publicly available annual report: http://d1lge852tjjqow.cloudfront.net/CIK-0001124610/67b316e9-d82e-4848-ade6-

e046775865be.pdf

VMW Q4 2015 Earnings Call: http://s2.q4cdn.com/112802898/files/doc_financials/2015/q4/Q4-15_earnings_w_tables_final.pdf

Exhibit 2

Workstation Revenues over the Years1

Source: Bookings Data (masked)

1 All numbers are directional and for illustration purposes only. The data shared is masked and only illustrative of real data. These have been done

to maintain confidentiality.

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 9 of 16

Exhibit 3

Workstation as a Proportion of Store Bookings* (masked)

Source: Bookings Data (masked)

Exhibit 4

Cross-Sell Behavior of Workstation (masked)

Source: Bookings Data (masked)

Workstation 6 Workstation 7 Workstation 8 Workstation 9 Workstation 10 Workstation 11 Workstation 12 Workstation 12 Player

Workstation 6 97593 10842 6604 5213 4420 2602 2179 109

Workstation 7 97431 24005 19858 15939 9376 8050 293

Workstation 8 67588 24326 21903 12319 10408 311

Workstation 9 65935 23648 15326 11683 373

Workstation 10 68998 18294 16665 508

Workstation 11 45851 13535 485

Workstation 12 41650 623

Workstation 12 Player 5139

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 10 of 16

Exhibit 5

De-anonymized/Anonymized Store Visitor Funnel (masked)

Source: Bookings Data (masked)

17 MM (# of Visitors to VMW Store)

5 MM (# of Visitors to Workstation in Store)

~4MM (store.vmware.com visitors with email id)

~1.7MM(Unique emails of Personal Desktop Buyers)

~500K unique

emails of

Workstation

buyers

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 11 of 16

Exhibit 6

List of Feature Buckets

Source: Bookings Data (masked)

Exhibit 7

Training Data

Data with 1,00,000 rows can be downloaded from the following link:

http://hrm.iimb.ernet.in/iimb/download/IMB_621.htm

Source: Bookings Data (masked)

Metrics for the

Dimension

Inputs

Workstation, Fusion, vSphere, vCenter, vSOM, Horizon,

vRealize

Activation, Download, Registration, Page Views, Cart Add/Remove/View,

Checkout, Purchase, Form Success, Form Abandon, Buy Now etc.

Internal, Paid Search, Email, Social Network, Search Engines etc.

Google, Bing, Yahoo, MSN, YOL etc.

OS like Android, iOS, Linux, Mobile iOS, OS X, Windows OS,

Windows Mobile and Browser like Apple, Blackberry, Google, Dolphin, Microsoft,

AOL etc.

Dig

ital D

ata

Digital & Non-Digital Feature Engineering (Offline + Online)

Store

Products

Event Wise

Search

Engine Wise

OS/Browser

Wise

Referrer Type

Marketing

Channel

DemandBase Data, IDM Data

De-

anonymizatio

n Features

Paid/Organic Vehicle Data

Non D

igital

Data

Revenue

History

Responses/Camp

aign Features

Marketing

Channel ShareOther Products

Bought

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 12 of 16

Exhibit 8

Sample Feature Names

Variable Meaning

Train_period_workstation_purchase_f

lag

Outcome variable (Whether the customer purchased

workstation (coded as 1) or not (coded as 0))

fswk_booking_pct Share of Fusion and Workstation bookings

total_bookings_amount Total bookings from this customer

personal_desktop_booking_pct Share of Personal Desktop Bookings

tot_windows_visits Total no. of visits to vmware.com webpage from Windows OS

days_since_first_personal_desktop_p

urchase_date

Length of Relationship with VMW w.r.t Personal Desktop

products

ftr_growth_personal_desktop_13_14 Growth in 'Personal Desktop' product bookings from 2013 to

2014

num_orders Total no. of lifetime orders this customer placed with VMW

num_order_lines Total no. of lifetime order lines this customer placed with VMW

ftr_growth_personal_desktop_14_15 Growth in 'Personal Desktop' product bookings from 2014 to

2015

idm_total_no_of_day_visits_to Total no. of visits to MyVMware Portal (required for customers

to interact with VMWare support)

ftr_growth_personal_desktop_12_13 Growth in 'Personal Desktop' product bookings from 2012 to

2013

tot_osx_visits Total no. of visits to vmware.com webpage from OSX OS

tot_apple_browser_visits Total no. of visits to vmware.com webpage from Apple Safari

Browser

idm_no_of_day_visits_to_home_page Total no. of visits to MyVMware Portal Home page

tot_microsoft_browser_visits Total no. of visits to vmware.com webpage from Microsoft

Internet Explorer Browser

tot_store_page_views Total no. of views to VMW Store Page

idm_no_of_day_visits_to_download_

page

Total no. of visits to MyVMware Portal Download Page

tot_page_views Total vmware.com page views

tot_first_touch_direct_views Total no. of page views by marketing channel

idm_no_of_day_visits_to_info_page Total no. of visits to MyVMware Portal Info Page

idm_no_of_day_visits_to_license_pag

e

Total no. of visits to MyVMware Portal License Page

tot_first_touch_natural_search_views Total no. of page views by marketing channel

gu_num_of_employees Total no. of employees in the customer company as per DNB

data

tot_google_browser_visits Total no. of visits to vmware.com webpage from Google

Chrome Browser

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 13 of 16

idm_no_of_day_visits_to_eval_page Total no. of visits to MyVMware Portal Eval Page

tot_visits Total vmware.com page visits

purchase_events Total vmware.com purchase events

tot_mozilla_browser_visits Total no. of visits to vmware.com webpage from Mozilla

Firefox Browser

tot_last_touch_direct_views Total no. of page views by marketing channel

tot_first_touch_internal_views Total no. of page views by marketing channel

tot_page_views_l90d Total vmware.com page views in last 90 days

ftr_growth_vsom_14_15 Growth in 'vSOM' Bookings from 2014 to 2015

tot_last_touch_natural_search_views Total no. of page views by marketing channel

num_any_campaign_responses No. of responses from this customer for all VMW campaigns

tot_last_touch_internal_views Total no. of page views by marketing channel

tot_visits_l90d Total vmware.com visits in last 90 days

ftr_growth_enterprise_desktop_13_14 Growth in 'Enterprise Desktop' product bookings from 2013 to

2014

Source: Data Analysis

Exhibit 9

Odds Ratio Explanation

Target = 0

Target = 1

Feature = 0

Feature = 1

Odds for feature = 1 is defined as d/c

Odds for feature = 0 is defined as b/a

Odds ratio = (d/c)/(b/a) = da/bc

Source: Data Analysis

A b

C d

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 14 of 16

Exhibit 10

Sample Averages of Features versus Target Variable

Source: Data Analysis

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 15 of 16

Exhibit 11

Purchasers of Workstation as of End of Each Quarter from 2013 (data masked)

Quarter No. of Workstation

Buyers

13Q1 2784

13Q2 2300

13Q3 3020

13Q4 4198

14Q1 2480

14Q2 2530

14Q3 1878

14Q4 3808

15Q1 2988

15Q2 2582

15Q3 3370

15Q4 4164

16Q1 2726

16Q2 2264

16Q3 2340

16Q4 1194

Source: Bookings Data (masked)

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.

Machine Learning Algorithms to Drive CRM in the Online E-Commerce Site at VMWare

Page 16 of 16

Exhibit 12

Sample Purchase Paths on E-commerce

Home Page → Product Detail Page → Cart → Checkout → Purchase

Source: VMWare

Exhibit 13

About the VMW Store

The store sells many products of which Fusion and Workstation are key to helping run Windows

on Mac and Mac on Windows, respectively. It is an e-commerce site in the truest sense and is

frequented for purchases both by consumers and businesses. The link to the store is provided

here: http://store.vmware.com/store/vmware/en_US/home

The store is a collection of pages. A sample purchase path for a user is indicated in Exhibit 12.

This is by no means the only path and there could be several paths but is shown to indicate how

the visitors purchase on the site.

Source: VMWare

For the exclusive use of M. Abouzahra, 2019.

This document is authorized for use only by Mohamed Abouzahra in 2019.