How Data Science Teams Are Going Deeper with Proof of Value

DAVE COLE

Hello, and welcome to another episode of the Data Science Leaders podcast. I am your host, Dave Cole and today's guest is Nimit Jain. He is the Head of Data Science at Novartis. Prior to his time at Novartis, Nimit was the VP of Customer Experience Analytics at DBS Bank in Singapore. Nimit, welcome to the Data Science Leaders podcast!

NIMIT JAIN

Perfect. Thanks, Dave, for the nice introduction.

DAVE COLE

Great. In today's episode, we're going to talk about crawling before you run. So the importance of starting with a POV, proof of value.

Next, we're going to talk about responsible AI. What is that? What does it mean?

And then, as I just mentioned, Nimit has some expertise in customer experience analytics. I think it might be a universal use case that any data science leader might want to embark upon. I'd love to hear Nimit's thoughts on what that is and what tips, tricks and advice he has for the rest of us.

Before we do that, I have one important question to ask you, Nimit. Looking over your background, you're now in New Jersey and have been for a couple of years. During your time at Novartis you did spend three years in Switzerland. Switzerland, I believe, is one of the most beautiful countries on earth. Why on earth did you leave Switzerland?

NIMIT JAIN

Yeah. I think I counter with your opinion and frustration as well. Some decisions, you have to make collectively. So I think this was a collective decision with my family. When I say collective, I mean whose decision it was and not what caused it.

DAVE COLE

Yeah. As you mentioned, we're data scientists and the weighting on where you live is probably smaller than it is for other members of your family, if you're anything like me. So I hear you. That's great. You also spent time in Singapore, so you've done data science around the world. Maybe we'll talk a bit about that in our episode.

Let's start off and just talk a little bit about your best practice when it comes to doing data science and the role that proof of values play. In your mind, what is a POV?

NIMIT JAIN

I think one of the favorite things for me is the challenge we have as an enterprise to scale the proof of values, and you call it last mile challenges. If you asked me what a POV is about five years back, I would have said it's building a predictive model to tell the business the power of AI.

Now, obviously, things are very different. It's more mature. Business is more educated. This field has really tremendously changed. For me, the proof of value is how well you define the business problem, how well you engage business at the start, how well you design and iterate with business, to make sure that we are solving the right problem—with the right metrics and mindset.

Also, working with business to prepare for the change once these models will be embedded into the business processes. It's a multi-dimensional perspective versus a unidimensional data scientist perspective. That's how I define the proof of value now. Put very simply: 5-10 years back, when you had any phone, you never expected it to have a touch screen; you were happy with Nokia. Now, whenever you buy any phone, the basic thing you expect is the touch screen. The expectation has changed and evolved as the industry has matured. The same thing now with the proof of values. The expectation has really matured from a business standpoint, and also what you deliver at the end. It has become more complex.

DAVE COLE

So the raised expectations can be a good thing, because that means that your user better understands your world and is more critical in a helpful way. What do you specifically see as being some of those higher expectations? What does that actually look like? You said, five years ago, it was just building a model and simply saying, "Hey, look, I have 83% accuracy on this model. Look at how cool this is."

Is there more, perhaps mainly around showing how that can be embedded into some production process? Is that the biggest gap? Instead of a show and tell it's more like wanting to understand how to productionalize this? Is there something more?

NIMIT JAIN

I think, right now, the business is trying to derive value. If they're really serious about industrializing the scale, they would want to understand, have a feeling of the value and the ROI on these investments. That’s where the proof of value has gone beyond statistical accuracy, to more life-testing of the models. Business touches and feels the results, sees the live performance of the model, evaluates even in some production scenarios and then takes a decision, gets confidence then also prepares for the future. How would this scale up look, embedded in the business process? They want to really use it now. It is not a good-to-have thing. It’s a must-have. That's where the mindset has very much changed.

DAVE COLE

Right. So their expectation is that they don't want to just see you show a model or the code, spin up some Jupyter notebook and present the accuracy of the results. They actually want to be able to see that model in production. If it could be an app, they want some app that they can interact and play around with. You even mentioned, potentially, that you can put it into production at a smaller capacity. Are you talking about that being in order to look at the value of an A/B testing framework, where there's a champion/challenger type approach?

NIMIT JAIN

I think, yes. So to a certain, again, it depends on the use case. For some use cases, it could be, as is versus the new way of doing things, which is the A/B testing scenario, where you see how the model performs with AI versus non-AI. The other would be where business takes a parallel decision, where we'll use this in a more human-in-the-loop scenario, where you embed AI then make a decision — a less risky decision, if you will. Based on that, once you really understand how to play, what you can and cannot do, then you really take it to riskier transactions and scenarios. There are different flavors of lift-testing. In some cases A/B testing fits and in some cases it doesn't.

DAVE COLE

You also talk about the role. Clearly the business user (your user) has a more active role in this. What gates do you put in place? Let's say the business user is playing around with some model that your team has created. You've worked with them to monetize it. You come up with some ROI calculation. What happens? Is it purely their decision as to whether or not it gets put into production? Is it a collaborative decision? Is it purely just based on ROI? What are some of the factors that make it go from POV to, "Okay, this is something that we are going to maintain into production?"

NIMIT JAIN

Yeah. Again, I think obviously, a business might get impatient on getting the ROI, but we cannot get the returns now. These things take time to really help understand the top and bottom line impact. It's more of a personal sense of comfort and how businesses really see this in their whole end-to-end vision. I think, definitely, all of our work which has gone industrial has been industrialized because of business. Whichever has not is a mix of business plus data science. It could be that your data was not predictive enough, and it's not able to beat human accuracy or at least be the same as what humans are doing. It could be because the cost of production is much higher versus the potential ROI in the next two to three years.

I think whatever eventually happens, even if the POV doesn’t go into production, I think there's always a benefit. We know a lot of those models can be reused in a different context of problem settings. A lot of the work doesn't go to waste. We ensure that as data scientists. Wherever we have productionalized the model, it is because business sees the value, both short- and long-term. Mostly, these are soft in nature at the start. I make some rough calculations while A/B testing in a smaller context. Then I think long-term: usage of the model, insights, if they’re really reaping benefits out of it.

DAVE COLE

Sure. We talked about the bar being raised in terms of the expectations on the business side. I imagine the quality of the POVs that you've created: how ready they are to be pushed into production. It could be code—standards have improved over time—just the overall knowledge of models needing to have a sub-second latency or whatever. That has improved. So has the speed with which the decision is made that, yes, this model needs to be put into production. I imagine that your models today are more production-ready than maybe they were. Is that safe to say?

NIMIT JAIN

I think that's where probably the responsibility of IPS also comes in, which we'll probably talk about. Industrialization is not just the model part of it. You also might be experimenting on a smaller scope. The smaller scope is good: you have one brand, one country, one region, one manufacturing site in whatever dimension you take. When they industrialize you, I've not seen it typically as apples to apples. Whatever you did in theory is not just taken into production. Obviously the scope gets enhanced and you have reason to believe that whatever you've done at POV can easily scale to the same archetype, whatever you've chosen. That, for us, is industrialization. It might take the same amount of time and effort that the POVs took. The idea is that the time doesn't increase linearly; it decreases in terms of the effort you have to put in.

DAVE COLE

Right. It should go down over time. That makes sense. Do you have any projects that are definitely going into production or do you always take this POV type of approach?

NIMIT JAIN

There are certain problems where, for example, if we know we have done this problem before and it’s just a different function/region, it provides me more of an extension. Most of the time, you have to contextualize the problem you're solving. As much as you want to reuse your views, contextualization itself is a big task.

DAVE COLE

Right. That makes total sense. You did mention responsible AI. I want to dive into that. That's our next topic. First of all, what is responsible AI? What does it mean to you?

NIMIT JAIN

If you look up the definition, I think Google, Microsoft, and a lot of these tech giants have been actively working on responsible AI. What we have done is we have a broader point of view on responsible AI because of being in healthcare. I would like to give you an example. There are some principles which we have defined and which are broad enough. One of the principles is environmental sustainability. What this really means as a data scientist is that when you build models, you question if you have to always have a complex model, which takes 60 GPUs to run for X number of days, versus having a simpler model, which might be a little bit less accurate but runs more efficiently. We are now making those trade-offs in a more responsible way versus a more unidimensional way.

DAVE COLE

There's a cost impact of having some models that require 60 GPUs. I'd love to know what that model is at any rate. Are you thinking about environmental sustainability literally, like that's a lot of GPU power and that's taking a lot of energy and not good for the planet. Do you think about it from that perspective?

NIMIT JAIN

Yes, exactly. When I gave you this example of 60 GPU, it might be just one model, but as you start putting more drops in, it becomes an ocean eventually. It might be a big cost. Obviously it has an impact on the environment. As data scientists, we are now making sure that we have a conversation with business at the get-go and the proof of values state themselves. When we make these trade-off decisions and show them, then really make a judicious choice on what is good for the project.

What we are doing with these responsible AI principles is that we are bringing on, early in the game, new tee length. There are very high chances that our POVs get industrialized in production and also used at the end of the day. But there are many principles. One of the interesting ones, which is environmental sustainability for us, is that there are always model biases, which is well understood out there. We typically leverage more of these open source tools.

DAVE COLE

Like what? What open-source tools? How do open source tools play a role in responsible AI?

NIMIT JAIN

When talking about biasness, I think it’s to understand if there are certain groups and cohorts within your whole data set, where the model is over-performing or underperforming. That's where we are trying to make sure we use some tools like, there's this Erudite AI from Microsoft. It's an open source tool, which can really dissect your model errors in the lowest granularity and tell you where the model is underperforming. And also, this biasness is about how you better explain the model to the business.

We are using a lot of these explainable AI models, which is a more realistic base: the Shapley-based models, either from Microsoft or there are a lot of other open source tools. We really bring that early on, even before business asks us. Initially we used to just give them predictions, but now we know they're not happy with just that. They also want to know why, and how they can change the future. There is this whole journey, which we have learned over multiple POVs if you want to bring it early on.

DAVE COLE

Right. So let me try to put it in my own words, Nimit. So explainability and bias. What you're saying is that they're very much related. If you just have a black box model that is spitting out a prediction, the business is no longer saying, "Oh, sure. Let's just roll this into production. Let's go." They want to understand the features that are part of this model. They want to open up the black box. They want to be able to explain why it has the predictive power that it has because, in part, they want to understand if there is any explicit bias going on. They’d want to know if there are any features being used, like gender, race or ethnicity, where there could be explicit bias and they might not be comfortable. They should not be comfortable putting a model into production that is based on features like that. Me tying that together, does that sound right to you?

NIMIT JAIN

You're right. The topic, which I thought to be about explainable AI also has to have transparency and explainability of the models. There's another principle biases and transparency are sort of interlinked. Transparency is very important now. Even if we use the black box model, we have to make sure we show what is driving those predictions. For some of these areas, the maturity curve is very high in the AI industry and for some, it is still maturing. Privately, it can be in the space of NLP, for example. We also custom designed those algorithms to answer some of these business questions versus using off-the-shelf algorithms.

DAVE COLE

Your team is actually creating their own algorithms so that they can help with that transparency?

NIMIT JAIN

Yep.

DAVE COLE

Oh, that's great.

NIMIT JAIN

And also developing those products, actually. We are also co-developing those open source products with Microsoft, for example, because we are using those products actively. The context would really make the products more mature.

DAVE COLE

Well, that collaboration is great. There are certain data science teams that are, I hesitate to say, sophisticated enough. A lot of data science teams focus on applied data science, like taking the algorithm and applying it to real world problems. It could be classified as well. Some teams focus on a research element where they might have read a paper and there isn't an open source version of the algorithm out there. They actually go ahead and build it. For those teams, there's more work there but, to your point, one of the benefits and why you might want to do that is increased transparency. Are there any other benefits in your mind other than increased transparency, to actually having a team build their own algorithms?

NIMIT JAIN

It also comes from the people in home UI in the team. All of these data scientists have been trained to contribute because they are not only the consumers there, but they also want to be producers of intelligence. It's definitely very important to make sure our data scientists remain excited and feel empowered and involved in the overall data science as a team.

That's the reason we focus extensively on people to contribute to the knowledge as well as be the first consumers of some of those new algorithms. First adopters is also not an easy thing. Now in NLP, everybody talks about BERT. Back then if you were a first consumer of it, really, you have seen the progression of it and you can talk much more intelligently. You can use it where it is necessary to be used versus using it as a Swiss knife: everywhere. So I think there are a lot of advantages for the data science teams to focus on innovation as producers, as well as consumers or both.

DAVE COLE

So quickly, for our audience who may not be familiar with BERT and how it's applied in NLP, please explain it if you can.

NIMIT JAIN

I would say transformer based network. The more you ask me to open Pandora’s box, the more complex it'll get.

DAVE COLE

Sure.

NIMIT JAIN

I can definitely tell you it was iPhone magic in the world of NLP in October 2018, if I'm not wrong. That’s when open source tools, a lot of them released by Google, had really high accuracy in a lot of the natural language tasks: machine translation; sentiment analysis; classification; whether the email is spam or not. Now I think the baseline for all of the NLP work has changed to be bought versus a simple different way of doing things before it. I can probably stop here, otherwise I'd get more complex.

DAVE COLE

Yeah. That's fine. I'm just Googling BERT myself. I'm somewhat familiar with it. The acronym is Bi-directional Encoder Representations from Transformers. It comes from the researchers at Google AI language. Correct me if I'm wrong, but it's an open-source framework that allows your teams out there to better do NLP-related use cases. That's a super high-level summarization from yours truly, but there was a lot there. It sounds like it's on the cutting edge. So I guess what you were saying, though, is that you've been following the evolution of BERT. It is fairly cutting edge. There are ways in which your team can actually take it upon themselves to come up with their own slight tweaks of the algorithm.

NIMIT JAIN

Yes.

DAVE COLE

Okay. That’s to make it more customized for your team's specific use case. That's a great way to keep your team actively engaged, especially your high performers who are very cutting-edge-minded and keep them highly interested in what they're doing. Obviously, it has a business value to it because it's being tailored to a problem that you have at Novartis. That makes a lot of sense. Well, let's move on to the next topic. Changing gears substantially here and talking about customer experience analytics, that's an area for almost all software companies, most retailers, restaurants, you name it. It's a fairly universal challenge for companies to ensure that customers are having a great experience. What is customer experience analytics? I want to dive into a little bit of some of the use cases and some of those challenges.

NIMIT JAIN

Sure. Customer experience analytics, I did, I would say a half a decade back. A lot of buzz words were around user experience, personas, thinking of being in the shoes of customers. Those things were out there and there were a lot of best practices a lot of the companies, which were more customer-focused, were trying to adopt. We were trying to solve some very interesting use cases. For example, how we could have a seamless customer onboarding if you were a customer of a bank. It's customer experience analytics I did for a bank in Singapore. There, the worst or the biggest problem is if you are a new customer, you have to open an account in a bank, there are so many hoops and hurdles you have to go through.

That's where we wanted to really think in the shoes of customers, trying to solve the problems of how to really have a seamless onboarding for a customer. Even simple things like if you go to an ATM and draw cash or whatever you want to do there, how you can have a personalized view on the screens versus the generic four things on the screen. All of the information in the banking world, customers give to us. We don't have to buy the information. You can really individualize the experience of the customer. The other small, interesting problem we also saw: let's say you're a customer and going to an ATM and your debit card gets stuck in an ATM. What is the first thing you do?

DAVE COLE

I freak out. I worry. I assume it's gone and I call the bank.

NIMIT JAIN

Exactly. This is a big cost to the bank. Even though we know that we have eaten your card, we cannot do anything until you call and inform us. This is a complete disjoint. That's where we did a lot of A/B testing and tried to change the customer behavior of calling the bank to be proactively telling them we know the card is stuck and it will be at their home in two days. A lot of customer experience analytics, we did primarily upstream of that whole customer journey. When you're trying to chart out who your customer is, what their experiences are, their pain points, which personas to talk about, that's where we can get more data-driven because we have a lot of historical data.

We used to really mine information to understand what is driving the customer sentiments across these different problem statements, which I talked to you about. You have a more factual customer journey versus a hypothetical customer journey. Then you can figure out, as part of the customer experience, which kind of personas you have, who to talk to for a focused interview and what the interventions should be. Then again, do A/B testing to see if we are able to influence the customer journey from a bad experience to a happy experience. It was all data-driven with a lot of qualitative analysis.

DAVE COLE

I took a bunch of notes here. A lot of the focus around customer experience analytics is in a few different areas. Personalization is a big one. The theory there is that the more personalized the experience, the better it is, the better the overall satisfaction a customer might have. Data science obviously can play a role there, simplifying processes. That could be anything from your onboarding process and what it takes to create an account to getting money out of an ATM and making that more easy and streamlined. This is fairly universal, but understanding the various cohorts and personas that you have, putting names to them and trying to see, from a personalization standpoint, each of those personas, you might want to have the different experiences based on what persona that customer might fall into.

Then understanding the overall customer journey and where they are in that journey: are they a new customer just being onboarded or are they a lifelong customer who hasn’t been using the full array of services? You might want to upsell or cross-sell them something. Those are just some areas of customer experience analytics. Did I miss anything there, Nimit?

NIMIT JAIN

I think the only fifth dimension would be the launches. When you do a lot of launches of new products, how to learn from past mistakes. That is another dimension you can use in this data-driven customer experience approach.

DAVE COLE

I forgot to mention the underpinning to a lot of this is doing A/B testing. Do you have any advice for the best ways to do this? Most of the time, when you're improving a process or trying to personalize an experience, you're testing out theories. You mentioned getting an email or a text right after your card gets eaten (hopefully not) in an ATM. Does that improve the overall satisfaction? In that example, like you mentioned, A/B testing, what did that look like from an A/B standpoint?

NIMIT JAIN

I remember at that time we had to really research a lot. I think there's a lot being written and done in the marketing context. You talk to Walmart. You talk to Target. All of those retail giants have been doing A/B testing for ages.

DAVE COLE

Right. And that's mainly around marketing content: the blue creative is better than the red creative, that kind of thing.

NIMIT JAIN

I hope to send an email. What kind of email? For example. In our context that's when we view it. When we've tried to really find analogs, it was very difficult to find. A simple question you were not able to answer is, “What should my sample size be, on which I need to experiment?” We had to do this A/B testing on what kind of cohort or for how many days we have to run this experiment so that we could statistically prove that we were able to change the customer behavior of not calling the call center when the card was stuck, to being happy once they received an SMS. We had to really research a lot. Therefore, this area is still a little bit underdeveloped. What should be the ideal sample size?

How long do you need to run your experiment? What defines a statistical significance? Obviously, you have these P values, but then, what is really the right level? All of that research we had done, but I think we were able to experiment really fast because there was less risk. You're just trying to send an SMS to people. You're not encroaching in their personal space. People have allowed us to send SMS to them.

DAVE COLE

Right. You have to opt in to say something like, "Yeah, send me an SMS." Then there are those not on SMS who opted in for other communication channels. I think what you were optimizing for in this specific use case was whether or not they actually called into the bank. I imagine also you want to make sure that they don't leave the bank. In severe cases you might say, "Okay, this is the last straw. I'm done with this bank because I'm so frustrated," but those are the things that you were measuring when you're doing some of your testing.

NIMIT JAIN

Again, that's where there is no end to the measure. You can measure whether you're calling the call center or not, but then you can also measure, have they increased the transaction? Have they decreased the transaction with us?

DAVE COLE

Yeah. I was going to say, you’ve got to be careful, because you don't want to just optimize for whether they call into the call center. That's a cost to the bank and time is money. You want to limit those calls coming in. Really what you're optimizing for in customer experience analytics: is the customer satisfied? Are they happy? That experience, despite it being negative on the whole, is hard to turn that into a positive. It can, if you overwhelm them with a very personalized experience and make them feel like there was some reason why your card was eaten and it was to protect them or what have you, here's what you're doing about it. Here's the card.

NIMIT JAIN

That's an interesting point. We also actually did A/B testing on the content of the message itself. As you said, simple things in English, people can interpret very differently, depending on how we perceive it. There's a lot to this experimentation and there's no end. It will never be over. I think you can make it as complex as possible. What we realize is that first, we have to do a very simple experiment. Once we approve the simple, then try to make it more complex. That makes sure that you highlight what assumptions you have made while seeing if your experiment is successful or not. I think it has also come to the culture. I remember in the company, we had a culture of saying we have 2000 experiments as a company. It also comes from the top down, that culture of experimentation and really trying to change and delight the customers.

DAVE COLE

You're doing all these experiments for what purpose? Not just to potentially cut costs or increase revenue, but actually to delight the customer. If you do that first, some of these other things hopefully will follow. One very specific thing, looking at the customer's behavior after one of these experiments happens. Were there times when you would just survey them directly and just ask, "Hey, what was your experience like," or do you just look at the customer events after the fact?

NIMIT JAIN

Again, it is both ways. You do look after the fact, because then you have a smaller sample size constraint because you have rich historical data, big data. You have to still do focused interviews. You cannot replace the qualitative aspect of this customer experience, which is where you really get closer to the customer. This big data approach helps you define, as you were saying, personas. Now you have these 10 personas, then you can have focused group interactions with each of these personas and they really lift the life of the customer and what he does the whole day.

Then you realize that he's been doing these 10 transactions in these different areas, he fits into this persona and so on and so forth. These are the needs of those customers. They want something which is touchless. Some customers still want cash transactions. Some customers like to use only a laptop for transactions, not even a Phone. Those are the things that you really get to touch and feel when you walk with the customer. Both go hand in hand to answer your question.

DAVE COLE

Got it. Some combination of looking at the data, looking at all the various events that a customer is creating as they interact with you as a bank in this specific scenario. Then also just reaching out to them directly and just asking them by way of survey. That's been very helpful, Nimit. I think that many of our listeners out there should be thinking about how to create a better customer experience and what role data science has to play, and I think we touched on a bit of that. Before that, we touched on responsible AI and the importance of starting with a POV, how the bar has been raised over the years in terms of the expectations of what a POV looks like. I think it’s a good thing. I really appreciate you coming on the data science leaders podcast, Nimit. If folks want to reach out to you and have additional questions, can they reach out to you via LinkedIn?

NIMIT JAIN

Yeah, sure. I'd be happy to see what I can do to help.

DAVE COLE

Great. Well, thank you for taking the time, Nimit. It was a blast having you on. Have a great rest of your week.

NIMIT JAIN

Thanks, Dave!