Elevating Your Team as Strategic Business Partners

DAVE COLE

Hello, and welcome to another episode of the Data Science Leaders podcast. I am your host, Dave Cole, and today's guest is Indy Mondal. Indy, how are you doing today?

INDY MONDAL

I'm doing great, Dave. How are you?

DAVE COLE

Great. It's a Friday, so I'm excited to end the day on a great note. I'm very pumped about today's episode. So, Indy is the Senior Director of Data Science, AI, and Product Insights at DocuSign. He also played a similar role at both Hulu and Getty Images. And one of the few things we're going to be talking today with Indy about, the first is using data science to perform strategy. So, one of the things we've touched on a bit in the Data Science Leaders podcast is what role... How proactive should data science leaders be in helping to form strategy, instead of being reactive and helping to validate strategy?

The next thing we'll talk about is using models to drive efficiency, and Indy's got a lot of experience there, and we'll learn more about what that means when we get there.

And then last but not least, the role of self-serve in data science. So, what role should a data science leader play in standing up a self-serve mechanism for users? Let's start right at the top. So, what are your opinions, Indy, on the role of data science to help formulate strategy?

INDY MONDAL

Yeah. Excellent question, Dave. Look, data should always be the foundation and guidance for any successful business strategy. And oftentime, it entirely depends on the role the data science leader played to establish that expectation in leadership within the organization. And you are right, oftentime what I've seen in the industry, that data science becomes more of a support or validation of strategy, rather than someone driving the strategy. But to be able to establish that role effectively, it requires a lot of effort from the data science leader to drive evangelization and thought leadership. And not only that, establish the structure of the organization in a way to be the strategic partner of business rather than the support organization.

So, there are various ways you can build a data science or a data organization, there are various models. And the way I think about it, there are primarily two buckets, if you will, that you can build your organization: One is a support or a delivery organization, or second, as an organization that focuses on building strategic partnerships, strategic product and capability for business. The first model is very much focused on the what part and in optimizing on supporting business needs and ad hoc requests as they come in. But the second one requires a very product and strategic mindset, where you are partnering with a business at the strategic level to understand the why, not so much the what, and then building product and capability to drive the business.

With the first model, it is really hard for data organizations to be strategic in both its own work, as well as establishing it as a strategic partner to the business. Let's spend some time understanding why. Say, Dave, you are a new data science leader. You join a company, you are starting to build a team. You hire a few data folks and you suddenly started getting requests from the business. Someone came to you and said, "Can you build me a dashboard? Can you build me a model?" Great. You did a very good job in terms of developing those, suddenly business started seeing the impact that you drive. You started getting more requests, you delivered those.

And what happens after some time, because your request, you are delivering these individual requests as it comes in, you suddenly started getting so many requests, the only way you could scale is to hire more data scientists. And then you started hiring project managers to manage all these requests. And not only that, because you ended up building so many artifacts, like your dashboards and models, you now have to manage, support all of those. And you start to get to a point where almost you're binding your own artifacts and the requests coming in, and you have very little time to be strategic.

On the other hand, think about the possibility, what would happen, you still would have to do at the beginning, like you're still reacting to the business, but at some point you pivot. Instead of reacting to the business, you start to meet them and start to ask why.

And you, trying to understand the commonality between all these somewhat disparate requests and figure out what are the core things that you need to build to support all of those? And so that's where I think as a data science leader, you can always try to be, if you really want to be strategic, where you are trying to think, instead of building 100 models, 100 dashboards, and building only a set of core capability that I can support effectively, which can be exchanged in the future, and I'm really treating all those as a product and capability rather than individual request.

And I'll give you an example, again, from what we do here in DocuSign. A big part of what I do in DocuSign is to support product data science. And in my previous life, generally, this is how product data science I've seen work. A product manager is launching a product and he's like, "Hey Indy, I need some metrics," or I've launched a product and I say, "Hey, can you tell me why my product is not doing well?" So you started looking into it, you realize, you know what? We actually never captured the telemetry to begin with. We never even defined what success metrics need to be that defines the success of the product.

So you are asked to do something at the end, and at that time you're reacting, you're just supporting a PM with the request. What we have done here in DocuSign, we partner with the PM at the planning process. And this is all of my data scientists and my team, the product manager, data product manager. So they're partnering with the product manager at the planning process. Even before we get any request, we sit down to understand what product is building and how we define success for this product, and is there a target that we can set? And once we do that, we start to look into, do we need to build telemetry? What telemetry do we need to capture, what pipeline needs to build, what dashboard needs to build, and what we have done over the last... My period here in DocuSign, we have turned all of those into a product.

So we now have an instrumentation system that has a very reinforced schema so that we can always capture data in a certain way. Our pipeline is scalable and extensible. We actually have a product that we call a launch dashboard. So no matter what product goes, it’s there. So think about it, instead of reacting to all the product managers on a one-on-one basis, we actually ended up building a set of products and a set of practices that allow us to be a lot more strategic. And I think this is what you can do as a data science leader to signal to your business that you're actually not here just to support them, but you are here as a thought leader and meet them at the very beginning.

And today my organization at DocuSign partners with product managers for almost all new product strategy, supporting existing products, and how we think about evolving these products. And to me, your question about how one can think about being more strategic, to me, it's all about how you build your organization as a leader, and what sort of expectation and structure you're putting in place that allow your team to be strategic, and otherwise you'll end up being reactive all the time. I don't know if that makes sense...

DAVE COLE

That makes a lot of sense. There's a lot of good stuff to unpack there. So let me try to summarize as best I can. So based on what you've seen historically, there's two approaches that a data science leader can take.

One is to be a little bit more of a support role. So I hesitate to use this phrase, but a little bit of an order-taker, right? So somebody who says, "Hey, you have a question, we have answers. You bring us the questions. We'll put together a deck, we'll put together a dashboard, we'll build a model, we'll help you get your answers."

The second approach, and the preferred approach clearly, by you, is to play more of a strategic role, play more of a partner role, a thought leader role. And your recommendation is to establish that, ideally in the very early days. So if you're a new leader within an organization, new data science leader within an organization, you should try your best to set up those partnerships, be involved early in the planning process, before a product is being launched, so you can hopefully stave off things like missing... If there's a certain thing that the PM would like to have measured, let's make sure that we're capturing that data element so that it actually can be measured and that we're not having to wait around for that data element to be added to the telemetry so that you actually get that question answered. That's a small example, right, of why you want to be involved?

INDY MONDAL

Yeah. You're absolutely right. If I want to wrap it in like two sentences, it starts with you're always asking why. Don't worry about the what, go ask why. And second, make sure you as a leader are making... You need to do a lot of legwork with your peers and your leadership to establish the... What data science brings on the table and how we work with the leadership. So don't wait for work to come to you, go meet your leadership and define the work at that level.

Another example I'll quickly give you, one of my past jobs, we had so many dashboards that the data team had built, we actually had to build an internal product just so that you can search the dashboard.

DAVE COLE

There's so many dashboards you needed to build a dashboard to find the dashboard that you needed.

INDY MONDAL

Right. And then we have to build a full stack product to be able to do that, because we need to have CSM, content management system, we need to have... It was a month of process building that, because we have dashboard and Tableau and MicroStrategy and this. You don't have to do that, but if you start focusing on building the capability, rather than one-off things, you'll never get there. And I think this is what I've seen data science and data teams fall into this. You have so many of these artifacts, you're pretty much drowning in it.

DAVE COLE

Yeah. And we've all seen it, right? The sprawl of reports and dashboards and even models too. And if you're not involved in understanding the why, like why this dashboard is getting created, then you're probably not going to have a very good idea as to how long its lifespan should exist as a dashboard, like when it's time to retire it and move on and replace it, how it should be improved. I think what you're putting your finger on, is as a data scientist and as a data science leader, you need to be more proactive.

You need to really understand the business. And if you're just taking the orders, I can imagine the back and forth is going to be much longer. Because you're not truly understanding what they're getting at, you're not asking the question behind their questions so that they can... Because you're a data scientist, most likely they're not. And maybe they understand a little bit of your world, maybe you understand a little bit of theirs. But to truly understand their worldview, you have to partner closely.

INDY MONDAL

You also have to spend time educating your business owner. Because not everybody is data literate. Oftentime a business has come to me and said, "Indy, can you set a target for this product? Can you set a target?" And I said, "No, as a data science team, we don't set targets. We provide you guidance, forecast, guardrail."

DAVE COLE

Right. We can forecast. Yeah.

INDY MONDAL

Right. And it is the business who understands their priority of the year and the initiative that marry the focus or the guidance we have given to come to the target, but that's an education. The last thing I would say, and this might be controversial; if you have a data science organization and if you have too many project managers in your team, you are probably doing the first model I've said. If you have an organization where you have a data product manager, you are probably a lot more strategic and focusing on the second type of the organization I just mentioned.

DAVE COLE

So, okay. That's a really salient point, so if you have project managers with a J and you have a lot of those, then you're probably in the support organization. So you have product, with a D, managers on your team, data product managers. Is that right?

INDY MONDAL

Yeah.

DAVE COLE

So how would you define their day-to-day? What does their role look like?

INDY MONDAL

So the data product manager in my team, they're constantly meeting with our stakeholders as they come in, and their job is to understand not the want, but the need. They're trying to figure out all the requests that are coming in. We're meeting with them, okay, what's the pain point? What are we trying to solve? They understand the business really well, and figuring out the actual business need, and then being able to draw between all these requests that are coming in so that we are able to come up with use cases and capability that goes into building this product or capability, rather than many small projects or many small one-off deliverables. I don't know if that makes sense.

DAVE COLE

No, that does make sense. So one thing I'm hearing though, let me push on this a little bit, is that by getting closer to the business, in this case, the product manager who's building a new feature, by getting closer to them, you understand a little bit of the world, so you can understand the types of data elements that you need, the types of models that you need, what needs to be measured, what types of analytics need to be performed. Now that is super helpful, but there's a difference between that and then actually being someone who's setting the strategy.

So then maybe for you, as a leader, are you ever turning around and saying, “You know what, we've been building a lot of products in this area, but I think that there's a whole other area of opportunity over here and I think we really need to be doubled down on this product based on what I'm seeing?” So do you actually meet with the head of product and inform product direction?

INDY MONDAL

I think that's a very good point. This is what I was trying to say at the beginning. This is what exactly my team does right now. Once you get out of this constantly reacting and you actually have time in your data science, you can do the exploratory work. I can go and talk about strategy, but I need to go there with some data and evidence to back it up. So this is where my team spends a lot of time doing deep dive analysis so that we can take that to business, and we've done that multiple times in the last year in DocuSign, presenting to the head of product and like, look, this is what we've been observing. And I think this is what, to be really honest as a data science driving strategy, your job is to bring up these things that no one is saying, and use that as an example, as a foundation to start having the conversation, like, look, this is the thing we are seeing a bottleneck here.

This product that we have launched is... We are really seeing not enough adoption, and this is why we think that is. And then taking that too, and you can do that two ways. One, you can wait for the head of product to come back and say, "Indy, can you tell me?" Or you can build a data science organization who is actually focusing on understanding this, because they have bandwidth to do this deep dive and go back to the product leaders proactively and have those conversations. And what I like to do with my team is do the second, that we are constantly doing that. One example, and I think this is going to be pretty resonant with most data leaders.

If you think about COVID, COVID is something that no one has ever seen in their life before. So there was no empirical data or empirical things to fall back. The only way, and at least here at DocuSign, we worked with our product team and engineering team almost on a daily basis, because the only thing we had to understand what's happening and to drive the business, because we don't even know if it is normal or not, is to be data-driven. And we spent a lot of time in the last two years, and even now, if you think about it, when you are trying to build a strategy for next year as any product, I mean, Peloton is a classic example, it's all over the news. How do you focus demand? How do you build a strategy? Is it, generally, you see my dividend is going up and you are going to make some assumption, but people actually bought more than they want to consume.

So you have these years of demand that's been pulled in, and the only way you can be really successful when your data team is actually working closely with the product. And we spent a lot of time understanding how to get rid of the entire outlier that is the one data point, two data point, months of data point, to be able to go in to build a model that allows us to normalize that. So we do a lot of that work with our product team, with our business team, only because we are able to meet them at the strategic level and not just reacting to their one-off questions.

DAVE COLE

Yeah. I love it. I think it sets the bar for what a data science leader should do, and if you're not doing it today, what you should aspire to do. I think it's not too late. This is my editorial here, but I don't think it's too late. I think it is harder if you've, to your point, if you build an organization that's predominantly supportive in nature, you have a large PM organization that's, say, just handling the intake of requests. It is hard to maybe at that point to change hearts and minds, it's much easier to set it at the outset. However, I don't think it's too late to start having those strategic conversations to change maybe how you work with your business counterparts to get involved earlier in the process, start asking questions, like why, asking questions behind the question.

And then you can start building that rapport, and then before long you'll be starting... You'll be invited to some of those early meetings, invited to those strategic meetings. And obviously come with your own opinions too, look at the data within your organization and come with your own opinions. I think that's another way to build rapport. Well, cool. Well, I wanted to move on to our next topic. I just have written down use models to drive efficiency, but I know you have more to say about that as it pertains to product analytics. Tell me what you mean by that.

INDY MONDAL

Yeah. I think when you and I were having the conversation before, generally what I try to say, a model's performance doesn't result in a product's success. A lot of the time, as a data science team or AI team, we, rightfully so, one of the things that we really focus on, like how accurate is our model. What's our position call, recall, F1 score, confusion matrix, you name it. That doesn't always result in a successful product, because a product is driven by customer experience and usability. So one of the things that I fundamentally believe, that you cannot really develop a successful AI solution without having the clear understanding and the context of the customer; how the customer would be using the product and solution within. So you make any model, the output of that should just make sense to the customer intuitively. It just has to work.

And that's one of the things I want to do with my data team. I want to make sure my data team spends time understanding the customer problem, the use cases and context. And to do so, you need to understand that all the product or model actually at the end of the day, results in the customer doing something with it. The question is, what is that? And can you start measuring on top of your model metrics, a set of customer success and business success metrics that align to that customer efficiency given or the business impact. So generally, I'll give you a little bit of how I do in my team and then I'll give you a couple of examples.

So one of the things that we do in my team, besides the model performance metrics, we also measure both qualitative and quantitative metrics from a product perspective. So yes, you can measure your influence score, your precision recall and confusion matrix to understand how good the model is, but now you start to understand how that translates to customers. So one of the things is obviously qualitative metrics. You make sure you're constantly getting in-product feedback, however you're doing it, from the customers; are they finding it useful? But that's in a qualitative sense.

DAVE COLE

And how are you getting that feedback? Is that surveys? How are you getting that stuff?

INDY MONDAL

Yeah. It depends. The most easy way for you to do that, like some P set or a C set. You have an in-product survey that goes, pops out, user experience, that makes sure to capture that when they are done using that, how was it? And the second was trying to define a set of measurable success metrics. So if your goal is to, at the end of the day, develop efficiency or a productivity gain for the customer, can you actually define a matrix that can measure that? And then instrument your product in a way that you're collecting that so that you can train and optimize that. And second, on the same quantity, can you... And many times it's true for us in a B2B sense, any model that you're putting in into your product that has to result in some business impact.

So instead of business metrics, it could be one of your conversion metrics, depending on what we're doing. It could be click through, add to cart, add to your watch list if you're a media company, whatever that is, can you tie the model performance to that? If you can do that, then you are building truly a successful product. And to me, for my team, when you're working on any model, I want to make sure we're not being only lopsided and thinking one end, like, okay, mathematically, this model is so sound, it's giving me 95% accuracy, but if it doesn't really translate to customer experience, it's not going to be a good product. I'll give you two examples. There's a media company that I know very well, had built a content recommendation system, and I tell this story to everybody.

But the model was really good, but it didn't do well with the viewers. It was a very high performing model, but it actually was not able to do really well the engagement or consumption that we're looking for. So when we started digging into why, it turned out, the model that was developed for the US market by a team who was in a different country. So they didn't have the understanding of the pop culture we have in the US, which means the output lacked the contextual awareness and the relevance and it didn't always make sense to the user. So if you think about that, you still have a model which is really high performing, but it didn't do well in the product because there were additional things we had to measure, we didn't measure.

DAVE COLE

Right. So this is interesting, because most data science leaders, to your point, they just care about the accuracy of the model. When they think, hey, this is what I need to... When I need to retrain a model, it's typically because the accuracy of the model has gone down. And if your model is a content recommendation system, the accuracy is basically based on, I guess, the top item that you're recommending actually being clicked on and viewed. I think your point though, is that models are embedded in products as well. And you need to think about the entire product experience, don't just think of your job very narrowly, as just being this cog in a larger machine where you're just responsible for recommending the top 10 pieces of content they should view next.

Instead, you need to also be seeing, is that overall experience, is what the... Maybe they are searching, finding that top item is spot on, but the other nine are completely irrelevant and useless and the users are seeing those and they're getting frustrated and they don't want to actually use it anymore, the content recommendation system, because of other factors that you haven't thought of. So, surveying the users and thinking about the user experience, that's not just the PM's job, that's not just a product manager's job, that's also your job, is to understand it.

INDY MONDAL

And also defining the product metrics. When you are defining the product metrics, and sometimes depending on how your organization is structured, it could be your job, could not be, because there might be a different product analytics team that’s doing it. But as a data team, you have to have some understanding of how the product is going to be considered a success. What's the metrics? And to me, as a leader, and this sort of the DNA you have to build in your team, if you are not doing this, it really doesn't matter. Your product is ultimately failing. And if the product fails, it doesn't matter how successful my model is.

We actually haven't been successful as a team. And if you are a data science leader or any data leader, you have to think end to end. And to me, I've seen a lot of them fail in my life and I'm very careful now, any time, wherever I go, great, what is the impact with the customer, and what consider is successful a customer point of view? And can you define that? Maybe it's already defined. If it is, great. Make sure when you are training your model and testing it, you also test that part. Because you can do that, right?

DAVE COLE

Yeah. You might have multiple dependent variables in a sense, to think about here. Well, that's great. I think there's a general theme here, which is don't think narrowly, broaden your scope. Think about the business, think about the customer. Don't just think that your team is there to churn out models or dashboards. Think about the business. Last but not least, speaking of the business, I want to talk a little bit about how you view self-serve in data science. We already have heard some pretty strong opinions with you around the job of a data science team needs to extend beyond just support, being a support organization. But I would think that maybe you'd be interested in setting up self-service, like that might help allow you to be more strategic if you are actually standing up a self-serve arm. So what recommendations do you have there?

INDY MONDAL

Yeah. And I think this is something that we talk about a lot in my team, because every time you hire a data scientist, every data scientist that I know wants to work on the coolest data science and ML problems. We heard all these things; 90% of data science is spending your life, like plumbing, all those things are true. And the only way you can get out of it, if you think about your analytics maturity, you go from this descriptive reporting all the way to ML and AI. The only way you could go there, if you can somehow automate or make self-service the rest of it. So one of the things, and the reason I have such a strong product focus in my data team, that's one of the reasons, because I want to make sure that we're building products that drive self-service.

My product managers are taking, understanding how we define, like what's the success metrics for the product that we are building, so that option and all those things. We are SLAs, the goal is we build instead of hundreds of dashboard, a set of core products and capabilities. In the past, my engineering team, generally when you have a data team, your data engineering team is building pipeline. I hate, in a strong word, that my engineering team builds pipeline. I would rather build services that other people can just build pipeline. It is to think about more... And I think it's also a shift from 10 years ago. 10 years, everything was at the data teams, you have all these people, ETL developers, people are building pipeline.

And I think modern data organizations think about it as a product organization, building capabilities and also services that can ... They're modular, but also they can talk to each other in a programmatic way. So if you're building a model in your data science, make sure that you have a set of exposed interfaces that other people can consume from. That's one level of self-service. You're building dashboards for the business to do self-service. So if I get to a point where I don't have to do any of these 80% question that come on a daily basis, and we can either automate that or create self-service, you have your data scientists who are much more qualified to do that, have the time to do deep dive building models and be more strategic, which is really hard to do, because they're focusing on all these things like, "Hey, can you send me this data? Can you send me that data?"

DAVE COLE

Yeah. When I hear self-service in the world of data science, I'm typically, in my head, I'm thinking about dashboards. But what you reminded me of, it's not just building out the end presentation layer and making that self-serve. I think of BI tools and other things, but it's also the data itself. Thinking, instead of asking for data and then having a data engineer build out some pipeline that creates a file or populates some table that you then need to use, think more from a service-oriented architecture approach, thinking holistically what are all the data elements that somebody might need? And then how do you create an interface to make it more self-serve to allow the business users to actually be able to retrieve that data in a way that hopefully keeps the data quality and data integrity very high and allows them to get to the right answers in a great way? Do I have that right?

INDY MONDAL

Yeah. And I think that the way to think about it, Dave, is to think end to end. Self-service doesn't really happen, like dashboard is not self-service, that's just one thing. Self-service means end to end, so you have self-service in your data capture there. And again, I understand not every data science leader works end to end, but I do. So you are building capability for capturing your telemetry data. You're building capabilities, self-service for pipelines. You're building self-service for your model, how those models interact with each other. Even when you're building models, if you are a data scientist who is constantly building models, and you have lots of features that you are constantly training your model on, think about turning those into some sort of a feature store that other people don't have to do the same thing that you've been doing. So thinking about the entirety of your stack and not just the self-service means I need to have dashboards for the business. They're just one stakeholder. One of the things that I did back in Hulu, I own an experimentation practice. And in Hulu, my team built the experimentation platform and capability of doing that in Hulu. We're doing that in DocuSign as well.

And we always talk about our goal is to make sure we are building the platform in a way, the first level of automation or the self-service goes to the engineering team who is actually configuring the experiments. Second layer, it goes to the analyst who is going to be doing that so anybody in the company can do the experimentation analysis without our design, without us being in the picture. And last, you build the capabilities where the product manager can launch their own experiments. It's not all going to happen on day one, but you are constantly thinking about self-serving from one layer to the next layer to next layer. And when all of it's done, you drove self-service for the entire system.

DAVE COLE

Right. That makes a ton of sense to me. I think it's a way to drive scale as a team. Like if your team instead said, "No, no, no, you send us the 27 experiments that you want to run and we'll go ahead and run them," then you're going to be creating this massive organization, and you're probably going to have a lot of frustrated PMs, engineers, et cetera, who are like, "No, no, no, we know exactly what we need. Can you just teach us how to fish, or give us the tools to fish, give us the rod, the reel and whatever?"

So building out that platform, to me, is a harder problem to solve and probably a more interesting problem, I imagine, for your team to solve. But it also allows you to scale as an organization and also gets your counterparts in other departments a lot closer to what you do, as well. I think it sounds like a win-win. It also sounds like it's harder to do, but I think... Sorry, go ahead.

INDY MONDAL

It's also more rewarding.

DAVE COLE

Yeah, exactly.

INDY MONDAL

Tell me one thing. If you have to scale and your definition of scale is going to hire more people, where are you going to hire all these people from?

DAVE COLE

These days? Yeah, good luck.

INDY MONDAL

And then the second thing is, think about the modern business doesn't have that sort of time that the businesses did 10 years ago. Like if somebody comes to you, like, "Indy, I need to understand this," you cannot really tell them, "Come back in a month, let me build a pipeline." You have to react, because in two weeks the business would've moved on. How we respond to the modern business, with the speed of the business, with the speed of the data and the quantity of the data, you can't really scale if our solution is it's the people who're going to scale. I always try to figure out how can I do more with a less amount of people, because that's the way you can scale.

DAVE COLE

Yeah. I think if your only approach is to try to scale by just hiring more... At least, I don't know about you, Indy, but personally, I'm ready to go interstellar in my recruiting efforts to try to find people. The market out there is pretty darn crazy.

So I think you're taking the right approach. It's smart for these times. I also think, to your point, it's more rewarding. Hey, I really enjoyed our conversation. We touched on a whole bunch of topics. I learned a lot. I, for one, would love to join your team. It sounds like, based on what I heard today, it sounds very interesting. If people want to reach out to you, can they hit you up on LinkedIn and say, hello?

INDY MONDAL

Yeah. Please reach out to me on LinkedIn. I'm very, very responsive and active on LinkedIn. So, yeah. I'd love to chat to any of you who would like to reach out to me.

DAVE COLE

Awesome. Well, Indy, it was a pleasure. Have a great weekend and take care.

INDY MONDAL

All right. Thanks a lot, Dave. Have a good weekend, you too. Bye. Bye.

DAVE COLE

Thanks.