How to Operationalize, Scale, and Measure AI in Life Sciences

DAVE COLE

Hello, and welcome to another episode of the Data Science Leaders podcast. I’m your host, Dave Cole and today's guest is Sidd Bhattacharya. Sidd, how are you doing today?

SIDD BHATTACHARYA

Good, Dave, how are you?

DAVE COLE

I'm doing great. It's a Friday so I'm looking forward to a nice weekend and having some fun outdoors. Living with COVID in the last couple of years has been good for podcasts, but everything else is pretty crappy. I am looking forward to spending some time outdoors.

For our listeners: Sidd is the director of healthcare analytics in AI at PwC. He has over 10 years of experience, primarily focused in the life sciences industry.

Today we're going to be diving pretty deep into the life sciences. I'm going to be doing my best with Sidd here, to try to connect some of the things that he has seen within the life sciences industry to the world outside of life sciences. I think there are some interesting approaches to data science and some interesting things that happen in the life sciences world.

Without further ado, let's just dive right in, Sidd. Tell us a little bit about how you got into the life sciences. I also want to know what you think is unique and interesting about the life sciences world.

SIDD BHATTACHARYA

Sure. Thanks, Dave. I'm an engineer by training. I went to school back in India and I came to the US for grad studies in engineering and business. I've always had an engineering mindset and point of view. Then I joined consulting companies. Right now I’m with PwC, which is a great management consulting practice.

I've always been in the life sciences space. I remember I started way back, building systems for sales process optimization. It was back when I had more hair; you can't see me but I'm bald. I've always been in life sciences; being in consulting gives you a very good rounding. I've been in technology, I've helped companies define their R&D strategy, commercial go-to-market strategy, post module integration. I have a broad range of experience but I have always been an engineer at heart.

Somehow, in 2016, we were doing a strategy for a large pharma life sciences company. Some of the problems they were encountering at that point, to me who had come from an engineering background, were just ripe for AI. They were talking about problems of ingesting data, having physicians and clinicians review the data and make interpretations on it. My mind went directly to, "Well, AI can do this right?" In 2016, with NLP and some other machine learning models, I got started. That's how I got into AI. I keep joking that logistic regression has always been around, and that of late it has become a fancy AI term. I'm not complaining. I'm riding the wave.

DAVE COLE

I think everyone who's ever been to a doctor's office sees how much paper they're pushing around. I think life sciences and the healthcare industry are ripe for disruption. What are some of the similarities and differences that you see between life sciences and other industries?

SIDD BHATTACHARYA

If you look at the use case, the application level, I would say they're very similar. In every company and industry, people consume data. They try to understand what's in the data and they try to make decisions based on it. Then they output the decisions. If you take that as the basic process, that’s how we work as humans. We read stuff, we analyze it and then we put it up: you have an output. At its core all use cases are the same. The difference with the life sciences industry versus a financial services or chemical industry, from my experience with those, is the nature of data.

In life sciences, you're dealing with patients’ sensitive data. Some of the decisions in certain cases can have a direct impact on people's lives. You had better be careful about the data and make sure it's manageable. That's one big difference I would highlight, versus other industries. You have to be extra careful and sensitive. That all leads to the regulatory aspects. This is one of the most regulated industries in the country. You have the FDA overseeing it and you have similar governing bodies in other countries, for data and regulations.

DAVE COLE

I have the answer to this question, but how does the FDA impact life sciences companies? Where do they intercede into data science projects?

SIDD BHATTACHARYA

The FDA has a big interest in data science projects. Even before AI became a thing, they were always into data science. When you run a clinical trial, for example, data is collected and then you have a group of biostatisticians in the pharma companies and at the FDA who would analyze the data. If you think about it: as your drug goes through a clinical trial process, what you're collecting is product data, and you have to submit that data to the FDA. The FDA has to have people to analyze and understand the data, charts and graphs that came out of it. The FDA was always into data science and bioinformatics: it has just become smarter now.

DAVE COLE

What are they looking for? Is it on the patient side or is it more on the production of the drugs? Where is their oversight with regards to the data science and the life sciences business?

SIDD BHATTACHARYA

The FDA adds scores and is interested in making sure that the data science outputs that are produced are within a controlled environment. You make sure that you use data in the right way, that it has no bias in it and that the models you're using are tested. And at the end of the day there's a human being overseeing the outputs of the model. They, at its core, are interested in making sure it's a controlled process and that safeguards are put in place. What they don't want to do is have a model learn on its own. We use the term ‘continuous learning’ in the industry.

DAVE COLE

It's not going to fly if the FDA's not going to approve that.

SIDD BHATTACHARYA

No.

DAVE COLE

So, human-in-the-loop approach: you mention your primary use case as being clinical trials. You also mentioned biostatisticians. How is a biostatistician different from a statistician or a data scientist?

SIDD BHATTACHARYA

That's a great question. I try to explain it to people by explaining that biostatisticians would work in regression models. They would work in computer engineering models. The simplest way to think about it in my mind is as data vs. language or, more appropriately, numbers vs. language. Statisticians essentially deal with numbers. That's what they run their models on. That's what they make their predictions on. Model data scientists, if I can use that term, try to understand human language. They're trying to understand humans' vision. That's also a type of a language. The simplest way to define it is as numbers versus language. Traditional biostatisticians deal with numbers. Modern computer engineers deal in language.

DAVE COLE

There's a bio element there too. Obviously, they have expertise in biology as well?

SIDD BHATTACHARYA

That domain expertise is core and that's where AI is making such a big impact. If you looked at AI a couple of years ago, people treated it as an IT project or something that’s implemented as a technology. A big change that I'm seeing in the industry: people have finally realized that this is extremely domain-specific. To your point, people need to have that biology understanding. They need to have medical understanding to do any work related to AI. I'm glad to see that companies realize it now.

DAVE COLE

It's the one role where we talk a lot, on the Data Science Leaders podcast, that you as a data scientist need to have a strong background in business understanding. The closer you get to understanding the business, the more strategic you can be seen in the eyes of your business counterparts. It's built into the name of a biostatistician, right? You have the statistical background but also you also understand the biology, which implies that you understand the nature of the business. I wonder if, in the future and outside of the role of the industry of life sciences, that there might be more of these hybrid roles.

SIDD BHATTACHARYA

Absolutely. In fact, I would say that it's the most important role. If you look at the overall data science and AI landscape you have data engineers and traditional data scientists. The role in the middle, which is the bilinguals or, as I call them, product leaders, is the most important role. You can get data scientists who are extremely good at data science, very smart people. You can also get data engineers who are extremely good at the engineering aspects. But, if you don't have someone in the middle connecting the dots, you’ll have nothing more. That is a role that's very important. That's where I feel the majority of the talent crunch would be. We'll get over the data science talent crunch because data science is now being taught at schools. A few years from now and we'll not have that problem, but we will always have the problem with respect to bilinguals.

DAVE COLE

Bilingual, I like that.

Let's shift gears. Let's talk a little bit about operationalizing AI within the life sciences industry. What tips and tricks do you have there?

SIDD BHATTACHARYA

One of the things that is a little frustrating to me is that a lot of leaders still think they can just do an AI project or put one model into production and declare victory. While you can definitely do that, it's a short-term way of thinking. Think back to when the Internet was just up and coming: Macy's had a website. Every company that we know, including all retailers, had a website. But just because you had a website, did not make you an Amazon. It requires a lot more than just having a project or just having a tool to say, "I'm able to operationalize data science." In my mind, I always question what are the key ingredients to make it happen. One is the top-down vision. You are able to link metrics across business, data science and IT, to success. You cannot just say, "I want to do data science." It has to be measurable.

You have to have an ROI on your projects. You have to identify the right use cases. The ROI part is very important because whenever you start an AI or data science project, you have to write ROI calculated for it and you can measure it. The other thing is data strategy. I hear a lot of people talking about the importance of data, its messiness and how you need to spend years getting your data ready. I don't believe that's possible. You can never ever get your data ready. It's a mindset shift where people have to be comfortable with working with imperfect ‘dirty’ data, releasing products and projects with it. It's waiting for Superman that results in it never happening or driving business success. That's another thing when you think of operationalizing data science: having that mindset shift is very important too.

DAVE COLE

I think there's probably something in the middle. You don't want to have data sitting out on an S3, all over the place, massive data. On the other hand, if you're looking for perfection, you’re at as much of a data disadvantage. You're going to have imperfections and it's more important how you handle it. How do you play defense? That probably is a better approach than looking for that perfection.

SIDD BHATTACHARYA

Going back to what we were discussing, that's where the bilingual role is very important. Ask a data scientist: they're never comfortable releasing anything into production until it's 99.99% accurate — their fun score. That's where you need the bilinguals to make it happen.

DAVE COLE

Understand tolerance. Talk about ROI. In order to compute ROI, maybe have a use case or a project that you worked on. What are some tips that you have with regards to measuring ROI. I'd love to hear if it's specific to your industry.

SIDD BHATTACHARYA

From an ROI perspective, it's very important. Let me give you an example. I work a lot in the pharma R&D space. One of the problems that we have in the R&D domain is the volume and variety of data. You have data coming in from different sources. Then you have trained nurses, clinicians, and physicians having to review the data and make determinations based on it. Then it needs to be reported out to the FDA or an internal governing body. This is most acute in a place called pharmacovigilance or safety case processing. We did a project back a few years ago, with multiple companies, and the results were all always consistent. We were able to automate the entire process of capturing data, analyzing data and then reporting on top of it, using a combination of NLP machine learning models and simple rules. From an overall timeline reduction perspective, it resulted in 40-50% reduction in the timeline that it took, vs. the old process. Those are huge numbers if you extrapolate them.

DAVE COLE

Sure. Let me slow you down. What is pharmacovigilance?

SIDD BHATTACHARYA

Every pharma company in the world has this organization they're accountable for, capturing adverse events as reported for their drugs. If you take drug X and you have a headache, you are not supposed to have a headache. Either you report it or your physician is supposed to be reporting it into the pharma company. When the reports come in through call centers, these are emails or they could be written on napkins. The pharma company is supposed to take it. It's a fiduciary responsibility. That's what the group does. It's a well organized group with extremely talented people, but a lot of the work they were doing was data entry, data capture and data analysis. This has changed with the advent of AI. Now they're just reviewing the recommendations from a machine and looking at those outputs.

DAVE COLE

Got it. Pharmacovigilance, my guess has always been around (understanding the adverse impacts of drugs). These days we're inundated with ads on TV that have these long disclaimers, that my kids always make fun of. "This drug may cause headaches, fever, even death in some cases." I have you to blame, partially, for that. I imagine you bringing that into the data that you give back to the FDA, as it comes in and as it was written down. That's why NLP makes a lot of sense.

SIDD BHATTACHARYA

It's unstructured.

DAVE COLE

It's unstructured. These are the symptoms that were seen after this patient took this drug. It used to be that a human or a nurse had to read through exactly each one, then look at the symptoms and categorize them in some way. Is that right?

SIDD BHATTACHARYA

Exactly right. What AI is able to do now, is called narrow AI. I also call it boring and narrow AI, but we are automating slivers of what a human being can do. Humans are good at reading and you have NLP models that are very good at reading now. As humans, we are good at making basic analysis decisions. You have machine learning models who can take those inputs from what you have read and make the decisions for you. That's how the process can be automated.

Dave, you're talking about pharmacovigilance, but this is applicable throughout. You have other use cases in pharma life sciences, in the commercial world, like all center data processing: instead of having a call center agent take notes, you can have an audio transcript and a model going into these annotations, creating a report for you. This is something that a human had to do.

You have examples in supply chain management. One of the coolest things I came across was one of the large pharma companies, global pharma companies, using computer vision for quality analysis. They have cameras on their production shop floor. The pill packs come out, you have computer vision, the camera is taking pictures to make sure that it's of quality, it's of the same level, the liquid is of the same level, or you have the right number of pills in place. That's pretty cool, right?

DAVE COLE

That's very cool.

SIDD BHATTACHARYA

You have actual machines augmenting human work.

DAVE COLE

All this is driving efficiency, unlike the call center example. If the transcript is slightly off or wrong, it's not that big of a deal. In the world of pharma, if it works or not, miscategorizing or not categorizing a certain symptom is something we certainly don't want to happen. That's where the humans come in and it sounds like you're seeing 40-50% gains. Is that how you're comparing ROI? Is it a pre/post? We look at the world beforehand, we look at the world afterwards, we look at the time savings. Is there anything else that we should be thinking about?

SIDD BHATTACHARYA

In my mind, ROI is calculated in three ways. There are three key parameters. One is cycle time or time that we talked about it. Second is cost, typically associated with cycle time, savings, cost savings — these also come into play. We have seen anywhere between 40-50% savings rates, again, and the third thing is quality and consistency of interpretation: making sure something is auditable.

We keep talking about explainability in AI and how models need to be explainable. In my experience, a lot of human decisions are also not explainable, but if you have a model in place that's consistently making that same mistake, you can tie a thread back and say that there was something wrong with the training data or in the way that the model was architected. That's the third aspect of measuring ROI. It's not quantitative, but qualitative. Quality and compliance is also very important.

DAVE COLE

If you had a single human categorize data, as long as that person didn't get tired, there's bound to be a certain amount of consistency there. If you have multiple people doing it, even with guidelines in place, it's never going to be perfectly consistent.

SIDD BHATTACHARYA

Never.

DAVE COLE

That's where NLP can really help. To your point, even if you do identify a mistake in your model, if you fix it, theoretically you can go back and retroactively clean up the data, which is also nice and you can do it at scale. It certainly makes sense to me that the results of these models are always being reviewed by humans in the end, just to double check, but what they're actually doing is just spending a lot less time than if they did all this manually from start to end.

SIDD BHATTACHARYA

Yes. The nature of their work has changed, from actually having to do the work, to reviewing the recommendations or the outputs of a machine and then providing feedback.

DAVE COLE

You touched briefly on computer vision and on the factory floors where the drugs are being made. If you look through, what are some trends that you're seeing within the life sciences, with regards to data science and AI?

SIDD BHATTACHARYA

There are two key things. I'm seeing a lot of investments, a lot of efforts, in the drug discovery space, not just clinical trials, around drug target identification. If you look closely enough, every major pharma company has made an investment to the tune of millions of dollars into an AI startup in the last year or so. You have a plethora of AI startups now trying to figure out and make drug discovery simpler. How do you do better protein identification? How do you do some drug simulations? The simulation of trials and drug interactions has been invested into by all these pharma companies. They have made investments in AI. That's another area that I'm seeing a lot of traction in.

DAVE COLE

Obviously this speeds up research and development?

SIDD BHATTACHARYA

Correct.

DAVE COLE

Is the bottom line to narrow the scope before you actually do a trial, having some idea as to the efficacy of the drug that has been manufactured and created for the trial itself? That's certainly interesting. I'm curious about there being multiple startups. Are they all trying to solve the same problem or do they specialize in slightly different things?

SIDD BHATTACHARYA

It's all of the above. There is a lot of money available now. I think I read a report somewhere, that in 2021, they invested over $10 billion globally in helping drug discovery and development. That's a lot of money available and every startup has a unique angle. Some of them are doing simulation modeling. Some of them are doing digital twins. Some of them are more on the science side, trying to figure out how to best sequence some molecules. That's why it is.

DAVE COLE

Great. That's fascinating.

Let's talk a little bit about the art of doing data science within the life sciences. Do you feel the way the teams are organized align with more focus on research and development? We talk a lot about the operational aspect of data science, but also doing that R&D type work. The R&D work that's being done by the life sciences industry is actually generating drugs. What recommendations would you have to a data science leader within the life sciences space, in terms of how they're organized and how they do data science?

SIDD BHATTACHARYA

You need to have what I call the foundations and the right data platform. You need to have solutions that can scale up the work that you do. The most important part of the recommendation would be around the operating model: making sure you have the right talent mix. Many times I see life sciences leaders just say, "Oh, I have a bunch of data scientists," or "I have 50 data scientists. I have 100 data scientists."

And in my mind, that's great if you're trying to do proof of concept or experimentation. If you're trying to scale up something, you need a different talent mix. I call it 50, 40, 10. 50% of your talent or effort needs to be focused on delivery, on understanding the use case, focused on change management during the ROI. 40% of your talent needs to be focused on the data engineering, the infrastructure elements, making sure that the pipelines work, all of that. Only 10% of your effort should be on data science.

If you're looking to scale it up 50, 40, 10 is the mix we would recommend. Oftentimes I see the opposite happening. People think that hiring data scientists who are absolutely important and amazing people can help scale up. That's one thing. The other aspect is just making sure that you think of this holistically. Having one use case or one model in production is not the end of it. You need to make sure people are taken care of. It's a different career path you need for data scientists and data engineers and bilingual. It's not your traditional career path that people have typically defined.

DAVE COLE

The 50, 40, 10, that's a hot take. I haven't heard that mix. Only 10% are data scientists.

SIDD BHATTACHARYA

If you're trying to scale up.

DAVE COLE

Sure, sure. The 40% who are data engineers and ML engineers, and your effort team, I get that. The 50% on the delivery side, though… tell me a little bit about who those people are. Are they some of the bilinguals, as you're saying, or are they project managers?

SIDD BHATTACHARYA

It's a combination of project managers and bilinguals, people who understand the ROI of things and can do change management. Those are the four key categories. Bilinguals, project management, change management, and those who understand the ROI of things. That's where the majority of the effort is. You can build the best models in the world with the best engineering pipelines. If people don't adopt it, it's of no use. If you're looking to scale up AI, to put AI in the hands of more people, you need to have that 50% effort/talent base to do that.

DAVE COLE

I'd like to drill into one of those areas, which is the change management aspect. How have you seen the life sciences industry embrace machine learning and AI? It certainly seems they have, if they invested the $10 billion in startups. I would've done that 10 years ago but, whatever, that's just me. What are some of the things that you would recommend to a data science leader who's maybe struggling to integrate data science in a real way?

SIDD BHATTACHARYA

In general, looking back on when we started in 2016, 2017, and where we sit today, people are more comfortable with the notion of AI. They realize Skynet's not happening and that it's pretty dumb AI for now. People realized that it's here to help. It's going to augment our work. We can do different things. Telling that story is extremely important because if you just talk about AI in abstract and say we are going to use AI to automate this process, people will not adopt it. We are humans. Our first instinct is survival. That's how we have survived for all these eons. Understanding that and making sure the folks are able to realize that AI is here to help augment their work is the first step.

I keep going back to this. We can have the world's best models trained on the world's best data, with a foundation in the best engineering. If people don't adopt it's of no use and you will not see your ROI. It's a big portion of making any AI project or any digital transformation successful.

DAVE COLE

That makes perfect sense to me. I think phrasing augmenting versus replacing is important, as in staying away from the R word.

SIDD BHATTACHARYA

For sure. Yes.

DAVE COLE

Automating or just saying, "We're going to augment this process with a couple models here and there just to make everyone's life easier and improve the quality,” is important.

I'm going to put you on the spot because with regards to what's going on in our world today. I don't know if you have a perspective on this but, certainly right now, there's a lot of discussion going on with regards to vaccines and COVID. Coming from the life sciences space and knowing how drugs are tested, what's been your impression with everything that's going on in the world today?

SIDD BHATTACHARYA

Honestly, if you go back to pre-COVID days, people had a very different perception of life sciences. For good and bad reasons, people did not think very highly of the industry. That is something that has always troubled me because I've worked in this space. In every single company I work with, I can see the passion that these people have for patients. I can see the passion that they have and the care they take in designing a drug, testing it and making sure it reaches the right patients. What COVID has done is not only highlighted the potential that American drug companies have in terms of research, but they have also highlighted the fact that they're producing stuff that's helping patients.

I'm very happy that this has happened, although the pandemic is obviously a horrible thing. People are able to now see life sciences in a different light. I hope that continues because the passion that people have in designing and delivering these molecules is tremendous.

DAVE COLE

You're taking a very optimistic view. I think if I were to step back, I think pre-COVID, maybe post-COVID too, the perception of the life sciences space of pharmaceutical companies is that they're all out for the money. Right?

SIDD BHATTACHARYA

Sure.

DAVE COLE

I think the big challenge there in what you probably recognize is that it takes a lot of money and it takes a lot of research to produce these drugs. That's what I think a lot of people don't quite recognize. Once the drug has been made, that's the tip of the iceberg. Although that's the final result, there have been years and years and years and thousands upon thousands of hours put into making this drug.

That's usually why it gets priced the way it is now. Other cynics in the audience would say it also has to do with the demand and other things. That's when it gets a little gnarly, but then you're also painting a very optimistic view, which I love, Sidd, of how pharmaceutical companies today are being perceived with regards to the vaccine. I agree with you. If you were to have told me that within the first year of this virus manifesting itself and spreading like it did, that we would have a vaccine, I would've said, that's crazy. There is definitely a lot of skepticism with regards to the vaccine and whether or not it was rushed to market and whether or not it was tested.

There are millions and millions of people who have used it. Obviously there are people who have had side effects, who have reported those, and that's going into your pharmacovigilance models, being categorized and so on and so forth. I think the cynics out there would say this process was just too quick. Do you believe machine learning played a role in the speed with which this vaccine was manufactured and that there's a new era, in terms of being able to produce these drugs with regards to time-to-market?

SIDD BHATTACHARYA

Absolutely. This is all information in the public domain. One of the things, not just a vaccine, was a company called Benevolent AI. It's a UK-based company. In the early days of the pandemic, they used AI to look through their database of publications and they were able to find a molecule that they thought could help in diminishing the viral load.

NIH took that on and they have computer studies on it. AI not only helped in delivering some of the vaccines faster, it also helped in identifying some of the medicines in the initial days. Based on my firsthand experience, AI played a significant role in automating some of these processes like looking at data that's coming back from the vaccine trial. Having hundreds of people look at it, your AI models go into the first layer of cleaning, standardization, predicting where COVID will be from a location point of view. Where would you find the maximum number of patients to test your drugs? That's another area where AI played a pretty big role. I know I'm being an optimist.

DAVE COLE

I love it. I mean, that's certainly the way I've looked at it. I think this is nothing short of a miracle. It's something that we should be celebrating. The statistics are out there in terms of if you've had the vaccine and if you've been fully vaccinated. The likelihood that you will have to go to the hospital is obviously much lower than those who have not been vaccinated. The likelihood that you have adverse effects, including death, is also much lower.

SIDD BHATTACHARYA

The other angle here is that we have never had a COVID pandemic, obviously. It's not that we had AI models pre-trained on this, versus any of the other diseases we see. Despite having those limitations, we have seen AI help. If you translate that to any known disease area, the impact would be much higher.

DAVE COLE

I also think that there has been a crash course on statistics. In the early days of the pandemic, I'm guessing you were the same: all of us were looking at the number of cases and there was a lag in terms of the number of deaths. There's a lot of parallels between that data, getting the ground truth and all that other good stuff. I think a lot of folks like you and I would say, "Oh, it's 93% effective," and these things.

We recognize that there's a 7% chance you still might get COVID or you still might have to go to the hospital. Each one of those cases where somebody goes to the hospital, there's a news headline, "Oh my gosh, this person was fully vaccinated, went to the hospital and..." Well the question is: are they part of that 7% or not? I'm not putting words in your mouth but when I read articles, I wish the author was a little better-versed in statistics, and the readership as well. That's why we have this podcast to help educate the world.

SIDD BHATTACHARYA

Yep, yep, yep.

DAVE COLE

Awesome. Well, Sidd, on that note I really appreciate you joining the Data Science Leaders podcast. If people can reach out to you, what's the best way to get a hold of you?

SIDD BHATTACHARYA

I'm on LinkedIn. I try to post pretty frequently on AI in the life sciences and healthcare industry. You can follow me on LinkedIn.

DAVE COLE

Awesome. Well, Sidd, you have a great weekend and thanks for your time. I really appreciate it.

SIDD BHATTACHARYA

Thanks, Dave. Great talking to you.