"AI Insights: Transforming Finance, Legal Systems, and Business"

For a video version of this podcast, please visit https://youtu.be/awA5M7qJdE0. 

Ever wondered how a quantitative strategist from Goldman Sachs transitions into a leading AI entrepreneur? Join us for an insightful conversation with Hari Balaji, co-founder of Egregore Labs, as he shares his journey from the world of high finance to the innovative realm of AI. Hari recounts his professional evolution and the pivotal moment that ignited his passion for artificial intelligence—using advanced data analysis to interpret client reactions. This episode offers a fascinating glimpse into Hari's career, his return to India, and his collaboration on groundbreaking AI projects.

Discover the technical and business hurdles Hari faced while co-founding Egregore Labs, a venture aimed at revolutionizing finance with data-driven solutions. Listen as he discusses the complex decision-making involved in aligning technological breakthroughs with practical financial problems. We explore the unique dynamics of the financial services sector, the strategic choices made, and the indispensable role of human-made rules and collective perceptions. Hari’s insights provide a deep understanding of the financial industry's idiosyncrasies and challenges.

In the latter part of our discussion, we explore the transformative potential of Large Language Models (LLMs) in finance and their limitations. Hari offers a candid assessment of generative AI, emphasizing the importance of human-machine collaboration. We also touch on the future of AI, the pitfalls of overfitting models, and the evolving landscape of AI technologies. Aspiring entrepreneurs will find Hari’s advice invaluable, as he stresses the significance of selecting the right problems to tackle and understanding the enterprise ecosystem. Tune in for a treasure trove of knowledge and practical advice on navigating the intersection of AI and finance.

For more insights and discussions on the intersection of technology, law, and business, subscribe to our podcast and stay updated. Connect with us on social media for live updates, behind-the-scenes content, and more. Thank you for listening, and don’t forget to share your thoughts and questions in the comments or reach out to us directly!

Mail us at hello@quoqo.com or visit our website at www.quoqo.com.

Chapters

• 0:02 Transition From Finance to AI Entrepreneurship
• 12:10 Technical and Business Challenges in Finance
• 28:16 Navigating Enterprise and AI Integration
• 33:12 The Future of Artificial Intelligence
• 43:30 Future Trends in Artificial Intelligence
• 54:12 Effective Orchestration in AI Implementation
• 1:03:31 AI Negotiation in Finance Industry
• 1:14:16 PDF Text Recognition and LLM Challenges
• 1:21:29 Advice for Entrepreneurs in AI

Transcript

Chetan: 0:02

Hi everyone, thanks for joining another episode of Code Console. Today I have my good friend and fellow founder, hari. Hari Balaji has a very illustrious career and has been involved very deeply in AI since a long time. So first of all, welcome Hari to the podcast.

Hari: 0:25

Thanks, thanks, chetan. Absolute pleasure to be here and to be able to speak to your audience.

Chetan: 0:31

And, just to add, hari has got a stellar academic career to both with the stellar AI credentials as well. Hari has had a very illustrious career on the addition to his stellar academics and he is a graduate of both IIM Ahmadabad as well as IIT Madras. He's also had a very stellar quant career and he started with Goldman Sachs and all the way then led to creating or being the co-founder of Egregore Labs and I'm tracing two decades of his career in a couple of sentences. But he's had a very storied background in this and he's probably one of the people who is very well qualified to talk of all the latest things in ai and how things are, uh, shaping up.

Hari: 1:28

So welcome once again, harry, and thank you for your time thank you for this yeah yeah, thank you for that uh sort of uh very warm introduction.

Chetan: 1:37

I hope I live up to that reputation today not at all, but you know, just to start start off, it's very interesting. I mean, you were at Goldman Sachs, you were a quantitative strategist there and then you ended up co-founding Egregore Labs. I mean, what inspired the shift, and can you just talk about a little bit of your journey?

Hari: 2:03

Yeah, sure sure about a little bit of your journey. Yeah, sure, sure. So, yeah, so I started my career at in, sort of in a very quantitative role right, and that sort of went on for about a decade and so I think in many ways sort of finance as a field was a bit accidental for me, in the sense that I didn't grow up wanting to be in the financial services industry, Sort of. I went naturally with the flow and I landed IIM Ahmedabad at and you know, the placement process took me into the financial services industry and particularly in I mean I would like to specify here that it's largely the capital, the secondary services industry, and particularly in I mean I would like to specify here that it's largely the secondary capital markets, right. So, spending a lot of time working with derivatives and you know, fundamentally I'm a nerdy guy and sort of like to sort of tinker around with things and work with numbers and so on, as I do Chetan guy and sort of like to sort of tinker around with things and work with numbers and so on as a as a you chetan, and so I think that sort of naturally made it a comfortable place to start my career and sort of things took off from there, right, and obviously you know, once I mean a huge respect for for goldman, amazing people you get to work with and obviously a global platform and very interesting sort of problems that you get to solve on a day-to-day basis, and that sort of kept me there for a decade, almost right, and I think that was a very interesting phase of my career.

Hari: 3:42

And sometime around the time when I was 35, I sort of, you know, wanted to move back to India. To a large extent, that was driven also by personal reasons and, you know, wanting to be close to family and my wife's burgeoning career in India and so on, and I also was very fortunate at that time to meet my co-founder so around 2015 is when I met him and he was on to some pretty interesting ideas at that point. So, being a quant, I had spent a lot of time working with numbers in the beginning, but then subsequently, beyond just building complex financial models and such, went on to also do a lot of selling of these ideas. Right, and that extended beyond just the idea of, I would say, building numerically interesting products, but also things that were problem solved to align with prevailing regulatory, accounting, tax, legal and other frameworks, right, and so essentially sort of trying to build something that is not just maximizing revenue for a given risk but also is, from a regulations perspective, aligning with many of these constraints and rules, right. So it was a fairly interesting thing to work on.

Hari: 5:07

But around 2015, 2016, I think, one of the earliest problems that I sort of encountered, which took me in the direction of what was then an LP, which will now get subsumed under AI, was the idea that, let's say, if someone, let's say you know, the top research person at Goldman, releases a new research report right Now, this is sort of you know, let's say extremely famous and well-regarded guy. Everyone wants to know what he is going to talk about and he releases a new report right Now, how do we sort of understand what the reaction of our client base is to this report, right? So I think that is sort of one of the, let's say, toy problems that I spend a lot of time thinking about, because the traditional way of doing this would be to disseminate that report and then to have you know people at different parts of the world who are sort of, you know, sending that out to their clients. Then, you know, reach out, ask them what did you think about it? Or like organically get some feedback, collate that, summarize that and then it sort of comes back 10 levels till, let's say, the head of research, after a week or two weeks, will get some kind of a summary right with some talking points. And the idea was that is there a way to sort of you know, organically be able to get this information by observing conversations right, by sort of looking at mail interactions, looking at, let's say, chat interactions, identifying when this particular report was shared and when it is being talked about as a topic, and being able to bring all of that back together in one place? Obviously, this is a huge data security and privacy nightmare, so it is something that would probably never actually happen.

Hari: 7:03

But, as a part of thinking about this problem, spent quite a bit of time teaching myself various aspects of. You know NLP and you know at that time it was all about parts of speech tagging and things like that Right, so very early Stanford NLP library. You know very and I think it was just sort of the. What was making it interesting was that you know very and I think it was just sort of the. What was making it interesting was that, you know, the idea of this whole CBOC program right, like a bag of words and sort of word to vectorization and sort of those sort of early ideas were just sort of coming about when I was starting to look at the space.

Hari: 7:43

When I was starting to look at the space so around then I sort of met my co-founder and he was in the business of market data, right. So the large number of financial services players in various capacities buy data right. It could be as simple as, let's say, what is the price of a stock, but also, you know, let's say, market depth or positioning. Or you know, let's say market depth or positioning, or you know, let's say, it could be dividend data, corporate data, like a variety of data gets sold, and most of that data is quantitative. And this gentleman was, you know, very interested at that time in exploring whether qualitative data feeds could similarly be created and sold right. Data feeds could similarly be created and sold right, and so it seemed like a very interesting problem to work on.

Hari: 8:35

We spent quite a bit of time, you know, thinking about whether that is something that we want to, you know, do something about right. So that's kind of where sort of the journey itself started right, sort of the journey itself started right and yeah, I mean, I think the sort of where it took us was in a completely different direction, but but essentially, yeah. So it is sort of in some ways sort of you know, let's say, an accidental start where I was looking to move back and work on you know, and do something a bit different, and this was an interesting and exciting space. I had some exposure to it and since I was anyway moving back to India and I wanted some kind of a reset, it felt like a good idea to sort of try this out and see if we can do something here.

Hari: 9:22

Now, obviously, I think the beginning the conversation was that, hey, I'll work on this for a couple of months and then we'll see where it goes. But very quickly we got excited about the idea. I think the most important thing is that at that time the co-founder relationship really clicked and it made sense to sort of, you know, try and build a company around it. So that's kind of how uh got started here. So that's sort of the long-winded answer of how uh the switch from, you know, being a quantitative strategist to becoming a founder yeah, no, I think that's very interesting.

Chetan: 9:57

Uh, one of you know, as an aside, uh, back in 20, uh, 2014, uh, one of the things and we've had similar paths in the sense that I used to. During the credit crisis, I used to work pretty closely back in London, especially in offshore funds, and during the crisis, a lot of these went down as well and there was a fair amount of liquidity crunches, if you remember, at that time, and one of the things that I happened to work on was something called private investor back in 2014. I got together a small team and actually built a secondary market for some of these things market, you know, for some of these things yes.

Chetan: 10:45

And we ended up, sort of like even you know getting some regulatory approvals for some of these things as well. But you know, the truth of the matter was it was too early for the market at that time. It still is and it's taken a very different turn. Now it's become a market for understood securities.

Chetan: 11:07

but if you look at it from a global perspective, correct? I think it is, uh, what you know, just building out. You know building out an entire engine at that time, if I remember. And we, you know, you know getting the valuation of a company correct. You know, this is something. How do you arrive at a valuation of an unlisted security?

Chetan: 11:28

All these things were fairly new then. The computing power was nowhere to what is available right now, and so the algorithms were literally handcrafted. So we know from all these things, but that's on the side. But very interesting. I happened to work with a bunch of fonts at that time as well all friends but it was quite an interesting field, even today. A lot of people did try and attempt it in India after that. But once again, the domain knowledge plays a very important role for all of these things as well. So I think that's one thing.

Chetan: 12:08

But back on this. So you started Egregore Labs and you basically started looking at the data-driven approach to finance, and it comes with a whole set of challenges, especially, and some of the libraries that we spoke about and things like that have undergone a sea change as well. And when Transformers came out from Google, that changed quite a few things. Some of the Python libraries available today, which all of us take for granted, were actually getting built at that time. Correct? And you know how did you solve some of these technical challenges when it came to sort of finance and I'll talk a little later about the LLM challenges when it comes to finance as well correct, and? But you know how did you initially sort of? You wanted to take a data-driven approach. How did you solve some of the engineering challenges that came with it?

Hari: 13:11

Yeah, so, yeah, I mean I'm trying to think about the right way to answer that question. So let me sort of at a very high level, talk about, you know, some of the issues that we encountered as we started this. So, first of all, I think the core challenge and I have to be very careful not to over-index on my experience too much here and sort of try to keep it a bit generic, right and so I think the core problem in this entire space, which we had when I started, as well as I think to some extent, we have today, is that this is really a solution, looking for a problem in some ways, right. So, you know, if I have to think back to sort of my start of this journey, right, I think what was really, I didn't sort of get into this saying that, hey, I want to make money or, you know, I'm very excited about this problem and I want to solve it right, though at that time I just got in saying that, okay, this is very exciting, very interesting. I have not encountered something like this before. It feels like the future and I have to be part of it and I have to understand and learn and I want to.

Hari: 14:30

Basically, you know, there was like a sort of a passion for it, right. So I think that's sort of how we got it Right. So, in many ways, I think the core challenge almost becomes and I think this is true today as well uh is that you start, uh, you basically sort of take a solution and then you sort of try to see how it can be fit into solving some problems that work uh for you right, and in our case, the chosen area, which we understood was really financial services, uh, right and uh, and you know, the idea was that sort of how do we sort of then, uh, you know, uh, resolve this problem and, um, what, what problems do we pick to solve here?

Hari: 15:12

right, so I think we picked the one which we understood the best at the starting point, uh, which was that okay, uh, you know there are people buying market data. They're going to use that data to, let's say, uh, trade in the market or make financial purchase decisions, you know, buy and sell stocks or derivatives or whatever. Now can we add something based on all the other textual data, all the other information, news, etc. Etc. Can we sort of come back and give people signals which tell them what to do in the market or which they can add to their you know, sort of repository of existing, you know data tools in order to make a decision? Right, so that's sort of the problem we set up for ourselves Right Now.

Hari: 15:58

Again, as it's true now, it was true back then the rate of evolution of technology has been, or, like, what is available in terms of tools has just massively exploded, right, so there's traditional approaches, which is probably what you've picked up in college or what you've picked up, you know, in the past and then sort of. Then there is basically things which you find in textbooks, right. So back then you would, if you sort of wanted to do a course, it would take you through something like parts of speech tagging right, like sure, there is something like vectorization or neural networks or LSTM, all of these stuff, things are coming up, but it really requires you to sort of spend a lot of time investing in, I would say, technology and ideas which are still very early and which could very quickly get discarded because the next wave came by. So the biggest challenge at some level is to sort of from a technology perspective, is to understand what level of depth you want to go into things. When do you want to sort of just say, okay, I have this library, I don't fully understand it, but let me think of it as a black box, throw stuff at it, see if it responds and, you know, move forward. Or do I really need to sit down and understand this deeply and what is the level of depth, right? So I think that is sort of one, I would say, technical challenge that we spent a lot of time on right, without sort of going into specifics or details.

Hari: 17:35

Second thing is also and this is very specific to sort of, I think, financial services, but also certain other fields, right, which is and sort of you know. Let me sort of start by actually talking a little bit about the name of the company Agrigor, right? So Agrigor actually means it's basically like a group thought or a collective fiction, right? And so one of the inspirations behind that name is that if you look at sort of the entire financial services space, a significant portion of it is made up behavior and made up rules. It's not gravity, right? Like in the sense that everything ranging from regulations to taxation to you know law, these are human rules made by humans. And especially when you look at you know, prices. I think what you're going to see is that, you know, it is a function of human behavior, right, and not particularly a representation of the truth. In many ways, right. So, as they say, the markets in the long term are a weighing machine. In the short term are a voting machine, right? So the problem in some ways is that you have, let's say, largely historic data to model the future behavior of the market. But as information keeps coming in, as new tech gets adopted, the model of human behavior and the universe itself is continuously changing, right. So it is actually a very difficult problem to solve, I mean, and I think it is basically never going to be completely able possible to model. You know the way prices work in this market Now, second thing being that in the investment universe you may have the best model, et cetera, but what people care about is whether what you're providing is able to create incremental information or value which can change that decision. So that's what we call alpha Now, here again in the markets that we were sort of trying to build a SaaS product around.

Hari: 19:40

One of the challenges is that if there's a piece of new or interesting information, as long as it's available to one person, it is very valuable, right. The moment it's disseminated to five again, it becomes table stakes and there's a lot of value, right. So trying to sell data or insight as a SaaS business, right. Then there's sort of a negative economies of scale right in the beginning, right. So where it's better to, at some level, uh, you know, take your own capital and trade the trade, the market, as opposed to try to build a data business, right. So there are all these sort of challenges.

Hari: 20:31

Um, that that came about. I know they're not particularly technical, but and we've gone a little bit all over the place, right but, um, essentially, from a business perspective, I think very, um, uh, you know, you know, unique and differentiated challenges come about if you're trying to build software for this particular part of the puzzle, which is people who are looking to make money by investing or trading or positioning themselves in the capital market. So that's kind of where we started and in some ways these are all zero-sum games started and in some ways these are all zero-sum games. Subsequently, I think we had that insight and we moved away from this market to looking at unlocking, operational efficiency in middle office, back office support services and things of that sort, and those are non-zero-sum games, I mean, those are basically pure value unlocks. And it became sort of, you know, let's say, from purely a technical problem perspective, they were far less exciting, but from a value delivery perspective, I think they sort of were much, much more fulfilling and valuable.

Hari: 21:42

And I think that was sort of also our journey and as we matured as entrepreneurs, I think that was sort of also our journey and as we matured as entrepreneurs, I think that was sort of how we moved away from, you know, working on this problem we started with towards in this direction, right, and there, I think the challenges were a bit different in terms of I think the biggest thing that we had to work with is that there are, again, I think it's a classic problem which all aspects of AI are facing today, which is you may have a cool piece of new technology, it's even delivering value, but you have to really solve for user adoption, right, right, yeah, and I think that is sort of where, uh, you know that was a a different problem, but and we can go into details of that but that sort of um, you know, was the biggest challenge when we moved in this direction.

Hari: 22:34

Uh, right, so again, I've sort of gone all over the place, but, yeah, I mean, uh, uh, let me know if you want to sort of uh, pick on any one of these things and unravel the threads a bit deeper.

Chetan: 22:44

Yeah, yeah, I think one of the things and unravel the threads a bit deeper.

Chetan: 22:49

Yeah, I think one of the things is you've talked about moving from a consumer-facing initial set of ideas that you had to a more institutional-facing set of products that you wanted to build Right, and this usually comes sort of inside and I've seen a lot of other founders in the fintech space.

Chetan: 23:08

You know it's sometimes it's very, I think it's it's more enamoring than anything else that sometimes your own capital comes into play, especially if you're building a B2C sort of like a business and you hope that some of the things that you're building actually and with your own capital actually plays out.

Chetan: 23:30

But the real value in what you're building, especially around some of the deep tech, around finance or some of the other functions that we spoke about about really lies in institutions, isn't it? And institutions, to some extent, may have their own internal teams to sort of solve some of these problems as well. Sometimes, as an institution grows bigger, you know that internally it may not be possible to build or innovate as much earlier, and therefore that becomes an opportunity of play for companies like this. So is that how you looked at the problem and said, okay, what were the steps that helped you decide, saying that I'm going to focus on institutions, this is what I'm going to be building on, and what are the products that you actually decided were the right things to possibly fit into the institutional space?

Hari: 24:33

Got it. So I think that. So again, we I think a little bit are not the, you know, the normal. We didn't have sort of the sort of little bit of exceptions to the rule there, right, Because both me and my co-founder had spent an extensive amount of time working with large enterprises, right. That meant that we sort of inherently understood the challenges that they face and in many ways I think the challenge for us was not that we didn't know how to pitch to them or sell to them, and in our previous avatars we had spent a lot of time talking to institutional buyers, so we had exposure to enterprises. So I think that was less of the challenge. That was less of the challenge.

Hari: 25:29

Obviously, the harder part for us was really putting something together which met the, let's say, the criteria, or which passed the smell test of an enterprise and could gain enterprise adoption, right. So that is the real challenge here, and I feel that for many entrepreneurs, you know, getting the foot in the door with an enterprise should be less of a challenge, right, because I think access today is, you know, quite easily available and if you want to have an opening conversation, I don't think that's very hard, right, and it's not sort of a world where, I mean, every large enterprise today has hackathons, has labs, has, you know, various initiatives and means to engage with startups. Therefore, I don't think that is ever. That is much less of a challenge today, I think the bigger the harder. Conversation is really, sort of what do you do once you get that foot in the door and how do you get across the line in the medium to long term and build a sustainable relationship with an enterprise? Sort of not very well discussed or described? So let me sort of make an attempt to sort of walk through some of them, right, I think the first thing really is sort of you know how do you deliver your product to the enterprise, right?

Hari: 26:54

So, typically, like, no enterprise is just looking at just your product, right, I mean, they have a problem. They already have, let's say, dozens, if not more, internal software, various softwares and databases which are already interacting with each other. Right, and therefore, sort of you are going to be an incremental player in that mix, right? So the first thing really is to understand the state of their technology footprint and to figure out if you can be in the I mean, if you're a young startup, like, figure out if you can be in the marketplace of any one of their existing softwares, right. Or find a way where you can natively integrate and operate piggyback on one of those existing things right, because that immediately means that you don't have to go through that. You know extra long enterprise sales cycle, which involves not just you know finding the right buyer, demonstrating value at various levels within the company first, you know doing various decks and demos and so on, but also lots of questionnaires, cybersecurity reviews, data reviews. You know lots of other, let's say, hurdles to cross. And even beyond all that, when you finish all of that, you might still find someone who is in charge of vendor consolidation, right.

Hari: 28:16

Enterprises are always in the mood saying that, look, we are dealing with too many vendors. Can we find a way to reduce the number of touch points? Saying that, look, we are dealing with too many vendors. Can we find a way to reduce the number of touch points? And if the business or the billing that you bring to the table and especially as a startup, you will basically be quite modest in your first engagement, it really requires a lot of willpower for them to spend time engaging with you and bringing you into the ecosystem, and then, beyond that, they'll also worry about whether, if they develop a dependency on you and you go belly up, how does that? Where does that leave them? A lot of these challenges. My recommendation always is to try and find an existing piece of software and to be able to piggyback on.

Hari: 29:00

But, that said, if you do decide to go and direct, then that brings its own set of challenges, which is you know, but you have a very clear idea of you know who the buyer is. How is the buyer going to internally look good because of your product? Then figure out who your user is, because your user and your buyer will not be the same person, right? And you need to sort of spend a lot of time thinking about you know what metrics you will track? How will you ensure that users adopt your product? Why should they care, right? So I think that is the second part.

Hari: 29:35

And then there might be various other gatekeepers. And what would you want to do in order to sort of, you know, navigate that ecosystem? Right, and see, you may have a gatekeeper who's an internal player want to do in order to navigate that ecosystem? You may have a gatekeeper who's an internal player who might say you know what? Why are you spending time on the startup? Give me the budget, I'll build this for you. Or there might be someone in the ecosystem could be a procurement person who has a favorite vendor already is basically fighting to give them a seat at the table when it comes to this particular mandate and to compete directly with you, right?

Hari: 30:08

So a lot of these things to navigate so beyond sort of it's not going to be product-led growth, right? It's not that you build a great product, it goes viral and people just adopt it. Never happens in the enterprise sort of ecosystem. It's always going to be this kind of slow grind and you will need mentorship, guidance, a champion in-house who's able to tell you the tricks of the trade and help you with the ropes for you to be able to navigate the ecosystem.

Hari: 30:40

But, that said, I think the good thing about enterprise business is that once you build and you land and you sort of manage to establish a relationship and get your first mandate very quickly, two things are going to happen. The first thing is that typically, because they don't want to keep adding new vendors, they will typically throw new problems at you which are related in other departments. Same problem in a different department, similar problem in the same department. You will get all of these opportunities to sort of expand right and enterprises are also very sticky. Typically they will pay on time, so all of these kinds of things become possible. So very sticky business, dependable, reliable business. Also, logos are incredibly valuable. But that said, it takes a significant amount of effort to get across the line. I mean, this is generally what you would see, right.

Chetan: 31:37

That's absolutely. I think you've hit on pretty much every hard spot in the enterprise journey. Hit on pretty much every hard spot in the enterprise journey, especially in the price change journey. But it's very hard to get in and even harder to get out after getting in Right right right.

Chetan: 31:57

That's essentially how you would conceptualize, but just shifting gears there a little bit Hari. There's a sea change since, uh, since then, ai came into the four, essentially right, it's impacted so many things. Uh, especially I mean there are. When it comes to products, the way that I look at it is older products, maybe even on the sas side. You know, trying to sort of include or move towards ai and ai native products, right, which basically you know were built straight up. You know, utilizing gen ai capabilities as well.

Chetan: 32:33

But you know we are probably I would like to think we're probably a couple of months away from having general ai. You know, come into, come to the fore. You know, if you look at some of the latest mobile phone launches as well and we're doing this in July 24, some of these things are even going to be integrated. If you look at WhatsApp, for that matter, there is an integration with Lama 3 right now. If you look at Ola, there's an integration with Crude Trim. So you're looking at existing products, showcasing some of the latest innovations as well in this space. But when it comes to finance and financial products, let's talk about this what's been your general experience on using LLMs for crunching financial data and coming at an analysis has it met your expectations? Because I have my own thoughts on this. I'm not going to say, I'm not going to sort of, but I want to hear from you, sure Saying, who has built both traditional products as well as looking at. So what do you think, hari?

Hari: 33:45

Right. So yeah, let me sort of step back and try to think about or provide like a framework, for you know how I think about the space, and then I think we can sort of then take the specific instance of financial services and and go deeper, right? So, the first thing being that I think obviously we need to completely respect the hype cycle, right? So, where everyone in the beginning was like you know what is this, j&a, I think, should I even care about it? And then a couple of, I would say, consumer-facing demos happen, and suddenly it feels like magic, right, and what happens is that we over-index on the few things that we see which completely wow us, and then there's a halo effect and we think that this can do everything. Now, and I think the specific thing that Gen AI has triggered, which other previous generation software have not been triggered, is this kind of Turing test sort of breach, which is that anything which can emulate a human being is suddenly interesting to us. So if I show you an amazing robot that is able to go and clear landmines, okay, but it looks like a tank, it's not interesting. But if Boston Scientific comes up with a robot that looks like humanoid, right, is doing some jumps and, you know, climbing stairs or something. Suddenly we're like wow, right, because in our heads, not just because it echoes like some Terminator or some other you know media franchise that we have been fed on, but also anything which is close to human or emulating human behavior is certainly interesting to us, right, and I think what Gen AI has managed to do is it is in that sort of inside that particular circle on the Venn diagram, right? So it has certainly garnered a lot of interest.

Hari: 35:56

Now, that said, I think the biggest challenge here is there are lots of unknown. Unknowns, right, in the sense that, you know, every day you can go to various forums on social media and you will find, like you know, posts of the form that, look at this, you know, really particular specific, anecdotal heart problem which I, you know, interacted with the LLM and it solved it for me. Or, like, here is some really, really easy thing which, you know, even my three-year-old can solve, but you know, look at how stupid this thing is, it's not able to solve it. Right, and you can extend that to Kingdom Come. There are so many parts, so many problems or so many challenges or tasks which either we sort of take such a human benchmark to the problem solving that if something that's hard for humans is being solved very easily by the LLM and something that's hard for humans is being solved very easily by the LLM and something that's easy for humans is unsolvable by the LLM, we find that as points of interest right and this is a completely false comparison. Right, it's happening because, for whatever reason, that whole chatbot experience or, like you know, subsequently the voice interaction experience, is fooling us into mapping LLM capabilities directly to human capabilities and doing that direct one-to-one comparison. Obviously that's also raising questions on whether human jobs will be wiped out and all of these kind of things.

Hari: 37:31

But I think, frankly, the way to think about it and I think this is the right model. And you know, I know Jeff Bezos has not spoken a lot about AI, but one of his interviews he said that this is not an invention, it's a discovery. Right, and that's very true in that it's very much like, you know, dynamite or penicillin, where we have sort of you know, found something in the lab and I mean its behavior is a bit difficult for us to understand. It's still being fully mapped out and therefore we have to kind of spend a lot of time researching it before we fully understand its capabilities. Right Now, what it cannot do is already reasonably apparent, and a lot of well-regarded doyens of the field have called it out which is is incapable of, you know, first principles, reasoning the way humans do. It is just a next token. You know, prediction capability, and while, uh, you know, you can, uh, you can say that, hey, what I mean?

Hari: 38:28

The counterpoint to that? So? So the idea here being that, given a particular problem, it is not sort of actually sitting down and reasoning through it. All it's doing is generating the next token right, and that creates this sort of emergent behavior which convinces us that there's some reasoning going on.

Hari: 38:54

Counterpoint to this is sort of what, uh, you know ilia talks about, uh, which is, if you basically created a novel, uh, a detective novel, where you had, uh, you know all the clues, everything in it, and you fed, and the last letter of the novel on the last page is the name of the killer right, and you feed everything except that, except the last word, to the LLM and with 100% certainty it generates the name of the killer right, then is that not reasoning? Because you know, you don't understand it, but it has used all the data provided to it to come up with that last word, and so, in effect, it has reasoned and figured out who the killer is right Now. Is that not? Is that not what reasoning is right? And now there are various ways to sort of counter that.

Chetan: 39:37

But this is sort of the ongoing debate, right, but coming back to finance right now, uh, because, before you start that, uh, harry, I'll just pause you there for a minute, right, and one of the things is you know about the reasoning which you talked about is that I, if you think about how a human arrives at a decision, the older you get, you have more data points on which you will take a decision as well. Your decisions may be, at most, based on six or seven data points at any given point in time, or even lesser. I'm just generalizing, but this is what the human brain is capable of doing at any human point in time. But if you take large volumes of data, where there are thousands of probabilistic intents that are available, on which data can be processed and provide the illusion of reasoning Correct, you know, even if you think of man as a reasoning animal, it's basically you know. You would think that it's based on the sum total of all the experiences that that person has Correct and it's very it would be very similar to say an LLM as well.

Chetan: 40:44

You know, the larger it gets, you know, I mean the answer to the universe and everything cannot be 42 always Right, but how you think about it, but how very different is it? You know, at the end of the day. It is made by humans, so it's very distinct and you know, the reasoning is very similar to humans as well, unless you know, it reaches a capacity where it's able to think of a completely different system. Correct, which we have probably also not thought about. I'm just saying, how difficult is it to basically? I mean, the larger the model, the more difficult it becomes to control it, and the smaller the model, the easier it is, because it's got limited information. Is that generalizing too much? What do you think?

Hari: 41:31

Just picking your brains on this one. So, yeah, generalizing too much or what do you think you know? Just just, yeah, so, so yeah, I mean, I think, yeah. So I think the most interesting thoughts and work on this has been done by francois chole, right, and he talks about how to measure intelligence, and I think the most interesting thing in the space of llms I've been following over the last couple of days is this arc agi price, right? So the idea being that here is a bunch of logic puzzles and you know they all sort of involve. You know there's a grid with some squares and you need to kind of. You know there's some coloring which is happening there and you need to kind of do the pattern recognition there. And the idea behind sort of, uh, you know, creating this price and uh, prize and throwing these problems at LLMs is because there isn't a large enough prevailing data set on which sort of an LLM could have, let's say, where.

Hari: 42:27

So a lot of the traditional benchmarks for LLMs. The challenge that they have is that as you keep training and as you keep sort of trying to optimize for benchmarks, so actually let's take a step back, right. So today, if I want to, let's say, come up with a new LLM and I wanted to get attention. Right, it will not get attention unless I publish benchmarks. And there are sort of these different benchmarks for LLM capabilities which have now become sort of you know, broadly accepted. Unless I score higher than prevailing models on those benchmarks, no one is going to really look at my LLM right, unless it is a very specific LLM. It's focusing on one very narrow set of tasks, which could be translation or which could be, let's say, only one domain like you know, let's say you know pharma or medicine, or you know molecular biology or one of these things.

Hari: 43:20

If I put out a general purpose LLM, the expectation is that it will beat prevailing, you know, commonly available, open source LLMs on those benchmarks. Now, that is actually, I mean, the reason because we've fallen into this sort of into this paradigm. It is very problematic because what you end up doing with any new LLM is you sort of end up training it to the point where the benchmark data starts leaking into the LLM training set and therefore what you've got is something that beats all the benchmarks but practically, when people try to start using it, it's not really performant, right? And this is a big challenge, I mean it's true of all kinds of problems where you keep solving to sort of I forget the specific quote but essentially the measure or the metric has become primary, rather than building a better LLM. So you just want to build an LLM that beats the metrics.

Hari: 44:26

So inference and legacy also play. Correct.

Chetan: 44:30

Also come into all of these things.

Hari: 44:31

Correct, you're overfitting to those benchmarks Right, and what the RKGI sort of initiative is trying to do is to come up with a problem to which sort of LLMs are not exposed in just crawling the internet or being trained on data sets and things like that. Here's a completely new sort of problem set on which a lot of information is not available and there isn't enough training data to make the LLM an expert at solving this problem, which means inherently you have to take a different approach. Right Now people have tried to brute force, in the sense that they take the existing data set or the problem set, they create lots of examples and they try to take the traditional approach to solving this problem. But the hope is that there will be sufficient incentive for people to unlock a new way of using LLMs or generally available technology approaches to solve it. But I think this is sort of the direction I think which is actually quite interesting, because otherwise we are building things which are more of the same, throwing more tokens and building larger neural networks and building larger neural networks.

Hari: 45:53

I don't know if more is just going to be better. I think we are sort of getting to that point and I think there are sort of some other interesting ideas as well, which is I do expect that Transformers itself as the unit of operation or unit of construction rather, could change, because we haven't gotten to the point where it's something that everyone understands. There's enough research on it. You can easily sort of pick up and build with Transformers, and I think that sort of made that an interesting choice. But it's not at a point where it's a QWERTY keyboard, right, in the sense that if you find a better, let's say I would say, a unit of I mean a better thing to sort of build on, then I think we will probably see that shift happening at some point, right.

Hari: 46:57

So we've seen that with I don't know, rnn, cnn, lstm, lstm, I think, is what I've personally spent a lot of time building on top of, before transfer models came by, and I think nowadays we hardly talk about them, right, and I think nowadays we hardly talk about them right, in fact, the, if you look at this right. I mean in some ways again, I think I digress, but it takes me back to my engineering days, where so I passed out of engineering in 2004, and at that time again, I think there was this passion to work on the latest and greatest and coolest technology right, and, if you remember, this was a period when there were no smartphones, but phones were progressively getting smaller right and we were packing more and more and more into one, into that tiny device, and, uh, that was the interesting bit, right.

Hari: 47:48

So I spent my uh, you know, final year in uh in in university, working on wireless technology. And the idea was and there were all these ideas, if you remember, gsm was the prevailing sort of technology back then, and then CDMA had come about, and I think it was Reliance, the other Reliance, which had launched a mobile network and a phone with CDMA.

Chetan: 48:17

Which was in fact better technology as well. No, no, completely.

Hari: 48:23

The point here being that, fast forward a couple of years, right Now, we know these acronyms 3G, 4g, 5g but there's a small universe of people around the world, who are a small subset, who actually are working on these technologies. The level of abstraction, or the layer at which you want to build technology for the mobile phone has become the App Store, it's become Android. It's become Android, it's become iOS, it's not. You know how do I build a better technology at the level of, you know, I mean, the BlackBerry sort of layer of tech is gone, right, that is sort of now monopolized by a few players who are sort of you know, spending a lot of time getting better at that, whereas everyone else is focused on building at the next layer.

Hari: 49:21

Now, if you take the same thing here, there's TSMC, on top of which there's Nvidia, on top of which there are the foundational model companies, and then there are people building one level on top of that, right, uh, now what can happen is that we could see a reshuffle in this as well, just like like we saw back, I mean, just like back in the day uh, the people who are able to extract the largest uh, you know unit of value are not the folks who are building smaller phones, uh, but the company that was able to create a new layer of experience on top of the phone and unlock a massive adoption. The same could happen here, right In the sense that today all the value capture is or rather valuation capture is happening at the level of LLMs and foundational models. We could see a change where those become quite commoditized and all the value capture is happening at the next level or the level even above that.

Chetan: 50:19

So we don't know. I think that kind of that's a very interesting thought, because if you look at some of the latest NVIDIA edge devices, these can run some decent LLMs. These can run 7 billion 8 billion parameter LLMs very easily. It's not far away that you can typically load. I think. If you've got an Apple 15 or a device beyond that, you can easily run an LLM on that and you don't essentially need anything more than a more. I think that's where we are heading towards as well.

Hari: 50:56

Yeah, completely, and to your point, what you were saying earlier the app store for LLMs. Maybe it's not the custom GPTs on OpenAI, but it's again the app store, the traditional app store. We could very quickly come to that world. It is lots changing in the space and remains to be seen how things settle down.

Chetan: 51:25

With Nvidia, especially with all that technology leapfrogging, they'll have two or three generations in a single year. I've seen this happen as well. It's unbelievable how some of the technologies have evolved and how tiny these have become as well. That can technically be deployed, even on-prem. That has a bearing on enterprises. It has a bearing on how you deploy for enterprises and back to financial services and things like that. You also know that, given your experience, that enterprises are a different ballgame, it's not necessarily always public cloud that they would prefer, and this has all sorts of bearings on how we would develop Gen AI. But before you start, I just wanted to also mention you know, I think it's also down to use cases that you would use for these elements as well Correct, correct, and on Google I just wanted to mention here one of the things is you know, in our country, india, there are a lot of cases, right, which are pending, cases which at this point in time, there are like 50 million plus, right, right, and you know you have also a set of people who cannot afford legal services as well, right, right, and you know it's an obligation of the courts to appoint lawyers, for that matter, to look into all these cases.

Chetan: 52:52

But just imagine, you know, a single LLM can be trained and we have the technology today and the capability today to basically make sure that each of these cases are properly handled. You know the support also will be given to the decadent and things like that, and actually quality support is better. It's not just somebody that the court thought straight to a point. So there are lots of use cases which we can typically solve, and one of the things that also comes up in our country is even today I see an article in the newspaper saying that, okay, there's going to be some legislation that puts guardrails. They're looking at it as a technology that will probably take away jobs. It's not being looked at as a technology that can really streamline quite a few things as well. But just as an aside, saying that the technology is available today, you can train it to be a judge on cases and it is possibly much faster than anything else that that that could come about as well. You know for all of these, but you know that's my thoughts, uh, that's my thought.

Hari: 53:59

Yeah, actually, I mean, I'm actually gonna, I'm actually gonna counter you on that, right? I mean, um, I'll actually let me. Let me use that same point right to segue into the challenge we face in financial services as well, right? So? So typically, when it comes to anything which is any automation, what you will see is that LLMs are no exception here. Unless they can genuinely reason, your performance is only as good as your model, choice of model, parameters and features and so on, and your training set. Now, what that always means is that it's very easy to build a model that is going to work, depending on the difficulty in the problem. Between 60 and 85 or 90 percent of the time, you can get there very quickly, but it is very hard from there to progress to something that is successful, from a 90 to a 95, to a 95, to a 98, 98 to 99, and then so on. That is a challenge in general with any kind of ML model or with all of AI, which is that there is always going to be edge cases that slip through the cracks, and especially if you have a bad actor who is hunting for those cases, they will be able to find it. So this is true.

Hari: 55:32

So, if you look at, where have we seen very immediate and quick adoption of llms? Uh, it is in your customer support, uh, in marketing, in in writing copy of various sorts, in doing summarizing uh, probably in replying to emails. Uh, in search, right, and even in search, for example, it is a b2C use case, right Today, like Google's charges, you know, exactly zero for someone to do a Google search and therefore for them, let's say for search, to get disrupted by AI. I mean because you're not charging anything, I mean you're not going to see. I mean it is sort of a low touch, high volume use case, right, and all of these will see very quick LLM adoption because the cost of a mistake is not very high, whereas when you come to legal accounting, finance, these are all situations where you know one mistake I mean especially in financial services right, all the value lies in that one exception and catching that before anyone else.

Hari: 56:44

Right, and you know that's when there's a huge opportunity, right, if everyone thinks the same way, then the market prices in fairly and there is no opportunity for you to make money, for example. All the value and all the wealth lies in that, in those situations where you got something right, but everyone has it wrong. Okay, and it's very difficult for, let's say, any kind of automation or AI or ML sort of situation to be able to catch this on its own. So that sort of leads immediately to what I would think is that all of these spaces will always be solved in a human plus machine way plus machine way right Manual.

Chetan: 57:31

Re-hybrid. But you know just to digress on that as well you know it's not that you will be directly plugging in an LLM output to a use case, especially in these functions, right Right, there is a fair amount of orchestration that's involved, especially in financial services, right Right? You cannot simply plug in an open AI or a Lama or something else and then expect that it will solve issues. So I mean, there's a fair amount of orchestration, there's a fair amount of prompt engineering, there's a fair amount of error checking and all of these things. But does it get you there? 90%? Would you rather have 1 million pending cases or would you have 50 million pending cases?

Hari: 58:18

Oh, 100%, 100%, yeah, no, I think this thing, yes, correct. I think it becomes a trade-off and I think the way you will probably get there is that the plaintiff and the defendant, you know, sign a release saying that I'm going to. I know that this is not a perfect process, but it's a quicker process, so I'm going to basically go ahead and allow for this situation to be tried with an AI judge or whatever. I think there's probably a way to get that going completely. I think the so to yeah, 100%. I think in that way I mean, if you put it that way, definitely I think there's value here.

Hari: 58:59

The other sort of thing which we've seen a lot in ops, which may also be a way to resolve this, resolve the problem in the legal universe, is kind of some kind of, I would say, polling approach. So, for example, what is called, let's say, a six-side check in financial services or in back office or what would be, you know, essentially involve some kind of a double-blind approach to it, where you have the same information presented to two people and they both evaluate it. They don't know each other's answers and if the response differs, then a third person is given the same problem to solve and that person is the tiebreaker. So I think there could be approaches like that where, uh you know where you have uh you, you purposely build, uh you know uh, two very different sort of uh approaches to problem solving, or two different. Let's say, ai is to keep things uh very simple, and if they both both align, then that's fine. But then, beyond that, their conclusions are presented as a summary to a human who then gets to resolve or make the final decision.

Chetan: 1:00:24

In fact, that's exactly one of the approaches that we've taken in one of our applications as well.

Chetan: 1:00:32

In terms of when you're actually drafting, let's say, a contractual document or any legal document, and you want a choice of different classes which is based on the context of whatever you're drafting, our system will actually give you a list of basically, it takes a weighted average of the three most important things that we think are there and they'll give you a choice, right, so saying you can choose whatever you want.

Chetan: 1:01:03

You know from any of those things as well, but you know that's basically emulating the human mind as well. You know, to an extent, right, so you'd always want some options. And then you know, to an extent, right, so you would always want some options. And then you know you would want to pick and choose. And it's not, and even the early days of CAD GPT used to give you like a couple of options, right, right. Similarly, you know, until it became, until the LRM became big enough that it could just ignore all of these things as well. But you know, I think you know that's an excellent approach as well to some of these things, where you're giving accurate answers but you're letting the user just make a choice in all of these things as well.

Hari: 1:01:47

Yeah, actually, I mean just to segue from there. I'm very curious to get your thoughts on one thing which I have been thinking about in the legal space, which is, if you remember, sort of this whole episode around AlphaGo, the sort of the AI which defeated Lee Sedol and basically became the best Go player in the world. So the way it sort of learned to play Go was essentially at least the second version of it. They just let the machine play against itself and progressively get better and better and better that way. Right, and progressively get better and better and better that way.

Hari: 1:02:30

I always wondered, if you know, I don't know if there's a way to correctly present this problem, but I think that there's a way to. There should probably be a way to teach negotiation, contract negotiation to AI is, in this way, right, where you know, essentially like it's sort of presenting the two competing you know bots, if you will, and they are presenting arguments to each other, and then you kind of iteratively keep, you know, taking the winning bot and making it play against itself until it gets really good, right, and to the point where in the end, I have a product where I can say here is a contract, here are sort of you know some clauses that I really care about, and here is what I'm willing to give up. You present that narrative to the bot and then now it runs the negotiation for you. Uh, you can do that today.

Chetan: 1:03:24

you know, with some of these, you know, and uh again, uh, pitch for myself on the products as well. You can have playbooks and you can basically run a set of tasks where you could figure out what are the differences. Is there a deviation? How would you tackle the deviation, and things like this as well? So all these things could already exist today, but this has been. Have you let? I mean, if you run a chess program on a Mac or a Windows machine or whatever it is, mac has all those things built in and you let it play by itself. The end result is always a stalemate. It's a draw, you know. And this is exactly what's going to happen if you let two bots, you know, argue it out, because there's no end to it, right?

Chetan: 1:04:20

So and then, you know, the human becomes arbiter. At the end of the day, Got it. So this is, and this is typically an approach where and we've experimented with this in our data analysis department as well it usually always results in a stalemate, With no side actually coming out on tops, Because each of the bots thinks that it has to protect its users'. Interests. But in an action negotiation it's always a give and take. So the give and take is not necessarily, and you can always program the bot saying that you can give this, you can take this and conclude things are correct.

Chetan: 1:05:02

But I think the give and take abilities in a contract negotiation are much more critical than legalistic points about If you just throw, you know this you throw a bunch of legalistic points about. Sure, If you just throw, you know this you throw a bunch of legalistic points the other side is going to walk away. So this is exactly what happens. And there are also situations in the real world where you deal with large enterprises where it's not necessarily that you have the negotiation or the bargaining skills. They will just take my paper or leave. This is exactly what's going to happen, sure.

Chetan: 1:05:35

So in both these cases the bots will not. It may work in consumer contracts, but in business in reality at least the way that it's worked now for several thousand years is there's always some give and take and it's very unlikely that you can take very hard positions unless you have a very large business, and it also depends on the discretion of how much you need the contract than anything else. But we've done this, but there are slightly different approaches that we took after this as well. But I'm happy to show you some of the stuff that we've done this. But there are slightly different approaches that we took after this as well. But I'm happy to show you some of the stuff that we've done. And it's actually in production, got it.

Chetan: 1:06:21

But back on this and it's. You know we can keep talking on this and all the greatest stuff, right. But back on finance, right, I mean using LLMs and finance. Right, and we talked about some challenges earlier as well. Have you looked into this? Have you thought? I mean, are there better ways or are there limited ways in which you can use LLMs and finance today?

Hari: 1:06:47

Right, yeah, I mean, I think definitely. Okay, let me start with sort of a little bit of explaining the state of where things were before LLMs came by, right, so definitely, I think, even with the previous generation technology, which would be some form of what we would call AI, but not large language models, we were making a significant amount of progress and being able to unlock various capabilities in various parts of the financial services industry. So if you look at I mean, if I have to sort of break down the challenges in the industry, two, three or four core problems, I think the first sort of, I would say, highly repeatable, solved problem, for which answers existed even the previous generation, was where you know what I would call some kind of form filling or structured data processing, right? So typical example could be that you know someone has filled out a tax form, right, the tax form doesn't really change from, you know, person to person to person, I mean you sort of year to year to year or instance to instance.

Hari: 1:08:07

Probably once in three years, let's say, the tax form might change. Someone has filled this out. Now how do you process that and how do you sort of automate the subsequent actions to be taken? Now, this level of problem, which is, I would say, processing reasonably semi-structured or highly structured data. I think this has been completely solved. It's now a commodity. There are many, many players in the market.

Chetan: 1:08:32

And there's a lot of RPAs.

Hari: 1:08:35

I mean, I would not even call this. We're not even at the level of RPAs, right. We're simply like if I have a simple API, it is accepting, I tell it that I'm processing a tax form. You send the PDF or the document or whatever. It figures out what tax form this is, gives you the response back right, this could be tax. Tax is one example. Then we move to things which are, I would say, structured, but where the format is not in your control, right, which could be let's say, bank statement processing, right, you know bank statements are highly structured.

Hari: 1:09:07

I understand, typically what a bank statement should look like, but you know if let's say unknown bank X, how their statement is actually presented. How do they sort of, what is the layout of it? I don't know that, right. And now if I have to extract data from that and, you know, do some numerical work on it, then you know how do I extract the data, right? So this is the next level of problem, and where this gets actually quite complex and difficult is actually in the insurance industry, especially in the US, right, where you may I mean everyone needs to have property and casualty insurance. You may typically change your insurance provider once every two or three years. You may typically change your insurance provider once every two or three years and what will end up happening is that your new insurer has to quickly process your application, read your entire claim history, which are presented in what I call loss-run documents across various past insurers that you've had, who have various formats. Understand all of that and then come back with a quote, right, and it's a time-sensitive process because you know you are it's like you know going to a policy bazaar or an insurance bazaar type of setup where you said I'm looking to buy insurance, here is my historic data. Hey, all five companies give me a quote and at the moment you get three quotes. You're going to pick the lowest and move forward? Right, so you can't be. You have to be quick. At the same time, you have to be accurate, and there is a winner's curse element involved here.

Hari: 1:10:39

So, again, this is the next set of problems, right, where the data is structured but you don't know what the format is. And then, finally, you come to things which are completely unstructured, right, which could be, you know, financial statement notes or, let's say, legal contracts, or any long-form document like, let's say, a sustainability report, where you literally don't know it's in human language. You don't know if the information in here is, what is the relevance of it, what do I need to do with it. And there's a lot of what I call stuffing that can happen here. Right, so you could, let's say that you know you have, as a company, put your hand up and said that you will meet certain you know sustainability targets, right, obviously there are some core numbers right there, but, for example, you could bury all information in like a 300 page sustainability report and that could basically, you know, confuse or hide information, right?

Hari: 1:11:45

So very classic example of this is let's say you have committed to less than one fatality per 100,000 personnel or one injury per 100,000 personnel in your factories, these 14 subsidiaries which we have gotten as a result of acquisition over the last five years, where processes are yet being revamped.

Hari: 1:12:17

Okay, this is a small carve out, but practically it means that you have not achieved your goal right now. Should you be penalized for that or not, as a subjective call? But if someone just threw that you knowpage report at the LLM where some headline number said goal achieved less than one fatality or one injury per 100,000 people, then it will seem as if that has been achieved. So now we are entering into territory where Gen AI can actually unlock some value, right, because traditional approaches are not going to work. But you have to contend with two things here, first thing being that the LLM is only as good as the questions which it has been taught to ask, right, or the prompts it has been fed, or the framework it has been given, or the framework it has been given to perform an agentic analysis of the document. That is definitely sort of the first idea or caveat here.

Chetan: 1:13:23

But even before that, Hari, I think you touched on a very important topic. I'll just pause you there, just hold on to the thought. Sure, One of the main things, especially in finance, is to ensure the accuracy of the data. You talked of form recognition, you talked of unstructured data and things like this. Right Even today, if you were to sort of look at, you know, if you fed, let's say, a PDF, you know, to an LLM which containing financial numbers, I think one of the main things is you may not be able to sort of like even read the table properly, correct or the numbers in tables and things like that, which is where your recognition really, I think LLM may be capable, but the form recognition libraries are still not up to the mark as well, Even if you use some of the latest stuff that is out there.

Hari: 1:14:16

Yeah, actually, I'll break that down into two problems. What seems like one problem is actually two problems. The first thing is that you've got this PDF and you need to do simple, let's say, handwriting recognition or text recognition in it and also understand the structure of information being presented. Now that is one problem. I think it is something that is going to be an open-ended problem. Whatever software or technology you give me, I can always create a PDF that is going to trip. It's an open-ended problem. Whatever you know, uh, software technology you can you give me, I can always create a pdf that is going to trip it up. Okay, uh, so there is, uh, there is no end to that problem. Someone who wants to really make your life difficult can, you know, create a fuzzy looking pdf which is not easily tractable to that ocCR or that recognition, right?

Hari: 1:15:10

The second order problem is, let's say, that we have extracted faithfully the contents of the PDF, the semantic understanding, in order to be able to pull out the data that you require. I think that's the next level problem, which I think is firmly in the world of LLMs, which I think is firmly in the world of LLMs. I mean, a very simple way of thinking about this would be does your LLM know that an income statement and a profit and loss statement are the same right, or a statement of operations and an income statement are the same right? Does it have the semantic understanding to be able to figure this out? That's the next level challenge, right. That lies firmly within the world of LLMs, and that is also the failure to do that, plus the idea that, at the end of the day, we are still in just a next token prediction model creates vulnerabilities and hallucinations, right.

Hari: 1:16:07

So, where you could either be pulling out the wrong data or you could be making up data, I think those two problems are firmly in the realm of LLMs and as we get better at both prompting checking, creating a better workflow we will find that a lot of these challenges get eliminated.

Hari: 1:16:30

And obviously, as LLMs get better as well, I think these kind of activities will get sort of better addressed. So, for example, there are now foundational models which are smaller models, which have been specifically built to be used for RAG or like these kind of retrieval, augmented generation problems. I think that will become a special case in itself. There are going to be people who just want to interact with an LLM without any retrieval capabilities. There are going to be people who want to use LLMs solely for retrieval and solely for data processing, and then there's going to be a mix and we may actually end up with a double-barreled approach, where we use one LLM for just data capture and processing and a second one to then sorry data capture and a second one to then process that and think about it and analyze it right.

Hari: 1:17:29

So I think all of these ideas will come about, yeah.

Chetan: 1:17:34

And also, you know RAG is. You know the efficiency of RAG can be a lot better. It is not. I mean, if you use RAG and you've gone through, it's not an issue of the databases that you use, it's more in the question of, I mean, rag models itself can be improved quite a bit as well, in the sense that if you use rag today and if you do not necessarily give the right prompts or are able to sort of like peddle with it, the answers that you get are just nonsense. Right. So it is just. I mean there was or are able to sort of like peddle with it. The answers that you get are just nonsense. So it is just. I mean there's been a lot of debate on how rags are great and things like that, but it's not always. I mean, if you're looking for high precision activities, you know you will need to really amp up the rag to be able to make sense.

Chetan: 1:18:29

I mean, that's been my experience as well for something like this. If you let it into production, you will get a lot of negative reviews.

Hari: 1:18:38

Right, right, right, no, no, 100%, and I think even here it is. I feel like it is a space that is quite prone to abstraction or disruption. So if you think about it in the 90s and probably even the early 2000s, like we would all get these assembled computers right, if you remember, you would never go to a shop and buy one.

Hari: 1:19:00

Yeah, you probably still do right, like where you kind of say, okay, this is the ram I want you know, I want you know so much ssd, this is a chip, uh, this. And you kind of put things together and in some ways like RAG is that. But majority of the world doesn't buy computers that way today. We just buy something off the shelf and the combinations have been put together and abstracted away and there are neat boxes and the choice is between do you want to buy this Mac or this Mac, or this PC or that PC?

Hari: 1:19:34

And I think we will get to a point where the same thing happens with RAG, right. So where you have a performant, you know, let's say, engine or a bot which works very well for one use case, right, and it's like you know that statement right, like we are not going to look under the hood, we are going to take it. It's going to work out of the box and we will just sort of you know, pay per use or license out that particular you know bot and maintaining it, keeping it up to date with the latest and greatest of technology as well as domain capabilities, will completely become the onus will be on the provider, right, I think that's where we will get to, and I'm sure, like Waco is also doing work in this space.

Chetan: 1:20:27

You know, the secret amongst Gen AI startups is it all runs on a rag model computer and since you need assembled computers to be able to really process the models and things like that are pretty high-end servers as well. But if you really want to get things done, you still rely on assembling a PC that you want with the GPU that you want, the CPU that you want with the GPU that you want, with the CPU that you want, and the boards that you want and things like that. So I think this is where but you're absolutely right saying that as things get better and you look at it, you're looking forward to more efficient RAG models, if that's the right term to use right, um, because that, even today, is uh, I mean, you can deploy models, but in production it's a hit or miss and that's not a good place to be in. Sure, especially, uh, if you look at, look at this. But finally, you know we've covered a lot of ground.

Chetan: 1:21:27

Hurry Hari on this. But you, being an entrepreneur, how would you advise, especially the latest gen of entrepreneurs, especially in the fintech space now with the advent of AI? Are there any? During the conversation, you've had a lot of product development insights as well, but is there something more that you'd like to add, hari, and that'd be really valuable, given your experience of building both the older gen models with the latest gen models as well? Right?

Hari: 1:22:03

Yeah, I mean, I think, look, rather than sort of talking specifically about fintech, let me sort of try to share some specific advice for people building in AI. In today's day and age, I think the most important thing you can do is to pick the right problem, and when I say the right problem, I think let me sort of qualify that in a few ways, right. So there are many things which were, you know, being solved with previous generation technology and therefore clients, customers and therefore clients customers, users are familiar with some degree of automation. So if Janiya is able to do an unlock in those spaces, you will find that customer adoption is a bit easier because people are already exposed to automation technology. So that's one idea. So are your users willing to accept an automated solution? If they have some exposure to that already, then I think that's definitely helpful. The second thing which you want to think about is and this is probably more important and I think we touched upon it earlier as well is are you at the right level of abstraction? Right? Every time you have ChatGPT or one of these companies come up with a new model, you'll see these posts saying here are 100 startups that got killed today by this new release level of in the layer of, you know, chips to assembled computers to you know fully fleshed out PCs or Macs, or in the world of you know, where you are sort of working on either you're working on wireless technology or you're working on chips, or you're working on building the phone, or you're working on the App Store. You've got to figure out where you want to be in that sort of spectrum, right, and obviously closer to the consumer is better. But also, if you pick the right layer or level in your space and in your universe or domain, then the evolution of AI will actually benefit you as opposed to eat you up, right. So being able to sort of sit down and suss that out and it's not an easy thing to do right that will really help you survive and make this evolution or Cambrian explosion of LLM tech that's happening below you lift you up as opposed to, you know, be a wave that you know drowns you, right. So I think that is very, very important, right.

Hari: 1:24:59

So I think these are some, you know, I would say things that are nuanced and specific to Gen EI and beyond this, I think there is lots of other generic advice, right. I mean, like finding the right co-founder, working on the right problems, figuring out, if you want to. I mean actually having a strong why, for why do you want to start up, why do you want to work on these problems and why do you want to do product and not services? So a lot of these sort of fundamental questions you have to have answered for yourself. So a lot of these sort of fundamental questions you have to have answered for yourself. But I think, specific to Gen AI, I think these two are quite important, which is a lot of. It is actually change management and customer adoption solving right, because you may have the greatest product, but if your customers have to spend a lot of time understanding it, you're going to have a problem seeing adoption right and, at the same time, if you're not at the right level of abstraction, you could become a feature in someone else's product or one of the foundational models will just wipe you out. So that is something to sort of spend a lot of time thinking about and figuring out.

Hari: 1:26:07

Now, coming back to sort of product development, I would say like look, I'm not a traditional PM. Like, coming back to sort of product development, I would say like look, I'm not a traditional PM, like I didn't sort of cut my teeth being a PM, I sort of was forced to sort of take on that role because as a founder I have to be sort of a product manager. I sort of everything that I've learned is from first principles. Here, I think the, I would say, the most non-intuitive things that I would focus on, which I would say people don't talk enough about, especially in enterprise, is that you really need to know if you're selling to enterprise, whose lunch are you eating? Right, because the way enterprises work is they have a budget and that budget has to be allocated across whatever software spends that they have. You need to really understand where you fit in into their jigsaw puzzle, as opposed to at an abstract level, what problem you're solving.

Chetan: 1:27:08

And I think that is very, very important.

Hari: 1:27:10

You need to figure out. Do you need to be on a marketplace? Do you have to interact with four other softwares? Are there APIs available? How would they think about someone who's building on top of them? All of these things are very important to figure out pretty early in the game.

Hari: 1:27:24

The second thing I think this is again beyond sort of you know, data privacy, security and so on. I think there's this classic analogy which someone gave me a long time ago I keep repeating it right which is like it is that elephant's teeth or hathi ke daan sort of approach, right, which is you need to know what are your product features which look great in a demo and create that wow effect, versus what is it which is a boring aspect of your product but is going to again and again be used every day by your users and therefore need to pay a lot of attention to and get it right, because otherwise it's going to be a nuisance for them, and it will be. It will. It will affect adoption, right? So a very clear idea of what is sort of bau low complexity but high volume usage which you need to really get right, versus what is this wow feature which will probably get used three times a year, but will be amazing, I think, being able to break down the two, figuring out what looks great in a demo or in a sales process versus what users will actually end up using a lot.

Hari: 1:28:33

I think making this kind of differentiation, having a clear idea of what your roadmap will look along both these dimensions, I think that's quite important, especially in an enterprise world where buyer is not the user. Buyer will want what you call wow features to kind of be impressed, sell upward all of those things and for you to appear differentiated, but the user will really use all the very mundane things which kind of get used every day and therefore need to be very perfectly done and need to be focused on. You need to obsess over yeah, so I think we'll leave you with those closing thoughts.

Chetan: 1:29:21

Absolutely. I think you covered such relevant topics as well. One of the other things which, when you were talking, I was uh, reminiscing is, you know, sometimes even users right, I'm not clear how they will use a very specific picture, correct, they're like I want this because this is something else, like this also, I'm like how many times are you actually going to be using this which we know in reality is like you know, nil, right, you know, this is another thing just coming up and it's more like tick the box. But in enterprise situations that are, you know, there are many players and you know, and many players, to be sort of like, kept happy, you know, keep the ball rolling until it finally comes together. But you know, there's so many other topics, hari, where you and I could have spent a lot of time on, and I really appreciate you taking the time to speak to Kodan Kansar today and I hope to catch up with you soon and show you all the latest stuff that we're building out, get your thoughts on this and really appreciate your time.

Hari: 1:30:37

No, completely, and thanks a lot for inviting me over, and this was a really interesting conversation for me as well. Gave me an opportunity to voice a lot of things that have been going on in my head, but I've probably not like expressed, didn't get the opportunity to express in this way. So like really appreciate.

Chetan: 1:30:52

Uh, you know the invitation as well and uh, thanks for having me over yeah, thanks ali and many, and I'll see you next time in bangalore as well. Thanks so much. Okay, thank you all, right bye.