Shay Banon is the Founder and CTO of Elastic (NYSE: ESTC) the open source leader platform that enables enterprise search, observability, and cybersecurity. On today’s episode, Jon Sakoda speaks with Shay on his path to starting Elastic, and his success building a commercial open source company in the era of cloud providers.
Shay Banon is the Founder and CTO of Elastic (NYSE: ESTC) the open source leader platform that enables enterprise search, observability, and cybersecurity. On today’s episode, Jon Sakoda speaks with Shay on his path to starting Elastic, and his success building a commercial open source company in the era of cloud providers:
SHAY BANON: The past is fake. The future is real. I know where I want to go. I’m excited about it. You know, it’s like you can find yourself thinking about the past or decisions, I should have gone with X or Y or Z. You can’t change that. The future is real. Focus on the future. Focus on where you want to go, imagine it. And to me, that’s the exciting part.
JON SAKODA: Welcome to the Decibel Podcast. I’m excited to welcome my friend Shay Banon, the founder of Elastic, to the show. Shay is one of the true pioneers in open source, and his company is at the center of enterprise data, cybersecurity, and AI. He is one of the forefathers of the search industry and still has a lot of unfinished business in the space, and I’m excited to welcome him to our show. Shay, it’s great to have you.
SHAY BANON: Thanks, Jon. Excited to be here.
JON SAKODA: Shay, you have a really unique founding story. If it’s okay, can we start from the very beginning? Where did you grow up? What was life like in your family? And when did you discover computers?
SHAY BANON: Yeah. I’m from Israel originally, and I grew up in a city called Ashdod, which is about 30 kilometers south of Tel Aviv, right on the Mediterranean beach. So I grew up going to the beach a lot, and surfing, and running around. It’s a relatively new town. It’s an interesting one. It started because Israel wanted to have another port next to Haifa. And my grandfather moved there to build the port, one of the first ones that did it. So it’s kind of like a city that grew up and kind of came into being very quickly, around the early days of Israel. And it’s supposed to be a very well-planned city.
It was a fascinating city to grow up at. Because it was such a new city, we got a lot of waves of immigration, because Israel had a lot of waves of immigration. So I had a chance to grow up with a lot of diversity when it comes to people coming from many different places. My parents came from Morocco and Turkey. But tons of people coming from Europe, and Russia, and Iran, and Iraq, and obviously Morocco, and other places.
So I grew up mostly with my grandparents and my mother. My grandfather worked at the harbor. A lot of appreciation for knowledge. And I’m always fascinated by those generations that just wanted their kids to be very well-educated, but didn’t come from an educated background. So it’s almost like always living in a conflict, if that makes sense. I grew up working at the harbor when I was 13, where my grandfather used to work. Always, “Study hard and make sure you succeed.” But then there was also the, “Well, you have a safe place to work at the harbor if things doesn’t work out for you.” So that was great, a very loving and caring family that just supported a lot of where we wanted to go.
JON SAKODA: I think a lot of people don’t know that they have a safe harbor to return to. But in your case, it sounds like, you literally grew up with one. And I think a lot of technical founders can look back and remember the moment when they discovered computers. They became gamers or hackers earlier in their life. Do you remember when you fell in love with programming?
SHAY BANON: I think so. You ask me when I started to work with computers. And to be honest, I wasn’t really a tech kid or something along those lines, that got a computer when he was seven and then immediately started hacking or something along those lines. I worked pretty hard to save for my first computer because I needed it for my studies in high school, because I studied electronics and computers. But I only studied electronics and computers because that was the most advanced courses that were there. And I wanted to excel. But I didn’t do it because – I didn’t spend my weekends programming or something along those lines. And then when I applied for university, I initially applied for electronics, because that was the main thing I studied when I was in high school. And then in one of the weekend newspapers, I counted the number of jobs in the weekend newspapers. And I still remember it. And computer science had one more job than electronic engineer, so I called university, and I changed my role to computer science. That’s how I started to get into computers.
And I remember arriving at university, and I was typing with two fingers very slowly, and people were typing extremely fast. And I think that this is a lot of what at least I learned when I grew up. You can find beauty in almost anything that you do. So it’s about finding the passion, the beauty in what you do, and then really being curious and understanding how things work, why they work, how you can improve them. And that’s something that I learned and took from.
And I think I bring it to a lot of the things that we do at Elastic. Because when you’re founding a company or starting a company, writing the software is only part of what you do. There’s a lot of other systems and processes that are fascinating to learn how they operate and how you can maybe hack them or understand them and try to make them better.
JON SAKODA: It sounds like you didn’t immediately know you would be a technical founder. And I know the Elastic founding story starts a little later. And we’ll eventually get to the famous recipe app that you wanted to write for your wife, which eventually became the inspiration for Elastic. But before we get there, tell us a little bit about the early years of your career leading up to your first open source projects.
SHAY BANON: When you live in Israel, you tend to go and work at startups.
JON SAKODA: This was the early 2000s, right?
SHAY BANON: Yeah. 1997. So there was – the dot com was happening. I actually ended up – my first, first job, fresh out of just starting the second semester, I applied for work helping build heads up display for helicopter pilots. So that was fascinating, building real-time systems, programming in PLM and C, and working with the US Army, and the South Korean Army, and the Canadian Army, so that was a lot of experience condensed into a very short amount of time. And then I started to work at many different startups. None of them were really successful.
But I learned a lot about how to build software, which was really interesting. It’s like you ended up just joining a company, typically as a software developer. Even though I would progress to be a team lead or an architect, I always loved going back to be a software developer and really understanding how the company works and what makes it tick, and then just doing whatever is needed to make it successful, trusting in my capabilities to progress.
I think one of the traits in Israel that it really encourages people to go out and be adventurous and take whatever they can, and then give back when they do that.
JON SAKODA: And I think a lot of people know this already, but usually everyone in Israel serves time in the military. Did you serve in the IDF?
SHAY BANON: I did not. I had a – I still have a kidney disease that it is pretty dangerous now. Thankfully nothing goes on, and doctors are very surprised that nothing happened as a result of it. But when I was 13, they identified it. And then the IDF did not enlist me.
JON SAKODA: Do you still live with it today?
SHAY BANON: It’s supposed to never leave you. It’s like it’s a chronic disease. So conceptually, by this time, I should not have any kidney left and be in dialysis. But apparently, it’s not.
JON SAKODA: I’m glad that you can laugh about it now. And I think the first time I heard this, I was in shock. Obviously this must have been a pivotal moment growing up. And I’ve always wanted to ask how it’s shaped you in some way. Is it part of what makes you unique as a founder?
SHAY BANON: I think that when you’re 13 and you’re getting this type of news, it’s kind of like shocking, when you get news that you have this chronic disease, and the doctor tells you that you might lose your kidneys and be in dialysis. And you have this shock, I guess, about how to deal with it, and what does it mean. And I was taking some drugs at the time that made me really bloat up. If you look at me when I was 13 or 14, you go like, “This is not a person that we really recognize.” And I don’t know. I think you end up just building this mental tenacity, I guess, of just trying to work through it. You have this big thing in your head, and you learn to live with it. And I think I’ve just learned to work with big stuff that are just happening around you and just work through them, if that makes sense.
JON SAKODA: Yeah. Do you feel like in some ways, you have had no choice but to learn to be fearless? Like live every day, live the moment?
SHAY BANON: Yeah.
JON SAKODA: Nothing really to be fearful about in everyday life because there is so much uncertainty that you’ve been exposed to, above and beyond what most people see in their own day-to-day.
SHAY BANON: I think I’ve learned to accept fear and live with it. I don’t know if you really can ignore fear, if that makes sense. You just need to accept it and generalize it, compartmentalize it to a degree, and then just learn to operate as it happens. And there’s a lot of things that you’re going to be afraid in your life. And I’ve had many, many moments like that in my life as well. And they still frighten me. But you just learn to work with them, if that makes sense. And that’s what I try to teach my kids as well. It’s like, it’s not about not fearing something. It’s about going through the process, even though you fear something along the process.
JON SAKODA: I know this is not something you share regularly with people, so I appreciate you sharing it now with our audience. I’m sure a lot of founders can appreciate that mental tenacity for all of us comes from somewhere deep within, and we don’t always want to talk about where that comes from. So I appreciate you doing it here.
Let’s transition to the fun part of the story. Tell us the Elastic founding story. And if you don’t mind, can we start with the famous recipe app that you wanted to build for your wife?
SHAY BANON: The first open source project is an interesting story. When I got married to my wife in 2004 or ‘05, she wanted to change her career. And she did a first degree in logistics. She was doing her second degree. And she really loved cooking. And I told her, “Just change your career. Go be a chef.” So she started to work in a restaurant in Tel Aviv, completely changing her career. And me believing that in order to excel, you have to go and study in the best places that you can reach, if possible, and then go through there, we ended up moving to London as a young couple. And she went to study at the Cordon Bleu. And I moved to London without a job, without anything, just trusting that we’ll be able to go and find one.
And for a month, I really tried to figure out how do I make myself interesting to the London job market. So I had my first shock, which was, when you’re in Israel, it’s all about startups, and you go and have these very easygoing interviews with very deep technical questions and things along those lines. And then in London, it was all finance, especially at the time. And I would go to these interviews, and I had to wear a tie and know about scrum or whatever, why waterfall is the agile processes, not only about software. But also about software. And I was hacking quite a lot about C and C++. But Java was becoming more popular, and there was a lot of open source tools around. So the best way that I think you can learn is by trying to build something. So I tried to build something.
And then I said, “Okay, I’ll go and try to build a recipe app for my wife. As she’s accumulating knowledge in her schools and everything, then she’ll go and capture it in this app.” I wrote probably one of the worst designed apps possible, because I shoved every piece of technology that I can into it just to learn it. But I wanted the experience to start with a search box. Google and search and everything, it just felt so simple. And I wanted to explore that simplicity in that context.
So I try to go and figure out how to build search in an app. And I learned about Apache Lucene. And I can talk for ages about Apache Lucene and the amazing community there. But I started to try to use it. And like any good software developer, I realized that it was not easy to use it in a Java application, so I created an abstraction on top of it. Then at the time, at least, you could take business objects and make them to the relational databases. It was like taking business objects and making them to a search engine. So I wrote this layer and this abstraction. And I thought, okay, well, what can I do with it?
And I said, why not open source it? I’ve learned how to write this code and how to use it with other open source projects like Hypernet and others, like in the Java ecosystem. And ended up open sourcing it, not knowing anything about open source. I had no idea. I had these foolish thoughts that I open source it, and if it’s successful, I’ll get like a thousand developers. But it ended up being very – not exactly the same. But that was my first foray into open source.
JON SAKODA: Do you remember the moment when you knew it was going to take off?
SHAY BANON: No. I don’t know if there was a moment. It was really very progressive. I open sourced it, and people started to use it. I didn’t even know when would something be successful, or how does it work. I created the RC channels, the forums, and things along those lines. People started to use it. I got invited for the first time to the ServerSide conference, and I gave a talk. And that was the last talk on the last day, and I was so happy and excited. And people were using it. And I got a chance to talk to some of the people that I really respected in the open source community. It ended up moving to OpenSymphony, where the Atlassian team was heavily involved. And it was like slowly becoming more and more useful and popular.
And to be honest, the biggest part there was, first of all, it was a lot of work, that project. It was like, I ended up writing most of the code most of the time. A lot of work that goes beyond just writing the code. Helping in the forums. You know, I was almost like obsessed. If someone would ask a question, I would just have to figure out how to help them and how quickly. Because someone was stuck, and I can’t have it on my conscience, if that makes sense.
JON SAKODA: Did you have a lot of committers in the early years?
SHAY BANON: No. I ended up working with a lot of people in terms of how they ended up contributing. But most open source projects, to be honest, having one or two developers working on them. And that was the case in this open source project, just me working. And there wasn’t a company behind it. There wasn’t a foundation behind it. Even when it was part of OpenSymphony, it was just still just a one-person project. But it was similar – because it was a library. So it was similar to a lot of other open source projects at the time. You have one developer helping out, and a lot of amazing, amazing people doing an incredible job. But whenever someone contributed code, I was all giddy, and I was like, let me go and try to figure out how to help you.
JON SAKODA: So Compass, your first open source search project, was wildly successful. But I know you eventually open sourced Elastic years later, so you must have had some unfinished business. Tell us that story.
SHAY BANON: Yes. I think the biggest part about Compass, the first part was just around learning about open source. The second part was around falling in love with search. Because I started to see people implementing Compass in many different applications and exposing searching to the users. You could see people almost like liberating their application through our search box, right? It’s like, I worked as an architect in various banks in the UK at the time, in London. And my trick was going to various teams and telling them, “Give me a couple of hours and I can add a search box to your application.”
And then the next day, a trader would suddenly load their application, and they wanted a search box, and go like, “I can’t believe that now I can explore my data, and I can understand what’s going on. I don’t need to browse through it,” if that makes sense. “I can just search and find trades, confirmations, back end systems, whatever.” And I just fell in love with the experience that it has. It has a very short distance between a search box and the user. The technology, it almost immediately gets exposed to the user. You search something, and you know that the search was slow. You know that the search was not relevant to you. Search was bad. You talk about the technology in emotional terms or in qualitative and quantitative terms. This is amazing. A technology that gets exposed early to the users is fascinating, and you work very hard to make it successful. And I just fell in love with that experience.
And yeah, it took about a year, I think, to gather the courage to write a new open source system. Because the first time, I had no idea what I was walking into. The second time, with a kid, with a young daughter, I was like, is this going to be again where I spend my weekends and nights, and second job and third job at the same time? But I felt very, very passionate about it, and I thought that it was something that people really wanted to use.
JON SAKODA: Well, I was going to say, it sounds like this time around, you knew it was going to be successful. You knew what you were signing up for.
SHAY BANON: I had a very strong conviction that this is going to be successful, because I knew that search was useful. I proved it through Compass, the library. I knew that the quote-unquote “the market,” the user base, the community, wanted something like that, because they were asking for it all the time. I saw the success of NoSQL systems, and I thought that none of them really focused about getting data out, right? It’s like all of these NoSQL systems were like, “We store it in calendar format” or things like that. And none of them was like, how can we make it easy for people to just put data in and search it and find it easily?
So I made a point of not calling Elasticsearch a NoSQL system, even though people really wanted to call it a NoSQL system, because I said search is useful regardless of where you store the data. If you put it in HDFS, put it in Elasticsearch. If you put it in MongoDB, put it in Elasticsearch. It doesn’t really matter, in a relational database, or you can just put the data in Elasticsearch itself as well. And that’s perfectly fine as well. I wasn’t trying to pick fights, if that makes sense. I think a big part of the Elasticsearch community that I wanted to create was a welcoming one.
JON SAKODA: Shay, can we go back in time? Because I think a lot of people don’t remember the state of enterprise search back in 2010. At this point, Google’s gone public. Everybody understands the power of search. But remind everybody, what was the state of enterprise search? Why was it so hard to get that magical search box in applications and on company data? Why didn’t Google just take the market?
SHAY BANON: I think it’s a great question. I had a chance to experience a different type of search. Because Compass was a library that people integrated into their applications. So I got a chance to see legal applications, and recipe applications, and finance applications. People just put Compass into any type of applications. And it was very obvious that search applies to any type of company’s data. And at the time, at least, enterprise search was mostly focused around making, I don’t know, SharePoint and Documentum. And it’s like, expose the company data to users, right, in your organization. It was less around just take any type of data and treat it like a data store, like a search engine.
And my point was, Elasticsearch was going to feel like an amazing document-based data storage around JSON, and document-based data storage, that starts with search as a technology or as an experience. And then you would inspire users to put any type of data. I don’t think that that was a very common message to say for search engines back there. And when it comes to Google, it’s a good question.
When I told my mother that I’m going to build a search engine or start a company around it, or things
like that, she was like, “But Google is already out there. What’s going on?” But Google, I think there’s a big difference between building a search engine that specializes in the web and serving ads and things along those lines, towards building a generic search engine for any type of JSON-based data. And Google didn’t have a product that said, okay, you can just go and take data and put it in any format and search it. They had the Google Appliance that focused on, guess what, enterprise search, where you would plug it into your organization and start to be able to search your documents and things like that. So nobody, I think, really treated search as just a generic way to interact with any type of data.
JON SAKODA: And I don’t know if you would give yourself credit for this, but I do believe after about 10 years of trying to sell the Google Search Appliance, they eventually end of lifed it and gave up. So I would say that in the beginning, maybe it didn’t look like it was going to be as easy taking on Google. But with the fullness of time, they ended up packing up and sunsetting that Appliance.
SHAY BANON: Yeah. And by the way, the system that they recommended to transition to was an enterprise search product that was built on top of Elasticsearch. That was a wonderful way to, I guess, close the loop there.
JON SAKODA: Yes. You deserved to declare victory on that chapter of the book. And it is one of my favorites. The entire Elastic story obviously has many, many chapters. Would you mind if we focused on some of the earliest ones? Can you walk us through how you pulled together all the pieces of the puzzle in the very beginning?
SHAY BANON: Sure. So when I started to develop Elasticsearch itself, I knew that I was going to open source it, and I cared a lot about bottom up adoption. And tons of it is around wonderful user experience. So I’m trying to study, as much as possible, other systems, and read as much as possible, not only understand how they work, but how do people feel about them? Making it extremely appealing to developers. And I think there’s an art to it. They love ease of use, they love things that are progressively hackable. Start simple, but they can go deep. So those types of things. And that’s how I tried to develop Elasticsearch.
When it comes to open source, I had more experience around it, so obviously I open sourced it. I don’t know, maybe I had a good hunch, or I had a good nose to smell successful things around. I think Elasticsearch was one of the first projects on GitHub. So the whole success of GitHub, Elasticsearch was carried by it, and the interface, and Git, and things along those lines. A lot of the experience and everything happened on GitHub, on mailing lists, on IRC. I was very available. I wrote a lot of code. And the time, 50% or 60% of the time was coding. A lot of the other time was – people maybe don’t appreciate it, but answering in the beginning, a few mails a day. But then it was like tens of emails and, questions every day, and questions on IRC. And it was a lot. And helping people, and they have bugs, and they fail, and they try to deploy it, and it doesn’t work.
And the cloud was happening, so making sure that Elasticsearch was being used in AWS easily. And I think we innovated on things like AWS Discovery, and making sure that Elasticsearch discovers itself using AWS APIs, and things along those lines. There was a lot of freshness to it that people feel like it’s interesting, it’s innovating, it’s fun, and it’s moving extremely fast, and it takes our input into account. I think those are characteristics that developers love, and those are things that we try to carry with us up until this point and into the future.
But I think at the end of the day, there’s almost a core aspect to why Elasticsearch was successful, which is search. If you have a good search system. People need search. It’s extremely liberative. If there’s a good search system around, people will end up using it. And then they’ll start to be very creative around the type of data, what type of analytics you can do with it. Because search is not only about top 10 results. It’s about analytics that you do as you search through data. So a lot of the early community members were part of the content creation around all of these type of use cases that people took and ran with it, which was amazing.
JON SAKODA: And now that you mention analytics, this is a nice transition. You started out with a vision for creating search in a recipe app. And then ultimately, you moved into analytics. Tell us about Logstash, Kibana, and the other major moves that you made to create the ELK stack.
SHAY BANON: So, in 2010, I open sourced Elasticsearch. And I already quit my job a few months before, I think. And a few months later, I decided to dedicate the next couple of years to Elasticsearch, because I had a strong conviction that it’s something that is going to be useful, and potentially something that I can maybe a start a business around it in the future.
And the ecosystem around it was amazing. And people started to use it. In the beginning, it was mostly very narrowed around why is it better than Apache Solar? How can I implement enterprise search, which is perfectly fine. But I was waiting for all of these other uses cases to come along. And then I remember Jordan that developed Logstash, pinging the Elasticsearch Slack channel and saying, “Hey, I’m trying to store logs in Elasticsearch, and I’m not terribly sure how to use the API.” And I started to help Jordan on IRC. And I joined the Logstash IRC channel as well. And I started to read up about logs. And I realized that logs is an amazing market. It’s huge.
First of all, it’s amazing for search, because it’s a lot of unstructured data. We developers get very creative in our log messages, in our errors, including log messages like, “How the hell do – it’s not supposed to get here. If it gets here, then something really bad happened, and things,” what have you. And you want to be able to search them. So that was an amazing market. Almost no players in the open source market around storing logs. So it felt like a market that is ripe for open source disruption.
And I started to really invest in it, because I wanted Elasticsearch to be the place that you store logs. Because Jordan created Logstash. And Logstash is a wonderful tool that takes data, collects logs, collects information, processes it. But it can store it everywhere. It can store it in HDFS, it can store it in Cassandra, it can store it in MongoDB, and it can store it in Elasticsearch. So it felt like a year of Hunger Games, of let’s make sure that the Elasticsearch is the best place for Jordan to store logs or for users of Logstash to store logs, in Elasticsearch.
And somewhere in Phoenix, in a newspaper back end system, in a newspaper in Phoenix, Rashid was standing there, and he was tasked with creating this monitoring system. And he was using Logstash and Elasticsearch. And he hated the Logstash UI, so he created Kibana, just to build a UI around logs that are stored in Elasticsearch. And that became, over time, the ELK stack and Elasticsearch Logstash in Kibana. But it all like totally bottom up, not very enterprise-y, if that makes sense.
JON SAKODA: I think this point is so important. You had a community of use cases that had evolved bottoms up and happened to lead you into enterprise use cases like log management, analytics, cybersecurity, and ultimately now, AI, which we’ll get to in a second. While we’re on the topic of the enterprise, I do have to ask you, because I can’t think of anyone better than you to ask this question, what were your choices for monetization at Elastic? And looking back, was there anything you might have done differently?
SHAY BANON: It’s a great question. I think the best open source projects are ones that keep the open source spirit and foundation alive. But you have to have something around open source. Even foundations have companies that contribute to the open source code. Elastic, the company’s biggest contributor to Apache Lucene. But can you guess which other open source projects were a big contributor too? Apache Solar. Because for a very long time, Apache Lucene and Apache Solar were together, merged together. And every time we would implement something in Apache Lucene, we would go and change Apache Solar to make it better.
So it’s not a zero sum game, if you will, when it comes to community and contributions and things along those lines. But you also want to have a good business model around it. It was very obvious to me, at least, that we’re not going to sell services. Selling services around an open source project is a really bad business model, for many different reasons. But one of them is like, it has this almost inherent contradiction, which is, in order to sell services, you need to have a software that is complicated. A software that is complicated is not loved by developers. So, if you have an opportunity to go and simplify the product, you suddenly have this wrong, almost incentives within a services company that says, well, if you simplify the product, we won’t be able to sell services for that product.
So we wanted to build commercial features, enterprise commercial features, around the project. And by the way, from the early days, we wanted to also provide a cloud product. So when we raised money, it was around both commercial features that we’ll end up developing into the product, but also end up building a cloud product. And then there’s the delicacy of which products, which capabilities are you going to make commercial, how do you think about a cloud product? That was all very, very early days. And I think we learned a lot during that time.
JON SAKODA: Can we go back to the famous moment in Elastic’s history when you created a new form of open source license, SSPL, which was specifically designed to defend you against Amazon, I believe? This is now a blueprint for how a lot of open source companies choose to defend their intellectual property. And I’d love for you to tell the whole story again from your perspective. Why did you need to create a new form of open source license?
SHAY BANON: When I created Elasticsearch, I created it under an open source license. And by the way, it’s funny, when I created Compass, I created it under an LGPL license, because Hypernet was LGPL. And you can get into all the licensing discussions back then, and it was very heated discussions. And someone said, “Well, why don’t you change it to Apache license?” And I changed it to Apache license, like Apache Lucene. It was fine.
Then when I created Elasticsearch, I created it under Apache license. And we wanted to go and create a cloud service in Elastic, and we launched our cloud service relatively early. But we were always thinking, by the way, even back then about what would happen if Amazon ends up taking Elasticsearch and providing it as a service, because obviously Amazon was very successful. And we were concerned about it, because Amazon was not known to be very helpful to their community or their ecosystem.
JON SAKODA: Partner-friendly.
SHAY BANON: Yeah, exactly. Exactly. You would read all the stories about diapers.com, people that end up selling on Amazon website and then changing the thing on them and whatever. And it was very obvious to me, at least, that the same mindset was existing within AWS. So it’s like very ruthless, if you will, and very zero sum game. It’s like, we’re going to take an open source project if it’s successful and make it work.
People don’t know it, but actually, Amazon tried to get into search before that. They built a product called CloudSearch. And it was built on top of their A9 acquisition, and that sucked. It was really bad. And Elasticsearch was successful. And then they tried to rebuild it on top of Apache Solar, and that also sucked. And then I think Elasticsearch service that they launched was I think one of the really first services that they launched taking an open source product and hosting it. Obviously, it was very scary as a small company. I think we were like two years old or three years old as a small startup. But yeah, it’s like they took Elasticsearch as it is. They call it Amazon Elasticsearch Service. And that sucked, because at least to my mind, it’s a trademark violation. And I took a loan to register that trademark when I was on my own.
And things that you felt, at least back then, that were established as a way to protect open source were being ignored by Amazon. It felt like Amazon kind of got into this enterprise software market very bluntly with AWS very quickly. And in the beginning, a nice way to look at it is that they didn’t know better. They would just take a service and call it Amazon Elastic Service. But one of the ways that people really appreciated about open source was around trademark and IP. And it was like a thing. That’s the way that you protect your code. You take a trademark. That’s why I initially trademarked Elasticsearch. But that was completely ignored by Amazon.
There was a lot of confusion in the market around whether Amazon works with us or not. I remember, again on that, there was some surprises to me in terms of the mindset. Wernon Vogels, in one of the first re:Invents, tweeting that the Amazon Elasticsearch Service is out, and we’re happy to work together with Elastic the company and try to make it better. And it was completely false because they didn’t work with us at all around it. And that mindset and that way of competition persisted for many years afterwards.
It was a lot in terms of trying to figure out how do we go and compete with big Amazons. And nobody wants to be the Diapers.com of open source world. And I think we’re – eventually, we’re successful. But we sent out on a path to compete with Amazon that was a multiyear path. And I think now, we’re in a very different place, and Amazon is a partner of ours. But there were quite a few tricky years over our history.
JON SAKODA: And I know you haven’t declared victory just yet. But at a minimum, I think we can say that you paved a path for other open source companies to navigate their relationships with cloud providers. And also, in some way, do you feel you’ve been a part of a wave of companies that has changed the way Amazon views open source?
SHAY BANON: First of all, yes. I think Amazon matured, and they understood that some of the things that they did in the early years when they moved fast and ignored trademark, ignored IP, were very liberal with the truth when it comes to whether they partner with companies or not, or those along those lines – now they’re much more responsible. I think they also understand as they get bigger and really understand what it means to operate in the enterprise software market is that they have to work with other companies in order to be able to be successful. So it’s not something that they can go and operate at this level.
I think they brought a lot of the practices that they had in the retail business into enterprise software. And now they realize that some of them don’t work. I think Elastic really helped when it comes to the education, the evolution of the cloud business, the understanding about how it works. And I was very proud to play a part in that evolution and try to make sure that these open source projects and companies and foundations that all exist around it end up finding a very successful route through cloud partners to be able to be successful together.
JON SAKODA: I do think I need to say thank you. This story is amazing. And you have given the open source community yet another commercial path. And without this work, I think licensing in this space would be a lot harder.
SHAY BANON: Yeah. And I can talk about licenses as well. It’s very interesting around licenses. People ask me about open source licenses a lot, for example, startups. And I think people fear things too early, if that makes sense. So people say, “Oh, but yeah, Amazon or X might take our software and provide it as a service, and then we’ll lose.” But you need to start with focusing on making sure that your software is actually useful and people adopt it. And try to go and progress and invest as much as possible in making it useful to others.
It pains me that we cannot call Elasticsearch open source today. But still, we have our community, and obviously, we start from a very strong foundation, and our community’s following with us. But on a personal level, I’d love to be able to call Elasticsearch open source. I think a lot of the way that we operate – the license is very permissive, that we created. The way that we operate is 100% the way that we’ve been operating in the past. But if you are a young startup, go and use AGPL or another open source license, and focus on adoption. And hopefully, us and other similar companies like us have paved the way for you to be able to compete better or work better with cloud providers compared to how it was in the Wild West, if you will, when we were just all getting started.
JON SAKODA: Well, speaking of the Wild Wild West, let’s talk about Elastic and the GenAI movement. In many ways, you’ve been building up to this moment. I personally think Elastic is one of the great untold stories in the next wave of large language models. Would you mind taking everyone through your vision for Elastic and your future in enterprise AI?
SHAY BANON: I think search has always been involved with machine learning and AI for years, from natural language understanding and analysis to algorithms that you end up implementing in one form or another. And we just announced the release of our own model that we’ve trained called Elsa. And that’s an exciting one, because it’s not an LLM. Obviously it’s not a large language model. But it is a large language model that you can use within Elastic and have semantic search that is very, very impressive. You should see demos of it.
On a romantic level, it was interesting. I love search. And we had this amazing use case in the beginning in the early days of Elasticsearch. But since then, we’ve focused also in observability, and logging, and security, and finding threats. But it feels like search is back. Like the search box is back, the textual interfaces, the workflows around it, the excitement about search, the excitement about discovering data through search interfaces, through chat interfaces. The power of search is kind of back, and people realizing how exciting it is.
For example, what are my health benefits? Because I need to use them to X, Y, and Z. My son hurt his elbow. And if I’m an employee asking that question, the LLM has no idea how to answer it. You need to be able to go and fetch the user’s information, feed it into the LLM, understand where they live, in a distributed company like ours, for example, and all of that. So that level of interactiveness and information is going to be driven by search systems. And I love it, because this is why I started the Elasticsearch. And it makes me very, very excited about the future of Elastic.
JON SAKODA: I know that this entire journey started with you imagining a search box for your wife’s recipe app. Is she going to get a chatbot soon, a conversational chatbot for recipes?
SHAY BANON: So, the funny thing is that I’ve never finished, obviously, writing that recipe app. But I think the good thing is that I empowered all the various recipe apps and systems that they use, the New York Times Cooking app. I can say all of those search boxes are powered by Elasticsearch, which is very humbling. I mean, if you see a search box somewhere, there’s a good chance that it’s being powered by Elasticsearch, which is amazing.
JON SAKODA: One of the great powers that come from LLMs is that they’ve been trained on open source code.
SHAY BANON: Mm.
JON SAKODA: It is now able to read code that we’ve contributed to the world, and now it is able to rewrite code based on what it has been trained on. As one of the godfathers of open source, how do you feel about this new relationship that we have with large language models, where we have trained them on our code, and they are now able to take our code and in effect re-author it in a new way?
SHAY BANON: Putting aside for a second the whole licensing, GPL, virality, or whatever, things along those lines, I learned how to code better thanks to open source, because I could look at the code of Apache Lucene, of Hypernet, of Spring, Framework, of many others. I got to be a better developer thanks to all the code around me that I got a chance to learn, see how better developers than myself end up writing software, and trying to imitate them as best as possible. If we think that we’re going to use – as humanity, we’re going to use LLMs as a way to make us better, faster, more productive, more innovative, than they should have access to the code. There’s no doubt in my mind. And it’s totally fine also that they write some some of it. Elasticsearch was a lot of boilerplate code, I can tell you that. Code that I would have loved not to write so I could help more of the community when I was answering questions.
But at least on my end, I’m sidestepping the more maybe philosophical question around what’s the role of an LLM in society.
JON SAKODA: I do really respect your opinion on this, so thank you for weighing in on it. If I could just ask you a couple of more questions that I think our founders will care about. You have acquired a lot of great startups and have made them successful. Tell us a little bit about how you think about the kinds of companies you want to acquire, and what’s been your recipe for success?
SHAY BANON: Our acquisition history was focused on our platform, if you think about Elastic and Elasticsearch as a platform. So our acquisition strategy was around making our platform better, making it more approachable, and helping us get into areas that we found that were very important for us strategically. The nice thing is that thanks to open source, maybe people under-appreciate it, companies were already integrating with Elasticsearch, were already using it. And that was an amazing thing, because you could already see the integration happening. These people already loved Elasticsearch. So when they joined Elastic, they joined a place that we all felt like we knew each other, because we worked so much together already.
And they were already working in markets that we knew that we could go and innovate in, right? Like when we got into logs, it was very obvious to us that we were getting to traces, because people started to put tracing data in Elasticsearch. So we went and joined forces with OPIT around APM and trace data. Same thing with security. We were being used for significant security use cases, storing just security threats and searching through them across the world. So we knew that we wanted to get into security. So that was the exciting parts.
Making them successful afterwards, we had the foundational aspects that I mentioned, but it’s a lot of work. A lot of the work happens afterwards, because you want to rewrite potentially the software a bit. You want to integrate it into your distribution. We focused a lot on rewriting the software almost always to integrate it into our distribution, versus letting it run on the side. Those are typically the two type of acquisitions, because we had a platform play. And we spent a lot of time with the team, a lot of time offsite and meetings and what have you to make sure that they feel welcome.
I also had this always sense, it’s funny, but whenever we joined forces with a small company, I kept in the back of my mind, are we good enough for them? That was my mindset. Are we good enough? Are they going to be happy? Versus, I think one of the reasons why acquisitions fail is because people say, “Oh, they’re joining this large, successful. They always should be very appreciative of the fact and thankful and what have you.” And my mindset was on the other extreme. Is Elastic good enough to have them join us? And I think that that’s an important one.
JON SAKODA: If I flip the question around, so you’ve been approached for acquisition numerous times, chose to go public and become a standalone company. I think some founders have this mindset of, there’s just no way I’m ever going to sell. And some founders maybe along the way have actually entertained the idea. What’s been your history with acquisitions, and how have you thought about it?
SHAY BANON: I think we were growing very well for years. And we never felt like we needed to entertain acquisitions because we felt like we were doing something right. It’s like, we were building products, users were loving it, building our community, and growing our revenues. So those type of things, obviously they required a lot of work. But it didn’t feel like we need to go and change the way that we think, or find a second act, or something along those lines.
So I’ll be honest, that made it very easy for us to go and say, “Hey, we want to go and try to figure out, where can we take it? Maybe it’s going to be nice to join another company. But we feel like there is something special here, and we want to see how far we can take it on our own, because it’s a lot of fun to go and try to do that.” There’s other companies in the world, right, where maybe they’re not growing as fast. Maybe they hit a ceiling. Maybe the market is not growing. There’s many different reasons why companies stall while still having great people or still having great products. And in that case, I think acquisitions totally make sense. We just – we were never in that stage to think about it.
JON SAKODA: Any words of wisdom or lessons learned for your younger self? Something that if you could go to the younger version of yourself, maybe at Compass, maybe in the early days of Elastic, is there something you would want to say?
SHAY BANON: I really don’t think like that, you know? It’s like, it’s fascinating. There’s a show called Succession. And obviously, the people in Succession are not the best people in the world. But I still remember a line people asked Logan Roy, which is a very well-known titan of the news industry personality, very big personality. They asked him, “Have you ever thought about where you came from and everything along those lines, and how you built?” And he says something very interesting. And obviously, coming from the news industry, it’s fascinating. “The past is fake. The future is real.”
And I was like, the past is the past, you know? It’s like, we made decisions with the knowledge that we had. And I want to say we look back and say, “Oh, I wish I would have made that decision,” or whatever. The future feels very real to me. I know where I want to go. I’m excited about it. And it’s like, you can find yourself thinking about the past or decisions, I should have gone with X, or Y, or Z. You can’t change that. The future is real. Focus on the future. Focus on where you want to go. Imagine it, and then just start to plan towards getting to that area. And that’s – to me, that’s the exciting part.
JON SAKODA: Shay, you’ve been amazing. Thank you so much for coming on the show.
SHAY BANON: Yeah. Thank you. Thank you.