Building Webhooks-As-A-Service In Rust

Drew: Hi Tom, thanks for taking the time to chat today. I’m excited to learn more about Svix. I try to start by building some context. So, let’s start with some background on Svix. What are the problems that Svix solves and how does it solve them?

Tom: What we offer at Svix is “Webhooks-As-A-Service,” and that just means an API for sending webhooks. It’s somewhat similar to Twilio and Sendgrid, but instead of sending SMS or emails, we send webhooks. Customers have come to expect webhooks, which means that many services are looking to send them. It’s, however, a pain to build your own webhooks sending infrastructure, so we just make sending webhooks really easy.

Drew: How did you stumble into this problem?

Tom: At my previous company our customers were begging us to add webhooks. When I started looking into it, I realized it was going to be quite a bit of up-front time to build, and it was going to be annoying to monitor and maintain. On top of that, we had this beautiful API that we were really proud of, and we weren’t going to just add some low quality webhooks to it, which meant even more work. It just felt untenable for us to even approach this with the resource constraints we had. So, we just kept telling our customers no.

Tom: I wish I could tell you I was smart enough then to realize that this was a business, but I wasn’t. It was only several months later when a friend of mine asked a question about webhooks in a Slack community I’m a member of that it clicked. I was replying to her, and she was just completely disinterested in all of the minutiae it takes to get webhooks up and running, all she wanted was to send webhooks. I ended up asking “if I build a product that makes it easy, will you pay for it?”, she said yes and someone else chimed in and said they would pay for it too. I knew there was something there. So, it was really seeing the pain myself and then also realizing that other people were having this pain as well.

Drew: Solving your own problems is textbook entrepreneurship, right?

Tom: Ha, it’s a great way to start, that’s for sure!

Drew: So, I may have put the cart before the horse here a bit. I realized in my research that not many people actually understand webhooks and what they’re used for. Can you just give the elevator pitch?

Tom: Webhooks are essentially API callbacks. I often call them a reverse API. So if an API is someone making a request to a service, a webhook is the service letting that someone know that something happened on the service’s end. The alternative to this is polling an API, like maybe the client checks every few minutes and asks the service if anything happened. So, for example, you might make an API call to Auth0 every few minutes to check if someone signed up. The alternative is for Auth0 to send you a webhook telling you that someone signed up. With polling, the computational load on both the client and the service is really high. You can solve that with webhooks. Though, there are a variety of other reasons why webhooks are better than polling.

Drew: When I first learned about webhooks, it seemed like a simple idea. But, as I dug in, I realized there are a lot of things like retries that make them tricky to implement. Can you explain some of those?

Tom: Sure. Webhooks are dead simple. That’s their super power. It’s just an HTTP call! But, the simplicity of the concept also creates a trap for a lot of engineers who start implementing webhooks without thinking through the details. Almost everyone has seen this same thing happen with authentication. It’s really easy to think that you can just get a username and password and store it in your database. But, it turns out there is a lot more to it. Webhooks are similar. For example, if you provide an API that includes webhooks, you’ll have customers that have a substantial amount of down time. So, the webhooks that you send while their system is down are going to fail. But, in order to trust webhooks, you need to know that they will be delivered. So, you need an effective retry mechanism. Another thing is that as a webhook sender, you need to be able to handle all of the weird HTTP implementations out there. Yet another thing is security. Server side request forgery (SSRF), spoofing, and replay attacks can all happen. The last thing that comes to mind is just all of the supporting infrastructure to do things like notify customers when things go wrong.

Drew: Before we talk more about the company, I’d love to get some background on you. What are some of the things you did before Svix?

Tom: I’ve been coding all my life. I started when I was nine. I wanted to go to a robotics class, and they forced me to learn how to code first. I initially thought it was annoying, because I just wanted to build robots. Eventually, I decided programming was better than robotics could ever be. The feedback loop with software is just insane! It was also an exciting time to program. The internet was still in its infancy (relatively), and most things were still very hackable. So, I got into security. All the cool kids in security were using Linux. So, I started using Linux, and that got me into open source. My career really took off from there. I led a team at the Israeli Intelligence Force, led a team at Samsung, and started a few companies. One connecting thread through all of my career is that I’ve been creating tools for developers the whole time. At this point, I think I understand how developers think and how to give them a good experience. That’s become a key part of how I approach software.

Drew: One of the things that makes you a little bit different is that you’re a “technical” CEO. That seems to be becoming less and less common. Are there any unique challenges associated with that?

Tom: First of all, I’m the solo founder. And, the founder role in a company is really just to pick up anything that needs to be done. So, I had to learn sales, marketing, and other things with no prior knowledge. But, there is something special that helps you succeed in those areas if you’re the founder. For example, even if you’re not the best writer, you can still write very effectively about the company because you know it better than anyone. So, even if I couldn’t sell a random product very well, I can sell this one well. You also have to get good at hiring. I’m always trying to “replace” myself by hiring people that are better than me in every specific role.

Drew: I’d like to hear a little about your tech stack. Obviously we’re very focused on Rust, but what is the whole stack and how does Rust fit in there?

Tom: When we started, everything was in Python. I was already a big Rust fanboy at that point, but I had a lot of code already written in Python, so I knew I could move faster if I just built on that. Also, Rust wasn’t as mature back then (the answer to “Are we web yet?” was still no). So, it was easy to start with Python, and it was also fairly easy to hire for Python. But, at some point, we just decided it was time to switch to Rust. We actually had a competition inside the company to decide if we were going to do Rust or Go. I was on Team Rust. Someone else was on Team Go. We each took a few hours and rewrote the same piece of the product. The Rust turned out better. So, now Rust powers our main API. We have some of the management layer left over in Python, but all of the core stuff is Rust. Our frontend is React with Typescript. For databases, we use Postgres and Redis. And, everything runs on AWS with Terraform.

Drew: What are the key things that Rust has done in terms of enabling your mission?

Tom: When people argue against Rust, they usually complain about how annoying the compiler is for yelling at you all the time. But, I think this is a massive benefit! We encode everything in the type system! For example, when we interact with Redis, the key is typed to the structure of the value, so we get very strong type safety on top of redis. All of this means we have this unbelievable type safety that makes it really easy for us to rewrite things and experiment while knowing that we’ll never go down. Uptime is super important for our product. Also, our cloud spend was through the roof when we were using Python. Rust really helped with that.

Tom: I think something that is a bit of a double-edged sword with Rust is that people who like Rust tend to be perfectionists more than pragmatists. The language makes this hard to resist sometimes because the whole ecosystem provides so much perfection already. For example, a lot of people will work really hard to make sure they’re getting lifetimes just right so that there’s no copying or whatever, but sometimes it’s just more important to get something done than perfect and it’s OK to clone() a bit.

Drew: I’d never heard that before, but that makes sense. The Rust ecosystem really does encourage you to write near perfect code. I can see the temptation.

Tom: I’m not against writing perfect code, but it can be counterproductive in a startup context where execution speed is so important. That caused some initial tension when some of our engineers joined. But, I think now we’re all pretty much aligned on the expected tradeoffs between perfection and speed of execution.

Tom: As an aside, I think it’s important to understand that at a company the purpose of code is to drive business value. This statement shouldn’t be controversial, but it apparently is. I think it’s also somehow worse with Rust, as Rust is so much fun to code in and perfect, so people write code for the sake of writing code. Don’t get me wrong, that includes me too. I love to make my code perfect. But, the code exists to make the business stronger, better, faster, etc. Engineers that understand that usually stand out from the crowd.

Drew: When I was reading up on Svix, I read that you strive to deliver an excellent developer experience. What are some of the things you do to achieve that?

Tom: I’ll try to be brief, but I can talk about this for hours! Let me start with an example that summarizes my thoughts: if you buy a new phone, you basically expect to be able to take it out of the box and start using it, right? For some reason, with developer tools, we say, “They’re developers. They should just read the docs and figure it out.” Sure, developers can do that, but why force them to go through it? We give developers the same seamless onboarding they are used to from consumer product, and it really makes the difference!

Drew: I noticed a lot of the Svix code is open source. What is open source and what is not? And, how did you decide which pieces should be open?

Tom: That’s the billion dollar question! Svix is “open core”, which means that most of the code is open source, but some parts are not. When deciding what to open we consider the following: would it be useful for people, what the maintenance burden for us would be, and how much more complex it makes the open source offering. One of the things we care about when it comes to Svix is making sure the open source project is easy to run, so for example, we simplified some of the parts that require the more complex infrastructure we have in the hosted service.

Drew: You and I talked before about how Svix has a set of company values that you try to maintain and adjust over time. Could you talk about that and maybe highlight some values that you think are unique?

Tom: I used to think company values were just a Silicon Valley cliche, sort of like inspirational posters that you just put on the wall and never think about it again. I’m sure that in a lot of a lot of cases, they are probably just that. But when I started building the team, I realized that I needed to find a way to make everyone understand who we are, what we stand for, and what we value as a company. We use the values every day, both in hiring and in our day-to-day decision making.

Tom: It’s hard to choose favorites amongst the values, but “impact” stands out to me. We just try to make an impact for our customers. Then there are things like “customer obsession” and “urgency” that fall in line with that. Those are my favorites. They describe why we exist, while the other values are maybe more about how we do things.

Drew: I’m really fascinated by the idea of maintaining and changing the values over time almost like an open-source project. Do you guys have a formal process for that or is it organic?

Tom: Yeah, it just happens naturally. When we onboard people, we always tell them that one of our values is learning and improving, and that extends to the values themselves. So if they see something they think should be changed, they should bring it up. We want to make sure that our values are not just empty platitudes, and evolving them, as we evolve, is the only way to achieve that.

Drew: Well thanks again for chatting Tom.

Tom: You’re welcome!