Getting A Read On Rust With Trainer, Consultant, and Author Herbert Wolverson

Drew: I did research on Ardan Labs before the interview, but as is often the case with consulting, it’s hard to get my arms around everything you do. Can you explain what Ardan does and what you do there?

Herbert: Ardan Labs started years ago as a company that taught the Go language. Bill Kennedy, who’s kind of a big deal in the Go world, has been there since the first release and has been shouting very loudly about it ever since. He founded the company to spread some of his knowledge. The teaching side turned into consulting, and we now have a mostly Latin-American consultant board. A lot of the client base is down there, with some in North America and a little in Europe. Bill goes to every Go conference that exists and gives workshops. I’m on the training side; I go to most of the rest, but not all, because I have a family and I like seeing them. Bill has a family too, but they’re more grown up.

Herbert: So this turned into corporate training where companies hire us to give them week-long boot camps or periodic classes, and that expanded into consulting. Typical training is “I’ve got a team that wants to learn Go,” or “I’ve got a team that wants to learn Rust—start at the beginning,” or “We’re stuck on something really specific; go as deep as we need.” The consulting side is the classic “We have a problem, we need something, so we'll pay for it to get done.” For example, I did an open-source C# port of our wrapper around LensDB (a big vector database) for a customer. There's less on the Rust consulting side as it's still ramping up; Go is busy, and the team Slack is full of people I don’t know.

Drew: If Bill started the business around Go, how did the Rust thing emerge?

Herbert: From my point of view, it was out of the blue. I got an email from Miki Tebeka, another author at PragProg, saying, “Would you like to earn some more money? Have a chat with Miguel next week.” He gave me nothing more to run on. I talked with Miguel and he told me, “We’ve had companies asking about Rust, and we hear you’re Mr. Rust, so let’s try it.” We did some YouTube Live test training. They were terrifying, but apparently I passed because they brought me on board.

Herbert: Two or three years ago there was a lot of buzz and everyone thought Rust would take over everywhere. That actually slowed down a bit because people have realized Go is still great for web services, and Rust is great when you need to optimize the living daylights out of something without using C++. Our focus on the Rust side has changed quite a bit. We've trained so many companies that have a C or C++ code base and are porting that. Lately we have a lot of companies that have a big Python code base and at least part of it is too slow because you can quite easily get 30 to 40x speed increases by porting native Python to native Rust just because of how the languages work. So Rust came about in a pretty organic way. People asked for it, they found me, and then it did well enough that I was kept around.

Drew: That Rust and Python thing seems to be a huge trend from what I can see. I think Python is growing quite a bit, probably driven by the AI stuff, and then there are all these occasions where you need something faster and Rust is just sort of a natural, modern choice.

Herbert: Python seems like it's a natural fit for the math and science types right now. It used to be that scientists would write Fortran and then somebody would have to turn it into something maintainable. Science code and enterprise code don't look anything alike. The science code looks like a math paper. The enterprise code is usually designed so you can maintain it a month later. Sometimes I'm helping the scientists write “science-y” Rust and then trying to teach them to write something maintainable but occasionally it's something they just run once.

Drew: I guess you already explained how you got your job at Ardan, but how did you get into Rust and end up as “the Rust guy”?

Herbert: I’m old. I wrote BASIC on 8-bit micros in the ’80s, then Pascal, C, C++, and a whole bunch of other languages. Around Rust 1.0 I’d just finished a game project that went badly and wanted something new. I was totally fed up and wanted to write another language for a while. I asked around and the cool kids were talking about Rust, so I tried it and liked it- maybe more than is good for me. I started blogging the journey. It begins with awful Rust and 84 chapters later, you have a full featured game that runs in a browser or console. I wrote an OpenGL backend to do the rogue-like console that I wanted and just kept writing Rust. That became my first book, Hands-on Rust, which did amazingly well. It's been on lots of bestseller lists, which was something I never even dreamed of. That turned into another book, Rust Brain Teasers and Advanced Hands-on Rust, which is in its final beta. That got my profile up and people started coming to me–which is how Miguel found me through another author at the same publisher. I was honestly having fun. I wrote everything, and the only funding I had for it was my Patreon. I didn't really honestly care that much if people funded me, but it was nice. You can’t say no to free coffee. And that turned into more than free coffee. I never once said, “I’m going to become the Rust guy.”

Drew: You just kept pulling the Rust thread and it kept coming, huh?

Herbert: There's a lot of people out there who are better at really advanced stuff than me, which might be why training goes so well. I tend to take the more pragmatic approach as opposed to some of the absolutely beautiful theory that people write. That’s been my experience with the language. You have the option to write some really intricate, interesting code, but you can also just write straightforward, pragmatic stuff.

Herbert: I heard one of my favorite analogies at Rust Nation UK. If you think of any enterprise stack as being like a skyscraper, at the bottom you've got the really terrifying stuff, unsafe code, and maybe one or two guys knows how that works. If they did it right, then the next floor up is all about interfacing with that in a way. You get all the way up the stack and you've got the poor junior programmer looking down with no idea what anything does beyond a few interfaces on the top floor.

Drew: You have an interesting perspective as someone that works in training and consulting. What do companies look like on the Rust-training side?

Herbert: It’s a spectrum. On one end are defense companies that heard the Department of Defense and the White House say, “Thou shalt not use unsafe languages,” so they went into a blind panic and started onboarding Rust as fast as humanly possible.

Herbert: I trained companies who make defense hardware where the equipment they've got has to run on pretty low-end stuff because it's going to be out in the field getting beaten up. You physically don't have room for a Java Virtual Machine or a Go runtime. On that end, it's kind of funny when you train them because you always have a split. Half the class are like, "Yay Rust," and the other half who are like, "We like C++." Usually you can win them over, but that can be awkward because you feel like you need to come in with a t-shirt saying, "I come in peace.” I totally get it. I used to do C++. But I like to say that's where my hair went.

Herbert: On the other end, for example, you have a very large research group doing Python code, that was mandated to use less CPU. The entire purpose of the training was to try and speed up some of the several thousand Python microservices that were being used.

Drew: On the training note, one thing you mentioned when we were preparing for this interview was the Rust learning curve. It's sort of infamous. Do you find that the people you train experience this steep learning curve?

Herbert: It's what they expect. Honestly, they come in looking terrified quite often. So, I structure the training really carefully to go the whole first day without mentioning memory once. You start with "Hello, World" and by the end of it you're serializing and deserializing data from the disk. So, at the end of the day, I can say, "Hey, did you notice we didn't write anything that looks remotely different to other languages? It's pretty straightforward stuff. It's just really fast, really easy to do. And we never once thought about memory management."

Herbert: Then the next morning, I leave a slide on the projector showing manual memory management with the ELEC system, and a little poll saying, "How many of you were expecting me to start with this?" In normal corporate training, you never get to that because Box, Vector, and the whole drop system mean you never need to touch that. Arc is as close as most people need to get.

Herbert: Where the terror kicks in every time is when I start trying to explain the difference between threads and async. Threads always go well. Async goes well if you've previously used an async system. And, I’m careful to not step outside of Tokio. As soon as you start to worry about memory—particularly Arc in a big Tokio-based service—that's where you lose people because that gets kind of scary. So most of the time I lean on Axum and its layers system to ease people into it.

Drew: That makes sense. Do you have any pithy, general insights about putting Rust in production after working with several different companies on this?

Herbert: Please write some unit tests, and make them tests where you're actually working on the right data. I actually saw someone lose something in production that worked perfectly in testing, and it turned out they were accessing the wrong dataset. Not my fault, but that happens. That applies to any language though. Measure before you cut. Also, you're not going to see massive benefits from a service that nobody uses. If you're going to port something, port something that's useful enough that you can make a comparison. I don't know how many big corporate American networks you've dug into, but when you've got thousands of services, nobody knows what some of them do.

Herbert: Another thing is please, please, please don't name everything A and B—including lifetimes and generics—because a future you in a month who's worked on other projects will have no idea what you're looking at. And, if somebody else joins the team, they're going to have no idea what they're looking at. That partly comes from helping a bunch of companies who have developed 25 years or more of C++ or C that they've just added to and never trimmed.

Herbert: I was at one company that has a library with a million lines in a header file and no comments. The fellow who wrote the first quarter of it apparently passed away. They had a whole team reverse-engineering what the library they rely on does. So don't go there. One guy writing some poorly commented code to a whole team reverse-engineering that same code—that's a bad trajectory. And it's more common than you might think too, because it's very easy to add code. It's really hard to take it away.

Drew: I know probably most or all of your work for clients is confidential, but could you speak in a general way or at the level of detail you're comfortable with about some of the interesting projects that you've seen or worked on?

Herbert: In the last six months, it seems like AI is the answer to everything. Everything I've been asked about has been in some way connected to AI, whether it's for coming up with custom models for a specific problem or just trying to pass data through a classifier very, very fast. Rust is making inroads into that because of the fast data moving part.

Herbert: You think it's written once in Python, and then the slow bits get ported over. Otherwise, outside of AI, I've been mostly seeing industrial interest on things like industrial control processes—things that absolutely have to be solid—that you might previously have written in Ada, but are usually written in C or C++, and they need a new version.

Herbert: I've talked to a few automotive groups. I haven't had a chance to train one yet because there's a certification that they need that I don't have yet. I’m working on that. I’ve also talked to quite a few defense companies like I mentioned. That's always a lot like the industrial side. There's almost always a control system that is a natural map—that naturally can be in almost any language—because it's basically pumping video streams, pumping data around. And then there's the equipment in the field, which is almost all in C because that's what's been used in embedded code for 30–40 years. We’re starting to see that ported over.

Herbert: What I'm not seeing Rust in, oddly enough, is website backend type things—except for the occasional finance service that just can't keep up.

Drew: Right, that's interesting. It seems like this has quieted down. It felt for a minute there was a big push from the Rust community to get Rust into the web, right?

Herbert: Into everything—for sure. But the web aspect does seem to have quieted down a bit.

Drew: I think people are realizing there are plenty of places for Rust to be applied in all of the areas you've just mentioned. And, I guess people are sometimes making the decision that the juice is not worth the squeeze in terms of using Rust for certain things.

Herbert: It depends what you're doing too. For your typical business web service type thing, Go is kind of a natural choice because the infrastructure is already there. You're not reinventing much of the wheel. I am sure you can do it in Rust. It'll be faster, but it'll take longer to develop. I've actually been working this week on a web service for an open source project I work on called LibreQoS. I'm doing it all in Rust, but that's because I know and love Rust. It's not because Rust is necessarily the best fit, but I'm going to be much more productive in Rust than I am if you put me down in front of Go.

Drew: I was going to ask about LibreQoS. What is that?

Herbert: LibreQoS is traffic shaping and scale for ISPs. The biggest one we have right now pushes about 100 gigs a second through a Linux box, and every packet that goes through gets classified, run through a shaping system called CAKE—which is Common Applications Kept Enhanced. It's been in the Linux kernel for about nine, ten years. We had Dave Täht, who was one of the brainchildren behind it, on our team, but he unfortunately passed away at the beginning of April. When we started, CAKE was great for home use, but it used a lot of CPU. We hacked away at it until we could get it up to—like I said—100 gigs a second if you've got beefy enough hardware, or deploy it pretty cheaply at ISP scale.

Herbert: It's focused on bufferbloat, which is basically—instead of trying to use bandwidth to throw all your packets through as fast as possible—it looks at the round-trip time of the actual connections that are going through and paces your packets out so as to avoid those little congestion spikes that give you bad Zoom calls when your daughter decides to download Peppa Pig during a meeting.

Herbert: That's the big goal there, and it helps the most on smaller connections. LibreQoS is open source and Herbert-funded largely, and we've been pushing hard. We have hundreds of deployments out in places like the Philippines, parts of Africa, where bandwidth is scarce. It's super helpful on a satellite link or massively congested link. We've even had cruise ships deploy it lately. The Linux kernel part for that is still in C, but we're looking to change that. We haven’t done it yet because the Linux kernel Rust story is still a little too full of drama for my taste. But the entire system for monitoring and everything is in Rust, and then there's a few control scripts in Python.

Drew: That's very cool. Do you come from a background in networking? How did you end up working on that?

Herbert: I’ve had way too many jobs. I worked for LibreQoS, I worked for Ardan Labs, I worked for an ISP in Middle-Missouri that I helped found about a decade ago. They specialize in wireless links out to the places nobody else can get to. Missouri's got some interesting terrain where you've got farm houses surrounded by swamp that—if you run fiber—the fiber is going to wash away or blow away, because it also gets tornadoes. Wireless just makes sense in areas like that. LibreQoS is kind of a nonprofit because the goal is to get service out to the places that can't get good service any other way.

Herbert: When the pandemic hit, everybody suddenly worked from home.The whole network changed. Suddenly business customers were in the same place as residential customers. Robert Chacon was just getting LibreQoS setup. I found it on GitHub, deployed it in a hurry, joined the project, and made it better. It pretty much saved our network, because when you suddenly have all your residential customers becoming business customers, it's a panic moment. Normal residential customers don't use a lot of bandwidth during the day. Everything's optimized on the basis that people will be somewhere else on your network.

Drew: That's really cool. Filtra is a hiring platform, so we always ask questions about hiring even if you’re not actively hiring right now. So, what qualities do you tend to look for in new people that you add to the team?

Herbert: I personally always look for people who have learned principles more than recipes. If you've memorized how to set up a web server or a web service following the same script every time, anyone can do that, including ChatGPT, sadly. If you understand the principles of how it works, when the next version of the server comes along, you're going to be way ahead of everybody who is copying the recipe. So that's my number one thing. My number two is creative problem-solving. A lot of the time, the solution that's picked is picked because it practically fits into what's available rather than, "Hey, this is perfect." You know? And I personally like to avoid infrastructure solutions that spend too much money, but that's just a personal belief. I think I have a deep fear of Kubernetes on some level.

Drew: Don't we all? Maybe the answer to this next question is obvious, but Ardan, from what I could tell, is fully remote. That’s different from a lot of companies that are doing the return-to-office thing. Is that because of the nature of consulting?

Herbert: Yeah, it would be kind of weird if we all had to live in the same place and then not be there half the year anyway. Bill and I are scooting all over the world. Prior to the pandemic, they did have an office, and quite a lot of the consulting happened from there. They went remote and found that remote worked and people were happier. As far as I know, there's no push to consolidate. I know they actually downsized the remaining official office they had because it was not being used very often. Miguel lives in North Carolina, Bill lives in Florida, I live in Missouri, Mickey lives in Israel. It would be kind of tough to get us all back in one place anyway.

Drew: Right, that makes sense. That would be a big challenge–Israel especially. I think for a lot of companies, it ends up being an advantage where they can hire someone like you that's not in the location that they would need, and they can get more quality hires by allowing flexibility that way.

Herbert: You just have to really keep up on the Slack because it's very easy to forget about your coworkers if you just get your head down and start working.

Drew: I wanted to ask about some of your more personal things. We just talked about LibreQoS, but you have written a couple of Rust books. We talked about Hands-on Rust and how that came about. What's the second book that you did?

Herbert: Rust Brain Teasers. The funny story there was they were setting up a series. It's the first time I've ever heard of an author being contacted by a publisher and asked, "Would you write this book for us?" which was kind of cool. But it's never likely to be a super bestseller. It's more a read-in-the-airport-for-fun type of book. It's short, it's a collection of puzzle-type things where it shows you some odd Rust and asks you what happens, and then takes a page or two to explain the underlying theory. Everything from everybody's favorite adding–point one, point two in a 32-bit float—the answer may not be point three—to a few of the more obscure memory ones. It was a tough one to write because Rust works so hard not to have the weird corners. I thought they should just put out C++ Brain Teasers. Isn't that the right language for that?

Drew: That's a testament to the quality of the Rust language I suppose. So you've got a new book?

Herbert: Advanced Hands-on Rust, which picks up where the first one stopped and teaches you the stuff that I couldn't fit in the book without adding another 200 pages and getting fired. Apparently nobody wants doorstop-sized books anymore!

Herbert: The first one gets you up and running. You pretty much get to threads, a little bit of traits and generics, but it's really just the basics. In Advanced Hands-on Rust, you're writing traits, writing generics, writing libraries, handling some threading and synchronization, a little bit of network code. So, it feels like I'm finishing the first book. It is also very late because I had a kid, and it's amazing how a little baby slows you down.

Drew: People will have to look forward to that coming out in the future. Any date?

Herbert: The beta is on the PragProg website. If you buy the beta of an ebook, you get all the future versions, including the final one. It'll actually be in print. Amazon is estimating September. That sounds about right because the final beta is with the publisher now, and it does take a long time to get all the steps before things are in print.

Drew: I tweeted before this asking people what questions they wanted me to ask you, and someone told me to ask you Emacs or Vim, and I laughed. But then in our private messages you were like, "Ask me about code editors."

Herbert: In my first draft of Hands-on Rust I included a whole chapter on how to set up Visual Studio Code with Rust Analyzer. It went out to technical review, and of 20 technical reviewers, I got 19 people complaining about that chapter because I hadn't picked their favorite text editor. I realized that you can never answer that question without being doomed. While I'm teaching, I use Rust Rover, and I use Rust Rover partly because JetBrains are kind enough to sponsor me and give me lots of free stuff. But also partly because I like it. It's got a really awesome presentation mode built in. When I’m using whatever projector the training hall happens to have, it gives me a pretty good chance of making it possible to read. So that's just really handy for that. When I'm at home, I have to admit I use NeoVim with the Rust Analyzer plugin. I sometimes use JetBrains’ stuff, but I tend to reserve that for the computers with more memory. Variants of Vim I can count on being on almost anything I connect to. So, that way most of the time I will feel at home. I did learn the hard way that you don't ever, ever, ever teach using something like NeoVim. Emacs too. People will appear in the chat asking you about your theme, asking you about your plugins, and so on. You won't get any teaching done. You will just be talking about the text editor for an hour, and then you're out of time.

Herbert: What I tell people, because people ask me this a lot, is use whatever you're comfortable with. I don't care. As long as it's got enough functionality to tell me things like where's this function defined and remind you of the parameters. Unless you have an amazingly better brain than me, you probably can't remember the right parameter order for every function. So that's the biggie. As long as it's got that, I don't really care if you're using Vim, Emacs, Nano, Notepad++. Anything. But readers please don't send me hate mail!

Drew: So you're not drawing any lines in the sand today?

Herbert: No, I'm going to leave that to the younger people to fight out. I will sit down at any computer and use whatever editor you give me if it's not Emacs. I have nothing against Emacs. I just cannot remember the keystrokes, and I feel like an idiot because I end up with a window reminding me what to press. When I was in college, Emacs was prohibited. In the computer labs, there were actually signs informing you to not run Emacs. It used too much RAM on old Sun workstations.

Drew: Fair enough. The other day I posted something where I was having an issue with my VS Code configuration. The first comment was, "You should use NeoVim." I responded with a gif that said "I've been expecting you."

Herbert: Yeah, the Vim keystrokes really take a lot of getting used to.

Drew: I actually was having a problem with the VS Code Vim extension. Maybe I should use NeoVim. The last thing that you mentioned you thought would be good to talk about is Bevy. What do you have to say about that?

Herbert: The Bevy game engine is some of the most amazing Rust engineering I've ever seen. I recommend looking at it and looking at how it works because it presents you with this really easy-to-use syntax as the user. It does dependency injection in real time in a game, and it still performs well. It also has crazy things like you can store basically a database into the component system of game data. You can then inject queries directly into your system code to control what happens in the game. It looks like black magic because it doesn't look like Rust anymore. You're filling out this variable with this definition, and then all of a sudden it's got all your data and there's no sign of where it came from. You can learn a lot from the Bevy code.

Herbert: You just get curious and you follow the definition of things and find out that they have this crazy type that matches tuples of up to 20, I think it is, trait bounds that are meticulously set up so that you can put them in any order and it'll still compile. It looks computer-generated. And then apparently 20 parameters wasn't enough for someone, so now they'll also accept any one of those can also be a tuple of up to 20 more. It's kind of scary looking at this trait that somehow manages to match a function. It can match a resource, something you stored for later use. It can match a query for the data engine. And, it can match parts of the graphics engine. So you start tracing down, and it is the most incredible nested map of traits I've ever seen to resolve what you're asking for at compile time. And then you schedule your systems into a DAG, a Directed Acyclic Graph, and assign them to CPUs at runtime so they run in parallel, but only if they're not touching the same data. It somehow pulls all of this off at speed.

Herbert: I've never seen anything like it. So, I recommend looking at it as a learning exercise. Start using it because it's a great example of how to make Rust beautiful. But also dig into how it works. I've never contributed to another project that the PR checker makes sure that your Rust documentation is correct too, and will reject it if you haven't commented precisely and beautifully. The whole project is like that. So, it is a beautiful learning source and just an example of what you can do. It's terrifying and beautiful at the same time.

Drew: That is cool. Do you think Bevy is on track to become a dominant game engine?

Herbert: I know there are two games on Steam that use it now. I don't think it's going to take over from Unreal and Unity anytime soon, at least not until it has a beautiful visual editor. A lot of game developers really like the visual editing approach. And, it probably needs to stabilize a little bit. I know in the next year or so they're hoping to get there with the editor, and then we'll see. It kind of had a knock-on effect. Them and the Godot engine seem to compete on features. As a result, open source just gets better for everyone.

Drew: Right. Competition is always the best for the user, right?

Herbert: At least as long as it's friendly competition.

Drew: Right. That's very cool. I'll have to maybe play around with Bevy. I haven't, but I continue to hear more and more good things. Seems like it would be a fun side project.

Herbert: It is. You can get Flappy Bird up and running in an hour if you know what you're doing. Actually, I suspect ChatGPT or Copilot could probably get it running in less than that.

Drew: That's right. That seems to be the thing that everyone vibe-codes, right, is a little video game.

Herbert: Yep. Vibecoding Rust has been an interesting experience. I've tried it a couple of times. It seems like Rust has too many fine details for the AIs to be getting them exactly right. It's getting closer every month. I'm just waiting to see how long I'm relevant.

Drew: Right. Aren't we all, honestly? Well, hey, thank you for doing this Herbert.

Herbert: No problem. Thank you.