Luca Palmieri, Mainmatter

Drew: I think a lot of people look at you as one of the people who “made it,” and was able to create a career in Rust. So, what was your path like? How did you get your first job in Rust?

Luca: Well, I didn’t. I did the other trick: you take your non-Rust job and make it a Rust job. When I joined TrueLayer, I was a machine learning engineer primarily working in Python. Then, I moved to a more backend engineering position where the primary language was C#. Over time, I built momentum and consensus inside TrueLayer around the idea that Rust was a desirable stack for backend development. The first projects were things like little CLI tools, CI tooling, and such. Nothing on the critical path. But it was enough to get people playing with the language so they could build their confidence. We organized some training sessions, and I gave some talks. We also started hosting the Rust London meetup. Eventually, the opportunity came for some greenfield projects where Rust was a really great fit. Specifically, we were developing a whole new core banking system. Once that project got underway, the rest was history. I wrote Rust for the next two years, and we onboarded more than forty developers using Rust while I was there. By the time I left, we had a hundred or so Rust services. So, the adoption was extremely successful.

Drew: That’s a pretty cool trajectory! I was going to ask if you have tips for people making a career in Rust, but given the history you’ve just shared, I want to ask something more specific. If you’re someone working in a non-Rust job, and you think Rust would be good for your company, what are the strategies for spurring adoption?

Luca: I think it’s very case-by-case. This is actually part of the advising I do in my current job with Mainmatter. Generally speaking, I think the first thing you have to do is look at it from the business perspective. So, set aside for a minute the fact that you and I love Rust. Think about whether or not it’s actually a good idea if you’re the CEO of the company. If you can come up with some solid benefits, write those down and try to explain how these tie to the bottom line.

Luca: Also, you should be honest. Nobody is going to take a pitch seriously if it’s all roses. So, if there’s retraining or hiring costs, be honest about that. Other common drawbacks are situations where you have to create integrations to existing systems. It will often increase complexity as well if you’re going from one language to two for example.

Luca: No matter what, the bottom line is that you have to pitch the business case. In some situations, you may come up with very compelling reasons. For example, if you’re in an environment that really values security, Rust is going to be an easier pitch, especially if you’re currently using a memory-unsafe language. Other times, the reasons may not be overtly compelling. TrueLayer was actually in this position. There was an argument that C# was good enough. Those cases are obviously harder.

Luca: In terms of building the business case, I would also include explanations of how you’re going to integrate Rust efficiently. Explain how you’re going to keep delivering during the adoption phase.

Luca: One more thought would be don’t go straight for the jackpot. Nobody is going to bet their critical systems on a technology they’ve never used before. So, find incremental ways to adopt where you can step-by-step derisk choosing Rust for the stakeholders.

Luca: Of course, all of this is very context-dependent as well. Seed stage startups move one way, more mature startups move another way, and enterprises move very differently. In all cases, you’ll have to be somewhat political. You’ll need to know how to navigate your office environment and decision-making chain. Overall, I would say your goal has to be to do what you can to make it easier for the people who have to sign off to sign off.

Drew: I think it can be hard to speak the language of business as you mentioned, but it makes sense that that’s what it takes. You mentioned one thing in there that I’m personally curious about because you do a lot of training and upskilling at Mainmatter. When do you think it makes sense for a company to train existing talent versus hiring people who already have the skills from outside?

Luca: Does the company have more work than development capacity? If so, you hire. If it’s just a matter of using a new technology, then you upskill. The other thing to keep in mind is that there are many cases where technology isn’t the bottleneck in delivering the result. Sometimes the bottleneck is domain knowledge or familiarity with the existing architecture. Even if you hire talent with specific Rust expertise, it’s usually a good idea to pair them with internal engineers so they can teach each other. We find that more often than not you end up wanting to train your existing people. You might start by hiring or bringing in contractors, but once Rust gets embedded in a company it tends to grow quickly and you’ll want existing engineers working on existing projects in Rust.

Luca: One other thing to remember is that knowing something well and knowing how to teach it effectively aren’t always skills you find overlapping in the same person. So, even if you have some great Rust talent, it may still make sense to bring in dedicated instructors.

Drew: That’s a good point. I’ve had some experiences where the person I hoped I could learn from wasn’t a great teacher or didn’t have the patience or time for it. Anyway, I got off on a bit of a tangent. Zooming out to your overall career, do you have any advice for creating a career in Rust?

Luca: If you’re hoping to land a job that uses Rust, especially in the current job market, I would say you need to meet at least half of the requirements. For example, say you’re applying for a Senior Backend Engineer position at a travel company. If you’ve never done backend before, you’re not familiar with the travel industry, and the only thing you know is Rust, it’s going to be tough. So, with that in mind, if you’re trying to jump from a non-Rust position to a Rust position, your best bet is to keep as many other variables the same. Find something that’s basically the same as your current job but using Rust. This is why moving inside the same company is a great path. Obviously, there’s not often another team in your company using Rust, but that is ideal. Once you have a couple of years of experience with Rust under your belt, you can change the other variables like the industry you work in. So, at this particular moment where it feels like there are a lot more candidates than positions, it’s important to exceed that 50% threshold. A few years ago, hiring managers were more willing to take bets. So, read the season.

Drew: I think that’s great advice. Changing gears a bit, your book Zero To Production In Rust is one of the things that’s gotten you a lot of visibility in the Rust community. What was the motivation for writing that book?

Luca: So, I was still at TrueLayer. Specifically, we had just finished a proof of concept in Rust. That had gone well, and so I needed to scale the team, training several engineers on how to do backend Rust. I started by googling. I found some scattered resources that covered a subset of the topics that I thought were important, but it was not a nice curriculum. So, I just wrote the curriculum I wanted. It started as a series of blog posts. At a certain point, the scope became large, and the resonance was significant, so I decided to turn it into a book.

Drew: I didn’t know that it started as a series of blog posts. That’s really smart!

Luca: That goes back to derisking as one of my life philosophies. If you set out to write a book, it feels super daunting and you risk never starting. Also, when you write a book, you write it in fragments like chapters and sections. You often don’t have something coherent until the end of the whole writing process. Blog posts are self-contained so that if you stop at any time, you still have usable material.

Drew: This is maybe a little off topic, but it caught my interest. Can you talk more about derisking as a life philosophy?

Luca: I think our industry has a fascination with the idea of risking it all. It’s almost a trope from classical literature that carries on today. People always talk about the idea of arriving in a new land and burning the ships behind you to force commitment. The tech version of it is singing the praises of people who drop out of school to do a startup. It’s all cool, but I think it’s more interesting to learn to take calculated bets. For example, if you want a job in Rust, that’s great. But, if you need the job right now, it’s probably smarter to just choose a very common tech stack for that first position. In the case of my book, I could have decided to go off in a cave and spend hundreds of hours writing only to come out and find out that nobody wants to read what I wrote. By releasing blog posts one at a time, I got to launch several times, gauge interest, and even improve things. Someone might have said that I wasn’t committed to the idea by taking that approach, but I don’t believe that. I think you start with some small bets. If they’re working, you double down. If they’re not working, you pivot.

Drew: I was listening to someone talk the other day about startups, and they talked about how everyone thinks about entrepreneurs as risk takers where in reality the best entrepreneurs derisk. So, that resonates with me.

Luca: Yeah, there will always be striking counterexamples, but I think the broad statistics show that derisking is a good approach.

Drew: I did want to ask you a little bit about your last job before we talk about what you’re doing now. You spent some time at AWS. I believe the team you were on was the Rust Service Framework Team. What was your work there?

Luca: As the name implies, we were building a service framework in Rust. I realize that doesn’t mean much on its own, let me elaborate. AWS has developed an interface definition language called Smithy, which is basically their version of grpc and protobuf. It’s a way to define what your service interface looks like. From that service interface, you can generate clients in multiple languages as well as generate service scaffoldings. This makes sure that all the clients work nicely with the server, and they can’t fall out of sync because they’re generated. It also makes sure that the server complies with the interface because the server scaffolding is generated from the schema. The schema is the ultimate source of truth. AWS invested very heavily in this early on. Smithy and the Rust Service Framework are both open source. My team was tasked with building a Rust server generator from a Smithy API definition. This was a way for teams inside and outside AWS to write servers using the Smithy definition language in Rust. There was a need for this at AWS because Rust adoption is quite strong, with more and more workloads that would historically be handled with Java transitioning to Rust.

Drew: Did you notice a big difference between working at AWS, versus TrueLayer, versus Mainmatter.

Luca: Of course. You’re looking at company sizes that are a few different orders of magnitude apart. Being so large, AWS works very differently. But at the same time, I was surprised by the level of decentralization AWS manages to deploy. Very rarely did I have to wait for somebody to sign off on something. It didn’t feel like I was part of a gigantic thing that moved slowly. In some ways it felt like I had more latitude to do things at AWS than at TrueLayer even. I guess it’s kind of a mixed bag.

Drew: So, this is a stupid question, but I can’t resist asking (laughing). It seems like so many of the former AWS Rust people have large social media followings.

Luca: Do they? I guess they do? (laughing) Or, at least we can say you know the ones that do!

Drew: Right! So, what is that about?

Luca: You only see the ones that have large followings. Therefore, it looks like they all do. But, I think a few factors have played into that. First of all, we all know each other. We’ve all been in the community a long time. When I was looking for a gig, I knew that Jon Gjengset was at AWS. So, it was natural for me to reach out and ask him about it. I assume the same was true for Tim McNamara. The other thing is that when you’re working on something like a service framework in a company like AWS, it kind of feels like working on a piece of internal open-source software. You write some libraries, and then a bunch of people throughout AWS need to like it, trust it, and adopt it. So, you end up doing a lot of education and documentation and that kind of outreach. So, I think the large social media following naturally falls out of that. Fundamentally, it’s just a group of people with strong technical communication skills.

Drew: That makes a ton of sense. Are you able to speak to what happened there when everybody left AWS all at once?

Luca: Yeah, it’s no secret. AWS abruptly changed their remote work policy. I signed that contract under the understanding that it was a remote-first position. I was in the process of moving back to Italy from the UK, and my relocation was denied. So I left. A bunch of other people were in similar situations.

Drew: Yeah, I remember hearing about that unfolding.

Drew: The way I see it is they lost a bunch of really good engineers. Is that the way you see it? What do you think motivated that decision?

Luca: I have no idea! (laughing)

Drew: So, you work at Mainmatter now. I have some specific questions about Mainmatter, but before that could you just give a high level overview of what Mainmatter does?

Luca: Sure, Mainmatter is a remote consulting company based in Germany. We’re scattered across Europe. Our services cover three main areas: training, team reinforcement (a bunch of us embedding on a team for a period of time), and custom development. Mainmatter covers the full stack. However, I obviously specialise in backend, and our backend work is primarily in Rust.

Drew: How did Mainmatter get started? It didn’t start with Rust right?

Luca: No, it didn’t. Mainmatter is 10 years old. At that time Rust was a footnote on Hacker News. Mainmatter started around ember.js. The company has then grown and diversified over time.

Drew: So, how do you go from ember.js to Rust? I know you weren’t there for the beginning of this, but do you know the story?

Luca: Every frontend eventually needs a backend. Mainmatter would work with whatever backend the client was using. They were and still are very flexible in that regard. But, at some point you also end up recommending technologies. So, two or three years ago, they identified Rust as a suitable technology to recommend for backend development. That’s when they decided to invest more in Rust.

Drew: I see. The other backstory I’m curious about is Mainmatter hosting EuroRust. How did that happen?

Luca: Mainmatter has a history of organizing events in the communities we participate in. They’re one of the organizers of EmberFest. Therefore, when they decided to start investing in Rust, they also started hosting an event. That was right after Covid, in 2022. We had another one in Brussels last year, and this year it’ll be in Vienna. That’s actually how I got involved with Mainmatter. In 2022, I submitted my fourth proposal to RustConf and got rejected, again. I was a little pissed off, and ranted on Twitter about how it had been nine years since they last accepted a talk about backend Rust at RustConf, even though backend is the largest cohort of Rust users, by far. Mainmatter was starting to work on EuroRust: they reached out to me and invited me to curate the schedule. I got together with Tobias and Stefan, and we’ve been curating EuroRust’s content since the first edition.

Drew: It seems like there’s another piece of career advice in there. You were developing your relationship with Mainmatter well before you thought about working for them.

Luca: Yeah, that’s important. Network matters, and you must cultivate your network well before you actually need it.

Drew: Absolutely. So, Mainmatter’s a consultancy, which means you get to see a lot of different companies. I think that’s an interesting perspective. What are the common threads you see in companies using Rust?

Luca: Well, the landscape is a lot more varied than you might think. For example, a lot of companies use Rust but never publicize that fact. Working at a consulting company, you actually get to meet a lot of these organizations. Part of our work at Mainmatter has been a series of interviews (now republished on the Rustacean Station Podcast) where we sought out companies like this. We felt that Rust podcasts and articles over-index on the Rust open-source ecosystem, without giving enough attention to production users.

Luca: Rust is all over the place for all different reasons. The driving forces are industry—and company-specific. Some come to Rust for speed. Some come to Rust for flexibility (the fact that it can be deployed on the server, in the browser, etc.). Others come for memory safety. There’s another group that now just sees Rust as an inevitability and wants to get on board early.

Luca: In terms of challenges, training is definitely one. People tend to rely on self-training, but things often reach a point where they feel more comfortable having professional training. This is especially true when they expand Rust out of that first early adopter group.

Luca: Ecosystem stability is also a challenge. The ecosystem is very large. There are lots of options. And, people tend to have a hard time deciding what dependencies to rely on. They wonder if things are going to work well for them or even if they’re going to be well-maintained over time. This also includes some challenges people have just navigating the cultural norms of the Rust ecosystem.

Luca: Compilation times remain painful. I hear that almost 100% of the time. The only exception are C++ developers, who don’t complain as much. Everybody else does.

Luca: With all that said, overall we don’t encounter major problems. As a general trend, people who choose Rust for their projects succeed at adopting it.

Drew: You talked a little about training just now. I know you have the Mainmatter workshops. That’s the training you do that I’m very familiar with. What are the other forms of training that you do through Mainmatter?

Luca: It’s primarily the workshops. We occasionally develop bespoke trainings for companies when they have specific topics they want to see covered. Then, the rest of the training is in the form of “team reinforcement.” That’s where we work alongside the developers on the client’s team and training happens as part of the job. So, that’s more of an ambient, continuous training style versus a list of things we want to teach over a set period.

Drew: How long do you usually spend with a company when you embed?

Luca: It depends on the project. It can span all the way from proofs of concept that last a week to projects that last multiple years. There’s really no fixed duration.

Drew: As far as the workshops, are you continuously adding new installments to the series of workshops?

Luca: Yeah, we are continuously expanding those. Generally, when a client asks for something bespoke, some of those things will eventually get massaged back into an offering suitable for a broader audience. We recently released something new called 100 Exercises To Learn Rust, which has become very popular.

Drew: So, the last thing I wanted to ask you about is your current side project, Pavex. As I was preparing for this, I came across an interview where you called Pavex “the weirdest of the web frameworks.” So, what’s weird about it?!

Luca: Weird probably wasn’t the best choice of word, but I was referring to the fact that the framework is quite different from the rest. If you’re coming from something like ASP.NET Core or Spring, it feels normal and sensible. It’s a framework built around dependency injection. It has an intermediate compilation step that’s used to do reflection, which Rust doesn’t natively support. So, there’s a lot of internal machinery that had to be developed to actually deliver a dependency injection interface that feels natural. So, that’s unusual. The other thing that’s weird is that Pavex aims to be a fully-featured web framework, or what some would call a “batteries included” web framework. To be fair, it’s primarily targeted at backend, but there are aspirations to deliver something for frontend as well. The current batch of Rust frameworks are very different in that they tend to follow the javascript tradition of being small and minimal, leaving most things to extensions. So, that’s where all the weirdness lies.

Luca: My feeling is that what the community needs is a framework that is easy to use, has good error messages, is feature-rich, and allows people to be productive with Rust on the web in a reasonably short time. I think Pavex and its peculiar architecture are an interesting and potentially successful solution to that set of problems. At a minimum, it’s an interesting exploration of a space that hasn’t really been touched yet in Rust.

Drew: So, what does the future look like for Pavex in terms of adoption? Do you have plans around fostering adoption?

Luca: Yeah, I’ve thought a lot about this. One thing I’ve done is a closed beta rather than putting it on crates.io and just opening the floodgates. The reason for a closed beta is to arrive at the open access stage with something that is reasonably polished and extensively tested. That way people can adopt the framework quite early without fearing that it will be completely different just a few months down the road. Beyond that, marketing doesn’t scare me. I’ve done it before, and I don’t see marketing developer tools as particularly difficult. At the same time, I’m not necessarily aiming for the broadest adoption. That’s not really my North Star metric. And, this is probably another weird thing. I think a lot of open source projects suffer from being widely adopted, because the popularity puts a lot of demand on the maintainers. And, maintainer groups don’t usually scale with popularity. Eventually some projects crumble under pressure or get bogged down by high demand. We’ve seen a few projects in the Rust ecosystem come apart this way. So, the goal with Pavex is for it to be a sustainable piece of software. In fact, although it’s open source, Pavex won’t be free. And, that certainly is going to slow adoption relative to alternatives. But, I don’t care so much about that. I have a specific target user in mind: businesses using Rust for production use cases. This will be a smaller crowd, but I’m fine with that.

Drew: That makes sense. So, last question… If I look at your body of work over your career, it’s really interesting to me that you’ve been able to hold down a full time job but then also be very prolific outside of your day job. For example, you wrote the book, and now you’re building Pavex. How do you do that?

Luca: I don’t know (laughing). I’m not a productivity guru with magic tips. I just do the things that interest me the most at any given point in time. I’m also strategic in the projects that I take on and how I approach them. For example, books have the advantage of requiring a lot of upfront work, but once they’re done they’re done. All I do in terms of maintenance is go back every few months and update libraries and things that may have gone out of date. A lot of my open source projects are also “done.” They address one specific problem, solve it, and people adopt it over time. There’s not a lot of feature growth over time. So, having projects like this allows me to “mentally” archive my projects. A common trap is doing a bunch of things where each thing adds one pebble of ongoing effort. At a certain point, you’re carrying around so many pebbles from existing projects that you just can’t do anything new. By the way, that’s why I was very reluctant to start Pavex. I think web frameworks are one of those things that are never done. That’s why I’ve also taken a different approach to setting up the project for sustainability. My hope is that the structure I’ve set up allows me to scale the project outside of myself if it ends up being very successful.

Drew: Thanks so much for sharing all your experiences and insights Luca!

Luca: Thank you!