A few years ago I joined a tools company, where I worked on a messaging framework. Without even realizing it, I had a lot of expectations and assumptions about how that kind of work looks like.
A few days ago @mathiasverraes reminded me about this with this Tweet:
I admit that I also used that believe that people working on infrastructure are smarter, have more interesting, fun and valuable project, etc. etc. etc. Some turned out to be true, but there were also a few surprises:
1. I spent (significantly) more time interacting with users and doing things other than coding than in any other previous job
That was to a large extent my choice and that probably varies a lot across companies… but when you think about it it makes total sense!
As a user, you want documentation, blog posts, educational content that is high-quality, informative, is actually useful and not trying to sell you anything. When you consider using some tool for your company you want your technical questions answered asap and have the ability discuss the architecture of your project with people who understand your questions, and, ideally, have done something similar before. When raising a support issue you’d expect the other person to be able to put themselves in your shoes and help you solve the problem instead of doing the minimum to satisfy the SLA contract, even if ultimately it turns out that the issue was in your code, not the framework.
In some companies, programmers don’t participate in those kinds of activities. Based on my experience I believe in the long run it’s a huge advantage for both customers and for the development team when they do. The feedback and insights you get are hard to get otherwise and help you develop better tools.
2. Good API design is sooo hard and frustrating
With every iteration, you have better ideas for improving the codebase, making APIs easier to use and more elegant. On the other hand, nobody likes having to deal with breaking changes! I remember how it felt when Angular 2 has been announced when we were in the middle of the huge project using AngularJS… Balancing those two forces requires a lot of skill, experience and there’s no way to make everybody happy.
It’s hard to resist adding a feature that can help 2 customers, but what if it will also make lives harder for another 20? Or what if you ship a feature that nobody uses but you can’t just remove it without any notice and have to do it across 2 major versions, according to your support policy? Or what if somebody relies on an undocumented feature or uses it in an unintended way and your “fix” will break their solution?
Similar dilemmas are your daily reality as a tools developer. There were many days when I wished we could forget about all the history and start from scratch instead of making changes in a less-elegant backward-compatible manner and patching 5 versions that were still officially supported, but this is what provides huge value to your customers.
3. Clarifying “business” requirements is even harder
When customers are highly technical it may be orders of magnitude harder to understand what problem they’re solving than when dealing with “business” users. What seems like a simple feature request on the surface may turn out to be a problem with the design, a rare edge-case scenario unique to the customer or a great suggestion for the next version. You never know until you dig deeper and get more context. You may also need to reach out to more customers to verify that the same solution won’t have a negative impact on them or will be beneficial for more than one team.
While doing so, you have to resist the urge to jump straight into the solution mode, ignore for a moment the first proposed solution, so conveniently offered by the customer, skillfully navigate through buzzwords and jargon to get to the essence of the underlying issue, and make an honest attempt to solve the problem without writing any code (at least at first :)).
Sometimes you just look dumb when you keep asking the same “please, can you explain to me what is the problem from the business perspective”, other times people can’t provide you that information simply because they don’t have the whole picture themselves. But often without that context and understanding, the solution you offer will be suboptimal.
4. Abstractions have their downsides
Contrary to the popular saying, there are problems in software that can’t be solved with an additional layer of abstraction.
In fact, many problems are way easier to solve when you operate on a lower abstraction level and can safely make certain assumptions about the environment, expected load, usage characteristics, etc. When working on a general-purpose tool you have to balance the needs of many different users and their contexts.
What does that mean in practice?
For example, reaching impressive limits of optimization is not a great investment of your time and effort, if it’s only beneficial in a few narrow edge-cases. You optimize for the majority. It’s reasonable to be (overly?) defensive in your design, to minimize the risk that some features will be used in unintended ways. When experimenting, you have to keep in mind your support policy and ensure you’re not taking on a large maintenance burden that’s not likely to provide a lot of value. You do lots and lots and lots of testing. And then some more.
In practice, sometimes it looks like it’s actually more “fun” to work on the business-facing app, where you have more context, can make educated assumptions and are free to experiment in a more aggressive way.
5. Value is harder to measure and often code is not the most valuable part
When you’re working on the business-facing apps you can directly see an impact of many things you shipped. You see that automating a process saved users 15 mins. here or increased sales by 5% over there. Even if you don’t have a specific number, the impact of a given feature is rather easy to verbalize and understand.
In case of tools, it’s a bit more complicated, because you’re in the middle of the process, providing just one little piece. A lot of things are out of your control.
Sure, you can say that it’s better if you can process 10’000 messages per second than 100, but how much impact it actually has for the end-user that’s clicking on a link to track his package? Would they be much worse off if the developer used the tool provided by your competition? How much time did they save or how much more money did they make by using your tool?
Measuring the impact on the end result of just one of the 1000s moving pieces is way more complex than measuring the impact of a single end-to-end feature. And you’d be surprised (disappointed? :)) if you knew how often we concluded that the highest impact thing we could provide is better documentation, call with a customer or a bug fix, rather than a new feature.
So, overall, is it better or not?
Don’t get me wrong, working for an infrastructure company is great! If you have an opportunity, give it a try and see for yourself how it is. For example, it indeed tends to attract ambitious, experienced, smart people. Every now and then you work on a problem that you’d likely not encounter in a typical BLOBA (Boring Line of Business App). Most of the time you talk to other technical people and can learn a ton of useful things from your customers.
However, at the end of the day, both kinds of jobs may be more similar than you’d expect. The grass is always greener on the other side of the fence etc., but the reality is a bit more complicated 🙂 Each line of work will come with their unique set of challenges that others don’t have.
So if you’ll ever find yourself daydreaming about working on an infrastructure project in the hope it’d solve all your professional problems, pause for a moment before you start polishing your resume. Give your current system a closer look, maybe it’s more interesting than you think.