Tag Archives: Design

What is good for everything, is not really good for anything

Some time ago I worked on an ecommerce system, which allowed for very flexible promotions setup. If I remember correctly, it was poosible to create roughly 30 types of promotions. As you might imagine, testing all of it was a pain, since for every single promotion type I would need at least 10 test cases. So instead I decided to find out what combinations are actually used and focus only on realistic scenarios.

One day later I learned that business used only 5 fairly static kinds of promotions. And thank you very much for asking, they are happy with that number. They don’t need more. But since we’re already talking, would it be possible to have a promotion name in a dropdown and prepopulate fields accordingly? Most of the setup was static per promotion type and there was sooo much clicking, wouldn’t it be easier to just select a name from the dropdown, enter one number and voila, new promotion is ready? Knowing that promotions are much less complicated than we thought would also simplify implementation, not to mention testing…

I remembered that situation when Udi Dahan told us a universal reporting tool story during his ADSD course. He asked if we were ever told to create a universal reporting tool with all sorts of fancy filtering, sorting and presentation features? Did it allow for saving all those uber-flexible setups and giving them names, so business can run the same reports over and over on regular basis? Maybe you also find it disturbingly familiar?

The problem with flexible tools is that building them is extremely complex, they need to perform well on huge amounts of data and using all possible combinations of available options. That’s exactly what MSExcel does and Microsoft worked on it for a number of years now. Yet, usually we’re asked to build ‘mini-excel’ in much shorter time and given much smaller team.

The sad part is that most of the time the flexibility is not really useful. Most users save their 5 or, in extreme cases, maybe 10 static setups and run reports on them for months. So first of all, it’s a wasted effort. The exception is small percentage of users that really don’t know what they’re looking for in data until they actually see it. They play with models, look for patterns. But they’re minority and would be probably better off using tools dedicated for that kind of exploration.

Then, knowing what information user looks for, we could give them much better experience, so it’s also a lost opportunity. In case of promotions, we could save them typing and clicking. Given they had to setup dozens of promotions each month that would add up very fast and result in much better experience. In case of reporting we could think about generating ‘real time reports’ and sending them alerts. For example if every day somebody needs to run a report to check whether there were any orders matching some parameters, wouldn’t it be great if they got an email every time it actually happened? I bet they would love you for that feature. Not to mention how much work you would save yourself by implementing simpler, narrow solution.

So if ‘the other way’ is so bright why we keep working on universal, super flexible, yet not fully utilized features? In my experience the following factors contribute to the problem:

  • We don’t talk (enough) to business. The most obvious one. Maybe we’re not interested in business side of things, or maybe we don’t have opportunity to talk to the right people. Either way it’s obvious that something is missing here.
  • Business has to constantly fight for IT resources. So since we’re already doing something for them, they tend to ask for anything they can think of. After all nobody knows when they get hold of any geek the next time. By having a flexible tool they can do their job without needing us that often.
  • We (as an industry) taught business to speak our language. As an industry we didn’t do a very good job in learning how to get requirements out of business users. Not stories, not features, not “I want Y shown when I click X on ABC screen”, not CRUD. I’m talking about getting to the source, to what problem business is trying to solve with the feature they dream about, about discovering their why. Only then we can help them to come up with a good solution. Blindly following the specification is just not enough. Business users don’t come up with best solutions and designs. That’s our job.
  • The more complicated the feature, the more fun it is to implement. Let’s face it, sometimes it’s just too tempting 🙂

Are there any more factors you observed? Please, share them in comments.

Performance myths and facts

Recently I came across a few performance experts that turned upside down what little I knew about performance. I thought this is boring, this is low-level, this is better left to some crazy assembler gurus, who think microbenchmarking in the middle of the night is pure fun. Turns out I was wrong on many levels and I want to share the most important things I’ve learned as far.

The subject turned out to be surprisingly exciting and at the same time quite challenging, so I’ll keep writing about this as my learning progresses.

Myth #1. Performance is a trade-off between clean design and low-level ugliness

I’ve always thought about “highly optimized C#” as impossible to understand IL tinkering. According to Martin Thompson that approach might be appropriate for maybe 5% code we write. Most of the time we should simply rely on clean design rules such as single responsibility principle, high-cohesion, keeping our functions small, etc. The things that we really should invest in understanding though, are data structures and algorithms, their characteristics and how they are implemented. Understanding how their choice impacts performance is the single most useful thing we can learn. Martin talks a lot about Mechanical Sympathy and it turns out that it’s possible to explain why clean code is faster when we analyze how modern computers work, especially CPUs and GCs. This knowledge is also very useful when deciding what data structures we will use.

A bit different perspective came from Gil Tene, who made me realize how intelligent modern compilers are, especially JIT compilers (that we have both on JVM and .NET). Knowing how advanced compilers are, Gil advises to not waste our time for trying to compete with them. It’s much better to focus on making our code clean, readable and easy to maintain, because compiler will take care of making it optimized anyway (e.g. it’ll remove redundant variables, inline functions, etc.). Most of the simple “optimizations” turn out to be what compiler would do anyway, so it doesn’t make sense to sacrifice readability for this imaginary “performance improvement”. It might seem otherwise though, because in some settings (e.g. in VS by default) JIT compilation is turned off in debug mode. Of course, it’s useful to understand how all this works under the hoods,  because we can realize which of our pseudo-optimizations are a clear waste of time. Besides, sometimes we might hit subtle issues when compiler tries to optimize our code too much.

Myth #2. Performance optimization is a combination of crazy tricks

Both Martin and Gil talk a lot about “behaving nicely”, being predictable, following best practices and understanding the assumptions underlying the platforms and tools we’re using. Lots of super-smart people work on making our tools intelligent, force them to do all the hard low-level work for us, so if we just understand how we’re supposed to use them, we should be allright most of the time. The crazy tricks might be sometimes useful, but unless we work on a real cutting-edge system probably we won’t need them often.

Again, a lot of those assumptions are related to clean design and simple fundamentals, for example:

– Methods inlining – .NET JIT compiler will inline our function if it’s below 32 bytes of IL (short functions!) and is not virtual (be careful with inheritance!) *;

– Garbage collection – in a “well-behaved” application your objects either live very short (0 and 1 generations) or live “forever” (2 generation). Contrary to what many developers believe, you can have lots of GCs and still have great performance, just make sure they’re not full collections (generation 2).

So before I get to those crazy tricks, that I can talk about over a beer to impress other devs, I’ll spend more time learning about .NET, especially with regard to garbage collection and JIT compilation. I’ll pay special attention to what design choices were made, why, what alternatives were dismissed, how all that impacts me and what expectations it sets for my applications (or in other words, what “well-behaved” application really means).

Myth #3. More threads = faster

I was really amazed when I first learned that all LMAX‘s business logic is executed in one thread. Michael Barker and Trisha Gee showed one of the reasons why it was designed this way. I didn’t realize that the cost of locks is that high and even though the specific numbers might be the result of some bug on MacOS (which Michael noted were very impressive), it left me with a lot of food for thought. Maybe the differences are not that high if bug is fixed, but still it’s not really obvious whether adding threads will make my application faster. What is sure though, is that it significantly complicates design, increases maintanance cost and can result in more bugs.

Right now we talk a lot about parallization, using multi-cores to the maximum, microservices… The hype is high and it’s very natural to pay attention to new, shiny silver bullets. However, it’s useful to keep in mind that most of the things done in really cutting-edge systems, are way more than we need in the old good BLOBAs. Chances are we don’t have to be “as good as Netflix”, simply because our system doesn’t need it. We can’t forget that increasing complexity of our solution significantly increases the cost of maintanance, training people, etc.

While most of the world started talking a lot about how much we really, ReAlLy, REALLY need multi-threaded systems to survive in the current market, the amazing team at LMAX used the completely not fancy approach – they focused on their specific requirements, measurements, architecture, data structures, using known patterns in a fresh way and came up with an amazing result. And they process 6 milions transactions per second on a single thread. Not sure about you, but seems that should be more than enough for what we need in the systems I work on at the moment.

Myth #4. First get your application working then uglify optimize

It’s been my experience that performance is treated as an after-thought. We only talk about performance when something bad happens, but not many teams really monitor it on regular basis or are even aware what is needed. Sometimes we have requirements that specify that system should handle specific number of requests per second under whatever load and this is the best I’ve seen as far. But as Gil Tene said many times, even when we have some “requirements”, very often they are either not precise enough, we fall prey to common misconceptions regarding measurement or we simply ignore “all the rest” that is not part of our current requirement (such as 99th percentile).

All performance experts agree that we have to start design with a clear idea what do we need. So it’s virtually impossible to design application without having precise expectations regarding the worst case scenarios and unacceptable behaviours and at least some rough estimates of important numbers (e.g. how many users we might expect, what are their expectations, what would happen if performance would be bad for a few people but most of the time it’s excellent, what metrics matter for us, how will we collect them). It’s also useful to determine at the very beginning what are the real limits and start formulating requirements from there (e.g. network latency, performance of other applications we integrate with, industry “best examples”).

Gil in his presentation noted that without having an understanding of what we really need, we use tools in inappropriate ways and waste time optimizing things that already are good enough or try to optimize them in a completely inappropriate way. Sometimes the good solution might be completely counterintuitive, such as to sacrifice performance, deliberately make it worse in some places, in order to guarantee that our system meets SLAs even in “exceptionally bad” moments. What I understood by that, is that performance is not a binary quality – our system’s performance is not just either good or bad. Moreover, there are no silver bullets and simple answers, what is good for one system and one team, doesn’t have to be good for another. In most cases there’s no one single way to improve performance, there’re plenty of them, each with their own advantages and problems. I need to understand their trade-offs and verify why my system doesn’t perform well. Maybe the problem is in architecture, maybe in data structure, maybe in a language construct, maybe the array doesn’t fit cache, maybe I need to tweak GC settings, introduce pooling for big objects or upgrade hardware. Very often I can solve the problem in multiple ways, each allows me to achieve my goal, but comes with different trade-offs.

All of this sounds very complicated and I think it really is in the beginnig. Just like everything in programming. There’s a lot of conflicting advice, experts disagree or find problems in standard approaches, there’s a lot of jargon or referring to concepts I don’t understand very well.  Last but not least, every system is different and I can’t simply blindly follow somebody’s solution, without first understanding the problem. What is easy is to use a random tool, follow “Getting started” tutorial, without spending too much time trying to understand what I’m actually doing and whether my optimizations are even required. So for me the lesson is that it’s important to know your application, have data regarding expectations and requirements, but also constantly collect information about performance in production and in test environments. That sounds like a lot of work, but without it the best I can do is count on my luck.

I really recommend to see slides from Gil’s presentation, he explains the issue with performance requirements (slides 15-29) and discusses the common problem with measurement tools (slides 30-46).

Myth #5. Microbenchmarks are the most basic tool

I thought that the first thing I should learn about performance are microbenchmarks, how to create them correctly and when they are really useful. But it turns out they’re not that important. This article concerns Java, but summarizes nicely what I’ve read in a few different places.

Seems that although microbenchmarks might be valuable from time to time, it’s way less often than I expected. They are prone to errors due to JIT optimizations (which most likely will be different for production and test environments), hardware differences, unrepresentative inputs, etc. It’s still useful to understand how to construct a meaningful benchmark, but definitely it should be used with caution and only when we’re really sure we need them and nothing else would do.

* Note there’re more rules than I mentioned, but those are most obviously related to clean code recommendations.

DDD Ultra-Lite

Some people say that DDD is heavy and expensive and as such must be used with care. Some teams adopt so called “DDD Lite”, which means they use “tactical patterns”, but don’t do much in terms of strategic design (or in plain English – they use software design patterns described by Eric Evans in the blue bible, but ignore the ideas that are considered truely important).

Truth be told, I’ve never worked on a “real DDD” project (yet!). However, DDD inspired me and I believe it made me a better developer. I used bits and pieces, most of them wouldn’t be recognized as “real DDD practices”, but I remember exactly where the inspiration came from.

I believe that the true value of what Eric Evans wrote years ago lies in the fact that virtually anybody can use some of the things he proposed. They don’t require any special tools, money, approval or reorganization. Those are simple things that you can start doing from next Monday (or tomorrow). I call those ideas “DDD Ultra-Lite”. Some of them are not new, but I believe Evans did a really good job of finding words and metaphors that stick and he certainly provoked a few AHA moments.

So here they are (in random order):

1. Listen to what business is saying

I’m not joking. The advice is probably as old as programming itself, but Evans has an interesting perspective.

He said that if business experts don’t understand your models or think they are awkward, then it means that you have to work more on that. Sometimes business people don’t openly oppose your idea, but you can see that they’re not very enthusiastic about it and that something bothers them, maybe they are confused or maybe they simply say that it doesn’t feel right.

There’s the opposite situation too. Sometimes when you finally find a good model and you communicate your idea to business expert he looks at you as you were the stupidest person in the world. Yeah, you are right, that’s obvious, what took you so long to understand this simple concept? That indicates that you finally got it right.

2. Do just enough design

Both over- and under-engineering are problems. Finding the balance between deep design and just “shipping features” is not a trivial task. Evans gives us a few tips that I found very useful:

  • Use the simplest possible model that allows for solving current problem. Don’t try to model reality, you will fail.
  • Remember about clean code, make your code and design easy to change and improve in the future. If your project is successful then you will change your design many, many, many times. As such, you don’t have to get it perfect first time (and probably even can’t).
  • If code starts getting in the way and you notice that making changes becomes more and more difficult, it might be a sign that your problem has outgrown the current model. Time for re-design of this piece.
  • Redesign small pieces of the system as need arises. Again, strive for the simplest model that solves the new problem.
  • Collaborate. Go for a coffee with a group of devs and/or architects. Discuss your problem with them, brainstorm, sleep it over and go back for another session.
  • Don’t make it heavier than it needs to be. Evans suggests multiple iterations, short discussions (1-1,5h), lots of learning and humility.

3. Ask powerful questions

Sometimes I work on a new feature or modify the existing one and the design starts getting ugly. It was just fine right before this change, but suddenly it becomes very complicated or just looks bad. This is a sign that the design should be changed, nothing special there. However, in Evans book I found a couple of questions that helped me on few occasions: Am I missing some domain concept? Does this change introduce or uses a concept that doesn’t match the existing model?

In other words instead of adding a few more if-s here and extra helper class there, maybe it’s time to revisit the design and find something more robust, more general or simply different. Something that explicitly caters for this new concept that just crashed our beautiful model. This is especially true if you know that there are more upcoming changes in this area or you work on a piece of code that is modified frequently.

4. Don’t reinvent the wheel

Evans encourages seeking external inspiration multiple times in the book. He mentiones browsing through patterns calatogs, to find out whether we can use one of them (possibly with modifications). He encourages investigating how similar problems are solved in other contexts (e.g. in other industries).

What still surprises me is that I haven’t experienced this approach in many places. I know some people look for solutions on the internet, browse through blogs, watch presentations from conferences, but as far I haven’t heard anybody say “I read it in XYZ book”. Usually the “design” sessions I attended were pretty much random brainstorming meetings, few people even prepare for them or know what exactly the problem is (e.g. has numbers describing non-functional requirements). Then months later we come across some random presentation and it turns out that we could have used something else and avoid a few problems.

I know that people talking at conferences and scientists make extensive use of solutions used in other industries. They try to find common issues and adapt solutions to specific needs. However, I think that in “real-life” very often we rely on personal experiences in other industry (e.g. in previous job) rather than extensive research.

The improvement I personally made is to make the “research” phase more organized and extensive and to revisit some “classics”. Blogs and presentations are extremely valuable, don’t get me wrong, but books tend to be better organized and contain more time-proved material. Not to mention that some of the “brilliant” articles are just plain wrong and relying on random Google search for unknown subject is not a good idea. If it’s written by a respected expert that should be all right most of the time. But even experts disagree and don’t always explain their unique context when describing solution.

If I remember something that might be useful I go back to the source of this information to refresh my memory. I try to find related articles, books, presentations or case studies. If I absolutely have no idea where to begin, then there are a few alternatives that proved useful for me: going back to the theory (I’m still surprised how valuable that is), browsing through the catalog (e.g. design or analysis patterns), while focusing on context and problem descriptions or finding a good “domain guide” for IT professionals for a specific industry.

That might sound trivial but I encourage you to really try it “as you meant it”. You might be surprised (I definitely was:)).