Tag Archives: Architecture

Microservices – Fred George’s style

2 Replies

It seems like everybody is talking about Microservices nowadays. People cannot agree on whether it’s just good old SOA done right or a significant mental shift, many are afraid that this is not a safe approach for most teams and the hype might cause disasters. Many discussions regard semantics and basic building blocks. People try to put Microservices in some box and say “here, this is how you do it”. I think this is not very useful.

Last December I attended YOW Conference. There were a few presentations of people that have experience with Microservices in production. What I found most interesting is that every single one of them had different approach. There’s no one-size-fits-all and it’s a really good thing. I also attended a workshop run by Fred George, whose experience with Microservices spans across at least 3 different companies and over 10 years. The approach he promotes seems to go against many familiar “good practices”, yet, I think it’s fascinating and valuable.

At the workshop we were working on functionality that picked the best offers for the given customer (let’s say there were rental cars specials). There was a central, global offers repository, there were also local, regional offers, there were rules that specified which offers are available for specific demographic or customer type (e.g. only for those with golden status), we also had some data specifying past behaviour of the customers, etc. Each part of the system had a separate storage mechanism and was completely independent of the others. They communicated via a single RabbitMQ channel. At the final stage another service took all offers suggestions for the given customer and picked a few that were the best match (the ranking was based on probability of promotion being used and potential profit). For more detailed description of the problem check out slides from Fred’s presentation.

We were given implementation of the first service. Our task, as a group of about 5-6 devs, was to add more of them. At first I wasn’t fully aware how much what I already know gets in the way. My first inclination was to create new channels, organize messages in topics, we wanted to have various message types, smart routing and all this stuff… Those are basics, right, that can’t be over-engineering. It will be easier to extend… Well, but we weren’t even sure whether there will be any extensions. Ouch… Turns out all that stuff wasn’t necessary at all.

The approach Fred proposed was so simple that it was hard to believe. It was really dumb and wasteful, it felt so wrong… In short: the original message was broadcasted to all services, each of them had a kind of built-in filter (usually one or two if statements) that decided whether the service should even bother to do anything (in practice most of the received messages were ignored), then service modified incoming message by adding some extra information to it and… sent it back to the same channel it came from. Each service used only small piece of information available in the message (e.g. only 2 out of 10 properties). And… pretty much that was it.

It’s surprising how uncomfortable people felt with this approach. You could see the resistance, there were lots of questions, mostly focused on trying to come up with scenarios where this approach wouldn’t work. We were smarter than that. We knew there are lots of patterns and tools that could make this better (more scalable, sophisticated, elegant… you name it).

But in the end none of this was necessary. It worked. It was good enough. It was easy to understand and elegant (although looked much better in Ruby than in C#:)). We wouldn’t hit performance issues any time soon, even though each service received plenty of irrelevant messages. We could easily introduce more channels or topics if our system would become bigger, but that was a distant future. Each service was minimalistic, clean and contained all relevant information, thus the risk of introducing bugs was low. If I needed to modify (or create) a single service, I wouldn’t need to understand all the others, so the maintanance costs would be low too.

uSwitch succesfully uses this approach in production for a few years. It gave them unique business advantages. Apart from things discussed above I was shocked to learn that they got rid of basically all tests. Only a few years ago they were a poster child for succesful adoption of Specification by Example. Instead, they invested heavily in KPIs monitoring. Seems to be working way better for them. They’re ultra-agile, super-fast, experiment a lot and started making lots of money.

To wrap up – I’m really impressed with Fred’s take on Microservices. Although I still don’t feel fully comfortable with this approach, I see that it perfectly matches to what guys at uSwitch (and few other companies) are doing. Generally people reject the idea, they don’t think it’s “how things should be done”, if feels wrong, even if they agree that might be good for some scenarios they’re sure it wouldn’t work in their specific setting. That might be true. It surely doesn’t fit every context (e.g. trading company could go bankrupt before getting any KPIs for the day). But I suspect that if we thought hard enough about our hidden assumptions we might discover that it fits more contexts than we would expect at first.

I’m not sure wheter I’ll be able to use this exact approach any time soon, but I’ve learned a few things that can be easily applied to any setting:

  • Even though I’m rather careful in this area, I still over-engineer. I jump to familiar solutions and patterns, before asking myself whether I even need them. Doing the “simplest thing that works” sounds very easy, but is extremely hard to practice and requires a lot of effort as our knowledge and experience grows.
  • Life teaches us that developers are not very good in predicting future (see discussions on estimates), so usually we don’t know which piece of code will be extended and how. Simplicity and minimalism have higher chances of paying off than “building for extensibility” (a.k.a. adding pre-mature abstractions and unnecessary complexity).
  • “Business” is often willing to accept much more than we would expect. It’s good to start with determining what Udi Dahan calls “anti-requirements”, before thinking about solutions to expressed requirements. Business might be ok with losing some money occasionally or even occassionally suboptimal solutions, if they make more money faster overall. We might think we know what “anti-requirements” are, but until we ask we just limit ourselves (and very often we’re wrong anyway).
  • The same things that give us unique advantages tend to have unique costs. Sometimes technical limitations force us to rethink business requirements and change the way company operates. Just because something is not a “standard cost”, don’t assume that business won’t be willing to give it a try. Ask. Explain. Discuss. Let them do their business things, such as managing risks and costs or defining success criteria. You just keep asking and exposing hidden assumptions.
  • Some “technical” problems are in fact people and/or process problems. Replacing tests with KPIs monitoring might seem like an interesting technical challenge. But somebody had to decide what we measure and how. Somebody had to agree to the risk and sacrificing small, temporary loses for the sake of long-term speed, fexibility and profitability. A lot of people had to think out-of-the-box, and not all of them were developers. Lots of interesting, original solutions appear, when a diverse group works on them. I think we often don’t appreciate enough how valuable is to discuss our challenges with people who have a completely different perspective.

DDD Ultra-Lite

5 Replies

Some people say that DDD is heavy and expensive and as such must be used with care. Some teams adopt so called “DDD Lite”, which means they use “tactical patterns”, but don’t do much in terms of strategic design (or in plain English – they use software design patterns described by Eric Evans in the blue bible, but ignore the ideas that are considered truely important).

Truth be told, I’ve never worked on a “real DDD” project (yet!). However, DDD inspired me and I believe it made me a better developer. I used bits and pieces, most of them wouldn’t be recognized as “real DDD practices”, but I remember exactly where the inspiration came from.

I believe that the true value of what Eric Evans wrote years ago lies in the fact that virtually anybody can use some of the things he proposed. They don’t require any special tools, money, approval or reorganization. Those are simple things that you can start doing from next Monday (or tomorrow). I call those ideas “DDD Ultra-Lite”. Some of them are not new, but I believe Evans did a really good job of finding words and metaphors that stick and he certainly provoked a few AHA moments.

So here they are (in random order):

1. Listen to what business is saying

I’m not joking. The advice is probably as old as programming itself, but Evans has an interesting perspective.

He said that if business experts don’t understand your models or think they are awkward, then it means that you have to work more on that. Sometimes business people don’t openly oppose your idea, but you can see that they’re not very enthusiastic about it and that something bothers them, maybe they are confused or maybe they simply say that it doesn’t feel right.

There’s the opposite situation too. Sometimes when you finally find a good model and you communicate your idea to business expert he looks at you as you were the stupidest person in the world. Yeah, you are right, that’s obvious, what took you so long to understand this simple concept? That indicates that you finally got it right.

2. Do just enough design

Both over- and under-engineering are problems. Finding the balance between deep design and just “shipping features” is not a trivial task. Evans gives us a few tips that I found very useful:

  • Use the simplest possible model that allows for solving current problem. Don’t try to model reality, you will fail.
  • Remember about clean code, make your code and design easy to change and improve in the future. If your project is successful then you will change your design many, many, many times. As such, you don’t have to get it perfect first time (and probably even can’t).
  • If code starts getting in the way and you notice that making changes becomes more and more difficult, it might be a sign that your problem has outgrown the current model. Time for re-design of this piece.
  • Redesign small pieces of the system as need arises. Again, strive for the simplest model that solves the new problem.
  • Collaborate. Go for a coffee with a group of devs and/or architects. Discuss your problem with them, brainstorm, sleep it over and go back for another session.
  • Don’t make it heavier than it needs to be. Evans suggests multiple iterations, short discussions (1-1,5h), lots of learning and humility.

3. Ask powerful questions

Sometimes I work on a new feature or modify the existing one and the design starts getting ugly. It was just fine right before this change, but suddenly it becomes very complicated or just looks bad. This is a sign that the design should be changed, nothing special there. However, in Evans book I found a couple of questions that helped me on few occasions: Am I missing some domain concept? Does this change introduce or uses a concept that doesn’t match the existing model?

In other words instead of adding a few more if-s here and extra helper class there, maybe it’s time to revisit the design and find something more robust, more general or simply different. Something that explicitly caters for this new concept that just crashed our beautiful model. This is especially true if you know that there are more upcoming changes in this area or you work on a piece of code that is modified frequently.

4. Don’t reinvent the wheel

Evans encourages seeking external inspiration multiple times in the book. He mentiones browsing through patterns calatogs, to find out whether we can use one of them (possibly with modifications). He encourages investigating how similar problems are solved in other contexts (e.g. in other industries).

What still surprises me is that I haven’t experienced this approach in many places. I know some people look for solutions on the internet, browse through blogs, watch presentations from conferences, but as far I haven’t heard anybody say “I read it in XYZ book”. Usually the “design” sessions I attended were pretty much random brainstorming meetings, few people even prepare for them or know what exactly the problem is (e.g. has numbers describing non-functional requirements). Then months later we come across some random presentation and it turns out that we could have used something else and avoid a few problems.

I know that people talking at conferences and scientists make extensive use of solutions used in other industries. They try to find common issues and adapt solutions to specific needs. However, I think that in “real-life” very often we rely on personal experiences in other industry (e.g. in previous job) rather than extensive research.

The improvement I personally made is to make the “research” phase more organized and extensive and to revisit some “classics”. Blogs and presentations are extremely valuable, don’t get me wrong, but books tend to be better organized and contain more time-proved material. Not to mention that some of the “brilliant” articles are just plain wrong and relying on random Google search for unknown subject is not a good idea. If it’s written by a respected expert that should be all right most of the time. But even experts disagree and don’t always explain their unique context when describing solution.

If I remember something that might be useful I go back to the source of this information to refresh my memory. I try to find related articles, books, presentations or case studies. If I absolutely have no idea where to begin, then there are a few alternatives that proved useful for me: going back to the theory (I’m still surprised how valuable that is), browsing through the catalog (e.g. design or analysis patterns), while focusing on context and problem descriptions or finding a good “domain guide” for IT professionals for a specific industry.

That might sound trivial but I encourage you to really try it “as you meant it”. You might be surprised (I definitely was:)).