Category Archives: Technical

What is good for everything, is not really good for anything

Some time ago I worked on an ecommerce system, which allowed for very flexible promotions setup. If I remember correctly, it was poosible to create roughly 30 types of promotions. As you might imagine, testing all of it was a pain, since for every single promotion type I would need at least 10 test cases. So instead I decided to find out what combinations are actually used and focus only on realistic scenarios.

One day later I learned that business used only 5 fairly static kinds of promotions. And thank you very much for asking, they are happy with that number. They don’t need more. But since we’re already talking, would it be possible to have a promotion name in a dropdown and prepopulate fields accordingly? Most of the setup was static per promotion type and there was sooo much clicking, wouldn’t it be easier to just select a name from the dropdown, enter one number and voila, new promotion is ready? Knowing that promotions are much less complicated than we thought would also simplify implementation, not to mention testing…

I remembered that situation when Udi Dahan told us a universal reporting tool story during his ADSD course. He asked if we were ever told to create a universal reporting tool with all sorts of fancy filtering, sorting and presentation features? Did it allow for saving all those uber-flexible setups and giving them names, so business can run the same reports over and over on regular basis? Maybe you also find it disturbingly familiar?

The problem with flexible tools is that building them is extremely complex, they need to perform well on huge amounts of data and using all possible combinations of available options. That’s exactly what MSExcel does and Microsoft worked on it for a number of years now. Yet, usually we’re asked to build ‘mini-excel’ in much shorter time and given much smaller team.

The sad part is that most of the time the flexibility is not really useful. Most users save their 5 or, in extreme cases, maybe 10 static setups and run reports on them for months. So first of all, it’s a wasted effort. The exception is small percentage of users that really don’t know what they’re looking for in data until they actually see it. They play with models, look for patterns. But they’re minority and would be probably better off using tools dedicated for that kind of exploration.

Then, knowing what information user looks for, we could give them much better experience, so it’s also a lost opportunity. In case of promotions, we could save them typing and clicking. Given they had to setup dozens of promotions each month that would add up very fast and result in much better experience. In case of reporting we could think about generating ‘real time reports’ and sending them alerts. For example if every day somebody needs to run a report to check whether there were any orders matching some parameters, wouldn’t it be great if they got an email every time it actually happened? I bet they would love you for that feature. Not to mention how much work you would save yourself by implementing simpler, narrow solution.

So if ‘the other way’ is so bright why we keep working on universal, super flexible, yet not fully utilized features? In my experience the following factors contribute to the problem:

We don’t talk (enough) to business. The most obvious one. Maybe we’re not interested in business side of things, or maybe we don’t have opportunity to talk to the right people. Either way it’s obvious that something is missing here.
Business has to constantly fight for IT resources. So since we’re already doing something for them, they tend to ask for anything they can think of. After all nobody knows when they get hold of any geek the next time. By having a flexible tool they can do their job without needing us that often.
We (as an industry) taught business to speak our language. As an industry we didn’t do a very good job in learning how to get requirements out of business users. Not stories, not features, not “I want Y shown when I click X on ABC screen”, not CRUD. I’m talking about getting to the source, to what problem business is trying to solve with the feature they dream about, about discovering their why. Only then we can help them to come up with a good solution. Blindly following the specification is just not enough. Business users don’t come up with best solutions and designs. That’s our job.
The more complicated the feature, the more fun it is to implement. Let’s face it, sometimes it’s just too tempting 🙂

Are there any more factors you observed? Please, share them in comments.

Why DDD is so damn hard?

5 Replies

In my latest post I mentioned the quote from Evans book. I met quite a lot of people interested in DDD, that either missed this quote or didn’t realize how significant it is. Yet, I think this is the single most important thing you should understand before you even attempt to start doing DDD. Even more so if you want to advocate it in your company and drive the adoption.

DDD is hard. Not because it’s complicated. In fact it’s a rather simple idea. Not because you have to be another Evans to get any value out of it. You can do a lot of tiny things, right where you are now, starting today. Not because there are no materials available or that the good advice is given only to the chosen few. Everything’s out there, easy to find, a lot of useful things you can get for free or pay a few dollars for a book.

I think that DDD is hard for two main reasons – it requires a completely different skillset than what is normally expected from developers and a completely new way of thinking. You need to become the soft geek and watch out to not slip into the more comfortable mode.

The soft skills can (and should!) be learned, also by means of formal training. There’s really a lot of good material out there on Emotional Intelligence, motivation and goal setting, communication, dealing with conflict and leading change. At the end I put links to my favourite resources, in my preferred “geeky style” – very pragmatic, based on solid research, with advice specific enough to get it into practice right away. I’ve gone through every one of them more than once and intend to do it again and again. I learn something new every time. But this is the easier part, if you know what you’re looking for, then eventually you’ll find a way to get it.

The other problem is that the way you approach DDD is very counterintuitive, it’s different from what usually works for us, it’s very, very, very uncomfortable (it really is!!!). There are some experts, but you need to learn to be your own expert, because DDD is messy and very sensitive to the context. What worked for Evans, might not work in your project. You can’t really get a good answer to DDD-related question on StackOverflow. There’s no book or a flowchart that will help you with making decision. You need to learn to observe, listen, become more self-aware, recognize hidden assumptions and… your own biases.

Evans casually mentions that the technical talent usually prefers to work on elaborate frameworks, rather than solve domain problems. Rather than treat software as a tool, means to an end, we learn to use it as a universal hammer. This is how we work, this is how we provide value, this is what we’re proud of doing, in essense – this is who we are.

Learn to see it, notice limitations it introduces. Then start asking why, why, why? Don’t be afraid to take a couple of steps back and question everything you “know” to be true. You might feel uncomfortable, because you suddenly stop “getting things done”, you feel unproductive. Even though you understand on an intellectual level that this way you provide more value, it’s still challenging on the emotional level.

When I first started asking a lot of questions, every single task was taking a lot of time. We had lots of meetings and discussions. Some colleagues told me that if the customer can’t get the requirements right, then it’s their problem, we should do our best, the way we understand it. I didn’t agree. I worked my own way… and felt very unproductive.

After all those discussions some work was even dropped, because we realized that it was in the backlog for too long and it wasn’t relevant anymore. It happened once for 3 tasks in the row. So my colleagues joked that they will give to me any task that they didn’t want to do and maybe I’ll convince customers that we shouldn’t do it. It was funny, but also… After a few days of work I had nothing to show, no results, no feature shipped. All those meetings and discussions were demanding, I had to think how to explain things, watch out for assumptions to question. It would be so much easier just to do as I was told and do my best to interpret requirements without asking any questions. If something was wrong, they’d have noticed, right?

There were other things too. Since I’m an introvert all that extra communication cost me a lot of energy. Some days I was literally exhausted, after work I just wanted to sit alone in my room and don’t talk to anybody. Then I didn’t know how “those things” should be done. I was new to the system. I asked a lot of silly questions, quite a few times felt like a total idiot and promised myself that I won’t ask any more. Then I felt that I’m simply annoying because I give all those people lots of extra thinking, extra work, they surely must have hated me for this, not to mention that I slowed down the whole process.

For some reason that I can’t explain I believed this to be a good approach. And over time it started slowly paying off. We had an important project that involved changes in 30 different systems at the same time. My team worked on only 2 of them. The project lasted a few months. We managed to catch a few important misunderstandings at the very early stage of the project… and then we wasted the whole day discussing how to ensure data is consistent at the end of the “processing window”, only to discover that the business is ok with temporary incosistency. We insisted on having enough time for tests and consulted test cases with business. We caught a few other misunderstandings. The project was successful, there were maybe 3 bugs found in the business testing phase (compared to literally thousands raised for those other 28 systems). Questioning everything and talking to business weren’t the only reasons for success, but I know that if we didn’t do it, we’d probably miss those misunderstandings and they’d be discovered at a very late stage. We had enough time to find alternatives and we avoided re-work.

Still… I changed jobs, went to another company at the opposite end of the world, worked with a different client and it all started again. Feeling like an idiot more times than I dare to admit, doing things differently than they “were done there”, not sure if and how fast I can show that it actually provides some value…

It’s so much easier to disconnect and just “do as you’re told”, hoping that somebody else will make sure we’re working on the right thing and get the requirements right. Once you get involved, you also assume more responsibility, it’s not “somebody else’s job” anymore. It’s also so much easier to focus on all the patterns, discuss superiority of repository over static CRUD-like objects or hold endless modelling sessions, looking for the perfect model. The trouble is all that doesn’t really matter if you work on the wrong thing. Doesn’t matter how perfect your model is, if you focused on some unimportant part of business and don’t understand the domain.

So be prepared. Quite likely, you’ll be a lot out of your comfort zone, doing things you have no idea about, pretending that you actually know what you’re doing, learning a lot from your own mistakes… There will be lots of them. Maybe some colleagues will get angry with you, they will tell you that what you’re doing is silly and a complete waste of time. Worst of all, they might be right, especially in the beginning. But you won’t learn all this just by reading a book and sitting in front of your screen all day. You can’t really learn all those things by doing your pet project, using some made-up requirements. You need to accept the discomfort and start learning those crazy things. The good news is, some day it won’t be that hard anymore.

Recommended materials:

“Influencers” great material if you want to lead the change in your organization, even if it’s “just” promoting automated tests and clean code

“Crucial conversations” absolute must-have on difficult conversations with high impact

“Crucial accountability” on how to keep each other accountable without ruining the relationship and being a jerk

“Succeed” a great book on goal setting and motivation (with the silliest title ever:)), solid research, no usual motivational crap, pure science and practical advice

“Inspiring Leadership through Emotional Intelligence” a Coursera course that proves that Emotional Intelligence can be learned and improved, fantastic material on how to listen better and how to avoid burnout

Evans got it wrong… or did he?

1 Reply

Recently I attended a local DDD meetup. I think the idea is slowly reaching its maturity, as after many “How to get started”, we’re moving slowly towards “How I/we/you got it wrong” kind of talks. It’s fantastic that people are openly discussing their mistakes and share lessons learned, but there’s something that really troubles me. It’s a growing DDD mythology.

Lenin is attributed with saying that a lie told often enough becomes the truth. Yes, this Lenin. Apparently he knew what he was talking about. We have quite a few of those lies already repeated over and over by well meaning people. I believe they’re trying to help, but unfortunately they didn’t check their facts before passing them on.

For example, if you’re interested in DDD, sooner or later you’ll hear that Evans got it wrong, that the order of chapters in his book is suboptimal. Some will even advise you to read the book in a specific, right order (skip chapters on tactical design and read them at the end).

People say that due to this silly error, some devs have done harm. Poor chaps got so excited with all the new ideas, that they decided to change the world right after chapter 7 (or sooner!). Before we get sidetracked into discussion on how is it even possible, that professional adults make serious decisions based on one book they didn’t even finish reading, go ahead, look at the table of contents. Do it now! It’s free.

Anything interesting?

Did you notice that tactical patterns are described only as far as part II?

Then go back to the sample content. Read it. Or if you’re in a hurry, go straight to the page 4 and scroll to the last two sentences. There. Read three sentences. Only three, I promise. The last two on page 4, and the first one on page 5.

“Technical people enjoy quantifiable problems that exercise their technical skills. Domain work is messy and demands a lot of complicated new knowledge [!!!] that doesn’t seem to add to a computer scientist’s capabilities. Instead, the technical talent goes to work on elaborate frameworks, trying to solve domain problems with technology [!!!].”*

Now let it sink in your mind.

The next time you hear somebody saying that Evans got it wrong, please, do world a favour and tell them to read the book in order. Tell them to stop reading at chapter 3, if they don’t feel like they can make it to the end. Tell them to start with the last two sentences on page 4 and the first one on page 5. Even if that’s all they’ll read about DDD, it’s worth it.

Hopefully that will save us a few “How I got DDD wrong” talks. Or if we’ll have them, then maybe at least they’ll be based on facts, rather than myths slowly becoming the truth. Because Eric nailed it, right from the very beginning.

* Emphasis is mine.

Microservices – Fred George’s style

2 Replies

It seems like everybody is talking about Microservices nowadays. People cannot agree on whether it’s just good old SOA done right or a significant mental shift, many are afraid that this is not a safe approach for most teams and the hype might cause disasters. Many discussions regard semantics and basic building blocks. People try to put Microservices in some box and say “here, this is how you do it”. I think this is not very useful.

Last December I attended YOW Conference. There were a few presentations of people that have experience with Microservices in production. What I found most interesting is that every single one of them had different approach. There’s no one-size-fits-all and it’s a really good thing. I also attended a workshop run by Fred George, whose experience with Microservices spans across at least 3 different companies and over 10 years. The approach he promotes seems to go against many familiar “good practices”, yet, I think it’s fascinating and valuable.

At the workshop we were working on functionality that picked the best offers for the given customer (let’s say there were rental cars specials). There was a central, global offers repository, there were also local, regional offers, there were rules that specified which offers are available for specific demographic or customer type (e.g. only for those with golden status), we also had some data specifying past behaviour of the customers, etc. Each part of the system had a separate storage mechanism and was completely independent of the others. They communicated via a single RabbitMQ channel. At the final stage another service took all offers suggestions for the given customer and picked a few that were the best match (the ranking was based on probability of promotion being used and potential profit). For more detailed description of the problem check out slides from Fred’s presentation.

We were given implementation of the first service. Our task, as a group of about 5-6 devs, was to add more of them. At first I wasn’t fully aware how much what I already know gets in the way. My first inclination was to create new channels, organize messages in topics, we wanted to have various message types, smart routing and all this stuff… Those are basics, right, that can’t be over-engineering. It will be easier to extend… Well, but we weren’t even sure whether there will be any extensions. Ouch… Turns out all that stuff wasn’t necessary at all.

The approach Fred proposed was so simple that it was hard to believe. It was really dumb and wasteful, it felt so wrong… In short: the original message was broadcasted to all services, each of them had a kind of built-in filter (usually one or two if statements) that decided whether the service should even bother to do anything (in practice most of the received messages were ignored), then service modified incoming message by adding some extra information to it and… sent it back to the same channel it came from. Each service used only small piece of information available in the message (e.g. only 2 out of 10 properties). And… pretty much that was it.

It’s surprising how uncomfortable people felt with this approach. You could see the resistance, there were lots of questions, mostly focused on trying to come up with scenarios where this approach wouldn’t work. We were smarter than that. We knew there are lots of patterns and tools that could make this better (more scalable, sophisticated, elegant… you name it).

But in the end none of this was necessary. It worked. It was good enough. It was easy to understand and elegant (although looked much better in Ruby than in C#:)). We wouldn’t hit performance issues any time soon, even though each service received plenty of irrelevant messages. We could easily introduce more channels or topics if our system would become bigger, but that was a distant future. Each service was minimalistic, clean and contained all relevant information, thus the risk of introducing bugs was low. If I needed to modify (or create) a single service, I wouldn’t need to understand all the others, so the maintanance costs would be low too.

uSwitch succesfully uses this approach in production for a few years. It gave them unique business advantages. Apart from things discussed above I was shocked to learn that they got rid of basically all tests. Only a few years ago they were a poster child for succesful adoption of Specification by Example. Instead, they invested heavily in KPIs monitoring. Seems to be working way better for them. They’re ultra-agile, super-fast, experiment a lot and started making lots of money.

To wrap up – I’m really impressed with Fred’s take on Microservices. Although I still don’t feel fully comfortable with this approach, I see that it perfectly matches to what guys at uSwitch (and few other companies) are doing. Generally people reject the idea, they don’t think it’s “how things should be done”, if feels wrong, even if they agree that might be good for some scenarios they’re sure it wouldn’t work in their specific setting. That might be true. It surely doesn’t fit every context (e.g. trading company could go bankrupt before getting any KPIs for the day). But I suspect that if we thought hard enough about our hidden assumptions we might discover that it fits more contexts than we would expect at first.

I’m not sure wheter I’ll be able to use this exact approach any time soon, but I’ve learned a few things that can be easily applied to any setting:

Even though I’m rather careful in this area, I still over-engineer. I jump to familiar solutions and patterns, before asking myself whether I even need them. Doing the “simplest thing that works” sounds very easy, but is extremely hard to practice and requires a lot of effort as our knowledge and experience grows.
Life teaches us that developers are not very good in predicting future (see discussions on estimates), so usually we don’t know which piece of code will be extended and how. Simplicity and minimalism have higher chances of paying off than “building for extensibility” (a.k.a. adding pre-mature abstractions and unnecessary complexity).
“Business” is often willing to accept much more than we would expect. It’s good to start with determining what Udi Dahan calls “anti-requirements”, before thinking about solutions to expressed requirements. Business might be ok with losing some money occasionally or even occassionally suboptimal solutions, if they make more money faster overall. We might think we know what “anti-requirements” are, but until we ask we just limit ourselves (and very often we’re wrong anyway).
The same things that give us unique advantages tend to have unique costs. Sometimes technical limitations force us to rethink business requirements and change the way company operates. Just because something is not a “standard cost”, don’t assume that business won’t be willing to give it a try. Ask. Explain. Discuss. Let them do their business things, such as managing risks and costs or defining success criteria. You just keep asking and exposing hidden assumptions.
Some “technical” problems are in fact people and/or process problems. Replacing tests with KPIs monitoring might seem like an interesting technical challenge. But somebody had to decide what we measure and how. Somebody had to agree to the risk and sacrificing small, temporary loses for the sake of long-term speed, fexibility and profitability. A lot of people had to think out-of-the-box, and not all of them were developers. Lots of interesting, original solutions appear, when a diverse group works on them. I think we often don’t appreciate enough how valuable is to discuss our challenges with people who have a completely different perspective.

Can CRUD be good (enough)?

1 Reply

When it comes to CRUD, I generally I agree with Matthias Verraes – it’s a great anti-pattern. However, everything we do, needs to be put in context. There are no silver bullets or absolute truths in our industry and I realized there are situations where CRUD is simply… good enough.

Some time ago we started re-designing our e-commerce project. It’s a legacy solution (some mean people even say it’s “uber legacy”) and we hit a lot of issues when trying to introduce automated tests. On a high-level we basically had two layers – data access and UI, business logic was randomly divided between controllers and data access objects.

The solution seems rather obvious – gradually isolate logic from data access and UI, extract new layer (let’s call it “business logic” or “domain” layer). That’s far from easy, given the current state of the solution, but possible.

Since our team is made mostly of novices and expert beginners (at least with regard to writing testable code and design), we needed very specific rules to follow. After long discussions and gradual improvements, we ended up with this general recommendation from “design team”:

controllers only manipulate parameters and maybe map domain objects to view models, they delegate all interesting work to domain objects
domain objects do all the heavy lifting, by default we have domain service here plus its interface, repository plus its interface (so we can mock it) and whatever other domain objects we need
all our current database access code is hidden in repositories.

Overall it looks good. Not perfect, not particularly sophisticated, but simple and way better than what we have now. There’s only one small catch. Over time developers started following this pattern everywhere, without ever questioning what is its purpose and whether it makes sense at all in the given context.

Our solution comprises of two main parts – customer site and management site. Rougly 70-80% of the management site are simple forms used for editing values: create new element, edit another, delete. The most advanced bit of logic in those are simple validations (e.g. value requried, regex match).

To me it seems like a perfect scenario to deviate from the general design guidelines. There’s not much logic to extract. There’s not much to test, so one end-to-end test per scenario should be more than enough. The changes in this area are not often and very simple. But we end up with at least 5 different objects involved, all just passing parameters and delegating calls to the layer below. CRUD should do here just perfect, it’s good enough, way simpler and faster.

The only problem is now I’m met with questions such as: “Isn’t it bad?”, “Aren’t we supposed to isolate data access?”, “How are we going to test it?” (but what to test here exactly without a database?)…

Lessons learned:

Context is king.
Rules and guidelines are no substitute for thinking. They’re only thinking aids, much like useful stereotypes and habits, without which we would be constantly overwhelmed and not able to do much valuable work. But if you’re too rigid with them… Well, it’s not much better than not having them at all.
If people don’t understand the two lessons mentioned above, then it doesn’t matter how great your rules will be. It’s impossible to determine all edge cases and exceptions up front. It’s not personal, it doesn’t mean you’re not smart enough, it’s just how things work in this world.
More important than what and how you’re doing something, is why you’re doing it in the first place. Make sure the motivation is clear for everybody and overcommunicate it at every opportunity. Make sure everybody understands that on our way to achieving the holy why, we might try various hows and whats. That’s ok. Make sure you focus on results of your why, and not verifying hows and whats (e.g. since we wanted to have testable code, we should focus on tests, not having extra abstractions everywhere in the codebase).
Last but not least, there are no silver bullets in technology. Even when it comes to “obviously good” practices, there are contexts in which they are not useful. So keep an open mind and be prepared to be challenged by reality. Sometimes CRUD is just good enough. Even though in general it definitely is a great anti-pattern.

Performance myths and facts

Leave a reply

Recently I came across a few performance experts that turned upside down what little I knew about performance. I thought this is boring, this is low-level, this is better left to some crazy assembler gurus, who think microbenchmarking in the middle of the night is pure fun. Turns out I was wrong on many levels and I want to share the most important things I’ve learned as far.

The subject turned out to be surprisingly exciting and at the same time quite challenging, so I’ll keep writing about this as my learning progresses.

Myth #1. Performance is a trade-off between clean design and low-level ugliness

I’ve always thought about “highly optimized C#” as impossible to understand IL tinkering. According to Martin Thompson that approach might be appropriate for maybe 5% code we write. Most of the time we should simply rely on clean design rules such as single responsibility principle, high-cohesion, keeping our functions small, etc. The things that we really should invest in understanding though, are data structures and algorithms, their characteristics and how they are implemented. Understanding how their choice impacts performance is the single most useful thing we can learn. Martin talks a lot about Mechanical Sympathy and it turns out that it’s possible to explain why clean code is faster when we analyze how modern computers work, especially CPUs and GCs. This knowledge is also very useful when deciding what data structures we will use.

A bit different perspective came from Gil Tene, who made me realize how intelligent modern compilers are, especially JIT compilers (that we have both on JVM and .NET). Knowing how advanced compilers are, Gil advises to not waste our time for trying to compete with them. It’s much better to focus on making our code clean, readable and easy to maintain, because compiler will take care of making it optimized anyway (e.g. it’ll remove redundant variables, inline functions, etc.). Most of the simple “optimizations” turn out to be what compiler would do anyway, so it doesn’t make sense to sacrifice readability for this imaginary “performance improvement”. It might seem otherwise though, because in some settings (e.g. in VS by default) JIT compilation is turned off in debug mode. Of course, it’s useful to understand how all this works under the hoods, because we can realize which of our pseudo-optimizations are a clear waste of time. Besides, sometimes we might hit subtle issues when compiler tries to optimize our code too much.

Myth #2. Performance optimization is a combination of crazy tricks

Both Martin and Gil talk a lot about “behaving nicely”, being predictable, following best practices and understanding the assumptions underlying the platforms and tools we’re using. Lots of super-smart people work on making our tools intelligent, force them to do all the hard low-level work for us, so if we just understand how we’re supposed to use them, we should be allright most of the time. The crazy tricks might be sometimes useful, but unless we work on a real cutting-edge system probably we won’t need them often.

Again, a lot of those assumptions are related to clean design and simple fundamentals, for example:

– Methods inlining – .NET JIT compiler will inline our function if it’s below 32 bytes of IL (short functions!) and is not virtual (be careful with inheritance!) *;

– Garbage collection – in a “well-behaved” application your objects either live very short (0 and 1 generations) or live “forever” (2 generation). Contrary to what many developers believe, you can have lots of GCs and still have great performance, just make sure they’re not full collections (generation 2).

So before I get to those crazy tricks, that I can talk about over a beer to impress other devs, I’ll spend more time learning about .NET, especially with regard to garbage collection and JIT compilation. I’ll pay special attention to what design choices were made, why, what alternatives were dismissed, how all that impacts me and what expectations it sets for my applications (or in other words, what “well-behaved” application really means).

Myth #3. More threads = faster

I was really amazed when I first learned that all LMAX‘s business logic is executed in one thread. Michael Barker and Trisha Gee showed one of the reasons why it was designed this way. I didn’t realize that the cost of locks is that high and even though the specific numbers might be the result of some bug on MacOS (which Michael noted were very impressive), it left me with a lot of food for thought. Maybe the differences are not that high if bug is fixed, but still it’s not really obvious whether adding threads will make my application faster. What is sure though, is that it significantly complicates design, increases maintanance cost and can result in more bugs.

Right now we talk a lot about parallization, using multi-cores to the maximum, microservices… The hype is high and it’s very natural to pay attention to new, shiny silver bullets. However, it’s useful to keep in mind that most of the things done in really cutting-edge systems, are way more than we need in the old good BLOBAs. Chances are we don’t have to be “as good as Netflix”, simply because our system doesn’t need it. We can’t forget that increasing complexity of our solution significantly increases the cost of maintanance, training people, etc.

While most of the world started talking a lot about how much we really, ReAlLy, REALLY need multi-threaded systems to survive in the current market, the amazing team at LMAX used the completely not fancy approach – they focused on their specific requirements, measurements, architecture, data structures, using known patterns in a fresh way and came up with an amazing result. And they process 6 milions transactions per second on a single thread. Not sure about you, but seems that should be more than enough for what we need in the systems I work on at the moment.

Myth #4. First get your application working then ~~uglify~~ optimize

It’s been my experience that performance is treated as an after-thought. We only talk about performance when something bad happens, but not many teams really monitor it on regular basis or are even aware what is needed. Sometimes we have requirements that specify that system should handle specific number of requests per second under whatever load and this is the best I’ve seen as far. But as Gil Tene said many times, even when we have some “requirements”, very often they are either not precise enough, we fall prey to common misconceptions regarding measurement or we simply ignore “all the rest” that is not part of our current requirement (such as 99th percentile).

All performance experts agree that we have to start design with a clear idea what do we need. So it’s virtually impossible to design application without having precise expectations regarding the worst case scenarios and unacceptable behaviours and at least some rough estimates of important numbers (e.g. how many users we might expect, what are their expectations, what would happen if performance would be bad for a few people but most of the time it’s excellent, what metrics matter for us, how will we collect them). It’s also useful to determine at the very beginning what are the real limits and start formulating requirements from there (e.g. network latency, performance of other applications we integrate with, industry “best examples”).

Gil in his presentation noted that without having an understanding of what we really need, we use tools in inappropriate ways and waste time optimizing things that already are good enough or try to optimize them in a completely inappropriate way. Sometimes the good solution might be completely counterintuitive, such as to sacrifice performance, deliberately make it worse in some places, in order to guarantee that our system meets SLAs even in “exceptionally bad” moments. What I understood by that, is that performance is not a binary quality – our system’s performance is not just either good or bad. Moreover, there are no silver bullets and simple answers, what is good for one system and one team, doesn’t have to be good for another. In most cases there’s no one single way to improve performance, there’re plenty of them, each with their own advantages and problems. I need to understand their trade-offs and verify why my system doesn’t perform well. Maybe the problem is in architecture, maybe in data structure, maybe in a language construct, maybe the array doesn’t fit cache, maybe I need to tweak GC settings, introduce pooling for big objects or upgrade hardware. Very often I can solve the problem in multiple ways, each allows me to achieve my goal, but comes with different trade-offs.

All of this sounds very complicated and I think it really is in the beginnig. Just like everything in programming. There’s a lot of conflicting advice, experts disagree or find problems in standard approaches, there’s a lot of jargon or referring to concepts I don’t understand very well. Last but not least, every system is different and I can’t simply blindly follow somebody’s solution, without first understanding the problem. What is easy is to use a random tool, follow “Getting started” tutorial, without spending too much time trying to understand what I’m actually doing and whether my optimizations are even required. So for me the lesson is that it’s important to know your application, have data regarding expectations and requirements, but also constantly collect information about performance in production and in test environments. That sounds like a lot of work, but without it the best I can do is count on my luck.

I really recommend to see slides from Gil’s presentation, he explains the issue with performance requirements (slides 15-29) and discusses the common problem with measurement tools (slides 30-46).

Myth #5. Microbenchmarks are the most basic tool

I thought that the first thing I should learn about performance are microbenchmarks, how to create them correctly and when they are really useful. But it turns out they’re not that important. This article concerns Java, but summarizes nicely what I’ve read in a few different places.

Seems that although microbenchmarks might be valuable from time to time, it’s way less often than I expected. They are prone to errors due to JIT optimizations (which most likely will be different for production and test environments), hardware differences, unrepresentative inputs, etc. It’s still useful to understand how to construct a meaningful benchmark, but definitely it should be used with caution and only when we’re really sure we need them and nothing else would do.

* Note there’re more rules than I mentioned, but those are most obviously related to clean code recommendations.

Why should you facilitate Code Retreat?

Leave a reply

Less than a week ago was another Global Day of Code Retreat. Few thousand people in few hundred cities spend a whole day working on improving their skills. That part doesn’t seem strange anymore, devs are doing such things all the time.

But during after party one of the participants asked me why am I doing it? Why spend a lot of time and effort on something nobody pays you to do? Participants have obvious benefit – they get better developers, they learn. Some openly admit that they treat it as a free training. But what about organizers and facilitators?

That question caught me off guard. I just wanted to organize this Code Reatreat for a while, even since I moved to New Zealand. But why?

1. Teaching (or facilitating) is a great learning tool

Some people think that teacher or trainer or even random CR facilitator has to be an expert and surely knows more than people who are “just” attending the event. That might be true, but more often it is not.

There is a good reason why it is said that if you want to truely learn something you should teach it to somebody else. Facilitators learn a lot during the day. Maybe sometimes even more than participants, because they have more time to observe and analyze approaches of multiple pairs. They see various approaches, compare them, think about them, try to come up with good questions.

We know there is more than one good way to solve a problem, but unless we have a big group of people working on the same task, we don’t realize how different ideas people have. Then we can discuss and compare multiple solutions, see their strenghts and weaknesses. This does not happen often at work, because we don’t give the same tasks to multiple groups.

Being a good facilitator is not easy, you have to practice asking smart questions – those kind of questions that guide, but don’t give away the answer. Also by trying to help people you have to organize your knowledge better. You are deepening your understanding of familiar concepts. You see them in a completely new contexts. You have to explain the same thing over and over, in various ways and by doing so, you discover what really matters, you get to the essence. In short – you deepen and distill your knowledge.

Apart from technical aspects, facilitators also have to learn basics of teaching, such as “crowd control” (make sure one person does not dominate the whole retro, try to encourage shy attendants to speak up), clearly explaining new concepts (might seem trivial unless you actually give it a try), providing encouragement, helping to get “unstuck” when somebody just sits there and stares at the screen, finding the middle ground between people “just doing it” and giving the answers away (also called proper guidance).

2. Moving the industry forward

Contrary to what is told, still too many developers seem to finish their education when they leave university. Some need to keep up to date with newest frameworks and buzzwords, but it still amazes me how many people in our industry never encoutered the basics such as proper testing or working in pairs. This is something we are not taught at the uni and not all companies pracitce it.

At every CR there are some people who never wrote tests or don’t feel comfortable with them (yet). It is really amazing that after just 5-6 sessions they say that it was easier than they expected. They discover the benefit of having more confidence in the implementation and are keen to introduce those concepts at work (or just practice it more diligently).

For some this is the first opportunity to try pair programming. Those people are very excited about the collaborative approach, they feel inspired by seeing in how many ways the same problem might be approached by different people, they discover that when they cooperate, the end result is way better.

Many participants commit to promote those two practices at their workplaces.

3. Bringing companies and communities together

In some cities companies are not eager to cooperate. Sometimes they say that if we work with X, they do not want to have anything to do with us. On the other hand, there are some companies that never got involved in community events, but if you ask them they are very keen to participate. Maybe nobody ever asked them, maybe they simply never thought about it. In such a case that might be the way to get them more involved in other activities as well.

For example one of our current sponsors insisted on booking few spots for people from their company. Now they are very keen to organize similar events. Since a few of their employees attended, they are in a very good position to make it happen.

The other thing I really like about Code Retreats is that they are technology agnostic. You meet people using C#, Java, Ruby, Clojure, Python, Haskell, JavaScript… Sometimes that means you just have a quick exposure to different paradigms and make a few interesting observations, but sometimes it inspires you to learn a completely new thing or become a member of another community group. It definitely broadens your horizons.

4. Spreading hunger

A while ago Steven Jobs gave a really great graduation speech. In it he encouraged people to “stay hungry, stay foolish”. I think this is a great state to live in. Always on the lookout to learn a few new things, to get better, to make world a better place. People do not always are lucky enough to be in a job where they have great role models, but they are very likely to meet fantastic people at event like CR.

Sometimes attending such an event opens up a completely new world for somebody. It turns out to be a first step in a very long journey towards mastery. It might be this tiny event that “awakes a hidden software craftsman within”.

Some time ago somebody inspired me and helped me to make that step. I never looked back. Now I want to give it back and share this experience with as many people as possible. I believe organizing CRs is one of the ways to make it happen.

P.S. If you’re thinking about organizing or facilitating the CR, but don’t know where to start, feel free to ask in comments or on Twitter. I’d be more than happy to help!

DDD Ultra-Lite

5 Replies

Some people say that DDD is heavy and expensive and as such must be used with care. Some teams adopt so called “DDD Lite”, which means they use “tactical patterns”, but don’t do much in terms of strategic design (or in plain English – they use software design patterns described by Eric Evans in the blue bible, but ignore the ideas that are considered truely important).

Truth be told, I’ve never worked on a “real DDD” project (yet!). However, DDD inspired me and I believe it made me a better developer. I used bits and pieces, most of them wouldn’t be recognized as “real DDD practices”, but I remember exactly where the inspiration came from.

I believe that the true value of what Eric Evans wrote years ago lies in the fact that virtually anybody can use some of the things he proposed. They don’t require any special tools, money, approval or reorganization. Those are simple things that you can start doing from next Monday (or tomorrow). I call those ideas “DDD Ultra-Lite”. Some of them are not new, but I believe Evans did a really good job of finding words and metaphors that stick and he certainly provoked a few AHA moments.

So here they are (in random order):

1. Listen to what business is saying

I’m not joking. The advice is probably as old as programming itself, but Evans has an interesting perspective.

He said that if business experts don’t understand your models or think they are awkward, then it means that you have to work more on that. Sometimes business people don’t openly oppose your idea, but you can see that they’re not very enthusiastic about it and that something bothers them, maybe they are confused or maybe they simply say that it doesn’t feel right.

There’s the opposite situation too. Sometimes when you finally find a good model and you communicate your idea to business expert he looks at you as you were the stupidest person in the world. Yeah, you are right, that’s obvious, what took you so long to understand this simple concept? That indicates that you finally got it right.

2. Do just enough design

Both over- and under-engineering are problems. Finding the balance between deep design and just “shipping features” is not a trivial task. Evans gives us a few tips that I found very useful:

Use the simplest possible model that allows for solving current problem. Don’t try to model reality, you will fail.
Remember about clean code, make your code and design easy to change and improve in the future. If your project is successful then you will change your design many, many, many times. As such, you don’t have to get it perfect first time (and probably even can’t).
If code starts getting in the way and you notice that making changes becomes more and more difficult, it might be a sign that your problem has outgrown the current model. Time for re-design of this piece.
Redesign small pieces of the system as need arises. Again, strive for the simplest model that solves the new problem.
Collaborate. Go for a coffee with a group of devs and/or architects. Discuss your problem with them, brainstorm, sleep it over and go back for another session.
Don’t make it heavier than it needs to be. Evans suggests multiple iterations, short discussions (1-1,5h), lots of learning and humility.

3. Ask powerful questions

Sometimes I work on a new feature or modify the existing one and the design starts getting ugly. It was just fine right before this change, but suddenly it becomes very complicated or just looks bad. This is a sign that the design should be changed, nothing special there. However, in Evans book I found a couple of questions that helped me on few occasions: Am I missing some domain concept? Does this change introduce or uses a concept that doesn’t match the existing model?

In other words instead of adding a few more if-s here and extra helper class there, maybe it’s time to revisit the design and find something more robust, more general or simply different. Something that explicitly caters for this new concept that just crashed our beautiful model. This is especially true if you know that there are more upcoming changes in this area or you work on a piece of code that is modified frequently.

4. Don’t reinvent the wheel

Evans encourages seeking external inspiration multiple times in the book. He mentiones browsing through patterns calatogs, to find out whether we can use one of them (possibly with modifications). He encourages investigating how similar problems are solved in other contexts (e.g. in other industries).

What still surprises me is that I haven’t experienced this approach in many places. I know some people look for solutions on the internet, browse through blogs, watch presentations from conferences, but as far I haven’t heard anybody say “I read it in XYZ book”. Usually the “design” sessions I attended were pretty much random brainstorming meetings, few people even prepare for them or know what exactly the problem is (e.g. has numbers describing non-functional requirements). Then months later we come across some random presentation and it turns out that we could have used something else and avoid a few problems.

I know that people talking at conferences and scientists make extensive use of solutions used in other industries. They try to find common issues and adapt solutions to specific needs. However, I think that in “real-life” very often we rely on personal experiences in other industry (e.g. in previous job) rather than extensive research.

The improvement I personally made is to make the “research” phase more organized and extensive and to revisit some “classics”. Blogs and presentations are extremely valuable, don’t get me wrong, but books tend to be better organized and contain more time-proved material. Not to mention that some of the “brilliant” articles are just plain wrong and relying on random Google search for unknown subject is not a good idea. If it’s written by a respected expert that should be all right most of the time. But even experts disagree and don’t always explain their unique context when describing solution.

If I remember something that might be useful I go back to the source of this information to refresh my memory. I try to find related articles, books, presentations or case studies. If I absolutely have no idea where to begin, then there are a few alternatives that proved useful for me: going back to the theory (I’m still surprised how valuable that is), browsing through the catalog (e.g. design or analysis patterns), while focusing on context and problem descriptions or finding a good “domain guide” for IT professionals for a specific industry.

That might sound trivial but I encourage you to really try it “as you meant it”. You might be surprised (I definitely was:)).

Legacy Code Retreat

Leave a reply

Recently we (together with Seb) organized a Code Retreat event in our company. It was not the typical event you see during Global Day of Code Retreat, for a few reasons. First of all, it was internal, only people from our company were present. Therefore, the diversity of tools and languages was not as high as usually. Then, it was organized during work hours. Finally, it was a Legacy variant.

What exactly was the point?

Some people might question, whether organizing internal company event of this type makes any sense. After all, one of the fundamental rules of retreats is attending them in your free time, only because you strive to improve your skills, not because your employer requested it. In our case there were a few reasons for choosing that format:

We knew there are a few skills that devs from our company needed to improve (refactoring, following simple design rules, writing tests and testable code, etc.). We knew exactly what issues come up in various projects and decided that getting familiar with those basics is the first step towards making our codebase better.
We didn’t have any training budget, therefore we couldn’t hope for getting a professional trainer. Even if we did, probably it wouldn’t be worth spending that money for learning the basics we intedned to cover.
Many people work long hours, so asking them to do something extra in their free time seemed a bit too much. Even then, getting some people out of a project even for one day turned out to be extremely difficult, so we needed an official, approved event to allow people to spend some time learning.
We wanted to provide opportunity for people to practice new skills. Making a presentation, organizing a “Lunch and learn” or anything of that kind, for sure wouldn’t work for us.

What did we do?

We’ve got source code for JBrains’ Trivia game from here. We paired with Seb for one hour to get GoldenMaster done, in case some people don’t finish it during first session (proved to be useful:)).
We organized a short intro to Git a day earlier (especially useful for short refactorings session).
We run sessions, each 45 min., followed by a retrospective and break. Since retros between sessions were very short, not everybody had to answer questions.
In the end we asked every person to share what they learned, which session they liked the most and what will change in their work from next Monday.
After that we hit the pub for a quick beer:)

How did it go?

We were surprised and very glad when people said how many new things they learned from each other. Without any suggestions on our side, a few people said exactly the things we wanted to address, regarding changes in how they would work from now on. People also got very excited about what they learned, some encountered new tools that their colleagues were using, seems like the event left them hungry for more (which was one of our two main goals).

Of course, work is much harder than what we do during reatreat, in reatreat we don’t have databases, codebase is tiny, we don’t worry about deadlines and pressure from management. So I’ll follow up on this, want to see whether people managed to implement those new ideas at work and what challenges did they hit.

Detailed description of sessions:

Session 1 : Golden Master – Create a test for the game, changing as little code as possible and covering the largest possible number of cases (challenge – change only one line). Keep the tests and use them in later sessions.

Retrospective questions:

How much did you change?
How many test cases did you cover?
What challenges did you hit? How did you deal with them?

Session 2: “Make the code better” – Using whatever techniques you’re familiar with make the code look better (whatever that means to you)

What do you think about this codebase? What problems did you hit?
How did you change code (e.g. extracting methods, extracting classes, removing duplicates)?
How many tests did you add?

Session 3: Pure functions – Extract methods that do not have any side effects and state.

Retrospective questions:

How many methods did you extract?
What methods did you extract?
How many tests did you add?

Session 4: Single responsibility principle – Make sure each class has a single responsibility.

Retrospective questions:

How many classes did you extract?
What classes did you extract?
How many tests did you add?

Session 5: Short refactorings – There’s an imposed 3 min. limit for refactoring and covering your change with a test. If you don’t manage to finish before time elapses, revert your changes.

Retrospective questions:

How many refactorings did you manage to finish?
What did you learn in this exercise?
Did you use Resharper and keyboard shortcuts?

Session 6: No mouse or short methods or eliminate conditionals – Pick one of the three tasks (or combine them:)): either do not use mouse or eliminate conditional statements from your code (ifs, switches, loops) or make your methods shorter than x lines.

Retrospective questions:

Which task did you choose?
How many conditionals did you eliminate?
What techniques did you use (popular ones are using dictionaries or modulo function)?
How many methods did you add?
In the end, how many of your methods were longer than x lines?
How many tests did you write?

Specification By Example applied

Leave a reply

Humble beginnings

This was a very ordinary day, ordinary meeting, when one of my colleagues came up with a proposition of writing tests using Specflow. If I remember correctly, he watched a presentation from NDC and the idea appealed to him for some reason. We decided to give it a try and write tests for one of the old modules in our system.

We’ve already had some existing tests in this module, but we wanted to increase the coverage. Since we didn’t start from scratch, Konrad managed to get the first tests running pretty quickly. Few days later he was showing us how to get started. Some time later we decided that we could use our slack time for writing Specflow tests.

A few months later I was thinking about making a presentation at a local .NET user group. I mentioned it to my team-lead in our one-on-one meeting. He suggested that I could investigate the concept of Specification By Example a bit deeper and share our practical experiences. I read a few books, I made a presentation, then another, the idea stayed with me since then for good and I still keep learning about it.

What’s the big deal?

Soon enough we started noticing benefits. The separation of abstraction levels (pseudo-natural language vs code) influenced the way we were thinking about the problems under test. We started discovering more and more inconsistencies. We documented them and discussed with client. We asked more questions than ever. After a while we questioned everything, just on principle.

Then we applied the concept to other areas, where we didn’t introduce Specflow and didn’t even want to. When requirements were unclear, we used examples and tables, we used natural language to provide specific scenarios, making sure that we and our business experts were on the same page. The side effect of this was that we started focusing more on the “why” part, on the big picture of the problem. We learned that some of our assumptions are incorrect, that we automatically add constraints that our experts didn’t came up with (e.g. they were ok with some kinds of temporal data incosistency, which seemed crazy for us), that sometimes we use the same words to describe different concepts (apparently “future date” sometimes might be equal to “last week”).

The scenarios provide a very neat framework for communication with business users. Our experts were in fact quite “techy”, which made things more difficult. Sometimes they used technical concepts in a different way than we. Sometimes they had “how” ready, before we even knew “what” we are supposed to do. Scenarios were a good entry point for getting conversations started. And those conversations were different from those we had before. They naturally kept us communicating on higher abstraction levels, talking about business, not ints and database indices. Thinking about the minimum set of key examples helped us spot potential issues even before we started implementation. We caught a few incosistencies in business rules without writing a single line of code.

We got it all wrong! Yet… it worked

I find it ironic, that when I started digging deeper I learned that we got it all wrong, we made all the possible mistakes, started from the completely false assumptions… and yet, it brought us a lot of value. How was that possible?

First of all, we had the “idea” champion. Konrad wored very hard to make it work, he started the whole process, provided some initial examples and occasional tweaks later, he tutored other people on the team how to use it. I think that if that idea “was given” to us from the manager or some external team, we would not care that much. But it was ours. Konrad wanted it to succeed and we wanted him to succeed in his efforts. So we tried hard to make it work.

Then, the tool itself was interesting. It was something new, something different, so a few of us were actually happy to spend our slack time writing more tests (by “us” I mean developers!). That meant that more tests were written, then would be if we used, say, NUnit for the same thing.

Our Specflow scenarios were far from ideal. We tested on the lower abstraction level than we should. We used too many examples. We didn’t share our scenarios with clients (sometimes we copied them to email, but they didn’t know it’s anything more than plain text). Tests organization probably could be better.

So we might say that even if we didn’t fail from the technical point of view, from the point of proper tool usage, we didn’t have much success with it either. But that wasn’t that important. We still could get the benefits the tool brings. I think that in the end we could just throw away all the tests and never use Specflow again, that wouldn’t matter that much. Even then we’d get some value from getting familiar with the tool.

To me the main benefit was that we learned to work differently in the process, now we don’t need tools and frameworks anymore to get the conversations started. I really experienced that I, as a developer, am also responsible for requirements and understanding the busines side of things. That I can (and should!) ask questions, clarify, propose alternatives and warn about difficulties as early as possible.

Of course, we could use a different tool or not use any tool at all. Also, there are other benefits to BDD or Specifications (I’ll write more about it in a separate post). But I think it’s pretty cool that we didn’t have to fight for budget, attend any formal trainings or look for communication experts, to learn all those things. If it worked for us, I guess it can also work that way for other teams.