The difficulties IT companies face in achieving excellence

LIB: Mary, could you introduce yourself to our readers?

Mary: I started my career as a programmer. I worked for 20 years at the University of Wisconsin dealing with process control equipment. Later, I became an IT manager in videotape manufacturing, where I took part in the transformation from the classic manufacturing system to just-in-time. We decided to adopt a pull system, invented by ourselves at the time, to stay competitive against the Japanese market. Eventually, I left 3M and decided to look at the market. I decided to take part in a government software project called “Waterfall”, which was a disaster. Then, around 2001 or 2002, I began studying lean in software development, Tom joined me, and we wrote our first book, “Lean Software Development”. The idea of the book was to describe why the project's idea had not been very good and how lean applies to software development. Since then, we have written three more books.

LIB: What benefits have lean principles and techniques brought to software engineering?

Mary: In any area where you develop something new, you do not apply lean as in manufacturing; you need, however, to learn. So, in product development in general, and particularly in software development, I have to come up with some solution to a problem and, as a software engineer, I have to think about what problem we need to solve. And if you look at the way lean was applied in software development, which only began in the late 1990s, it was a bit different from production: it has to do with going slowly, with deferring decisions until you have as much information as you can, it has to do with working in small batches and, basically, it has to do with information flows as opposed to development processes. And we need to make these information flows as smooth and fast as possible and constantly innovate. And it is also necessary to make sure we build something with quality. There is a custom in development, which we created some time ago, that says: “let's build our system and then set aside 30 or 40% of the development process to test it”. This is completely contrary to the concept of lean software development, which says: “let's build the software and test it all the time”. These kinds of principles are the ones that work well in software development.

LIB: In your opinion, how should an IT department or organization start applying lean thinking?

Mary: I prefer to talk about an IT organization rather than an IT department. And for a good reason: when I was an engineer, building process control systems, I was not part of an IT department. In fact, I have never worked in an IT department. And I am always puzzled by IT departments; I mean, I am not puzzled by software engineering in general, but by the fact that the IT department has terrible results. Consider the idea of technology development. Everything improved when executives and senior managers began to understand technology. You do not find companies like that until the late 1990s. One company we must mention is Amazon. Amazon's IT department is revolutionary. One thing you really need to know about lean is to understand the customer's problem, and that is not exactly what an IT department does; it thinks about its colleagues, who are also customers, but not the customers of the business. What needs to be done is to understand the business problem, the problem of the business's customer, instead of thinking of your colleagues in the organization as your customers.

Tom: One thing that happens in an IT department is that people are supposed to do what they are told to do, and that is not a very exciting role. The emerging trend is that people with competence in technical areas become part of something called the “key to all the statistics”, which is responsible for the entire customer experience, everything the product delivers to the end customer; this includes people who understand the market, the technology, delivery, sales marketing, those elements. If everyone works together to deliver value to the external customer – and the company keeps a share of it – that is the paradigm that dominates today in what we call IT.

Mary: There are some CIOs who understand this, and they are taking their companies down the right path to improve the business itself and the customer experience.

LIB: What difficulties are encountered when we apply lean in IT?

Mary: Let's talk about product development rather than IT, because I think of this company that has technical capability, which may or may not be called an IT department. If you want to apply lean principles, there are many aspects of lean that are counterintuitive. We used to think that to have high quality, we had to be very slow and deliberate, to give ourselves time to discover what the defects in our software were at the end of the process. If you want to apply lean thinking to software development, you will realize that there is something wrong with that concept. Why do we look for defects only at the end of the process, instead of looking for them as we develop the software? The thing about lean, at least as far as software engineering is concerned, is that we need to automate everything from the start so that we can test the software as we develop it. The second counterintuitive thing is deferring decisions as long as possible. We need to create software that is changeable and that allows testing immediately after changes. So we did not need to know everything right at the start, but we deferred decisions as long as possible. These concepts of ease of change, of running experiments instead of writing requirements, and of discovering what is useful as the process evolves instead of knowing everything right at the beginning are difficult for people who have spent their whole careers in different processes.

LIB: Speaking of your presentation at the Lean IT Summit, can you tell us what you will talk about?

Mary: We believe it is about the difference between flow efficiency and resource efficiency. There is a book called “This is Lean”, which is one of the best sellers in Sweden. It is not a book about software, but about business. It explains that what it does has to be about ensuring efficiency – you do not define efficiency by making sure everyone is busy; all those people with so much work to do is not efficiency. If you look at lean, it has a very clear notion of what flow is. Lean understands that large batches create a lot of variation. Let's use a highway as an analogy: if you pack a highway with cars, what happens? Traffic stops. So this concept of creating congestion by keeping everyone busy is something managers do not seem to understand; they say: “let's load everyone up with work; that way, we will make better use of our resources”. If you overload people with things to do, the consequence is stress, it is switching from one activity to another; it is much better if you have the right amount of things to do at the right time. In software development – or in any other process –, if you look at the slow unit and maximize the speed at which it moves through the flow, that is much better than keeping everyone busy. Think of a hospital: do we want to make sure that all the doctors, all the X-ray machines, all the expensive equipment are always busy while patients wait, or do we want to take a patient from the emergency room and move them through the system as fast as possible? In lean, you see the patient as the unit of flow; in the other paradigm, you see the doctor, who has to be busy all the time. In lean, we say: “we will have the best flow, the best use of all our resources if we do not constantly have long queues and large batches, if we can move the patient through the system as fast as possible”. You should look at the patient's problem and solve it as quickly as possible, instead of making sure everyone is busy. You should look at the unit of flow, at the customer's problem. And when you do that, you get higher quality and better solutions.

Tom: The concept of flow is right at the center of lean thinking. There has been an interesting convergence recently, in the last few years, between lean thinking and design thinking. Designers do not talk about flow; they talk about friction, which is anti-flow. I think a very good alternative to talking about flow is identifying the sources of friction. The designer's goal, when designing a new product, is to identify where the friction is. What problem is the customer facing? Where is time being wasted? Where are resources being wasted? Finding potential solutions to these problems is much more likely with lean thinking than, for example, creating a hypothesis and testing it. That is the direction IT is heading in. If you go to technology conferences nowadays, you will notice that they are very standardized in terms of processes, practices, tools. The software market is shifting from large models to something like a system that enables a small group of people to deliver a service very quickly; they have to carefully find interfaces.