Three levels of maturity – The invalidation maturity model

Sharing: facebook, twitter

The validation maturity model

Opening note: This blog post is already nearly two years old and out of any or no reason, I never published it. (Well, I actually thought it is too complex and not quite approachable.) Anyways – Marc Burgauer had a huge role in reviewing it and then bringing input into it. So, I found it unfair and by now publishing it, I wanted to appreciate all the work and input that went into it from Marc. Consider his 50% the better and clarifying part.

Levels of Validation and three levels of maturity

I observed that organizations seem to have quite different maturity regarding the level on which they validate or invalidate assumptions. This becomes more apparent the more methods of validation like Lean Startup enter business life. Categorizing maturity along these levels has interesting effects that explain certain aspects of organizational behavior, rooted in culture and values. The model I will talk about is quite implicit and seems to be a blind spot in most organizations – hidden in plain sight. It took me some time to realize it and find a way to make it explicit and communicate it. What makes this model important is that it relates practices (what do we do) to maturity and resulting capabilities (how do we know why, what and how we do things?)

The model itself is quite simple: Companies, it seems, come in three maturity levels:

  • Maturity level 0: Doing things, no validation, gut feel, intuition driven
  • Maturity level 1: Solutions are (in)validated
  • Maturity level 2: Problems are (in)validated

Let’s go through the levels one at a time. While we do so: Let’s not be judgmental. While the maturities are ordered zero to 2, this does not mean a company on Level 0 is unsuccessful or dumb. The company may indeed be quite successful. The risk is more in the dimension of resilience. We think maturity level 2 really makes companies a learning organization and thus much more resilient towards hostile changes in their complex environment.

Maturity Level 0 – Doing things

A Level 0 company is doing things. This means, the company is not very conscious about which things it is doing or why. Lots of companies on Level 0 are doing well because they already have perfected their business model in the past. But such a company has only limited capabilities in steering and strategy. The company has no self awareness. When things go wrong in their environment, things will go wrong on the inside.

For startups, Level 0 means they will be doomed. The very role of a startup is to consciously and quickly iterate through many problem and solution hypotheses. If this is not done consciously, the company is gambling. While they might get lucky with one or the other idea, in general, they don’t know why they are doing things. The reason for this might be rooted in the social system of the startup such as a rhetorically (or charismatically) strong, but methodologically weak founder.

In Level 0 companies we see that they are attracted to methods but don’t want to invest the work to “get” the methods, they think they ‘know’ what to do anyway. When we have a workshop in such an environment, they know the outcome right from the start. Clients brought no new insights, they just validated the existing bias. Level 0 environments also believe in Genius product managers etc. that just have the right insight anyway. Again, this does not mean Level 0 means death, but death is more probable and sooner to be expected.

The key problem with Level 0 is that orgs living here have no conscious, explicit model of what they are doing. Consequences are that Level 0 orgs cannot know why things work or do not work. They also cannot acquire knowledge on what works and what does not. Worse, they also have no model or framework in which to digest feedback. And without being able to reflect feedback, a company is basically not self-conscious. The company does not know what it is doing, it is blindly iterating. It is driven by intuition, which is a) not an economical approach and b) very risky. In essence, Level 0 orgs are gambling, throwing the dice.

If you don’t know what you are doing and don’t understand why it works, you can’t scale. Also, if boundary conditions change, you cannot consciously react, because, again: You don’t understand, as you don’t have a model. So, companies on level zero are unstable towards change from the outside. Level 0 orgs are a fantastic target for disruption.

Maturity Level 1 – (In)validating solutions

Maturity 1 environments dare to question their solutions. We call this “(in)validating solutions”. Many of the Level 1 environments I observed developed from a Level 0 environment through the deployment of agile values and methods. This leads to questioning, “what we are doing” and a higher level of discussion between business and technology.

The rising level of curiosity and having had a taste from the tree of knowledge leads to all sorts and ways of doubt in proposed solutions: Does the product owner really know the client, did he really track a user down for an interview or is it all opinionism. First, opinionism is countered with opinionism, leading to arguments like “I am sure he wants X”, “No, I think he wants Y”, “I asked my wife and she immediately said F” and so on.

After a while, consensus arrives that UX testing is a good thing. In some companies even the boss or former Genius gives in to learning and opens up to his genius solutions being tested and finally (in)validated.

There are 2 dangerous side effects of Level 1:

  • Starting with (in)validation on the solution level is still late. Late invalidations are expensive. If we want to invalidate, then as early as possible, before developing the solution.
  • On a social, cultural level it bears the danger that the Genius, being open to (in-) validation, stops here and does either not see the next level and opportunity for improvement or is afraid of losing Genius by extending (in)validation to the problem state.

Also, the pain of change while going to Level 1 is immediately tangible, while the benefits are so far ahead in the future and intangible, that slide-back into Level 0 can happen.

I observed highly agile environments, thinking they are already giving their best (which they do on the perspective of doing things better!) but are blind to the fact that they only do so on the solution level.

On this level, the worst can still happen: The boss or Genius can still take a shower in the morning, thinking he has a great idea, comes into the company, and even though his solution is challenged and iterated on, the company will still end up with a product no one needs – simply because the problem addressed is irrelevant. This is especially dangerous when other employees are already further developed and try to ask if this is real problem. Limiting their curiosity leads to the worst kind of disengagement – they no longer care.. Having new ideas is not to be mistaken with what will be described as Level 2 – everyone and every org has more than enough ideas. They just don’t (in)validate.

Level 1 means also the entry of models (= consciousness) into the daily life. Level 1 is the land of efficiency (Doing things right).

The real danger of Level 1 orgs is their dangerous partial knowledge, as their mental model is still quite limited. They go half the way of challenging this model. In terms of situational leadership, they are the talented but dangerous breed of highly motivated, willing, unable. The risk is that they already work on a model and think that it is complete. But: Only iterating on solutions will still end up in solving the wrong problems. This leads to instability and a high risk of being disrupted by external changes.  Level 1 is the dangerous state of advanced beginners.

Maturity Level 2 – (In)validating problems

Level 2 extends Level 1 by (in)validating already on the earliest stage of problems. Already the hunch of a potentially relevant customer problem gets questioned and (in)validated before investment in potential solutions starts. Our solution minded culture is not used to this and this is what makes Level 2 so rare. It hurts. And leads to highly non-linear discovery processes.

Level 2 marks the separation between efficiency and effectiveness. A Level 2 company only wants to solve the most relevant customer problems and tracks them down aggressively (do the right thing).  The economical benefit is that money will not be spent on products that no one needs. Level 2 brings with it a built-in guarantee for effectiveness rather than just effectiveness. Peter Drucker will love Level 2 orgs, but more importantly also their customers and environment The challenge for leadership here is much harder. Nothing will be taken for granted, everything be questioned. But this is just the beginning: The managers role will change towards encouraging a culture where every assumption will be questioned and, after invalidation, smashed to pieces only to be replaced by real findings of autonomous teams. The next step will then be that ideas and problems being initiated and stated by Management will become a bottleneck. In future, the teams will be empowered to do the necessary research, synthesis and analysis to come up with the next new problems to be (in)validated and then abandoned or solved. Here, many necessary ingredients for a self-conscious, successful company come into the picture: Diverse, T-shaped, multi-disciplinary teams, aligned in following a coherent vision, and finally empowered through trust from the leadership, which is based in the self consciousness earned through being a Level 2 environment. Finally, the long journey pays off!

Level 2 means a truly learning organization. (In)validating problems has customer empathy  built-in, as most of your problems can only be validated through heavy client contact. In the bigger picture, Level 2 ensures learning on all levels.

There are simple things one can do as a leader to support this: Simple sentences like “what would we have achieved once we solved that?” can have great effects. They make clear, that Management itself is thinking on the level of questioning their own assumptions.

The model is fractal

An interesting aspect of the maturity model is that it is fractal. It works on every organizational level from team over department, company and finally the whole enterprise. This also means, that you can observe different maturity levels within one organization-.

Some Observations

Some observations along this model:Orgs hire staff or onboard coaches that appear to be on their level. A Level 0 org is looking for a product guru, who – by intuition or experience – simply and somehow seems to know what’s right. Arguing on Level 2 in a Level 0 org will not get you a job – even though you might be right. A Level 0 org, when discussing an engagement with us, will ask for a quick solution, but not how to get there. They want to hire experience in solutions, but not experience in deriving those solutions. Deriving solutions is new and scary territory to them. Level 1 orgs will typically love experienced UX guys, testing and iterating on solutions. Level 2 orgs will embrace the knowledge of an experienced Lean UX or Lean Startup guy who really challenges the assumptions on the problem level. They are looking for combined expertise in theory and practice.

Also, orgs will only accept tools, techniques and methods that are one maturity level above them as a maximum. A hard core crash course on deriving, writing and (in)validating problem statements makes little sense in a Level 0 org. It may – for a limited time – serve as an eye opener. But it may also be regarded as esoteric, useless or it may simply be used to simply deepen existing biases. The same workshop, in a Level 2 org makes absolute sense in a Level 1 org. While it will be a hard time to question core assumptions or realizing that it is hard to formulate the very purpose of the company or what customer problems are solved, they often take the bait and use the workshop as a gateway drug to look into Level 2 practices and values. Coaches and orgs should take care if they are working and ‘ordering’ on the right level.

Empathy for the customer will require orgs to reach Level 2, but also practicing Level 2 will bring empathy for the customer and with it effectiveness.

A hint to getting there: Ritual Dissent

The basic element that is preventing orgs from climbing the ladder from Level 0 to Level 2 is a lack of open feedback loops. Candor is not easy to digest. Especially in organizations on Level 0, where leaders are leaders because they have the ideas, it is not easy to challenge those ideas against the hierarchy or chain of command. Negative feedback will be seen as personal criticism. To get out of this trap, it is important to install feedback channels that depersonalize the feedback and give the employees an opportunity to give open feedback on the issue that is not taken as personal and possibly hurts the vanity of the leaders, but that is really seen to be feedback on the issue at hand.

The more creative work gets, the more such candor is necessary. So. Lets have a look at how a creative company like Pixar is handling feedback. Pixar has the metaphor of the film to be created being the patient. And no single person and her vanity in Pixar is more important than protecting the sanity of the patient. The patient needs to be protected and helped at all cost. So all feedback relates to saving the patient rather than personal ideas on saving the patient. This is even more important, as of course, in the beginning, shortly after having the first sketches of a movie after staring on a blank page, every director and writer and director at Pixar is tense and nervous. It is a very instable situation for everyone involved. But it is essential that feedback will be listened to and digested. On the other hand, it is important that those doing the work keep a clear mind and are protected.

At Pixar, they have a feedback mechanism called the BrainTrust, which meets regularly and reviews the prototypes of movies being worked on and gives feedback on them. The feedback is mostly on what does not work in the storylines or what does not yet feel authentic. So the feedback is mostly on problems, not solutions. The next important ingredient is that while the BrainTrust is composed of all kinds of potentially intimidating persons from Pixar, high up in hierarchy and merit, including John Lasseter. The point is, though, that these people created a culture of candor, openness, and established the value of protecting the movie. Also, they have no operational lever on the people involved. They are not giving solution hints and they are not checking if a certain problem was solved or how that was done. They simply watch the next iteration of the movie and give new feedback notes. The BrainTrust is simply a high potential feedback mechanism. The team then works with all kinds of feedback notes after filtering and prioritizing them. The team knows best that the movie is not yet working.

Establishing a culture like that at Pixar takes ages. A small start might be to include a round of Ritual Dissent in some of your activities. Ritual dissent means to offer a forum for open feedback on an issue (not on people or their opinions). I have learned this practice from Will Evans, who set into practice a concept described by Complexity Thinker Dave Snowden. In our Design Studios and other activities, we now include rounds of ritual dissent, in which we critique outcomes of creative processes. As there is a clearly identified spot in the process, where it is made clear that critique will arise, everybody knows what is coming and the stage is set in a way that is not taken personally.

It shows that these rounds of feedback often bring up what is rumored in water cooler discussions but is never being brought up in the open and channeled to where it belongs – to where decisions are made. This kind of feedback shows huge value and, taken seriously, will stop many useless approaches early on. These feedback rounds deliver nearly as much value in our observation as feedback from the outside. Also, and this is why we mention them, they are a good practice for open feedback, which again is the essence of getting from Level 0 to Level 1 and then off to Level 2.

As a final question: Where do you think your organization lives in this maturity model?



Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>