Will Management Succeed at Killing Agile?

If one believes the latest cover story from the leader of promoting management trends, The Harvard Business Review, Agile is both everywhere and can be used for everything. Considering the business that I am in, I should be pretty happy as well. But I’m not. HBR’s Agile at Scale looks more like a tombstone to me.

The HBR article is touting the latest management-fad version of Agile, Agile as Anything Management Wants. Their Agile is all about dozens of random organizational changes that are labeled “Agile,” and are cited as proof of Agile’s greatness. Someone says they are doing or being Agile? Ipso facto, they must be.

Agile as Anything Management Wants is bad Agile, and it is not only destroying the good Agile, but is also destroying a moment in which Agile might have actually made things better…truly better.

There are actually many styles of Agile out there, but they are very different. There was one that emerged naturally through the 1980’s, which then exploded as a profoundly powerful management technique in 2001. This peak moment coincided with many learnings on organizations and human behavior from the previous 50 years.

In that moment, Agile shouted a battle-cry against the way we traditionally manage…and it won!

As I explain the other Agiles, I’m hoping you’ll see what I see: that most people don’t understand what Agile was in the first place — a very specific solution to a large and very rare problem — and in that misunderstanding it has been mis-applied over and over, with lackluster results. It has been further diluted in meaning by hyperbolic marketing and packaging, and finally now, its meaning lost, misunderstood and over-commercialized, it has fallen into the hands of managers and consultants, where it will be dealt its final blow, becoming a mere label for the practices it opposed in the first place.

Why care? If you own or manage a business there are three big reasons you should care: productivity, quality and retention. Because good Agile, what I’ll call Humanist Agile, really makes a difference. I’ve seen it in the wild — yes, you can do it yourself — and I’ve seen it in the 100+ clients whose companies I’ve helped to transform. Good Agile will not only make your organization better, but it will teach you to be a better manager and person. In the end, good Agile makes us better together.

Yes, I make a business of helping companies apply Agile, but this isn’t about that. I discovered something that I loved about myself and all of us when I finally discovered Agile. I’ve never believed in anything like I believe in the need to for us to save Agile and what it really means. I’m not sure that Agile is dead…yet. But I fear for its soul.

Here’s my journey.

Pre-Historic Agile

I never wanted to be a project manager. I was really good at coding software. But they asked me if I wanted to be a “PM” and said it paid like 30% more. It was 1982, I was in my 20’s living at the beach in LA, and 30% more money and not working late hours or weekend sounded pretty fine. But I really didn’t want to do what a PM had to do.

Within the next year, a good friend pointed me toward an IEEE article on Iterative Enhanced Project Management. Basically, the idea was to stop planning so much, make a list of the features that you want to build along with the requirements (just a simple list) and run the big project as a bunch of short, two-week projects, showing your work to your stakeholder or client every two weeks. It wasn’t called Agile, but many of you will recognize this approach as the Agile backlog, sprint and review process — 18 years before they slapped the Agile label on it. It worked really well. I got a letter of commendation from the VP of IT for “my” innovative technique. And a raise.

I want to be clear: I didn’t give a damn about the technique. I hated doing Gantt charts and large (or any) project specifications. This IEPM thingy called for neither, which meant I could live the life of a slightly overpaid and underworked 20-something project manager whose teams did all the hard work. I was lazy, not inspired. I liked going home early and playing volleyball at the beach, not managing.

I spent the next 20 years as a project and program manager, using other Agile-like techniques, like rapid prototyping, concurrent development, RUP, XP and others — all of them as just easy ways of managing. I now call it “lazy managing,” but I mean it in a good way.

Software Agile happened in 2001, but I didn’t even notice it. I only came to know that Agile years later, after it had become popular. And it was almost a decade later before I really came to understand it and care about it.

Software Agile, the Original Humanist Agile

The first real Agile should be called “Software Agile,” and is best understood by reading the very brief Manifesto for Agile Software Development. Software Agile was created to solve one very big problem as identified in 1978’s book, The Mythical Man-Month: software projects were getting too large to complete. In fact, many of them were going down in flames. Our traditional, hierarchical plan-and-control (keep these words in mind) methods of managing projects and people simply did not scale.

The word Agile came from the solution to the planning part of that problem: the first plans for any project are written at the point of maximum ignorance; we know the least when we first start. You want to write an accurate plan? Just wait until the project is over. Your ability to plan well increases as a project progresses.

The planning problem grows geometrically with project size; twice as large of a project and four times as difficult to plan. In a 300 staff-year project, any attempt to plan the solution ran into a challenge: the more time you spent planning it, the longer it took to get started. And the larger the plan grew, the more ignorant and less understandable the whole thing became. Big projects had very ignorant plans.

Software Agile replaced plan-and-control with a test-and-learn approach. Basically, it prescribed don’t plan all the way to the last thing, just plan the next thing…and you’ll know when you get to the last thing. If you’ve heard of Agile as being “open-scope,” which means we will just get working and see how far we get by X date, that’s where this comes from. Using this technique, projects started faster, and teams would often attack the ignorance,by focusing on what the team knew the least about.This method of dealing with massive ignorance was great — but the problem itself, the unknowability of a very large project, is not actually all that common in the business world. Think about your own organization…is the work so hard to plan that you can’t take even just a few minutes, hours or days to do it? Most work does not have a geometrically-massive planning problem like the one that Software Agile solved.

And hey, abandoning planning is actually pretty easy to do, as well — we often do things in an unplanned way…even when we should have planned.

More profound, I believe, and yet very subtle, Agile also dictated that there be a release of control over the team. With the team in control, work got done differently. I didn’t understand how important this was until I learned a lot more about Agile, and then had the profound experience of seeing what a difference it can make.

Nonetheless, Software Agile was pretty much a genius masterwork. After more than a decade of experimentation and refinement, in 2001, a group of 17 Software Developers met in Snowbird and agreed on The Manifesto, a set of principles to be used for these very large projects. The fact that we have very large software in our lives now is a direct result of Software Agile.

The Software Agile principles are amazing in their clarity: trust the team to work with the client/end-users, build the right thing, and trust they’ll find their way. It worked ridiculously well, and yet, we’ll see how difficult simple ideas can be to implement.

The Plan-and-Control model is how we traditionally manage: the people above you — your manager(s) — figure things out, and then tell you what to do and how to do it. Sure, you have a bit of control, but not near as much as you might think. But letting go of control when you’re a manager? Really difficult to do.Ten years later, I would summarize this new style of managing as, “Stay the F — k away from the team.” Those early gurus of Software Agile knew this was a bit of a dark art; their techniques were evolved by working in collaboration for over a decade. They and I had something in common — they were not managers, but rather very senior software engineers who didn’t want to be managers. I think it was easier for me and probably for them, as a result, to have the first revelation of letting go. When Software Agile was practiced by those wizards, the results were astounding.

In the 2009 book, The Business Value of Agile, which studied Software Agile, research showed that software development velocity, as measured by how much good code was written per hour of work (called a KLOC), increased by 500–1,500%. That’s a wicked-large number, like 1 person accomplishing what 5 to 15 might. Solving the planning problem helped a lot, yes, but I’m pretty sure a good part of that result was from the control part.

Software Agile shone a light on what teams could really do when we stop managing the way we normally do. Your organization likely doesn’t have a massive complexity or uncertainty challenge like Software Agile fixed, but your performance may suffer from over-control…and that’s a spot where you may learn some things.

Agile Spreads

Four years after the Manifesto, in 2005, it was de rigueurfor all high-quality technology job candidates, especially senior program managers like me, to have the word “Agile” on their resume. Word on the street was that Agile was magic, and the magicians were in demand. While I didn’t have formal training, I could do many of the techniques better than most who had some. That year, I implemented my IEPM-style Agile and boosted a 20-person development team’s delivery rate and reliability dramatically, like 50% or so.Two years later, in 2007, the agency-consultancy Sapient hired me to run its 100+ person Los Angeles office and implement Sapient’s version of Agile there. A full six years after the Manifesto, I finally had to take Agile seriously. We implemented it and it didn’t really work. Nothing. Nada. Just a few different rituals, but projects didn’t go any better or anything. In fact, I think people were slightly annoyed that they had to do things differently.

I learned another valuable idea: the chaos of real life intrudes in most businesses in ways that never happen in a large-scale software project. Software Agile had been a solution for massive ignorance, but not chaos — a Software Agile project was pretty calm, with the same team working on the same project for multiple years.Link to article here (http://bit.ly/MD-NOCO-Test)

The less your project (or business) looks like a 300 staff-year software development project with dedicated teams and a very patient client, the less likely it is that Software Agile will work.

How do we usually solve for chaos in an organization? More managers! We’ll come back to that point in a minute.

A few years later, I came up with the list of the Seven Forms of Chaos that Will Make Your Agile Fail. All of them can be found in ad agencies, which are what I often work with, but also most organizations have pretty high chaos levels.

The Agile Toolbox

So what was Software Agile really? In reality, Software Agile was just a name — pretty much everything that it prescribed was based on techniques from yesteryear: the Delphi method, developed by Rand Corporation in the 1950’s to improve cold-war policy analysis; the Kanban, developed by Taiichi Ohno of Toyota in 1953 to help smooth automotive assembly line planning and operations; and the stand-up meeting, developed by our caveman ancestors standing around a morning campfire, planning what they will do that day. And of course, the Iterative Enhanced Project Management that evolved in the early days of software project management.

You’ll notice that most of those techniques have nothing to do with software. Instead, they unleash better results by leveraging cognitive and behavioral models that underlie our humanness. You can think of Software Agile as a moment in 2001 when a group of software folks decided that these very good techniques, arranged a specific way, might work well together in a very specific situation.

In reality, what most people call Agile during the last 10 years should be called “Toolkit Agile,” using Software Agile components –the stand-up meeting, the backlog, planning by using sticky-notes, etc. — as they see fit, but not in a comprehensive system.

Just like at we did at Sapient, companies seem to pick and choose various techniques…if something appears to make sense, they use it. But choosing some new tools doesn’t really mean that you’re fixing anything. It doesn’t mean that you’re actually getting results.

But the flexibility of Toolkit Agile — or you could call it fuzziness — cracked open the door to today’s “Agile is Anything Management Wants.” Increasingly, Agile happened as a result of managers with too little knowledge and too much time wanting to “just try something Agile.” It can feel better than doing nothing, like buying some plants or painting the walls can cheer a place up.

Feel-Good Agile

In at least one of my previous articles I have referred to a moment in 2015 when I spoke at a large IT and software developers conference with 600+ people in the audience. I was on a very large stage with huge monitors hung from the ceiling on each side of me — about as Rockstar as I have ever felt. My keynote was something like, “Turbocharge your Agile Projects.” I designed the whole presentation around a thought, “almost 15 years since the Manifesto, a room filled with Agile Wizards, let’s have them share their awesome numbers!” When I asked for a show of hands for who was using Agile, all hands shot up. Check.

I then cited the data on how well Software Agile could work (5–15X) and then asked how many had (only) gotten a 2X (200%) increase in team velocity as a result of implementing Agile — I figured shoot low and I’ll get a whole bunch of people.

Nobody raised their hand. What?

I worked my way down from there, increasingly worried that my whole presentation concept was going to blow up; 1x (100%), 50%, and still not a single hand. Six hundred-plus people who had moments ago demonstrated that their arms did, in fact, work. Shit.

Finally, out of sheer desperation, I asked how many of them got a 25% or more greater improvement in team velocity, only six raised their hand.

Six out of six hundred. What?

Only five slides into my keynote presentation, I was reeling. What does this mean? Why are they doing Agile, then…or are they? With all that feel-good going on, shouldn’t there at least be a placebo effect?

In desperation, I asked how many people felt that they, “…prefer Agile to the old methods and generally like it better, though you haven’t seen any measurable performance improvement?”

All of the hands in the room went up. Wow.

I was amazed, not only because of the absence of wizard-like results, but also because, frankly, what I had been doing, albeit somewhat ignorantly of “real Agile,” did move the needle. At the time of that event, we had trained about 1,000 people in 20+ organizations, and all had benefitted, but I had at least one client that was going 100% faster as a result their Agile implementation, and many others were doing almost as well. I had assumed there would be a LOT more of them in this audience.

From that day on, I started investigating what people were doing with Agile and asking what results they had achieved. It’s not a pretty picture. I’ve heard the aftermaths of failed attempts inside hundreds of companies, and been partial witness (usually over drinks, so I take them as fact) to scores, maybe hundreds, more. If we include several other large-group raise your hand polls I have conducted since that day in 2015, the numbers are even greater.

So, everyone agrees it is better. But almost nobody can point to real results.

Foolproof and Feckless Agile

This is where things got interesting for me. Pretty much everyone had tried to implement some key parts of Software Agile, and it didn’t really seem to fit very well, but things didn’t really get worse either. I would call it failure, but most of them didn’t. Many of them had also abandoned some of the pieces that they had tried, a la the Toolbox Agile model.

I drew the conclusion that failure, outright failure, is rare when you implement pieces of the Agile toolbox. Why? Because the toolbox is filled with best practices for humans, mostly better management practices that bring out the best in us by creating flatter organizations and more inclusive conversations. If you get it 5% right, you’re doing it 5% better than before. So it is hard to truly fail.

But you’ll be lucky if you get anything. The tools appear to be simple, but because they have underlying cognitive and behavioral components, they are actually a quite tricky to get right. The stand-up meeting, for example, works better when managers add new information at the end, rather than at the beginning. Counter-intuitive, yes, but it has basis in the research, and we’ve demonstrated it at dozens of companies. It is really easy to do a bad stand-up meeting, and really hard to do it well.

But I still wonder, does implementing a stand-up meeting, a caveman-era technique, really make you Agile?

Wondering if your implementation sucks?

Here is what the Foolproof and Feckless Agile implementation looks like:

• Teams are a bit happier (it is hard to mess this one up)
• Stakeholders or clients don’t really participate or “get it”
• No change in managerial style or interaction — managers still manage the same way, assigning work to people and telling them when and how to do it
• No reduction in the intensity of managing
• No change in velocity of work or quality
• New excuses for why it is late (“your work didn’t make it into this sprint, sorry!”)
• Stand-up meeting has poor attendance or low energy

If more than a few of the above are true for you, then you’re not doing it right. Eating a barely-edible, over-cooked steak is better than nothing…but for the cook it is a failure. If I show the list above when speaking at a conference, I see a lot of nodding in agreement. And I don’t think that warrants the cover of HBR.

I think that victory is often claimed by contrast rather than conquest. The sad reality of many organization and process transformations is that they don’t create results. A few years back, the number was greater than 50% but I’m not sure what it is now. How many training programs have you seen or paid for that only months later nobody was using? An Agile non-failure could be a success for many organizations; changing their process and not having it become a shitshow may be what constitutes a win for many companies.

If you’ve implemented some pieces of Agile, and are feeling better about it, good for you. You’ve succeeded at not failing, not messing up the engine of the business, and that’s worth something.

Commercialized Agile

Some of the blame for what happened to Agile lies in its commercialization.

There was nothing really new about Agile. As you can see in Craig Larman’s IEEE paper from back in 2003, Agile was merely a waypoint in the development of iterative and integrated development methods. But what made this Agile different was that it had a business model and some great marketing behind it. And some good branding: “Agile” was way better than XP, RUP, IEPM and many others the preceded it.

I’m pretty sure that most Feckless Agile implementations have their roots in the training model defined by the then newly-formed Scrum Alliance — this organization was launched by some of the original Manifesto signatories to create a proprietary version of Agile that could be packaged, licensed, protected…and IMO, capitalized upon. One Agile coach I know from the early days of Agile said, “Scrum is a business model, not a project management method.”

The entry-level Scrum credential is called a Scrum Master certification. The training is two days long and includes some terminology, some somewhat silly games, and then a final exam, after which (until recently) you would automatically get your certificate, regardless of your score. Everyone became a Scrum Master. All you had to do was pay your fee, sit in class for two days, and you were a master. Not.

Two days of training never made anyone a master of anything. You want someone to implement a complex set of cognitive/behavioral methods after two days of training? Would you go to a coach or therapist who had only two days of training?

Even as lame as that may be, many organizations instead naively buy “Agile” software. The industry’s sales line, delivered with a Hubspot-marketing sort of glee, typically declares that it “XYZ software will make you Agile in XX days!” Of course, almost all of these companies existed prior to the latest Agile craze, and many have literally just bolted on the word Agile with some half-baked features, most of which don’t even support the hidden magic in Agile.

As noted a few years ago in a The Economist article titled Digital Taylorism, virtually all project management software is distinctly non-Agile, decreasing the empowerment of teams. Project management software won’t make you Agile. Ever.

If you want more color on this topic, please see my article, How Marketing is Killing Agile.

The Scrum Master and commercial software models fail because they are in complete conflict with one of the most important ideas in Software Agile: teams get better through removing the managers from the room. Agile is about teams finding their way, not having managers or pieces of software telling teams how they should be.

Agile co-founder Dave Thomas says much the same in his 2014 blog titled, It’s Time to Kill Agile, that highlighted that most providers were on the right-hand side of the manifesto, not the left. By “right-hand side” he means that they are doing things the same way they have always been done, but calling them “Agile.”

How I Discovered Humanist Agile

These “fewer managers” and “release control” ideas are not at all obvious, and while I sort of understood them from being a lazy (but effective) manager, in 2010, their importance became etched into my psyche because of a serendipitous moment of great desperation.

I had a bad project on my hands. Not just any project, but a project that was more like an airborne fuel tanker in flames heading toward the unforgiving granite mountain of a launch date. As a last resort, I completely handed a project over to the team. Here’s the deal.

It was staffed 15 people and 3 project managers on it. I removed 2.5 project managers and made it all the team’s problem. I gave them one very important instruction that I repeated over and over. And they pulled it off. They went from a dispirited, overworked group that was slogging through things, to being a team of warriors that were killing bugs, and releasing features with a Midas touch. Solid gold.

No manager could have managed that one to success. Only the team could. And in that moment, it verified the one big idea in Agile: empowered teams are far more powerful than managing and managers.

This moment changed my life. I can tell you that when I manage this way or teach others the Humanist Agile Way, managing becomes a noble profession — all of those buzz-phrases like “believing in your people” and “developing potential” all come true, but in a way that I never would have expected. It struck me than that gardeners don’t make plants grow…great gardeners provide the right environment and inputs, but they can’t make anything grow. Same goes for great managers.

It hit me then that 2001’s Software Agile had been a vivid demonstration of what we’ve kept learning, and forgetting, over the last 50 years: that the way we manage, our default societal notions of managing, as well as most patterning that we receive, are all wrong. Software Agile was a striking repudiation of the management techniques still in widespread use today, as defined in the 1920’s by Frederick Taylor.

A manager finds out that there is a change in priorities and that a piece of work being done by worker X is of lower priority than that of another piece of work, Y. He/she then goes to that worker, tells them to stop working on X, shift to Y, and also gives directions on how to work on Y, as well as on when it should be done. In the end, the manager’s and the worker’s effectiveness is judged by whether the worker got the work Y done in the way and by the time the manager expected it.

Sound familiar? That’s Taylor’s Scientific Management, and it is how most managers think we should operate. Unfortunately, pretty much everything in that paragraph is cringe-worthy. Especially, when you really understand how an organization should operate and what the true function of a manager is.

How did I learn this? Well, one year before that 2010 project-in-flames, when I was laid off from Sapient during the Global Economic Whatever, someone suggested that I not only was I getting laid off, but I had been a pretty bad manager. On one level it was a very fair comment, for I had never really liked being a manager. But I took offense, and while the job market recovered, I read about 20 really heady books and piles of research on how to really make teams effective…how to manage really well.

The books I chose included many techniques that had been assembled into Software Agile, the humanistic ideas of worker empowerment had been out there for a long time…but nobody had really put them together as a management theory.

Oh, and yes, I also discovered that I had been a pretty bad manager…but no worse than many others…that’s not an excuse; I just didn’t know the difference.

So what did I tell that project team in 2010 that enabled them to save the project? I channeled my inner anti-Taylor…I relinquished control, and asked the team what they should do. My direction was specific and empowering: You should only do what you, the Team, think you should do. I put them in charge.

Empowerment like this is the heart of Agile. It’s a different approach to managing, a humanistic one. Here’s the Humanist Agile version of the Taylor vignette:

A manager finds out that there is a change in priorities and that a piece of work being done by worker X is of lower priority than of another, Y. At the next day’s stand-up, the manager provides the priority information to the team so that they can assess how the change impacts their work plans. If they have further questions, they ask the manager. The team chooses when and how they will incorporate the priority change, and then they proceed with their work. The manager’s and team’s effectiveness is judged by how much end user value they create, the quality of their work and how well they feel their process is going.

The creators of Software Agile tapped into key concepts of the human empowerment movement. You can find these notions as far back as the 1950’s in management literature and in the decades that followed various writers have brought it to the front.

The Japanese demonstrated clearly the value of these ideas in the 1960’s when Taiichi Ohno empowered 5th-grade educated factory workers to become the quality managers for the Toyota production lines. They still lead in automotive quality, fifty years later. That’s how powerful empowering people can be.

More recently, seminal research around Self-Determination Theory(SDT) has defined the sources of human potential, while also showing us how much our traditional notions of managing, and Taylor’s notion that some workers were more like “oxen,” only perpetuate bloated management hierarchy, while failing to bring out the best in us.

Guess what? Let go of control, empower the teams, and they’ll pretty much do better without you. That’s sort of a hard pill to swallow after your promotion to manager…and most managers would rather not.

The original Software Agile delivered a holy trinity of results, faster delivery, improved quality and higher team and stakeholder satisfaction. That’s achievable today, but it requires a huge change in the way we manage.

(Edit: numerous requests for more info on this section, so here is an article I just republished from AdWeek onto on Medium: Calming Agency Chaos by Un-Managing and Agile)