Top Down or Bottom Up - Large Scale Agile Adoption

So I've worked in both large and small organizations where we have gone through an Agile adoption or the phrase of the day, Transformation. Having seen both sides of the coin I started realizing that you have two paths to take when considering moving your organization to an Agile delivery methodology.

I use the phrase delivery, because I think at the end of the day that is what we are talking about.  Strip away the manifesto's goals of conversation over documentation, accepting change, etc...What are are really talking about is moving an organization from a Project delivery methodology to one that is Product delivery oriented.

What is the difference?  From a Management perspective actually quite large.

1. Project Delivery - These have very strong controls which move people to a new project.  Budgets are set up for the specific time period of the project and then up front requirements and design are completed in order to ensure that the project is fully ready to be engaged.  Project Managers manage all facets of the project via extensive project planning and plans.  Management receives up dates as to the project progress on a regular schedule, usually weekly. Resources are assigned either in full or in part, yet no one actually monitors nor can they really manage whether or not someone is working 25% on a particular project.  Projects tend to focus on reporting and there is high pressure to ensure that individual projects are green, which drives teams to deliver on the easiest and often less valuable parts of the project first and only at the end of the project is the hard work tackled, which reveals itself as budget overruns or timeline delays (or worse delivery of reduced scope).  Project reporting is elaborate and management receives project reports that are often sanitized.  Value is typically not delivered to the organization until the end of the project.
2. Product Delivery - The work that is done for a Product is centered around a value stream it delivers and the work is ongoing.  Teams are funded as a whole and are kept together long-term in order to maximize their productivity.  Work is planned out in short increments called Release Plans that span anywhere from 6-12 weeks, with 2 week sprints.  Management receives regular updates (2 weeks) but can access information radiators at anytime, transparency is the key and goal with Agile Product Delivery.  Teams commit to work in 2 week sprints and their commitment is key to building trust with Management.  As time goes by management can trust that both a teams abilities and productivity can be counted on.  Teams are focused on delivering high quality code to production every two weeks which brings value to the organizations investment in them along with the increased value in terms of new sales, reduced costs, etc...Feedback loops via Product Demonstrations provides management the ability to assess where they are going with the product and deliver not what was asked for up front (Project Delivery) but rather what is actually needed (Product Delivery).

So what does the difference between Project and Product delivery approaches have to do with Agile adoption, well everything.

Most Agile adoptions begin at the bottom of the organization with the teams tasked with developing new software.  These efforts are borne, as the Agile manifesto was, out of frustration with how software was being developed in their respective organizations.  Often management is aware of the issues these teams face but are unable or unwilling to make any changes to how things are currently working and why would they?  You learn very early in your career that rocking the boat is not something that goes over well with organizations, my first boss told me I couldn't by computers that weren't from IBM because you don't get fired if you buy IBM. The message is that if anything goes wrong you need to point to well-known names, processes, etc.. with which to blame or use as support.  To think that this doesn't go on today is to place your head in the proverbial sand.

Though we can have great success with bottom up Agile adoptions with respect to improved productivity within small groups/teams, the overall Project oriented organization is typically still in place.  Management still wants to see project plans, have things 'planned' out for up to an entire year, they aren't comfortable with the fuzzy feel of product roadmaps.  They want commitments, even false ones, so that when things fail they can point to the fact that they had all of their planning in place.

For Agile to really take hold, Sr. Leaders need to change the way that they manage both their people and the work.  It starts first I think in understanding that we have not learned how to speak to management very well yet from an Agile/Scrum perspective.

We need to understand what management is really concerned about and then center our product delivery efforts around that.  One of the problems that we face with some of our leaders is that they themselves don't always know what to be focused on, they are looking at multiple balls in the air but at the end of the day as a Sr. Leader I think I have just a few things I should be focused on:

1. Growth - This is often related to sales, market and revenue.
2. Profits - Tightly aligned with the first item, our ability to make a consistent profit is what helps us continue to reinvest in our company.  Ever increasing revenue or sales without corresponding profits will eventually lead a company into bankruptcy, money isn't free and it is not endlessly available, in spite of what we think we see with new technology organizations.
3. Organizational Excellence - Because none of the above can happen unless you have a great organization.

Agile actually addresses all of the above, yet we spend more time talking about how we will improve our individual work efforts which causes us to  fail to tie this to the needs of management.  Management on the other hand often views the improvements that come from the bottom up approach as more of an anomaly rather than an organizational improvement worth adopting.

Trust is the missing component when it comes to conveying how Agile will make the entire organization better.

Agile isn't easy and it requires skills that frankly many of our Sr. Leaders lack or don't fully utilize and the politics of most organizations reward behavior that doesn't align with Agile principles such as transparency, open honest conversation and openly questioning the status quo.  We have people in power who got there by way of the non-Agile status quo and changing that means they have to learn how the new game is played in order to stay on top, it's much easier to keep things they way they are over learning how to navigate the new.

So how do we speak to our Sr. Leaders with respect to Agile?

1. Better ROI - Talk to any Finance executive about what they look at when purchasing a piece of equipment that will deliver revenue and you will hear them talk about Net Present Value of the investment, Positive Cash Flow and Depreciation costs.
   1. We improve ROI in Agile due to our focus on only the most valuable items.  In non-Agile project work often are working on features/functionality that may be important to someone inside the organization yet will bring little or no value to the organization.
   2. When we are able to start talking about the value streams of our organization, be they revenue, cost reduction or improving our brand image we begin to be able to have a better ROI conversation with management.
   3. We also positively impact ROI via higher levels of productivity gained with dedicated teams.
2. Flexibility - One of the most important elements that 21st century organizations require is the ability to be flexible enough to react to market forces or reactions.  Financing large projects far out into the future with the expectation of some level of return and no we don't really have great track records of predicting future ROI out very far into the future.  With Agile we provide the framework to identify the most valuable work for the business in small planning windows.
   1. Sr. Management needs to understand that this flexibility comes with an obligation to have consistent short term review windows as the team progresses so that we deliver what is actually needed and not what we thought we wanted.  You may have thought you needed a Ferrari when in fact what you needed was a mini-van, we provide the framework to course correct via Sprint reviews every two weeks.  If you plan all of your project up front for the Ferrari, we'll certainly try to make that happen, but in reality as the end of the project nears you will probably get the car but with a lawnmower engine and no brakes, it may look like a Ferrari but it won't operate like one nor will it provide the value the organization really needed.
3. Predictability - Another key element that we deliver with Agile is predictability and accountability.  Your teams will be much more accurate in planning and delivering in short-term 2 week sprints with a planning horizon of 6-8 weeks.  What management needs to look for is consistent delivery of the committed work that the team makes, commitment is everything.  What Wall Street analysts look for is a business that can provide a solid ROI, react responsibly to their competitors or even better be a market leaders and provide predictable results year in and year out.

So the question at hand is what is better a Top Down or Bottom Up adoption?  My money for long-term success is on the organization that can consume what Agile really means, not just to their development teams but the organization as a whole.  You can't BE Agile if you don't make the paradigm shift from command and control to one of collaborative and collective delivery.

Agile isn't Easy

Over the years I have seen many teams and organizations who start on their journey towards Agile product delivery make the mistake of thinking that Agile is easy, promotes freely changing direction and worse will fix all of your issues and make everything better quickly and easily.  The truth couldn't be further from the reality. There is no such thing as a perfect software development delivery process.  Unlike production lines for things such as automobiles where you have the same pieces going to together each and every time and each piece has a specific functionality, tolerance and timing, software development is the exact opposite.

Software development is about meeting changing needs across the dynamic nature of business  For example - You wouldn't see an automobile company add anew feature to a car on their assembly line over a 2-4 week time period.  They need time to design the entire process and ensure that the production line is capable of accepting the new change.

Traditional software development tends to align a bit more to the automobile example and in fact there are times where this type of rigid, pragmatic approach to product delivery is actually the correct process (Agile isn't for everyone nor for every situation).

However for those that are going to move to Agile you need understand that the type of discipline that you might use in the automobile example actually needs to exist in your Agile processes, seriously you ask? Yes - You need to be able to deliver high quality, well-tested and fully functional pieces of software every two weeks. Now are you seeing how difficult Agile can be?

If you think Agile is easy, then you are already on the path to failure and unmet expectations.

It is very common for teams who are moving to Agile to take their interpretation of the Agile Manifesto to the unhealthy extremes, for example:

• Working Software Over Comprehensive Documentation

Many teams I've worked with take this to mean NO documentation and that couldn't be further from the truth.  We value working software over the need for documentation but I've never believed that you can have long-term success with your product without delivering some levels of documentation.  Without documentation your software knowledge becomes tribal and when your tribe leaves the team or your organization, well so does their knowledge.

There are way for teams to build documentation as part of their daily Sprint development work.  Using the combination of user stories and Behavior Driven Development (BDD) acceptance criteria as the foundational elements of your work you are creating your documentation as part of the work needed to deliver quickly.  The great value in BDD is that the acceptance criteria is written in English syntax and then translated into test automation.

This process of writing stories with BDD is supported by Gojko Adzics book, Specification by Example and allows us to deliver light weight documentation during our sprints.  I often see teams adding a story to a future sprint to handle their 'documentation' requirements of their previous work but I haven't seen this work long-term.  Functional development, dealing with bugs, etc... will ultimately push these stories down into the backlog, never to be seen again.

The process described above is not easy, but it can be done and the teams that can get to this level of capability will succeed in driving value to your organization every two weeks.

One of the things that I consistently tell organizations moving to Agile is that it will highlight every current weakness of your product/software delivery methods.  Agile is a game changer, it requires a mind shift in how you look at your product, the work that supports it and how you see the visualize the value your product delivers.  If you are a leader who is going to be uncomfortable finding out truths about your organizations inefficient manner of product delivery then you need to think twice about moving to Agile.

Do you have TITL (To Important to Lose) people in your organization?  If you do, then you need to really look at how they influence any changes in your organization.  Do you need to get their approval, gain their support, etc....?  If so you will find more often than not that Agile will scare the hell out of them.  Successful Agile teams/organizations understand that self organizing teams take away the need for many of the day-to-day management decisions our middle management layer makes.  Agile speed comes when you remove the friction of management layers and provide teams with a clear vision of what you want them to deliver.

You must be prepared for the resistance that you will face from your product delivery teams, not everyone wants to go to Agile  We become comfortable with what we know and do in our daily work life and in that comfort comes stasis.  If you are not prepared to lose people, especially your TITL people, then you need to question if Agile is really for you.

I know it sounds heartless, but I've been working for over 30 years and one of the first things I learned right out of college was that technology was disruptive and if you weren't on board with how it changes how you work you would be left behind. At one company I worked for many years ago, I saw Regional sales managers who had been with the company for over 30 years and when we rolled out a sales force tracking/marketing tool (which I led) there were several who refused to even turn on the computer, they were all let go within months.  As employees we have the obligation to continue to grow and learn and continue to make ourselves valuable to our organization and if we don't, then you as a leader have an obligation to make tough decisions that ensure that the organization is continuing to grow and not stagnate with old processes.  It's not personal it's business.

As you move to Agile you also need to understand the investment that needs to be made in your technology stack.  Many organizations have decades old technology stacks which have been shoe horned into the future and though you get by you won't be able to become Agile until you have a strong Continuous Integration framework, high levels of Unit, Integration and Functional test automation.  Getting to this will take time (and using the stories and BDD disciplines mentioned above will help you get there).  You simply can't go fast if you don't have the technology backbone that supports it.

Agile isn't easy by any stretch of the imagination, it requires thoughtful introspection, focus on continuous improvement, disciplined delivery and a tenacity for value and quality that is never satisfied.

Agile Planning and Delivery

Agile is often viewed as the way that small organizations and teams can build product quickly and in fact I believe that to be true. Being quick and nimble allows smaller/growing organizations the ability to get into a market quickly and deliver features that larger organizations haven't delivered yet or haven't thought of.

This very ability to deliver new features to a market that desires them is the very reason that larger organizations need to change the way that they deliver their product.  As an organization becomes larger, the entropy that comes from that growth causes the organization to stop being a market leader and start being a market follower.

In the technological world we live in, you can go from market leader to out of business in just a few short years.  I read an article recently that suggests that Walmart is starting to show signs that its dominance in retail may be coming to an end.  I wouldn't be surprised, as I think they have lost sight of the fact that low prices don't always solve the things we as consumers value.  Walmart isn't a market leader anymore, they may just not have come to that realization.

It's this lag in reality when big companies start falling behind  Somewhere along the way almost every organization will lose its way, lose site of the very thing that caused them to be great in the first place.

When you look around in the Agile learning space you see many of the original creators of Agile and Scrum trying to figure out how to help large organizations get past their own entropy.  I don't think that Agile can help, it can highlight where your planning and delivery processes are inefficient but it doesn't provide the language that senior management needs to hear in order to effect real change.

If we want to help organizations transform and be 'agile' then we need to speak in the langauge of management, which is money.  I've yet to have a cogent conversation with management regarding how their lack of focus costs them money and more importantly fails to deliver the speed to market that they need in order to stay market leaders.

If you think that Agile provides you with a means to quickly change product direction in order to play defense or catchup when new features that you have been planning on are released by a competitor, then you are missing the point.  Agile does accept that change will happen, but we expect that the change will take the form of measured change in how we can make the product more valuable, not wholesale changes in direction.

Changing direction within a Sprint is something I've seen several times and it's almost always driven by management.  Whether you have too many idea's/opportunities or are just trying to stay relevant give you team time to work on one thing at a time.  Context switching reduces productivity and that you can translate into money.

The example I've been using recently is this:

If you managed your Sprints the way you manage your money you would go broke.  Take this example for consideration (note this is what good Agile teams should be forcing into the planning conversation).

Consider a team that has planned 20 points for the current sprint and someone (Product Owner, Sr. Management) has decided that something else is more important, like completing a feature for a key client.  The appropriate response from your Agile team is 'Great, let the team review the request and provide an estimate'.  Right there you have probably lost 2-3 points in analysis and estimation by the team.  Once they have completed their estimation, they come back to you and tell you that it's a 5 point effort.

You think to yourself great we can do that in this sprint,  let's do it.  A good Agile team will tell you yes we can do it but you need to remove 2-3 points for the estimation effort AND 5 points for the new work that you want us to take on in the current sprint.  So you have LOST 7-8 points of productivity, aka value) to get 5 points of value.

If you invested 8 dollars into a business and only were able to get 5 dollars back you would quickly realize that you can't sustain that monetarily, right?

Put this to your executives next time you have a priority conversation.  I didn't even bring in the  technical debt we typically take on to hack something together and the lack of testing that will happen because we weren't ready to take on the work (read no automation).

As a Finance guy I would say that this is a bad investment.  Yes I want to satisfy my customer but if they are a good customer you should be able to have an ability to manage their expectations so that the team can take on the work in the next Sprint.  Depending on when the request came in they may only have to wait 2 weeks.

Effective teams I've worked with use this process as a first line of defense against changing priorities within a Sprint.