As more and more larger organizations look to Agile as a means of delivering software to their customers the one thing that keeps coming back to me is that for any of this to work there has to be transparency and acceptance that a move to Agile will change your organization, not understanding this will court almost inevitable failure. Agile in itself is an aspirational desire to change the way that we deliver software, one that does away with the project processes that evolved over the years from Waterfall. I know that waterfall feels sound and safe with all of its up front business analysis, project planning, Steering Committees and that all important Change Management process, but in reality it really is more of a facade than foundation.
Having managed work and projects in both worlds I have seen how it all works. Recently I was told (in an Agile organization) that Project Managers are responsible for delivery and it was at that point that my thoughts crystallized around my own journey, Project Managers don't deliver, Teams do.
At it's heart Agile is about everyone doing their part to make our product delivery better, whatever that looks like for your organization in that moment in time. Agile by itself is not prescriptive, the Scrum/Lean techniques and processes that have evolved from our Agile journey may be a bit more prescriptive and become more so when we add things like Scrum certifications to people's palmares. We need remember that one of the key reasons that the manifesto came into being was an intense desire to find better ways to deliver software, which means the journey has no end and certifications and such are merely ways for us to have a shared language. Let me repeat, once you commence on your Agile 'journey' it doesn't have an end, you will always be evaluating what you can do better.
So back to the question at hand, to succeed or not to succeed in Agile, what needs to happen?
- Trust - First and foremost we need to begin to build trust between Sr. Management and our Product Delivery teams. If we have a history of delivering late, with fewer features and cost overruns it is really hard for that same management team to flip the switch once you say you are Agile and trust the very teams who haven't delivered in the past. Trust is a two-way street and the great thing about Agile is that once you begin to master the techniques and methods that successful teams utilize trust is almost a by-product of that.
- Commitments - In Waterfall we are making a 'commitment' to deliver a set of value/features in a specified period of time based upon a business requirements document as guidance for what the business/customer is asking for. When we make a commitment farther out into the future we become less and less accurate with our plans, it is the nature of the unknown unknowns in life. Things change, they always do, so to expect that our business and software development teams, in today's ever-changing world, can predict 9-12 months in the future exactly what they are going to deliver is simply living in delusion. Manage reality or it will manage you. Commitments in Agile are much shorter in time, basically every 2 weeks. These commitments however are based upon the Vision that YOU management need to provide. You say you can't plan for the future every two weeks I would argue you can't plan much further out. By planning and committing in shorter delivery increments, management get an opportunity to change direction without causing massive pain from already planned out work. We need to be able to change direction or refocus efforts on the things that are most valuable to our business, not what we made big plans for last year that we thought we needed. Locking in feature development that doesn't meet customer needs, simply wastes money and loses market share. When teams make and keep their commitments to you, they gain confidence and you gain trust.
- Context - In most waterfall projects we end up asking for absolutely everything we think we might need, when in reality sometimes 70% of what we asked for (or even less) is more than enough to meet the needs we were trying to address. Focus on the most Valuable things your customer wants and move on to the next highest value work, not diminishing returns on things customers may not value as much as we might. Teams want to work on what brings the most value to the organization, development teams when provided the transparency of why we need to do something can do amazing things.
- Self Organizing Teams - This one really causes I think the most concern for management. You in essence are saying that the teams that you currently manage are better able to make decisions about how to delivery our products. Guess what, they are. These are people who work in the trenches every single day, know every single issue, impediment, risk, etc... associated with your current product delivery processes. And every one of them has probably said something to the effect of, 'If I was in charge I'd to do X to fix this'. I've been a Sr. Manager for many years and have built several high performing teams and one of the best things I can do for them is to provide guidance and vision for what I'm looking for and letting them solve the issues that deliver the solution. I'm a smart guy but I don't have all of the answers and if you are the type of manager who believes that in order to be 'respected' you have to be the one to manage how Agile will change your product delivery processes you will fail personally and the organization will suffer as a whole. Teams of people can solve major problems much more easily than one person can, so let them have the ability to self organize and empower them with making the necessary changes to improve your product delivery.
- Context - With great power comes great responsibility. By ceding some level of daily control to your teams you are also doing so with the expectation that they deliver on what they commit to.. If they don't then they must provide a game plan based upon the Retrospective on how they will solve their inability to hit their commitments.
- Commitments - In Waterfall we are making a 'commitment' to deliver a set of value/features in a specified period of time based upon a business requirements document as guidance for what the business/customer is asking for. When we make a commitment farther out into the future we become less and less accurate with our plans, it is the nature of the unknown unknowns in life. Things change, they always do, so to expect that our business and software development teams, in today's ever-changing world, can predict 9-12 months in the future exactly what they are going to deliver is simply living in delusion. Manage reality or it will manage you. Commitments in Agile are much shorter in time, basically every 2 weeks. These commitments however are based upon the Vision that YOU management need to provide. You say you can't plan for the future every two weeks I would argue you can't plan much further out. By planning and committing in shorter delivery increments, management get an opportunity to change direction without causing massive pain from already planned out work. We need to be able to change direction or refocus efforts on the things that are most valuable to our business, not what we made big plans for last year that we thought we needed. Locking in feature development that doesn't meet customer needs, simply wastes money and loses market share. When teams make and keep their commitments to you, they gain confidence and you gain trust.
- Investment - Agile won't come without an investment cost associated with it. If like many large organizations you have a mess of legacy code mixed with attempts at migrating to newer technology stacks, business requirements and rules imbedded in code with tribal knowledge scattered through the organization. Agile requires speed in your product development processes which translates into several investment areas:
- Automation - When we say automate we mean across the entire spectrum of the organization, if it can be automated you should probably evaluate whether it can/should be. More specifically we want to automate:
- Unit Tests - Developers should be writing unit tests for everything they build, preferably using XP techniques such as TDD (Test Driven Development). These are not really hard processes but if you are starting from scratch there is both discipline and framework that needs to be built-in order to get to a mature state for test automation. Unit tests need to be executed with every build, because with that comes fast feedback if something broke, fix it early and you increase speed and profitability.
- Integration Tests - Probably one of the harder areas to get high levels of automation in, primarily because the organization hasn't invested in the appropriate product like test environments. Be prepared in your Agile journey to spend heavily on getting the right environments in place and highly available. Testing the performance of your system right before you deliver is a recipe for disaster and delays (remember time is money).
- Functional Tests - These are the tests management is probably most familiar with and may even have reviewed at one point or another. These are typically the manual tests that QA will execute at the end of your waterfall project, where we are not baking in Quality but testing out defects. Building high levels of automated testing at the functional level gets to what I call, Progressive Regression. Instead of running a final full regression at the end of a 6-9 month project, why not do it every single night? You will need to again invest in physical environments but also in people training as many in the QA world are not equipped to handle the new role of a Software Engineer in Test.
- Word of Caution - Don't rely only on Test Automation in your functional testing efforts, you still need real people with hands on testing capabilities because at the end of the day automated testing cannot always tell you when something is bad from an experience or product flow perspective.
- Deployments - One of the hardest things that teams fail at is planning for deployments to their environments. Making your deployment process as frictionless as possible is a high value target for your organization. Many organizations don't have a fully functional DevOps org and many in this field are still struggling to figure out how to operate in an Agile world. Let them figure it out and provide them with the tools that will support automation of critical deployment processes and hold them accountable to doing it. To many times we purchase tools and then never make the time to actually use them, invest and utilize, that is the key to your ROI.
- None of the above work comes without an investment in both hardware and people. Current requirements processes will change substantially as you move to an Agile cadence. People will struggle to find their way, some level of productivity reduction is expected in the beginning of an Agile transformation. You as a Leader need to set a clear vision of why the organization needs to change, make it clear that the teams have accountability to deliver the work that they commit to and that you as a leader have accountability to provide them with space to learn, fail and finally improve. Successful Agile includes failure because without it we aren't really learning from our mistakes, rather hiding the truth. Agile done right makes everything visible, especially who is accountable for what.
- Automation - When we say automate we mean across the entire spectrum of the organization, if it can be automated you should probably evaluate whether it can/should be. More specifically we want to automate:
- Patience - Nothing great was ever built-in a day or a week. Odds are good that this will take more time than you thought it would, but also make it clear to the organization that an Agile death march is not something that you are taking on either. Agile is about accountability and commitment, use these values to your benefit and identify those that simply can't or won't work in an Agile environment.
From a management perspective you need to understand your role in the success of any Agile transformation, it must also change the way you look at and manage your business.