Agile and Waterfall can live together in harmony

Agile is challenging to use on certain projects, like those bound by fixed-price, fixed-date contracts. Agencies, governments, and other organizations who are bound by contracts struggle to implement Agile effectively. Contracts force software developers to adopt less flexible approaches to development in order to satisfy negotiated requirements and deadlines. Even worse than taking an alternate approach is to use the wrong approach entirely. This can manifest itself as a completely bastardized version of Agile, which is unfortunately the most common version in use today.

Understanding the history of Agile and Waterfall

Agile is not new. The earliest form of Agile, IIDD (iterative and incremental design and development), was in use by engineers over 75 years ago. Plan-Do-Check-Act (PDCA) was later made popular by Dr. W. Edwards Deming in the 1950s, but the core concepts behind PDCA (later Plan-Do-Study-Act or PDSA) were based on the scientific method and the work of Francis Bacon in the 1600s. The core concepts behind Agile are not simply old, they’re ancient.

Waterfall is a comparatively modern methodology first used on software development projects in the 50s. The term itself was coined in the 1976 paper Software Requirements: Are They Really A Problem? by Bell and Thayer.

Waterfall was originally embraced on software projects by companies involved in the military and aerospace industries that needed a rigorous process to recognize the importance of delivering perfect software the first time. When lives are at stake, up-front research and rigorous quality control are far more important than the flexibility to respond to changing requirements late in a project. Waterfall reflected the times. Before the Internet made pushing out patches to broken software somewhat trivial, a rigorous process was needed to ensure that patches were the exception and not the rule.

Early CMMI adopters were developers of large-scale, risk-averse, mission-critical sytems, often with high levels of management oversight and hierarchal governance; whereas the early adopters of Agile methods generally focused on smaller single-team development projects with volatile requirements in a software-only environment.

Contrary to popular opinion, modern forms of Agile got their start in large corporations. Kent Beck created XP (eXtreme Programming) while at Chrysler in the 90s, while FDD (Feature Driven Development) was created by United Overseas Bank in Singapore. These are massive, bureaucratic companies with established cultures that were already managed by mature processes, but they created and implemented new forms of Agile in order to plug gaps they identified in their existing Waterfall processes.

Using Agile when Waterfall is more appropriate

The word Agile is hot. The word Waterfall is not. This polarization attracts a large number of practitioners and managers to Agile with a fundamental misunderstanding of what Agile is, let alone how to apply it and when to apply it. There are a great deal of companies misusing Agile rather than properly using existing Waterfall processes simply because Agile sounds better on paper.

While many software developers are familiar with the Manifesto for Agile Software Development, few managers are aware of the Project Management Declaration of Interdependence (DoI), a set of six management principles that help to encourage better leadership of Agile in the software engineering profession.

The title “Declaration of Interdependence” has multiple meanings. It means that project team members are part of an interdependent whole and not a group of unconnected individuals. It means that project teams, their customers, and their stakeholders are also interdependent. Project teams who do not recognize this interdependence will rarely be successful.

Practitioner-driven (bottom-up) Agile only addresses software construction, while management-driven (top-down) Agile can address the other inter-dependencies that impact an entire organization; everything from writing contracts that support Agile to ensuring that all disciplines required to deliver software are on the same team. In an ideal world Agile is understood and embraced by both practitioners and managers.

Warning signs that your organization has a fundamental misunderstanding of Agile:

• High pressure to release a fixed-scope project on a specific date due to contractual obligations
• Developers included on the Agile team while QA, Operations, and Strategy/Creative are managed separately
• Overtime to meet deadlines rather than adding additional sprints
• Tweaking metrics such as velocity and estimates to meet objectives
• A lack of retrospectives and process improvements
• Team members not empowered to self-organize, such as team members being assigned work by managers
• An inability to remove team members who actively sabotage the process

The effects of poorly implemented Agile are terrible. Abused Agile is essentially Waterfall without any research, analysis, estimation, or quality control processes in place. This is exactly why some experienced software engineers consider Agile to be nothing more than cowboy coding.

The best of both worlds

Waterfall is far from perfect, but nobody can argue against up-front analysis and research, especially when the result of that research makes its way into a contract. Where Agile falls short on delivering contractually-bound projects, Waterfall recognizes those constraints and provides a framework to address them. On the other hand, requirements can never be defined with absolute accuracy up-front, which is why newer software engineers consider Waterfall processes to be bloated and burdensome.

What we really need is a hybrid process that blends the best of both worlds. Both Agile and Waterfall have valuable things to offer software development teams, so instead of choosing one or the other, we should keep an open mind and evaluate the strengths of both in order to address our specific opportunities and constraints.

What Agile has to offer:

• Placing people first by recognizing capacity and adapting to change rather than forcing overtime and burnout
• Empowering teams to self-organize and work to their full potential without micromanagement
• Placing value on working software over pie-in-the-sky documentation
• Striving for constant process improvement through retrospectives

What Waterfall has to offer:

• Recognizing the importance of dates, especially when bound by contractual obligations or government regulations
• Placing value on getting it right the first time in industries where failure can be catastrophic
• Effectively scaling up and out on large teams

A balanced approach

Firm release dates always present a challenge. The less flexible a release date is, the more up-front work is required to mitigate risk. Some projects must launch on a specific date no matter what, especially when regulated by the government or other market forces. That said, firm release dates cost money; “measure twice, cut once” isn’t the most efficient way to launch a modern software product when the market shifts in minutes rather than months. The more flexibility with a release date, the more flexibility with the overall process. An ideal balance for some teams involves Waterfall up front followed by Agile during construction and post construction. Looking at a project in two-phases makes sense, as one phase of the project may dominate over the other depending on the amount of flexibility with scope and timelines.

If you’re able to adopt Agile in its purest form, kudos. For the rest of us who are bound by constraints outside of our control, the best approach is to keep an open mind and address our individual circumstances. As long as the process you adopt is open and transparent, it will reflect reality rather than hide from it. The team and ultimately the quality of software produced by the team will benefit greatly. The software engineering profession will always benefit from less dogma and more skeptical, rational thinking.