I've come accross many agile resources or attended education events where presenters essentially suggested that you just need to get permission to set up an agile process, and they go on to describe what those agile process elements "should be", without addressing the problem that, in many corporate development environments, there are factors that prevent you from adopting a full-blown agile approach all at once. There are truly dysfunctional corporate development environments (I suffered under one for almost six years) that would benefit substantially from ANY degree of progress toward agility, but there are a couple major barriers to achieving this kind of gradual transition.

First, there is the subculture of agilists who seem fixated on an "all or nothing" approach, reflected in statements along the lines of "unless you're doing X, Y, and Z, you're doing it wrong and you're not agile". Many senior agile dev. community members have explicitly spoken out against this attitude, as counter-productive and just plain wrong. Kent Beck included an apology in the second edition of his book on XP, specifically to try to address this.

"Critics of the first edition have complained that it tries to force them to program in a certain way... I'm embarrassed to say that was my intention... in this edition, I have tried to rephrase my message in a positive, inclusive way" -- Kent Beck, in the preface to his second edition of "Extreme Programming Explained"

Published works by other industry thought leaders including Mary Poppendieck, Allen Shalloway etc. have made similar points. IBM has pushed for ALL its developers to adopt SOME form of agile, while explicitly REJECTING the idea that you need to follow any specific set of practices or methodology.

Transition to agile and lean at IBM

A related topic of discussion in the community has been on trying to eliminate the squabbling between proponents of different methodologies, in recognition of the fact that, as long as the adopted practices support agile PRINCIPLES, it represents a huge advance over the traditional "plan and direct" approach that still cripples efforts of development groups in many organizations.

The second impediment to charting a gradual path toward agility is figuring out how to do this. What does a "partial" agile process look like? At my last company, we finally managed to get buy-in on adopting some specific agile PRACTICES (continuous integration, unit testing, a project post-mortem, and some amount of static code analyis to try to reign in code quality issues...), but we failed utterly to get commitment to the more important agile PRINCIPLES, with disasterous results. Management still insisted that we divide our three year project into "tasks" which were distributed to 15+ developers in two globally-distributed teams to "estimate", despite the fact that only three of us had significant first-hand experience with the technologies we would be using, and many developers suffered from questionable coding skills (which was something we had specifically hoped to address by moving AWAY from the traditional "go off in your corner and complete your assigned task" approach). In that case, we ended up six months later with half-a-dozen independently implemented feature "fragments" that had never been code reviewed, with no idea of when we might ever see an integrated end-to-end running solution, while we were forced to sit through multiple hours-long meetings where panicked managers tried to figure out how to reign in our schedule, which was already running double the original "estimates". Complete disaster!

One of the things that I like in what I've been reading about Kanban, is in how it tends to overcome initial management (and developer) resistence, by making very MINOR changes to the existing development process, and how the teams seem to naturally adopt more and more agile techniques over the course of the project, as the productivity improvements become obvious. While I've also heard people say essentially the same thing about Scrum ("adopt it and the benefits will quickly become clear"), the problem is that jumping directly from a legacy staged process to full-fledged Scrum is simply too drastic of a change for managers at many organizations, especially if some of those non-agile process elements exist for other reasons, such as a need to conform to process and documentation requirements in a regulated industry, for example. A company may have multiple layers of managers responsible for software project success whose background lies in a product domain rather than software, who have no basis for managing development OTHER than charting detailed estimates and tracking "variation to plan" (which is what most published project management guides are telling them they should do anyway). A corporate dev. group might also contain a large number of developers who still aren't familiar with basic engineering best practice concepts like S.O.L.I.D., who will need alot of mentoring and oversight before they even know how properly "refactored" code should look! These, and many other real-world conditions will inevitably force you to compromise the "purity" of whatever agile methodoloy you may be wanting to apply, but there should still be some way to help organizations make SOME progress, in the hope of getting all the way there EVENTUALLY.

My own take on how to effect a gradual transiton would be to initially forgo some useful agile practices while trying to conserve core agile PRINCIPLES. Setting up peer code review and mentoring to ensure code is kept clean, maintainable, and extensible is more important than insisting that a large team of developers with poor dev. skills write (sloppy) unit tests to support their (convoluted) code. Generating some amount of unecessary documentation is preferrable to having your product fail a customer audit, or to have your project remain unfunded by corporate managers who don't buy into your "Trust us" argument. By choosing your battles and focusing on a few core issues like maintaining high code quality, organizing short iterations and being able to sqeeze in at least SOME amount of customer proxy feeback (usually from a busy marketing rep. who doesn't understand why you can't just go off and build the entire app. as specified), limiting the number of tasks under development to force developers to leave their silos and collaborate, holding regular informal reviews to try to improve the dev. process based on observed results... you are surreptitiously "infecting" the organization with traces of agile, so you can breakdown resistance and eventually take over the host.

Has anyone else had first-hand experience with "partially-agile" projects?

Andrew,
My own answer to your question, "Is it possible to go agile gradually?" is that it absolutely is, and further more it must be a gradual process in many, especially large and established, organizations. I worked for a very large company that, to this day, is primarily a Waterfall shop and saw first hand that Agile is not simply a process-shift but it is in many ways a culture shift. As you have pointed out many Project Managers are familiar and comfortable with the idea of having long-term estimates that they can use as a baseline to measure progress. I have also seen a lot of resistance to the idea of a Backlog, primarily because they believe that "everything" must be delivered to be successful. Suggesting that some tasks might not be completed, simply does not sit well. Convincing an establishment of waterfall-minded managers to try Agile will not be done all at once. It will be painful and slow and a single failed project could set the "movement" back significantly.

As for the "All or nothing" point of view, I will agree that certain tenants are a requirement for any development group to call itself "agile". For example, locked Sprints with well defined and immutable backlogs are core to the process and if this level of support cannot be given to the team then it is unlikely that an Agile process will take root and be successful. On the other hand I think TDD and Continuous Integration are supporting techniques that complement an Agile methodology but are not central to its success. In the end culture is the catalyst, if it is resistant to Agile, adoption will be slow or not at all.

Hi, Kevin.

Although its usually associated with unit testing, I'd put a high priority on adopting Continuous Integration, because it is trivial to set up, free, and doesn't impose any obvious changes to the existing dev. process... while it solves the serious problem of undisciplined developers checking in code that breaks the build, eliminates manual integration "build day"s, and gives you an infrastructure to let you gradually introduce unit testing later.

The value of a "locked sprint" was something I always took for granted too, until I started reading how experienced Scrum teams have been discovering they no longer needed them once they started tracking work in progress, by extending Scrum with Kanban. Using this technique (it seems too lightweight to call it a process or methodology), teams continue to schedule weekly or bi-weekly retrospectives to critique and refine their process, and continue to uphold all the traditional agile practice expectations of self-organizing teams, with rigorous attention to code quality etc. etc., but attempts to estimate how long work items will take in order to try to fit them to a "locked" timebox of arbitrary length gradually gives way to just tweaking the number of items permitted to be in progress at any one time, and tracking the actual average time each item took to complete.

From the product owner's perspective, rather than being expected to agree what should be done in the next two weeks (which may involve feature FRAGMENTS that are difficult for them to evaluate or even care about), they are simply restricted in the number of backlog items they can have "on deck" at any time, for the developers to work on next. Review of completed items can be scheduled separately from developer retrospectives, at points that may be "more natural" to product owners, since they involve entire features.

I found it interesting that teams reported they didn't set out to explicitly try to eliminate estimates, but that the need for them seemed to fall away naturally, as managers were satisfied to receive the information they needed in the form of actual work item "cycle time", and "process" was essentially reduced to seeking to eliminate delays in processing individual backlog items.

Of course, I haven't had the chance to try this in a production environment yet, but it sounds interesting.

* NOTE: description of Kanban above has been copied to another thread, Extending Scrum With Kanban

Just came across this post by Mark Needham, which discusses the question:
can a company be partly Agile, or is it all-or-nothing?
Mathias