Fighting the Last War — Software Development Beyond Agile and Scrum

"If you don't show Kansas, Oz isn't all that special." 

~ Frank Decker, on adding a dystopian LA backstory to the movie Demolition Man

"Generals always fight the last war"

~ Edward P. Warner (1934)

1. The Scientific Revolution Ahead

For the next Jonas Salk to be a mathematician we need to use new tools to provide a mathematical focus on previously unsolvable problems.  Software lies between that vision and the practical world we live in, but software has long been a gap that has kept mathematical insights from becoming real world solutions.   We have 200,000+ clinical trials' worth of data in storage, but what use are they if you can't search them, analyze them for results and plan next steps from there?   "Big Data" is a catch-all term for the new kinds of analysis that are tailored to massive or genomic data sets, but our software deficiencies only start with wrestling the data.   The world ahead needs two things:  1) a way to channel the myriad human communications  (spoken, texted, emailed, webbed, blogged, fitbitted, jawboned and more) into an Interstellar-tesseract library for review,

and 2) a better way to write the tools and applications that we'll need to navigate and draw wisdom from that tesseract.

2. The World We've Come From

It's said that "generals always fight the last war," so it shouldn't be surprising that while big data tools are new, our software development approaches are still the ones we developed when "the internet changed everything."   Extreme Programming and the Agile Manifesto got us started, and the backsides of the manifesto authors is an appropriate image for how that manifesto suits us now in the Age of Analytics.  Time will make fools of us all, but the principles from 2001 read like platitudes now -- they are not wrong, per se, but a lot has changed since 2001, and we can choose better rules today.  Let's start with some things that have changed:

• Teams are more likely to be distributed now.   Daily stand-ups will have a different flavor when no two people are on the same continent or in the same time zone, rather than the same room.
• Web development has matured.   In the late 90's everyone was a "web newbie."   The game had already changed when David Heinemeier Hansson released a video building a weblog in 15 minutes, and even that is almost 10 years ago now.
• Web tools have matured.   Java, .Net, Rails, Python and PHP are old hat now, and RCS got replaced by Subversion and then Git as the tools raced ahead even faster.
• Communications tools are richer now.   By now most developers have video-conferenced with Facetime or Skype or Lync, IM'd with Yammer or Campfire, wiki'd with Sharepoint or Basecamp, and done code reviews through Git (or read the sordid code history with Git Blame).
• The competition to your software is a lot greater than it was then.   The godfather of agile methodologies, Extreme Programming (XP), was introduced in 1996 by Kent Beck when he was working on a Chrysler payroll project.  Chrysler payroll?   Not exactly the hair-on-fire environment that Jamie Zawinski lived in at Netscape, much less edgier parts of Facebook or Google.    Kent Beck 1) had an internal audience 2) who had no competitor software to choose from, and 3) couldn't effectively decline to use his software on release.    Apple iStore now has 1.5 million iPhone apps, available for review and updated continuously.    Kent Beck did great work, but the iStore represents a massively more competitive environment than he faced in Chrysler.

According to Conway's Law, modern software designs match the communication structures of the organizations that built the software.  Cobol mainframe software was thus the product of The Organizational Man age.  Here I'll posit Repko's corollary, which states that modern software methodologies match the prevailing organizational mores of their era.   Thus it was that 2000-era technology mores that gave us Scrum:

• Here in Payroll...
  • We're all co-located, so we'll improve communications with morning stand-up meetings
  • We don't know how to write requirements that are a) rigorous enough to be developed swiftly, nor b) flexible enough to stand the test of time, so we'll have a Product Owner that we can rap with on a daily basis
  • Our leader is a consultant who doesn't work here, so we'll define the role of Scrum Master and (like our consultant) that role will have power but not authority
  • Web development is new and Java is even newer, so our entire team are newbies.   We'll handle this by
    • letting everyone own all the code -- easier to spot problem-children, and to rebalance the team if we have to "cull the herd"
    • tracking performance daily with check-ins and velocity (what other profession monitors itself daily?)
    • eliminating any concept of seniority -- if nobody knows anything about web or Java, what's the benefit of seniority?
  • We'll prepare code in sub-projects called sprints that last from 2-4 weeks each
    • it's only payroll -- it's not like we're building an operating system or air-traffic control
    • if it were something big like an operating system, we'd just have the sprints go on forever
    • we don't have any functionality (like architectural advances) that can't be fit into a 2-4 week sprint
  • Our goal is to have deliverable software at the end of each sprint.
    • the software doesn't have to be good (or compete with a dozen similar apps on the iStore), it just has to be deliverable

It's not that I don't like Scrum -- it is easily grasped, it provides a structure to software projects, and it's a positive step forward from the Waterfall software development approaches that preceded it.   It is, though, the product of its times and environment, and competitive-market communications and advanced analytics software needn't be saddled with Internet-bubble-Chrysler-payroll software practices.

While better than nothing, many of the "agile" practices that comprise Scrum don't make a lick of sense anymore.   Their era has passed, and as much as people still bow to Imperial Science of Scrum and its Certified Scrum Masters, late-90's-payroll approaches just aren't going to cut it in 2015.

Scrum is the development methodology of the last war.  A quick note about the image at the header of this posting.   Many readers might suspect that the Pearl Harbor attack on December 7, 1941 may have been a turning point in naval warfare -- the specific point at which aircraft carriers (with their bombers and torpedo planes) became THE capital ship for Navy planning.   Some with greater perspective might even flag the earlier Battle of Taranto (Italy) of 11-12 November, 1940 as that turning point.  In fact we're approaching the anniversary of that turning point, the warfare revolution of its day, which came far sooner.

US General Billy Mitchell came out of World War I with the belief that his aircraft -- 1920's-era biplanes seemingly little-advanced beyond what the Wright brothers flew at Kitty Hawk -- could sink any ship of any navy, and he pleaded for a chance to prove it.   His bombers did surprisingly well in early tests, and so he was presented the ultimate challenge:   The Ostfriesland -- pride of the WWI Germany navy, due for demolition but still thought by many (stop me if you've heard this before) to be "unsinkable."   In an earlier effort, US Secretary of the Navy Josephus Daniels smugly proclaimed:

"I'm so confident that neither Army nor Navy aviators can hit the <ship> when she is under way that I would be perfectly willing to be on board her when they bomb her!"

Good thing he didn't.  He would soon discover the truth of Victor Hugo's words "No army can stop an idea whose time has come."  When the paradigm shifts, you're either on the right side of the shift or you are lost to history.    The first bomb fell at 12:17pm on that day 94 years ago -- July 21, 1921.  The 6 bombers came at two minute intervals, with the last bomb falling at 12:27pm.   The new era of warfare dawned just 13 minutes later, when at 12:40pm the mighty Ostfriesland slipped beneath the waves.

It's time to change the way we write software, and I'll write about that new paradigm:

Beyond Scrum -- The Evolution of Software Development

in my next posting.