Agile, by First Principles

Agile gives me the warm and fuzzy, because like all good systems it can be characterized by a small set of rules, first principles, from which we can derive all other relevant lessons.

first principles

Arguably, the Agile Manifesto, is the most famous attempt to lay out the first principles of the agile approach to software development.  The Agile Manifesto comes in two forms – an elegant and concise one, which is shown below, and a slightly longer, more expository one that I will leave you to discover.

Shorter Agile Manifesto

The problem with first principles

First principles are distillations of wisdom that practitioners gained only after a lot of experience.   For that very reason, they often make little sense to the newbie engineer.  We read the words and some of us ask, “Really?  Why ?”.  We don’t have an answer.  I know that is how it was for me.   Eventually, I acquired the critical mass of hard knocks that were necessary to see the inevitability of those first principles.

These days I seem to practice Agile almost instinctively.   So much so, that I struggle to explain what I am doing, and why I am doing it.   In trying to unpack how I go about software development,  I tried to write down the working set of principles that I work by.

Once the words were on paper, I could see how they derive from the Agile Manifesto.  It reminded of a line from Gandhi, “I’ve traveled so far. And all I’ve done is come back… home.”

A very personal agile manifesto

Below, you will find my working principles for agility.  Notice that my words don’t exactly match those of the Agile Manifesto.  That is as it should be.  My own personal manifesto, so to speak, is necessarily colored by my particular work experiences, and my personal strengths, weaknesses and prejudices.

Again, let me emphasize, this is not meant to be consumed blindly.  Ask yourself why this is valid. Think about what implications these ideas have.  Don’t be surprised if you do not understand or even agree with some of these.   This works for me; this, among other things, makes me an effective IT worker.

  • The only acceptable ‘status’ is working software that delivers business value.
    • Don’t tell me the status. Prove it.
      • Where is the code? Let me read the code. Let me see it running.
      • Where is the test? Is the test correct? Is it adequate? Let me see the test working.
  • The earlier you know the status the better.
    • Short iterations.
    • Frequent feedback.
    • Continuous integration.
  • If you can’t answer the question, “what is done, and what remains”, in terms that the business understands, you’ve got nothing.
    • User stories, in strictly business terms.
  • Change is the only constant.
    • You will never get correct nor complete business requirements at any one instant.  The same applies to solution specifications.  Roll with it.
    • No design nor solution will ever be right the first time.  Roll with it.
    • Priorities will change.  Roll with it.
  • Information is indispensable.   Documentation, and meetings are incidental.
    • Putting words on paper, and communicating are two different things.
    • Just because you talk, talk, and talk, does not mean you are communicating anything useful.
  • If you are not putting software in production, for business to do business with, you’ve got nothing.
    • All the process in the world, all the fancy tools, the “best and brightest” people, mean nothing, if you are not delivering.
    • The answer to every question of the kind, “is this the correct process, is this Scrum, is this Agile”, is one simple thing.
      • If it helps you deliver, yes.  If you are not delivering, none of it matters.
  • No battle plan survives contact with the enemy“.
    • Human beings are unreliable.  Human judgment is unreliable.
      • Estimates are less correct, the farther out into the future they extend.
      • You will never anticipate everything that can go wrong.
      • Every process will break down.
      • Remember, Eisenhower’s advice, “In preparing for battle, I have always found that plans are useless, but planning is indispensable”.
    • You must be able to adjust and keep moving forward.   How?
      • There are no short-cuts.
      • This comes entirely from the attitude, ownership, knowledge, and skill, that people on the ground can bring to bear.

Story points of a user story that spans sprints

We start a story in sprint 7, but we finish the story in sprint 8. Should we count those story points against 7, or against 8?

Barking up the wrong tree

We are asking the wrong question.

It does not matter whether we add those story points to sprint 7, or 8. That line of thought tells us nothing useful about how we are performing the primary task at hand – deliver correct, well built software in a timely manner.

Make no mistake, when a user story slips into a second sprint, you have a problem with how the work is going. That problem, whatever it is, demands examination and remedy. However, deciding which sprint to stick the story points in neither helps the examination of what went wrong, nor does it contribute to correcting the failure.

Big story

The user story may have been too big. Perhaps we bit off more than we could chew. We should have broken the user story into smaller ones. Watch out for that next time.

How do you split a story into smaller ones? That is a separate question. There are a lot of references on that, which I’ll leave you to research.

We are not efficient

Say we decide that we cannot break the story into smaller pieces. Perhaps something is not quite right with our productivity.

In my current place of work, we get issues that can be finished (analyzed, fixed, tested) in a couple of days by a developer that knows the domain, and the system. However we also have folks that take a couple of weeks to finish the same story. Their work often span sprints. These folks have to come up to speed. There is your problem. Address it.

Perhaps the developer finishes his part, but the issue languishes with the testers because they are swamped with other things. This exposes problems with how we have organized our teams. Protect team members from being dragged into work that is extraneous to the current sprint.

We should have known early

We should have known that a story would not finish, well before the end of the sprint, and we should have taken remedial action right then.

If we know our team, and our engineering environment, we should have been aware even before we started, that we would not finish a story. If we cannot make such judgements even after a couple of sprints, the sprints are not teaching us anything. If we are not gaining a more accurate estimate of our team’s capability, sprint by sprint, we are not being Agile. What obstacles keep us from understanding our team’s capacity to deliver? Investigate that.

Sometimes we misread a user story. We clearly see the story only after working on it for a couple of days. At this point it should become clear whether we will finish the story or not. This kind of early feedback is the cornerstone of the Agile way. If you become aware of an out of control story only at the end of a sprint, you are not being Agile. Why is this feedback not happening? Or is it happening, but the information is worthless? Perhaps your team is dissembling, and the scrum master cannot tell fact from fiction. Look into this.

Velocity – Weather or Climate?

It is not hard to see why we fixate on which sprint earns a partially completed story’s points. We are trying to figure out how much our team can deliver in a sprint; the so called Velocity.

Do not get too hung up on Velocity. As Mike Cohn of Mountain Goat Software points out, Velocity is characterized by a dichotomy that is very much like the one between weather and climate. On any given day, weather can be good or bad. You can make valid generalizations about the weather of a particular location, only after observing it over a large span of time. We have a word for that. Climate. Think of Velocity as climate, not weather.

The number and size of user stories that you start and finish within a single sprint, will vary from one sprint to another. Don’t worry about it too much. Over several sprints, you will get a good measure of how much work you can do. Over several sprints, Velocity will reveal itself.

Here are three slides from Mike Cohn’s Scrum Master training, which also make the above point.

Velocity Fluctuates from Sprint to Sprint

Velocity Fluctuates from Sprint to Sprint

Velocity - Some are Outliers, Some are Typical

Velocity – Some are Outliers, Some are Typical

Infer Reliable Velocity From Sample

Infer Reliable Velocity From Sample

Focus on learning from the trials and tribulations of each sprint.  Address these issues as you become aware of them.  Velocity will take care of itself.

User Stories and Building Blocks – Another Difference

An earlier post described how the Agile philosophy implicitly divides software development into two types of work – user stories and building blocks. In addition, the post delved into how to distinguish between the two types of work. Here, I point out another seminal difference between user stories and building blocks.

How much is done?

During the course of a project, a critical question comes up again and again. Business stakeholders, and managers, keep asking, how much is done? What exactly is completed? What remains?

What answer do you think is most useful for them? Investigating that question reveals a defining difference between user stories and building blocks.

Status by User Story

This is a list of features that we must build. Clearly these are user stories.

As a customer service rep, I want to be able to generate an auto insurance quote.

As a customer service rep, I want to be able to email, fax, and mail a paper copy, to the customer.

As a customer service rep, I want to order a customer’s credit report.

As a customer service manager, I want to see all enrollments that are in process at this moment.

When a business stakeholder wants to know what is completed and what remains, we can simply point her to this list of business features. Is an item production ready? Is there a metaphorical green check mark against the item?

This list of work represents the value that business users are looking for, in the system we are building. Hence, the list of user stories communicates status that is immediately relevant to business, in an unambiguous, even visceral manner.

Status by Building Block

Now consider this list of work.

Build the data acces layer of the enrollment sub-system.

Implement the business logic of enrollment as production rules.

Build a REST service wrapper around the Enrollment API.

Build a web based UI for managing enrollments. This is meant for internal processors and customer service reps. This UI will sit directly on top of the business layer.

Build a mobile app for use by customers to manage their enrollments. This will talk to the REST web service.

If you tell business users, you have completed the data access layer, and business layer, does that tell them anything about what they really care about, the business features that we are building? It does not. Business stakeholders really have to wait till all of the above tasks are done before they can begin to evaluate what shape, and how much of the business features, and in what shape they have been delivered. Until then all is uncertain.

Out of this uncertainty is born magic status measurements like 35% done, 60% done, etc. These numbers convey little information that is useful for business stakeholders. Confidence in such status’ are low and always feel pathetically forced.

Thus

You can differentiate between a user story and a building block by what information they convey when we complete them. Completion of a user story unambiguously tells business what features are done, and what remain. Completing a building block leaves those questions unanswered.

User Stories and Building Blocks

When I look at software development from the Agile point of view, I perceive two types of work. One is what Agile calls a user story, and the other, for lack of a better term, let me call, building block.

All of us who have had any truck with Agile, have run up against these two types of work.

Further, many of us are stumped by these questions below.

How to distinguish between the two types of work?  “Is this a user story, could that be a user story, shouldn’t this be a user story”, is a familiar song.

If a task is not a user story, and yet it is a significant effort, how do we fit it into a plan of action whose purpose is to only implement user stories?   In other words, how do I fit the second type of work, the ‘building block’, into an iteration, sprint, time box, or whatever your particular flavor of Agile calls it.

Should this be a user story?

One of the first principles of the Agile philosophy provides the answer.

If you have not delivered value to the end user, you have not done the job.  
The driving focus of the Agile way is delivery to the end user, of a feature 
that she requires, asked for, and can recognize as such.

To illustrate, let us consider two different kinds of projects.

Auto Insurance Policy Administration System

You are building a policy administration system that can manage Auto Insurance policies.

Consider these tasks.

User Story

As a customer service rep, I want to query the DMV for a driver’s motor vehicle report.

This is a business function, expressed strictly in terminology that comes from the business user’s world. When you build this completely, from UI to backend, you will have a full vertical slice of business functionality, which you can set aside. Now, you can change focus to another vertical slice of business functionality, perhaps, querying a vehicle owner’s credit score.

Business folks know the auto insurance business. They do not know software engineering. Business folks can only evaluate the value of the system you are building, in terms of the business functions it helps them perform.

Can I query for the Motor Vehicle Report? Check.

Can I query for the Credit Report? Check.

We can write the sweetest code imaginable, but unless we place these business features in the business users’ hands, our job is incomplete. Such features are user stories.

Building Block – One

Build a library of UI widgets, backed by Twitter Bootstrap, which allows entry of entities in our domain model, like postal addresses, a vehicle description, an accident report, and so on. You should just be able to refer to these widgets, as necessary, in various screens of the UI

Creating such a widget set requires considerable skill, and time.  However what can a policy admin representative do with a UI widget kit, however sophisticated it is? Nothing. The policy rep wants to take a driver’s information over the phone, and prepare an insurance quote for the driver. That is a user story, which will be implemented using, among other things, the above UI widget set. Creating the widget set itself, is not a user story. The widget set is a ‘building block’.

Building Block – More

Build a fault tolerant API that can send any query to DMV.  The API must tolerate connection failures up to a certain number of times, which must be configurable.

When we have built said API, does the business user have a feature that they can do business with? No. However, you can build the ability to request a Motor Vehicle Report, on top of this API.  Only then do you have a tool that a business user can clearly identify as an essential cog of their business.

Here are some more tasks that are merely building blocks in the Policy Adminstration project.

You have to build a data access API that hides the details of your application’s datastore, MongoDB, from the rest of the application. In other words, this is one of the layers in your application.

Your application has to schedule various activities. So you create a simplified facade around some proprietary, and complex scheduler that your company just bought.

After several sprints of work, you decide that a critical API method (EnrollmentService.enroll, for instance), has become too unwieldy and must be refactored, as defined by Martin Fowler et al.   However this is a high risk task, which will require significant skill, effort, and time.

To summarize, a project often has tasks which you require to reliably implement user stories. Completing such a building block puts you on the road to producing a business feature that users recognize, but it does not get you all the way there.

Re-engineer the enrollment, and billing sub-systems

Now, let us consider a different kind of project.

Your policy administration system was delivered and has been in use for 
over a year. Periodically, problems have arisen, but you have kept the system 
going through bug fixes, work arounds, cleaning up data directly, and so on. 
After a year or so, the code has become so baroque, that even bug fixes are 
becoming difficult to do.

So you decide that two critical sub-systems, enrollment, and billing must 
be re-engineered. The UI cannot be touched. All features that the user has 
been using for a year now must stay intact. Any changes you make must be 
under the hood.

Can this project be planned, and executed in an Agile manner? How can we break up the work into user stories, when we are not adding any business features at all?

Agile is not so much about any one kind of project, as it is about how to drive a project to completion.

  • Figure out the primary stakeholder that asked for the project
  • Figure out how that stakeholder views the deliverables of the project.
  • Slice up those deliverables into smaller goals (sometimes called user stories), such that the stakeholder is always able to unambiguously see, in terms that matter to him, what has been completed and what remains to be done.
  • Pick a fixed interval (aka sprint) of no more than 2 or 3 weeks, within which you can start and completely finish one or more of the above user stories
  • Prioritize, and schedule the user stories across the sprints
  • Execute a sprint, examine how you did at the end of the sprint, adjust, and execute next sprint

I can see no reason why we cannot apply the above method to a purely technical project, like the one under consideration. Some technical lead will be responsible for defining what the problem is, and what success looks like. This technical lead should be able to break up the larger goal into smaller ones which are easily measurable.

Perhaps, the enrollment and billing subsystems consist of 3 interfaces, and 3 implementation classes. The interfaces must stay untouched. The 3 implementation classes must be refactored. Perhaps each class can be a ‘user story’.

If those ‘user stories’ turn out to be too big to fit into a sprint, we can break them up into individual interface methods. Perhaps each ‘user story’ is simply the re-engineering of a couple of interface methods.

Perhaps, the enrollment and billing subsystems support a dozen business use cases, and we know exactly which service methods support which use case. Perhaps we could break up the refactoring work according to these business use cases. Each user story consists of refactoring the service methods that support one of those dozen business use cases.

The larger point is that all projects have to be delivered. All projects have some stakeholder who knows what shape that delivery must take. Slices of that delivery are user stories. All other work that must happen in order to deliver those user stories in good order, are building blocks. In this refactoring project, say you want to write automated integration tests, which will ensure that user functionality has not been broken. This would be a ‘building block’. A whole class, refactored to our satisfaction, which passes those integration tests, would be a ‘user story’.

How to fit ‘building blocks’ into sprints?

If the ‘building blocks’ do not already exist, before you start the Agile project, let them be subsumed by the user story.

Say that first story, of the first sprint, is the Motor Vehicle Report query that we saw earlier.

As a customer service rep, I want to query the DMV for a driver’s motor vehicle report.

Building Block – One

Before you are able to start implementing the user story, you need a development environment, which might include a source control project, a build script, private DB schemas for each developer, a continuous integration server, and a test environment where the latest build can always be found.

Already available?

It is possible that much, if not all of this, is already available. You have established a development environment over time, and across previous projects. As part of the first user story, in the first sprint, simply do what little is required to setup your development environment for this particular project.

Starting from scratch?

Perhaps you have decided that from this project onwards, you are switching to Git for source control, and your build scripts will be in Gradle instead of the Maven that you used previously.

As part of the first user story, you prepare your Git repositories, and write whatever Gradle script is required to get you through the first user story. Since this is new, you will spend more time on this, than you would have, if you had stuck with the previous technologies. That is fine. It simply means that it will take longer to complete that same first user story.

Is this Greek to you?

You have never written a build script before. You have never used ANT, nor Maven, and hence it will signficantly longer to learn the Gradle necessary for your build scripts.

This just means that it will take you even longer to complete that first user story, since it will take even longer to create your development environment.

Building blocks simply get factored into Velocity

The more time you spend getting your building blocks in place, the more time it takes to complete a user story. The faster you are able to complete your building blocks, the faster you will complete your user story. That is really all there is to it.

Many factors determine how much time you will spend on the building blocks. Do you already have them built, before starting the user story? Are you accomplished at designing and building the particular building block in question, which means you can get through it quickly?

After using Agile in some form or the other for over 10 years, not always rigorously, nor effectively, I finally took a class on Scrum a couple of weeks ago. It was taught by Mike Cohn, of Mountain Goat Software, one of the earliest practitioners of Scrum for software development, and a prominent proponent of the methodology.

As you might imagine, folks brought up the questions we have been considering. Here are a couple of slides from the class, which speak to the matter.

Should 'building blocks' be assigned their own sprint?

Should ‘building blocks’ be assigned their own sprint?

Let 'building blocks' be subsumed by 'user stories'

Let ‘building blocks’ be subsumed by ‘user stories’

Mike Cohn used the term ‘Architecture’ to refer to what I have been calling ‘building blocks’. Mike posited that you do not want to think of ‘Architecture’ as separate from ‘user stories’. Consider ‘Architecture’ an intrinsic part of ‘user stories’. You will typically spend more time on ‘Architecture’ (my ‘building blocks’) earlier in a sprint, and in the earlier sprints. As you move through the project, if you are doing your engineering right, you will simply use already built pieces, or simply apply previously made design decisions, more and more.

As the above image shows, how good you are at getting the building blocks out the way, affects how much ‘user story’ you complete in a sprint, which in a word, is Velocity.

Look to the first principles

All through the four days of Scrum training with Mountain Goat Software, I heard question, after question, which would prompt the same thought in me. Just look to the first principles of Agile. The answers are there.

Of course, the challenge, for them and for me, is knowing those first principles.