Agile, by First Principles

Agile gives me the warm and fuzzy, because like all good systems it can be characterized by a small set of rules, first principles, from which we can derive all other relevant lessons.

first principles

Arguably, the Agile Manifesto, is the most famous attempt to lay out the first principles of the agile approach to software development.  The Agile Manifesto comes in two forms – an elegant and concise one, which is shown below, and a slightly longer, more expository one that I will leave you to discover.

Shorter Agile Manifesto

The problem with first principles

First principles are distillations of wisdom that practitioners gained only after a lot of experience.   For that very reason, they often make little sense to the newbie engineer.  We read the words and some of us ask, “Really?  Why ?”.  We don’t have an answer.  I know that is how it was for me.   Eventually, I acquired the critical mass of hard knocks that were necessary to see the inevitability of those first principles.

These days I seem to practice Agile almost instinctively.   So much so, that I struggle to explain what I am doing, and why I am doing it.   In trying to unpack how I go about software development,  I tried to write down the working set of principles that I work by.

Once the words were on paper, I could see how they derive from the Agile Manifesto.  It reminded of a line from Gandhi, “I’ve traveled so far. And all I’ve done is come back… home.”

A very personal agile manifesto

Below, you will find my working principles for agility.  Notice that my words don’t exactly match those of the Agile Manifesto.  That is as it should be.  My own personal manifesto, so to speak, is necessarily colored by my particular work experiences, and my personal strengths, weaknesses and prejudices.

Again, let me emphasize, this is not meant to be consumed blindly.  Ask yourself why this is valid. Think about what implications these ideas have.  Don’t be surprised if you do not understand or even agree with some of these.   This works for me; this, among other things, makes me an effective IT worker.

  • The only acceptable ‘status’ is working software that delivers business value.
    • Don’t tell me the status. Prove it.
      • Where is the code? Let me read the code. Let me see it running.
      • Where is the test? Is the test correct? Is it adequate? Let me see the test working.
  • The earlier you know the status the better.
    • Short iterations.
    • Frequent feedback.
    • Continuous integration.
  • If you can’t answer the question, “what is done, and what remains”, in terms that the business understands, you’ve got nothing.
    • User stories, in strictly business terms.
  • Change is the only constant.
    • You will never get correct nor complete business requirements at any one instant.  The same applies to solution specifications.  Roll with it.
    • No design nor solution will ever be right the first time.  Roll with it.
    • Priorities will change.  Roll with it.
  • Information is indispensable.   Documentation, and meetings are incidental.
    • Putting words on paper, and communicating are two different things.
    • Just because you talk, talk, and talk, does not mean you are communicating anything useful.
  • If you are not putting software in production, for business to do business with, you’ve got nothing.
    • All the process in the world, all the fancy tools, the “best and brightest” people, mean nothing, if you are not delivering.
    • The answer to every question of the kind, “is this the correct process, is this Scrum, is this Agile”, is one simple thing.
      • If it helps you deliver, yes.  If you are not delivering, none of it matters.
  • No battle plan survives contact with the enemy“.
    • Human beings are unreliable.  Human judgment is unreliable.
      • Estimates are less correct, the farther out into the future they extend.
      • You will never anticipate everything that can go wrong.
      • Every process will break down.
      • Remember, Eisenhower’s advice, “In preparing for battle, I have always found that plans are useless, but planning is indispensable”.
    • You must be able to adjust and keep moving forward.   How?
      • There are no short-cuts.
      • This comes entirely from the attitude, ownership, knowledge, and skill, that people on the ground can bring to bear.

Is it a Bug or an Improvement – Irrelevant to agility

I see development teams struggling with the question – is this a bug, or an improvement.  A team that is agile (exhibits agility) would not care.

All the same to me

Tickets come my way.   Some are labelled bugs, and others, improvements.   To me, whose only mandate is to get the damn work done (aka agility), they all look the same.  What I do in response to either a bug or an improvement, is exactly the same.

All of the work requires analysis.   Gather all of the information that is available (documentation from client and analysts, production data, existing code, other tickets,  tests), and separate the essential business requirement from the solution that supports the requirement.  Then make sure that all stakeholders (clients, analysts, managers, developers, testers) have the same view of the matter, problem and solution.  Often, what they ask for is not what they need, and it is sometimes necessary to improve the solutions they specify. This requires constant and nimble communication.   Next construct the solution, which includes testing what we have constructed.  Finally, deliver the solution.

Every ticket is this same dance.

Why care, exactly

I can only think of two reasons for fixating on the difference between a bug and improvement.

First, the green-eyeshade brigade use the difference to figure out who to charge for the work.  A bug forces the development team to swallow the cost.  An improvement can be thrown into the client’s bill.

Second, it might help us figure out who to blame, or to use a more charitable perspective, it might point out where there is room for improvement.  A bug implies the developer messed up.  In cases where the bug is discovered in production, the finger sometimes points to the tester.   An improvement implies the business analyst, or perhaps the client missed something the first time around.

Of the above two reasons, I have sympathy only for the former, the billing problem.  This is mostly because I am low on the totem pole, and know very little about money matters.   Allow me to punt on this.

The latter reason leaves me cold, as you will see below,

What am I gonna believe – statistics or my lying eyes

Numbers can lie.

It is hard to have faith in statistics that classify work as bugs and improvements, when everyday I see my comrades-in-arms struggle to distinguish between the two.  More often than not, we shove tickets into one bucket or the other so work can move on. Typically, some power-that-be is breathing down our necks, or the task management tool is forcing us to make a selection.  Come time to review the work, we all know that the pretty pie charts, which the managers pass around, are nonsense.

I don’t need statistics to know the folks that I work with.  

I work with business folks, development managers, analysts, developers, testers, tech-writers, and so on.   After I spend a couple of weeks in the trenches with them, I know what their strengths and weaknesses are (don’t look now, that’s agility).

Take a developer for instance.   I read a developer’s code.  That tells me how she thinks through a problem.  I know if she can tell a story in simple, straightforward terms.   Can she find the shortest, most direct path between points A and B?

Consider a business analyst.   I read the documentation he creates.  That tells me if he simply knows the business or if he indeed is good at observation and analysis.  Does he see only the surface of things, or can he lift the hood, and identify the patterns and sub-structure that the surface covers?  I will know if his communication skills are adequate.   Can he accurately, clearly, describe what the business is, and what they are asking for?  The more I know the business (agility) the easier this gets.

I use the applications that they specify and build.  I know if they have empathy.  Do they know their users?  Can they walk in the user’s shoes?

I watch how they negotiate one on one conversations, and meetings.   This tells me if they know how to listen.

You get the idea.  Every little thing that a person does is a breadcrumb I can follow, especially if I know how to do that work myself (agility).

My recommendation – chuck it

If at all, only the bean counters need to worry about whether a piece of work is a bug or an improvement. Hide this question from the people that actually do the work. This question is irrelevant to the conduct of the work.   If you hear someone in your development team kvetching about bugs and improvements, make them buy lunch for the whole team (agility).

Focus on Agility, not Agile

For some time now, folks have been emphasizing that we have lost sight of agility, while frantically pursuing Agile.   There is little value in dogmatically following one Agile methodology or the other, Scrum for instance, while loosing sight of underlying purpose of such methods(in a word, agility).

So much Agile, so little Agile

To put it another way, Scrum, in and of itself, is not important.  Rather, why Scrum makes the recommendations that it does, is the important thing.

Take just one small example.

Scrum recommends we have a daily standup meeting.   I have been in teams that hold these meetings religiously.   However, it soon becomes clear to everyone attending the meeting that we are just going through the motions.   The meeting is just another way to waste 15, to 30 minutes of time every day.  In these environments I learned precious little in standup meetings.  Well, I did learn how many different ways there are to talk a lot without saying anything.

When I really want help from a team member I find a way to talk to that person on my own, one on one.   If someone else wants something from me, they come and find me, either electronically, or physically.

If I want to learn the status of the work, in my capacity as a tech or team lead, I pull down the code, build and run it.   I test the code myself, and I learn things that nobody tells me in standup meetings.

Information is essential.  The meeting is not.

If I was working with peers whose skills I had confidence in, and whose word I could take implicitly, the standup meeting may give me the information I am seeking.   However, more often than not, what I hear in a standup meeting falls well short of a full wallet.   Fine. I am not an engineer for nothing.  I don’t need that standup meeting to know exactly where each of my team members are.  I have other ways of skinning this cat.  This is not Scrum.  However, it surely is agility.

From our man in India

Man walks to the barber through rush hour traffic.  Out of 8 million ways to die, man feels like he has escaped 6 million of them.

Man has random thought on Day 5 – Is every f:@($ing family as fiercely funny as mine?

Startling notion passes through man’s head – My lime rice is better than my Mom’s. To his surprise, he is not incinerated by bolt of lightning.

Man has family in India.  Man has family in the US.  Man learns unwanted lesson – he cannot do justice to both at the same time.

Man’s Peddamma passed away in India.  She was a force of nature.  She was a pain in the ass (all forces of nature are pains in the ass).   Peddamma and Peddananna were like one of those large banyan trees, which provided shade and sustenance to a large extended family (10 aunts and uncles, 33 first cousins).  Man’s immediate family had even more special treatment.  His Dad is the youngest of the 10 siblings.  Grand Dad died early, and Peddamma and Peddananna raised his Dad to manhood.  That seamlessly extended to man’s sister and him.  Till they were about 10, they resided under his Peddamma’s stern and attentive hand.  However, for a long while now, his life and his Peddamma’s life did not intersect except for short periodic visits.  He left the protection of that banyan tree years ago.  She had been ailing for a long time.  And she was 92.  So when she finally said the big Enough, man felt reverence, and affection, but little pain.  Her passing was sad, but it was in the natural scheme of things.  It was not a tragedy.  If anything, the moment called for a celebration of her life.  My Peddamma lived a big life.

Meanwhile.  In the US.  God liberated a beloved aunt from her pain.  In return her family had to let her go.  To the Sweet Hereafter.

User experience in a back-office application

I can’t ever seem to explain what ‘user experience‘ is, leave alone make the case that it is important.  I say ‘user experience‘ and people hear ‘user interface‘.

User experience in an in-house, back-office application

Let’s try this.

Imagine I am speaking to a bean counter type middle manager in some sclerotic insurance company. I say this to him.

You make sure that the user knows the business.

I will build a system that a user can run the business with.

The user will not require training to use the system.

The system is not the business.

The system is a transparent tool that anyone with business knowledge will simply ‘get’.

The system will ensure that the user executes the business correctly, and efficiently.

New Enrollment into a Medicare Prescription Drug Plan

Consider new enrollment into a Medicare Part D plan (Prescription Drug Plan).  The insurance processor must know the business, described below.

  • A person comes to us, asking to enroll in a prescription drug plan.
  • In response we send a query to CMS (Centers for Medicare and Medicaid Services) to confirm that the applicant is eligible for the benefits she is asking for.  In industry jargon, this is the BEQ (Batch Eligibility Query).
  • If CMS finds that the applicant is eligible, we complete the application, and formally notify CMS that the person has enrolled into a Medicare Prescription Drug Plan.   This happens through the MARx Batch Input Transaction File, which we send to CMS every day.
  • Then we wait for CMS to tell us that it has accepted this person’s enrollment.  This is through the Transaction Reply Report (aka TRR), which CMS sends us daily.
  • Now, the member is completely enrolled.

Say we build a software system that allows the processor to execute the above workflow.   In addition, say we promise that the processor can simply be put in front of the system, and she will be able figure out how to perform the business without any further training from us.

More than ‘user interface’

If we can build such a system, we will have mastered user experience.

Think about what it takes to accomplish that.

Clearly we are talking about more than proficiency in HTML and Javascript.  At the very least, we the engineers, have to know the business, and we have to know the business users.

Process is very often, much ado about little

The first principle in the Agile Manifesto is “Individuals and Interactions, over processes and tools”. A recent development task illustrated that point starkly – skill trumps process.

Time’s a wastin ….

I was working through a weekend, exhaustively specifying the UI (mockups, WORD docs, UML State diagrams) for a recent user story, when it occurred to me that we were wasting our time.

Here is more or less the user story I was working on.

As a documentation processor, I must be able to review all of the documents in a mailbag. Upon review, I must be able to accept, or reject each of these documents.

In the time that it was taking to describe the UI to its last gory detail,  I could create a working prototype of the UI, which would be close to 80% of what the user needs.  The business folks could take the prototype for a spin, and give me feedback, which could quickly iron out the remaining 20% of the work.

See, we are building this for processors in an insurance company.   They are not looking for sophisticated, space age UI.  They just need something that is easy to understand, learn, and allows them to do the work efficiently.

Between the business analyst and me, we could have the UI squared away in a couple of weeks – a single sprint.  Add a tester if you must.

I will test every damn line of code I write anyway, before this leaves my hands.   The tester is only a second line of defense.   Even without that tester, I can get this very close to done.

So why am I specifying the UI when I can just build the thing myself?  I am having to do this because, collectively, the primary resources that are assigned to this task cannot or will not, produce an effective UI on their own.  That amazes me.  I am shaking my head as I write this.   If you have several years of experience with this problem domain, with these business analysts, with these end users, and with the same UI technology, you should be able to take the user story, fill in all the missing details, and end up at the UI.

Again, here is the user story, for reference.

As a documentation processor, I must be able to review all of the documents in a mailbag. Upon review, I must be able to accept, or reject each of these documents.

Let’s think through this.

Analysis

  • Users have to review all the documents in the mail bag.
    • This means you have to list the documents to the user
  • So how do you present a list of documents.
    • A table, where each column is some attribute of the document, and each row is a document.
  • How do you make it easy for a user to study a large list of data that is in a table?
    • Allow filtering and sorting by various attributes.
    • Break up the list into several pages, and allow the user to page
      • Or simply allow the user to scroll up and down the large list.
  • How do you help the user make changes to large amounts of data safely, and correctly?
    • You must ask the user to confirm a large change.  Give her a chance to back out.
    • You must provide the user a way to undo a change.  Mistakes will happen.
  • What attributes (columns) of the documents must be included in the list
    • Leave out the attributes of the documents that are implementation-centric, which users have no need to know.
    • Pick the attributes that are business information.  Make a guess on which of these the user might be interested in.   Changing this list after review should not be hard.

Can’t the developer do this?

Analysis

If you have been developing UI for any time at all, you should be able to do the analysis I just did, and arrive at a similar, if not the same place.

Business Info

You have been working in this domain for several years.  That means you must already have all the business information you need.

You must know the business-centric attributes that characterize the mailbag and documents.   These become the columns of your listing, and suggest what the user can sort and filter on.

Language and jargon that end-users will recognize must also be familiar to you by now.  This is where static labels, feedback messages and such will come from.

Implementation

You have been working with your UI framework for several years.  So you should know how to implement the various UI features mentioned above – a table of data, sorting, filtering, paging, scrolling.

But none of this happened

But none of this happened, and a UI designer had to be brought in to spell everything out.   Why?

The Developer

Though they were well-versed with the UI framework, the developers have little knowledge of, or interest in, what makes an effective UI.

They are happy to be told what to build.  They want to be told what to build.  When left to their own initiative earlier, they produced a UI that so defied common sense and business reality, that the end-users’ job turned into a labor-intensive, error-prone, hell.

The Tester

If the testers know the lingua franca of the business (English, in this case), if the testers know the business, and if they know the end-users, the testers can judge if the UI satisfies the business need, simply by relying on the 2 sentence user story.

Unfortunately, testers’ familiarity with the business is uneven.  They are simply kids that compare a spec to delivered software.  If the spec is incomplete, or incorrect, these testers are thrown off course.

Further, more often than not, the testing leadership subscribes to the view that testers must only work off an exhaustively detailed spec.  Applying their own knowledge of the business, their own analytical and communication skills, is considered risky. The focus on learning the business themselves is weak.

Manager

Well, managers have all the power, which makes them responsible for everything, doesn’t it?

Who are they hiring?

The UI developer does not exhibit knowledge of effective UI design.    So how did we end up hiring him?

Say the manager that is evaluating the developer does not know anything about UI design himself.   So the manager cannot determine the developer’s competence in the matter.  That is reasonable and common.   A manager is constantly confronted with problems in which he does not have direct expertise.   Yet, she must hire people that will be responsible for solving that problem.    In that situation, a manager has to be able to recognize competence (and conversely, incompetence) as a characteristic separate from concretely observable skill.   Competence has an aura, a body language, a speech pattern, a look in the eye, as does incompetence. You have to be able to smell the bullshit before you see it.

When managers neither have concrete expertise, nor do they have a nose for fakery, we end up with these half-baked developers that require to be spoon fed.

Bad drivers

Many resources are able to learn and do more if we create the right environment.  This is usually some combination of large grained carrot and stick, coupled with the pushing of buttons that we can only learn by getting to know the resource personally.

Effective UI design for the corporate world, is a skill that can be learned by just about anyone in a few weeks, or over the course of a few UI development tasks.  If a developer has worked on UI development for going on three years, and still does not have a coherent, useful view of the matter, the leadership has been asleep at the wheel.

The same argument applies to the testers.   The management enabled them to be unthinking, ‘garbage in garbage out’ type grunts.

Skill trumps process

What a business analyst and a skilled engineer can take care, is now distributed over a whole platoon – The business analyst, the UI designer, the developer, the tester, tech and test leads to oversee the grunts, managers to oversee all of this.

Where the work could simply be a conversation between two people, now is a multi-step workflow involving many more people than that.  Two people getting a task done is just work.    Eight people, each with their own inputs, and deliverables, working on narrow parts of the whole, and that with their vision constrained by blinders, is a process.

What does the process buy us?   Nothing that I can see.  I see overhead. An 8 to 10 person team requires more management and book-keeping than a 2 person team. I see a machine with more moving parts, which exponentially increases the complexity, the amount of co-ordination required, and the number of ways the process can fail.

One or two committed, skilled engineers can replace the whole process. Why not learn to identify, hire and hold on to such resources? Why the fixation on process? Do you want get work done?  Or do you want to run a process?

Story points of a user story that spans sprints

We start a story in sprint 7, but we finish the story in sprint 8. Should we count those story points against 7, or against 8?

Barking up the wrong tree

We are asking the wrong question.

It does not matter whether we add those story points to sprint 7, or 8. That line of thought tells us nothing useful about how we are performing the primary task at hand – deliver correct, well built software in a timely manner.

Make no mistake, when a user story slips into a second sprint, you have a problem with how the work is going. That problem, whatever it is, demands examination and remedy. However, deciding which sprint to stick the story points in neither helps the examination of what went wrong, nor does it contribute to correcting the failure.

Big story

The user story may have been too big. Perhaps we bit off more than we could chew. We should have broken the user story into smaller ones. Watch out for that next time.

How do you split a story into smaller ones? That is a separate question. There are a lot of references on that, which I’ll leave you to research.

We are not efficient

Say we decide that we cannot break the story into smaller pieces. Perhaps something is not quite right with our productivity.

In my current place of work, we get issues that can be finished (analyzed, fixed, tested) in a couple of days by a developer that knows the domain, and the system. However we also have folks that take a couple of weeks to finish the same story. Their work often span sprints. These folks have to come up to speed. There is your problem. Address it.

Perhaps the developer finishes his part, but the issue languishes with the testers because they are swamped with other things. This exposes problems with how we have organized our teams. Protect team members from being dragged into work that is extraneous to the current sprint.

We should have known early

We should have known that a story would not finish, well before the end of the sprint, and we should have taken remedial action right then.

If we know our team, and our engineering environment, we should have been aware even before we started, that we would not finish a story. If we cannot make such judgements even after a couple of sprints, the sprints are not teaching us anything. If we are not gaining a more accurate estimate of our team’s capability, sprint by sprint, we are not being Agile. What obstacles keep us from understanding our team’s capacity to deliver? Investigate that.

Sometimes we misread a user story. We clearly see the story only after working on it for a couple of days. At this point it should become clear whether we will finish the story or not. This kind of early feedback is the cornerstone of the Agile way. If you become aware of an out of control story only at the end of a sprint, you are not being Agile. Why is this feedback not happening? Or is it happening, but the information is worthless? Perhaps your team is dissembling, and the scrum master cannot tell fact from fiction. Look into this.

Velocity – Weather or Climate?

It is not hard to see why we fixate on which sprint earns a partially completed story’s points. We are trying to figure out how much our team can deliver in a sprint; the so called Velocity.

Do not get too hung up on Velocity. As Mike Cohn of Mountain Goat Software points out, Velocity is characterized by a dichotomy that is very much like the one between weather and climate. On any given day, weather can be good or bad. You can make valid generalizations about the weather of a particular location, only after observing it over a large span of time. We have a word for that. Climate. Think of Velocity as climate, not weather.

The number and size of user stories that you start and finish within a single sprint, will vary from one sprint to another. Don’t worry about it too much. Over several sprints, you will get a good measure of how much work you can do. Over several sprints, Velocity will reveal itself.

Here are three slides from Mike Cohn’s Scrum Master training, which also make the above point.

Velocity Fluctuates from Sprint to Sprint

Velocity Fluctuates from Sprint to Sprint

Velocity - Some are Outliers, Some are Typical

Velocity – Some are Outliers, Some are Typical

Infer Reliable Velocity From Sample

Infer Reliable Velocity From Sample

Focus on learning from the trials and tribulations of each sprint.  Address these issues as you become aware of them.  Velocity will take care of itself.

User Stories and Building Blocks – Another Difference

An earlier post described how the Agile philosophy implicitly divides software development into two types of work – user stories and building blocks. In addition, the post delved into how to distinguish between the two types of work. Here, I point out another seminal difference between user stories and building blocks.

How much is done?

During the course of a project, a critical question comes up again and again. Business stakeholders, and managers, keep asking, how much is done? What exactly is completed? What remains?

What answer do you think is most useful for them? Investigating that question reveals a defining difference between user stories and building blocks.

Status by User Story

This is a list of features that we must build. Clearly these are user stories.

As a customer service rep, I want to be able to generate an auto insurance quote.

As a customer service rep, I want to be able to email, fax, and mail a paper copy, to the customer.

As a customer service rep, I want to order a customer’s credit report.

As a customer service manager, I want to see all enrollments that are in process at this moment.

When a business stakeholder wants to know what is completed and what remains, we can simply point her to this list of business features. Is an item production ready? Is there a metaphorical green check mark against the item?

This list of work represents the value that business users are looking for, in the system we are building. Hence, the list of user stories communicates status that is immediately relevant to business, in an unambiguous, even visceral manner.

Status by Building Block

Now consider this list of work.

Build the data acces layer of the enrollment sub-system.

Implement the business logic of enrollment as production rules.

Build a REST service wrapper around the Enrollment API.

Build a web based UI for managing enrollments. This is meant for internal processors and customer service reps. This UI will sit directly on top of the business layer.

Build a mobile app for use by customers to manage their enrollments. This will talk to the REST web service.

If you tell business users, you have completed the data access layer, and business layer, does that tell them anything about what they really care about, the business features that we are building? It does not. Business stakeholders really have to wait till all of the above tasks are done before they can begin to evaluate what shape, and how much of the business features, and in what shape they have been delivered. Until then all is uncertain.

Out of this uncertainty is born magic status measurements like 35% done, 60% done, etc. These numbers convey little information that is useful for business stakeholders. Confidence in such status’ are low and always feel pathetically forced.

Thus

You can differentiate between a user story and a building block by what information they convey when we complete them. Completion of a user story unambiguously tells business what features are done, and what remain. Completing a building block leaves those questions unanswered.

User Stories and Building Blocks

When I look at software development from the Agile point of view, I perceive two types of work. One is what Agile calls a user story, and the other, for lack of a better term, let me call, building block.

All of us who have had any truck with Agile, have run up against these two types of work.

Further, many of us are stumped by these questions below.

How to distinguish between the two types of work?  “Is this a user story, could that be a user story, shouldn’t this be a user story”, is a familiar song.

If a task is not a user story, and yet it is a significant effort, how do we fit it into a plan of action whose purpose is to only implement user stories?   In other words, how do I fit the second type of work, the ‘building block’, into an iteration, sprint, time box, or whatever your particular flavor of Agile calls it.

Should this be a user story?

One of the first principles of the Agile philosophy provides the answer.

If you have not delivered value to the end user, you have not done the job.  
The driving focus of the Agile way is delivery to the end user, of a feature 
that she requires, asked for, and can recognize as such.

To illustrate, let us consider two different kinds of projects.

Auto Insurance Policy Administration System

You are building a policy administration system that can manage Auto Insurance policies.

Consider these tasks.

User Story

As a customer service rep, I want to query the DMV for a driver’s motor vehicle report.

This is a business function, expressed strictly in terminology that comes from the business user’s world. When you build this completely, from UI to backend, you will have a full vertical slice of business functionality, which you can set aside. Now, you can change focus to another vertical slice of business functionality, perhaps, querying a vehicle owner’s credit score.

Business folks know the auto insurance business. They do not know software engineering. Business folks can only evaluate the value of the system you are building, in terms of the business functions it helps them perform.

Can I query for the Motor Vehicle Report? Check.

Can I query for the Credit Report? Check.

We can write the sweetest code imaginable, but unless we place these business features in the business users’ hands, our job is incomplete. Such features are user stories.

Building Block – One

Build a library of UI widgets, backed by Twitter Bootstrap, which allows entry of entities in our domain model, like postal addresses, a vehicle description, an accident report, and so on. You should just be able to refer to these widgets, as necessary, in various screens of the UI

Creating such a widget set requires considerable skill, and time.  However what can a policy admin representative do with a UI widget kit, however sophisticated it is? Nothing. The policy rep wants to take a driver’s information over the phone, and prepare an insurance quote for the driver. That is a user story, which will be implemented using, among other things, the above UI widget set. Creating the widget set itself, is not a user story. The widget set is a ‘building block’.

Building Block – More

Build a fault tolerant API that can send any query to DMV.  The API must tolerate connection failures up to a certain number of times, which must be configurable.

When we have built said API, does the business user have a feature that they can do business with? No. However, you can build the ability to request a Motor Vehicle Report, on top of this API.  Only then do you have a tool that a business user can clearly identify as an essential cog of their business.

Here are some more tasks that are merely building blocks in the Policy Adminstration project.

You have to build a data access API that hides the details of your application’s datastore, MongoDB, from the rest of the application. In other words, this is one of the layers in your application.

Your application has to schedule various activities. So you create a simplified facade around some proprietary, and complex scheduler that your company just bought.

After several sprints of work, you decide that a critical API method (EnrollmentService.enroll, for instance), has become too unwieldy and must be refactored, as defined by Martin Fowler et al.   However this is a high risk task, which will require significant skill, effort, and time.

To summarize, a project often has tasks which you require to reliably implement user stories. Completing such a building block puts you on the road to producing a business feature that users recognize, but it does not get you all the way there.

Re-engineer the enrollment, and billing sub-systems

Now, let us consider a different kind of project.

Your policy administration system was delivered and has been in use for 
over a year. Periodically, problems have arisen, but you have kept the system 
going through bug fixes, work arounds, cleaning up data directly, and so on. 
After a year or so, the code has become so baroque, that even bug fixes are 
becoming difficult to do.

So you decide that two critical sub-systems, enrollment, and billing must 
be re-engineered. The UI cannot be touched. All features that the user has 
been using for a year now must stay intact. Any changes you make must be 
under the hood.

Can this project be planned, and executed in an Agile manner? How can we break up the work into user stories, when we are not adding any business features at all?

Agile is not so much about any one kind of project, as it is about how to drive a project to completion.

  • Figure out the primary stakeholder that asked for the project
  • Figure out how that stakeholder views the deliverables of the project.
  • Slice up those deliverables into smaller goals (sometimes called user stories), such that the stakeholder is always able to unambiguously see, in terms that matter to him, what has been completed and what remains to be done.
  • Pick a fixed interval (aka sprint) of no more than 2 or 3 weeks, within which you can start and completely finish one or more of the above user stories
  • Prioritize, and schedule the user stories across the sprints
  • Execute a sprint, examine how you did at the end of the sprint, adjust, and execute next sprint

I can see no reason why we cannot apply the above method to a purely technical project, like the one under consideration. Some technical lead will be responsible for defining what the problem is, and what success looks like. This technical lead should be able to break up the larger goal into smaller ones which are easily measurable.

Perhaps, the enrollment and billing subsystems consist of 3 interfaces, and 3 implementation classes. The interfaces must stay untouched. The 3 implementation classes must be refactored. Perhaps each class can be a ‘user story’.

If those ‘user stories’ turn out to be too big to fit into a sprint, we can break them up into individual interface methods. Perhaps each ‘user story’ is simply the re-engineering of a couple of interface methods.

Perhaps, the enrollment and billing subsystems support a dozen business use cases, and we know exactly which service methods support which use case. Perhaps we could break up the refactoring work according to these business use cases. Each user story consists of refactoring the service methods that support one of those dozen business use cases.

The larger point is that all projects have to be delivered. All projects have some stakeholder who knows what shape that delivery must take. Slices of that delivery are user stories. All other work that must happen in order to deliver those user stories in good order, are building blocks. In this refactoring project, say you want to write automated integration tests, which will ensure that user functionality has not been broken. This would be a ‘building block’. A whole class, refactored to our satisfaction, which passes those integration tests, would be a ‘user story’.

How to fit ‘building blocks’ into sprints?

If the ‘building blocks’ do not already exist, before you start the Agile project, let them be subsumed by the user story.

Say that first story, of the first sprint, is the Motor Vehicle Report query that we saw earlier.

As a customer service rep, I want to query the DMV for a driver’s motor vehicle report.

Building Block – One

Before you are able to start implementing the user story, you need a development environment, which might include a source control project, a build script, private DB schemas for each developer, a continuous integration server, and a test environment where the latest build can always be found.

Already available?

It is possible that much, if not all of this, is already available. You have established a development environment over time, and across previous projects. As part of the first user story, in the first sprint, simply do what little is required to setup your development environment for this particular project.

Starting from scratch?

Perhaps you have decided that from this project onwards, you are switching to Git for source control, and your build scripts will be in Gradle instead of the Maven that you used previously.

As part of the first user story, you prepare your Git repositories, and write whatever Gradle script is required to get you through the first user story. Since this is new, you will spend more time on this, than you would have, if you had stuck with the previous technologies. That is fine. It simply means that it will take longer to complete that same first user story.

Is this Greek to you?

You have never written a build script before. You have never used ANT, nor Maven, and hence it will signficantly longer to learn the Gradle necessary for your build scripts.

This just means that it will take you even longer to complete that first user story, since it will take even longer to create your development environment.

Building blocks simply get factored into Velocity

The more time you spend getting your building blocks in place, the more time it takes to complete a user story. The faster you are able to complete your building blocks, the faster you will complete your user story. That is really all there is to it.

Many factors determine how much time you will spend on the building blocks. Do you already have them built, before starting the user story? Are you accomplished at designing and building the particular building block in question, which means you can get through it quickly?

After using Agile in some form or the other for over 10 years, not always rigorously, nor effectively, I finally took a class on Scrum a couple of weeks ago. It was taught by Mike Cohn, of Mountain Goat Software, one of the earliest practitioners of Scrum for software development, and a prominent proponent of the methodology.

As you might imagine, folks brought up the questions we have been considering. Here are a couple of slides from the class, which speak to the matter.

Should 'building blocks' be assigned their own sprint?

Should ‘building blocks’ be assigned their own sprint?

Let 'building blocks' be subsumed by 'user stories'

Let ‘building blocks’ be subsumed by ‘user stories’

Mike Cohn used the term ‘Architecture’ to refer to what I have been calling ‘building blocks’. Mike posited that you do not want to think of ‘Architecture’ as separate from ‘user stories’. Consider ‘Architecture’ an intrinsic part of ‘user stories’. You will typically spend more time on ‘Architecture’ (my ‘building blocks’) earlier in a sprint, and in the earlier sprints. As you move through the project, if you are doing your engineering right, you will simply use already built pieces, or simply apply previously made design decisions, more and more.

As the above image shows, how good you are at getting the building blocks out the way, affects how much ‘user story’ you complete in a sprint, which in a word, is Velocity.

Look to the first principles

All through the four days of Scrum training with Mountain Goat Software, I heard question, after question, which would prompt the same thought in me. Just look to the first principles of Agile. The answers are there.

Of course, the challenge, for them and for me, is knowing those first principles.

Software Engineers, Clerks, Baby-sitters

What kind of a software engineer am I, I often wonder.

Consider a Java developer, Tom Ws, who has worked on SOAP based web services for over a year, yet does not know how XML schema and WSDL are related.

Tom gave the front-end guy, Dick Ui, an updated WSDL. Dick then tells Tom, ok, now I need the XSD. Tom Ws thrashed about for a day, giving Dick Ui this XSD file and that, which Dick kept rejecting. Finally, Tom goes to the purported architect, Harry Bs, who, wonder of wonders, asked Dick Ui the right question. The WSDL contains all the XSDs you need, embedded within it. What on earth are you talking about? After this, the situation resolved itself fairly rapidly.  It turned out that Dick Ui was looking for the schema of JMS payloads, a related but different unit.

Engineers

There are resources, that know your domain, technologies, tools coming in.  These are few and far between. You cannot always count on finding these folks.

However, there is viable alternative.  There are resources that will not know your problem space, nor the technologies and tools that you use, but upon arrival, they are perfectly able to dig into the existing system, discover what they have to learn, and learn it themselves.

They will not have answers right away, but they know the correct questions to ask, and can find the answers with minimal hand-holding.

Clerks

They begin work at pretty much the same gate as the engineer.  There is much they do not know about the business domain, and the technical solutions of your choice.  That is where the similarity ends.

They learn nothing that you do not teach them explicitly.   They wait to be told what to learn.  Then they wait to be taught what they have to learn.    They cannot fill in any of the gaps themselves.

When they run into a roadblock, they will send out an email and are happy to wait, and wait, and wait.  You see, they believe “answers” are someone else’s responsibility.

Baby-sitters

Someone must spoon-feed the clerks.   Enter, the baby-sitter.  You know those people that say to you pityingly, “Do you still code?”.   That’s a baby-sitter.

The baby-sitter thinks that programmers are simply draftsman, translating a perfectly conceived blueprint into Java, C#, or what have you.  That writing computer code is an act of communication, which requires thought and practice to master, a baby-sitter considers a fanciful concern.   The baby-sitter believes that the coder will never have to make a consequential choice when he writes code, and actively encourages the coder to dumb himself down.

If you got rid of the clerks, you would not need baby-sitters.

Imagine

Imagine an engineering outfit that has only engineers. Imagine a development team that religiously avoids both clerks and baby-sitters.

This seems like a reasonable goal to me, especially, if you factor in the trope that a good developer is usually worth 4 or 5 mediocre developers (a case that I must learn to make one of these days). There will be human resource challenges; how to hire and keep high-skilled, and hence high-valued resources, but you have man-management challenges with low-skilled workers too.

As for me

What kind of a developer am I?

I suppose I have been around long enough to be some kind of an Engineer.

On some days, though, I am happy to be a baby-sitter.  That is all the energy I can muster. Further, some poor soul is happy that I am making his job easier, and that is not a small thing.

On other days, I dream about a rigorous development outfit that is entirely, of and by, engineers.  I would very probably bring up the caboose in a place like that, but that may not be such a bad thing.