As a generalist, what must I know regarding ‘performance’

I believe, a generalist, or a team of generalists, must offer these skills, related to the performance of software systems.

We are either going to be working with new software, which we write, or we will be working on a preexisting, complex and heterogeneous system.

A new system

On a new system, I am going to need the following skills.

  • Create a flexible profiling infrastructure.
    • Be able to log the turnaround time on relevant functionality, and at relevant layers of the system.
    • Be able to turn these logs on and off, increase and decrease the granularity of these logs, etc.
    • Create log output that is easy to analyze.  Bottlenecks must be easy to find.
  • Architect the system such that, bottlenecks can be replaced, without affecting any other part of the system.
  • Load test the system.  Throw large amounts data, and users at the system, in order to expose its fault-lines.

An existing system

More often than not, you will have to deal with preexisting systems which you cannot instrument.   Hence performance measurement will essentially have to be a black-box affair.  In such situations, these skills are needed.

  • Load test the system from whatever interface is exposed to you.
    • Throw users, and data at the available GUI.
    • Throw users and data at available APIs.
  • Specialized knowledge on profiling the various parts of the existing system will help.  For instance, say you have a PHP app running in Apache.   Knowing how to manage Apache profiling, will help.   But you cannot anticipate what tools a client will use.  So this expertise is something we have to be able to acquire quickly, as needed, and on the job.

What platforms and tools exactly?

My interest is in two platforms – the Java eco-system, and node.js.  Much of my programming experience has been in the Java world, and I am newbie to node.js.    In any event, the above requirements translate into the following concrete skills, I believe.

  • Implement profiling (logging by another name) which can be configured at run-time.
    • In a system that runs on the JVM, which includes these languages – Java, Scala, Groovy
      • Through common logging frameworks – log4j, JDK logging, SL4J
      • Through aspects – AspectJ
      • Through meta-programming – Groovy, Scala
    • In node.js
  • Break the system into modules whose boundaries are milestones that are relevant to performance.  Implement these modules such that we can look into and change the performance on any one of them, without affecting anything else.
    • For instance, presentation, service, data access layers.
    • Or web service access to an external site, a computation performed in a rule engine, a search query against a text-based index, a MapRequce query against a NoSQL db, etc.
  • Get turnaround times out of at least commonly used tools, which are listed below.  Clients may use other tools.  So this is often going to be something you figure out on the job.  Also, getting profiling information is only one problem.  Actually administering these tools to improve performance is often a complex, and specialized job.   It is almost impossible for a generalist to master all these tools.  At most, when the need arises, we need to know how and where to look for a solution.
    • Web / App Servers
      • Apache HTTP server
      • Jboss Application server
    • At the client
      • Rendering engines, and Javascript engines on web browsers (Chrome, Safari, Firefox, IE, Opera)
      • A mobile app – Android, and iOS
    • Data repositories
      • Oracle
      • MySQL
      • Lucene
    • Others
      • JBPM (Business process management)
      • Jboss Rules (Rule engine)
      • CXF, Axis (Web service frameworks)
  • Load (data and users) test.  There are plain old test automation tools, and there are specialized load and performance testing tools.  I, in fact, do not know what is adequate for this purpose exactly.   I have used simple Junit based tools in the past for this.
    • By driving a web UI that is in some browser.
    • Through a mobile app – Android, and iOS
    • By driving an API (JVM language, or node.js) directly, without going through a UI.

     

Now what?

On second glace, this is no trivial list of skills.  If nothing else, there should be little room for boredom.

User experience in an enterprise system

I have a decided fixation on ‘user experience‘.  However, I couldn’t tell you what I am looking for exactly.  This is an attempt to remove the fuzziness.

It seems to me that there are two primary questions.

  • Who are the users anyway, of an enterprise system?
  • What should these users be able to expect from a system?

The users of an enterprise system

Anyone who interacts with the enterprise system in any way at all, I consider its users.

  • Employees who use the system to conduct the business of the enterprise.
  • Tech folks who create, maintain, enhance, and run the system.
    • Managers
    • Programmers
    • Testers
    • Dev Ops (Software configuration folks, System Admins)
    • Software Operators
  • Business partners who use the system
  • Customers who use the system

All of these folks, in their various capacities, must have good ‘user experience‘.   What does ‘user experience‘ mean to each of these folks?

Common to all users

In this post I will only list behavior that I believe applies across all classes of users.  In subsequent posts, I’ll go more into what might make sense for various individual classes of users.

Business functions must exhibit the following characteristics.

  • They must be performant – must execute just as fast as necessary, regardless of the load on the system.
  • Failure of a function must not leave the system in an incorrect state, which might require invasive, and manual cleanup.  Rollback as necessary.
I've been involved in exercises where dealing with the errors that 
production code left in its wake, became a full time job.  There are 
many reasons why a development task ends up here, including holes in 
business analysis, code construction, and testing.  None of this is 
rocket science.  The relevant skills can be acquired, and these potholes
can, and must be avoided.
  • Every business command must be available through three avenues – a GUI, a DSL that you use at the command line, and as pluggable units in external systems like schedulers, ESBs, and workflow management systems.
A single API must support all these various methods of invoking business
functionality.

You are taught to create interfaces that are appropriate to all levels 
of expertise.  Stepping through 8 web pages in order to make one edit 
on the 9th page might be acceptable for non-tech savvy customer service 
rep.  However, a developer, who is dealing with a customer support issue, 
does not have to deal with the overhead of all that UI.  Two lines of 
script ought to do the same job just nicely.  You could use SQL and change 
data directly in the system's database.  But wouldn't you rather go 
through the application's infrastructure, which presumably has essential 
checks and balances in place. That is where the domain specific language 
(DSL) comes in.

A DSL would also enable functional testing, while avoiding the drudgery of 
dealing with the UI.
  • You must be able to access the business function from the device of your choice – desktops, laptops, tablets, phones.
This is where the world seems to be heading.  Either get on the bandwagon, 
or be left behind.
  • Command line access must be in the form of a domain specific language (DSL) with which you can write scripts to tie together various functions in a workflow.
  • Long lived commands must have these behavior.
    • Run them in the background, and bring them into the foreground as necessary.
    • Expose current status, and progress.
    • Allow you to abort in the middle of execution, and return the system safely to the previous state.
    • Allow you to stop and resume where you left off.

Required skills 

Not every project, at every client, will ask for all the features described above.  However, I believe it represents a set of skills that an engineering outfit can reasonably be expected to have.  I want to be able to deliver this functionality.

There is little here that is exotic.   I only see three or four main streams of skills, all of which are well understood, and well supported.

  • Learn to measure, manage, and deliver performance.
  • Learn to architect business functions such that they lend themselves to manipulation from the command line, and GUI.  This mostly consists of creating well-designed APIs.
  • Learn to create GUI in a browser neutral fashion, and for various mobile devices.
  • Learn to create well-engineered and expressive DSLs.

Is it practical for a small and competent team to be able to develop these competencies, and employ them  successfully?   I would have to think so.

 

Look to the culture, Luke

Five of us went to Tirumala last week. We walked up Srivari Mettu, which is a flight of steps (2400, I think they said) that goes straight up the back of the mountain on top of which Tirumala sits. The climb was challenging, but not impossible.

At Tirumala, accommodations had been arranged in lodgings that were represented to us as “very good”. We got to the rooms, and to little surprise, found them, not so good.

The room, an apartment really, was large. It had three bedrooms, one of which had a window air-conditioner, two bathrooms, a dining room, and what appeared to be space for cooking. The rooms were fairly clean, and the sheets looked washed.  That is the good news.

The paint had peeled off in several places, and these spots were patched in different shades of the original color. There were several large stains on the walls. One bathroom’s geyser did not turn on, and the other bathroom did not have a knob on the hot water tap. The doors to both bathrooms were pockmarked with rust, and the bottom of the doors were actually eaten away.

The rooms looked, and felt, sadly, rather shabby.

The room cost 2000 rupees I think, which ain’t exactly peanuts.

Tirumula is reputedly one of the richest religious institutions in the world.  I will bet my life they have the resources to maintain those rooms well. They have the money, and God knows there is never a shortage of labor.

Why didn’t they do better? After all, they have the means. It has to be because they felt it was not necessary. The people who rent those rooms will gladly accept them as is.

Or, is it possible that the powers that be, think that the rooms, as they exist now, are indeed well maintained?

My mother, who lives in India, travels a lot, and almost surely has seen worse, thought the rooms were par for the course. Her sister, my aunt, who lives in Florida, had a major freakout. I, who travel between the U.S and India regularly, am a little bit more inured to the unevenness of Indian life, and staked out a middle ground – moderate irritation, tempered by grimly amused resignation.  There is a T-Shirt out there that says, “Surrendering to India”.  That is my mantra.

In fact, I don’t believe that the management really believes that the rooms are well-maintained. Everyone knows that more is possible.  You can find good maintenance in India. Maintenance where attention is paid to detail. Management that walks in the shoes of the guests, anticipates their needs, and satisfies them, efficiently, quietly, even imaginatively, and elegantly. In Tirumala itself, we found a restaurant, called Sarangi, which was a perfect example of this.

The difference between an establishment like Sarangi, and our rooms, is cost. That was an expensive restaurant. Only clientele of a certain economic class would frequent Sarangi.

So good maintenance, as defined above, is only available to the affluent. A very large swath of Indian society is never given good maintenance as the norm.  Not even when you have the means to provide it.  This slice of society accepts that lackluster maintenance is normal, and good enough. This is what they receive, and in turn, this is what they themselves are conditioned to provide.

You have a roof over your head, so what if the walls are stained and badly painted. You can close and lock the bath room door, so how does it matter if the door looks like it might give you a disease if you touch it. You have running water, so who cares if the water heater is mere ornamentation, and an ugly one at that.

The software you build performs the business function you need. So who cares if the user interface is confusing, error prone, and has poor finish.  The code runs, so who cares if it is unintelligible. The reported bug is fixed, so who cares if fix adds more strands of spaghetti.  The enhancement came in on time so who cares if you copied and pasted thousands of lines of code, rather than analyze, and re-factor.

Those inexpensive programmers that you outsource your work to – where do you think they learnt what is important and what is not? Where do you think they learn how to perform a task?   They learn it everyday, ever since they are in knee pants, in places like those guest rooms in Tirumala.

You see this everywhere.  Here is an another example.

There is a multiplex / food court /entertainment center, called Mayajaal, in Chennai. The multiplex has 16 screens. The tickets range from 120 rupees to 150 rupees I think. This place makes money hand over fist. Exactly like Tirumala, Mayajaal does not lack resources.

My parents and I went to a movie there, and took the elevator up to our screen. The linoleum floor of the elevator had a huge gash in it. Part of it was just gone. The railing that runs along the wall of the elevator had been ripped off. The restroom had those automatic air dryers, but the electrical sockets were hanging out of the walls. I mean there was a hole in the wall, and the board with the outlets had just been pushed into the hole.

I repeat, Mayajaal makes a lot of money. It makes enough to fix the elevator, and actually install the damn electrical outlets.

So why don’t they? Because neither Mayajaal, nor the folks that frequent Mayajaal, think it is truly necessary.

What do you believe these folks might think about “user experience”, or “software engineering”? I would guess, they consider such notions quaint, but ultimately unimportant. You get to see the movie, so who cares what the elevator looks like. You have a place to relieve yourself, so who cares how a fixture is installed.  God is not in the details. It simply is not part of their day to day life. They don’t see it. They don’t expect it. They don’t know how to supply it. They cannot understand how someone might think otherwise.

Many of your inexpensive programmers, are exactly these folks.

These past few years I’ve had occasion to work with such resources.   They were almost universally smarter, and more hard working than me.  They consistently delivered the goods in acceptable time.  However, I believe I consistently produced better software, soup to nuts – business and technical analysis, design (internal, and external), and construction.   The quality of their work made me very frustrated, angry, would leave me in despair, really.   I don’t think I will ever feel that way again.   You see, now I know where these folks are coming from.

How do you take a bright, enterprising, middle class, or lower middle class kid, from some small town like Khamman, or Villupuram, who has a degree from one of those hundreds of engineering colleges that have mushroomed up all over Andhra, and Tamil Nadu, pay him 2 or 3 lakhs a year, and get him to care about detail?   That is the challenge.   How do you inculcate a decidedly alien culture in someone like him?

A small thing that used to drive me, and some of my fellow travelers crazy, was the chronic misspelling of variable names in the code.   There is only one cure for it.  Visit India.  In the past few weeks, I’ve seen misspelt shop signs, and traffic sign boards.  My Dad’s letter-head has issues.  My cousin’s wedding invitation had an obvious error.  Note, these are all significant documents, affecting branding, and important life events.  Why did they not proof-read this stuff?   The money surely is available, and how much time does it take to proof-read a shop sign, or even a wedding invitation.

The answer – they don’t believe it matters.  How can we expect a programmer who grows up with this, to care about correctly spelling an identifier in computer code?   The user doesn’t even see it, for God’s sake.

My favorite so far is this road sign, on a bend in the Puthur – Oothukottai route from Tirupathi to Chennai: “Cures Ahead.  Goo Slow“.

Today, on my morning walk down Vandalur Road, I saw the local trash collection service in action.  They wore brown uniforms, and were dumping the trash into a metal cart, which they were pulling behind a bicycle.  Nice.   But.  The carts were open.   I could already see flies buzzing around, and in the cart.  Soon the cart would fill up, and trash would start to spill out.

Why did it not occur to them to put a lid on that small cart?   How are we going to get them to care about that missing lid?  I am coming around to the idea, that this is the primary challenge of outsourcing.  Everything else is just technical detail.

 

I want to create software that is like Tom Binh bags

or  I want to create software like Tom Binh makes bags.

I am impressed by Tom Binh bags.

It makes me happy to carry around a Tom Binh bag.

I am so glad that it has been possible for someone to make a success out of working in just that manner.

Craftsman

Is Tom Binh a craftsman?  Is that what it is?   Do I want to be a software craftsman?  That is possible, isn’t it?  There are people out there that see value in that, don’t they?

I have been doing computer programming of some sort or the other for almost 17 years.   I don’t feel very accomplished.  I feel a distinct lack of skill.  There are so many questions, which I don’t know the answer to.  I am not sure I am an expert at anything.  I know a little bit of a lot of things.  I can be relied on to ask good questions, I think.  I have been known to work hard.  I play well with other people.

At best, I think I have the aptitude to be a craftsman.  I keep thinking that you are a craftsman when the skill that you ply is fairly complete, and second nature.  I am not that, yet.  I have promise.  That is the most I can say for myself.

How can I work at something for 17 years, and still not be very good at it?  Are you ready for this?   It turns out that I don’t have to be.  In software engineering, this strange slice of the working world, I can provide a lot of value, without being very skilled at software engineering.  Despite my shortcomings, if software engineering skills were graded on a curve, I am sure I would fall at a decent spot.  I am good enough for a lot of environments.

Quality

When I see a Tom Binh bag, I think, “That is quality“.

A Tom Binh bag is an inspiration.   No, it is more.  A Tom Binh bag is an aspiration – for me, carrying one is an act of prayer.

But what am I aspiring to?  What does “quality” mean to me?

I am not sure I can explain it.   I could resort to that old saw about pornography – we can’t define it, but we sure know it when we see it.

Tom Binh’s bags show attention to detail.  “God is in the details“; sign me up for that church.

Every little part of the bag has a well thought-out purpose.  Every little part of the bag fits, as intended, into its surroundings, and all the little parts work together, as intended, to serve a larger, well-defined, goal.  Unmistakably, the bags are a rigorous architecture, and design, made flesh.

I want to create software that exhibits the same characteristics.  Is that too much to ask?  The answer of course, is yes.

Why don’t more people carry Tom Binh bags?

Good, but not worth the money

The bags cost more money than some folks want to, or can spend, on luggage.  Regardless of the value the bags provide, the finances just don’t make sense to these folks.

Software engineering shares this issue.   Higher skilled engineers are expensive.   Business, often does not want to spend that money.  The work produced by inexpensive, but low skilled engineers, is deemed good enough.

There is a school of thought that holds that a higher skilled software engineer is worth several low-skilled engineers.  Alas, I don’t know how to make this case, even though I have seen anecdotal evidence to support this idea.

This is not an entirely lost cause, but close.

I am looking, and I am not liking

Some people simply are never going to like Tom Binh bags.   Even when money is no object, they see nothing there to their liking.

Have you ever had to deal with a business stakeholder who wants a completely inexplicable UI?   The customer is always right.  You can only walk away from this.

Craftsmanship dictates low volumes?

I might be wrong about this.  Very little of Tom Binh’s production process seems to be mechanized.  Tom Binh is not in the mass manufacturing business, I think.  He creates quality with meticulous, manual, effort, by a small number of highly skilled folks.   Naturally, this means they can produce only so many bags.

Software does not have this limitation.  Mass manufacturing is trivial.   It is the master copy, however, which is the challenge.  Craftsmanship can indeed be brought to bear in creating and maintaining that one master.  However, we all know that is easier said than done.  Some days I understand why.  Some days I do not.

Colors

Oh man, do I love those colors!

tombinh-colors

I wonder, what color is the code I write?

What color is your code?