As a generalist, what must I know regarding ‘performance’

I believe, a generalist, or a team of generalists, must offer these skills, related to the performance of software systems.

We are either going to be working with new software, which we write, or we will be working on a preexisting, complex and heterogeneous system.

A new system

On a new system, I am going to need the following skills.

  • Create a flexible profiling infrastructure.
    • Be able to log the turnaround time on relevant functionality, and at relevant layers of the system.
    • Be able to turn these logs on and off, increase and decrease the granularity of these logs, etc.
    • Create log output that is easy to analyze.  Bottlenecks must be easy to find.
  • Architect the system such that, bottlenecks can be replaced, without affecting any other part of the system.
  • Load test the system.  Throw large amounts data, and users at the system, in order to expose its fault-lines.

An existing system

More often than not, you will have to deal with preexisting systems which you cannot instrument.   Hence performance measurement will essentially have to be a black-box affair.  In such situations, these skills are needed.

  • Load test the system from whatever interface is exposed to you.
    • Throw users, and data at the available GUI.
    • Throw users and data at available APIs.
  • Specialized knowledge on profiling the various parts of the existing system will help.  For instance, say you have a PHP app running in Apache.   Knowing how to manage Apache profiling, will help.   But you cannot anticipate what tools a client will use.  So this expertise is something we have to be able to acquire quickly, as needed, and on the job.

What platforms and tools exactly?

My interest is in two platforms – the Java eco-system, and node.js.  Much of my programming experience has been in the Java world, and I am newbie to node.js.    In any event, the above requirements translate into the following concrete skills, I believe.

  • Implement profiling (logging by another name) which can be configured at run-time.
    • In a system that runs on the JVM, which includes these languages – Java, Scala, Groovy
      • Through common logging frameworks – log4j, JDK logging, SL4J
      • Through aspects – AspectJ
      • Through meta-programming – Groovy, Scala
    • In node.js
  • Break the system into modules whose boundaries are milestones that are relevant to performance.  Implement these modules such that we can look into and change the performance on any one of them, without affecting anything else.
    • For instance, presentation, service, data access layers.
    • Or web service access to an external site, a computation performed in a rule engine, a search query against a text-based index, a MapRequce query against a NoSQL db, etc.
  • Get turnaround times out of at least commonly used tools, which are listed below.  Clients may use other tools.  So this is often going to be something you figure out on the job.  Also, getting profiling information is only one problem.  Actually administering these tools to improve performance is often a complex, and specialized job.   It is almost impossible for a generalist to master all these tools.  At most, when the need arises, we need to know how and where to look for a solution.
    • Web / App Servers
      • Apache HTTP server
      • Jboss Application server
    • At the client
      • Rendering engines, and Javascript engines on web browsers (Chrome, Safari, Firefox, IE, Opera)
      • A mobile app – Android, and iOS
    • Data repositories
      • Oracle
      • MySQL
      • Lucene
    • Others
      • JBPM (Business process management)
      • Jboss Rules (Rule engine)
      • CXF, Axis (Web service frameworks)
  • Load (data and users) test.  There are plain old test automation tools, and there are specialized load and performance testing tools.  I, in fact, do not know what is adequate for this purpose exactly.   I have used simple Junit based tools in the past for this.
    • By driving a web UI that is in some browser.
    • Through a mobile app – Android, and iOS
    • By driving an API (JVM language, or node.js) directly, without going through a UI.


Now what?

On second glace, this is no trivial list of skills.  If nothing else, there should be little room for boredom.

User experience in an enterprise system

I have a decided fixation on ‘user experience‘.  However, I couldn’t tell you what I am looking for exactly.  This is an attempt to remove the fuzziness.

It seems to me that there are two primary questions.

  • Who are the users anyway, of an enterprise system?
  • What should these users be able to expect from a system?

The users of an enterprise system

Anyone who interacts with the enterprise system in any way at all, I consider its users.

  • Employees who use the system to conduct the business of the enterprise.
  • Tech folks who create, maintain, enhance, and run the system.
    • Managers
    • Programmers
    • Testers
    • Dev Ops (Software configuration folks, System Admins)
    • Software Operators
  • Business partners who use the system
  • Customers who use the system

All of these folks, in their various capacities, must have good ‘user experience‘.   What does ‘user experience‘ mean to each of these folks?

Common to all users

In this post I will only list behavior that I believe applies across all classes of users.  In subsequent posts, I’ll go more into what might make sense for various individual classes of users.

Business functions must exhibit the following characteristics.

  • They must be performant – must execute just as fast as necessary, regardless of the load on the system.
  • Failure of a function must not leave the system in an incorrect state, which might require invasive, and manual cleanup.  Rollback as necessary.
I've been involved in exercises where dealing with the errors that 
production code left in its wake, became a full time job.  There are 
many reasons why a development task ends up here, including holes in 
business analysis, code construction, and testing.  None of this is 
rocket science.  The relevant skills can be acquired, and these potholes
can, and must be avoided.
  • Every business command must be available through three avenues – a GUI, a DSL that you use at the command line, and as pluggable units in external systems like schedulers, ESBs, and workflow management systems.
A single API must support all these various methods of invoking business

You are taught to create interfaces that are appropriate to all levels 
of expertise.  Stepping through 8 web pages in order to make one edit 
on the 9th page might be acceptable for non-tech savvy customer service 
rep.  However, a developer, who is dealing with a customer support issue, 
does not have to deal with the overhead of all that UI.  Two lines of 
script ought to do the same job just nicely.  You could use SQL and change 
data directly in the system's database.  But wouldn't you rather go 
through the application's infrastructure, which presumably has essential 
checks and balances in place. That is where the domain specific language 
(DSL) comes in.

A DSL would also enable functional testing, while avoiding the drudgery of 
dealing with the UI.
  • You must be able to access the business function from the device of your choice – desktops, laptops, tablets, phones.
This is where the world seems to be heading.  Either get on the bandwagon, 
or be left behind.
  • Command line access must be in the form of a domain specific language (DSL) with which you can write scripts to tie together various functions in a workflow.
  • Long lived commands must have these behavior.
    • Run them in the background, and bring them into the foreground as necessary.
    • Expose current status, and progress.
    • Allow you to abort in the middle of execution, and return the system safely to the previous state.
    • Allow you to stop and resume where you left off.

Required skills 

Not every project, at every client, will ask for all the features described above.  However, I believe it represents a set of skills that an engineering outfit can reasonably be expected to have.  I want to be able to deliver this functionality.

There is little here that is exotic.   I only see three or four main streams of skills, all of which are well understood, and well supported.

  • Learn to measure, manage, and deliver performance.
  • Learn to architect business functions such that they lend themselves to manipulation from the command line, and GUI.  This mostly consists of creating well-designed APIs.
  • Learn to create GUI in a browser neutral fashion, and for various mobile devices.
  • Learn to create well-engineered and expressive DSLs.

Is it practical for a small and competent team to be able to develop these competencies, and employ them  successfully?   I would have to think so.


Halfway through a weekend visit to Hyderabad ….

On Saturday, I visited family spread all through Hyderabad.

In turn, the visits served up a smorgasbord of family dynamic – pleasant surprise; genuine affection; guilt trips; polite, but disinterested cordiality;  joy, which is sweeter for being unexpected; voluble, and surprisingly articulate grief of family that lost its matriarch suddenly, and too young; the inchoate sadness of the old ones, struggling against the fade out of everything, including, the most heart-rending of all, memory of the lived life; love, only nurtured fitfully and carelessly, yet still alive; the crushing awareness of the limits of what people can do for each other; and a few surreptitious tears.

As you might imagine it was exhausting.

But that was the least of my problems.

Every visit means food.  Food that is very hard to refuse, and not necessarily because it is just plain good.  With every mouthful, you get nutrition, and a dose of apprehension about the next.  After all these years, you know the feeding will be relentless, and unending – it started at 7:30 AM and ended at 11:00 PM.  See, you can ride the emotional roller-coaster with some grace.  You are older now, and you have learnt to fake maturity with the best of them.  But where can you ever hope to fit all that food?  That space is limited, and inexorably decreasing with age.

Nothing diverts them from the feeding.

So wonderful to see you.  Eat.   How have you been?  Eat.  We are getting by as best we can, like everyone else.  Eat.  It has been such a struggle since your aunt passed.  Eat.  So sorry to hear about your Dad’s health.  Eat.  You know, I am also diabetic. Eat.  What, you too?  Eat.  I don’t remember things so good anymore.  Eat.  What’s her name, my grand-daughter’s oldest, your niece?  Eat.  Real estate prices went through the roof, and I am told we are now, like, millionaires.  Eat.  Pity you did not hold on to your old house.  Eat.  Young people these days, they have their own ideas about everything.  Eat.  Who knows when she will have kids, I’ve stopped asking.  Eat.  Why are you still single, you doofus.  Eat.  You can’t leave already!  Eat.  So glad you stopped by.  Eat.   Make sure you visit again before you leave town.  So sad, you hardly ate anything.

In the middle of all this, there is a moment of frisson, and Chinese water torture is no longer a mystery.

At the end of the day, all consideration of weighty matters like life, death, the inevitable, self-made purgatory that awaits us all, and so on, had drowned, and disappeared in a sea of food.   The pain in the tummy obliterated the pain in the heart.  Knowing that more visits remained, my head hit the pillow around midnight, with only one clear thought.  Holy crap, I have to get up in the morning and eat again.



Can you drive a gofer?

There is this thing, a box, a basket, a very minor burden, at my feet, and it has to be moved a 100 feet or so.

I can do the thing myself, in less time than it will take to complete this thought.


I can summon a gofer whom I will instruct to perform the same work.

In India, in some circles, the latter is the way to go.  There are always gofers; if you cannot find your own, you must use someone else’s.  I can have all the time in the world, but I must not do the work myself.  I did what came naturally a few times, and caused significant consternation up and down the totem pole.

I had always fetishized a sort of low footprint self-reliance.  Get through the day with a minimum of dependence on other folks.   Here, an entirely opposite skill is prized.   I must be able to acquire, maintain, and drive, a small crew of gofers, who will do all of my personal work, except of course, the very personal ones.

This morning, I noticed cobwebs at a couple of corners in our house.   I stood there looking at them, and then called the young lady gofer that works for my parents.  I showed her the cobwebs and started to turn away before it occurred to me to raise my arms 45 degrees and clear away the cobwebs myself.  Predictably, the lady gopher said, “Don’t, don’t, I will take care of that”.

After trying to conserve bandwidth for a few days ….

I looked around for help on slowing down the speed at which I was eating up my monthly allotment of bandwidth, and found that many other people have tread this path before. Here is one pretty nice reference at Million Clues.

As various references suggested, I switched to FireFox, and installed two Add-ons, AdBlock Plus, and ImgLikeOpera.

I don’t have any scientific proof that this has reduced my bandwidth consumption, but I can certainly see that both banner ads (AdBlock Plus), and images (ImgLikeOpera) are not fetched.   Pages are loading faster.

ImgLikeOpera has the more significant impact on my user experience.  I have set it to block all images by default.   It has been interesting going about my business on the web without any images.   Some types of sites accommodate the missing images much better than others.

EspnCricinfo looks bad without images.  They use images for backgrounds of menus, without the background image, the menu items simply get mixed up with the other text.

Shopping sites were hard to deal with, which I suppose is natural.  After about 10 minutes on the Croma site, I gave up and turned on all images.   Large shopping sites like Zappos, and Amazon, will not work very well on low bandwidth.   On the other hand, I imagine that their mobile sites are designed with bandwidth constraints in mind.

News sites did much better.   The New York Times site was lovely.  Without images it continued to look very familiar.   Missing images were clearly marked, and stayed out of the way of the text.  The text itself was in exactly the same places that it always is.  Washington Post was a little uneven.  Talking Points Memo was clean.   Political Wire never has news images, and isn’t that a nice decision.

I have a renewed appreciation for the tech and design folks at the New York Times.   They probably wouldn’t even hire me as a janitor.

Facebook was interesting.   At first, it was very hard to use with all images turned off.   After a couple of days, it is beginning to feel natural.  I guess the brain adjusts.  It gets used to the visual cues that the images provide.   After a couple of days without images, you begin to learn the cues that the other content provides.

I have deliberately stayed away from many of the Indian sites.   My experience these past few weeks at these sites has not been very pleasant.   Their attention to user experience, to put it mildly, is uneven.   Dedicated E-commerce sites (FlipKart, InfiBeam, SnapDeal, etc.) have been generally much better than enterprise sites (Tamil Nadu Electricity Board, Tata Docomo, Karur Vysya Bank, etc.).

This is all very interesting.  I have never had to take available band-width into account when coming up the design for a web UI.   I wonder how seriously this is addressed by developers who cater to folks in the first world.   However, everybody is moving towards ‘mobile design first’.   One of the basic constraints of that world, is limited bandwidth.  Decisions made for the mobile design will bleed into the desktop, and that will address bandwidth shortage in a natural manner, I expect.



Dealing with low Internet bandwidth, in India

In India, Internet bandwidth is limited, and the more you want, the more expensive it is.  We know that carriers in the U.S. are going in this direction too.

I got myself what is known as a “data card” (essentially a barebones phone, which plugs into your USB port, and allows you to dial out to the internet) with a pretty high band-width plan.  The promise is 3 GB per month, at more than decent speeds, and after I go past the 3 GB limit, the speed is throttled down significantly.

3 GB per month seemed adequate when I signed up.   In the U.S my mobile plan only has a 2 GB limit, and I barely use half of it each month.   I had that in mind when I took the 3 GB plan.   However, in the two weeks since I got the data card, I have used up close to 2 GB already.  I did not stream any video, or audio.  All I have been doing is reading news, and trying to get some work done online.  Needless to say I was shocked.   It appears that I forgot that most of my usage in the U.S happens over my landline, and through my work.   Now, I am forced to pay attention to how I use the Net.

This morning, I was paying some bills through my bank’s online bill payment service.  For the first time I noticed how I heedlessly waste bandwidth.   I would enter the amount for one bill, submit that, which took me through a couple of pages, and then start my next bill, which would take me through those pages all over again.  In fact, my bank allows me to enter payments for several bills at the same time, which would take me through those same two or three confirmation pages, but I would be visiting these pages only once.

I was reminded of how my mother gathers several errands together before she ventures into the city.  Any trip from our home in the southern outskirts of Chennai, a place called Kelambakkam, into the city and back, burns through a whole day.  It simply is not practical to expect to travel into Chennai two or three times a week.  Nothing else would get done at home.   You have to try and get a lot of work done on any one trip.

Another thought occurred to me while I was at my bank’s web site.  All I really want to do is specify a biller, a payment amount, perhaps a payment date, and say, go.   If the bank had a RESTful API, which it exposed to me, I could get this done in 3 lines of script.  Pick your poison – Javascript, Python, Groovy, PHP, whatever.  Imagine how little bandwidth that would require.

However, this will work only for the geeks.   Regular folks need a UI.

Ah, and suddenly, mobile apps make sense.   With mobile apps, presentation happens completely at the client.   None of the artifacts (images, bloated Javascript, CSS, etc.) that make up the presentation will travel between the client, and the server.   Rather, only the data, and a command or two will, just like with an adhoc script I might cook up to talk to a RESTful API.

I should find out exactly what is eating up the bandwidth.  I wonder how much the damned advertisements consume.

Update on the driving, in India

In the past couple of weeks, I have continued to drive on and off. Daytime driving has become significantly easier, while driving at night remains a lost cause.

Don’t think, just drive

During the day, driving has become almost instinctive. Naturally, the traffic has not changed, but it seems to bother me less. I am noticing more. If there are 10 vehicles in front of me, I am now able to, almost unconsciously, plot a way through them.  You can guess the relative speeds of the various types of vehicles, judge who is about to make what move, and in turn make a decision yourself.   The blind spots exist, but I don’t seem to worry about them.

How all this happened is not clear.  I don’t consciously understand what it is I am seeing, and what I am not seeing.  Now and then deliberate thought suddenly intrudes, which, to little surprise, kills the fluency.   Do you remember that advice Kevin Costner’s catcher gives Tim Robbins’ talented but green pitcher, in Bull Durham?  Don’t think, just throw, or something to that effect.   That is where I am with the driving, I think.

Remember, one of the two simple rules that govern driving here – if you see a space in front of you that will fit your vehicle, you must fill it.  There is a little bit more to it.  You must fill that space fast.   You see a hole, gun the motor, rush into that hole, and slam the brakes. This is how our driver does it, which I found puzzling, until now.   If you don’t fill that space when you have the opportunity, somebody else will, which slows you down.   Moreover, being decisive in that manner provides clarity to the vehicles around you. Remember, you are constantly surrounded by other vehicles, each confronting the same problems you are.   Several times, I could clearly see my hesitation cause confusion in the other guy – he is forced to slow down and wonder what the heck is going on with me.  When this spreads to several vehicles, you have gummed up the works.

3 …. 2 …. same difference

I’ve driven on the Chennai Bypass a few times now.   This is a 6 lane highway, 3 in each direction, with a divider in between.   As you can imagine driving on this road is a little less challenging, but not entirely without the Indian touch.    I noticed that the larger vehicles rarely used the far left lane.  It would be largely empty, save for a few slower, smaller vehicles, like the two and the three wheelers.   It is not like folks maintain any kind of lane discipline; they usually drift into whatever space is available, so that almost free lane was incongruous.   I realized what the answer was this morning (we are on the road again this weekend).

Some sons of India drive down your side of the highway, in the opposite direction.  That far left lane is the one they use.  You don’t mind that; you must leave room for bending the rules, because someday you will need to do that yourself.

Even more interesting, at night, large, cargo carrying trucks simply park themselves on that lane.  We were on the bypass early this morning, and large stretches of the far left lane had become a rest area.   Perfectly reasonable position, if you ask me.   You take two lanes, and we’ll use one, for whatever.  Didn’t your Mom teach you to share?

The problem isn’t the sharing.  The issue is that this is a violation of the promise that a ‘three lane highway’ makes to you.  When they built that highway, a civic contract came into being, which said what you could expect from a ‘three lane highway’.  Such promises are almost never completely kept.  These civic contracts are satisfied only partially.  If you cannot temper your expectations accordingly, God help you, and the people around you.

So it goes.

Done with the night

At night, I am okay as long as I am on one of those roads where opposing traffic is hidden from view by a divider.   When I am forced to deal with the lights of the oncoming traffic, I am toast.   Maybe my eyes have just grown old, or maybe the creeping diabetes has rendered them less effective than they could otherwise be.  I can barely see the road in front of me.  On a drive from Alwar Thirunagar, to Kelambakkam, through rush hour traffic, around 8:00 PM, my eyes became tired by the time we hit Vandalur.   I have never felt that before.   I had to fight the urge to shut my eyes, and lay my head back on the seat.  We still had about 15 kms to go.

I went walking one night, on Vandaur road, which turned out to be exactly like driving at night.   You don’t really have a sidewalk, and hence you are virtually part of the traffic on the road.   I could barely see 5 feet in front of me sometimes.   It was unsafe.

I will drive at night only if I absolutely have to.