Tag Archives: craftsmanship

by

As a generalist, what must I know regarding ‘transactions’

I believe, a generalist, or a team of generalists, must offer these skills, related to the implementation of ‘business transactions‘.

The failure of a business transaction must still leave the system in a safe, and valid state. As a ‘generalist’ I must either know how to achieve that goal, or know where to quickly find a solution.

Platforms

As always, I want to be able to solve this problem in two platforms – the Java eco-ssytem, and Node.js.

 

Short-lived business transactions

Most of us are familiar with ACID transaction support in relational databases.  Relying on this support is only recommended for short-lived business transactions.

Database transactions are typically implemented by locking database tables for a particular user, which forces all others to wait till the locks are released. This necessarily slows the system down, among other complexities. Hence the recommendation that ACID transactions be very short-lived.

Here are some examples of short-lived business transactions.

  • Change the address of a customer. Typically you have already gathered the new address, and you simply have to update a few tables with the new data.
  • Apply a payment against a policy. Again, the whole transaction is typically an update to a few tables.

In the Java eco-system, this sort of short-lived transaction, when applied against a single database, is implemented with the JDBC API. We must be able to able that, using Java, Scala, and Groovy.

I must be able to talk to relational databases, and manage ACID transactions against a single database, using node.js.

 

Long-lived business transactions

However, often, there are business transactions that are long-lived, and must behave gracefully.

Here is one example of a long-lived transaction – migrating old insurance policies from one system to another.

This is historical data, often several years worth and can be voluminous. It contains many different parts, like contacts, coverages, changes made to the policy over the years, documents, etc. Accepting all of the policy into a new system can take a significant time. We used to run into policies that took 20 seconds to process completely. Things will go wrong, and when they do, you would like to cleanly rollback all of the incoming data. However, you cannot keep an ACID database transaction open for 20 second, 10, or even 5 seconds. That locks up database tables, which in turn will severely diminish your ability to handle load.

Here is another example – business workflows that extend over several days.

They are initiated, passed around to several folks, and then eventually completed. If for some reason this business process ends in some kind of rejection, or failure, you may want to discard, or perhaps archive data that this workflow created.

Consider that new accident information is received for an automobile policy. Maybe some documents are uploaded. Some premium adjustments are made. Some bills are generated. Underwriting, and billing managers review, and sign off. This process may take several days to finish. Say after doing a lot of work, you discover all this work was done on the wrong policy – perhaps an ex-husband’s. How do you roll back this work, and data that has been accumulating over several days? Surely not with a database transaction.

So how do ensure that such operations are well behaved? If necessary, how to ensure that these long-lived business transactions exhibit ACID properties?

As a ‘generalist’ I must know standard approaches, and solutions to this problem. Further, I must be able to implement these solutions in the Java eco-system, and in node.js.

 

Single database vs. multiple

In a large, heterogenous environment, you are often working with several databases. Perhaps all documents are in some legacy SQL Server DB, and day to day transactional data are in a fast MySQL DB.

How do you implement ACID when a business transaction, even a short-lived one, works with data that is distributed across more than one database?

In the Java world, there is the JTA API, which supports so-called ‘distributed transactions’. But very few people seem to use this. As a ‘generalist’ I must know what the alternative is.

Similarly, I must know how this problem can be handled in node.js.

 

Polyglot persistence

This is just a special case of the multiple database scenario. The data repository can be anything at all – relational DB, NoSQL DB, text based index, messaging end point, flat disk file, etc.

How do you implement ACID when the business transaction works with many different kinds of data repositories?

For instance, say you are recording an auto accident. Photos of the car might go into a noSQL database, like MongoDB. A description of the accident is saved to Oracle, and this information is also parsed and pushed into Lucene. Finally, a notification is dropped into a queue that a claims adjuster is watching. If this transaction dies for some reason, you want to rollback the changes you made to each of these very disparate data repositories.

by

As a generalist, what must I know regarding ‘performance’

I believe, a generalist, or a team of generalists, must offer these skills, related to the performance of software systems.

We are either going to be working with new software, which we write, or we will be working on a preexisting, complex and heterogeneous system.

A new system

On a new system, I am going to need the following skills.

  • Create a flexible profiling infrastructure.
    • Be able to log the turnaround time on relevant functionality, and at relevant layers of the system.
    • Be able to turn these logs on and off, increase and decrease the granularity of these logs, etc.
    • Create log output that is easy to analyze.  Bottlenecks must be easy to find.
  • Architect the system such that, bottlenecks can be replaced, without affecting any other part of the system.
  • Load test the system.  Throw large amounts data, and users at the system, in order to expose its fault-lines.

An existing system

More often than not, you will have to deal with preexisting systems which you cannot instrument.   Hence performance measurement will essentially have to be a black-box affair.  In such situations, these skills are needed.

  • Load test the system from whatever interface is exposed to you.
    • Throw users, and data at the available GUI.
    • Throw users and data at available APIs.
  • Specialized knowledge on profiling the various parts of the existing system will help.  For instance, say you have a PHP app running in Apache.   Knowing how to manage Apache profiling, will help.   But you cannot anticipate what tools a client will use.  So this expertise is something we have to be able to acquire quickly, as needed, and on the job.

What platforms and tools exactly?

My interest is in two platforms – the Java eco-system, and node.js.  Much of my programming experience has been in the Java world, and I am newbie to node.js.    In any event, the above requirements translate into the following concrete skills, I believe.

  • Implement profiling (logging by another name) which can be configured at run-time.
    • In a system that runs on the JVM, which includes these languages – Java, Scala, Groovy
      • Through common logging frameworks – log4j, JDK logging, SL4J
      • Through aspects – AspectJ
      • Through meta-programming – Groovy, Scala
    • In node.js
  • Break the system into modules whose boundaries are milestones that are relevant to performance.  Implement these modules such that we can look into and change the performance on any one of them, without affecting anything else.
    • For instance, presentation, service, data access layers.
    • Or web service access to an external site, a computation performed in a rule engine, a search query against a text-based index, a MapRequce query against a NoSQL db, etc.
  • Get turnaround times out of at least commonly used tools, which are listed below.  Clients may use other tools.  So this is often going to be something you figure out on the job.  Also, getting profiling information is only one problem.  Actually administering these tools to improve performance is often a complex, and specialized job.   It is almost impossible for a generalist to master all these tools.  At most, when the need arises, we need to know how and where to look for a solution.
    • Web / App Servers
      • Apache HTTP server
      • Jboss Application server
    • At the client
      • Rendering engines, and Javascript engines on web browsers (Chrome, Safari, Firefox, IE, Opera)
      • A mobile app – Android, and iOS
    • Data repositories
      • Oracle
      • MySQL
      • Lucene
    • Others
      • JBPM (Business process management)
      • Jboss Rules (Rule engine)
      • CXF, Axis (Web service frameworks)
  • Load (data and users) test.  There are plain old test automation tools, and there are specialized load and performance testing tools.  I, in fact, do not know what is adequate for this purpose exactly.   I have used simple Junit based tools in the past for this.
    • By driving a web UI that is in some browser.
    • Through a mobile app – Android, and iOS
    • By driving an API (JVM language, or node.js) directly, without going through a UI.

     

Now what?

On second glace, this is no trivial list of skills.  If nothing else, there should be little room for boredom.

by

User experience in an enterprise system

I have a decided fixation on ‘user experience‘.  However, I couldn’t tell you what I am looking for exactly.  This is an attempt to remove the fuzziness.

It seems to me that there are two primary questions.

  • Who are the users anyway, of an enterprise system?
  • What should these users be able to expect from a system?

The users of an enterprise system

Anyone who interacts with the enterprise system in any way at all, I consider its users.

  • Employees who use the system to conduct the business of the enterprise.
  • Tech folks who create, maintain, enhance, and run the system.
    • Managers
    • Programmers
    • Testers
    • Dev Ops (Software configuration folks, System Admins)
    • Software Operators
  • Business partners who use the system
  • Customers who use the system

All of these folks, in their various capacities, must have good ‘user experience‘.   What does ‘user experience‘ mean to each of these folks?

Common to all users

In this post I will only list behavior that I believe applies across all classes of users.  In subsequent posts, I’ll go more into what might make sense for various individual classes of users.

Business functions must exhibit the following characteristics.

  • They must be performant – must execute just as fast as necessary, regardless of the load on the system.
  • Failure of a function must not leave the system in an incorrect state, which might require invasive, and manual cleanup.  Rollback as necessary.
I've been involved in exercises where dealing with the errors that 
production code left in its wake, became a full time job.  There are 
many reasons why a development task ends up here, including holes in 
business analysis, code construction, and testing.  None of this is 
rocket science.  The relevant skills can be acquired, and these potholes
can, and must be avoided.
  • Every business command must be available through three avenues – a GUI, a DSL that you use at the command line, and as pluggable units in external systems like schedulers, ESBs, and workflow management systems.
A single API must support all these various methods of invoking business
functionality.

You are taught to create interfaces that are appropriate to all levels 
of expertise.  Stepping through 8 web pages in order to make one edit 
on the 9th page might be acceptable for non-tech savvy customer service 
rep.  However, a developer, who is dealing with a customer support issue, 
does not have to deal with the overhead of all that UI.  Two lines of 
script ought to do the same job just nicely.  You could use SQL and change 
data directly in the system's database.  But wouldn't you rather go 
through the application's infrastructure, which presumably has essential 
checks and balances in place. That is where the domain specific language 
(DSL) comes in.

A DSL would also enable functional testing, while avoiding the drudgery of 
dealing with the UI.
  • You must be able to access the business function from the device of your choice – desktops, laptops, tablets, phones.
This is where the world seems to be heading.  Either get on the bandwagon, 
or be left behind.
  • Command line access must be in the form of a domain specific language (DSL) with which you can write scripts to tie together various functions in a workflow.
  • Long lived commands must have these behavior.
    • Run them in the background, and bring them into the foreground as necessary.
    • Expose current status, and progress.
    • Allow you to abort in the middle of execution, and return the system safely to the previous state.
    • Allow you to stop and resume where you left off.

Required skills 

Not every project, at every client, will ask for all the features described above.  However, I believe it represents a set of skills that an engineering outfit can reasonably be expected to have.  I want to be able to deliver this functionality.

There is little here that is exotic.   I only see three or four main streams of skills, all of which are well understood, and well supported.

  • Learn to measure, manage, and deliver performance.
  • Learn to architect business functions such that they lend themselves to manipulation from the command line, and GUI.  This mostly consists of creating well-designed APIs.
  • Learn to create GUI in a browser neutral fashion, and for various mobile devices.
  • Learn to create well-engineered and expressive DSLs.

Is it practical for a small and competent team to be able to develop these competencies, and employ them  successfully?   I would have to think so.

 

by

I want to create software that is like Tom Binh bags

or  I want to create software like Tom Binh makes bags.

I am impressed by Tom Binh bags.

It makes me happy to carry around a Tom Binh bag.

I am so glad that it has been possible for someone to make a success out of working in just that manner.

Craftsman

Is Tom Binh a craftsman?  Is that what it is?   Do I want to be a software craftsman?  That is possible, isn’t it?  There are people out there that see value in that, don’t they?

I have been doing computer programming of some sort or the other for almost 17 years.   I don’t feel very accomplished.  I feel a distinct lack of skill.  There are so many questions, which I don’t know the answer to.  I am not sure I am an expert at anything.  I know a little bit of a lot of things.  I can be relied on to ask good questions, I think.  I have been known to work hard.  I play well with other people.

At best, I think I have the aptitude to be a craftsman.  I keep thinking that you are a craftsman when the skill that you ply is fairly complete, and second nature.  I am not that, yet.  I have promise.  That is the most I can say for myself.

How can I work at something for 17 years, and still not be very good at it?  Are you ready for this?   It turns out that I don’t have to be.  In software engineering, this strange slice of the working world, I can provide a lot of value, without being very skilled at software engineering.  Despite my shortcomings, if software engineering skills were graded on a curve, I am sure I would fall at a decent spot.  I am good enough for a lot of environments.

Quality

When I see a Tom Binh bag, I think, “That is quality“.

A Tom Binh bag is an inspiration.   No, it is more.  A Tom Binh bag is an aspiration – for me, carrying one is an act of prayer.

But what am I aspiring to?  What does “quality” mean to me?

I am not sure I can explain it.   I could resort to that old saw about pornography – we can’t define it, but we sure know it when we see it.

Tom Binh’s bags show attention to detail.  “God is in the details“; sign me up for that church.

Every little part of the bag has a well thought-out purpose.  Every little part of the bag fits, as intended, into its surroundings, and all the little parts work together, as intended, to serve a larger, well-defined, goal.  Unmistakably, the bags are a rigorous architecture, and design, made flesh.

I want to create software that exhibits the same characteristics.  Is that too much to ask?  The answer of course, is yes.

Why don’t more people carry Tom Binh bags?

Good, but not worth the money

The bags cost more money than some folks want to, or can spend, on luggage.  Regardless of the value the bags provide, the finances just don’t make sense to these folks.

Software engineering shares this issue.   Higher skilled engineers are expensive.   Business, often does not want to spend that money.  The work produced by inexpensive, but low skilled engineers, is deemed good enough.

There is a school of thought that holds that a higher skilled software engineer is worth several low-skilled engineers.  Alas, I don’t know how to make this case, even though I have seen anecdotal evidence to support this idea.

This is not an entirely lost cause, but close.

I am looking, and I am not liking

Some people simply are never going to like Tom Binh bags.   Even when money is no object, they see nothing there to their liking.

Have you ever had to deal with a business stakeholder who wants a completely inexplicable UI?   The customer is always right.  You can only walk away from this.

Craftsmanship dictates low volumes?

I might be wrong about this.  Very little of Tom Binh’s production process seems to be mechanized.  Tom Binh is not in the mass manufacturing business, I think.  He creates quality with meticulous, manual, effort, by a small number of highly skilled folks.   Naturally, this means they can produce only so many bags.

Software does not have this limitation.  Mass manufacturing is trivial.   It is the master copy, however, which is the challenge.  Craftsmanship can indeed be brought to bear in creating and maintaining that one master.  However, we all know that is easier said than done.  Some days I understand why.  Some days I do not.

Colors

Oh man, do I love those colors!

tombinh-colors

I wonder, what color is the code I write?

What color is your code?