“No more batch processing! I’m serious!” For many months now, this has been the mantra of our forward-thinking (and frustrated) IT director. He will not be deterred even slightly, despite numerous protests of a “You can’t be serious” nature.
He has a point. Overnight batch processing is the number one cause of really ugly system errors in our organization. One little thing goes wrong while we’re all sleeping soundly, and it starts a domino effect – the next step in the batch run goes a little more wrong, and so on and so forth until a nasty morning of record fixing is created for some unsuspecting slumberer.
Our good director’s biggest gripe about batch jobs is that they just don’t make sense – that is, unless you’re thinking like a programmer. There is no business need to stockpile transactions to later do them all in one run; this was a concept invented by programmers to circumvent technology limitations, many of which no longer exist.
For example: if we want to sell books through an online retailer, we have to send them a title load containing product descriptions, an image of the book cover, pricing, availability, etc. If the price or stock on hand changes, we send them an update. Updates are sent in batches on a regular basis, with frequency depending on the agreement with each retailer. The problem with this is obvious – we sell books every day, so how can their system have the right on-hand quantities for our products if we send batch updates, say, every two weeks? The customer is fed misinformation and may actually order a book that has gone out of stock since the last system update.
Programmers are entirely to blame for this mess. No business executive would suggest such a process. Ask the executives how the system should work, and I’m sure they’ll say, “The customer should be able to see our inventory as it is right now.”
Today we have technology that allows the executives to get their way. If a customer visits an online book retailer’s site and looks up a title published by my employer, the retailer’s system should fetch the latest information from us in real-time and present it to the customer. If the customer chooses to order the book, the order should be fed directly to our system as well. This is Web services architecture in action.
As programmers, we are often accused of being unable to see the big picture because we immediately get scared off by the technical complexity of systems that make logical sense. Right away we’re asking questions like, “How can the page load fast enough for the customer if it’s querying our server first?” or “What happens if the customer tries to place an order while our system is down?” We have to learn to view these questions as issues that are resolved by the solution, not issues that dictate the process.
The successful programmer is someone who can embrace the dichotomy of our profession – someone who can have limitless vision and realize it using building blocks of ones and zeroes.
Cooney works as a programmer/analyst for a major Canadian book publisher. He can be reached at cooney@hotmail.com.