David Brown is worried. As managing director of the IT transformation group at Bank of New York Mellon, he is responsible for the health and welfare of 112,500 Cobol programs — 343 million lines of code — that run core banking and other operations. But many of the people who built that code base, some of which dates back to Cobol’s early days in the 1960s, will be retiring over the next several years.
“We have people we will be losing who have a lot of business knowledge. That scares me,” Brown says. He’s concerned about finding new Cobol programmers, who are expected to be in short supply in the next five to 10 years. But what really keeps him up at night is the thought that he may not be able to transfer the deep understanding of the business logic embedded within the bank’s programs before it walks out the door with employees who retire.
More than 50 years after Cobol came on the scene, the language is alive and well in the world’s largest corporations, where it excels at executing large-scale batch and transaction processing operations on mainframes. The language is known for its scalability, performance and mathematical accuracy. But as the boomer generation prepares to check out of the workforce, IT executives are taking a fresh look at their options.
In a recent Computerworld survey of 357 IT professionals, 46% of the respondents said they are already noticing a Cobol programmer shortage, while 50% said the average age of their Cobol staff is 45 or older and 22% said the age is 55 or older.
“Organizations are trying not to get backed into a corner because of the skills issue,” says Paul Vallely, mainframe sales director at software vendor Compuware. “I haven’t seen companies move off mainframes due to the Cobol skills shortage yet, but it’s looming.”
For Bank of New York Mellon, which bought its first mainframe in 1955, keeping the core Cobol applications that run the business on the mainframe makes sense. Modernization efforts have made BNY Mellon’s Cobol-based programs more accessible through the use of Web services and up-to-date user interfaces.
But for some noncore applications, and for smaller workloads, organizations have been gradually migrating off of mainframes — and away from Cobol. In several cases, Cobol programs are simply rehosted on Linux or Windows servers; in other cases, they’re rewritten in object-oriented languages; and some programs are being replaced with packaged software.
“Over the past five years, there has been an acceleration of [some] businesses moving off host platforms,” says Adam Burden, global application modernization lead at Accenture. That often means leaving Cobol behind by either rewriting it for J2EE or .Net or moving to packaged software.
Cobol Training
Closing the talent gap
Where do you find Cobol programmers these days? College graduates with Cobol training are in short supply. In Michigan, for example, state schools that offer Cobol programming education have cancelled classes because of a lack of interest. “They can’t get anyone to enroll,” says Jonathan Miller, director of information systems and services for the government of Michigan’s Saginaw County. But some colleges are still providing Cobol training — with help from IBM. The mainframe vendor has developed curricula in association with more than 80 colleges and universities ranging from Brigham Young to Texas A&M.
“We donate hardware and software, help with the curriculum, and they graduate hundreds of people every year,” says Kevin Stoodley, an IBM fellow and CTO.
Guardian Life Insurance has recruited Cobol programmers from Workforce Opportunity Services, a nonprofit that collaborates with business clients and local colleges to train economically disadvantaged students to work in less popular technology disciplines such as Cobol programming. “They take kids from disadvantaged neighborhoods and provide them as consultants,” says former Guardian CIO Frank Wander, who now has his own consultancy, IT Excellence Institute.
“It’s sort of a work-study program. We have over 200 consultants today in five states, and we’re expanding,” says Workforce founder Art Langer.
BNY Mellon and many other organizations also increasingly rely on outsourcers to pick up maintenance and support duties. But for many users, an offshore locale is not the place to keep the institutional knowledge of the business rules behind the code. David Brown, managing director of BNY Mellon’s IT transformation group, says the bank wants those skills in-house. Fortunately, it’s not all that difficult to cross-train programmers in Cobol. “Right now, it’s pretty easy to hire programmers. And if they understand Java, I can bring them back to procedural languages like Cobol,” Brown says. The trick is to develop a curriculum that teaches not just Cobol, but the business rules behind the code that runs the company, he says, adding, “We need to make sure we can roll that forward.”
— Robert L. Mitchell
Gartner estimates that the world has seen a decline of about 5% in total Cobol code over the past few years. Much of that involved migrations by small and midsize mainframe shops that move off what they see as a legacy language when they retire the hardware, says analyst Dale Vecchio. They’re using other building blocks to develop their systems. “Cobol is no longer needed,” Vecchio says. “There are alternatives.”
Rehosting can get code off the mainframe quickly. One vendor that caters to users considering that option is Rockville, Md.-based Micro Focus, whose offerings include a system that will support Cobol programs on a Microsoft Azure cloud.
But rehosting is often seen as just an intermediate step on the way to completely modernizing and transforming Cobol systems.
Cobol’s Image Problem
A procedural language, Cobol is not perceived to be as agile as object-oriented languages for modern programming needs such as mobile apps and the Web. And despite the availability of state-of-the-art Cobol development environments — including IBM’s Enterprise Cobol on the mainframe and Micro Focus’s Visual Cobol, which integrates well with Microsoft’s Visual Studio development suite for .Net — Cobol is widely viewed as a legacy language.
Nearly half (49%) of the Computerworld survey respondents whose organizations don’t use Cobol said the reason is that the language is simply outdated.
Not everyone agrees, of course. “Cobol has had lasting value, and it’s not broken,” says Kevin Stoodley, an IBM fellow and CTO of enterprise modernization tools, compilers and security at IBM.
A majority of the Computerworld readers who took part in our survey seem to concur with Stoodley: 64% of the respondents said their organizations still use Cobol — more than any modern language except for Java/JavaScript and Visual Basic. That figure is actually slightly higher than the response rate to a similar question in our last survey on Cobol use, from 2006. Some 62% of respondents to the 2006 survey said they still used Cobol.
In the more recent survey, over 50% of respondents said Cobol represents more than half of all internal business application code.
“There has been no renaissance for Cobol,” says Accenture’s Burden. “There’s not a whole lot of new development going on. But our clients are enhancing their core applications and continue to maintain them.” Indeed, 53% of the survey respondents said they’re still building at least some new business applications in Cobol. The vast majority of that code is still being written for mainframes.
But the fact is that many IT organizations don’t have much choice but to continue using Cobol. Migrating large-scale systems built in Cobol is costly and risky. “They might want something more flexible, but they just can’t do it. They’re captive to Cobol,” Burden says.
The down economy has helped put off the inevitable, says Compuware’s Vallely. “Economic issues provided everyone with a hall pass because not as many folks were looking to retire,” he says. But as the economy improves, retirement plans may pick up too. “Organizations are trying to be more proactive,” he adds.
“No other language has seen as big an impact from changes in the demographics of the workforce as has Cobol,” Vecchio says. Going forward, it will become more difficult to maintain a Cobol portfolio. “The inflection point will come when enough Cobol programmers have retired that an organization can no longer tolerate the risk,” he says. At that point, most of those programs will migrate — but not all.
Rightsizing Cobol
For BNY Mellon, those Cobol batch and transaction processing programs on the mainframe represent an enormous investment. And while Gartner says it’s technically possible to move individual mainframe workloads of up to 3,000 MIPS, the bank’s aggregate workload, which relies heavily on Cobol, uses a total of 52,000 MIPS of horsepower, spans nine mainframes and is growing at a rate of 10% each year.
“The business wants us to make investments in programming that buys them new revenue. Rewriting an application doesn’t buy them any value-add,” Brown says.
Instead, the strategy is to “rightsize” some noncore applications off the mainframe where there’s a business benefit, try to keep mainframe MIPS growth under 5%, and stay the course with the bank’s core Cobol applications by passing on the business knowledge to younger programmers the bank will need to recruit and train (see “Closing the Talent Gap”).
Other functions, such as general ledger and reporting, are moving to distributed computing platforms, where they are either replaced by packaged software or re-engineered into Java or .Net applications.
But Brown still needs Cobol programmers to replace those expected to retire, and the learning curve can last for a year or more. That means adding staff and having a period of overlap as Cobol’s secrets get passed on to the next generation. “I’m trying to get those people on board and do the knowledge transfer sooner rather than later,” Brown says.
But that kind of proactive approach, and the extra costs it incurs, can be a hard sell. “We haven’t gotten to the point of feeling the pain yet. When we do, it will happen,” he says.
Brown wouldn’t specify the number of people he’s hoping to hire, but he says the “real heavy need” will happen in the next five to 10 years, when the original mainframe programmers are expected to retire en force. BNY Mellon currently employs “a few hundred” Cobol programmers, he says.
Brown’s concerns are well placed, says David Garza, president and CEO of Trinity Millennium Group, a software engineering firm that has handled code transformations for large businesses and government organizations. “Almost every job we get has Cobol in it,” he says, and most of the calls come from organizations that have already lost their collective knowledge of the business logic. At that point, he says, a migration is “a big risk.”
The Cost of Waiting
Trinity Millennium Group and similar vendors have established processes for analyzing and extracting the business rules embedded between the lines of Cobol code. “The solutions have come a long way in terms of the ability to extract logic and rules,” says Burden.
But the process is time-consuming and costly. One Millennium client recently spent $1 million to have its Cobol programs analyzed and business logic reconstructed as part of a migration off of a mainframe. “If they had the legacy programmers there and we had done the exercise with them, it would have cost $200,000 and taken one-tenth of the time,” Garza says. If you wait until that institutional knowledge is gone, he warns, the costs can be as much as 10 times higher than they would have been beforehand.
Compounding the loss of skills and business knowledge is the fact that, for some organizations, decades of changes have created a convoluted mess of spaghetti code that even the most experienced programmers can’t figure out. “Some systems are snarled so badly that programmers aren’t allowed to change the code at all,” Garza says. “It’s simply too risky to change it. They’re frozen solid.”
Jim Gwinn, CIO for the U.S. Department of Agriculture’s Farm Service Agency, faced that type of situation. The USDA’s System/36 and AS/400 systems run Cobol programs that process $25 billion in farm loans and programs. “We have millions of lines of Cobol, and there’s a long history of it being rewritten,” he says. “It has become increasingly difficult to change the code because of the complexity and the attrition of the knowledge base that wrote it.” That’s a big problem because laws that govern farm programs change every year, driving a need to update the code to reflect those changes.
Gwinn hired consultants from IBM, who concluded that rewriting the programs in a different language or rehosting them on a distributed computing platform would be complicated and costly. But the System/36 hardware had to go, so Gwinn decided to bite the bullet: The FSA will move off of its end-of-life mainframe systems by rewriting some of the code in Java and replacing the rest with packaged software from SAP.
But Gwinn says he’ll miss Cobol. “It has been very stable and consistent, with little breakage due to code changes, which you see with Java-based changes,” he says. “And in a distributed environment, you have to balance your workloads a little more carefully.”
Going for a Rewrite
The anticipated exit of institutional knowledge and the resulting shortage of Cobol programmers were also primary drivers of NYSE Euronext’s decision to re-engineer 1 million lines of Cobol on a mainframe that ran the stock exchange’s post-trade systems. While Cobol was dependable, it wasn’t viewed as maintainable in the long run.
Steven Hirsch, chief architect and chief data officer at NYSE Euronext, cites the need to make changes rapidly as another reason the stock exchange abandoned Cobol. “Ultimately, the code was not easily changeable in terms of what the business needed to move forward. We were pushing the envelope of what it took to scale the Cobol environment,” he says.
So NYSE rewrote Cobol programs that run its post-trade systems for Ab Initio, a parallel-processing platform that runs under Linux on high-end Hewlett-Packard DL580 servers. The new environment allows for more rapid development, and the rewrite has eliminated a substantial amount of unnecessary code that had crept into the original Cobol programs over the years.
If a business’s Cobol code doesn’t need to change much — as is the case for many batch and transaction processing programs — then the code can be maintained on or off of the mainframe indefinitely. But that wouldn’t work for NYSE Euronext. “We are a rapidly changing business, and we needed to move faster than our legacy code,” Hirsch says.
As for the stock exchange’s trading systems, they’re all built with proprietary NYSE Euronext software. “There’s no Big Iron or Cobol,” Hirsch explains. “There’s been no use of mainframes in the trading environment for many years.”
Rehosting: Lift and Shift
When it comes to hiring new Cobol programmers, Jonathan J. Miller, director of information systems and services for Saginaw County, Mich., is struggling. “We’ve lost our systems programming staff,” he says. And like many government IT organizations that have suffered from budget cuts, he doesn’t have much to offer those in-demand Cobol programmers.
Generous government benefits packages used to attract candidates even though salaries were lower than they are in the private sector. Now, he says, “our pay hasn’t increased in eight years and benefits are diminished.” The county has been forced to contract with retired employees and outsource Cobol maintenance and support to a third party — something that just 18% of Computerworld survey respondents said they’re doing. (See the full survey results here.)
The Cobol brain drain is becoming critical for many government organizations, says Garza. “It’s a high-risk problem in many countries [Trinity Millennium is] doing work in. The people have retired. Even the managers are gone. There’s no one to talk to,” he explains.
Saginaw County found itself hemmed in by the complexity of its Cobol infrastructure. It has 4 million lines of highly integrated Cobol programs that run everything from the prosecutor’s office to payroll on a 46 MIPS Z9 series mainframe that is nearing the end of its life. With mainframe maintenance costs rising 10% to 20% each year, the county needs to get off the platform quickly.
But commercial software packages lack the level of integration that users expect, and Miller’s team doesn’t have the time or resources to do a lot of integration work or to re-engineer all of the program code for another platform.
So the county is starting a multiphase project to recompile the code with Micro Focus Visual Cobol and rehost it on Windows servers. An associated VSAM database will also be migrated to SQL Server. Miller hopes that the more modern graphical development suite will make the Cobol programming position, which has gone unfilled for two years, more attractive to prospective applicants. But he acknowledges that finding talent will still be an uphill battle.
A Legacy Continues
Is there a role for Cobol off the mainframe? “I don’t believe there is. Cobol and the mainframe run well together, and that’s where I want to keep it,” says BNY Mellon’s Brown. The bank still creates new Cobol components on the mainframe and will continue to do so.
That’s a common sentiment among Accenture’s large corporate customers, says Burden. Cobol will continue its gradual decline as midrange systems are retired and businesses continue to modernize legacy Cobol code or move to packaged software. Today, Cobol is no longer the strategic language on which a business builds new applications. But it still represents the “family jewels” of business, Burden says. “They’re enhancing existing applications and adding functionality to them. I’ve seen no slowdown in those activities,” he explains.
If companies can’t find talent to keep that infrastructure going, third-party service providers such as Accenture are ready, says Burden. The scale of Accenture’s support operation is large enough to provide a career track for Cobol programmers, and he notes that it’s easy to cross-train on the language. “We can turn out new programmers quickly. If clients can’t support Cobol, we will,” he says.
“People make too much of that trend that we’re not graduating enough Cobol programmers,” says IBM’s Stoodley. Preserving the institutional knowledge is what’s critical. “You can make a problem for yourself if you don’t keep your team vibrant,” he says. But as long as there’s a demand for it, “businesses will find people willing to work on Cobol.”
Cobol may have been created for simpler times in application development, but it remains the bedrock of many IT infrastructures. “You have to respect the architecture of Cobol,” Burden says. “I don’t see that changing for another 10 years, or even longer.”
Mari Keefe, editorial project manager, provided research assistance for this survey.
