Service-oriented architecture (SOA) can demolish the status quo. Decades of siloed system design have left most government organizations with antique, rickety systems that don’t play well with others. By putting new SOA wrappers on old proprietary applications, modular interfaces can be built, shared, linked, reused and recombined as needed, to create an infinitely interoperable IT utopia.
No need to rip and replace old systems; instead, they can be refurbished and extended internally and even externally via the Web. This is where SOA shows promise well beyond rejuvenating legacy enterprise systems, says Bill St. Arnaud, senior director of advanced networks at Ottawa-based CANARIE Inc.
“SOA is now seen as a key component in a broad range of fields beyond enterprise IT: chemistry, biology, everything,” he says. “Whether it’s a traditional payroll application or radio telescope research, it makes sharing, mapping and transferring data, and creating new mash-ups, simple.”
SOA can also have a profound impact on business processes. Many complex processes that require human back-and-forth can be automated as SOA-based Web services, which in turn can invoke other Web services, and then others, throughout the service chain. “If GM orders a phone line from Bell Canada [for example], it has to be validated, checked, tested, delivered and invoiced by many people,” says St. Arnaud. Instead, all the specialized steps in the transactions can be itemized, agreed in a contract, and automated as interlinking Web services between both companies.
Take-up of SOA is stronger in more competitive markets, he says. In the U.S., about 70 per cent of companies say they plan to invest in it over the next two years, according to IDC Canada research. In sluggish Canada, the figure is 40 per cent, with the public sector lagging still further behind the private sector.
Building this SOA utopia won’t be easy. There are many impediments, ranging from making the business case to fix systems that aren’t entirely broken to governance and liability issues to standards wars, notes St. Arnaud. Nevertheless, SOA is slowly but surely creeping into many areas of Canadian government.
The way SOA works
In 2005, the Alberta Ministry of Justice deployed a SOA-based enhancement to its maintenance enforcement program (MEP), driven by new legislation enacted the year before. Dubbed “Deadbeat Dad” legislation, the Ministry was concerned with finding ways to track and enforce adherence to court orders by withholding access to government services, explains Stuart Charlton, enterprise architect at BEA Systems Inc. in Toronto. “So if someone doesn’t pay child support, they might suspend their driver’s licence.”
As in other provinces, Alberta’s ministries and government departments are far from integrated. Tracking thousands of these MEP cases over time, sometimes more than 10 years, and building in the triggers for enforcement actions based on patterns of misbehaviour was a major system undertaking within the Justice Ministry. But developing processes to ensure actions are executed by a multitude of external ministries would have been a monumental inter-departmental undertaking.
Instead of labour-intensive back-and-forth between multiple departments – phoning, faxing, exchanging forms and information – the Ministry of Justice used Web services to automate the workflow. “The new approach is to agree on a shared contract that meets the needs of the consumers and producers of services and information,” says Charlton. “The biggest challenges in SOA are coming to those contractual agreements between parties, and the processes and change management around that.”
But once these issues are settled, the technology itself is relatively straightforward. A producer posts the services it has made available for common use – for example, an application that identifies an Albertan as an MEP case – through an electronic interface based on SOA standards, which can be used by any authorized consumer using the same Web-based technology. All the various conditions for an exchange between departments, including exceptions that require human judgement, are agreed and scripted in advance.
“So this is a conversation between the consumer’s and producer’s systems. The consumer says, I would like to request this service, and will validate the information I provide,” says Charlton. “It will then grab and use the service automatically if there is no exceptional condition. If there is, it’ll provide a worklist to another interface for human review. But all this communication is happening behind the scenes.”
Like a high-tech version of an Amish barn-raising, all the departments touched by the MEP program share and link the applications, processes and data that relate to it, instead of building a specific system for it.
Back-seat drivers
While the quest for competitive advantage may be a clear driver for SOA uptake in the private sector, the scenario is more complicated in the public sector. Without a profit motive, SOA is typically a hard sell that must be tackled indirectly and incrementally.
“There’s no imperative to adopt it. Instead, it comes up in IT projects after budgets are approved to renew applications or to build new ones. SOA is almost incidental,” says Michael Turner, an e-government strategy consultant and former ADM with Public Works.
The Treasury Board’s CIO branch has provided direction on SOA, he says. “It has done some ground-breaking work, providing overarching policies and standards,” says Turner. “But there’s no requirement for departments to comply or demonstrate they’re doing it to get budgets approved.”
Turner acknowledges that without a compelling need, it would be virtually impossible to obtain funding to rearchitect a system with SOA. “That implies jacking up the house to redo the foundation. People don’t do that. Instead they say, the next time we develop this application, we’ll build a different kind of foundation with more flexible components that can be re-used.”
But the consequences of putting off repairs to faltering IT infrastructure are higher costs in the long run, he warns. “The operational costs of application management are significantly higher than they need to be. And it’s more complex to make modifications or add-ons to non-SOA systems. We’re paying for the fact we’re not getting on board.”
Even a trivial change like adding a new field to a legacy system can cost millions, agrees Tom Metzger, director of solutions engineering at Vancouver-based Make Technologies Inc., which specializes in modernizing legacy systems. “Future maintainability of systems is a big driver in the public sector. And many government organizations are on the hook to respond to legislative changes.”
Some crown corporations have been early adopters of SOA, says BEA’s Charlton. “We’ve seen some aggressive adoption at Farm Credit and Canada Post.” He points out that Treasury Board is actively pursuing a strategy for IT transformation across the public sector. The challenge is overcoming the inertia to get started by focusing on agencies with a pressing need to undergo change.
“Agencies tasked with counter-terrorism are great examples,” says Charlton. “The RCMP and DND are starting to adopt SOA. They’re already doing it in front-line applications and they’re modeling their approach after the U.S. Department of Defense, which is based on a SOA foundation.”
The lack of a clear vision to sell to politicians and constituents contributes to funding issues, suggests Michael Kuhbock, founder and CEO of the Integration Consortium, an international industry association based in Calgary.
While creating citizen-centered services may be a worthy cause, the public doesn’t perceive any urgency. “They say: Everything seems to work okay, so why spend millions on SOA when there are other needs?” says Kuhbock.
Kuhbock believes an issue that fires passions and unifies efforts, such as environmental stewardship, may play that role in the future. Every service the government provides, and every supplier the government uses, has an environmental impact. “Right now, the data around that is not tracked,” he says, pointing out green issues are gaining traction with the public. “In the future, the public may say: We won’t stand for this anymore. We want to ensure we know what every service organization in government is doing to the environment.”
SOA is still a work in progress, and its capabilities are ill-defined and confusing. SOA facilitates system integration, but it doesn’t actually do the integration.
“It’s an architecture, nothing more,” says Kuhbock. “In the construction industry, you engage an architect to design the building so it has a sustainable infrastructure and everything works together – but then you engage construction crews to build it.”
SOA design is counter-intuitive and difficult for IT staff trained in classic waterfall systems design, adds Turner. “The traditional way is to look at application suites as a whole, design in the most integrated way possible, internally within the system, and use programming logic that produces tightly-bound functions with minimal duplication.”
While it may be an elegant system initially, the costs to make any modifications or additions later are high, since the system isn’t designed to change or grow over time, he says. “Then SOA comes along, intentionally breaking things down into blocks and duplicating functions so they can work as loosely coupled modules. It’s less elegant, but in the end, they’re cheaper to put together.”
Governance and accountability issues are also stumbling points when sharing SOA infrastructure and interlinking modules – a problem that’s more acute in the public sector versus the private, he says. The horizontal shared services movement in government butts directly against the brick wall of vertical organizational structures.
“The cabinet system we have in Canada is built on the intentional design of silos,” says Turner. “Each set of programs is enabled by legislation and statutes, and each organization supports a minister who’s accountable to Parliament for that set of programs. Natural silos are designed into the system, so when you try to apply common technology across departments, it goes against the grain. It almost comes down to, do you want efficiency or accountability?”
The tendency to treat SOA as an IT project rather than a strategic architecture is a consequence of these silos, says Kuhbock. At a recent presentation he made to government IT officials, many of them said they had the same central issue: “They can do SOA in their own compartmentalized area, but they can’t get it broadly driven; so they’re just stuck in their silos again. It’s like architecting a town, district by district, without thinking about how the whole town will interact, or mapping out the whole community.”
Vendors and standards wars
Beyond the public sector, there are some industry-wide SOA issues that need to be settled. “SOA is on a hype cycle,” says Kuhbock. “There’s confusion about it in the marketplace across all industries. There’s no universally accepted definition, so it’s different from one industry to the next.”
IT vendors feed this confusion by treating SOA as a selling point in their marketing strategies, suggests Kuhbock, with many branding their particular product or flavour around their own definitions.
“It’s all based on what they’re selling,” he says. “Everybody has different reasons for being active in this space, be they producers or consumers of solutions. All have different objectives, so it’s hard to get consensus, and this bleeds into every area of its use.”
As a consequence, the number of SOA standards is rapidly proliferating. “At last count, there were over 110 SOA standards,” says Kuhbock.
St. Arnaud agrees this lack of common standards is a major issue. “There are standards wars going on. Web services have exploded into a multitude of different variations and special sector types. Now suddenly, this great nirvana solution that could make everything compatible has itself become incompatible with different versions.”
But various government entities – the federal Treasury Board, the Ontario Government, Canada Health Infoway Inc. – have done extensive work in defining SOA standards and providing technical guidance for their constituent areas. So any lack of standard definitions should in theory be less of an issue in government relative to other sectors that don’t have top-down leadership.
Not so, says Kuhbock. “SOA has been muddled by the vendor community,” he says, pointing out public sector IT people still need commercial software tools to re-architect their systems. “If the government is dealing with vendors A, B and C, and each of those has a set of standards around their SOA-based tools, then the government is back into analysis paralysis sorting it out. No one’s really come out with best practices, methodologies and so on, and the vendors keep redefining the standards around their tools.”
Nor is any one major vendor’s version emerging as a de facto standard in the public sector, since the government uses a wide range of vendors to encourage competition. “You don’t just have IBM and BEA, you have Microsoft, SAP, Oracle, Accenture, Deloitte – all moving in different directions,” says Kuhbock. Industry groups like the Integration Consortium are trying to facilitate consensus, and vendors are starting to agree to disagree, but this process is proceeding slowly, he adds.
Kuhbock emphasizes SOA is not a software tool or an IT project, but an architecture that enables communication between services, data and processes. “If you’re constructing a building, you need concrete, beams and pylons, but if you don’t have an architecture, you’ve created an unstable environment that can crash. Many IT systems were built as silos, so they’re like buildings that don’t have staircases or elevators to join the first and fifth floors.”
Adding Web services to link the floors without a master plan may not improve the situation, he warns. Should you just add a stairwell to link the two floors, or do you need an elevator shaft for the entire building?
Software can be used to enable SOA, but it doesn’t create the architecture: a comprehensive strategy does that. “Once you map out the SOA strategy for an organization, then you can take out bite-sized pieces that are SOA-enabled projects. If you don’t do that first, you’re just adding another hairball to systems,” he says, pointing out many SOA silos are starting to pop up in organizations.
“If you’re using it to service-enable a couple of functions, but those services aren’t spread over the entire organization with the proper governance around them, they’ll just get lost in all the other hairballs in the IT infrastructure.”
SOA in health care
Canada Health Infoway is promoting SOA to evolve e-health records across the health care sector, says Michael Martineau, e-health practice leader at the Branham Group, an Ottawa-based research consultancy. “Infoway has produced reference architecture and provincial blueprints as guidance for this. All the provinces reference it in their plans,” he says, noting it’s all based on health care organizations exposing a suite of services that others can use.
The SOA framework developed by Infoway is not fundamentally different from the federal government’s approach, explains Martineau. “A framework can imply a variety of standards. Health care has some unique messaging standards and other differences in nomenclature. For example, when a service returns data, how it’s interpreted will be different for lab data compared to an e-commerce transaction. But the conceptual framework is the same.”
It’s possible to develop e-health records without SOA, but Martineau points out calling them “records” is misleading. “It’s a bad term because it’s not one single record. The data needed for it resides in different systems such as medication, labs, etc. An e-health record is really a system to pull data from different sources, so this initiative is a great big integration project. SOA is the best way to do it, and that’s why Infoway is pushing it.”
But uptake of SOA is relatively low in the sector. In a Branham survey of health care senior IT staff conducted last fall, about half said they had no plans for SOA. “This has to do with the proprietary architectures and complete vendor suites that are prevalent in the sector,” suggests Martineau. “Their view is that whatever the vendor gives us, that’s what we’re going to use, so SOA doesn’t make sense for us.”
Infoway faces many challenges in developing this area, he says. Many large health care IT shops such as hospitals are spending most of their limited budgets on maintaining their systems, so moving on SOA is very difficult. Nor is there any formal mandate to adopt SOA, beyond requiring it in project plans if organizations want to access Infoway dollars.
“But most of Infoway’s projects to date have been provincial or regional projects, and I haven’t seen much trickling down to individual health care organizations,” says Martineau. “And there’s the chicken-and-egg problem of vendors, who have to adopt SOA as well. IBM has been aggressive in moving to SOA, but some vendors are resisting as it serves their interests to lock in their customers.”
In Ontario, the regional reorganization into local health integration networks (LHINs) is providing a driver for SOA adoption in some areas, he says. There are two main approaches. Where there are many versions of a proprietary platform, health care organizations are consolidating their systems around one vendor’s platform. In Northern Ontario, for example, many hospitals are Meditech shops, and these will eventually be consolidated into a single version for the region, he says.
“We’re seeing the same thing in other parts of Canada. Until Meditech moves on SOA, it’ll be harder for these regions to move as well. But there are products from other vendors that wrap around Meditech to expose services as SOA to other apps, so it’s still possible.”
In areas such as the GTA that have multiple vendors and platforms, health care organizations are starting to adopt SOA to integrate within their regions, he says. SOA is the best approach for building modular interfaces around disparate platforms to glue health care systems together within a region, without getting bogged down in technical specs, says Martineau.
“For the first time, technology and business people can talk the same language,” he says. “The process for conducting lab tests can be defined as a series of services and functions, and you don’t need to care about the technical details or the programming language.”
There are back-end advantages as well, he adds. “You can take a service and split it across several servers, do load balancing and so on. People don’t care about behind-the-scenes adjustments. The old way, you did have to worry about this, as the business architecture was different from the technical architecture.”
SOA in municipalities
Vendor issues also play a role in the municipal sector, says Roy Wiseman, CIO of the Region of Peel. Municipal software is a very niche market with small vendors providing specialty software tools. “They’re not big-10 names,” he says. “It’s software for tasks like the collection of payment for parking tags and registration systems for voters.”
Municipalities are primarily looking for off-the-shelf products, but their specialty vendors are slower to adapt to new technology such as SOA. Nevertheless, Wiseman says, he sees significant changes afoot in the future. “SOA drives the software development industry more than it drives us.”
Larger municipalities such as Calgary and Edmonton, which have SOA projects under way, have more money and influence, and can attract vendors who will customize their wares or undertake software development projects themselves. But smaller municipalities don’t have the same clout or resources, he says.
SOA is starting to trickle down to the municipal level. “Over the next year, we’ll start seeing it referenced as a regular feature in RFPs for off-the-shelf and custom solutions,” says Wiseman. But most vendors aren’t ready to respond at present. “We’re shooting ourselves in the foot if we make it a mandatory requirement now, as none of the providers are at that point.”
Once it gains traction, Wiseman believes SOA will play a powerful role in sewing together the individual geographic information systems (GIS) across municipalities. Searching for GIS-based information across a province or region would become quick, simple queries instead of labour-intensive exercises.
“If you look at Peel or Ontario, there is really only one geography,” he says. “All of us have one subset of information about that region, be it street networks, gas and electricity lines or land parcels. We can build apps to pull that in from different sources, but we all need to do that in a standardized way. Building one-to-many interfaces is where there’s real complexity.”
Simplifying interaction between organizations is where SOA has the greatest value, says Wiseman. “We can all build one-to-one interfaces internally, but when you get into inter-jurisdictional dealings, then you need a common language.”
SOA can provide that Esperanto. But due to all the hype, some believe SOA is just another technology fad. While Wiseman doesn’t believe SOA is revolutionary, he does believe it’s here to stay. “It’s the evolution of the same idea that’s been around for years: object-oriented programming, CORBA and so on, which are all about fitting together software components so they interoperate.” SOA will likely continue to evolve, perhaps into something with a new name, but the fundamental approach to unifying systems will remain the same, he says.
Rosie Lombardi is a freelance writer based in Toronto. Contact her at rosie@rosie-lombardi.com
Related content:
SOA: A better ballgame with BTEP
SOA: It’s architecture, not technology
SOA: Understanding the architecture