Behind the scenes at the world’s largest online marketplace for used, rare and out-of-print books, the IT system handles 55 million books/SKUs and, daily, about 10 million static home page hits, about 1.3 million book searches, two million Java servlet hits, plus generates about one gigabyte of transaction information per hour and executes 1000 SQL statements per second. Having grown tenfold already in under five years, Abebooks of Victoria, B.C, is preparing for continued explosive growth.
“We’re discovering we’re dealing with the tip of an iceberg; we’re not too sure how big the iceberg is,” reports Marci Crossan, Abebooks manager of marketing and communications. “We’re the only company selling books this way online. We’re seeing the growth potential of online sales of rare and used books is just enormous. So when you’re talking about planning for growth, we see it coming – we just don’t know how huge the possibilities are.”
The tale of Abebooks is a classic read on identifying a business need/opportunity, providing adequate support – including IT – to successfully make the most of that opportunity, then adapting to change.
Advanced Book Exchange Inc. began in May 1996 as hardly more than an experiment by four Victoria, B.C., antiquarian booksellers looking for a place to list their books online. It provides independent booksellers with business support, technological infrastructure and international marketing, enabling booksellers of any size to compete on an equal footing with book superstores. In addition to English, French and German versions of Abebooks.com, the company recently added Spanish-language books and authors.
The company has grown to 80 employees in Victoria, B.C. and another 10 in Dusseldorf, Germany. The latter is a result of a 2001 acquisition of JustBooks, a small online startup in Germany and France. One programmer in Germany provides that office with IT support and works with developers in Victoria developing new products for Abebooks.com. Between the Victoria and Germany sites, Abebooks can provide customer service support for most of the day.
“In the six or seven years, we’ve grown to list books for 12,000 independent booksellers in 42 countries,” says Crossan. “Everyday on our Web site, 20,000 books change hands. A bookseller will list a book and a book buyer around the world will buy a book. If you’re looking for a copy of the Great Gatsby, if you do a search, you’ll probably come up with 20,000 different copies from a $1 beat-up old paperback copy at a bookseller in Idaho to a signed first edition for $100,000 from a bookseller in Oxford, England.”
Predicting the growth is critical to ensuring adequate IT support which in turn directly affects the company’s bottom line. “Whenever we’ve made our site faster, it appears that our sales go up,” says John Snider, manager of operations. “The faster someone can order a book, the more likely it is that they will order a book. The more responsive you are, the more likely that a customer will stay and conduct business.”
However, the company’s track record for predicting growth has been deplorable.
Explosive growth
“The growth was so explosive that every year we had to replace our hardware,” admits Snider. “We would buy what we thought at the time would last two or three years by just adding additional CPU and RAM and disks. But every year we maxed out the capacity.”
So for each of the past four years, they bought new larger, monolithic symmetric multi-processing (SMP) hardware. However, a crisis last September prompted them to make longer-lasting improvements. As students flocked to the site to buy textbooks, Abebooks’ Web infrastructure could not handle the expected but nonetheless dramatic jump in volume. Parts of the site crashed, customer response times were unsatisfactory and ultimately Abebooks lost sales.
To handle the September load, Abebooks moved all of its databases into production, doubling its processing power by tapping into the processors on the redundant business continuity/disaster recovery system. Obviously a better and long-term solution was needed.
Originally, when the Web site was developed by the two founders, the development system FoxPro was used for the front end and the database. As the company grew, so did its inability to handle the volume of data. About five years ago, they turned to the Oracle database, first on NT, then on Unix. They had earlier considered Oracle Real Application Cluster (RAC), but could not justify the cost. Snider reports that last fall, however, they found the costs and benefits weighed favourably for the cluster compared to adding two more CPUs to their latest previous upgrade, an RS6000 IBM server. They chose to implement the Oracle RAC and did so last November.
Database migration
One advantage to their annual upgrades prompted by unforeseen growth is that Abebooks’ 25-member IT staff gained what Snider calls “a wealth of experience in how to migrate one database server to another.”
Five years of experience in moving live, online databases with no transaction loss and practically no downtime resulted in remarkable efficiency in going from an Oracle SMP environment to an Oracle RAC.
“From the time the vendor said ‘there’s your hardware environment,’ we basically had an Oracle RAC up and running in three days,” Snider recalls. He says that although Oracle originally offered to provide consulting services and estimated it would take 10 days, including data migration, they took only about five days to migrate almost 300GB of data. “We didn’t lose a single transaction.”
And all with no noticeable downtime for the Web site.
They migrated the data by using replication which saved all the transactions. In copying over the books table, which is the biggest database, for example, “we’d basically throw a peg in the sand and say all the transactions from this point forward, we have to save and apply them once the books table has been copied over.” Then they switched from the SMP box to the cluster box, losing only up to a minute’s worth of sales going from one database to the other, according to Snider.
He notes that their cluster architecture has addressed their growth issues and will provide Abebooks with the vital IT support for performance, availability, fault tolerance and flexibility/scalability.
He feels confident that they will “be able to upgrade and be responsive to our users’ requirements for performance in the future at a much lower cost than we’ve experienced. When we max out our growth, we foresee that we’ll just have to add another node to the cluster. Instead of spending up to $250,000 for a new SMP box, we’ll spend maybe $30,000 for a new node.”
Now Abebooks relies on three IBM Xeon Intel-based boxes, redundant switches to connect the cluster to the network and StorageTek disk array for shared disk storage in the cluster. When the cluster was installed, Abebooks was using 10 per cent of the processing power in their online environment, providing room for seasonal sales spikes and near future growth.
IT performance and availability affects three user groups. Book buyers searching for books on the site want a fast response. Booksellers want to be able to quickly upload data. The company’s internal users, like the finance department running a query or some invoices, want quick processing capabilities. Crossan says all groups have noticed the faster processing speed of the latest IT upgrade.
Online performance
Abebooks attributes a recent 25 per cent increase in sales volumes to book buyers who were previously frustrated by the old system having been won back because of improved customer services, such as faster searches through the books in the company’s database. “We’ve got 16 books sold a minute, so every minute that our site is down or slow really impacts our bottom line,” Crossan stresses.
Clearly, lost sales due to web site crashes are unacceptable for a company whose main business is entirely online. Lost sales for Abebooks also means lost sales for the 12,000 independent booksellers who rely on the site to sell their books.
Internal users are finding that day-to-day business processes like financial reports, batch orders and database maintenance tasks that were taking days or hours to complete are now taking hours or minutes. For example, the maintenance and defragmentation of the database’s search index – a crucial component for consumers to easily and accurately locate the book they’re seeking – now takes a few hours, compared to three days in the old environment.
“As soon as we went live, everyone seemed to notice [the improved performance],” agrees Snider.
Another performance issue is being able to separate the online processing component of the Web site where the sales take place and the batch component where booksellers provide the books to be listed. “We have this huge volume of books come in from booksellers to list so we have to crunch through all these books to get them into our database,” says Snider. Counting updates and changes, he estimates they do a half million book changes every day. Formerly, the batch processing would steal CPU cycles away from the online processing. So, what people were doing behind the scenes could impact the sales cycle!
Now with the cluster, they do all their batch processing on one node and all sales or online transactions on another node. There will be some contention at the disk array level for the database block but that is very rare, says Snider. “It could occur that a batch process would want to touch the same database block as an online process. They will have to transfer the block between themselves, but that is fast with the new Oracle technology. The cache usually transfers the block from memory to memory rather than going through disks. We don’t see very much contention at all. The fact that we have online and batch separated means that our online processing is much faster.”
Fault tolerance
Another priority is fault tolerance. A monolithic SMP environment is a single point of failure, Snider stresses. The cluster environment permits fully redundant nodes. Abebooks’ three-node cluster includes two online nodes and one backup in case of failure.
Already, the cluster has saved the day. In January, a system board failed on each of the two online nodes; the second one happened two hours after the first. “We looked at the serial numbers on the two boards that failed and they were 10 numbers apart so there was basically a bad production run from the manufacturer,” Snider suggests. “We were still able to keep our Web site up and running because we were using a cluster.”
Regarding flexibility, Snider appreciates the fact that Oracle runs on different architectures so it will handle the next generation of chips that comes out from whoever – whether it be AMD or Intel or HP or IBM, he says. That’s important because of the need to offer users the fastest performance possible.
Staff members at the Victoria office seem to be upbeat about their latest processors. Describing Abebooks staff as a combination of “bibliophiles and tech-heads,” Crossan reveals that they used to call their RISC servers Poe, Dante and other classic authors and poets. Now the IT department has named their latest Xeon processor-based boxes after comic superheroes.
Supporting the frenetic world of Abebooks.com where the churning, volatile inventory must always be current and the site never allowed to sleep does indeed seem like a challenge for superheroes. But, this is no fantasy world. Like other innovative e-commerce operations, it is a living business, requiring constant care to remain successful.