Information Technology (IT) managers around the world should brace themselves for a staggering surge in the volume of digital information within the next three years, according to a recent study by research firm IDC in Framingham, Mass.
The study, called “The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010,” predicts that increased use of rich media will push data volumes to more than 988 billion gigabytes in the next three years.
The survey of companies and government organizations was sponsored by storage tools developer EMC Corp. of Hopkinton, Mass.
Current digital data volume is around 161 billion gigabytes.
IDC notes that this is equivalent to three times all the information in all the books ever written or equal to 12 rows of books stacked end to end, extending from the earth to the sun.
“The year 2007 will be the first time the volume of information created will theoretically surpass available storage capacity,” according to David Reinsel, program director of storage research at IDC.
He said while there was no cause for alarm, the survey results did reinforce the need to rationalize data storage and management.
“A lot of data is needlessly being copied and stored, and it is not cheap.”
Although the survey did not cover cost structures, the IDC analyst said storage of unnecessary data drained cooling and power, real estate, software and management resources.
Both IDC and EMC traced digital information growth to increased Internet usage and the ever-increasing conversion of physical data to digital format.
According to Ken Steinhardt, chief technology officer for EMC, there’s been a dramatic change in the nature of data being stored, with digital data volumes growing significantly.
About 30 years ago, Steinhardt said, the bulk of communications dealt with structured data.
Today, he said, the ubiquity of multimedia technology had led to an explosion of unstructured data [categories] such as music and video.
“We’ve moved away from the simple transactions of years past to the swapping of jpeg and mpeg files.”
While 30 years ago a typical enterprise user file might have represented a row of figures in the database, today a similar file may contain digital photos and images of documents or digitized audio recordings.
More than 95 per cent of current digital information is unstructured data. In organizations, unstructured data accounts for more than 80 per cent of all information.
The IDC study also indicated that 70 per cent of the digital information created by the year 2010 would be generated by individuals.
For instance, IDC estimates there will be more than 500 billion images captured by digital camera users by 2010. To this has to be added the growing number of e-mail messages from 1.6 billion e-mail boxes and 250 million instant messaging accounts.
Enterprises and other large organizations would, in some way, be involved in the collection or handling of this data, said Reinsel. These institutions would be responsible for meeting privacy, security and compliance issues connected with at least 85 per cent of the information, he added.
Today, 20 per cent of all digital information was subject to compliance rules and standards, while security practices and procedures would apply to 30 per cent of corporate digital information, said IDC.
The consulting firm estimates, however, that less than 10 per cent of organizational information is currently classified or ranked according to value.
“It is important for enterprises to realize they will be trusted as stewards or custodians of this data,” said Reinsel.
“This ever-growing mass of information will put a considerable strain on IT infrastructures; it will change the way IT professionals do their job and the way consumers use information,” added Mark Lewis, vice-president and chief development officer, EMC.
Steinhardt said to effectively manage data, IT managers should adopt the following best practices:
– classify data according to importance, need for access, relevance to business operations and compliance;
– create defined policies for storage, access and eventual destruction of data; and
– choose the appropriate software and hardware to accomplish data management goals.
Reinsel suggested IT managers could look to server consolidation and virtualization tools to increase data storage capacity.
He said various strategies could be used to streamline data management processes.
These include “de-duplicating” software that actively guards against replication of saved data, as well as using applications that associate access policies with documents.
While the cost of storage technology has gone down, the cost of data management continues to rise, according to Parag Suri, category manager, StorageWorks division, Hewlett-Packard (Canada) Co., based in Mississauga, Ont.
“Data management and operation could actually cost twice as much as technology by 2008,” said Suri.
For instance, a typical bank could be dealing with at least 2,000 terabytes of data, he said. The same organization might have to dedicate at least one IT personnel for every 40 terabytes of data.
“There is a growing drive to reduce the number of people managing the storage environment.”
One of the key solutions, Suri said, would be to consolidate technology and to resort to virtualization.
Like Reinsel and Steinhardt, he recommends the setting up of data protection and retention policies. This he said should be accompanied by an enterprise-wide information and training campaign.
“What good is a policy if people are not aware of it or not applying it?”
Suri also said IT managers should “view data storage as a utility.”
Storage – like utility services – has seasonal characteristics, he noted. There would be periods when storage was at peak or low demand.
“Recognizing these service peaks and valleys will enable IT managers to plan data storage allocation and reduce cost.”
Related content:
Institutions unveil high-performance computing network
Storage: it’s really about filing cabinets
