Over the last week there has been no shortage of online articles, blogs and tweets about Prism, the program used by the United States National Security Agency for collection data in real time from companies such as Apple, Facebook, Google and Microsoft.
For instance, just as Google and Facebook have cheaply developed the technologies and techniques to glean from our online interactions how to calculate an individual’s preference for clothes, they have also help bring down the cost of analyzing a person’s chat logs.
U.S. spy chief defends phone, Internet surveillance
Mozilla launches anti-snooping Web site
It is also often thought that government agencies would likely chose proprietary Windows systems over Linux operating systems because open sourced systems could introduce risks since their underlying structures are open to other researchers. However, open sourced software benefit from the myriad of researchers working on it who can spot more bugs and work together to fix them.
The NSA and other covert organizations still spend millions of dollars in super computers such as those produced by Cray for pattern matching, says Bloomberg writer Ashlee Vance. But for co-relating nuggets of data culled from a myriad of online, mobile and other sources, open sourced software is a spy agency’s special weapon of choice, she reported.
Is it any coincidence then that Facebook, Google and Yahoo, all avid users and propagators of open source programs, also happen to be on the list of companies alleged to have complied with court orders to release information to federal agencies?
Vance also cites that for years, companies such as IBM, Microsoft and Oracle have sold their revolutionary database technologies to so-called “three-letter agencies.” More recently, Hadoop, the open source software framework for big data and MapReduce, Google’s programming model for processing large data sets, have were included in the leaked NSA slide which identified the technologies as vital in the NSA’s data mining operation.
Both technologies were open source applications develop by Google and Yahoo and later modified by thousands of developers. Such collaborative works gave rise to the big data era and its accompanying data analytics tools.
As early as 2009, the NSA had admitted it had taken source codes from companies such as Yahoo, Facebook and Twitter which have open sourced their underlying infrastructure. The NSA said it modified some of these codes for data analytics purposes.
In so doing, NSA was able to drink from the spigot of intelligence of thousands of computer scientists and programmers from all over the world.