Nicholas Carr has an article in The Atlantic titled Is Google Making Us Stupid? that got me thinking over the weekend. I’ve noted before how Google’s ubiquitous nature can have unintended consequences, such as the de facto standardization of spelling and grammar. In his article Nick notes that the nature of searching for information is having profound impact on the way in which he and others read:

“. . . what the Net seems to be doing is chipping away my capacity for concentration and contemplation. My mind now expects to take in information the way the Net distributes it: in a swiftly moving stream of particles. Once I was a scuba diver in the sea of words. Now I zip along the surface like a guy on a Jet Ski.”

I’ve noticed this myself as well, a feeling that the Internet is condensing all information into Cliff’s Notes. In particular, this has been quite concerning in the blogosphere, where it often seems folks expect hard issues to be discussed and resolved as quickly as they can post. In a previous post on standards for cloud computing I cited a few threads where folks expected publication of specifications and standards as easily as posts to a blog. Of course, that simple isn’t possible, but I believe that the speed with which we access information online builds that expectation.

It appears that one unintended consequence of unhindered access to information is a sort of commoditization of thought. Information is simply so easy to come by that we tend to value it less. Worse, ideas and the effort to communicate them are discounted by the sheer volume of information that floods our senses at every waking moment.

As an example, when I was an undergrad some 25 years ago at CSUF, research was painful. I might spend hours scouring through microfiche, racks of books or the dreaded manual stack for the CDC Cyber mainframe trying to find information. Having unearthed what I was looking for, I was certain to write it down or copy it.  Today, however, I write down as little as possible. Written information is the first thing I lose. I also almost never copy of print information. Rather, I rely on my ability to search for it again. Google has become an integral part of my process for consuming information.

I can’t share Nick’s more pessimistic view of the situation though. Humans are resilient and have a way to adapting technology to their needs even if at first it appears the other way around. I, for instance, picked up an Amazon Kindle a couple months ago. Using the same technologies that can commoditize thought, the Kindle places entire books at my disposal even when I’m unconnected at 30,000 feet. Since getting it I’ve read two novels, three business books, a science book, and numerous magazines and newspapers. That would have been more than a year’s reading before. Plus, I’ve noted more young kids carrying books with them recently than laptops. They’ve got their phone for texting, and their iPod, but no laptop. Perhaps, the market is already pushing back.

Kevin Kelly has a post on The Technnium about whether google sized data sets will lead to a new way of doing science:

“There’s a dawning sense that extremely large databases of information, starting in the petabyte level, could change how we learn things. ”

Kevin’s post was pointed out to me by Jonah Stein and caught my attention because a couple AppLogic users are starting to build very large data sets and as a result we’re ocassionaly asked about how to access and distribute them. However, the post really is less about the technology of dealing with these data sets and more about whether they’ll change the way science is conducted, which is another subject I’m interested in so I read on. Kevin’s post was inspired by a Wired cover story “The End of Theory” by Chris Anderson who writes:

“There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.”

So, Chris’ theory is that if we have sufficiently large data sets we’ll be able to act merely on the finding of correlation rather than waiting to understand the actual mechanism that relates the data - the cause and effect. However, IMHO data without understanding isn’t knowledge. In fact, it can be dangerous because the conclusions reached through statistics are very susceptible to influence by what data is collected.

In the 70’s Harvard medical school researchers had reams of data correlating breast feeding infants with juvenile cancer. Simply accepting the statistics could have lead to some horrible decisions. Fortunately, they didn’t accept the statistics and further research showed breast feeding doesn’t cause cancer, but that carcinogens in the mother’s environment and diet were being passed through to the baby.

The Eugenics movement a century ago was based on statistics correlating skull shape of different races with intelligence test scores. This psuedo-science was in part responsible for Nazi atrocities. Of course, we understand now that the intelligence tests were geared towards white Europeans, but without that knowledge, the statistics seemed compelling to people of that era.

Even the low birthrate issues in Europe today could provide miscues if you just accept the statistics. Is the cause that people are less religous than in previous generations? Is it that government subsidies are higher than other countries, or that they’re lower than 50 years ago? Is it pollution or economics? There are statistics to prove each of these.

In pure mathematical sciences huge data sets will provide scientists with interesting insights on where to focus their research. Used properly they will provide a sort of short cut to new theories to test and those theories will provide feedback into the system on what data to collect in the future.

After a couple decades building technology, you come to realize that technology development is really a cycle rather than a vector. For instance, the way in which AppLogic packages an application and its infrastructure is analogous to the way you’d launch an app on an SMP system with a shell script that starts processes and connects them through sockets. Jason Brooks has a short but interesting piece about this subject, and the way in which the iteration adds value.

Often the parallels aren’t obvious at first, because you start out trying to solve a problem rather than recreate a technology in a new space. As an example, we weren’t trying to create an operating system when we started AppLogic - we simply wanted to help folks scale web services. We wanted to give them a way to express the application structure so it could be implemented by a system rather than operators. Only after we had the first prototype running did it became clear we’d built a new kind of operating system, a meta operating system.

What’s unique is that a meta operating system has no APIs of it’s own, because the code runs in guest OS’s. Still, the meta OS controls resources, schedules processes and provides the user a console.

Realizing this was an important step and made completing the project easier. Realizing we had an OS meant we now had a template to refer to whenever we weren’t sure how to solve a problem.