I just got back from San Francisco, where I spent the last few days at the Web 2.0 Expo.

I’ve always found tradeshows to be enjoyable because it’s a chance to meet partners and users face-to-face and find out what’s developing. For the past couple years, though, shows and conferences have lacked the energetic buzz about new gear, apps, and business models that makes you feel like you’ve had too much espresso.

IMHO, the O’Reilly folks did a great job with the Web 2.0 Expo this year. The aisles were crowded and people were busily bantering about new services. We were demonstrating AppLogic at the invitation of our friends at Layered Technologies and we routinely had three or four folks at a time in the small booth eager to learn about utility computing and how to get their virtual data center. All-in-all, an invigorating way to spend a few days.

When I first got into the computer industry the fastest computer in the world was the Cray-1, recognizable to this day for it’s unique C shape cabinetry. The Cray-1 marked the height of the supercomputer era and pushed the limits of engineering. Even the distinct shape was an example of the extreme engineering that went into the system. Arranging the cabinets in a C shortened signal times so the machine could run faster. Equally unique, though less visible, was the liquid cooling system that allowed packing the maximum number of components in the smallest space.

Supercomputers like the Cray eventually faded into history with the development of microprocessors, PCs and eventually parallel computing. At the time, I thought I’d seen my last liquid cooled computer as well. I was wrong.

A recent quote from Google’s Rhett Weiss in the Charleston Post and Courier, shows that liquid cooling is strategic again – though this time at the data center level. When asked how the company would use water and electricity at it’s new data center, Mr. Weiss responded:

“We’re in a highly competitive industry and, frankly, one or two little pieces of information like that in the hands of our competitors can do us considerable damage. So we can’t discuss it.”

Mr Weiss makes it clear that Google, the quintessential internet company, sees operations as strategic. So much so, that they even assemble their own servers. Steve Duplessie touched on why building data centers and computers is strategic when he wrote recently

“the overwhelming majority of data centers contain infrastructure products that were all designed for a different era.”

“This could well be the biggest inflection point in tech history…”

I spent an hour this morning trying to figure out why a cohort here at 3tera couldn’t access our Google Analytics account.
Eventually I deleted him and restarted the whole process only to realize as I did so that I’d used a different email address for him
in adwords. Problem solved.

However, there’s still the interesting question of why the email addresses have to be the same between two such loosely coupled systems. It’s not noted anywhere on the site. There’s no technical reason for that constraint. It’s just a business or engineering decision that was made along the way.

Such constraints are common, of course. As another example, have you ever tried upgrading Windows Media Center Edition to XP Pro? Again, there’s no technical reason, but that upgrade path simply isn’t allowed. The only allowed upgrade is to Vista, and unfortunately for the shop I returned my new laptop to I ran afoul of that rule months before Vista was released.

Why do such rules exist? I’ve encountered a couple sources. First, developers and product managers feel they know how their
product should operate. They’re fanatically dogmatic about the use cases for the system and feel comfortable with enforcing what they think of as best-practices. The second source is a desire to cause customers to behave in a way that’s profitable to the vendor. Nuff said about that.

If you’re at all creative, then for almost any interesting task you can usually come up with more than a few practical ways to get it done. After all, just think about how many ways you’ve used a hammer or screwdriver that the manfacturer probably never considered. Have you ever considered the impact on the manufacturer. How should the engineers designing your tools prepare their wares for your unpredictable usage. How would you react if your hammer actually prevented you from crushing walnuts.

Decisions about where to limit users’ freedoms are a regular point of heated discussion between any good product management team and their enginners. That’s certianly true of our engineering staff and myself. I’m a firm beliver in building tools instead of rules and dislike restricting options for users unless there’s an exceptional benefit. Sometimes, of course, that benefit is simply that the system can be built. Engineers, on the other hand want to limit the number of permutations they need to support. Too much freedom bloats the QA plan. More often, though, they feel there’s a “right way” to do something and want to enforce it.

Part of my bias toward building tools comes from being a geek. Like all geeks, I love finding new ways to use gadgets. There’s a serious aspect to this bias too though. After many years of experience building systems sufficiently advanced that they change user behavior I’ve learned that predicting use cases can be dangerous. Imagine trying to map use cases for traditional auctions onto the development of eBay for instance. It would constrict users so much they might not use the system at all. When I was a product manager for one of the first laptops, this was exactly what happened – we patterned our use cases on desktops. Eventually I got rid of my desktop and worked with just a laptop (pre 1990) and I learned why users weren’t buying.

I’m writing about this today, because I just learned about the first case of a user losing a volume on an AppLogic grid.
The customer has a virtual private server with one of our partners and the grid operator decided to remove some servers for maintenance. Ordinarily, if you follow procedures, this doesn’t result in any loss of data. In this case, unfortunately, the operator circumvented the procedures and the system operated as a tool. After warning about potential lost volumes, it allowed the operator to proceed with removing several servers at once.

Our engineers stepped in to help and it looks like only a swap volume was lost – no real data. None the less, it raises the question of whether allowing the operator that freedom was the right behavior for the system. I’m comfortable that alerting the operator was sufficient, but I’ll definitely be thinking about this instance the next time the subject of rules versus tools comes up.

Allan Leinwand has an interesting post on GigaOm yesterday about the CTO of a Web 2.0 company making a VC pitch. Alan was surprised the CTO didn’t have a plan for dealing with load balancing, networking and the racks of gear needed to deliver the solution.

“The Internet is like electricity. We plug into it and all of the things that you mention are already there for us. We don’t spend any time at all on network or server infrastructure plans.” was the CTO’s response.

Allan was an early employee at Cisco and CTO at Digital Island, so I’m not surprised he’d ask someone about their networking plans. “I’m old school,” he confesses.

IMHO, though, expecting the CTO of a Web 2.0 startup to have a networking plan misses the point. In the 90′s web startups had to have a CTO with knowledge of routers, switches, load balancers, vpns, caches and servers because much of the infrastructure was new and unproven. A decade later, can building yet another co-lo cage really add value. That’s exactly the reason I’ve been working on utility computing.

As an investor I’d be interested in how the CTO plans to increase the company’s value. Does he know his customers? Does he understand the the problem they’re trying to solve and the existing solutions customers are using. What about the competition? What technological shifts are happening that allow for a new solution. How will the new company change the landscape? What companies will be threatened by them?

In making a decision to invest, networks and servers would only be a line item on the financials.

Over the past couple days Jessie Smart, our engineering services manager, and I have been bringing up a grid for our webinar tomorrow and I think it provides a concrete example of what utility computing is really all about.

Two guys, one of them a marketeer, are running a system hosted by a partner with 86 processors, 150GB of RAM, 28TB of usable storage and 4.4Gbps of bandwidth to the internet. We’ve never seen the hardware, nor have the folks in the data center seen any of our applications. We now have 33 n-tier applications running, comprised of 297 virtual appliances and 759 volumes – all of which are consuming 64 processors and 67GB of memory.

It hasn’t all gone exactly as planned. First, not all of the servers were ready on schedule. We had a couple servers that kept restarting because of a misconfiguration. Plus, yours truly screwed up and we had to reboot the grid earlier today.

None the less, the total time invested is less than 40 man hours. (But, of course, many of them in a row and late at night.) More importantly, though, it’s 40 man hours from two guys who don’t make a living running servers or applications; Jessie and I have regular roles here that have generated numerous interuptions along the way.

In case you’re tempted, the cost of this system is less than $30K/month.

This afternoon we’re busily bringing up the grid for our upcoming utility computing webinar, and it’s a beast. 48 nodes, 96 dual-dual core processors, 192GB of RAM, 36TB of storage and almost 5Gb of bandwidth available to the internet. Best of all, for the next few days it’s the personal playground of yours truly -
courtesy of the event cosponsor Layered Technologies’ GridLayer division.

After the Webinar, of course, it’ll be up for sale.