Wright PC Consulting, LLC

This week technology followers including Slashdot and Data Center Knowledge have been reporting yet another move by the search, SaaS, storage and email giant Google that may have many wondering whether the air inside Googleplex is too thin or just filled with a special creativity element.

Google has reportedly filled a patent application for the development of offshore data centers being dubbed the “Google Navy.”  This discovery has raised a few eyebrows and generated some discussion along with questions:  What is an off-shore data center?  Is it tethered like many casinos?  Who will protect them?  Where will they be located?  How will they connect to the land loving networks?  Will this too fall short like their shipping container concept or thrive using it?

Only time will tell.  However, while not he first to conceive the concept of a floating data center, they may be the first with the deep enough pockets to bring the concept to a fully deployed non-tethered conclusion.  The concept of placing your data permanently in the water might sound a little counterproductive, it certainly has some truly green and finical merits, if not incentives.

One of the more obvious is the lack of required capital to acquire the land and develop the infrastructure to build and maintain their land based data centers.  In addition, the cost savings in land and other related taxes alone must have the pencil pushers excited.  Finally, simply finding the adequate space needed for the big players in the data center build-out race must be getting more difficult as the data centers get larger and larger, not to mention the resources and infrastructure required to support the data giants.

Putting the financial benefits aside, there are also other green reasons that this seemingly wacky plan may just pay off for Google.  Google is reportedly looking at both cooling and powering the floating centers using green technology that harvests the natural wave motion of their floating foundations and converting that energy into usable and unlimited, re spendable resources.  By placing your data on top of easily harnessed energy resources, Google can power and cool its center without using more costly, dirty and earth destroying traditional sources.

So, perhaps the concept of floating data centers may seem a little bit of a stretch, so too did the idea of indexing the infant internet 10 years ago when two MIT students tried to conveince us otherwise.

A flaw in many DNS servers leave users vulnerable to a redirect exploit that could send them to the darker side of the web. The flaw was discovered by security researcher, Dan Kaminsky had been advising ISPs and others to patch their DNS servers to protect themselves and their customers. Through an unfortunate mistake details of the flaw were released into the wild this week when those details were posted on a blog. That post has long since been removed. However, the cat is already out of the proverbial bag.

So, what does this mean for you? Hopefully your ISP was one of the more proactive and has patched their servers. Not all have been so swift and have left customers vulnerable. To see if your DNS servers have been fixed, go here for an instant analysis.

If you have followed my earlier advise and are using OpenDNS, then you can skip the test. OpenDNS servers are not vulnerable to the flaw as shown by the test results:

My advise to customers who are either unsure about thier DNS servers or if thier servers fail is to switch to OpenDNS at least temporily.  Who knows, with the feature set that OpenDNS offers, you may decide to use them as a perminent solution. I did a long time ago and have been a happy with the speed, features and security that I now expereince. You may too.

It appears that Surpass Hosting has experienced yet another outage, on a weekend. Many customers, including Wright PC Consulting noticed server loss of connectivity just after 8AM CDT.  Once users were able to reconnect to their servers, many of us noticed that they had been rebooted, some more than once.

So, we must ask again: “What happened at the Dime?” Customers at WebHosting talk and at HostDime’s community forums even reported that once again that the voice lines were out during the network/data center outage as well. That makes since because they use VoIP for their support lines.

While this incident was a shorter event that the last one, downtime is downtime and means potential loss of profit, and even more damaging for many– our reputation. Unlike the outage in May, were there was a pretty steady stream of information, this time so far all that we have heard was this from staff:

“Our facilities are still experiencing some intermittent connectivity issues at this time. Our engineers are on-site diligently working on this issue. We expect to give a more detailed report within the hour. We greatly apologize for the inconvenience this is causing, but please know that we are doing everything in our power to get everyone back to full operation. Thank you for your patience during this time.”

Hopefully, Surpass and DimeNOC will have more information on the outage and what is being done to reduce he probability of it happening again. Things like why customer servers have to be rebooted to fix a networking issue within the data center would be a good place to start.

07/14/2008-Update from Surpass/HostingDime:

“Greetings to all,

I apologize for the delay in getting this up, per the power issues we ran into May 23 2008 where we mentioned that things were being done to be pro-active on ensuring no such outages occurred again, we had scheduled a planned UPS / Battery maintenance to replace / add some things to our systems. This was scheduled to be done Saturday July 12 2008 between 6AM EST and 10AM EST. There was no public announcement made about this because this maintenance was not to effect any clients or internal systems. However just in case, we planned it on a weekend during early morning hours in case something did go wrong we would be on standby and being the weekend the least possible impact it would have on clients.

Shortly before 10AM est when the maintenance was about to be completed there was what we thought at the time a DNS network wide issue which we thought effected some servers in the data center. We noticed the call volume increase and immediately began to investigate the issue. We advised staff taking calls to let everyone know it was an intermittent connection issue as we didn’t know what the real cause was just yet. Shortly after that we noticed that the EATON UPS maintenance staff had completed the maintenance and attempted to move our entire DC Floor load back to the UPS. The whole data center runs off 3 phases, when the Eaton UPS staff switched the load to the UPS one of the 3 phases failed to fully switch and caused over 33% of the servers in the data center power cycle. It was so sudden and so fast we initially thought it was a DNS issue and thus why it was said initially as a possible network issue when in reality it was one of the 3 phases failing to fully switch over to the UPS. During this time there was about 2-3 intermittent loss of power to the 3rd phase of the load thus why some of you saw your servers go on and off more then once. We immediately went to the Eaton Power UPS staff and advised what happened as this maintenance was not to suppose cause any service interruption to any clients. One of the tasks of the maintenance was to replace the capacitors in the system. The UPS system has over 50 of these capacitors which provide power via 3 electrical phases to the entire NOC, he immediately admitted that one of those capacitors was not wired correctly from the factory thus why 33% of the DC load power cycled on and off during attempts to move the load to the UPS as it shorted it out the phase.

They immediately corrected it and shortly after that the entire NOC power load was fully running on the UPS. A very very mis fortunate situation and we have scheduled a call with EATON today to get an explanation from them of how something this important was not wired correctly from factory. Eaton is a public company http://www.eaton.com/ and we are shocked how this happened and are determined to get to the bottom of it. We had an entire team of 25 staff on standby in case anything went wrong and immediately got 98% of systems online within 1-2 hours.

The situation is still unacceptable under no circumstances but out of our control. Our culture is built upon integrity and everything stated is 100% true. The only good news that can come out of this is that all the needed maintenance/changes/upgrades to our power systems is completed 100% and we are looking forward to the next 5 years ++ of uptime
We take downtime seriously and have acted pro-actively to ensure you get 100% uptime each and every single month. Services have remained 100% since then and are expected to remain 100% from here on out.

If there is anything we can do for you please just reach out to us and we will make it possible. We are determined to make your business relationship with us something you can count on with peace of mind. We put our entire soul and mind into this company with our entire 100+ staff team with a desire to give you the best level of service and support. We are at your disposal at anytime.”

__________________
Emmanuel :: Surpass Hosting Network Admin
http://www.SurpassHosting.com

Manny
Sys Admin HostDime