High End Computing Network Redesign

Technocrat

High End Computing Network Redesign

zogger

Mon Aug 25 19:12:00 -0700 2008

Computers.

manage

University of California at San Diego computer engineers have come up with a proposal to change the way data centers and clusters could be configured to give them better scalability at a cheaper cost for the interconnect speeds by altering the topology. They say they can use the same hardware, and drop costs to one seventh of what current examples are using.

"Large companies are putting together server farms of tens of thousands of computers - even approaching 100-thousand, and the big challenge is to interconnect all these computers so that they can talk to each other as quickly as possible, without incurring significant costs," said Amin Vahdat, a professor of Computer Science and Engineering (CSE) in UC San Diego's Jacobs School of Engineering. "We are proposing a new topology for Ethernet data center connectivity." SIGCOMM paper in PDF: A Scalable, Commodity, Data Center Network Architecture

High End Computing Network Redesign

Timothy Brownawell

Tue Aug 26 10:05:14 -0700 2008
manage

It looks like this is cheaper because it provides more redundancy, you have 10 1Gbit switches more-or-less in parallel instead of 1 way more expensive 10Gbit switch.

High End Computing Network Redesign

itwerx

Tue Aug 26 12:50:06 -0700 2008
manage

Agreed, they're not doing anything new at all. This is actually a fairly common architecture in HA data centers. (I guess that's what happens when you spend most of your time in academia instead of the real world. :)

High End Computing Network Redesign

mcrbids

Tue Aug 26 23:51:55 -0700 2008
manage

So, where do I go to learn more about this type of architecture?

High End Computing Network Redesign

Charles E Hill

Tue Aug 26 17:10:25 -0700 2008
manage

Academia, meet Real World.

Quoth the PDF:

"We assume a cost of $7,000 for each 48-port GigE switch at the edge..."

Dell sells their 6248 GbE switch, with additional 48 Gbps stacking module and a 10 GbE uplink for $3,000. And before anyone whines that it is a Dell and not a "real" switch like a Cisco, HP or Juniper, you need to realize that GbE switches are now commodity. These Dells support everything that the big boys do. In many cases, they're made in the same factories.

I've installed 3 of these just recently, with 2 connected via the stacking module (48 Gbps) and the third via 10 GbE coax. I'm in the middle of monitoring traffic patterns to determine the best layout, but I'm 99% certain that workstations <--> servers and servers <--> servers, but not a lot of workstations <--> workstations. Servers are on the "core" switch, and each of them (5) are connected via quad-port NICs.

My main question is do I assign separate NIC interfaces, or just aggregate the quad into a single 4 Gbps bonded connection. I'm also tempted to include the 2 on-board NICs in that equation, making it a 6-port bundle per server.

High End Computing Network Redesign

itwerx

Thu Aug 28 23:07:32 -0700 2008
manage

The specific benefits of aggregated vs assigned will depend a bit on the switch fabric's own max capacity and how it responds to being saturated, (might be worth a little more testing and architecture tweaking to handle burst cases), but as a general rule aggregated is always better.