Shared Nothing Storage in Open HA Cluster

Two years back I led and designed a project to make Solaris Cluster easy to use. The wizards that resulted from this effort is a key element of the Solaris Cluster user experience now. Many new projects want their features supported using the wizards today. The architecture of the project even made it to the IEEE Cluster conference.

This past year I led another important effort for Solaris Cluster. This feature, Shared Nothing Storage, that was released as part of Open HA Cluster 2009.06 removed a major hardware requirement for the cluster: the necessity to have a shared storage. This was achieved by configuring the iSCSI protocol stack present in COMSTAR in a particular fashion and layering a ZFS mirror on top. This feature allows a user to use any local disk present in the system as a storage for the service and to make that service highly available.

There is no need to turn disk fencing on for this configuration and therefore it also removes the need to have SCSI reservations. Here is a picture of the configuration, with detailed configuration instructions here.

The key challenge in providing this feature was to make the cluster device subsystem robust enough to handle devices that are attached via the network. The design details are present here.

This configuration becomes more interesting when I/O multipathing is configured, because it shows the flexibility and the power of the COMSTAR architecture. With COMSTAR, a single logical unit of storage can be accessed via multiple port providers, multiple iSCSI targets in this case. These multiple iSCSI targets can be used to create multiple paths to the same logical unit. This provides fast mirroring of data in the cluster configuration. If you want to understand the different configurations with multi-pathing, Aaron Dailey and Scott Tracy have a excellent white paper on using MPxIO on Solaris. Here is a picture of the cluster configuration with I/O multipathing.

 Try it out. Join the discussions at ha-clusters-discuss@opensolaris.org

IEEE Cluster 2007 Conference

Ira and I attended the IEEE Cluster 2007 conference last month.  This conference was held at Austin, Texas from September 17-20. This was a technical conference with hands-on tutorials, paper presentations, poster sessions and panel discussions related to cluster computing.  Cluster computing means both high-performance cluster computing, and high-availability clustering in this conference.

 A poster paper that I co-authored, "CHAF – An Object Oriented Framework for Configuring Applications in a Clustered Environment" was accepted for the conference. This framework was implemented in Sun Cluster 3.2. I gave a live demonstration of this implementation on my laptop with a lab cluster at the back end. My session and demo were well received, to the extent that a Sun customer referred to it in his email to Sun later.

Andy Bechtolsheim gave the opening keynote, at the conference, on "Scaling to Petaflops". He talked about challenges of peta-scale, the opportunities and Sun’s work in this area. He said that the primary challenge is memory speed scaling to meet the extra compute power delivered by the multiple cores.

 The two main topics that were the focus of the research papers and discussions, were multi-core and virtualization.  The panel discussion topic was multi-core computing, and the panelists were from IBM, Intel, UT Austin, nVIDIA and AMD. Prof. Steve Keckler used the phrase "termites, chainsaws and bulldozers" to refer to the different numbers of cores per chip, and it was clear by the end of the panel session that this phrase had caught on among the panelists and the audience!

 Ira and I also got an opportunity to visit Texas Advanced Computing Center at the University of Austin. This center is building a new supercomputer using Sun machines and the new Sun Magnum switch. It will be the largest supercomputer in the world when it becomes operational at the end of this year. It will have about 4000 nodes in the system. The whole site and the system was very impressive.

 The conference organizers had arranged a social outing with a barbecue dinner and live music (two live rock bands) at an Austin landmark restaurant, Stubbs. You might have seen their barbecue sauce in a local Safeway.

 One thought that stayed with me after the conference was from the closing keynote about "The Challenges and Rewards of Petascale Clusters", by Mark Seager from Lawrence Livermore National Labs. Mark Seager mentioned that different technologies that are in main stream today were present in the research community at least 20 years back. Some examples that he gave were garbage collection, virtual machines, and object oriented design. He stated that parallel programming was a technology that was not mainstream yet. I look forward to seeing Sun playing a big part in this.