NexentaStor storage appliance

In the last few weeks I’ve had to find an alternate solution for storing and serving VMware images to a big ESX cluster at work. I looked at a lot of open source tools and options, and some commercial ones, and have settled on the NexentaStor.

Our original solution is about 2 years old, and is a Dell NX1950 system running Windows Unified Storage Services 2003.  It had a single MD3000 head unit connected to the NX1950, and two MD1000’s chained off of the MD3000.  The MD3000 and one MD1000 each had 15 400GB 15K SAS drives in them for a total of 30 drives, and the remaining MD1000 had 15 1TB 7500RPM SATAu drives in it.

The first two trays were setup to serve out virtual disks via the iSCSI target provided in Windows Unified Storage Services.  The last tray of drives was for file shares and backups.

Suffice to say, there were many, many problems with performance, reliability, and interoperability.  We had at least 3 major outages attributed to the iSCSI target freezing or becoming overwhelmed.  Several other situations arose when drives failed in the array, and the horrible PERC+MD3000 combo tried to use a spare drive, which drug the usability of the array to a crawl.  In general, my experiences with Windows Unified Storage Services has proven to me that it’s definitely not worth the money at all, especially given that you can only get it (AFAIK) from OEMs.

The iSCSI piece is absolutely horrible in it – slower than anything else I benchmarked, and really quite buggy.  NFS is not even worth deploying as it’s at least twice as slow as iSCSI, and not even on the charts compared to traditional NFS.  Plus it has so many odd quirks about configuring it, that it makes it a big barrier to actually use with ESX.

We have several other solutions at work already – NetApp, EqualLogic, and various types of NFS/iSCSI hybrids.  I went about testing the following solutions before making my decision:

 

In the end, NexentaStor was the fastest, most reliable, and had all of the options I needed for my environment.  It’s based on OpenSolaris with a bit of Debian, but what I really was impressed with is ZFS and all of the benefits it provides.  By far, it’s NFS and iSCSI implementations were faster than any of the other vendors, even the purpose built EqualLogic array.

EqualLogic was the fastest next to Nexenta, but it’s really expensive, and only iSCSI.  I wanted the option to do NFS too.  NetApp is slightly slower than EqualLogic, and well, is by far highway robbery in terms of cost.  OpenSolaris is unstable as all get out, unless you go back far enough, and then there are missing pieces of functionality, such as iSCSI.  OpenFiler is really nice, and we use it in some other areas already, but it’s RAID technology is just as slow as Dell PERC or other crappy controllers.  It’s NFS and iSCSI implementations are fast, but not as fast as NetApp/EqualLogic and nowhere near NexentaStor.

Nexenta is still small compared to some of the other solutions, but it’s very well thought out.  I am looking forward to a lot more interesting things from their product line.  I’ll write more about the specifics later…

About Steve Mitchell

I live in Seattle, WA and love sailing, technology, & playing and composing music. I started playing the piano when I was 3, and ended up figuring out many other instruments along the way. I'm an avid sailor and have a 2000 Beneteau 311 named Grace, and sail it whenever and wherever I can.

  • Sumit Gupta

    Within NexentaStor iSCSI, which iSCSI implementation did you use ?

  • Hi,Thanks for sharing the great info.Just as a note I’m running OpenSolaris 08/11 and it has not faltered since it was installed in November 2008. So I’m wondering why you say it’s not stable?VMware introduced a bug in 3.5.0 143128 that takes down the iscsi connection but thats not Opensolaris stability.

  • erast

    I probably should mention that upcoming NCP2.0 RC1 (hence NexentaStor v1.1.7 too) includes interoperability fixes for VMware ESX / COMSTAR iSCSI target and MacOSX / GlobalSAN initiator. It also includes bunch of CIFS fixes – in hope that AD environments works better now

  • steve

    Erast,Thanks for the note. I am running NexentaStor 1.1.6 which was just released this last week, and have been using the COMSTAR and non-COMSTAR iSCSI stuff in both 1.1.6 and 1.1.5. COMSTAR was broken in 1.1.5 with ESX, but works in 1.1.6. While it’s faster than many of the other implementations, it’s still not as fast as NFS. For sure the COMSTAR implementation is faster by far than the default or old iSCSI one. Its good to know there are upcoming fixes coming as well…

  • Werner

    On what hardware (server + disk storage) did you deploy NexentaStor? I’m considering using Nexenta as well, and was thinking of using Perc6/E and a few MD1120 units from Dell for storage front-ended by a Dell 2900 III (2 x Quad) + 16 GB RAM.

  • steve

    @WernerWerner,I plan on posting my full hardware setup later this week, but I am using almost the same as what you’re considering.I have a Dell PE 2900 III with dual quad core processors, each at 2.5 Ghz. 8GB of RAM, and dual 73GB 15K RPM drives mirrored for the Nexenta install.I have two PERC/5i controllers, each with dual ports, each connected to a fully populated Dell MD1000 tray. Each tray has 15x 400GB 10K RPM drives in them, and I have them in split mode so that one port on the controller controls 7 drives, the other 8.I don’t see the need for a huge amount of ram – I think 8GB is more than enough. The PERC controllers provide more than enough cache, although I’m looking into a SSD solution for later this year.

  • Allan

    Thanks for your opinion, especially valuable and rare from someone with access to all that high dollar stuff. I am looking for shared vm storage and backup space and this is most helpful. I like to try and give back where I can, so here’s a tip I came across today for increasing iSCSI performance tenfold in windows registry…http://blogs.sun.com/constantin/entry/x4500_solaris_zfs_iscsi_perfect

%d bloggers like this: