My All-in-One Datacenter In A Box

For the past two years I had two lab servers at home. One was a VMware ESXi host, the other was a Solaris-based fileserver. Then I learned about an interesting way to combine the two into one when I came across the “All-in-One Server,” a concept written up by the author of Napp-It.

virtualized_by_vmware1I would describe it as this: Your install ESXi on a server like you normally would, and create a virtual guest. You will use this guest to install your favorite fileserver OS (I chose OpenIndiana). This virtual fileserver will be given direct access to the hard drives for you to configure software RAID like mdadm, ZFS, or any choice of your own. This is thanks to Intel VT-d or AMD-Vi which gives your virtual guest direct access to the PCI disk controller (host bus adapter) that your disks are attached to. Your virtual fileserver then presents this storage back to ESXi for you to install all of your other virtual guests on. And there you have it, your virtual machines will now reside on a virtual NAS (NFS) or SAN (iSCSI) with virtual 10Gbe connectivity to ESXi.

zfs_logoOf course there are some downsides. For one, it is absolutely not a viable solution to host a production system. I’d venture to say the concept is pretty radical and no vendor would ever support it. Naysayers may complain that combining hypervisor and storage is a single point of failure. Or that if the virtual fileserver crashes then all the guests crash with it.

However for my home lab it still seemed almost perfect. Replacing two boxes with one would save space, clean up a mess of cables, save on power consumption, and reduce heat and noise. I could now increase the speed from hypervisor to storage ten-fold from 1Gbe (physical) to 10Gbe (virtual). It allows me flexibility by virtualizing the fileserver which is something I never thought I would have. And the best part, no need for a big expensive hardware RAID controller anymore. We all know how picky ESXi is about that.

So, I decided to do it. Here is my build sheet. It is based almost completely off of Gea’s manual (from the napp-it project I mentioned earlier) . Also many thanks to [H] forums and it’s members for their wealth of information regarding ESXi, VT-d and PCI-passthrough, the IBM M1015 SAS HBA, ZFS, and pretty much anything else surrounding this project.

1CPUIntel Xeon E3-1230 3.2GHz LGA 1155 Quad-Core
1MotherboardSUPERMICRO MBD-X9SCM-F-O LGA 1155 Intel C204
16GBRAM4x 4GB (16GB) ECC Unbuffered DDR3 1333
1ChassisFractal Design Define Mini Black Micro ATX
1Power Supply
1RAID UnitOrico 229rsh 2 x 2.5" SATA with RAID
22.5" SATA7200RPM SATA 2.5"
1Hotswap BayStarTech 5.25" to 4x2.5" SAS Hot Swap Bays
2HBAIBM M1015 8-port flashed with IT firmware
4SAS disksSeagate Savvio 10K.3 146GB 10K RPM 2.5" SAS
6SATA disksWestern Digital WD Green WD20EARX 2TB SATA

Here is a breakdown of reasons why I chose these items:

  • CPU- Only server grade processors (Xeon) support VT-d
  • Motherboard- Single socket motherboards are awesome for a low power server. This one has IPMI for remote management over LAN too.
  • RAM- ZFS loves ECC RAM
  • Chassis- I wanted a real server chassis but couldn’t find one that was good enough for my home. I wanted something small, sleek, sexy, and quiet with only 120mm fans or bigger. I “power searched” NewEgg for days and couldn’t find anything. I chose Fractal Designs based on my previously mentioned requirements, plus it’s small MicroATX form factor, it’s capacity for 6x 3.5″ drives, and 2×5.25″ bays for accessories such as the RAID unit and 2.5″ drive bays.
  • RAID Unit- This  stores your bootable ESXi installation and the VMFS datastore for your virtualized fileserver. I had options:
    1. Just a single SATA disk. Very simple. but ten years ago I vowed never to trust a single hard drive (or SSD) for any important function
    2. USB flash drive. ESXi is designed for low writes but the fileserver OS that I chose was not. Therefore a USB flash drive would soon reach it’s write limit.
    3. Two disks mirrored with the mobo’s onboard software RAID. This won’t work. ESXi requires hardware RAID card.
    4. Hardware RAID card. Expensive. Hot. It’s just one more device prone to failure and BBU requires replacement every 2-3 years.
    5. Dual disk enclosure with built-in RAID. These are software based RAID controllers with port multiplication. They perform RAID functions upstream from the SATA port they are plugged in to. Rather than configure it via the BIOS or OS levels, it is configured via a DIP switch usually on the rear of the unit. It is completely invisible to the OS so ESXi will never know that there is a RAID array behind the “SATA device” that it is booting off.
  • SAS/SATA Host Bus Adapter- This device will be configured for PCI-passthrough and presented directly to your virtual fileserver. If you want to use a Solaris-based or Linux OS, then check out Zoraniq’s blog post on this subject. I chose the LSI SAS 2008 card that IBM re-branded as M1015. System pulls can be found on eBay for much less than the LSI branded card. LSI 1068e is also a good choice but has a 2TB drive limit.
  • SAS and SATA disks- These disks will be connected to the SAS/SATA HBA which makes them visible to your virtual fileserver to configure for RAID (ie Linux) or ZFS (ie Solaris). I chose ZFS with wanted two storage pools aka zpools:
    1. Nearline Storage (SATA). This is your somewhat average storage with low I/O requirements for infrequently accessed archive data (movies, music, photos, etc). I bought six of cheapest 2TB drives I could find and, after downloading Grant Pannell’s zpool binary replacement for 4k disks, created a RAID6-like zpool with this command:
      # zpool create tank raidz2 [disk1] [disk2] [disk3] [disk4] [disk5] [disk6] # Change [diskN] to match your system
      # zpool status tank
      zpool status tank
        pool: tank
       state: ONLINE
        scan: scrub repaired 0 in 16h42m with 0 errors on Sat Sep  8 16:34:09 2012
              NAME                       STATE     READ WRITE CKSUM
              tank                       ONLINE       0     0     0
                raidz2-0                 ONLINE       0     0     0
                  c2t50014EE2B08EB028d0  ONLINE       0     0     0
                  c2t50014EE2B08EBDCDd0  ONLINE       0     0     0
                  c2t50014EE25B9AED71d0  ONLINE       0     0     0
                  c2t50014EE25B38FD69d0  ONLINE       0     0     0
                  c2t50014EE25B390134d0  ONLINE       0     0     0
                  c2t50014EE05826FA2Cd0  ONLINE       0     0     0
    2. Fast Storage (SAS). I wanted storage for I/O intensive data, like the virtual guests, databases, etc. I bought four 10K SAS disks for a RAID10-like zpool that I created with this command:
      # zpool create sastank mirror [disk1] [disk3] mirror [disk2] [disk4] # Change [diskN] to match your system
      # zpool status sastank
      zpool status sastank
        pool: sastank
       state: ONLINE
        scan: scrub repaired 0 in 0h25m with 0 errors on Sat Jul 28 13:06:52 2012
              NAME                       STATE     READ WRITE CKSUM
              sastank                    ONLINE       0     0     0
                mirror-0                 ONLINE       0     0     0
                  c2t5000C50005BDC58Fd0  ONLINE       0     0     0
                  c2t5000C5000AC59CD3d0  ONLINE       0     0     0
                mirror-1                 ONLINE       0     0     0
                  c2t5000C50005BDD7FBd0  ONLINE       0     0     0
                  c2t5000C5003BA6376Bd0  ONLINE       0     0     0
  • 4×2.5″ SAS Hotswap Bay- This is simply an enclosure for the 2.5″ SAS drives mentioned above.

After it was built I referred to Gea’s manual for the process of installation and configuration of software, storage, and virtual networking. It was up and running in one weekend. OpenIndiana exports NFS storage to ESXi over a virtual 10GBe connection. It is fast and stable and I am very happy with it.

There are several more things I would like to add to this post in the future

  • Photos of the build, and an illustration or infographic of the “all-in-one” concept
  • Benchmark results to prove my claim that it is “fast and stable”
  • The concept of virtualizing networking and virtualizing software-based firewall’s like m0n0wall or pfSense
  • vSwitches and VLAN-tagging, and using them to create WAN, LAN, DMZ segments to secure and speed up your network
  • The possibility of using SSD for ZIL and L2ARC cache