I started this build because my external 1TB drive was beginning to fail. It's a LaCie "BIG DISK" 1TB. Reports I've seen around the web indicate most everything is good about these drives EXCEPT the power supplies are a bit underpowered, which can lead to data loss. Based on my experience with the drive and the power supply, I highly suspect it is my power supply that has failed. I'm not certain, but I'm trying to get a warranty replacement now (as it's only 9 months old).
Fortunately, it was not storing anything unrecoverable. It was the primary backup for my iMac, and I have secondary off-site backup. I had a number of videos on that drive, but I can resurrect those from other sources. What the drive failure has made me realize, though, is that I wanted something less prone to failure, since I had planned to store greater amounts of data on that external drive.
Looking about, I saw a lot of home users buying or building NAS systems. I initially looked at the Drobo and thought it would easy and simple. I also saw reports questioning reliability that led me into a cursory study of various RAID configurations, the problems, software vs. hardware, etc. (Note that those Drobo reports were old and likely a small minority of their customers, and I understand that the Drobo is improved since its first iteration.)
That's when I found ZFS and RAIDZ. I can't say that ZFS/RAIDZ is the ultimate end of filesystems and RAID, but it was definitely "cool", looked reasonable to manage, and resolved some of the questions I had when looking for an external storage solution. I also ran into FreeNAS; while OpenSolaris might be a better choice, I didn't have a whole lot of time to tinker, and I suspected I would have an easier time getting compatible hardware for FreeBSD/FreeNAS than I would for OpenSolaris. (Quite possible a wrong assumption, I admit.) Also, being a Mac OS X user and command-line semi-junkie, I felt a little more comfortable going with FreeBSD. (Also quite possibly a bad assumption, but hey... gotta start somewhere and make guesses at some point.)
So, I decided to try building my NAS, rather than going with the Drobo or some other preconfigured solution which couldn't do ZFS/RAIDZ anyway. My goals for this project were:
I've built systems before, but those were generic Windows boxes without any special needs. Case, motherboard, memory, drives, video card, etc, etc, etc... Grab parts, slap 'em together, and you've got a box. I've never fine-tuned my builds for truly optimal performance, but generally the performance of the components I've put into my boxes have been beyond sufficient for my needs and never required fine-tuning.
For the NAS, though, I needed to put a little more thought into it. During the build, I made choices that forced me to compromise my goals. In the end, I have a NAS that fulfills some of the goals I set; some of the compromised goals I can attend to over time. I've also learned a number of things for future builds, and I am sharing my experiences with others to help them make better decisions.
This was the most annoying part of designing the NAS. I didn't purchase any other parts until I picked a case. It took so long, and frustrated me so much, because the kind of case I was looking for is quite rare, apparently. I had in mind to buy a 5-in-3 hot-swap drive cage (i.e. fits five 3.5" HDDs into the space for three 5.25" HDDs), but the cases that had at least three 5.25" external bays were usually towers. The smaller cases usually only had two such bays and one 3.5" external. Internal bays weren't useful here, since I wanted to hot-swap.
The cases that I found that would work tended to be custom and cost twice as much as I would have liked. The more mainstream cases that cost a decent amount really weren't configured properly. Like I said... annoying. In the end, I finally decided upon the Apevia X-QPACK2-GD and the iStarUSA BPU-350SATA hot-swap drive cage.
The combined price of these two items came to $210, but that was giving me five externally accessible and hot-swappable drives. And since the Apevia case included a 500W power supply, that was one less thing I needed to worry about.
Of course, you can see that the Apevia case has only two external 5.25" HDD bays. Studying the design (as best I could via the web), I noticed underneath those two bays was a single external 3.5" HDD bay, a digital temperature readout, and a handle. None of those were needed; I figured I could pull them out, and that would effectively give me the space for the drive cage.
Well, I would have gotten away with it if not for those meddling kids! I mean, meddling internals! Basically, I had hoped that I could pull out a Dremel and mod the kit a bit, enough to get that cage in. And for the externals (i.e. removing the handle, 3.5" drive bay, and temp readout), it would have been fairly simple and successful. However, to fit that cage inside would require cutting a cross beam so much as to sever it. That cross beam supports the power supply, provides a bit of cable routing, and probably helps to strengthen the case.
I studied the case and cage configuration for a long while, but in the end, decided that it wouldn't work. Actually, for a talented case-modder, it could probably be made to work. For myself, however, I had to realize that I didn't know how to approach such a mod, didn't have the tools, and couldn't spend the time. Time to change plans.
I knew I could get a similar drive cage: 3-in-2 instead of 5-in-3. But now I had to make a decision. If I wanted them hot-swappable, that would limit me to three drives in the RAIDZ. If I wanted five drives, there was no point in getting a hot-swap cage, since the other two drives would be internal anyway. (The other option was to find a different case, but I was in no mood to go case-hunting again.)
My needs are somewhat light... I'm mostly doing backup and media storage; these drives are not being banged on constantly such as those that would be in a server room. So, for me, hot-swap was cut from the plans. What that means is that when it comes time to replace a drive, it will require I shutdown the NAS to replace the drive. Since this is only for my needs (i.e. I'm not serving customers), that's an acceptable compromise. So I returned the iStarUSA hot-swap cage and instead got iStarUSA's iStorm7 3-in-2 cage. It has a fan to push air past the drives to keep them cool, and slid right into the Apevia case without mods. It was also much cheaper.
Now that I had the case, power supply (bundled with case), and drive cage, I could start ordering other components. For drives, I was going to have five drives in RAIDZ. I wanted about 3TB storage, which meant using 750GB drives, which was a good spot to be in, I think, for price and stability. I ordered five Western Digital Caviar Black 750GB drives. I actually picked up three from newegg and two from Amazon (same price); I thought ordering a few each from different retailers would get drives from different manufacturing batches (and, theoretically, would be less likely to have simultaneous faiulres). Who knows... They all seem to be working just fine; hopefully they won't all die at the same time.
I was initially going to get "green" drives to reduce power consumption, but I had seen some reports that suggested "green" drives were not really suitable for RAID. I ordered these drives, and then realized that I picked up 7200RPM drives (instead of 5400RPM); I worried that the higher-speed drives might run too hot, but they've been maintaining a decent 35°C.
Initially, thinking of power, I wanted to get a small, simple motherboard with a few things built-in: simple video (just to configure; it will run headless), dual gigabit ethernet, 64-bit and up to 2GB RAM (both good for ZFS), and six SATA ports. Well, there really aren't any like this. I found one or two rather expensive ones, and certainly these things are easier to find on larger motherboards, but I needed a Micro-ATX or Mini-ITX, and had no luck there.
In the end, I found some good comments about the Intel D945GCLF2 Mini-ITX. It has integrated video, one gigabit ethernet port, and two SATA connectors. It comes with a dual-core Atom CPU @ 1.60GHz, so it has more than sufficient power to run a NAS. It accepts DDR2-667 RAM, so I bought a single 2GB stick from Kingston. ZFS can run on a 32-bit CPU with low memory, but apparently it excels with 64-bit and as much RAM as available; 2GB should be plenty.
Dual gigabit ethernet would have been cool, but probably unnecessary.
Since the motherboard had only two SATA connections, I needed four more. (Well, three more, but they come in powers of two.) I started off with a Highpoint RocketRaid 1740 card, which has four SATA300 connections and can provide various kinds of RAID; however, as I was aiming for a ZFS/RAIDZ build, I didn't want any RAID; I just wanted the drives passed through to the operating system.
However, this does not seem to be possible with the 1740. I was able to get around it by creating three JBOD arrays with one disk in each array. The problem here is that, to FreeNAS, these don't look exactly like drives. They work, but you can't get S.M.A.R.T. information out of them. Also, if I ever decided to move this array to another case, I would be forced to continue using the same RocketRaid 1740 card, or lose all the data (or be forced to backup and restore). If the controller card passed the drives through, the drives could be moved elsewhere without any downtime (by simply telling the new setup how they are already configured as a RAIDZ set.)
I decided to look around a bit for a different controller card, and eventually found good reviews and comments for the Promise TX4 card: four SATA300 ports and no RAID; just drives. I ordered it, swapped cards, and rebuilt the system. (I had put little data on the NAS in the past week, so there was no great loss in just starting over.) The drives connected to the Promise card even showed up in the BIOS setup screens, and it was zero work to see them and use them from FreeNas.
I expected some of my purchases would come with SATA cables, but I also expected that they would be more suitable for a larger case. After some quick measurements, I figured I could get by with cables around 10-12 inches in length. I looked around a bit; cables are often marked up high, especially on large name shops like Amazon. In the end, I found a good assortment of cables at PCH Cables Inc. In particular, I bought six 10-inch straight-to-straight black SATA cables.
I also realized that my power supply had only two SATA power connectors. Searching around, I found a number of convertors from 4-pin molex to SATA power; PCH had good prices on those as well, so I bought two of them. These cables and power adapters are working without problem.
When I had all the components, I started putting things together, first with the motherboard, memory, and a spare IDE hard drive I had intended to use as a boot drive. Well, it turns out that drive was dead; I'm not sure why I still had it. So I needed something else to boot from.
After a lot of trial, error, reading web pages, and retrial, I finally got the motherboard to boot off a 2GB USB flash drive. It wasn't the fastest boot around, but the system isn't intended to be rebooted regularly. It was a decent solution for boot.
However, I wasn't completely happy. It sticks out the back (or the front, whatever), which means you have to accommodate that and make sure not to snap it off. It was also very large for FreeNAS; the software doesn't need much more than 32MB to boot, and here I was "wasting" a 2GB flash drive on it. Searching for alteratives, I found a 128MB flash drive that plugs directly into the IDE connector on the motherboard.
Awesome solution! Makes use of the empty IDE connector and cheap! It wasn't the first nor only solution I found.
Also cool are drives that plug directly onto unused USB headers on the motherboard. I was initially going to buy one
of those, but I found I could get the IDE above from Amazon, and not have to register at yet another Internet
retailer. If you are considering a similar build, you can try something like the 128MB IDE flash drive above;
or, if you'd rather use a USB header and/or your IDE connector is occupied, search the
web for "USB vertical flash storage", and you will find flash drives that plug directly into the motherboard. A
couple of examples:
The Apevia case motherboard tray slides out completely, so you can do memory, IDE, and PCI installation with little hassle. The 3-in-2 drive cage is easy to load drives into, along with the case's removable side-bracket for two more drives.
Cabling became the most difficult part for a few reasons. First, once everything is loaded in, the area above the motherboard can become very tight. The PCI card blocks access from one side, and the power supply and drives partially block access from the top. My hands are not "dainty", so some connections are a struggle, but in the end, I got it all together.
Second, the drive cage fits in nicely and includes a fan (w/ washable filter) to keep the drives cool. Having the fan, however, pushes the drives back such that they are a hair over 1 inch from the power supply. The bottom drive in the cage isn't affected so much, nor are the power connectors going to the drives. However, the straight SATA data cables going into the top two drives barely fit.
This is especially true given the third problem: that's where the power supply's cables exit into the cage! And this is the power supply that came with the Apevia case: it's probably a decent power supply, but certainly not modular, so I can't remove cables I'm not using. I had to really jam them sideways in order to get the SATA data cables into place, and I'm still not comfortable with that region.
Moving the power cables to the side like that means that, in some cases, I had to route them around below the power supply and drive cage. In the end, it wasn't so terrible, since they have plenty of length for the case.
To remove some of the clutter, I disconnected almost all the cables from the front panel:
The cable for the Power LED had a connector that didn't match the motherboard, so I used the Drive Activity LED cable as the Power LED (since drive activity would be pretty much meaningless here anyway). But later, I pulled both LEDs, because I wanted to reduce illumination (see goal #5). I disconnected the temperature display for the same reason.
That left only the USB ports and the power switch on the front panel connected. I had even considered disconnecting the USB ports (since there are four on the back sitting empty), but having removed a bunch of cable clutter, I figured leaving that one in place was okay for the minor convenience of putting in a USB flash drive (if I ever need that).
Update (2009 July 24): Actually, the idea of playing music directly from the NAS isn't a bad idea. I have already considered adding a print daemon to manage my printers; why not also plug in speakers and let it play music? At this point, the primary reason is because I am using iTunes and an iPhone and am fairly comfortable with that situation. Still, it's another option to keep in mind.
It certainly is external. And in theory, it is stable, and it has redundancy enough to survive one drive failure in addition to all the error prevention and correction that ZFS provides. Actual stability can only be determined with time.
Five 750GB SATA II drives in RAIDZ for a hair under 3TB storage: 1TB set aside from Time Machine backups, with 2TB for media. The sixth "drive" is a small flash drive, all that's needed for FreeNAS. I had to compromise, and gave up hot-swap.
It is reasonably small: approx 9"H x 11"W x 15"D (1,485 cubic inches). Compare this to the 4-bay Drobo: 6" x 6.3" x 10.7" (404 cubic inches). My build certainly isn't as tight and compact as a Drobo: you can fit three-and-a-half Drobos into my case! Of course, I have an extra drive and generic components compared to their custom build. But what I have isn't a tower case, is less than one cubic foot, and I didn't have to get custom.
I purchased a Kill-A-Watt and did a couple initial measurements. At 89.7 hours, it has measured 7.74kWh, which comes out to about 86.3W. (Does this seem right?) Somehow, I'm not sure whether that's correct or not, but then again, it certainly spends more time sitting there looking pretty than doing any intense work. This needs more study, but under 90W isn't too bad... is it?
I disconnected the front panel lights, since they weren't really needed. There is an exhaust fan in the back with a bright blue LED; the case was designed to be a gamer's LAN box, so blue LED fans are cool, but it's brighter than I'd like, so I'll get to replacing it at some point.
It's not quiet; it's not terribly annoying either, but you certainly notice the silence when it is powered down. The drives, I think, are pretty good, but there are five fans in there making noise. (I don't know their actual dimensions at the moment; I will measure those later.)
I made sure all fans were running, because I wanted to ensure the drives didn't overheat in such tight quarters. That doesn't seem to be a problem, so I may be able to shut down one or two fans. The noise isn't terrible, but is certainly noticeable. When I shut it down recently to reconfigure a few cables, the silence was nice.
See down below for a parts list/costs. In the end, without drives, my price was $324. I have seen Drobos on sale for $370. So yes, I beat a Drobo in some ways.
UPDATED 2009 July 24: In theory, it is fast enough. FTP transfer to the NAS is currently around 300Mb/s; however, I need to examine CIFS and/or AFP settings to figure out why they are providing a pathetic 10-15Mb/s. I need to do some proper testing and tuning.
UPDATED 2009 Aug 01: The pathetic 10Mb/s rate is gone; the usual is 200-300Mb/s. Still would like to see it higher.
This was definitely an experiment and definitely fun!
Now that it's up and working, I'm already thinking of ways to make it better.
The problem with the RocketRaid is that I could not pass-through the drives: they had to be in a RAID, even if it was just each drive in their own JBOD. That's what I did at first, but it's not optimal. Since I'm using ZFS/RAIDZ anyway, I want the bare drives without any JBOD overhead (since I could no longer move the drives elsewhere). Also, being in the JBODs on the RocketRaid prevents S.M.A.R.T. information from reaching FreeNAS. I could install Highpoint's utility for accessing this information, but I'd rather just do it right in the first place.
Completed on July 23: I did have to build the RAIDZ again, which meant losing whatever was already stored on the drives while using the RocketRaid, but I intentionally put very little on them, so only a little loss of time.
UPDATED 2009 Aug 01: Setting this up was only slightly more difficult that setting up the USB flash drive. With the USB drive, I could install the embedded version of FreeNAS onto it from a command shell running on Mac OS X, using the dd program. I was initially going to borrow an IDE DVD-ROM drive and plug both it and the IDE flash drive into the NAS's motherboard... except the IDE flash drive has the wrong connector for a cable! (It's designed to plug directly into the motherboard.) And the only other PC I had available had only one IDE connection as well. So no booting from CD.
Of course, I soon realized my stupidity and copied the FreeNAS embedded image over to the NAS, booted from the USB drive, and proceeded to install the image to the internal IDE flash drive.
One other thing I wanted to try was to install OpenSolaris to the IDE flash drive and import the existing ZFS vpool; from there, I could run a few performance tests to compare against my FreeNAS configuration. However, the default OpenSolaris install needs approximately 3.5-4.0 GB of space on the boot drive, but my boot drive is a mere 128MB! Plenty for FreeNAS, not so plenty for OpenSolaris. Granted, the default OpenSolaris install includes a graphical user interface (the same as Ubuntu, I believe) and probably a lot of software and drivers that a NAS doesn't need. In the future, I may go back and create a stripped down version of OpenSolaris — or just boot from USB — to do performance comparisons between OpenSolaris and FreeNAS.
Since I can't hot-swap drives, if one fails, I need to identify it, shut the machine down, remove, replace, reboot and rebuild. If I had a six drive installed, but not in use, I could get to the rebuild stage immediately by dropping the failing drive from the RAIDZ and adding in the ready spare. I would still need to reboot and replace the failed drive at some point, but I could do it after I ensure the data was safe.
UPDATED 2009 Aug 01: It would appear that ZFS supports spare drives. Tell the pool which drive is a spare, and it'll automatically replace a failing drive. Cool... This makes this idea more likely to happen, although I need to see if I can add a spare to an existing RAIDZ pool, or if it has to be created with the spare. (If the latter, I could still put the drive in the machine ahead of time: I'd just have to manually run the commands to replace the failing drive.)
Yeah, well, I keep checking around. I really should be spending time on more important things, but the case issue just bugs the hell out of me.
Because the drives back up to within an inch of the power supply, and because the power supply's cables exit into the case right in the same area, it's very crowded right at that point, and puts some pressure and tension on everything there. Getting 2-3 right-angle SATA data cables would help with this area.
Contrary to most modders, I don't want this to be noticeable. It should be easy to replace that fan with a basic black one. If that works, I might reconnect the power LED on the front panel as the sole light / power inidicator.
It's not painful, but silence is bliss, and this project isn't silent. I need to test disconnecting some fans to see what happens to drive temperatures; I probably have more fans than needed, so hopefully I can get away with disconnecting one or two of them. If not, then I need to find some quieter fans, if possible.
Initial calculations (using a Kill-A-Watt) seem to indicate the NAS using under 90W; that's probably mostly "idle", since I've not done any lengthy, intensive operations with the system. More measurements are needed to see how high this will go, but 500W is probably overkill. If I could replace the power supply with a smaller, lower wattage supply, that would help with airflow (and reduce noise if I can disconnect a fan or two), less cable congestion, etc. A modular supply would also be cool, by having even fewer cables (and more SATA power connectors, so I could remove the molex-to-sata adapters). I don't know if modular power supplies go below 500W, but it's something to look into.
The decorative front panel piece of the Apevia X-QPACK2-GD case is metal, and I've seen a few reports indicating that touching it can reset the system, because it isn't grounded. Well, I believe I just experienced that first-hand: the NAS reset itself while I was trying to see in through the case window. Grounding that panel shouldn't be difficult.
Need to research how to measure the various parts of the system, determine bottlenecks, and make them faster. Even though I am not requiring the ultimate performance, I do want to see how it stacks up so far and remove any huge bottlenecks.
UPDATED 2009 July 24: This may be necessary sooner than I thought. My first time around, I was getting decent write performance over CIFS, but when I started a backup last night, I was getting a pathetic 10Mbps. (That's mega-bits.) AFP was only slightly better, but FTP was an amazing 300Mbps. My primary backup method has been to turn on Apple's Time Machine and forget about it: Time Machine works and has saved my skin in the past (when I accidentally trashed the partition map playing with things I should not have played with.) But using Time Machine will require I get CIFS or AFP to write faster; otherwise it could take days to backup the system!
UPDATED 2009 Aug 01: I'm not sure why I had that one evening of 10Mbps, but I'm getting 200-300Mbps regularly for the past week. So it's working well. Could be faster, so I still will do some performance testing. ZFS compression is an interesting idea to reduce I/O by using the CPU (mostly unused right now), but I'm not sure I'll get much out of compressing video data, which is highly compressed as it is. Fortunately, ZFS makes creating datasets/mounts that can be compressed independent of the rest of the filesystem, so testing this will be easy.
I made a newegg.com shared wish list that contains almost all of the components (except the IDE Flash Drive and the SATA cables). With the exception of those items and two of the five drives, I did purchase everything from newegg.com.
The cost I am going to list here is the final cost of the NAS itself; I'm not including the cost of the RMA or the loss on resell of the Highpoint RocketRaid card. This is only the NAS in its final configuration (as of 2009 Jul 24). Rounded to nearest dollars, shipping not included:
Apevia X-QPACK2-GD case: $ 89 (assuming $10 mail-in rebate clears) iStar iStorm7 Heat Terminator bracket: $ 30 Intel D945GCLF2 mini-itx board w/CPU: $ 81 Kingston 2GB DDR2 667 RAM: $ 25 Promise SATA300 TX4 controller card: $ 70 Transcend 128MB IDE flash module: $ 9 SATA cables & power adapters: $ 20 (approx) TOTAL for NAS without drives: $324 5x WD 750GB hard drives ($80 ea): $400 (RAIDZ: provides redundancy and 3TB usable storage) TOTAL for NAS with drives: $724