Trying To Find The Ideal File System

[Chip]

I've spent half the day trying to find a file system or software-based data management system to meet my needs, and it does not appear to exist. Here's my situation:

I have just under 5TB of storage space spread between 10 physical drives between 200GB and 1.5TB in size. 6 drives are SATA internals, 3 are USB externals and one is an external firewire drive. The drives are all between 50% and 80% full, and I have no redundancy at this time.

What I would like to have is a RAID5-type setup with with all drives appearing as one large disk, with single-failure parity redundancy.

The obvious problem with this is that standard RAID types only work with identically sized drives. As stated, my drives are all of varying sizes.

Windows Home Server is very nearly what I need, but not quite. It will create a single large virtual disk from any number of physical disks of any size and allows you to add drives to the pool on the fly, but it only supports mirrored redundancy. This means that I'd only be able to use 1/2 of the actual storage capacity, which I couldn't do without purchasing more drives right off the bat. With this many drives, I'd much rather use parity redundancy, which would only cost me the size of my largest drive (1.5TB) total.

The only perfect solution I've found is the Data Robotics DROBO. The problem with the DROBO is that it is a really expensive piece of hardware and only works with internal drives. I know the functionality could be replicated purely with software, but I can't find anybody who has actually done it.

Is there something I've missed? Some obscure file system or utility that will do what I want? I've looked into the various Linux file systems, but none of them seem to be able to meet all my requirements. Surely somebody besides me and the guys at Data Robotics has thought of this before.

 

[lingenfr]

Chip,

I think that RAID-5 on that configuration will be extremely difficult and probably overkill. I don't think it fills the bill for you, but I have been looking at LVMS for my file server. It can make many drives appear as one. Definitely a bit easier to manage and back up, etc. I will take a look around as I am interested as well.

 

[todd]

This sounds like a job for ZFS! Has that made it into the real (i.e., anything besides Open Solaris) world yet?

 

[Chip]

I'm not looking at RAID5 for a performance increase, I'm looking at it (or something like it without the limitations) for the minimal-overhead-redundancy. Logical volumes won't help here, as they don't provide any redundancy at all.

I looked into ZFS, which is very interesting, but it still has the same-size-drive limitation for RAID arrays.

 

[Chip]

I've given up on "ideal" and settled for "as close as it gets". I installed Windows Home Server on my spare box earlier today and have started migrating things over. I'll swing by Microcenter tomorrow and pick up a couple more 1.5TB drives to make up for Microsoft's inability to provide efficient redundancy. Migrating all my data over is going to take most of the week, but at least everything will be nice and organized (not to mention safer) once it is done.

 

[gp32rich]

I've given up on "ideal" and settled for "as close as it gets". I installed Windows Home Server on my spare box earlier today and have started migrating things over. I'll swing by Microcenter tomorrow and pick up a couple more 1.5TB drives to make up for Microsoft's inability to provide efficient redundancy. Migrating all my data over is going to take most of the week, but at least everything will be nice and organized (not to mention safer) once it is done.
If you were going to do that anyway, why not go for RAID?

 

[WizardStan]

I don't know why I can't find any information to help you. I've never thought about it until now, but a RAID with different sized drives should be possible, in theory. I can think of two ways to do it off the top of my head. You lose a bit of efficiency, but it's a small price to pay. And yet there must be a reason it can't work like that, otherwise someone would have already done it, right? This is really going to start bothering me in a little while.

 

[Chip]

If you were going to do that anyway, why not go for RAID?
Because RAIDs do not allow the use of varying sized disks, nor are they expandable.

 

[Javacat]

I've got an idea! Although it could get a bit crazy and I'm not sure what speeds you'd end up with + Not sure if your RAID hardware would let you do it either

You could merge a few of the drives together using a few RAID 0 setups, so that you've got clusters that are all similar sizes (eg 3 500GB drives totalling 1.5TB, 4 300GB totalling 1.2TB, then the 1.5TB drive). You could then combine them together using RAID 5, giving 2.4TB of storage and redundency. The downside would be that if one of the smaller drives failed in their "cluster" then the group of drives would be useless, but you then have the excuse to go out and get a new 1.5TB drive to replace them. If it was one of the 300GB drives that had died and was replaced you'd then be able to expand the total RAID storage to 3TB.

BTW, me likes RAID 10, although it can be a bit expensive for home setups.......

 

[Ravnos]

This sounds like a job for ZFS! Has that made it into the real (i.e., anything besides Open Solaris) world yet?

It's supported in FreeBSD and usable as the root fs, there's a FUSE implementation available for Linux (due to license discrepancies), and I believe the next version of Mac OS X should have it as an officially supported file system. Basically, everything but Windows, though there may be a file system driver available for it too, I haven't checked.

 

[Chip]

You could merge a few of the drives together using a few RAID 0 setups, so that you've got clusters that are all similar sizes (eg 3 500GB drives totalling 1.5TB, 4 300GB totalling 1.2TB, then the 1.5TB drive). You could then combine them together using RAID 5, giving 2.4TB of storage and redundency. The downside would be that if one of the smaller drives failed in their "cluster" then the group of drives would be useless, but you then have the excuse to go out and get a new 1.5TB drive to replace them. If it was one of the 300GB drives that had died and was replaced you'd then be able to expand the total RAID storage to 3TB.
I considered a multi-level RAID for about 10 seconds, but it's just too complicated and even more prone to failure than other varieties. Also, it doesn't allow for expandability. WHS really is the only thing out there that offers what I want, which is a shame because it doesn't do it all that well. The way it handles drive balancing is horribly inefficient, and copying files to the data pool is ass-achingly slow. I'm talking 12 hours per TB slow, and that's from a drive connected directly to the server box, with mirroring and media serving turned off. With those turned on, it's half as fast.

I've been digging through MS forums and there's not anything wrong with my setup, WHS is just naturally dog slow. Their implementation of an expandable data pool is just plain crummy, but my choices are between crummy and nothing, so I'll take crummy.

 

[lizard808uk]

Btrfs?? I only suggest this as I found it as an install option in Fedora as: "i-can't-believe-it's-not-btr" \^_\^

 

[Chip]

Btrfs?? I only suggest this as I found it as an install option in Fedora as: "i-can't-believe-it's-not-btr" \^_\^
Their wiki is a bit sparse on exactly what BTRFS can or cannot do, but they did have this banner in bold letters on the front page: QUOTE
Btrfs is under heavy development, and is not suitable for any uses other than benchmarking and review.
I think I'll stick to WHS for now.

 

[Phawx]

@Chip

Be careful with WHS and Large archives. I have four 1.5 TB HDD's in my WHS and it's 84% full. I tried putting something on the mirrored portion that was a solid 30GB file and WHS complained that it couldn't duplicate the file.

I don't know why it couldn't duplicate it. I had 764GB left. So that 30GB file should have taken 60GB of that 764GB pool. For whatever reason large contiguous files when you are close to full space will *not* mirror.

Just a heads up.

 

[Chip]

Be careful with WHS and Large archives. I have four 1.5 TB HDD's in my WHS and it's 84% full. I tried putting something on the mirrored portion that was a solid 30GB file and WHS complained that it couldn't duplicate the file.
Luckily I don't have any single files larger than about 2GB, and they've been copying fine so far. I am concerned about other hidden limitations that it may have though. Nobody seems to know exactly how their "folder level mirroring" works, and I am concerned it could break if I have folders larger than any one physical disk. I have mirroring turned off to speed up copying (relatively), and I am very concerned that it will wait until I turn it on to decide that there is a problem. Another day or two of transferring and I'll find out.

 

[Javacat]

Old point that you brought up a couple of times but many different levels of RAID do allow for expandability.

 

[Chip]

Old point that you brought up a couple of times but many different levels of RAID do allow for expandability.
RAID5 and variants of it allow you to expand the array, but only with identically sized drives. It's still a no-go.