But let's face it - some of the best things are quick and dirty.
After reading this article on Slashdot: Building a Massive Single Volume Storage Solution?, I thought about something I'd always meant to try out, and as I had a few hours to kill, I did it. (I'm not saying it's the answer to the question in the article - just that it was that that inspired me.)
(I actually think that the guy was wanting a way of mounting a single filesystem across a network of nodes, but hey...)
modprobe -v nbd <--- This will give you the /dev/nbd* devices you need. (TODO: Do you need this on the server?)
touch /tmp/disk1
touch /tmp/disk2
touch /tmp/disk3
nbd-server 5001 /tmp/disk1 50M
nbd-server 5002 /tmp/disk2 50M
nbd-server 5003 /tmp/disk3 50M
(Adjust ports, files, and size limits as you like).
Open up your firewall to allow your other machine to connect on those tcp ports.
modprobe -v nbd
nbd-client your.server.address 5001 /dev/nbd1
nbd-client your.server.address 5002 /dev/nbd2
nbd-client your.server.address 5003 /dev/nbd3
Make sure you have md ( software RAID support ) in your Linux kernel.
modprobe -v raid5
If you all is good, cat /proc/mdstat should say:
Personalities : [raid5]
unused devices: <none>
"C"reate your new array with this command:
mdadm -C /dev/md0 -c 128 -l5 --raid-devices 3 /dev/nbd1 /dev/nbd2 /dev/nbd3
mdadm -D /dev/md0 will "D"isplay all about your new RAID array. It should look something like this:
/dev/md0:
Version : 00.90.01
Creation Time : Wed Oct 26 15:45:33 2005
Raid Level : raid5
Array Size : 102144 (99.77 MiB 104.60 MB)
Device Size : 51072 (49.88 MiB 52.30 MB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Oct 26 15:45:33 2005
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Rebuild Status : 58% complete
UUID : 8b1bf4f2:59fbcb5c:24476e9d:02827b3c
Events : 0.1953
Number Major Minor RaidDevice State
0 43 1 0 active sync /dev/nbd1
1 43 2 1 active sync /dev/nbd2
2 0 0 - removed
3 43 3 2 spare rebuilding /dev/nbd3
Although it looks like 2 RAID members, and a spare, this is apparently the fastest way for the new array to build.
Add these lines to your /etc/mdadm.conf file so that you can "A"ctivate it later with mdadm -A /dev/md0.
DEVICE /dev/nbd[0-9]
ARRAY /dev/md0 devices=/dev/nbd1,/dev/nbd2,/dev/nbd3
mkfs /dev/md0
(This too can take a while. You can use any filesystem you like - Reiser, XFS, ext3 (blech) )
mount /dev/md0 /mnt/raid -o sync
The sync option makes sure changes to /dev/md0 are written at the time, not when you unmount. It seems faster without the sync option, but it's just delaying writing until later.
Voila. You now have a RAID 5 setup across your network.
Obviously, in this example, you have all your raid devices on the same server. You can move them to seperate machines quite easily.
Your network will be limited to the slowest network link. This sucks, but is just the way it is. You get a fully internet distributed filesystem, but you pay the cost in speed. Or you have them all on your local LAN, and pay the price in redundancy.
Fast Ethernet gave me about 4MB/s. dd if=/dev/zero of=/mnt/raid/test bs=1M count=50 took about 13.1 seconds with a chunk-size of 128k. Gig Ethernet would give a lot more (I assume).
Read more about Linux Software RAID.
For more auto-failover, add hot-spares.
Also, you can combine raid levels. Perhaps two RAID5 arrays, mirrored?
Learn about what happens in Linux software RAID if a "disk" dies. Hint: mdadm -r /dev/md0 /dev/nbd1.
When you have your nbd1 back, mdadm -a /dev/md0 /dev/nbd1, and let it rebuild.
Or you could run dm-crypt over the resultant md0 filesystem to provide encryption.
Or for that matter, LVM.
This isn't (as far as I know) a distributed client model. I.e. it's not for mounting the resulting filesystem on multiple machines. It's about having many remote "disks" so that your data is spread all over the place.
Thanks to Neil, who didn't realise that I was using his machine to test it out. :)
Todo El feedbacko aqui, por favor.
Notes for me about lvm: not finished
pvcreate -v /dev/md0
pvscan (Note size)
vgcreate mdvg /dev/md0
vgscan (Look for "Found volume group "mdvg" using metadata type lvm2")
lvcreate --size $sizefromearlier --name mdlv mdvg
mkfs /dev/mdvg/mdlv
mount /dev/mdvg/mdlv /mnt/raid -o sync
umount /mnt/raid/
lvremove /dev/mdvg/mdlv