Topher’s Scribbles

I’ve been trying to set up a backup file server using FreeBSD 6.2-RELEASE amd64, Areca RAID, and using a partition that is >2TB. My first install crashed the server under heavy writing. I was rsyncing 3 servers simultaneously. Checking the logs, there were some “kernel: g_vfs_done():…Error: 5″ errors before the crash. I searched google and found out that it was a common error on Areca cards using FreeBSD. So I found a solution on the Areca website, http://areca.tw.com, and it said to edit the driver source code. I recompiled the kernel and started over again. This time, I am rsyncing 4 servers simultaneously and updating ALL the ports via cvsup (I had updated them earlier so there was nothing to really update. What I did was remove the ports tree, then I cvsup again). So far, no g_vfs_done errors and no crashes (knock on wood).

Here’s how I installed the server step by step:

1) I have 10×1TB Hitachi HD’s. I created 1 Volume at 100GB using RAID6 and another Volume at approximately 6.3TB using RAID6 as well. Make sure you use LBA 64 for any volume greater than 2TB. You can obviously create your own volumes depending on your usage. However, the 100GB Volume is just for FreeBSD.

2) Install FreeBSD on the first Volume, which should be da0. Do not touch the da1 volume.

3) When that’s done, install all the sources using sysinstall, and navigate to Configure > Distributions > src, and choose ALL.

4) Now we edit the driver. Open up /usr/src/sys/dev/arcmsr/arcmsr.c with any editor. You will see the following if statement at or around line 1304.

if(acb->srboutstandingcount >= ARCMSR_MAX_OUTSTANDING_CMD) {
pccb->ccb_h.status |= CAM_SCSI_BUSY; //<-- REMOVE THIS LINE
arcmsr_srb_complete(srb, 0);
return;
}

All you need to do is remove one line and add 2 more. The new if statement should look like this:

if(acb->srboutstandingcount >= ARCMSR_MAX_OUTSTANDING_CMD) {
pccb->ccb_h.status &= ~CAM_STATUS_MASK; //<-- ADD THIS LINE
pccb->ccb_h.status |= CAM_REQUEUE_REQ; //<-- ADD THIS LINE
arcmsr_srb_complete(srb, 0);
return;
}

5) Recompile the kernel and reboot (I was going to go into detail on the kernel recompile, but I will post this elsewhere).

Hopefully you have no errors up to this point. Now we partition the >2TB filesystem, mount it, and add to the fstab. From this point, I will use my volume set up. Just adjust to your settings if yours are different.

6) We will be using gpt to create a >2TB partition table.

# gpt create -f /dev/da1

7) Next, we add a gpt partition.

# gpt add /dev/da1

You can view the partition by using this:

# gpt show /dev/da1

8 ) Now we format the partition.

# newfs /dev/da1p1

9) Create a folder, let’s just name it /HUGE_STORAGE and then we mount it.

# mkdir /HUGE_STORAGE
# mount /dev/da1p1 /HUGE_STORAGE

10) Finally, we add this to fstab. Open up /etc/fstab and add this line to the bottom.

/dev/da1p1 /HUGE_STORAGE ufs rw 2 2

This exact set up has so far been a success. I will post anything that comes up, good or bad, here. Good luck!!

Done!!

UPDATE: Feb. 21, 2008 

I am happy to say that I have been monitoring this server and haven’t received any errors at all!! It runs about 3 rsyncs simultaneously nightly, and I’ve already tar/gzip’d  a 96GB file, mounted 2 samba mounts (which doesn’t have anything to do with the RAID card or hard drives), and moved/copied/removed/hard linked directories that were approximately 100Gb.