Sunfire V120 Fibre Channel Booting
I recently discovered you can boot older IDE/SCSI Sun SPARC systems off of fibrechannel, giving us an option for using modern drives on these machines without complicated and underpeformant netboot setups or drive emulators.
Fibre Channel
During an unlikely project to use a 650MHz SPARC Sunfire V120 as a ZFS file server I found myself compelled to make use of Fibre Channel. FC is a SCSI transport channel implemented as a SAN, Storage Area Network, allowing multiple computers to make use of storage interfaces with addresssable resources. PCs can boot over fibrechannel with option rom cards, and for Sun machines, cards exist with forth code(FCode) that allows them to boot over FC as well.
Background
The Sunfire V120 in use only supports SCSI drives, and only has a single 32 bit 5V PCI slot. A standard PCI slot of this sort actually has about a gigabit of bandwith which is more than enough for the needs of a storage server with two 10/100 ethernet ports. I intended to use SAS/SATA with this storage directly, but discovered that 5V tolerant PCI-X SAS cards either did not exist, or were completely unavailable on ebay. What I did discover in browing ebay is that 5V tolerant FC cards were widely available.
Hardware and installation
I purchased a Sun 375-3102 2GB SG-XPCI1FC-QF2 X6767A 5V tolerant one port PCI-X FC card and installed it into the PCI slot on my Sunfire. In obp you should find it with show-devs and should be able to get output like this showing it's recognized.
cd /pci@1f,0/pci@1/SUNW,qlc@5
ok .properties
.properties
assigned-addresses 81022810 00000000 00c00200 00000000 00000100
82022814 00000000 0000a000 00000000 00002000
82022830 00000000 00020000 00000000 00020000
port-wwn 21 00 00 e0 8b 14 09 d1
node-wwn 20 00 00 e0 8b 14 09 d1
reg 00022800 00000000 00000000 00000000 00000000
01022810 00000000 00000000 00000000 00000100
02022814 00000000 00000000 00000000 00001000
compatible pci1077,2300.1077.106.1
pci1077,2300.1077.106
pci1077,106
pci1077,2300.1
pci1077,2300
pciclass,0c0400
pciclass,0c04
manufacturer QLGC
#size-cells 00000000
#address-cells 00000002
device_type scsi-fcp
name SUNW,qlc
version ISP2300 Host Adapter Driver: 1.14.09 03/08/04
I do have some thoughts on this card now that I'm more familiar with the technology. The first thing is that 2G FC is very legacy at this point, 4G is the slowest speed which is widely available so compatability must be kept in mind. The second concern I now have is that I would have preferred to get a card with two FC ports, because a FC port can only be a target or initator, not both.
Operating system
The only OS I've used so far with FC on SPARC is Tribblix, the "retro" Illumos (Solaris) distro. It's small enough to use on an underpowered system like this has all our drivers, supports legacy Solaris packages like the Light Out Manager cli/driver. It uses SRV packagind and is overall simple and easy to understand. It's mostly a personal choice but I've appreciated it so far.
Partitioning
Support was more or less out of the box. Boot the installer off the CD (which takes an exceptional amount of time on IDE) and then the firt thing you need to do is fix the formatting. You're most likely using PC formatted drives, and the installation will fail with weird errors, while 'format' will silently fail. The command you need is format -e, run it and select the disk you intend to format, writing a new partition layout and selecting the sun layout when asked.
For devices where you intend to use ZFS as a full disk, you can skip this set, ZFS does not mind how disks are formatted on PC or SPARC.
Installation
Installation is straightforward folliwng the standard steps for Tribblix or your distributin, the format command will list the drives like normal and installation will mostly proceed as normal.
On the Sunfire V120, the installer will fail to update the boot device, which is where we'll have to step in. List the drives and their WWN (World Wide Name) and write down the WWN of the target drive.
root@tribblix:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t3d1 <HP-EG1200JEHMC-HPD5 cyl 48638 alt 2 hd 255 sec 189>
/pci@1f,0/pci@1/SUNW,qlc@5/fp@0,0/ssd@w210000108663881c,1
Specify disk (enter its number): ^C
It's there at the end of the path: w210000108663881c.
Booting
On some hardware, the obp has some native abilities to list FC drives with a verb like probe-fc-devices, and the installer would probably succeed in setting the boot drive. The V120 can't do this but thankfully we can refer to Correcting device paths when replacing fiber boot disks after ufsrestore
These tech notes helpfully document that older SPARC machines can boot FC devices by embedding the WWN in the device path. For example
nvalias fc0 /pci@1f,0/pci@1/SUNW,qlc@5/fp0,0/disk@w210000108663881c,0:a
setenv boot-device fc0
From there you should be able to reboot into your new Tribblix installation off of a fresh drive, congratulations!
Fibre Channel Targets
The setup I have is a Sunfire V120 connected to an Atto 6500E SAS Fibrebridge, connected to a 25 tray SAS expander. This thing is awesome, it automatically maps and exposes SAS drives as FC targets. You probably shouldn't go this route.
- They're rare
- The N version that's widely available is the "netapp" version, meant for use in IBM Metroclusters. Half the ports are disabled and it may not be usable for these purposes at all. Also does NOT support tape drives.
- If you can find one it's around $90 or more
Instead, the best option is another PC with another FC cards. Cards can be purchased for around $20 and are widely available. Using SCST/LIO on Linux or COMSTAR on Solaris, your FC ports can be configured to act as fibre channel targets. You can host real disks, disk images, or ZFS vdevs over the fibre channel. I have not actually worked with any of these unfortunately, as I have my bridge appliance.
Closing thoughts
FC seems to be a cheap and surprisingly easy way to use modern drives on Sun machines that only have vintage interfaces, I've found it more than sufficient for my project and I am working to extend my setup into a sort of FC lab. I've only tested this on the Sunfire V120 but I think this could be used with any number of PCI, PCI-X, and PCI-E machines.