In our environment ZFS has become a critical component of our Storage Infrastructure. We have been able to provision Fibre Channel storage from our ZFS file systems which give us the benefit of data integrity, deduplication, performance through use of the ARC, L2ARC, and ZIL as needed. Additionally the real benefit to ZFS is our storage can run on commodity hardware which reduces the TCO significantly over the major Storage vendors. Now administering ZFS is really quite simple, however getting Fibre Channel can be a bit more work. However after reading this you ought to be able to handle this with your eyes closed, well maybe not with your eyes closed. But you will be much better prepared.
You will need
- Working ZFS Implementation on Solaris 11 Express.
- QLogic FC HBA which supports the QLT driver. We are using the QLE2562.
- Of course a working fabric and clients in order to utilize the storage.
Install Prerequisite Packages
# pkg install stmf fcoet srpt target Packages to install: 4 Create boot environment: No Services to restart: 1 DOWNLOAD PKGS FILES XFER (MB) Completed 4/4 38/38 0.8/0.8 PHASE ACTIONS Install Phase 138/138 PHASE ITEMS Package State Update Phase 4/4 Image State Update Phase 2/2 PHASE ITEMS Reading Existing Index 8/8 Indexing Packages 4/4
Ensure that the Fibre Channel HBA is Detected
# prtconf -D | grep qlc pci1077,15d, instance #0 (driver name: qlc) pci1077,15d, instance #1 (driver name: qlc)
Determine the PCI Address of the Fibre Channel HBA
# mdb -k Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sd ip hook neti arp usba uhci stmf stmf_sbd qlc fctl sockfs md lofs random idm crypto smbsrv nfs sppp ipc ptm fcp cpc fcip nsmb ufs logindmux ] > ::devbindings -q qlc ffffff01cb184040 pciex1077,2532, instance #0 (driver name: qlc) ffffff01cb183d90 pciex1077,2532, instance #1 (driver name: qlc) > $q
Now we will need this from the above output pciex1077,2532.
Deactivate the QLC Driver for the Fibre Channel HBA
Remove the default driver (qlc) from the adapter.
# update_drv -d -i 'pciex1077,2532' qlc
Activate the QLT Driver for the Fibre Channel HBA
Add the target driver (qlt) to the adapter.
# update_drv -a -i 'pciex1077,2532' qlt
Reboot the Machine
# reboot
Confirm the HBA is using the QLT Driver
# prtconf -D | grep qlt pci1077,15d, instance #0 (driver name: qlt) pci1077,15d, instance #1 (driver name: qlt)
# mdb -k Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sd ip hook neti arp usba uhci stmf stmf_sbd fctl sockfs lofs random idm crypto cpc smbsrv nfs fcip sppp ufs ptm ipc ] > ::devbindings -q qlc > ::devbindings -q qlt ffffff01cb184040 pciex1077,2532, instance #0 (driver name: qlt) ffffff01cb183d90 pciex1077,2532, instance #1 (driver name: qlt) > $q
Configure the SCSI Target Service
# svcs stmf STATE STIME FMRI disabled 13:27:08 svc:/system/stmf:default
# svcadm enable stmf
# svcs stmf STATE STIME FMRI online 13:29:54 svc:/system/stmf:default
Display the Status of the SCSI Target Service
The ALUA Status is what allows a LUN to be served via FC and iSCSI at the same time. We do not require or want it.
# stmfadm list-state Operational Status: online Config Status : initialized ALUA Status : disabled ALUA Node : 0
Create a File System (Volume) to be Provisioned
In order to present a LUN we will need to create a ZFS file system as a volume.
# zfs create -V 10G tank/test_lun0
Notice the lack of a mount point, this is due to it being provisioned as a volume.
# zfs list tank/test_lun0 NAME USED AVAIL REFER MOUNTPOINT tank/test_lun0 10.3G 1.15T 23.9K -
Create a LUN for the Volume
We must create a LUN with the Volume, this basically gives it a WWN.
# sbdadm create-lu /dev/zvol/rdsk/tank/test_lun0 Created the following LU: GUID DATA SIZE SOURCE -------------------------------- ------------------- ---------------- 600144f0e8efcb0000004e52bcc60001 10737418240 /dev/zvol/rdsk/tank/test_lun0
Retrieve the WWN of the Adapters
This will be needed to add members to the Target Group.
# fcinfo hba-port HBA Port WWN: 21000024ff309840 Port Mode: Target Port ID: 0 OS Device Name: Not Applicable Manufacturer: QLogic Corp. Model: QLE2562 Firmware Version: 5.2.1 FCode/BIOS Version: N/A Serial Number: not available Driver Name: COMSTAR QLT Driver Version: 20100505-1.05 Type: unknown State: offline Supported Speeds: 2Gb 4Gb 8Gb Current Speed: not established Node WWN: 20000024ff309840 HBA Port WWN: 21000024ff309841 Port Mode: Target Port ID: 0 OS Device Name: Not Applicable Manufacturer: QLogic Corp. Model: QLE2562 Firmware Version: 5.2.1 FCode/BIOS Version: N/A Serial Number: not available Driver Name: COMSTAR QLT Driver Version: 20100505-1.05 Type: unknown State: offline Supported Speeds: 2Gb 4Gb 8Gb Current Speed: not established Node WWN: 20000024ff309841
Lets trim the output for readability.
# fcinfo hba-port | grep "HBA Port WWN" HBA Port WWN: 21000024ff309840 HBA Port WWN: 21000024ff309841
Create Target Group
The Target Group is used to control which adapters are used to present LUNs.
# stmfadm create-tg ust4zfs201
Add Members to our Target Group
Any WWN that we add to this group will be able to present the LUNs we associate with this Target Group via a view.
# stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309840 stmfadm: STMF target must be offline
Since the SCSI Target service is already running we must stop it to add members to the Target Group.
# svcadm disable svc:/system/stmf:default
We want to associate all of our WWN with this Target Group since we won’t be restricting LUNs to a specific adapter.
# stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309840
# stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309841
Now we can restart the SCSI Target service
# svcadm enable svc:/system/stmf:default
Create Host Group
The Host Group is used to allow access to LUN(s) by Client(s). I name my Host Groups functionally, so for example if the storage would be associated with a single client named testhostg then I would name the group that. If it were to be used as shared storage for a VM then I would name it as the VM instead, since it would attach to multiple clients.
# stmfadm create-hg testhostg
Add Members to the Host Group
Here we simply add the WWN of the client(s) to the Host Group.
# stmfadm add-hg-member -g testhostg wwn.2100001b32878b01
Create a View
The view will tie everything together. It associates the Host Group with the LUN and the Target Group. Without this the LUN will not be usable.
# sbdadm list-lu Found 1 LU(s) GUID DATA SIZE SOURCE -------------------------------- ------------------- ---------------- 600144f0e8efcb0000004e52bcc60001 10737418240 /dev/zvol/rdsk/tank/test_lun0
600144f0e8efcb0000004e52bcc60001 is what we needed. Now lets wrap everything up neatly in a View.
# stmfadm add-view -h testhostg -t ust4zfs201 600144f0e8efcb0000004e52bcc60001
Now from your client you should see your storage as a block device. You will need to perform a scan or a refresh from the client.