Centos – Routinely unable to remove qla2xxx module under CentOS

I routinely cannot unload the Qlogic fibre module qla2xxx under CentOS 5 or 6. These machines are running StorNext cvfs, EMC powerpath and hostagent. Nothing else should ever be touching the fibre. All three of these processes have been stopped before trying to remove the module. The interesting thing of the output below is the lsmod: qla2xxx is being used by 45 things but none are listed. There are no "fibre-ish" processes visible (that I can tell) after running sudo ps ax. I have not tried a "force (-f)" removal Force removal (-f) does not help the situation.

# service cvfs status
Active SNFS mountpoints:
Error: fsmpm not running!

# service PowerPath stop
Stopping PowerPath: PowerPath is not running done

# service hostagent status
hostagent is stopped

# modprobe -v -r qla2xxx
FATAL: Module qla2xxx is in use.

# rmmod -v qla2xxx
ERROR: Module qla2xxx is in use

# lsmod | grep qla
qla2xxx               366555  45 
scsi_transport_fc      52241  1 qla2xxx

# modprobe --show-depends qla2xxx
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/scsi_tgt.ko 
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/scsi_transport_fc.ko 
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/qla2xxx/qla2xxx.ko ql2xfailover=0

# modprobe -r -v scsi_transport_fc
FATAL: Module scsi_transport_fc is in use.

# modprobe --show-depends scsi_transport_fc
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/scsi_tgt.ko 
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/scsi_transport_fc.ko

# lsmod | grep scsi_
scsi_transport_fc      52241  1 qla2xxx
scsi_tgt               12173  1 scsi_transport_f

# modprobe -r -v scsi_tgt
FATAL: Module scsi_tgt is in use.

# modprobe --show-depends scsi_tgt
insmod /lib/modules/2.6.32-220.7.1.el6.x86_64/kernel/drivers/scsi/scsi_tgt.ko 

Update: There is a kernel thread persisting for each HBA on the host, which should be killed when I attempt to remove the module.

# ps ax | grep qla
  551 ?        S<     0:00 [qla2xxx_0_dpc]
  557 ?        S<     0:00 [qla2xxx_1_dpc]
  563 ?        S<     0:00 [qla2xxx_2_dpc]
  569 ?        S<     0:00 [qla2xxx_3_dpc]