Is this message familiar to someone ?
zfs-clu-01a rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots
free
zfs-clu-01a last message repeated 75220 times
We have a lot of trouble using NFS on ZFS .
Actually we cannot copy big files (>500mo) over NFS and we have performance
issues.
NFS client using NFSv3 over TCP
**--NFS Server Configuration :**
Concurrent NFSD Servers: 4096
NFSD queue length: 512
Concurrent LOCKD Servers: 4096
LOCKD queue length 512
**-- System configuration**
**1/ Disable ZIL**
Settings -> Preferences -> System
Sys_zfs_nocacheflush Yes (default: No)
Warning: it is unsafe without UPS
**2/Disable Nagel algoritm**
Settings -> Preferences -> Network
Net_tcp_naglim_def 1 (default: 4095)
**4/ disable sync**
Data Management -> Shares -> Folder xxx
Sync disable (default: Standard)
I can provide more information if required.
——– UPDATE 01 ————-
I have a new interesting thing :
The problem is only or most seen with "centos" system ( release 6.5 )
Same thing with Debian (last version) is working good on this file copy as the same time period
Here is more information :
OS: Nexenta Ver. 31-5-0
Load Average 1.25, 1.26, 1.21
NFSv3 Server (configuration in my last email)
1) NFS CLIENT
Clients Number: 15 with very high access demand and then we have more than 20 others low access
Centos 6 and Debian 7
NFS Client v3
Mout options :
nfs (rw,noatime,nodiratime,vers=3,rsize=16384,wsize=16384,proto=tcp,mountproto=tcp,nolock)
Nics: 1x 1g
Dedicated NFS vlan
2) zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
syspool 464G 181G 283G 39% 1.00x ONLINE -
tank1 6.52T 2.38T 4.13T 36% 1.00x ONLINE -
3) zpool status
pool: syspool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
syspool ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tank1
state: ONLINE
scan: resilvered 190G in 81h49m with 0 errors on Sun Jan 26 15:35:10 2014
config:
NAME STATE READ WRITE CKSUM
tank1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c6t5000C5005F0C34A7d0 ONLINE 0 0 0
c6t5000C5005F1261CBd0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c6t5000C5005F531E2Fd0 ONLINE 0 0 0
c6t5000C5005FD1677Fd0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c6t5000C5005FD18BF7d0 ONLINE 0 0 0
c6t5000C5005FD1907Bd0 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
c6t50000394B820EA30d0 ONLINE 0 0 0
c6t5000C5005FD1D133d0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c6t5000C5005FD73F97d0 ONLINE 0 0 0
c6t5000C5005FD7518Bd0 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
c6t5000C5005FD7C2A3d0 ONLINE 0 0 0
c6t5000C5005FDC123Bd0 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
c6t5000C5005FDE1D13d0 ONLINE 0 0 0
c6t5000C5005FDEAFDFd0 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
c6t5000C5005FDEB127d0 ONLINE 0 0 0
c6t5000C5005FDEB513d0 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
c6t5000C5005FDF88CBd0 ONLINE 0 0 0
c6t5000C5005FDF9C4Fd0 ONLINE 0 0 0
mirror-9 ONLINE 0 0 0
c6t5000C50067802A17d0 ONLINE 0 0 0
c6t5000C500678037A7d0 ONLINE 0 0 0
mirror-10 ONLINE 0 0 0
c6t5000C5006780D6F7d0 ONLINE 0 0 0
c6t5000C5006780DEB3d0 ONLINE 0 0 0
mirror-11 ONLINE 0 0 0
c6t5000C5006780E90Fd0 ONLINE 0 0 0
c6t5000C5006780E99Bd0 ONLINE 0 0 0
cache
c0t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
spares
c6t5000C5006BF8C91Fd0 AVAIL
errors: No known data errors
4) hddisco
=c6t5000C5006BF8C91Fd0
driver sd
driver_instance 37
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS08
serial S0M19QGA
guid 5000c5006bf8c91f
device_id id1,sd@n5000c5006bf8c91f
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5006bf8c91d
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5006bf8c91e
P end
=c6t5000C5005F531E2Fd0
driver sd
driver_instance 26
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M04DCN
guid 5000c5005f531e2f
device_id id1,sd@n5000c5005f531e2f
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005f531e2d
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005f531e2e
P end
=c6t5000C5005FDEAFDFd0
driver sd
driver_instance 27
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M061KF
guid 5000c5005fdeafdf
device_id id1,sd@n5000c5005fdeafdf
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdeafdd
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdeafde
P end
=c6t5000C5006780E90Fd0
driver sd
driver_instance 29
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M06GV9
guid 5000c5006780e90f
device_id id1,sd@n5000c5006780e90f
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5006780e90d
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5006780e90e
P end
=c6t5000C5005FD1677Fd0
driver sd
driver_instance 28
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M05WCR
guid 5000c5005fd1677f
device_id id1,sd@n5000c5005fd1677f
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd1677d
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd1677e
P end
=c6t5000C5005FDF9C4Fd0
driver sd
driver_instance 30
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M03SZR
guid 5000c5005fdf9c4f
device_id id1,sd@n5000c5005fdf9c4f
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdf9c4d
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdf9c4e
P end
=c6t5000C5005FD7518Bd0
driver sd
driver_instance 25
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M06JCM
guid 5000c5005fd7518b
device_id id1,sd@n5000c5005fd7518b
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd75189
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd7518a
P end
=c6t5000C5005F1261CBd0
driver sd
driver_instance 24
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M03LZY
guid 5000c5005f1261cb
device_id id1,sd@n5000c5005f1261cb
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005f1261c9
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005f1261ca
P end
=c6t5000C5006780E99Bd0
driver sd
driver_instance 20
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M06GV8
guid 5000c5006780e99b
device_id id1,sd@n5000c5006780e99b
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5006780e999
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5006780e99a
P end
=c6t5000C5005FD1907Bd0
driver sd
driver_instance 23
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M0657D
guid 5000c5005fd1907b
device_id id1,sd@n5000c5005fd1907b
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd19079
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd1907a
P end
=c6t5000C5005FDF88CBd0
driver sd
driver_instance 22
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M04CQ7
guid 5000c5005fdf88cb
device_id id1,sd@n5000c5005fdf88cb
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdf88c9
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdf88ca
P end
=c6t5000C5005FDC123Bd0
driver sd
driver_instance 21
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M05WMD
guid 5000c5005fdc123b
device_id id1,sd@n5000c5005fdc123b
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdc1239
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdc123a
P end
=c6t5000C5006780D6F7d0
driver sd
driver_instance 19
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M05TVK
guid 5000c5006780d6f7
device_id id1,sd@n5000c5006780d6f7
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5006780d6f5
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5006780d6f6
P end
=c6t5000C5005FDEB127d0
driver sd
driver_instance 13
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M061JX
guid 5000c5005fdeb127
device_id id1,sd@n5000c5005fdeb127
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdeb125
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdeb126
P end
=c6t5000C5005FD73F97d0
driver sd
driver_instance 16
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M06J86
guid 5000c5005fd73f97
device_id id1,sd@n5000c5005fd73f97
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd73f95
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd73f96
P end
=c6t5000C5005F0C34A7d0
driver sd
driver_instance 14
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M0405D
guid 5000c5005f0c34a7
device_id id1,sd@n5000c5005f0c34a7
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005f0c34a5
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005f0c34a6
P end
=c6t5000C500678037A7d0
driver sd
driver_instance 15
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M03FNP
guid 5000c500678037a7
device_id id1,sd@n5000c500678037a7
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c500678037a5
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c500678037a6
P end
=c6t5000C5005FD18BF7d0
driver sd
driver_instance 17
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M0659G
guid 5000c5005fd18bf7
device_id id1,sd@n5000c5005fd18bf7
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd18bf5
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd18bf6
P end
=c6t5000C50067802A17d0
driver sd
driver_instance 18
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M03WJV
guid 5000c50067802a17
device_id id1,sd@n5000c50067802a17
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c50067802a15
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c50067802a16
P end
=c6t50000394B820EA30d0
driver sd
driver_instance 32
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor TOSHIBA
product AL13SEB600
revision DE06
serial 43L0A02HFRD3
guid 50000394b820ea30
device_id id1,sd@n50000394b820ea30
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w50000394b820ea32
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w50000394b820ea33
P end
=c6t5000C5005FDE1D13d0
driver sd
driver_instance 11
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M060LE
guid 5000c5005fde1d13
device_id id1,sd@n5000c5005fde1d13
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fde1d11
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fde1d12
P end
=c6t5000C5005FDEB513d0
driver sd
driver_instance 6
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M061HL
guid 5000c5005fdeb513
device_id id1,sd@n5000c5005fdeb513
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fdeb511
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fdeb512
P end
=c6t5000C5005FD7C2A3d0
driver sd
driver_instance 9
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M06HG0
guid 5000c5005fd7c2a3
device_id id1,sd@n5000c5005fd7c2a3
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd7c2a1
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd7c2a2
P end
=c6t5000C5005FD1D133d0
driver sd
driver_instance 7
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M064EN
guid 5000c5005fd1d133
device_id id1,sd@n5000c5005fd1d133
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5005fd1d131
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5005fd1d132
P end
=c6t5000C5006780DEB3d0
driver sd
driver_instance 10
device_type disk
removable no
media_present yes
size 600127266816
size_str 558.9GB
caching_supported no
vendor SEAGATE
product ST600MM0006
revision LS06
serial S0M060TL
guid 5000c5006780deb3
device_id id1,sd@n5000c5006780deb3
is_ssd no
mpxio_enabled yes
parent mpxio
parent_instance 0
path_count 2
P start
P parent mpt_sas
P parent_instance 5
P path_state ONLINE
P target_port w5000c5006780deb1
P end
P start
P parent mpt_sas
P parent_instance 4
P path_state ONLINE
P target_port w5000c5006780deb2
P end
=c0t2d0
driver sd
driver_instance 34
device_type disk
removable no
media_present yes
size 511503761408
size_str 476.4GB
caching_supported no
vendor DELL
product PERC H710
revision 3.13
serial 0022d14c0cc2ee9e1900ac67e7a0a38c
guid 6b8ca3a0e767ac00199eeec20c4cd122
device_id id1,sd@n6b8ca3a0e767ac00199eeec20c4cd122
is_ssd no
mpxio_enabled no
lun_no 0
parent mr_sas
parent_instance 0
=c0t1d0
driver sd
driver_instance 33
device_type disk
removable no
media_present yes
size 511503761408
size_str 476.4GB
caching_supported no
vendor DELL
product PERC H710
revision 3.13
serial 0027a4060badee9e1900ac67e7a0a38c
guid 6b8ca3a0e767ac00199eeead0b06a427
device_id id1,sd@n6b8ca3a0e767ac00199eeead0b06a427
is_ssd no
mpxio_enabled no
lun_no 0
parent mr_sas
parent_instance 0
=c0t0d0
driver sd
driver_instance 5
device_type disk
removable no
media_present yes
size 499558383616
size_str 465.3GB
caching_supported no
vendor DELL
product PERC H710
revision 3.13
serial 003f0f9308bb2c521900ac67e7a0a38c
guid 6b8ca3a0e767ac0019522cbb08930f3f
device_id id1,sd@n6b8ca3a0e767ac0019522cbb08930f3f
is_ssd no
mpxio_enabled no
lun_no 0
parent mr_sas
parent_instance 0
Best Answer
Hmms, look like the system (Nexenta) is getting too many NFS connections or you're hitting a bug. You could try to up the maxdupreqs, that could be masking the issue, but let's try that first:
To check what it's current value is use:
But maybe it helps.