we have several DOS applications (Clipper) which share dbase files on a file server. The applications run under XP. This has worked for about two decades with Netware and for years with Samba (member server) without problems.
Some weeks ago I upgraded openSUSE from 11.4 (samba-3.6.3) to 12.2 (samba-3.6.7) and changed the hardware (to AMD E-450 with 6 GiB RAM). To make it worse (from a debugging point of view) about that time the switch was changed (from 100 Mbit to a 48 port Gb switch).
Since then (it is not clear since which change exactly because the users don't tell us immediately…) a few users face severe problems with certain of these DOS applications which are not precisely reproducible. This seems to be about access rights or (more probable) file locking. As far as we know these applications do byte range locking on the files. I do not know whether (and how) I can get this kind of debugging information from samba. There are no general problems accessing these files. Oplocks are enabled (disabling is inacceptable and does not solve the problem, too).
Then I changed the server structure: Earlier Samba ran on real hardware. I made the host OS a simple installation just serving as a host for VMs) and put Samba into a VM, using the openSUSE 11.4 installation which worked without problems before. The problems have not disappeared since. An upgrade of the Samba VM (to 12.2) seems to have made it even worse. Regular Windows share access seems to not have been affected in any of these configurations. ifconfig shows that about one of every 4000 RX packets is dropped on the interface which seems OK to me.
Any ideas, either for the real problem or for a precise Samba debugging / tracing that shows me what exactly the problem is in the communication between Samba and the XP clients?
Without better ideas I will probably first try a different NIC. Years ago that has solved a (general, not DOS related) Samba problem for me.
Best Answer
Make sure to isolate the error from the client perspective. You must understand what the client tries to do before you debug at the server side.
Note that DOS doesn't know anything about Oplocks - so I don't really see how they're affecting you with this particular issue. When a client locks a file with the standard DOS system call for this, it will get locked as a whole. A second client will then encounter the "Error 5" described.
Since this worked earlier, it suspect that the application does not use the standard locking mechanism, but instead uses it's own - whatever it is. This would mean that some other process locks the file. You can hunt for open files (locks) with
lsof.