Using Folder Redirection to get the "My Documents", "Desktop", and potentially "Application Data" folders out of the user's roaming user profile will help matters tremendously.
I'd review this document from Microsoft: "Managing Roaming User Data Deployment Guide". It's dauntingly in-depth, but it's filled with very good information.
Some people only use Folder Redirection and don't use roaming user profiles. I tend to disagree with that strategy because, to me, the user's profile is user data, too, and needs to be backed-up with the same degree of stewardship as their "overtly" data items. Roaming user profiles makes moving the user to a new PC a lot easier, too.
Whether you use "Offline Files" to cache data client-side is dependent on your environment. Windows XP doesn't handle Offline Files well when the user's redirected folders get larger than 2GB in size. Windows Vista and Windows 7 have a much-improved caching engine and do a better job. To my mind, if the user's computer isn't portable there isn't a "win" in using "Offline Files". Others' views may vary.
Finally, I tend not to use the default security paradigm that Microsoft "recommends". I set the permissions on the root of a shared folder hosting user redirected folders to something like: SYSTEM/Full Control, Administrators/Full Control, Authenticated Users/List Folder Contents-This folder only. Then, I pre-create each user's folder and add a That-User/Full Control permission. I change the default settings in "Folder Redirection" to prevent granting the user "exclusive access" (which really means "mess up my folder permission hierarchy and turn off inheritance"). I also set the group policy setting to "Do not check for ownership of Roaming Profile Folders" to enabled to allow me to set the security on my roaming profile folders the same as above.
I do all this w/ permissions for two reasons. Microsoft's defaults "break" the permission inheritance hierarchy on my filesystem, and I find that both irritating and an obstacle that I'll invariably have to fight with at some point in the future. Secondly, the "Microsoft way" invovles setting the share-root folder to allow users to create subfolders. At best, this is just lax security. At worst, one user could launch a denial-of-service attack against a new user by pre-creating the folders for that user before they logon and setting the permissions to deny the new user access.
Best Answer
My recollection was that starting with Windows 2000 the roaming profile client would compare the dates / times of files in the server copy of the profile with the local copy and, if the local copy was up-to-date, it would not copy the server-side file back to the client. Finding documnetation from Microsoft of this behaviour has proven impossible. (Yet again, we have one of those situations where access to the Windows source code would put the matter to rest, but instead we're forced to poke at it with sticks and stones like cave men...)
My recollection was based on the story that prompted my behavior of always using Folder Redirection on the "Desktop" folder. At a Customer site, a user was saving several multi-gigabyte CAD drawings to their "Desktop", rather than in the appropriate folders on the server computer. The hard disk drive in their PC died, and my then-employer dispatched a technician to replace the hard disk drive and re-image the PC. I got a call because the replacement image was "freezing up" at the "Loading your personal settings..." dialog. I observed, using "Network Monitor" on the server computer, that files were being copied from the server to the client. A little more sniffing around uncoverd the 20GB+ of files in the user's server-side profile's "Desktop" folder that were copying down to the client. The technician was being impatient and kept rebooting the PC, causing the process to start over with the file that was partially copied on the prior attempt. Once we let everything copy the user could logon and logoff with no problems or delays.
So, with this story in mind, I just did a little mockup in a virtual machine using Windows Server 2003 R2 as the file server computer and Windows XP Professional SP3 as the client.
I logged-on to the VM with a user account who already had a roaming user profile to allow the profile to be cached by the client, then I logged-off. This caused a 12MB roaming user profile to be cached to the client.
I logged-on to the clent again, this time capturing the traffic between the client and the server with Wireshark (running on the server computer). I saw rougly 2,500 Ethernet frames go by during the logon. Under 1.5MB of traffic moved between the client and the server during this logon, though you'll recall the profile directory is over 12MB. This is a pretty good indication that a full copy isn't taking place on each logon (but I wanted more proof, as you'll see).
The behaviour I observed in the capture was a recursive traversal of the folders in the profile, starting with the root directory, using the "Firstfirst" API. After that traversal was complete, the client perfomed a folder-by-folder walk through the profile performing an "open" and then "query file info" on each folder.
Nowhere during this logon did I see the contents of the files in the profile actually traverse the wire (and, as I said, the entire conversation was under 1.5MB, whereas the data that makes up the profile is 12MB).
I dropped a 56MB file into the profile and logged-off. I verified that the file appeared on the server copy of the profile after the logoff completed on the client, then deleted the file from the cached copy on the client computer's hard disk drive (Via the "C$" share).
I logged-on to the client again while watching with Wireshark and observed a 60MB transfer between the client and the server during logon. I could see clearly in the capture where the client requested the contents of the 56MB file from the server.
I logged-off and logged-on again, this time leaving the locally cached 56MB file on the client alone. In that logon, the total transfer between the client and the server was again just under 1.5MB.
This seems to confirm my memory of the behavior, at least in Windows XP Professional SP3.
So, why are you seeing the long logon delays? Besides being "slowish" (I'd argue they're just "slow"), your WAN links are probably fairly latent (as compared to the local Ethernet). The recursive directory traversal I observed in the logon above took about 3 seconds to complete on a LAN. It would be a huge multiple of that on a WAN, even though very little data was actually crossing the wire. SMB sucks like a vacuum cleaner over latent links.
I suspect that if you "cleaned up" a user's profile of extraneous files in the "Application Data" folder, the "Cookies" folder, etc, you'd see a decrease in logon and logoff time. Unfortunately, junk tends to pile up there and it's difficult to keep in check.
Some ideas:
Immidio Flex Profiles creates a ZIP file that contains all the user profile data and uploads that to the server computer. It would work better on latent links than the built-in profile replication engine. This toolkit used to be no-cost, but I believe it's now a for-pay product.
If users don't move between off-site offices frequently you could put a "server" in each off-site location to be the "Roaming Profile Server" for those users who work in that office, and back it up remotely. You could use a Windows client PC in this role, so long as more than 10 users don't try to connect to it at once (and it doesn't get turned off). Alternatively, since you're already familiar with Samba you could put a low-end Linux machine out there hosting the roaming profiles in each office, and use something like rsync to copy the files back to the hub location for backup.
If you want to gold-plate the solution you could put a Windows Server machine into each office, replicate the roaming profiles share with DFS-R, and users would be able to move freely between all the offices.