Poor SMB Download Speed over VPN

This is most likely due to the SMB protocol being very "chatty", and requring many requests/acknowledgements before and during transfer. Here a couple of things that may help:

• Adjust TCP windowing to be optimal for your network. A google search will bring up many tutorials
• Upgrade to Server 2008 AND Vista machines (requires both to take advantage), as they use SMB2.0 which specifically targets this issue.
• Install a WAN accelerator on either end, be it a dedicated device like a Riverbed device, or the BranchCache feature in Server 2008 R2.

It sounds like you're accessing SMB file shares on a standalone (non-Domain member) Windows Server machine over the Internet w/o using a VPN. This is a very odd configuration and not one that I'd recommend to anyone. I'm not sure I understand your reasoning behind not wanting to use a VPN.

You are going to find that various ISPs, wireless hotspot providers, etc, block NetBIOS over TCP and SMB over TCP (TCP ports 139 and 445, respectively) as a matter of policy. Setting security aside, using a VPN allows you to have arbitrary communication between your servers and clients w/o worrying about network providers filtering your traffic. This, alone, should be reason enough to be using a VPN.

Name resolution via NetBIOS broadcast (unqualified names) isn't going to work at all over the Internet. DNS name resolution will work (fully qualified name or unqualified name if you've hard-set the DNS suffix on the client) if the server computer's name resolves in DNS. Specifying the server's name as its IP address in the UNC should always work anywhere that NetBIOS over TCP and/or SMB over TCP aren't blocked.

If you're going to use a VPN on Windows and want unqualified names to resolve you'll either need to use WINS or configure the clients to qualify names based on the domain name of the server computer. I feel like part of your reluctance to using a VPN might have to do with spotty name resolution. Putting up a WINS server would go a long way toward making name resolution via a VPN work better.

Your architecture here is really, really bizarre and not at all a "best practice". If your users can't handle using a VPN then you might want to consider using IPSEC (albeit, in your non-domain environment you're going to have to hand-configure a lot of things to make that work).