Too broad question.
I won't speculate on minimal specs, but if you are going to serve large files to two dozen customers, then SATA+RAID5/6 seems like the way to go performance-wise.
NAS vs custom built Linux box. Two factors here: 1) Availability of admins experienced with Linux whom you trust enough to believe if they say "Yes, we can do this". 2) HW support of the parts you order. Still, if you decide to build your own system, then you may be able to have 3 boxes for the price of 1 NAS system (according to your own calculations), so you could have 2 in production and one idle, ready to be cannibalized in case of any failure.
NAS boxes give you: External support -- somebody else to blame if things go pear-shaped ;). Usually more user-friendly interface (click here to setup your box) and, usually, ability to replace failed disk by a trained monkey. They also have a limited feature set, which you cannot realistically expect to be changed on your request.
Custom-made Linux system will be much more flexible, give more bang per buck, no artificial limits of feature lists and ease of expansion (throw more HDDs into it). OTOH they will require somebody, who knows what he is doing, to handle them.
Also, if your main board/disk controller dies terribly, you can just move the HDDs into any Linux box and have your data available. This may not be the case with a NAS box if it uses a HW RAID solution.
As far as OS-X support goes I think you should be happy with NFS, but last time I was configuring a file server for Macs it was XXth century outside and the boxes talked AppleTalk.
For some background:
The reason you typically get a NAS appliance is to get something that will serve up storage and "just work". Of course you need to do an initial configuration, but after that, you pretty much just leave it alone until a drive fails, or there's a critical firmware update. For a Windows file server, you'll need to do regular maintenance (patching, etc).
NAS devices can also offer excellent performance, and nice features like hot-swap/hot-add of drives. They also will handle building your RAID volumes for you with minor effort on your part.
You can join most NAS's to corporate domains, so additional authentication is not an issue.
For your particular scenario:
If you already own the Windows file server (hardware and licensing), then you're pretty much good to go. It should work OK for your needs and there's no reason to replace it with a NAS. My only question would be, does it at least have a RAID1 for the storage drive? I would recommend that in addition to backups ... it will save you trouble in the long run.
HOWEVER - If you don't already own the server, than it is worth looking at SMB level NAS units. You can get a good quality 4-disk NAS for MUCH cheaper than a new server+licensing. ($1000 or so). The only main question here would be if your particular backup solution will work OK if backing up off a remote target (you won't be able to install a backup client directly on the NAS).
You mentioned FreeNAS - that and Openfiler are viable options for your scenario. I have built an Openfiler box used for a similar scenario as yours and it works well. I was able to join it to an AD-based domain and it works fine. If you are comfortable building a rig from scratch this method would be cheap and effective, but keep in mind you're on the hook for hardware repairs.
Best Answer
I recommend using FreeNAS, which provides support for CIFS protocol (SMB). CIFS is supported natively on Windows and Linux clients. You can also use Ubuntu, but remember that this is a general purpose distribution, so it may require more work in setting than FreeNAS.