I'm not sure that this is something you'd want to get into for a live site - for a hobbyist project, it would be fun and a great learning experience. If you miss one simple thing in a front facing ecommerce website, your entire customer base could be compromised, and that's irreversible damage.
Like the others, working with one of the existing platforms (and making sure it is updated constantly) is a better idea if you're going to be deploying this on a live site. You should be familiar with PCI-DSS Compliance, which is a way to better protect your customer base from potential attacks.
I've personally worked quite a bit with Magento - for academic purposes, you should take a look at their schema and overall structure. The architecture is very interesting and the database structure is vast. I believe it is pretty PCI-DSS compliant out of the box, though I haven't deployed it recently.
I hope this isn't too blunt, but the task you are undertaking here is extremely difficult, and the odds of you getting it right are slim. Security flaws are most of the time caused by mistakes in implementation, not in the underlying technologies. In order to make a system like the one you've described secure, you have to use the correct tools and the correct methodology and account for all of the edge cases or the security of the entire system will be compromised.
That's not really a helpful answer though, is it? When you are building a system like you are building the question you should be asking shouldn't be "How do I do this?" It should instead be "What is the way I can do this that relies the least on myself?" The answer to that question is to use tried and tested systems wherever possible, and to roll your own solutions only as a last resort.
To answer your first point about encryption, it doesn't make sense to worry too much about securing a key in memory of the server. If an attacker has enough access to a machine to read your keys out of memory, you are totally and completely hosed and any solutions that you have coded up aren't going to help much any way. In other words, favor securing data at rest and data that is moving over the internet, since that is where most attacks are going to occur.
As far as storing the data goes, I don't see any reason why asymmetric crypto needs to be involved here. I would use something like PBKDF2 to derive a key directly from the user's password, then encrypt the data and store the encrypted blob in a database. I would recommend a database over a flat file because managing a folder full of flat files is tedious at the best of times. Databases may not show any solid benifits in speed or security over flat files, but they come with many other features such as pooled connections and they also make backing up data much easier than flat files. Use the simplest system you can to minimize your attack surface, and use thoroughly tested open source tools whenever possible. If you can find a way to use GPG for the encryption and key derivation part of things, I would recommend it.
As far as transfer goes, I believe that you are thinking about things the wrong way. Don't do any encryption client side. Browser javascript is not suitable for cryptography, as explained in this article. So long as you make sure that you use TLS/SSL for all connections to your site, you shouldn't need to worry about transmitting data unencrypted. For an example of why it is hard to do client side encryption, do some googling about the security of MegaUpload's successor, MEGA.
Finally, I wouldn't trust any one dude you get an answer from on the internet, including myself. I would do a lot of research about this sort of thing before committing to a solution. Also, I might recommend asking this question over at the IT Security Stack Exchange.
-- EDIT --
Somehow, I totally missed the fact that there are three parts to your system, the client (browser), the server (database), and the connector that imports data from the VisualFox Database. This actually makes the whole system a lot more complex, because there are essentially three parties that need to share a secret, instead of two. What I would recommend is not to encrypt the data based on the users password, but to instead encrypt it based on some server password. I'm having a little bit of trouble thinking of a good way to describe this process, so I'll give you an example workflow instead.
Server Side
- Admin starts server.
- During start up, server code asks for a password.
- Server uses PBKDF2 to derive a key which is stored only in memory.
- Server spawns a thread that will poll the VirtualFox Pro server every X (days/hours/minutes) for updated data.
- Server enters loop awaiting requests from browser clients.
Updating database
- Main Server's child thread requests an update of data from the Virtual Fox Pro server.
- VirtualFox Pro server dumps a report containing data for client's with modified entries.
- VirtualFox Pro server opens secure connection to main server (ssh, sftp, etc) and transmits zipped data.
- One by one, the main server uses the PBKDF2 derived key that is stored in memory to decrypt blobs stored in a database, update them with new data, reencrypt them, and store them back into the database. This process should all happen in-memory.
Browser client connects
- Main server receives https request from client.
- Main server uses some third party authentication framework to check clients credentials. This framework should use bcrypt to hash passwords and only store the hashes on the file system.
- If the authentication framework positively identifies a user, the main server will decrypt the user's blob using the PBKDF2 derived key in memory and send the data to the user.
- When the user's authentication cookie expires, the main server will stop using the PBKDF2 derived key to decrypt data, and will instead prompt the user to re-authenticate.
This model is more in line with how traditional websites work (which means that you can rely on third party, bug tested frameworks), but data is encrypted/decrypted in memory before touching the database. Ideally, you could use GPG or some other keystore for managing the encryption keys on the main server as well.
Best Answer
Every time you call session_start() php either creates or reads (if already exists) a cookie named PHPSESSID. This cookie stores a unique identifier for the browser session, its php's way of telling one session from the other. A bot can easily refresh this cookie with a new identifier on each request and mislead php into recognizing each failed login as a first time try, so no its not secure.
Exactly the same can be done with the IP, all $_SERVER variables can be manipulated. A clever bot can send different IP on each request. So, if you are battling a malicious bot, requests are unidentifiable.
Two solutions, an easy one and a paranoid one:
Easy solution
Add CAPTCHA by default on your login page.
Paranoid solution
Create a table in your database to log the timestamp of each and every request, regardless of IP or any other identifier. Whenever you get a request to your login page decide if you are going to show the CAPTCHA based on the elapsed time from the immediately previous request. Make sure to have a cron job (or manually) delete old rows from the table, so it won't get very big very soon.
You will have to chose a sane interval, based on the traffic of your site. Make sure the interval is configurable, cause hopefully your traffic will rise over time. The success or failure of the solution depends on choosing a balanced interval.
For extra paranoid points log and compare microtime().
Ordinarily a login script does not need a CAPTCHA, as the strain on the database to check two indexed fields against a table of a few thousand users shouldn't be much. But if your site does have more than a few thousand users, or you have valid reasons to suspect that a bot is targeting you the paranoid solution should be sufficient.
Update:
As Dean notes, faking the IP is not trivial, that's what I meant by a clever bot, didn't really thought it was valuable to elaborate further. There is a related question at serverfault.stackexchange with a detailed accepted answer on the matter. (Related articles at Symantec and Linux Security)
Of course it would be somewhat secure to go with an IP based solution, especially if your site is not likely to be an intended target (but how can you know that?). Extra things you have to factor in an IP based solution would be dynamic IPs and how to distinguish between firewalled users.
Personally, I always go with the paranoid approach, and never absolutely trust $_SESSION, $_COOKIE and $_SERVER variables.