I hope this isn't too blunt, but the task you are undertaking here is extremely difficult, and the odds of you getting it right are slim. Security flaws are most of the time caused by mistakes in implementation, not in the underlying technologies. In order to make a system like the one you've described secure, you have to use the correct tools and the correct methodology and account for all of the edge cases or the security of the entire system will be compromised.
That's not really a helpful answer though, is it? When you are building a system like you are building the question you should be asking shouldn't be "How do I do this?" It should instead be "What is the way I can do this that relies the least on myself?" The answer to that question is to use tried and tested systems wherever possible, and to roll your own solutions only as a last resort.
To answer your first point about encryption, it doesn't make sense to worry too much about securing a key in memory of the server. If an attacker has enough access to a machine to read your keys out of memory, you are totally and completely hosed and any solutions that you have coded up aren't going to help much any way. In other words, favor securing data at rest and data that is moving over the internet, since that is where most attacks are going to occur.
As far as storing the data goes, I don't see any reason why asymmetric crypto needs to be involved here. I would use something like PBKDF2 to derive a key directly from the user's password, then encrypt the data and store the encrypted blob in a database. I would recommend a database over a flat file because managing a folder full of flat files is tedious at the best of times. Databases may not show any solid benifits in speed or security over flat files, but they come with many other features such as pooled connections and they also make backing up data much easier than flat files. Use the simplest system you can to minimize your attack surface, and use thoroughly tested open source tools whenever possible. If you can find a way to use GPG for the encryption and key derivation part of things, I would recommend it.
As far as transfer goes, I believe that you are thinking about things the wrong way. Don't do any encryption client side. Browser javascript is not suitable for cryptography, as explained in this article. So long as you make sure that you use TLS/SSL for all connections to your site, you shouldn't need to worry about transmitting data unencrypted. For an example of why it is hard to do client side encryption, do some googling about the security of MegaUpload's successor, MEGA.
Finally, I wouldn't trust any one dude you get an answer from on the internet, including myself. I would do a lot of research about this sort of thing before committing to a solution. Also, I might recommend asking this question over at the IT Security Stack Exchange.
-- EDIT --
Somehow, I totally missed the fact that there are three parts to your system, the client (browser), the server (database), and the connector that imports data from the VisualFox Database. This actually makes the whole system a lot more complex, because there are essentially three parties that need to share a secret, instead of two. What I would recommend is not to encrypt the data based on the users password, but to instead encrypt it based on some server password. I'm having a little bit of trouble thinking of a good way to describe this process, so I'll give you an example workflow instead.
Server Side
- Admin starts server.
- During start up, server code asks for a password.
- Server uses PBKDF2 to derive a key which is stored only in memory.
- Server spawns a thread that will poll the VirtualFox Pro server every X (days/hours/minutes) for updated data.
- Server enters loop awaiting requests from browser clients.
Updating database
- Main Server's child thread requests an update of data from the Virtual Fox Pro server.
- VirtualFox Pro server dumps a report containing data for client's with modified entries.
- VirtualFox Pro server opens secure connection to main server (ssh, sftp, etc) and transmits zipped data.
- One by one, the main server uses the PBKDF2 derived key that is stored in memory to decrypt blobs stored in a database, update them with new data, reencrypt them, and store them back into the database. This process should all happen in-memory.
Browser client connects
- Main server receives https request from client.
- Main server uses some third party authentication framework to check clients credentials. This framework should use bcrypt to hash passwords and only store the hashes on the file system.
- If the authentication framework positively identifies a user, the main server will decrypt the user's blob using the PBKDF2 derived key in memory and send the data to the user.
- When the user's authentication cookie expires, the main server will stop using the PBKDF2 derived key to decrypt data, and will instead prompt the user to re-authenticate.
This model is more in line with how traditional websites work (which means that you can rely on third party, bug tested frameworks), but data is encrypted/decrypted in memory before touching the database. Ideally, you could use GPG or some other keystore for managing the encryption keys on the main server as well.
First of all, could you provide me with some clarification: are you coding a chat room (one-to-many) or individual (one-to-one) chats?
There are a couple of obvious flaws in your current system's design that I would like to point out. First, I'll start out with a brief analysis and then explain what is wrong with it and what you can do to fix it. Obviously if you want to run a successful project you should start from step one and get your hands dirty with systems analysis and design.
Problem
Website receives a high volume of traffic and eventually crashes.
Requirements
- Sustain high volume of web traffic
- Display previous 150 messages
- Secure communication pathway between clients
Problem Analysis
Right away it is obvious that your site is crashing because the code that opens the file is being called hundreds of thousands of times per second. Opening files and writing to them is very memory intensive.
The FILE Probem: Files do not handle concurrency very well at all. In fact, they're terrible with concurrency. Think about it like this, essentially you've opened the same file with notepad with hundreds of thousands of open windows/processes and you're changing the content in all of them simultaneously. When you try to solve a problem with this type of solution you end up with non-deterministic results. Basically, it is impossible to predict what data will be in the file.
Fortunately, there is a way to get deterministic results while still using files if you lock them properly. Unfortunately, this is not a solution to your problem. In your case, only one person would be able to send a message at a time. Surely that is NOT the solution you want!
Wait... there IS a solution:
You CAN USE a Database!
Databases are particularly good at solving this sort of concurrency issue! Depending on what database/engine you use your table may lock or only a single record might lock. In you case, I would suggest a free database like MySQL and a record-locking engine like InnoDB. If you're not a database rookie, you might want to look into MariaDB as well, it is a fork of the MySQL project by the original developer and is a binary drop in.
Basically, there is no way around it using a database for this type of solution. In fact, databases are very powerful and you can program procedures with them. From your query you can choose to select only 150 messages and then order them by most recent very easily. All users will be able to send messages at the same time with a record-locking database engine like InnoDB.
I would like to additionally point out that I would be a little troubled to find out what the code for the rest of the application looks like. It is very easy to write PHP code that looks fine but performs terribly. I'm not sure if you're familiar with asymptotic analysis or unit testing but I highly suggest that you thoroughly test your code before pushing it to production. Given the size of your userbase you should be concerned about code optimization and runtime. If your application/problem/solution was properly analyzed, designed, implemented, tested, and debugged you would have a much better handle on your problem.
It is also very easy to write code in PHP that is insecure. I would like to advise you to test your code thoroughly (try to break it) when writing modules that interact with the database. Poorly coded web applications can be very easy to exploit and given your user base of 300-400k I wouldn't doubt it if Cindy Lou Who suddenly decided to give it her own security audit. If a white hat hacker discovers the flaw in your system they will likely encourage you to fix it. If a black hat hacker discovers the flaw they will likely use it to spread malware and steal information.
Best Answer
I would alter that approach to allow all users to log in, but if the number of active users exceeds 100 then redirect the new user to a holding page. Their session is created but they are now in a queue to leave the holding page.
This makes giving users access a lot easier, because you no longer have to track the 100 logged in users. You can simply order by the record ID of session table. This approach assumes you are tracking sessions with a session table in your database. When a user logs out you delete their session record from the table.
It's assumed you are using an auto-increment integer for your ID column.
To check if the current user should be place on hold is very easy.
Using the first
id
in the table you can quickly check if the user is in the top 100 sessions.As soon as a record above the current use is deleted. They automatically move up in the queue.