Javascript – Using Hash Bang URL’s. Best practices? Best libraries? Important pitfalls

I don't know about "best practice". I do know the most common mistakes.

First Mistake: DOS Yourself

You use the webhandler to process the long running job. This can be bad or extremely bad depending on your percentage of hits that become long running jobs, how long they run and how much sustained traffic you get.

You want to make sure that you aren't getting more than 1 long running job within the period of time it takes for that long running job to complete. If you do you DOS yourself. It will also get worse the more traffic you get assuming the percentage and time stays consistent. It's one of those problems which self-imposes a limit on traffic growth.

Second Mistake: Spawning from the webhandler

Spawning a process from the web handler to handle a long running process can be tricky, and as a result also error prone.

• You have to dissociate from the parent correctly otherwise the webhandler waits for the child to complete.
• When you fork a child in unix it inherits open handles from the parent. These will automatically be closed unless they are overwritten. This includes things like database connections, filehandles, other open network connections. All of which get closed when the child process completes.

Options

I usually use at(1) to cleanly dissociate from the webhandler without forking.

You can also use a polling implementation with cron.

You can communicate to another server process that handles the processing. That communication can be done with sockets, pipes, or higher level abstractions like a REST http call or routing a queue message.

I've worked in multiple projects in a similar environment to what you describe. In my experience something like this works pretty well:

• Each developer should have their own development setup - servers, databases, etc.
  • You need to be able to test your work quickly and without disturbing other people's work
• You also should have a "testing" or "staging" server.
  • This is where you should try to keep an environment as close to your production system as possible.
  • You should test your code here before moving code into production
• The source control repository can be kept on a server as well. (This could be your staging server too, you don't necessarily need a separate one)
  • Although mercurial keeps copies of the repos on the dev machines, it's easier to keep everyone updated if everyone pushes their changes to a central repository.
  • You can also do deployments to staging/testing and later production more easily if all the code is in a central location.

To address some of your problems...

At least with SQL databases it's relatively simple to keep the database shared. You can just take dumps of the data and copy it over to the other people.

You don't necessarily need to have the same exact data on all developer's computers. It should be enough they can perform testing etc. with the data they do have.

The testing/staging server should preferably have the same data as your production server - this is to ensure your code gets tested in a realistic environment before it's moved to production.