It is my understanding that any general-purpose programming language can be used for server-side development of a website.
Am I right in thinking that a server just needs some kind of interface such as CGI to make the server and the programming language work together? If so then why are some programming languages (such as php) more popular than others?
Best Answer
In the early days of the web, CGI was indeed the only (practical) way to have dynamic content (you could do named pipes of files -- and those were used in days before cgi, but that wasn't practical at all).
CGI works by sticking a bunch of information in the environment of the process that is forked and then exec'ed (and possibly some in stdin) and then takes what comes out of stdout and spits that back to the requestor.
This doesn't care one bit about what the implementation language is. Indeed, I wrote my early CGIs back in the day in C or C++. It was kind of painful. I later learned some perl in the early 90s and that was much less painful.
This works, up to a point. The problem is scale. Each CGI request is a fork and exec of a process. Thousands of requests means thousands of processes. That really doesn't work well.
The solution to this is to remove forking and execing by either moving it into a thread in the web server itself, or dispatching the request to another process that handles the request without needing to fork and exec. mod_perl is one such tool to do this (a plugin moving perl into apache). Php (late 90s) also did this with implementing the language as a plugin in the web server itself rather than something that was forked and exceed. This became quite popular as it was perl-like (which was the early dominant web programming language) and could outperform perl cgis. There is still quite a bit of momentum from this period of time in the mid-90s -- before the more enterprise-grade application servers started to take hold with more formalized languages behind them. If you dig around, you can find a lot of failed attempts in the late 90s to early 2000s too -- languages and frameworks that just didn't stick.
This brings us to the application servers where internal threads are spawned (or other approaches -- this isn't the case for everything) to handle requests rather than entire new processes -- which can help with scale. As an external process this could be seen with FastCGI and then later became prevalent with other application servers. Note that with this the line between application server and web server got a bit blurry -- many application servers could double as web servers, though weren't optimized for handling static file IO in the way that traditional web servers are.
The generic application server has also paved the way to solutions where instead of a generic application server, you have the application itself either running an embedded web server or otherwise being the entire deployment. In such situations one doesn't deploy a web application on an application server - it just is running itself and handling requests. Again, the goal of this model is to avoid the heavy price of launching new instances of the application and instead handle the requests inside the application with much lighter weight threads or similar approaches.
Here's the thing though -- all the solutions are deficient in some way, shape, or form. CGI, while easy has serious problems with scale. Plugins in the web servers get bound into the web server itself (apache vs nginx vs IIS vs ...) and lose the common functionality of the language. Microsoft has its own parade of technologies it would like to promote. And if you know one language, wouldn't you rather keep programming in it rather than have different languages in different parts of the stack (javascript in the client and Node.js)?
And so, you've got today. Some people work in a Java stack (with scala and clojure becoming not uncommon). Others in a C# stack. Others in a JavaScript stack. There's quite a bit of php stacks out there. Lots of python. You can still find some perl stacks out there (and if you look at some low volume sites, you'll still find CGIs). With cloud computing, Google has also promoted Go as a viable server side web language.
Each has its advantages, disadvantages, its frameworks and its servers. The relative popularity of these ebbs and flows as technologies around them change. They do different things well.