Your understanding is correct, if you're from the past. You're pretty much describe as it looked like in 1990s.
Yes, many languages can be executed directly by a web server plugin. Right on for PHP, mod_php for Apache is still the most popular way to host it. However, high-traffic sites use more modern approach, using web server only as a proxy for FastCGI (in case of PHP it's PHP-FPM)
the output is embed into HTML pages is then sent back to the client.
I think you're referring to early 90's so called spaghetti code, however modern approach is to use one of many MVC frameworks. In case of PHP that would mean for example Zend Framework (there are numerous alternatives).
As for ASP, you're probably referring to so called "classic ASP", which is obsolete. Currently it's ASP.NET, which can use any of .NET languages (C# being the most popular), and of course the .NET framework.
C and C++ are not typically used for web application. If so, such services are implemented either as stand alone servers, as module for web server or as FastCGI.
Perl can be executed directly from web serve module using mod_perl. There is also PSGI, which is basically clone of Python's WSGI.
Python is very popular language for web apps. It can be directly executed from Apache web server via mod_python, however that is obsolete and not recommended. Currently the way to go with Python is either via WSGI server module. WSGI server implemented in Python (eg. CherryPy, WebPy) or using stand alone Python web stack (Tornado and Twisted's Web module are fine examples). And of course again, you'd most probably will be using WSGI compatible MVC framework, Django is the most popular one (again, multiple alternatives available).
Ruby, again very popular language for web apps. Best known for web framework Ruby on Rails, which again is MVC. You can execute Ruby directly from server module via mod_ruby or via FastCGI.
Servlets/JSP are executed in stand-alone J2EE application servers, such as JBoss or Tomcat. It's more commonly used to add web interface to business system rather than to create stand alone web apps.
Classical CGI (ie. spawning process on each request) has become obsolete many years ago. It has been replaced by FastCGI (where process is long-running, rather than spawned on each request), server modules, interfaces such as WSGI and clones and stand-alone solutions.
Also the paradigm of the request processing has evolved, with CGI it was process per request. Then was process pool (or thread pool), each process (thread) handling one request at a time. However now, most modern approach is for web servers and stand-alone frameworks to use event-driven programming.
This is an unsolved problem in our field. There's no way to be sure that your code will indefinitely work. Even if your code was truly perfect in the forwards-compatible sense (and if it is, please come work for my company! ;) ), if it runs on, uses, or is used by any other software which gets a bug or changes in any way, your code may not work.
So I can't give you a list of things to do that, if you follow them, will guarantee success. But what you can do is minimize the risk of future breakages and minimize their impacts. A more knowledgeable Pythonist would be able to give you advice more specific to Python, so I will have to be more general:
write unit tests. Even for things that you know don't need them.
using popular/well-designed/stable libraries and technologies, avoiding unpopular (and thus likely to soon be unsupported) ones
avoid writing code that exploits implementation details. Code to interfaces, not implementations. Code against multiple implementations of the same interface. For example, run your code in CPython, Jython, and IronPython and see what happens. This will give you some great feedback about your code. This might not be helpful for Python3 though -- last I heard, some implementations were still in Python2.
write simple, clear code that is explicit about its assumptions
write modular, composable code. If some code must do something dangerous (in the future-proof sense), separate it so that even if it has to change, the rest of the code doesn't.
have a specification of some form. This is similar to the points about unit tests, if you use tests as a spec, and interfaces, which can also be used as specs. (I mean interface in the general sense, not the Java keyword sense).
Doing any of these things may/will increase the amount of work you have to do. I think that makes sense -- a lot of these points can also be made for how to write good code, which is quite difficult (in my opinion). Sometimes you may need to violate some of these suggestions. That is perfectly acceptable, but be aware of the costs.
It's great that the Python team is thinking about this, and for sure they are far more talented and skilled than I will ever be. Still, I would estimate there's a 100% that somebody's code somewhere will stop working the way the want it's intended to when Python is upgraded.
Best Answer
Python is pretty easy to embed and has good documentation on how to do it.
Also, Python has a pretty approachable syntax, even for new users. Perl tends to have obtuse syntax making it less approachable for new users.
Another common language for embedding is Lua. It is known to be fairly easy to embed and has low operating overhead.
Python is well known and used in the Scientific community thanks to SciPy and NumPy, which may influence your particular observations.