Messaging tends to concentrate on the reliable exchange of messages around a network; using queues as a reliable load balancer and topics to implement publish and subscribe.
An ESB typically tends to add different features above and beyond messaging such as orchestration, routing, transformation and mediation.
I'd recommend reading about the Enterprise Integration Patterns which gives an overview of common patterns you'll tend to use in integration problems which are all based above a message bus (though can be used with other networking technologies too).
For example using open source; Apache ActiveMQ provides a loosely coupled reliable exchange of messages. Then you can use Apache Camel to implement the Enterprise Integration Patterns for smart routing, transformation, orchestration, working with other technologies and so forth.
It's a fairly high level concept of abstraction. The central concept is that the ESB provides the middleware and interfaces that allow businesses to connect their applications without writing code.
This could include mediation to reconcile incompatible protocols, data, and interaction.
The idea of a central bus on which everything passes gives opportunity for additional layers of abstraction. Using industry standards to "plug" other applications, clients, and such into this bus makes it so that connecting new services, data sources, clients with disparate needs is relatively easy.
Actual implementations
As far as actual implementations, that's the domain of very large enterprise support businesses. While it's very buzzwordy, the goal is an ideal that on some small level can be understood through comparison with the internet:
Similarity to Internet
One big communications bus with widely different uses and data, but all running standardize protocols.
One can, in fact, write an HTTP to FTP connector that would allow browsers to access FTP sites without invoking an FTP client (usually built into the browser now).
Mashups
Mashups demonstrate an interesting implementation - take some bus route data from the San Francisco authority, map from google, and sushi bar locations from yahoo with ratings and run a simple query that gives you the closest sushi bar, weighting it so that you'd be willing to travel a little further for a better bar.
All completely different services, incompatible by themselves, but using standard connectors (yahoo pipes, for instance) they can be pulled together into a cohesive and useful whole.
-Adam
Best Answer
Good question - the crucial difference between a service bus and messaging system is the data convention on your messaging system. A messaging system typically let's you sent everything: binary blobs, XML, comma-separated lists, etc. So application A can send a comma-separated string to application B and B sends some XML to application C and C sends some other XML to app D. That's messaging, but not a 'service bus'. You could say a messaging system is 'untyped' (dynamic structure) while an ESB is 'typed' (static structure).
In a 'service bus' you have a common data definition for all applications and adapters on that bus (could be XML with a shared XSD). Common data objects (CDOs). Anything that connects MUST send it's information adhering to this data definition. The ESB should support loading, sharing and versioning this common data definition. The big advantage is that you can connect a component (e.g. a Message Broker) and it does it's thing without having to know which application sent this data and where this data is going to.
The trade-offs of Messaging vs. ESB are similar to other untyped/typed choices: REST vs. SOAP, unvalidated XML vs. XML with XSD, Groovy vs. Java, ... Some people will enjoy the additional structure (looks good on paper - managers like it) - some will hate it (stuff breaks when versions change, for a little addition you have to update everything - hackers don't like it so much ;-)
Coming back to your questions (reordered)
Message Oriented Middleware (MOM): software libraries for various languages with a broker (or not) to communicate 'messages' between applications. One step up from TCP/IP communication. 'messages' are structured objects or text strings or binary data. Usually you have additional reliability over TCP/IP or UDP. Some examples: TIBCO RV and EMS, IBM MQ, Apache ActiveMQ, ZeroMQ, ...
Java Messaging Service (JMS): the definition of a common API for MOM's - people complained that when your application switches from MOM 'X' to MOM 'Y' you need to rewrite the messaging code. If you code against JMS, you can just switch the libraries and the same application that used to work with TIBCO EMS suddenly works with ActiveMQ (or vice versa)
Enterprise Messaging System (EMS): TIBCO's implementation of JMS (product name: TIBCO EMS)
Enterprise Service Bus (ESB): an ESB uses a message oriented middleware to integrate applications, databases, brokers, etc. An ESB is a MOM with added data structure and structure definition management. When connecting a new component to an ESB, you can expect more 'compatibility' out of the box than when connecting it to a MOM. In an ESB there are higher standards on what a component must do to connect. TIBCO's ESB is called ActiveMatrix, I think.