I rather like Scheme, if you want to work with the JVM you should check out Clojure, which is a lisp that is designed to work in the JVM. And yes Lisp is still worth learning to see how powerful such a minimal design can be! The folks who created lisp got some things really right. Its amazing how many of the cool new features of modern languages lisp had in the 1960's!
For an embeded scheme try guile: http://www.gnu.org/s/guile/
There's a number of inter-related questions here, I'll try to separate them as best I can.
Why do other languages build on LLVM IR and not clang AST?
This is simply because clang is a C/C++ front end and the AST it produces is tightly coupled to C/C++. Another language could use it but it would need near identical semantics to some subset of C/C++ which is very limiting. As you point out, parsing to an AST is fairly straightforward so restricting your semantic choices is unlikely to be worth the small saving.
However, if you're writing tooling for C/C++ e.g. static analysers, then re-using the AST makes a lot of sense as it's a lot easier to work with the AST than the raw text iff you're working with C/C++.
Why is LLVM IR the form it is?
LLVM IR was chosen as an appropriate form to write compiler optimisations. As such, it's primary feature is that it's in SSA form. It's quite a low level IR so that it is applicable to a wide range of languages e.g. it doesn't type memory as this varies a lot across languages.
Now, it happens to be the case that writing compiler optimisations is quite a specialist task and is often orthogonal to language feature design. However, having a compiled language run fast is a fairly general requirement. Also, the conversion from LLVM IR to ASM is fairly mechanical and not generally interesting to language designers either.
Therefore, lowering a language to LLVM IR gives a language designer a lot of "free stuff" that is very useful in practice leaving them to concentrate on the language itself.
Would a different IR be useful (OK, not asked but sort of implied)?
Absolutely! ASTs are quite good for certain transformations on the program structure but are very hard to use if you want to transform program flow. An SSA form is generally better. However, LLVM IR is very low level so a lot of the high level structure is lost (on purpose so it's more generally applicable). Having an IR between the AST and the low level IR can be beneficial here. Both Rust and Swift take this approach and have a high level IR between the two.
Best Answer
It's mostly legacy now. Many critical business systems are still in COBOL simply for the fact that they are so big and integrated that the cost of rewriting doesn't seem worth it. Writing a new system in COBOL is probably not feasible anymore, as most COBOL developers are so scarce that they can pull in a considerable amount of money for the specialized skill (similar to a Foxpro developer now). There are few to no reasons to keep a COBOL app around, but unfortunately the common reasoning is when the COBOL app is already in-place, trusted, and tightly coupled with other systems to where it's near impossible to replace. That reasoning is exactly why it should be replaced before it gets to a situation where the only hardware that runs the app has to be custom built from Ebay parts from the 80/90s.