That is a lot of questions! It sounds like you're trying to proxy GlassFish through Apache so users can access your applications on standard ports (80 and 443), and you've got multiple applications, multiple domains, and you want to use SSL.
Well, you've got a lot of work ahead of you then! You're probably going to need to look into virtual hosting on Apache; in particular, one virtual host for webmail.mydomain.com and then another for mydomain.com.
If you don't have two IP addresses (two NICs) on your webserver then you'll have to use name-based virtual hosting. Be aware that name-based virtual hosting and SSL don't work together easily; you'll probably have to use an SSL certificate with common name mydomain.com and alias webmail.mydomain.com (altSubjectName extension).
Information on configuring Apache can be found here:
http://httpd.apache.org/docs/2.2/vhosts/
Information on using name-based virtual hosting with SSL can be found here:
http://wiki.apache.org/httpd/NameBasedSSLVHosts
Information on configuring GlassFish can be found here:
http://download.oracle.com/docs/cd/E18930_01/html/821-2416/gfaad.html
I'm assuming that what you're after is to use client-certificate authentication in your JBoss application, to authenticate browsers (not Apache Httpd as a client to JBoss).
If you want to have Apache Httpd as a reverse proxy in front of you JBoss container, you need to configure Apache Httpd to request and handle client-certificate authentication. In particular, you should use its SSLCA* and SSLVerifyClient directives, part of mod_ssl.
Whether you configure SSL/TLS between Apache Httpd and your JBoss worker node is independent. It's often unnecessary if you're on a trusted network at that point. If you do want SSL/TLS there, use Apache Httpd's SSLProxy* directives to configure which CA to trust. This being said, doing so will certainly create more confusion within your application, since there would be an ambiguity as to where the client-certificate information comes from: either a real-client certificate authentication at the JBoss container level, or a relayed certificate, as handled by Apache Httpd.
Indeed, you'll need to pass on the client-certificate information to your container for further information, as described in this answer.
If instead, what you're after is client-certificate authentication between Apache Httpd and JBoss, where you need this connection to be secured with SSL/TLS and to make sure if comes from the reverse proxy (which would present its certificate), you should be able to to this using SSLProxyMachineCertificateFile available in Apache Httpd 2.4. (This configuration is certainly unusual.)
EDIT: (Following changes to the question.)
As per my understanding, Apache is configured to use 2 way mutual
authentication with SSLVerifyClient optional_no_ca meaning that client
may or maynot provide the certificate.
SSLVerifyClient optional_no_ca means that Apache Httpd will only check that the client has the private key for the certificate it presents: it won't verify that the certificate is trusted (making SSLCACertificateFile useless). If you want presentation of the certificate to be optional, but still verify it against your PKI, use SSLVerifyClient optional (with SSLCACertificateFile).
Now jboss is configured to one way SSL authentication. Now what I
understand is ,when browser send request apache,apache will respond
with the certificate and browser will try to authenticate using its
root CA or throw an exception asking user to store it.
Yes, and the connection between the browser and Apache Httpd has nothing to do with JBoss.
And when apache will route request to jboss,here apache will act as
client and jboss as SSL server,jboss will send its certificate from
keystore which will be verified by the Apache using
SSLCACertificateFile directive
No, it's SSLProxyCACertificateFile (or ...Path)
And if jboss has to redirect to itself ,it will have to go through the
reverse proxy as we have set proxyPassReverse.In that case jboss will
act as SSL client and Apache http as SSL server and Apache will will
send its certificate which jboss verify using the CA certificate in
trustore. Am I right in interpreting the config files?
I'm not sure under which circumstances JBoss would "redirect to itself" (or what you mean by this). There's nothing to suggest JBoss has a role as a client here. This is not what ProxyPassReverse is for.
Also I dont exactly understand the use of optional_no_ca in
SSLVerifyClient.Will apache request the certificate from browser or
not or it depends upon the browser ?
When you set SSLVerifyClient to optional, optional_no_ca or require, the server will request a certificate. With require, it will terminate the connection if the client doesn't send one (that it trusts).
As your configuration stands, there's nothing to suggest that the client-certificate is conveyed to the JBoss container. It's not verified in any way at the Apache Httpd level (anyone with a self-signed client-certificate could connect here). You could in principle use optional_no_ca, let any certificate through and only verify them once they arrive to the Java container, but you'd certainly need some extra custom code to do so in your JBoss application. You would also need to convey the certificate itself somehow (e.g. via mod_header and a custom header or more directly using mod_proxy_ajp. It would also be harder to break the connection to make the client try another certificate if the one it presented initially was incorrect.
Best Answer
You're confusing two things.
If you want port forwarding from port 443 to 8443, don't go via Apache Httpd, just forward the port (for example, via
iptables). In this case your JBoss container must be configured to handle the SSL/TLS connection (all the certificate settings).If you want a reverse proxy from Apache Httpd (listening on port 443) to your JBoss container, you don't need to enable SSL/TLS on your JBoss container (especially on localhost), just proxy the request to Apache Httpd in plain HTTP (or via AJP). For this, you'll need to configure Apache Httpd to handle the SSL/TLS connection.