Using self signed SSL for mail

Yes, you are correct that if you don't specify --ssl-ca then the client does not check the server certificate at all. Since it works without that option the most likely reason for the failure is that the client doesn't trust the server certificate.

If you are using self-signed client and server certificates then the ca.cert file should include both these files. That way the client will trust the server certificate and the server will trust the client certificate.

For example:
Generate the server key and certificate:

$ openssl req -x509 -newkey rsa:1024 \
         -keyout server-key-enc.pem -out server-cert.pem \
         -subj '/DC=com/DC=example/CN=server' -passout pass:qwerty

$ openssl rsa -in server-key-enc.pem -out server-key.pem \
         -passin pass:qwerty -passout pass:

Generate the client key and certificate:

$ openssl req -x509 -newkey rsa:1024 \
         -keyout client-key-enc.pem -out client-cert.pem \
         -subj '/DC=com/DC=example/CN=client' -passout pass:qwerty

$ openssl rsa -in client-key-enc.pem -out client-key.pem \
         -passin pass:qwerty -passout pass:

Combine the client and server certificates into the CA certificates file:

$ cat server-cert.pem client-cert.pem > ca.pem

SSL X509 certificate file is a file containing a certificate in the format specified by the ITU in their X series of specifications - specifically x.509. There is no private-key information there as it is a certificate format so things like names, public keys, signatures, validity dates and other goodies go in there - but not private-keys.

PEM is a set of specifications for Privacy Enhanced Mail - it offers for the most part packaging standards that are build off of the PKCS specs (PKCS specs were developed by RSA and RSA labs themselves for public use to enable interoperability between various implementations - this was done again by the IETF and by the community in other arenas). You can use openssl to convert formats around - see here http://www.openssl.org/docs/apps/openssl.html . Concatenation won't help you any.

If the certificate is self-signed then it's kind of odd that a CA gave it to you - that makes me think that you are telling about feeding the Gandi CA certificate to openssl which is telling you that cert is self signed (ergo it is a root or end entity certificate but certainly not an intermediate aka chain certificate).

To address some of your points: 1 Self signed certs will always generate warnings in good clients because they lack trust until the user decides they are trustworthy (or for most users.. until Microsoft , Mozilla Foundation, Google, or Opera) decides the user should trust that CA). If your clients (browsers or whatever) have the CA stored and marked as trusted then this warning won't pop-up. 2 Possibly 3 Looks like you used the right key and CSR with Gandi 4 Looks to me like you need to load the domain key and cert and possibly CA into OpenSSL either via OS level trust configuration or via Stud configuration.