Silent Outlook popups regarding self-signed certificate when connecting to pop3-ssl server

Found the solution. It was caused by the Microsoft LiveMeeting add-in. It created a message form in the C:\Users\[current user]\AppData\local\Microsoft\FORMS folder that was corrupted. Deleted it and it fixed the problem.

Yes, you are correct that if you don't specify --ssl-ca then the client does not check the server certificate at all. Since it works without that option the most likely reason for the failure is that the client doesn't trust the server certificate.

If you are using self-signed client and server certificates then the ca.cert file should include both these files. That way the client will trust the server certificate and the server will trust the client certificate.

For example:
Generate the server key and certificate:

$ openssl req -x509 -newkey rsa:1024 \
         -keyout server-key-enc.pem -out server-cert.pem \
         -subj '/DC=com/DC=example/CN=server' -passout pass:qwerty

$ openssl rsa -in server-key-enc.pem -out server-key.pem \
         -passin pass:qwerty -passout pass:

Generate the client key and certificate:

$ openssl req -x509 -newkey rsa:1024 \
         -keyout client-key-enc.pem -out client-cert.pem \
         -subj '/DC=com/DC=example/CN=client' -passout pass:qwerty

$ openssl rsa -in client-key-enc.pem -out client-key.pem \
         -passin pass:qwerty -passout pass:

Combine the client and server certificates into the CA certificates file:

$ cat server-cert.pem client-cert.pem > ca.pem