Mac verify error: invalid password? when trying to convert .pfx to .pem

You could use BouncyCastle (assuming C#, because you mentioned .NET).

Let's say localhost.pem here contains both the certificate and the private key, something like this should work:

using System;
using System.Collections;
using System.Linq;
using System.Text;
using System.IO;

using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.OpenSsl;
using Org.BouncyCastle.Pkcs;
using Org.BouncyCastle.X509;
using Org.BouncyCastle.Security;

namespace TestApplication
{
    class Program
    {
        static void Main(string[] args)
        {
            StreamReader sr = File.OpenText("localhost.pem");
            IPasswordFinder passwordFinder = new PasswordStore("testtest".ToCharArray());
            PemReader pemReader = new PemReader(sr, passwordFinder);


            Pkcs12Store store = new Pkcs12StoreBuilder().Build();
            X509CertificateEntry[] chain = new X509CertificateEntry[1];
            AsymmetricCipherKeyPair privKey = null;

            object o;
            while ((o = pemReader.ReadObject()) != null)
            {
                if (o is X509Certificate)
                {
                    chain[0] = new X509CertificateEntry((X509Certificate)o);
                }
                else if (o is AsymmetricCipherKeyPair)
                {
                    privKey = (AsymmetricCipherKeyPair)o;
                }
            }

            store.SetKeyEntry("test", new AsymmetricKeyEntry(privKey.Private), chain);
            FileStream p12file = File.Create("localhost.p12");
            store.Save(p12file, "testtest".ToCharArray(), new SecureRandom());
            p12file.Close();
        }
    }

    class PasswordStore : IPasswordFinder
    {
        private char[] password;

        public PasswordStore(
                    char[] password)
        {
            this.password = password;
        }

        public char[] GetPassword()
        {
            return (char[])password.Clone();
        }

    }
}

You'll probably need something a bit more subtle for the IPasswordFinder and if you want to handle certificate chains correctly. For more advanced features, you may be able to find more details in the BouncyCastle examples.

I think the trick is to do a little bit with openssl and then get keytool to do all the real work. So with openssl convert your existing certificate and key into a PKCS12 file. Using openssl you'd have something like.

openssl pkcs12 \
  -export -in cert.crt \
  -inkey cert.key \
  -certfile ica.crt \
  -name "yourKey" \
  -out cert.p12

And then the magic is to import the .p12 into your keystore as if it was another keystore.

$JAVA_HOME/bin/keytool \
  -importkeystore -deststorepass secret \
  -destkeypass secret -destkeystore $KEYSTORE \
  -srckeystore cert.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass secret \
  -alias "yourKey"

You have options for using this within java but I would expect a full answer to involve the following.

import java.security.KeyStore;
KeyStore trustStore  = KeyStore.getInstance(KeyStore.getDefaultType());
FileInputStream instream = new FileInputStream(new File("/your/keystore"));
trustStore.load(instream, "secret".toCharArray());

Hope it works!