R – What format signature does Openssl pkeyutl expect

digital-signatureencryptionopensslrsa

I'm trying to verify a file that was signed by hashing with SHA-1 and encrypting the hash with an RSA private key.

Obviously I'm using the RSA public key to verify. The key is in DER format.

The signature verification works correctly using Java's Signature class.

The openssl command I'm trying (and the result) is:

       ~/Downloads/openssl-1.0.0-beta3/apps/openssl pkeyutl -in encryptedZip.bin 
-keyform DER -verify -sigfile savedDigitalSignature.txt -pubin -inkey public.der
    WARNING: can't open config file: /usr/local/ssl/openssl.cnf
    Signature Verification Failure

I don't see anything in the openssl configuration file that would apply, so I don't think that warning is significant.

The savedDigitalSignature.txt file contains the signature bytes.

My theory is that openssl is looking for the digital signature to be in some specific file format, but I haven't found anything in the documentation indicating what that should be.

Thoughts?

Best Answer

This command is very low level. You have to make sure everything is in the right format for it to work,

The input signature (-sigfile) must be the binary signature. For RSA, the padding must be PKCS#1.
The input data must be the binary digest. If you sign it with SHA1, this file can only contain 20 bytes.
Public key must be in X.509 encoded SubjectPublicKeyInfo in DER or PEM format.

Related Solutions

How to generate a self-signed SSL certificate using OpenSSL

You can do that in one command:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365

You can also add -nodes (short for no DES) if you don't want to protect your private key with a passphrase. Otherwise it will prompt you for "at least a 4 character" password.

The days parameter (365) you can replace with any number to affect the expiration date. It will then prompt you for things like "Country Name", but you can just hit Enter and accept the defaults.

Add -subj '/CN=localhost' to suppress questions about the contents of the certificate (replace localhost with your desired domain).

Self-signed certificates are not validated with any third party unless you import them to the browsers previously. If you need more security, you should use a certificate signed by a certificate authority (CA).

RSA Public Key format

You can't just change the delimiters from ---- BEGIN SSH2 PUBLIC KEY ---- to -----BEGIN RSA PUBLIC KEY----- and expect that it will be sufficient to convert from one format to another (which is what you've done in your example).

This article has a good explanation about both formats.

What you get in an RSA PUBLIC KEY is closer to the content of a PUBLIC KEY, but you need to offset the start of your ASN.1 structure to reflect the fact that PUBLIC KEY also has an indicator saying which type of key it is (see RFC 3447). You can see this using openssl asn1parse and -strparse 19, as described in this answer.

EDIT: Following your edit, your can get the details of your RSA PUBLIC KEY structure using grep -v -- ----- | tr -d '\n' | base64 -d | openssl asn1parse -inform DER:

    0:d=0  hl=4 l= 266 cons: SEQUENCE          
    4:d=1  hl=4 l= 257 prim: INTEGER           :FB1199FF0733F6E805A4FD3B36CA68E94D7B974621162169C71538A539372E27F3F51DF3B08B2E111C2D6BBF9F5887F13A8DB4F1EB6DFE386C92256875212DDD00468785C18A9C96A292B067DDC71DA0D564000B8BFD80FB14C1B56744A3B5C652E8CA0EF0B6FDA64ABA47E3A4E89423C0212C07E39A5703FD467540F874987B209513429A90B09B049703D54D9A1CFE3E207E0E69785969CA5BF547A36BA34D7C6AEFE79F314E07D9F9F2DD27B72983AC14F1466754CD41262516E4A15AB1CFB622E651D3E83FA095DA630BD6D93E97B0C822A5EB4212D428300278CE6BA0CC7490B854581F0FFB4BA3D4236534DE09459942EF115FAA231B15153D67837A63
  265:d=1  hl=2 l=   3 prim: INTEGER           :010001

To decode the SSH key format, you need to use the data format specification in RFC 4251 too, in conjunction with RFC 4253:

   The "ssh-rsa" key format has the following specific encoding:

      string    "ssh-rsa"
      mpint     e
      mpint     n

For example, at the beginning, you get 00 00 00 07 73 73 68 2d 72 73 61. The first four bytes (00 00 00 07) give you the length. The rest is the string itself: 73=s, 68=h, ... -> 73 73 68 2d 72 73 61=ssh-rsa, followed by the exponent of length 1 (00 00 00 01 25) and the modulus of length 256 (00 00 01 00 7f ...).

Best Answer

Related Solutions

How to generate a self-signed SSL certificate using OpenSSL

RSA Public Key format

Related Topic