SSH How to ignore checking IdentityFile

This is expected behaviour according to the man page of ssh_config:

 IdentityFile
         Specifies a file from which the user's DSA, ECDSA or DSA authentica‐
         tion identity is read.  The default is ~/.ssh/identity for protocol
         version 1, and ~/.ssh/id_dsa, ~/.ssh/id_ecdsa and ~/.ssh/id_rsa for
         protocol version 2.  Additionally, any identities represented by the
         authentication agent will be used for authentication.  

         [...]

         It is possible to have multiple identity files specified in configu‐
         ration files; all these identities will be tried in sequence.  Mul‐
         tiple IdentityFile directives will add to the list of identities
         tried (this behaviour differs from that of other configuration
         directives).

Basically, specifying IdentityFiles just adds keys to a current list the SSH agent already presented to the client.

Try overriding this behaviour with this at the bottom of your .ssh/config file:

Host *
  IdentitiesOnly yes

You can also override this setting on the host level, e.g.:

Host foo
  User bar
  IdentityFile /path/to/key
  IdentitiesOnly yes

in order to connect to an SSH server and authenticate with your public/private keypair you have to first share your public key with the server.

this is done by copying the public key for your private key to the server, and adding it to ~/ssh/authorized_keys either by copy/paste, copying id_rsa.pub to ~/.ssh/authorized_keys on the server or with cat id_rsa.pub >> ~/.ssh/authorized_keys, appending it to the list.

when you connect, the server uses your public key to sign a challenge, and your client uses your private key id_rsa to decrypt the challenge, re-encrypt it with the server's public host key and send it back.

the host verifies that you decrypted the challenge properly, by decrypting your response with its private key, and the client/host establish an encrypted connection, based on the shared data, not on your public/private keys.

at NO POINT in the exchange is your private key, or the host's private key exchanged or revealed to one another. your public key IS stored on the server, but that's why it is a PUBLIC key.