GitLab SSH – Can’t Establish SSH Connection to GitLab CE Setup

I've a Gitlab CE setup running on my server(CentOS 7 + Apache 2.4). Currently, the site works fine and I can fully interact with Gitlab on the browser, however, I can't properly setup my environment on my shell via SSH in order to push my repos to my server.

Currently, when I try to push, I get the following error:

git@[MY_GITLAB_SUBDOMAIN]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Since I've a custom port for my SSH in general, on my local machine, I made the addition to my SSH config file like below:

Host gitlab
Hostname [MY_GITLAB_SUBDOMAIN]
Port [MY_CUSTOM_PORT]
User git
PreferredAuthentications publickey
IdentityFile [MY_PUBLIC_KEY]

Therefore, on the current local repo I'm trying to push, I edited the git config file like so:

[core]
  repositoryformatversion = 0
  filemode = true
  bare = false
  logallrefupdates = true
  ignorecase = true
  precomposeunicode = true
[branch "master"]
  remote = origin
  merge = refs/heads/master
[remote "origin"]
  url = gitlab:can/[PROJECT]
  fetch = +refs/heads/*:refs/remotes/origin/*

I've added my public key to my account on my Gitlab preferences. I've also manually checked the authorized_keys file on my server for the user git along with it's permissions to see if it's working fine. Permission are good(.ssh folder: 700 & authorized_keys: 600) and the key is present there.

What am I missing?