How to have Postfix send email to another server if user lookup fails for specific domains

TLS just enables encryption on the smtp session and doesn't directly affect whether or not Postfix will be allowed to relay a message.

The relaying denied message occurs because the smtpd_recipient_restrictions rules was not matched. One of those conditions must be fulfilled to allow the message to go through:

smtpd_recipient_restrictions =
    permit_sasl_authenticated
    check_recipient_access hash:/etc/postfix/filtered_domains
    permit_mynetworks
    reject_unauth_destination

To explain those rules:

permit_sasl_authenticated

permits authenticated senders through SASL. This will be necessary to authenticate users outside of your network which are normally blocked.

check_recipient_access

This will cause postfix to look in /etc/postfix/filtered_domains for rules based on the recipient address. (Judging by the file name on the file name, it is probably just blocking specific domains... Check to see if gmail.com is listed in there?)

permit_mynetworks

This will permit hosts by IP address that match IP ranges specified in $mynetworks. In the main.cf you posted, $mynetworks was set to 127.0.0.1, so it will only relay emails generated by the server itself.

Based on that configuration, your mail client will need to use SMTP Authentication before being allowed to relay messages. I'm not sure what database SASL is using. That is specified in /usr/lib/sasl2/smtpd.conf Presumably it also uses the same database as your virtual mailboxes, so you should be able enable SMTP authentication in your mail client and be all set.

Why doesn't the routing happen before the recipient existence check? Isn't there is way? When I route emails, I mean "The current Postfix doesn't handle anything and just forwards email to another server which is fully responsible for the config".

No, that isn't way postfix gonna works. Recipient existence checks and email routing was two different activities for postfix. Recipient existence checks was performed when accepting email. The process who performs it was smtpd. Email routing was performed when delivering email. The process who perform it was trivial-rewrite

In the comment, you mention that you use virtual delivery agent from postfix to delivering email. I'm gonna tell you that it possible to achieve no-redundant-information in postfix configuration. BUT you must change current postfix setup.

To achieve your goal, we must move all domains in virtual_mailbox_domain class to relay_domains class. Then using dovecot LDA to deliver email in the right mailbox. Relay domains class has special parameter called relay_recipient_maps to checks the recipient existence.

The good news is you can use relay_recipient_maps together with transport_maps. Why? Because relay_recipient_maps only used to check existence of user. The value/right hand side string in transport_maps was ignored when recipient existence check. Snippet from man postconf.

Optional lookup tables with all valid addresses in the domains that match $relay_domains. Specify @domain as a wild-card for domains that have no valid recipient list, and become a source of backscatter mail: Postfix accepts spam for non-existent recipients and then floods innocent people with undeliverable mail. Technically, tables listed with $relay_recipient_maps are used as lists: Postfix needs to know only if a lookup string is found or not, but it does not use the result from table lookup.

For the completeness, here the postfix configuration for this setup. Credits to sebokopter for the idea.

#main.cf
# rename virtual_mailbox_domains to relay_domains
relay_domains = mydomain.example.com
# put the parameter value of virtual_mailbox_maps AND transport_maps parameter
relay_recipient_maps = hash:/my/virtual_mailbox_maps, hash:/my/transport_maps

transport_maps = hash:/my/transport_maps
relay_transport = dovecot:

#master.cf
dovecot   unix  -       n       n       -       -       pipe
    flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${recipient}