Linux – How To Tell SendMail To Explicitly NOT Try TLS For A Domain

you can use the recipient ACL and use the encrypted condition, see here.

deny
    domains = secure.mail.org
    ! encrypted = *

EDIT

To force sending encrypted mail to some domains, you can create a "required tls" transport (specifying host_require_tls), and then create a router for the domains you want, e.g (untested):

begin router

tls_router:
    driver = accept
    domains = secure.mail.com
    transport = tls_smtp

begin transport

tls_smtp:
    driver = smtp
    hosts_require_tls = * 

Sendmail may be performing an MX lookup on the domain, which is not necessarily what you want. Enclose the hostname in square brackets to prevent this.

define(`SMART_HOST',`[foo.bar.com]')dnl