Question

What Linux tool can immediately (without queueing) relay e-mails to recipient SMTP server and provide valid bounce messages? Or, if there is better solution of this problem, what tools should I look at?

The postfix itself has MULTIPLE INSTANCE feature. It will lets you define separate instance of postfix that have its own queue and configuration file. You can use this instance as last chain of your SMTP processing. And yes, because it has separate queue, this special instance can send your email without waiting and tinkering another queue.

Delivery Reputation: Would using php mail() vs SMTP make any difference with the delivery rate? From what I understand, Postfix is going to be using SMTP protocol when communicating with other servers on the internet when delivering emails, so therefore is it safe to assume that it doesnt matter if the email was sent to postfix via php mail or smpt from localhost? Does any of these 2 methods affect the email reputation is any ways even in a small way?

No related of all. Postfix can receive email via both mail() and SMTP. After processing it, postfix will send via SMTP.

Performance Load: Is there any performance difference between php mail and SMTP when sending emails? Say, if I am sending like 10,000 emails, which method would take the most resources (or time)? My assumption is that both can take a bit of time like: php mail for compiling the emails with headers, etc.. and SMTP for making connections each time. Which one consumes the most server resources?

Benchmark it! I don't have any data on this.

Here what process involved in mail() and SMTP?

In mail() command, PHP invoke sendmail command and the program puts your email on a file in maildrop queue directory. Pickup daemon scan that directory, and move the email to cleanup daemon.

In SMTP one, PHP create connection to your postfix server via SMTPd. After finishing SMTP ritual, SMTPd will do checks if the emails allowed or not. If a email allowed, it will pass it to cleanup daemon.

Source: official documentation of postfix

Security Issue: When I searched for the difference between the two, many sites says about the security issue with php mail since a hacker can upload php script to send out spams. But I can also see another security issue with SMTP as well since the SMTP username and passwords are stored in configuration file in text which is not secure neither. Since there are security issues for both methods, does one weigh better than the other in any aspects?

As @Tutul said in another answer, you likely expect some spam script use mail() to send out the spam. Yes this is one of consideration to block mail command in php.

However, one principle holds: once someone success placed the script in your web-and-mail server, he can send use ability of PHP to send email either via mail() and SMTP.

In php itself, there are no protection to limit how many SMTP connection or mail() can be invoked by spammer. One place to place the defense line is in MTA (postfix). Unfortunately, you can't throttle incoming mail invoked from mail() command. However you can throttle how many sending request can made from SMTP connection. Policyd or postfwd can help postfix throttle it.

Note: Above explanation is about throttle process when receiving email. Of course you can throttle it when sending email. For example, you limit 20 email per minute to @gmail.com to avoid GMAIL blacklist daemon block you. See the docs about Postfix Performance Tuning

Final note

IMHO, I prefer to send email via SMTP. You can use MTA as additional protection from spam outbreak. You may have to handle another mail queue in PHP as @Sanmain said in another answer though :)