After having tried to get a yubikey setup to do this for some time (though not with OpenVPN) I came to the conclusion that if this is to work, it has to be supported both by the application and by PAM. That is, the application has to know to ask for three things (instead of the usual two) and the underlying PAM library has to know to expect three things to be passed to it (and to use them appropriately).
The yubikey PAM library doesn't seem to have that support, or at least it didn't have it reliably while I was still trying to make this work.
Instead, I decided to change to using my yubikey in OATH mode, and I found that proper three-phase authentication was handled much better by both sshd and the underlying pam_oath library.
I can't say whether OpenVPN support this better, as I haven't tried it, but you may wish to investigate it as an option if you can't get yubikey mode working properly. It has the added advantage that if any of your users can't use a yubikey for some reason (eg, OpenVPN from an endpoint with no USB port) there are a number of other OATH implementations which can be used e.g. on a smartphone to bail those particular users out without having to completely overhaul your two-factor infrastructure.
In case it's of any interest to you, my writeup on sshd/yubikey/OATH/two-factor/three-phase authentication can be found here.
Edit: no, by application I meant OpenVPN. OpenVPN must know to ask for (effectively) two separate passwords and a username, and the backing PAM module must know to expect those three tokens, and to combine them in a way that is agreeable to FreeRADIUS. It's almost immaterial what that agreed-on method is, as long as the tokens are validated; what's important is that the client-facing side of the whole authentication engine knows how to ask for, and how to deal with, the three different tokens.
Trying to roll your own by suborning PAM into calling the RADIUS plugin twice, with different arguments each time, and hoping that it will somehow magically come out in the wash, seems doomed to failure to me (as well as fraught with potential security holes).
My bigger-picture point was that you were more likely to find a designed-in solution using OATH than the yubikey-specific token handlers, since I know from what I tried that the yubikey-specific handlers didn't seem to like three-token approaches, preferring the catenate-password-and-OTP approach (which I also don't like).
I figured it out. The problem was in the PAM stack for the /etc/pam.d/password-auth file. Specifically this line:
auth sufficient pam_unix.so nullok try_first_pass
What was happening is that the token for google-authenticator was being accepted, but then pam_unix.so was trying to use that code as the system password because of the "try_first_pass" option. I'm not sure why, but this was causing the entire authentication chain to start all over again, asking for the google-auth password.
Getting rid of the "try_first_pass" option fixes the issue and gives me the desired behavior.
Best Answer
I figured this out myself. If anyone is interested, it's related to the configuration in /etc/pam.d/radiusd
First, follow one of these tutorials to set up Google Authenticator as the second factor on your freeRADIUS server:
https://networkjutsu.com/freeradius-google-authenticator/ https://www.supertechguy.com/help/security/freeradius-google-auth/
When it comes to making changes to /etc/pam.d/radiusd, use one of these configurations:
To prompt for password AND Google Auth OTP:
auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass
account required pam_unix.so audit
account required pam_permit.so
To prompt JUST for the Google Auth OTP (i.e. no password):
auth required pam_google_authenticator.so
account required pam_unix.so audit
account required pam_permit.so
Note that this doesn't send a challenge response - it just means that the password doesn't need to be entered in the first place.