Apache 2.4 – Load Specific PHP Module for Directory on CentOS 7

Inside /etc/httpd/conf.modules.d/ I have a conf file called 10-php54-php.conf which I use to specify what PHP version I want to use.
I can freely switch between PHP 7.2 and PHP 5.4 by commenting line 2 and uncommenting line 3.

Is there a way to specify that I want to use PHP 5.4 in a specific directory and leave the rest of Apache using 7.2? I saw a few guides on our beloved Google telling me to use php-fpm but I'd like to use these .so files if possible.

/etc/httpd/conf.modules.d/10-php54-php.conf:

<IfModule prefork.c>
     LoadModule php7_module modules/libphp7.so
     #LoadModule php5_module modules/libphp54-php5.so
</IfModule>

I already tried some variations inside httpd.conf but apache won't start:

<Directory "var/www/html/directory">
        LoadModule php5_module modules/libphp54-php5.so # tried with LoadModule and Include
</Directory>

Apache 2.4.6, CentOS 7.4.

EDIT: When putting LoadModule inside my virtualhost, it fails to restart when running apachectl graceful.

/etc/httpd/sites-enabled/domain.conf:

<VirtualHost *:80>
        DocumentRoot /var/www/html/
        ServerName www.domain1.com
</VirtualHost>

<VirtualHost *:80>
        ServerName domain2.com
        ServerAlias *.domain2.com
        ServerAdmin admin@domain2.com
        DocumentRoot /var/www/html/domain2
        ErrorLog /var/log/httpd/domain2.com-error.log
        CustomLog /var/log/httpd/domain2.com-access.log combined
        php_admin_value default_charset UTF-8

</VirtualHost>

<VirtualHost *:80>
        ServerName service.domain1.com
        ServerAdmin admin@domain2.com
        DocumentRoot /var/www/html/service
        ErrorLog /var/log/httpd/service.domain1.com-error.log
        CustomLog /var/log/httpd/service.domain1.com-access.log combined
       #LoadModule php5_module modules/libphp54-php5.so
</VirtualHost>

journalctl -xe:

Mai 22 11:33:08 host.host polkitd[700]: Registered Authentication Agent for unix-process:14527:379023715 (system bus name :1.15226 [/usr/bin/pkttyagent --notif
Mai 22 11:33:09 host.host httpd[14533]: [Wed May 22 11:33:09.001953 2019] [so:warn] [pid 14533] AH01574: module php5_module is already loaded, skipping
Mai 22 11:33:09 host.host kernel: httpd[14533]: segfault at 0 ip 00007f46635326b7 sp 00007fff2214f1d8 error 4 in libphp7.so[7f4663284000+45a000]
Mai 22 11:33:09 host.host systemd[1]: httpd.service: control process exited, code=killed status=11
Mai 22 11:33:09 host.host systemd[1]: Reload failed for The Apache HTTP Server.