Security – How to setup an IP whitelist for a subdirectory

A client of mine complained of an attack. I checked the access log and found a massive number of requests for the admin login page from seemingly random IP addresses. I created an .htacces file in the /administrator directory and populated it with the following (IP addresses obfuscated):

order deny,allow
deny from all
allow from 96.xxx.xx.xxx #my IP address
allow from 97.xx.xxx.xxx #my client's IP address

I then went to a free proxy server and typed in the URL for the admin page. The page didn't load any of the assets (images), but it did load the actual page itself.

Joomla! does some SEF stuff in the .htaccess file that is in the DocumentRoot. It looks like this:

## Begin - Joomla! core SEF Section.
#
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
#
# If the requested path and file is not /index.php and the request
# has not already been internally rewritten to the index.php script
RewriteCond %{REQUEST_URI} !^/index\.php
# and the request is for something within the component folder,
# or for the site root, or for an extensionless URL, or the
# requested URL ends with one of the listed extensions
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|vcf|raw))$ [NC]
# and the requested path and file doesn't directly match a physical file
RewriteCond %{REQUEST_FILENAME} !-f
# and the requested path and file doesn't directly match a physical folder
RewriteCond %{REQUEST_FILENAME} !-d
# internally rewrite the request to the index.php script
RewriteRule .* index.php [L]
#
## End - Joomla! core SEF Section.

I am guessing that the reason that the .htaccess file in the administrator directory isn't working properly has something to do with the main Joomla! .htaccess file. Is that true? I tried adding this to my main .htaccess file, but it resulted in a 500 error:

<Directory /var/www/vhosts/sweathelp.org/httpdocs/administrator>
    order deny,allow
    deny from all
    allow from 96.xxx.xx.xxx
    allow from 97.xx.xxx.xxx
</Directory>

How can I effectively block all access to the administrator directory, excluding the two whitelisted IP addresses?