Fail2ban-regex matches with strings, but not with log and conf files, why

You're almost there, the * isn't doing what you think it is as it matches 0 or more of the previous character try

^<HOST> -.*(AhrefsBot)

for example

fail2ban-regex '5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"' '^<HOST> -.*(AhrefsBot)'
Running tests
=============

Use   failregex line : ^<HOST> -.*(AhrefsBot)
Use      single line : 5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> -.*(AhrefsBot)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed

So it seems that you're not very familiar with regexes, you have a steep learning curve. The fail2ban utility uses python regexes, it's worth reading that page a little.

Part of the problem you are having is this part of your failregex

^<HOST>

This says look for the pre-defined <HOST> regex at the beginning of the line (or immediately after a newline), that's that the ^ is for.

Looking at your log examples they all begin with a date/time, this is removed by fail2ban before the regex is applied to the rest of the line. The line doesn't begin with anything that '^' would recognse so that's why your regex is failing.

A simple example using your errorlog entry. If you want to take action for scriptunknown errors (that may or may not be a good thing) you could use a failregex like

failregex= scriptunknown", clinet: <HOST>

You can test this by running it past your log file using fail2ban-regex(1) e.g.

fail2ban-regex /path/to/logfile 'scriptunknown", client: <HOST>'
Running tests
=============

Use   failregex line : scriptunknown", client: <HOST>
Use         log file : /path/to/logfile
Use         encoding : UTF-8


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] scriptunknown", client: <HOST>
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|  [1] Year(?P<_sep>[-/.])Month(?P=_sep)Day 24hour:Minute:Second(?:,Microseconds)?
`-

Lines: 2 lines, 0 ignored, 1 matched, 1 missed [processed in 0.00 sec]    
|- Missed line(s):
|  [01/Oct/2015:09:15:52 +0800] - 60.18.17.206, 113.21.15.23  "POST/httprl_async_function_callback?count=121 HTTP/1.1" 200 1351 "-" "Drupal (+http://drupal.org/)" "-"

Ok so that may do what you want but it may be too broad, you'd have to look at the results and make those calls.

by the way i dont use iptables software on my server (do i need to install one in order for fail2ban to work?)

You need some sort of firewall that is compatible with fail2ban installed and working on your system. As you tested it and

i manually baned my own IP and it works

Then I guess there is something there doing the job.