Iptables – flushed and removed iptables rules, but CSF restores them

To answer your question succinctly, no: there would not be any "leftover" rules after flushing every table. In the interest of being thorough however, you may want to set the policy for the built-in INPUT and FORWARD chains to ACCEPT, as well:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

Clear ip6tables rules:

ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT
ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X

...and that should do it. iptables -nvL should produce this (or very similar) output:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Have a look at the "SECTION:Login Failure Blocking and Alerts" and set the whished settings.

More specific, LF_POP3D and LF_IMAPD for the amount of attempts before its blocking the IP address.

Furthermore you need to check if the log paths are set correctly.

Go way back down into the config, and see that these settings are correct:

POP3D_LOG =
IMAPD_LOG = 

For me both are /var/log/mail.log, but check your system.

In the file csf/regex.pm you can see which attempts are being filtered.