Iptables – Limit incoming connections using iptables per IP

I need to limit access to some port per IP. Let's say 5 connections per minute – not more.

I've seen iptables recent, connlimit and limit, but all of them are not fitting exactly what I need.

Suppose you have a client trying to connect every second. In my scenario I need to allow 5 packets each minute.

recent: If some IP tries to connect every 1 second, –hitcount 5 will memorize this IP and keep it in the list until no packets comes within –second 60 time. So, it will limit the client permanently in my scenario.

limit: This one limits as I wish with –limit 5/min, but for all IPs – no way to specify this per IP.

connlimit: Limits number of simultaneous connections, not per some time.

In fact, I need a mixture of limit + recent. Who knows how to do it?