Linux – SynProxy can not return syn ack packets with asymmetric dual bridge topology

I have a asymmetric dual bridge topology as shown below
when i connect from 172.16.11.5 and 172.16.10.6 with ssh
but i can not connect because of SynProxy.

                  -------
                  |     |
                  ---o--- 172.16.11.5
                     |
                     |
                -----o----- 172.16.11.6
                |         |
                |         | default gw 1.1.1.1
                |         |
     1.1.1.2/30 --o----o--- 2.2.2.2/30
                  |    |
                  |    |
                  |    | (enp10s0f0)
              ----o----o-----
              |             |
              |     XXX     |
              |             |
              |  br1   br0  | synproxy
              |             |
              ----o----o-----
                  |    |
                  |    |
                  |    |
     1.1.1.1/30 --o----o--- 2.2.2.1/30
                |         |
                |         | default gw 2.2.2.2
                |         |
                -----o----- 172.16.10.1
                     |
                     |
                  ---o--- 172.16.10.6
                  |     |
                  -------

On all machines between 172.16.11.5 and 172.16.10.6
"rp filtering" is off and "ip forwarding" is on.
There is a machine at the middle of topology which is
called "XXX" machine. XXX has two bridges and a SynProxy.

When SynProxy is turned OFF on XXX, I can ping from
172.16.11.5 to 172.16.10.6 and icmp packets follow
this path: br1->172.16.10.1->172.16.10.6->172.16.10.1->br0.
In addition, I can access from 172.16.11.5 to
172.16.10.6 with ssh. So the TCP traffic works as I expect.

However, when SynProxy is turned ON on XXX, I can ping
from 172.16.11.5 to 172.16.10.6 and icmp packets follow
the same path. But I can not access from 172.16.11.5
to 172.16.10.6 using ssh. This is because synproxy can
not send syn ack replies through br1 iface. If I add a
route for synack packets on XXX, i can connect from
172.16.11.5 to 172.16.10.6 with ssh.

route add 172.16.11.5 dev enp10s0f0

SynProxy rules for XXX:

iptables -t raw -A PREROUTING -i br0 -p tcp -m physdev --physdev-in 
enp10s0f0 -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CT --notrack
iptables -t raw -A PREROUTING -i br1 -p tcp -m physdev --physdev-in 
enp11s0f0 -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CT --notrack

iptables -t filter -A FORWARD -i br0 -p tcp -m physdev --physdev-in 
enp10s0f0 -m tcp -m state --state INVALID,UNTRACKED -j SYNPROXY 
--sack-perm --timestamp --wscale 7 --mss 1460
iptables -t filter -A FORWARD -i br0 -m physdev --physdev-in enp10s0f0 
-m state --state INVALID -j DROP
iptables -t filter -A FORWARD -i br1 -p tcp -m physdev --physdev-in 
enp11s0f0 -m tcp -m state --state INVALID,UNTRACKED -j SYNPROXY 
--sack-perm --timestamp --wscale 7 --mss 1460
iptables -t filter -A FORWARD -i br1 -m physdev --physdev-in enp11s0f0 
-m state --state INVALID -j DROP

But this is not acceptable because 172.16.11.0 network
is cloud. So i could not add route all cloud network
to route table and could not add mac address to arp
table.

How can I connect from 172.16.11.5 to the 172.16.10.6
machine using ssh when SynProxy is turned ON on XXX?
or Is It possible?

Thanks in advance,