Pfsense multi site-to-site wan with policy based routing

We want to create a Hybrid multi WAN site to site connection between two (and more?) remote offices.

The office has two WAN connections to the HQ. One MPLS and one ordinary Internet connection. In the end policy based routing should utilise both connections (For example, send important traffic like VoIP over MPLS and everything else over the cheaper line)

We want to send the traffic on each interface only encrypted over each wire, so we will later create an IPSec or OpenVPN connection on each link. For testing purpose, we started with a GRE tunnel.
Both sides see and can communicate with each other.
We defined two gateways on each side on the pfsense devices. Each with the GRE IP of the remote pfsense device. We also put both into a Gateway group and configured different tiers.
After that we created a floating rule, that matches incoming Traffic on the LAN side to the remote network, and sends it to the gateway group. This works as expected, on both sides.
But the remote side, that receives the packet, does something wrong.
Pflog tells us that a state is created with a default rule on the GRE interface (match in on GRE0). Wer also see traffic on the remote host, and it's answer, but for some unknown reason, no traffic is going back over the tunnel. It seems that the pfsense device somehow forgets/ignores the state? (Further investigation needed)

I can not present a configuration as I am writing this from home, and the machines are in our office (currently in a lab environment)
We see that packets arrive on the remote site, and server, but to coming back to the sender.
From what I understand, the packet should be automatically returned on the gateway it arrived on, as the state is created there. But I also read that the redirect-to rule in pf should also have an return-to rule, on the other side. But no such rule was found in pfctl -sr. And no option in de pfsense webgui.

We are not doing any kind of NAT here!

I would appreciate any ideas on how people solved such a setup, and what I could have done wrong?

I am sorry that I cannot provide more information, but I am trying to solve a non private problem in my private time 🙂