How to load balance Kafka Boostrap with haproxy

I have a kafka cluster consisting of 3 machines running on AWS:
kafka1 to kafka3

I'm using the new style kafka consumers (>0.8).

I know that the kafka client connects to one of the kafka servers, grabs server metadata and then connects to the brokers directly.

I want to ensure that in the case of a broker failure that the clients are still able to grab metadata.

For this I have an HAProxy load balancer with the following config:

listen kafka
      bind *:9092
      mode tcp
      balance roundrobin
      no option clitcpka
      option forceclose
      timeout check 5s
      server kafka1 kafka1.example.com:9092 check inter 3s fastinter 1s
      server kafka2 kafka2.example.com:9092 check inter 3s fastinter 1s
      server kafka3 kafka3.example.com:9092 check inter 3s fastinter 1s

The idea is that if one of the brokers goes down it is removed from rotation and metadata is grabbed from one of the others. It also allows me to add more brokers to the cluster transparently.

However, this is causing issues with my kafka clients. PipelineDB straight up refuses to consume from topics, so does the confluent kafka library for Python. Kafkacat consumes, but starts giving errors after a while:

% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Connection closed

I can't find any info online on how to load balance the Kafka bootstrap procedure. An alternative would be to just configure a DNS entry with multiple A records, but I run into the problem of one of the brokers being down.