To answer your question directly, this is expected behavior by OSPF and not a bug in Quagga.

So first, let's take a look at the DR/BDR section of the RFC.

Designated Router
    The Designated Router selected for the attached network.  The
    Designated Router is selected on all broadcast and NBMA networks
    by the Hello Protocol.  Two pieces of identification are kept
    for the Designated Router: its Router ID and its IP interface
    address on the network.  The Designated Router advertises link
    state for the network; this network-LSA is labelled with the
    Designated Router's IP address.  The Designated Router is
    initialized to 0.0.0.0, which indicates the lack of a Designated
    Router.

Backup Designated Router
    The Backup Designated Router is also selected on all broadcast
    and NBMA networks by the Hello Protocol.  All routers on the
    attached network become adjacent to both the Designated Router
    and the Backup Designated Router.  The Backup Designated Router
    becomes Designated Router when the current Designated Router
    fails.  The Backup Designated Router is initialized to 0.0.0.0,
    indicating the lack of a Backup Designated Router.

If the BDR field in the Hello packet header is set to 0.0.0.0, it means you do not have a BDR elected.

In your case this is because you have your other router set to a priority of 0, this makes the router ineligible to become a BDR (this is why you see "DROther" and not "BDR"). You just need to set the priority on your other router to something that isn't 0.

Here is the other piece from the RFC for some more context.

Router Priority
    An 8-bit unsigned integer.  When two routers attached to a
    network both attempt to become Designated Router, the one with
    the highest Router Priority takes precedence.  A router whose
    Router Priority is set to 0 is ineligible to become Designated
    Router on the attached network.  Advertised in Hello packets
    sent out this interface.

https://www.ietf.org/rfc/rfc2328.txt

Pete said:

I can't think of a situation where you would need this. I just wanted to know what the logic was that makes this an explicit check in these routing protocols.

Short answer

Routing protocols are some of the most fundamental building blocks on the internet; we need them to be very reliable in every possible case. It does no good to bring up an OSPF or EIGRP adjacency on a mismatched MTU.

Routing protocols must remove any potential mismatched MTUs from the router's forwarding path.

Long answer

I can think of three possible situations where you'd find mismatched IGP MTUs...

1. Unintentional MTU mismatch at Layer2 (for instance, if someone accidentally mismatched MTUs on a serial line, or different vendors had different default MTUs on the same media)
2. Matching Layer2 MTUs, but a the router implementation has bug which miscalculates the required interface IP MTU
3. Intentional MTU mismatch

IP MTUs are directly correlated to Layer2 MTUs (at least for Case 1, above). No matter what we do, we are always at the mercy of mitigating the problems from unintentional Layer2 MTU mismatches, since there is no Layer2 MTU discovery mechanism (unlike IP, which has ICMP error messages).

This means that we have to do everything possible to avoid Layer2 MTU mismatches, even if Cases 2 and 3 above are casualties of mitigating problems with Case number 1. Case 1 has colossal implications unless we solve it; i.e. black-holing all traffic just because we permitted mismatched MTUs.

We're always limited to the least common denominator on the link. Frames larger than the receive MTU of an interface are silently discarded, and the router has no way of knowing whether the MTU was intentionally mismatched, or whether it happened accidentally.

Consequently, EIGRP and OSPF require valid Layer2 adjacencies ^{Note 1} (including MTUs).

What would(Could) the consequence be without matching MTU?

Quoting John Moy (OSPF's author) in RFC 2329 Page 4:

• Problems with all IP forwarding
• OSPF Problems

Also quoting him from the OSPF mailing list:

^{Note 1} some people misunderstand the meaning of adjacency as strictly an IP routing protocol concept. This assertion misses the reality that everything (including IP) requires matching layer2 MTUs, for Layer2 domains to work properly.

One of the most important functions of a routing protocol is building a valid FIB / CEF / forwarding table. That table maps the information learned via routing protocols to layer2 rewrite info. Those Layer2 relationships on the same physical link are what Cisco also calls adjacencies.