BGP TCP Ports – Why BGP Uses TCP 1027 Instead of 179

which peer will send the open message first?

Normally, the speaker that opens the socket sends the first OPEN message. But it actually doesn't matter (ref the DelayOpen timer), because BGP also provides a way to delay the OPEN message so the opposite peer can send first:

    Option 1:     DelayOpen

    Description: The DelayOpen optional session attribute allows
                 implementations to be configured to delay sending
                 an OPEN message for a specific time period
                 (DelayOpenTime).  The delay allows the remote BGP
                 Peer time to send the first OPEN message.

         Value:       TRUE or FALSE

In the event that both speakers open duplicate TCP sessions and send OPEN messages on each socket simultaneously, the BGP Identifier is used to resolve which socket should be closed. See RFC 4271, Section 6.8:

6.8. BGP Connection Collision Detection

If a pair of BGP speakers try to establish a BGP connection with each other 
simultaneously, then two parallel connections well be formed. If the source IP address 
used by one of these connections is the same as the destination IP address used by the 
other, and the destination IP address used by the first connection is the same as the 
source IP address used by the other, connection collision has occurred. In the event 
of connection collision, one of the connections MUST be closed.

Based on the value of the BGP Identifier, a convention is established for detecting 
which BGP connection is to be preserved when a collision occurs. The convention is to 
compare the BGP Identifiers of the peers involved in the collision and to retain only 
the connection initiated by the BGP speaker with the higher-valued BGP Identifier. 

Is there any good BGP Peer fsm diagram?

Wikipedia has this simplified BGP FSM.

The difference is that R4 is originating the route to 192.168.0.4/32 (with the network statement), whereas the route to 192.168.0.2/32 is being received from another BGP router.

As BGP on R4 is originating the 192.168.0.4/32 route it would never attempt to install it in its routing table as it has a better local route that it sourced the information from. It will advertise this route to other BGP routers, but would not use it itself as it is the originator of the route.

The route to 192.168.0.2/32 on the other hand was received from another router via BGP and also another protocol. The other protocol had a lower AD so a RIB failure is shown in BGP for that route.