Electronic – Car Battery Voltage

Your alternator has a non-zero impedance, which is higher than the impedance of the partly-charged battery in this case. That's what causes the voltage to drop when the battery is connected.

Or, in other words: The alternator cannot deliver as much current as would be required to put the battery's terminals to 14V, so it delivers as much as it can and the voltage drops. - With higher rpm the picture (voltage) will likely change. Besides, note that an alternator -as the name suggests- produces (rectified) alternating current (AC) where peak voltage and RMS may differ, so be sure what you are measuring. (This is, by the way, one of the reasons why you should never run your car's engine without connected battery; the battery, like a giant capacitor, smoothes the system's voltage ripple and spikes away which would otherwise put significant stress on other electric components.)

LiFePO4 vs LiIon vs LiPo

Some people have commented on LiIon batteries, but the question and this answer are about Lithium Ferro Phosphate batteries / LiFePO4 which I'll abbreviate in places as LFP4. .
These are related to LiIon and LiPo batteries but have major differences.
Notable, compared to LiIon and LiPo (which are chemically similar) LFP4 has an added internal inactive matrix which the Lithium resides within. This reduces mass and volume energy densities but also makes them free from the destructive melt down modes, gives them a much longer cycle life and a much longer calendar life, lower capacity, lower maximum voltage, better temperature operating and storage range, better overall energy efficiency - and a whole of life cost below to far below that of any lead acid battery.

LFP4 (LiFePO4) 4 cell batteries do make excellent replacements for normal 12V lead acid car batteries BUT proper attention to care and feeding is necessary to achieve the very long cycle and calendar lifetimes that they are capable of. They are far more "whole of life cost effective" than any lead acid cells in (probably) any application when properly used. Wrongly used they will die an early and expensive death.

A few manufacturers of cells or batteries suited to this use are

Thundersky, Winston & SinoPoly

There are others which I can list but a search on those names will get you into the right area. All those brands started out as Thundersky but there have been partings of the way and expensive litigation is happening.

Winston make 12V 4 cell batteries with no access to the inter-battery connections. Their 40 Ah and 60 Ah sizes are claimed to be well suited to automotive use. Sinopoly an Winston both make single-cell batteries in sizes from around 10 Ah up to vastly-huge_Ah and you can strap these together as required. I am currently experimenting with 2 x SinoPoly 12v, 40Ah batteries each comprising 4 x 40 Ah single cells with strapping to produce a physical pack. A 12V, 40Ah 4 cell battery is about the size of a Ford-Prefect battery and less weight but is equal to about the largest LA battery you will see in a say 4 cylinder car.

HOWEVER

Claims from various battery makers overlap but are not identical, charging specs are suspect, claimed lifetimes vary depending on sales path and even reputable sellers disagree. The Winston battery charging specs use suspiciously high end point voltages - higher than I'd expect for LFP4 but below LiIon- almost like LiIon being run super conservatively to get good cycle life. Sellers warrant most brands of 10 Ah + cells for 5 years or 10 years use subject to various conditions and probably subject to use of charge/discharge management electronics. Criteria for adequate monitoring varies - many claim that Vmax and Vmin are enough as long as C rates of charge and discharge are within spec BUT a local supplier demands a gas-gauge type monitor and a low voltage cutoff. (I'd consider the gas-gauge excessive and the cutoff or similar protection essential)

Charging voltage. Since most alternators produce around 14V volts when charging it should be fine with LiFePO4 battery since maximum voltage for those is 3.6V which gives us little headroom up to 14.4V (4*3.6V=14.4V).

Some claim Winston LFP4 needs higher V than car systems provide. Others use them regardless.

SinoPoly OK on car voltages BUT ...

Individual cell balancing can be done with simple dissipative BMS which dumps excessive charge to resistor.

Winston cannot balance as sealed but claim superbly balanced cells are OK. Numerous dealers sell them and say this but at least one large site says not to use for deep discharge due to this inability. I'd be very wary. I made a Winston/SinoPoly initial choice for experience gaining based on this and strange Winston voltages and bought SinoPoly - despite apparently superior Winston specs.

But we don't have possibility to disconnect alternator when battery is charged,

Don't start. You must be able to manage your energy source - AND it is easy enough to do - but if your system insists on violating battery specs (which may not be the case) do not use LFP4.

so here is first Question: is it OK to hold LiFePO4 battery at floating charge with voltage which is close to maximum battery voltage? Or this will significantly decrease battery life?

Opinions vary
Small LFP4 - say < 1 Ah to few Ah MUST NOT be floated or they DO die early.
Some LFP4 large battery makers claim floating is OK.
Thou shalt not float LiIon or LiPo or else ... and while LFP4 has differences I'm uneasy.

Charging current. Some LiFePO4 cells can be configured so they can easily take all charge current produced by alternator. (assuming 70A output and 3 parallel 40152S cells) So second question: Can it overload alternator or modern alternators are smart enough to lower voltage to avoid being overloaded?

This deep-ends entirely on battery and alternator specs. Batteries have clear specs. Exceed them not. If your battery has NOT got good specs available do not buy it.

No over discharge protection. Currently have no idea how to deal with it besides not letting it to discharge completely.

This is essential. There are various ways of doing this but a LFP4 large Ah cell MUST NOT be taken under around 2.75V. If you cannot be SURE of that, do not use them. They cost far too much and are too light per volume to make good boat anchors.