FFmpeg concat video and audio out of sync

Joining multiple files using ffmpeg concat seems to result in a mismatch of the timestamps or offsets for the audio. I've tried with several videos and noticed the same problem for h.264 / MP4.

Using concat and encoding the video seems to work fine. The audio stays in sync as ffmpeg does the full conversion calculations and seems to get everything right.

However, simply concatenating the videos without any transformation or encoding results in a slowly increasing sync issue. Obviously, encoding the videos rather than simply joining them will result in a loss of information/quality so I would rather find a way around this problem.

I've tried several flags to sort out this problem that appears to be based on the timestamps. None of these seem to correct the problem though.

ffmpeg -f concat -fflags +genpts -async 1 -i segments.txt test.mov
ffmpeg -auto_convert 1 -f concat -fflags +genpts -async 1 -i segments.txt -c copy test2.mov
ffmpeg -f concat -i segments.txt -c copy -fflags +genpts test3.mp4
ffmpeg -f concat -fflags +genpts -async 1 -i segments.txt -copyts test4.mov
ffmpeg -f concat -i segments.txt -copyts test5.mov
ffmpeg -f concat -i segments.txt -copyts -c copy test6.mov
ffmpeg -f concat -fflags +genpts -i segments.txt -copyts -c copy test7.mov

Note: all other questions that I could find on SO seem to "fix" the problem by simply encoding the videos over again. Not a good solution.

Update

I realized the concat wasn't the problem. The original set of clips had mis-matched timestamps. Somehow concat + encoding fixed the issue, but I don't want to re-encode the videos and loose quality each time.

ffmpeg -y -ss 00:00:02.750 -i input.MOV -c copy -t 00:00:05.880 output.MOV

Which resulted in the following data

ffprobe -v quiet -show_entries stream=start_time,duration output.MOV

start_time=-0.247500
duration=6.131125
start_time=-0.257333
duration=6.155333

Since then I've tried to use -tom and -t in different places along with -af apad -c:v copy and I've still failed to get the duration to be the same.

Here is the full ffprobe output

Here is the original (red) vs the segment (green)

Detailed Sample Files

I recorded a sample video, added the commands to chop it up, then concat it. http://davidpennington.me/share/audio_sync_test_video.zip