As you've noted, there are three categories that divide these up resulting in 8 different feeds:
15 vs 50 - This just tells you how many items appear in the RSS feed. Depending on how busy the group is and how often your RSS reader checks it, collecting only 15 messages could mean missing out on some if 20 had been posted. On the other hand, if the group gets very large postings a 50 item feed could overwhelm a basic RSS reader.
Messages vs Topics - Like you say, the difference between getting all messages or the first one of each thread.
RSS vs Atom - As a user there isn't really any difference between one or the other. Most of the differences only matter to someone programming the feed creation or reading process.
Best Answer
In the image included in the question, the option appears as "Display at the top".
References
Create a new topic - Groups Help