C# – Fluent NHibernate Many-to-Many

I am using Fluent NHibernate and having some issues getting a many to many relationship setup with one of my classes. It's probably a stupid mistake but I've been stuck for a little bit trying to get it working. Anyways, I have a couple classes that have Many-Many relationships.

public class Person
{
    public Person()
    {
        GroupsOwned = new List<Groups>();
    }

    public virtual IList<Groups> GroupsOwned { get; set; }
}

public class Groups
{
    public Groups()
    {
        Admins= new List<Person>();
    }

    public virtual IList<Person> Admins{ get; set; }
}

With the mapping looking like this

Person: …

HasManyToMany<Groups>(x => x.GroupsOwned)
    .WithTableName("GroupAdministrators")
    .WithParentKeyColumn("PersonID")
    .WithChildKeyColumn("GroupID")
    .Cascade.SaveUpdate();

Groups: …

 HasManyToMany<Person>(x => x.Admins)
    .WithTableName("GroupAdministrators")
    .WithParentKeyColumn("GroupID")
    .WithChildKeyColumn("PersonID")
    .Cascade.SaveUpdate();

When I run my integration test, basically I'm creating a new person and group. Adding the Group to the Person.GroupsOwned. If I get the Person Object back from the repository, the GroupsOwned is equal to the initial group, however, when I get the group back if I check count on Group.Admins, the count is 0. The Join table has the GroupID and the PersonID saved in it.

Thanks for any advice you may have.