Java – JList – deselect when clicking an already selected item

If a selected index on a JList is clicked, I want it to de-select. In other words, clicking on the indices actually toggles their selection. Didn't look like this was supported, so I tried

list.addMouseListener(new MouseAdapter()
{
   public void mousePressed(MouseEvent evt)
   {
      java.awt.Point point = evt.getPoint();
      int index = list.locationToIndex(point);
      if (list.isSelectedIndex(index))
         list.removeSelectionInterval(index, index);
   }
});

The problem here is that this is being invoked after JList has already acted on the mouse event, so it deselects everything. So then I tried removing all of JList's MouseListeners, adding my own, and then adding all of the default listeners back. That didn't work, since JList would reselect the index after I had deselected it. Anyway, what I eventually came up with is

MouseListener[] mls = list.getMouseListeners();
for (MouseListener ml : mls)
   list.removeMouseListener(ml);
list.addMouseListener(new MouseAdapter()
{
   public void mousePressed(MouseEvent evt)
   {
      java.awt.Point point = evt.getPoint();
      final int index = list.locationToIndex(point);
      if (list.isSelectedIndex(index))
         SwingUtilities.invokeLater(new Runnable()
         {
            public void run()
            {
               list.removeSelectionInterval(index, index);
            }
         });
   }
});
for (MouseListener ml : mls)
   list.addMouseListener(ml);

… and that works. But I don't like it. Is there a better way?