Python – extract unique elements from a 2D python list and put them into a new 2D list

listpythonunique

Right now I have a 2D list with three columns and numerous rows, each column contains a unique type of stuff. The first column is UserID, the second column is timestamp, the third column is URL. The list looks like this:

[[304070, 2015:01:01, 'http:something1'],
[304070, 2015:01:02, 'http:something2'],
[304070, 2015:01:03, 'http:something2'],
[304070, 2015:01:03, 'http:something2'],
[304071, 2015:01:04, 'http:something2'],
[304071, 2015:01:05, 'http:something3'],
[304071, 2015:01:06, 'http:something3']]

As you can see, there are some duplicate URLs, regardless of userID and timestamp.

I need to extract those rows which contain unique URLs and put them into a new 2D list.

For example, the second row, third row, forth row and fifth row all have the same URL regardless of userID and timestamp. I only need the second row (first one appears) and put it into my new 2D list. That being said, the first row has a unique URL and I will also put it into my new list. The last two rows ( sixth and seventh ) have the same URL, and I only need the sixth row.

Therefore, my new list should look like this:

[304070, 2015:01:01, 'http:something1'],
[304070, 2015:01:02, 'http:something2'],
[304071, 2015:01:05, 'http:something3']]

I thought about using something like this:

for i in range(len(oldList):
    if oldList[i][2] not in newList:
        newList.append(oldList[i])

but obviously this one does not work, becuase oldList[i][2] is an element, not in newList is checking the entire 2D list, ie, checking every row. Codes like this will just create an exact copy of oldList.

OR, I could just eliminate those rows having duplicate URLs, because using a for loop plus append operator on a 2D list with one million rows really would take a while.

Best Answer

A good way of going about this would be to use a set. Go through your list of lists one at a time, adding the URL to the set if it's not already there, and adding the full list containing that URL to your new list. If a URL is already in the set, discard the current list and move to the next one. 

old_list = [[304070, "2015:01:01", 'http:something1'],
            [304070, "2015:01:02", 'http:something2'],
            [304070, "2015:01:03", 'http:something2'],
            [304070, "2015:01:03", 'http:something2'],
            [304071, "2015:01:04", 'http:something2'],
            [304071, "2015:01:05", 'http:something3'],
            [304071, "2015:01:06", 'http:something3']]
new_list = []
url_set = set()

for item in old_list:
    if item[2] not in url_set:
        url_set.add(item[2])
        new_list.append(item)
    else:
        pass

>>> print(new_list)
[[304070, '2015:01:01', 'http:something1'], [304070, '2015:01:02', 'http:something2'], [304071, '2015:01:05', 'http:something3']]
Related Topic