Python – Get unique values from pandas series of lists

pandaspie-chartpythonsetunique

I have a column in DataFrame containing list of categories. For example:

0                                                    [Pizza]
1                                 [Mexican, Bars, Nightlife]
2                                  [American, New, Barbeque]
3                                                     [Thai]
4          [Desserts, Asian, Fusion, Mexican, Hawaiian, F...
6                                           [Thai, Barbeque]
7                           [Asian, Fusion, Korean, Mexican]
8          [Barbeque, Bars, Pubs, American, Traditional, ...
9                       [Diners, Burgers, Breakfast, Brunch]
11                                [Pakistani, Halal, Indian]

I am attempting to do two things:

1) Get unique categories – My approach is have a empty set, iterate through series and append each list.

my code:

unique_categories = {'Pizza'}
for lst in restaurant_review_df['categories_arr']:
    unique_categories = unique_categories | set(lst)

This give me a set of unique categories contained in all the lists in the column.

2) Generate pie plot of category counts and each restaurant can belong to multiple categories. For example: restaurant 11 belongs to Pakistani, Indian and Halal categories. My approach is again iterate through categories and one more iteration through series to get counts.

Are there simpler or elegant ways of doing this?

Thanks in advance.

Best Answer

Update using pandas 0.25.0+ with `explode`

df['category'].explode().value_counts()

Output:

Barbeque       3
Mexican        3
Fusion         2
Thai           2
American       2
Bars           2
Asian          2
Hawaiian       1
New            1
Brunch         1
Pizza          1
Traditional    1
Pubs           1
Korean         1
Pakistani      1
Burgers        1
Diners         1
Indian         1
Desserts       1
Halal          1
Nightlife      1
Breakfast      1
Name: Places, dtype: int64

And with plotting:

df['category'].explode().value_counts().plot.pie(figsize=(8,8))

Output:

For older verions of pandas before 0.25.0 Try:

df['category'].apply(pd.Series).stack().value_counts()

Output:

Mexican        3
Barbeque       3
Thai           2
Fusion         2
American       2
Bars           2
Asian          2
Pubs           1
Burgers        1
Traditional    1
Brunch         1
Indian         1
Korean         1
Halal          1
Pakistani      1
Hawaiian       1
Diners         1
Pizza          1
Nightlife      1
New            1
Desserts       1
Breakfast      1
dtype: int64

With plotting:

df['category'].apply(pd.Series).stack().value_counts().plot.pie()

Output:

Per @coldspeed's comments

from itertools import chain
from collections import Counter

pd.DataFrame.from_dict(Counter(chain(*df['category'])), orient='index').sort_values(0, ascending=False)

Output:

Barbeque     3
Mexican      3
Bars         2
American     2
Thai         2
Asian        2
Fusion       2
Pizza        1
Diners       1
Halal        1
Pakistani    1
Brunch       1
Breakfast    1
Burgers      1
Hawaiian     1
Traditional  1
Pubs         1
Korean       1
Desserts     1
New          1
Nightlife    1
Indian       1

Related Solutions

Javascript – Get all unique values in a JavaScript array (remove duplicates)

With JavaScript 1.6 / ECMAScript 5 you can use the native filter method of an Array in the following way to get an array with unique values:

function onlyUnique(value, index, self) {
  return self.indexOf(value) === index;
}

// usage example:
var a = ['a', 1, 'a', 2, '1'];
var unique = a.filter(onlyUnique);

console.log(unique); // ['a', 1, 2, '1']

The native method filter will loop through the array and leave only those entries that pass the given callback function onlyUnique.

onlyUnique checks, if the given value is the first occurring. If not, it must be a duplicate and will not be copied.

This solution works without any extra library like jQuery or prototype.js.

It works for arrays with mixed value types too.

For old Browsers (<ie9), that do not support the native methods filter and indexOf you can find work arounds in the MDN documentation for filter and indexOf.

If you want to keep the last occurrence of a value, simple replace indexOf by lastIndexOf.

With ES6 it could be shorten to this:

// usage example:
var myArray = ['a', 1, 'a', 2, '1'];
var unique = myArray.filter((v, i, a) => a.indexOf(v) === i);

console.log(unique); // unique is ['a', 1, 2, '1']

Thanks to Camilo Martin for hint in comment.

ES6 has a native object Set to store unique values. To get an array with unique values you could now do this:

var myArray = ['a', 1, 'a', 2, '1'];

let unique = [...new Set(myArray)];

console.log(unique); // unique is ['a', 1, 2, '1']

The constructor of Set takes an iterable object, like Array, and the spread operator ... transform the set back into an Array. Thanks to Lukas Liese for hint in comment.

multi array

var a = [[2,4],[2,3],[1,3],[1,3],[1,8]]

var unique = (value, index, self) =>{
  var findIndex = (element) => element[0] == value[0];
  return self.findIndex(findIndex) === index;
}

console.log(a.filter(unique))

Python – Get difference between two lists

To get elements which are in temp1 but not in temp2 :

In [5]: list(set(temp1) - set(temp2))
Out[5]: ['Four', 'Three']

Beware that it is asymmetric :

In [5]: set([1, 2]) - set([2, 3])
Out[5]: set([1])

where you might expect/want it to equal set([1, 3]). If you do want set([1, 3]) as your answer, you can use set([1, 2]).symmetric_difference(set([2, 3])).

Best Answer

Update using pandas 0.25.0+ with explode

Related Solutions

Javascript – Get all unique values in a JavaScript array (remove duplicates)

Python – Get difference between two lists

Related Topic

Update using pandas 0.25.0+ with `explode`