Eliminate Duplicates of a List (Python)

I have a list of lists and need to eliminate all lists that have the same 3rd element, and save 1.

For example:

x=[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]

could turn into:

x=[[1,2,3],[2,4,6],[8,5,9]]

I've attempted a solution that eliminates all lists that have the same 2nd index using lambda, but I don't know how to save one of each element like set() does--I just get:

x=[[2,4,6]]

Answers


Python sets should still work for you:

>>> seen = set()
>>> result = []
>>> for s in x:
    e = s[2]
    if e not in seen:
        result.append(s)
        seen.add(e)


>>> result
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

You could also use a dict here, which could be a bit more flexible later:

data =[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
one_of = {el[2]:el for el in data}.values()
# [[10, 5, 9], [5, 6, 3], [2, 4, 6]]

You can use set if you define a helper class

>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> class Element(list):__hash__=lambda s:hash(s[2]);__eq__=lambda s,o:s[2]==o[2]
... 
>>> set(map(Element, L))
set([[8, 5, 9], [1, 2, 3], [2, 4, 6]])

You can do this with list comprehension and helper set:

>>> t = set()
>>> y = [l for l in x if l[2] not in t and not t.add(l[2])]
>>> y
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

The itertools docs come with a nice set of recipes, one of which, unique_everseen does exactly what you want: "List unique elements, preserving order. Remember all elements ever seen."

You can copy it paste it from the recipes, or you can pip install more-itertools and just use it as more_itertools.unique_everseen.

Or you can look at the code, and see that it's essentially just a slightly more flexible and slightly optimized version of exactly what Raymond Hettinger's answer does.*

And that's exactly what the recipes are there for—when you want to do something to an iterable, there's often a recipe for it, which not only does what you want, but is so simple that if you take a couple minutes to work out how it works, you'll know how to do it yourself in the future.


* This shouldn't surprise anyone who knows anything about Python, or just looks at the source to itertools…


>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> s = set(i[2] for i in L) 
>>> [s.remove(i[2]) or i for i in L if i[2] in s]
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

Need Your Help

Detect if a specific textbox has focus

javascript forms greasemonkey userscripts

I will first say, it has been a couple years since I have written Greasemonkey/Userscripts last and I am a bit rusty with my JavaScript, and this should be a simple question.

Accessing Azure blob storage using bash, curl

bash azure curl openssl

I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do...