Fastest way to Remove Duplicate Value from a list<> by lambda

Question

what is fastest way to remove duplicate values from a list. Assume List<long> longs = new List<long> { 1, 2, 3, 4, 3, 2, 5 }; So I am interesting in use lambda to remove duplicate and returned : {1, 2, 3, 4, 5}. What is your suggestion?

Solution 1

The easiest way to get a new list would be:

List<long> unique = longs.Distinct().ToList();

Is that good enough for you, or do you need to mutate the existing list? The latter is significantly more long-winded.

Note that Distinct() isn't guaranteed to preserve the original order, but in the current implementation it will - and that's the most natural implementation. See my Edulinq blog post about Distinct() for more information.

If you don't need it to be a List<long>, you could just keep it as:

IEnumerable<long> unique = longs.Distinct();

At this point it will go through the de-duping each time you iterate over unique though. Whether that's good or not will depend on your requirements.

Solution 2

You can use this extension method for enumerables containing more complex types:

IEnumerable<Foo> distinctList = sourceList.DistinctBy(x => x.FooName);

public static IEnumerable<TSource> DistinctBy<TSource, TKey>(
    this IEnumerable<TSource> source,
    Func<TSource, TKey> keySelector)
{
    var knownKeys = new HashSet<TKey>();
    return source.Where(element => knownKeys.Add(keySelector(element)));
}

Solution 3

There is Distinct() method. it should works.

List<long> longs = new List<long> { 1, 2, 3, 4, 3, 2, 5 };
var distinctList = longs.Distinct().ToList();

Solution 4

If you want to stick with the original List instead of creating a new one, you can something similar to what the Distinct() extension method does internally, i.e. use a HashSet to check for uniqueness:

HashSet<long> set = new HashSet<long>(longs.Count);
longs.RemoveAll(x => !set.Add(x));

The List class provides this convenient RemoveAll(predicate) method that drops all elements not satisfying the condition specified by the predicate. The predicate is a delegate taking a parameter of the list's element type and returning a bool value. The HashSet's Add() method returns true only if the set doesn't contain the item yet. Thus by removing any items from the list that can't be added to the set you effectively remove all duplicates.

Solution 5

List<long> distinctlongs = longs.Distinct().OrderBy(x => x).ToList();

View more solutions

Comments

[Saeid]

Thanks, so I think longs = longs.Distinct().ToList() is correct. right?

[Jon Skeet]

@Saeid: So long as nothing else already has a reference to the original list, that should be fine. You need to distinguish between mutating the list itself, and changing the variable to refer to a new list (which is what that code will do).

[Jeppe Stig Nielsen]

If it is important to mutate the same list, couldn't we just say: var newTmpList = longs.Distinct().ToList(); longs.Clear(); longs.AddRange(newTmpList);

[Jon Skeet]

@JeppeStigNielsen: Yes, that's a possibility - but it's not a terribly nice way of doing it...

[dyslexicanaboko]

+1 Excellent answer - the all encompassing answers are always my favorite! This is exactly what I was looking for. I love how there is always that disparity between primitives and complex types. It is almost as bad as learning a new language and only having the #(*%$()*@ useless hello world example! Okay, off my soap box, great answer!

[ZoolWay]

I also prefer this solution because it uses a lambda as the OP requested in the title (note: Linq's Distinct() does not) so it can easily be used on other datatypes without having to implement Equals/GetHashCode or an IEqualityComparer

[Nolmë Informatique]

Very elegant solution but after ? We have a sourceList and a distinctList. How can we update the dbSet to reflect changes on the Database ?

[Tscott]

This worked for me. I'm my case, I needed to update the list, so I just did the following: long = long.Distinct().ToList();

[Al-Hanash Moataz]

fantastic answer ! and u want to disticts by many keys , just call it multiple times with different key selectors :)