Remove duplicates in the list using linq

Question

I have a class Items with properties (Id, Name, Code, Price).

The List of Items is populated with duplicated items.

For ex.:

1         Item1       IT00001        $100
2         Item2       IT00002        $200
3         Item3       IT00003        $150
1         Item1       IT00001        $100
3         Item3       IT00003        $150

How to remove the duplicates in the list using linq?

Solution 1

var distinctItems = items.Distinct();

To match on only some of the properties, create a custom equality comparer, e.g.:

class DistinctItemComparer : IEqualityComparer<Item> {

    public bool Equals(Item x, Item y) {
        return x.Id == y.Id &&
            x.Name == y.Name &&
            x.Code == y.Code &&
            x.Price == y.Price;
    }

    public int GetHashCode(Item obj) {
        return obj.Id.GetHashCode() ^
            obj.Name.GetHashCode() ^
            obj.Code.GetHashCode() ^
            obj.Price.GetHashCode();
    }
}

Then use it like this:

var distinctItems = items.Distinct(new DistinctItemComparer());

Solution 2

var distinctItems = items.GroupBy(x => x.Id).Select(y => y.First());

Solution 3

If there is something that is throwing off your Distinct query, you might want to look at MoreLinq and use the DistinctBy operator and select distinct objects by id.

var distinct = items.DistinctBy( i => i.Id );

Solution 4

This is how I was able to group by with Linq. Hope it helps.

var query = collection.GroupBy(x => x.title).Select(y => y.FirstOrDefault());

Solution 5

An universal extension method:

public static class EnumerableExtensions
{
    public static IEnumerable<T> DistinctBy<T, TKey>(this IEnumerable<T> enumerable, Func<T, TKey> keySelector)
    {
        return enumerable.GroupBy(keySelector).Select(grp => grp.First());
    }
}

Example of usage:

var lstDst = lst.DistinctBy(item => item.Key);

View more solutions

Comments

[Admin]

Hi Christian , What will be the change in code if i have a List<my_Custom_Class> and List<string>. My custom class has various items in which one is DCN number and list<string> has only DCN number. So I need to check the List<Custom_Class> contains any dcn from List<string>. For example suppose List1 = List<Custom_Class> and List2 = List<String>. If List1 has 2000 items and list2 has 40000 items on which 600 items from List1 exists in List2. So in this case i need 1400 as my output List as list1. So what would be the expression. Thanks in advance

[Admin]

Also one more case is here since List1 contains various items , other items values might be different but the DCN must be same. So in my case Distinct failed to give desired out put.

[Jen]

Thanks - was looking to avoid writing a comparer class so I'm glad this works :)

[Christian Hayter]

I find comparer classes extremely useful. They can express logic other than simple property name comparisons. I wrote a new one last month, to do something that GroupBy could not.

[Adriano Carneiro]

+1 This solution even allows for a tie-breaker: eliminate duplicates with criteria!

[Amirhossein Mehrvarzi]

But a little overhead!

[atconway]

Works well and got me to learn something new and investigate the XoR operator ^ in C#. Had used in VB.NET via Xor but had to do a double take to your code to see what it was at first.

[sobelito]

@nawfal, I was suggesting FirstOrDefault() in lieu of First()

[CyberHawk]

But, as Victor Juri suggested below: use FirstorDefault. can not believe, that solution can be so simple (without custom equality comparer)

[user8128167]

This is the error I get when I try to use Distinct Comparer: "LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DataAccess.HR.Dao.CCS_LOCATION_TBL] Distinct[CCS_LOCATION_TBL](System.Linq.IQueryable1[DataAcces‌​s.HR.Dao.CCS_LOCATIO‌​N_TBL], System.Collections.Generic.IEqualityComparer`1[DataAccess.HR‌​.Dao.CCS_LOCATION_TB‌​L])' method, and this method cannot be translated into a store expression.

[user8128167]

OK had to add AsEnumerable, see stackoverflow.com/questions/19424227/…

[Roy Tinker]

If I am correct, using FirstOrDefault here offers no benefit if the Select immediately follows GroupBy, since there's no possibility of there being an empty group (the groups were just derived from the collection's contents)

[Fereydoon Barikzehy]

There is no DistinctBy() method with Linq.

[Ademar]

@FereydoonBarikzehy But he is not talking about pure Linq. In post is linq to MoreLinq project ...

[Nuri YILMAZ]

I should notice that default comparer works if collection member types is one of value types. But which default equality comparer select by csc for reference types. Reference types must have own comparer(s).

[Sumit Joshi]

You can group with multiple properties : List<XYZ> MyUniqueList = MyList.GroupBy(x => new { x.Column1, x.Column2 }).Select(g=> g.First()).ToList();

[Steven Ryssaert]

Very clean approach

[Kefka]

@SumitJoshi thanks this solution worked when comparing two observable collections of my custom type as well!

[Suncat2000]

LINQ to Entities doesn't support methods that use IEqualityComparer. Supported and Unsupported LINQ Methods (LINQ to Entities).

[Kellen Stuart]

Stuff like this is what makes me love C#

[Christian Hayter]

@KolobCanyon It is used in hash table lookups. There is a good write-up at stackoverflow.com/a/4096774/115413

[mig_08]

Thanks this is exactly what I was looking for, works well.