How to remove duplicates from a list?

Question

I want to remove duplicates from a list but what I am doing is not working:

List<Customer> listCustomer = new ArrayList<Customer>();    
for (Customer customer: tmpListCustomer)
{
  if (!listCustomer.contains(customer)) 
  {
    listCustomer.add(customer);
  }
 }

Solution 1

If the code in your question doesn't work, you probably have not implemented equals(Object) on the Customer class appropriately.

Presumably there is some key (let us call it customerId) that uniquely identifies a customer; e.g.

class Customer {
    private String customerId;
    ...

An appropriate definition of equals(Object) would look like this:

    public boolean equals(Object obj) {
        if (obj == this) {
            return true;
        }
        if (!(obj instanceof Customer)) {
            return false;
        }
        Customer other = (Customer) obj;
        return this.customerId.equals(other.customerId);
    }

For completeness, you should also implement hashCode so that two Customer objects that are equal will return the same hash value. A matching hashCode for the above definition of equals would be:

    public int hashCode() {
        return customerId.hashCode();
    }

It is also worth noting that this is not an efficient way to remove duplicates if the list is large. (For a list with N customers, you will need to perform N*(N-1)/2 comparisons in the worst case; i.e. when there are no duplicates.) For a more efficient solution you could use a HashSet to do the duplicate checking. Another option would be to use a LinkedHashSet as explained in Tom Hawtin's answer.

Solution 2

Assuming you want to keep the current order and don't want a Set, perhaps the easiest is:

List<Customer> depdupeCustomers =
    new ArrayList<>(new LinkedHashSet<>(customers));

If you want to change the original list:

Set<Customer> depdupeCustomers = new LinkedHashSet<>(customers);
customers.clear();
customers.addAll(dedupeCustomers);

Solution 3

java 8 update
you can use stream of array as below:

Arrays.stream(yourArray).distinct()
                    .collect(Collectors.toList());

Solution 4

List → Set → List (distinct)

Just add all your elements to a Set: it does not allow it's elements to be repeated. If you need a list afterwards, use new ArrayList(theSet) constructor afterwards (where theSet is your resulting set).

Solution 5

Does Customer implement the equals() contract?

If it doesn't implement equals() and hashCode(), then listCustomer.contains(customer) will check to see if the exact same instance already exists in the list (By instance I mean the exact same object--memory address, etc). If what you are looking for is to test whether or not the same Customer( perhaps it's the same customer if they have the same customer name, or customer number) is in the list already, then you would need to override equals() to ensure that it checks whether or not the relevant fields(e.g. customer names) match.

Note: Don't forget to override hashCode() if you are going to override equals()! Otherwise, you might get trouble with your HashMaps and other data structures. For a good coverage of why this is and what pitfalls to avoid, consider having a look at Josh Bloch's Effective Java chapters on equals() and hashCode() (The link only contains iformation about why you must implement hashCode() when you implement equals(), but there is good coverage about how to override equals() too).

By the way, is there an ordering restriction on your set? If there isn't, a slightly easier way to solve this problem is use a Set<Customer> like so:

Set<Customer> noDups = new HashSet<Customer>();
noDups.addAll(tmpListCustomer);
return new ArrayList<Customer>(noDups);

Which will nicely remove duplicates for you, since Sets don't allow duplicates. However, this will lose any ordering that was applied to tmpListCustomer, since HashSet has no explicit ordering (You can get around that by using a TreeSet, but that's not exactly related to your question). This can simplify your code a little bit.

View more solutions