How to remove duplicates from a string in SQL

Question

I got a column in my SQL table where some values are seperated bij ". Some values in this string are duplicated which I want to remove. Here is an example of my data:

---------------
| qw"qw"er"er |
---------------
| q"w"w"q     |
---------------
| f"k"s"g     |
---------------

Now The result should replace any duplicates:

---------------
| qw"er       |
---------------
| q"w"        |
---------------
| f"k"s"g     |
---------------

So first I want to split the string and then remove duplicates. Could anyone help me with this issue?

Solution

Option 1 with a Parse Function

Declare @YourTable table (ID int,YourCol varchar(50))
Insert Into @YourTable values
(1,'qw"qw"er"er'),
(2,'q"w"w"q'),
(3,'f"k"s"g')

Select A.ID
      ,A.YourCol
      ,DeDuped   = Stuff((Select '"' + RetVal 
                           From (Select RetSeq=Min(RetSeq),RetVal 
                                  From  [dbo].[udf-Str-Parse](A.YourCol,'"') 
                                  Group By RetVal) P  
                            Order by 1 For XML Path('')),1,1,'') 
 From  @YourTable A

Returns

ID  YourCol      DeDuped
1   qw"qw"er"er  qw"er
2   q"w"w"q      q"w
3   f"k"s"g      f"k"s"g

Option 2: Without a Parse Function

Declare @YourTable table (ID int,YourCol varchar(50))
Insert Into @YourTable values
(1,'qw"qw"er"er'),
(2,'q"w"w"q'),
(3,'f"k"s"g')

Select A.ID
      ,A.YourCol
      ,DeDuped   = Stuff((Select '"' + RetVal 
                           From (Select RetSeq=Min(RetSeq),RetVal 
                                  From  (
                                            Select RetSeq = Row_Number() over (Order By (Select null))
                                                  ,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
                                            From  (Select x = Cast('<x>' + replace((Select replace(A.YourCol,'"','§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A 
                                            Cross Apply x.nodes('x') AS B(i)
                                        ) P1
                                  Group By RetVal) P  
                            Order by RetSeq 
                            For XML Path('')),1,1,'') 
 From  @YourTable A

Returns

ID  YourCol      DeDuped
1   qw"qw"er"er  qw"er
2   q"w"w"q      q"w
3   f"k"s"g      f"k"s"g

The UDF if Interested

CREATE FUNCTION [dbo].[udf-Str-Parse] (@String varchar(max),@Delimiter varchar(10))
Returns Table 
As
Return (  
    Select RetSeq = Row_Number() over (Order By (Select null))
          ,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
    From  (Select x = Cast('<x>' + replace((Select replace(@String,@Delimiter,'§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A 
    Cross Apply x.nodes('x') AS B(i)
);
--Thanks Shnugo for making this XML safe
--Select * from [dbo].[udf-Str-Parse]('Dog,Cat,House,Car',',')
--Select * from [dbo].[udf-Str-Parse]('John Cappelletti was here',' ')
--Select * from [dbo].[udf-Str-Parse]('this,is,<test>,for,< & >',',')
--Performance On a 5,000 random sample -8K 77.8ms, -1M 79ms (+1.16), -- 91.66ms (+13.8)

Comments

[Hassan Voyeau]

Can this answer be updated with a shorter/easier answer? Or is it still this involved?

[John Cappelletti]

@HassanVoyeau If you are on 2016+ you can use string_split() Take a peek at dbfiddle.uk/…

[John Cappelletti]

@HassanVoyeau The Parse UDF is perhaps the most concise version avail prior to 2016

[John Cappelletti]

@HassanVoyeau If 2017+ you could replace the XML STUFF portion with STRING_AGG()

[John Cappelletti]

@HassanVoyeau One last point. The XML Parse function is VERY performant. On a sample size of 15,000 records it was 13ms slower than the tally approach, and it is not limited to 10K bytes