How do I convert a string to and from JSON with escaped/special characters using DBXJSON?

Question

I'm having trouble converting a string with escaped characters to and from a TJsonString. (I'm using Delphi XE 2, Update 4, Hotfix 1).

NOTE: I am familiar with the SuperObject, but my requirements are to use the DBXJSON unit.

It looks like the TJSONString is not correctly escaped when returning the JSON representation via the ToString() method.

What (if anything) am I doing wrong and how do I correctly convert a string with special characters to/from its correct JSON representation?

Perhaps I missed something, but none of the following Q&As seemed to address this directly:

• Delphi decode json/utf8 escaped text
• Delphi JSON library for XE2 available for object serialization
• Delphi: JSON array
• How to parse nested JSON object in Delphi XE2?
• Nested json object deserializing using Delphi 2012

EDIT:

As it turns out, the examples below were indeed working as expected.

What wasn't clear to me was that when creating a TJSONString via it's constructor and adding it to a TJSONObject, the ToString() method will return an escaped representation. However, after parsing a TJSONObject, the ToString() method will returned the un-escaped representation.

The only other caveat was that the EscapeString() function in the sample code below was handling the double-quote. Although I wasn't using the double quote here, some of my other code was, and that caused the parsing to fail because TJSONString already escapes that character. I've updated my sample code to remove this handling from the EscapeString() function, which is what I've been using in my own classes.

Thanks again to @Linas for the answer, which helped me to "get" it.

Raw String Value:

Text := 'c:\path\name' +#13 + #10 + 'Next Line';

Text: c:\path\name
Next Line

What DBXJSON produces (NO ESCAPES):

JsonString: "c:\path\name
Next Line"

JsonPair: "MyString":"c:\path\name
Next Line"

JsonObject: {"MyString":"c:\path\name
Next Line"}

Parsing UN-escaped Text FAILS:

Text to parse: {"MyString":"c:\path\name
Next Line"}

Parsed JsonObject = *NIL*

What I EXPECT DBXJSON to produce:

Escaped String: c:\\path\\name\r\nNext Line

JsonString: "c:\\path\\name\r\nNext Line"

JsonPair: "MyString":"c:\\path\\name\r\nNext Line"

JsonObject: {"MyString":"c:\\path\\name\r\nNext Line"}

Parsing ESCAPED Text (INVALID) (Text to parse validated with JSONLint):

Text to parse: {"MyString":"c:\\path\\name\r\nNext Line"}

Parsed JsonObject.ToString(): {"MyString":"c:\path\name
Next Line"}

I've noticed that the only special character TJSONString seems to process correctly is the double quote (").

Here is the code I'm using:

program JsonTest;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils, DbxJson;

function EscapeString(const AValue: string): string;
const
  ESCAPE = '\';
  // QUOTATION_MARK = '"';
  REVERSE_SOLIDUS = '\';
  SOLIDUS = '/';
  BACKSPACE = #8;
  FORM_FEED = #12;
  NEW_LINE = #10;
  CARRIAGE_RETURN = #13;
  HORIZONTAL_TAB = #9;
var
  AChar: Char;
begin
  Result := '';
  for AChar in AValue do
  begin
    case AChar of
      // !! Double quote (") is handled by TJSONString
      // QUOTATION_MARK: Result := Result + ESCAPE + QUOTATION_MARK;
      REVERSE_SOLIDUS: Result := Result + ESCAPE + REVERSE_SOLIDUS;
      SOLIDUS: Result := Result + ESCAPE + SOLIDUS;
      BACKSPACE: Result := Result + ESCAPE + 'b';
      FORM_FEED: Result := Result + ESCAPE + 'f';
      NEW_LINE: Result := Result + ESCAPE + 'n';
      CARRIAGE_RETURN: Result := Result + ESCAPE + 'r';
      HORIZONTAL_TAB: Result := Result + ESCAPE + 't';
      else
      begin
        if (Integer(AChar) < 32) or (Integer(AChar) > 126) then
          Result := Result + ESCAPE + 'u' + IntToHex(Integer(AChar), 4)
        else
          Result := Result + AChar;
      end;
    end;
  end;
end;

procedure Test;
var
  Text: string;
  JsonString: TJsonString;
  JsonPair: TJsonPair;
  JsonObject: TJsonObject;
begin
  try
    Writeln('Raw String Value');
    Writeln('-----------------');
    Text := 'c:\path\name' +#13 + #10 + 'Next Line';
    Writeln('Text: ', Text);
    JsonString := TJsonString.Create(Text);
    JsonPair := TJsonPair.Create('MyString', JsonString);
    JsonObject := TJsonObject.Create(JsonPair);
    // DBXJSON results
    Writeln;
    Writeln('What DBXJSON produces');
    Writeln('---------------------');
    Writeln('JsonString: ', JsonString.ToString);
    Writeln;
    Writeln('JsonPair: ', JsonPair.ToString);
    Writeln;
    Writeln('JsonObject: ', JsonObject.ToString);
    Writeln;

    // assign JSON representation
    Text := JsonObject.ToString;
    // free json object
    JsonObject.Free;
    // parse it
    JsonObject:= TJsonObject.ParseJsonValue(TEncoding.ASCII.GetBytes(
      Text), 0) as TJsonObject;
    Writeln('Parsing UN-escaped Text *FAILS* ');
    Writeln('----------------------------------');
    Writeln('Text to parse: ', Text);
    Writeln;
    if (JsonObject = nil) then
      Writeln('Parsed JsonObject = *NIL*')
    else
      Writeln('Parsed JsonObject: ', JsonObject.ToString);
    Writeln;
    // free json object
    JsonObject.Free;
    // expected results
    Text := 'c:\path\name' +#13 + #10 + 'Next Line';
    Text := EscapeString(Text);
    JsonString := TJsonString.Create(Text);
    JsonPair := TJsonPair.Create('MyString', JsonString);
    JsonObject := TJsonObject.Create(JsonPair);
    Writeln('What I *EXPECT* DBXJSON to produce');
    Writeln('----------------------------------');
    Writeln('Escaped String: ', Text);
    Writeln;
    Writeln('JsonString: ', JsonString.ToString);
    Writeln;
    Writeln('JsonPair: ', JsonPair.ToString);
    Writeln;
    Writeln('JsonObject: ', JsonObject.ToString);
    Writeln;
    // assign JSON representation
    Text := JsonObject.ToString;
    // free json object
    JsonObject.Free;
    // parse it
    JsonObject:= TJsonObject.ParseJsonValue(TEncoding.ASCII.GetBytes(
      Text), 0) as TJsonObject;
    Writeln('Parsing ESCAPED Text (*INVALID*) ');
    Writeln('----------------------------------');
    Writeln('Text to parse: ', Text);
    Writeln;
    Writeln('Parsed JsonObject.ToString(): ', JsonObject.ToString);
    Writeln;
    Readln;
  except
    on E: Exception do
    begin
      Writeln(E.ClassName, ': ', E.Message);
      Readln;
    end;
  end;
end;

begin
  Test;
end.

Solution 1

You can try to define your own TJSONString type and escape json strings there. E.g.:

uses
  DBXJSON;

type
  TSvJsonString = class(TJSONString)
  private
    function EscapeValue(const AValue: string): string;
  public
    constructor Create(const AValue: string); overload;
  end;

{ TSvJsonString }

constructor TSvJsonString.Create(const AValue: string);
begin
  inherited Create(EscapeValue(AValue));
end;

function TSvJsonString.EscapeValue(const AValue: string): string;

  procedure AddChars(const AChars: string; var Dest: string; var AIndex: Integer); inline;
  begin
    System.Insert(AChars, Dest, AIndex);
    System.Delete(Dest, AIndex + 2, 1);
    Inc(AIndex, 2);
  end;

  procedure AddUnicodeChars(const AChars: string; var Dest: string; var AIndex: Integer); inline;
  begin
    System.Insert(AChars, Dest, AIndex);
    System.Delete(Dest, AIndex + 6, 1);
    Inc(AIndex, 6);
  end;

var
  i, ix: Integer;
  AChar: Char;
begin
  Result := AValue;
  ix := 1;
  for i := 1 to System.Length(AValue) do
  begin
    AChar :=  AValue[i];
    case AChar of
      '/', '\', '"':
      begin
        System.Insert('\', Result, ix);
        Inc(ix, 2);
      end;
      #8:  //backspace \b
      begin
        AddChars('\b', Result, ix);
      end;
      #9:
      begin
        AddChars('\t', Result, ix);
      end;
      #10:
      begin
        AddChars('\n', Result, ix);
      end;
      #12:
      begin
        AddChars('\f', Result, ix);
      end;
      #13:
      begin
        AddChars('\r', Result, ix);
      end;
      #0 .. #7, #11, #14 .. #31:
      begin
        AddUnicodeChars('\u' + IntToHex(Word(AChar), 4), Result, ix);
      end
      else
      begin
        if Word(AChar) > 127 then
        begin
          AddUnicodeChars('\u' + IntToHex(Word(AChar), 4), Result, ix);
        end
        else
        begin
          Inc(ix);
        end;
      end;
    end;
  end;
end;

Usage example:

procedure Test;
var
  LText, LEscapedText: string;
  LJsonString: TSvJsonString;
  LJsonPair: TJsonPair;
  LJsonObject: TJsonObject;
begin
  LText := 'c:\path\name' + #13 + #10 + 'Next Line';
  LJsonString := TSvJsonString.Create(LText);
  LJsonPair := TJsonPair.Create('MyString', LJsonString);
  LJsonObject := TJsonObject.Create(LJsonPair);
  try
    LEscapedText := LJsonObject.ToString;
    //LEscapedText is: c:\\path\\name\r\nNext Line
  finally
    LJsonObject.Free;
  end;
end;

And this is how parsing should be done:

//AText := '{"MyString":"c:\\path\\name\r\nNext Line"}';
function Parse(const AText: string): string;
var
  obj: TJSONValue;
begin
  obj := TJSONObject.ParseJSONValue(AText);
  try
    Result := obj.ToString;
    //Result := {"MyString":"c:\path\name
   //Next Line"}
  finally
    obj.Free;
  end;
end;

Solution 2

We just ran into this lovely problem, where our backslashes weren't being escaped (but our double quotes were, apparently lol)...

The solution was to stop using TJSONObject.ToString(), and use TJSONObject.ToJSON() instead. This gives you the correctly escaped string, as opposed to the human readable format that ToString() returns.

Hope this helps someone :)

Comments

[Doug]

Thanks @Linas, but I'm able to do the escaping myself as per my sample code. The other problem is that TJsonObject doesn't seem to be able to parse the JSON string correctly when it contains escaped strings. I'm hoping that it's still me that's doing something wrong. :)

[Linas]

@Doug Just tested with Delphi XE and everything works as expected. You are probably parsing json string in a wrong way. See my answer edit how it should be done.

[Doug]

Thanks @Linas. After examining my code in more detail, I found that my escape routine was handling the double quote (\"). The problem is that the TJSONString already handles this case, and ended up adding an extra escape (\\"), which caused the parsing to fail. After removing this case, everything worked fine. I've updated my example to fix this.