Converting Stream to String and back...what are we missing?

Question

I want to serialize objects to strings, and back.

We use protobuf-net to turn an object into a Stream and back, successfully.

However, Stream to string and back... not so successful. After going through StreamToString and StringToStream, the new Streamisn't deserialized by protobuf-net; it raises an Arithmetic Operation resulted in an Overflow exception. If we deserialize the original stream, it works.

Our methods:

public static string StreamToString(Stream stream)
{
    stream.Position = 0;
    using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
    {
        return reader.ReadToEnd();
    }
}

public static Stream StringToStream(string src)
{
    byte[] byteArray = Encoding.UTF8.GetBytes(src);
    return new MemoryStream(byteArray);
}

Our example code using these two:

MemoryStream stream = new MemoryStream();
Serializer.Serialize<SuperExample>(stream, test);
stream.Position = 0;
string strout = StreamToString(stream);
MemoryStream result = (MemoryStream)StringToStream(strout);
var other = Serializer.Deserialize<SuperExample>(result);

Solution 1

This is so common but so profoundly wrong. Protobuf data is not string data. It certainly isn't ASCII. You are using the encoding backwards. A text encoding transfers:

• an arbitrary string to formatted bytes
• formatted bytes to the original string

You do not have "formatted bytes". You have arbitrary bytes. You need to use something like a base-n (commonly: base-64) encode. This transfers

• arbitrary bytes to a formatted string
• a formatted string to the original bytes

look at Convert.ToBase64String and Convert. FromBase64String

Solution 2

I have just tested this and works fine.

string test = "Testing 1-2-3";

// convert string to stream
byte[] byteArray = Encoding.ASCII.GetBytes(test);
MemoryStream stream = new MemoryStream(byteArray);

// convert stream to string
StreamReader reader = new StreamReader(stream);
string text = reader.ReadToEnd();

If stream has already been written to, you might want to seek to the beginning before first before reading out the text: stream.Seek(0, SeekOrigin.Begin);

Solution 3

a UTF8 MemoryStream to String conversion:

var res = Encoding.UTF8.GetString(stream.GetBuffer(), 0 , (int)stream.Length)

Solution 4

Try this.

string output1 = Encoding.ASCII.GetString(byteArray, 0, byteArray.Length)

Solution 5

When you testing try with UTF8 Encode stream like below

var stream = new MemoryStream();
var streamWriter = new StreamWriter(stream, System.Text.Encoding.UTF8);
Serializer.Serialize<SuperExample>(streamWriter, test);

View more solutions

Comments

[Ehsan]

shouldn't Stream be MemoryStrea?

[drzaus]

could you use a BinaryFormatter, similar to this weird example?

[drzaus]

@drzaus hm...maybe not: > "Any unpaired surrogate characters are lost in binary serialization"

[Mortennobel]

Use ToArray() instead. The buffer may larger than the size of used data. ToArray() returns a copy of the data with the correct size. var array = stream.ToArray(); var str = Encoding.UTF8.GetString(array, 0, array.Length); . See also msdn.microsoft.com/en-us/library/…

[Levi Botelho]

@Mortennobel ToArray() allocates a new array in memory and copies data over from the buffer, which may have serious implications if you're dealing with a lot of data.

[PRMan]

And don't forget a using block around StreamReader reader = new StreamReader(stream);

[Wolfgang Grinfeld]

Note the use of stream.Length, rather than stream.GetBuffer().Length. And Levi did correctly note the reason for not using ToArray().

[FTWinston]

This worked because it was tested on a string. It wouldn't work for streams containing arbitrary bytes: see Marc Gravell's answer below, regarding Convert.ToBase64