Serialization of struct

Question

Suppose i have a struct whose member values i want to send over the network to another system using winsock 2. I'm using C++ language. How do i convert it to char * keeping in mind that the struct has to be serialized before sending and also how do i deserialize the char * into struct at the other end? I found boost serialization as a suggestion to similar question but can anyone illustrate with a small code snippet for both serialization and deserialization ?

This question might seem very basic but the other answers to the related posts did not help much.

Solution 1

Following example shows a simplest way to serialize struct into char array and de-serialize it.

#include <iostream>
#include <cstring>

#define BUFSIZE 512
#define PACKETSIZE sizeof(MSG)

using namespace std;

typedef struct MSG
{
    int type;
    int priority;
    int sender;
    char message[BUFSIZE];
}MSG;

void serialize(MSG* msgPacket, char *data);
void deserialize(char *data, MSG* msgPacket);
void printMsg(MSG* msgPacket);

int main()
{
    MSG* newMsg = new MSG;
    newMsg->type = 1;
    newMsg->priority = 9;
    newMsg->sender = 2;
    strcpy(newMsg->message, "hello from server\0");
    printMsg(newMsg);

    char data[PACKETSIZE];

    serialize(newMsg, data);

    MSG* temp = new MSG;
    deserialize(data, temp);
    printMsg(temp);

    return 0;
}

void serialize(MSG* msgPacket, char *data)
{
    int *q = (int*)data;    
    *q = msgPacket->type;       q++;    
    *q = msgPacket->priority;   q++;    
    *q = msgPacket->sender;     q++;

    char *p = (char*)q;
    int i = 0;
    while (i < BUFSIZE)
    {
        *p = msgPacket->message[i];
        p++;
        i++;
    }
}

void deserialize(char *data, MSG* msgPacket)
{
    int *q = (int*)data;    
    msgPacket->type = *q;       q++;    
    msgPacket->priority = *q;   q++;    
    msgPacket->sender = *q;     q++;

    char *p = (char*)q;
    int i = 0;
    while (i < BUFSIZE)
    {
        msgPacket->message[i] = *p;
        p++;
        i++;
    }
}

void printMsg(MSG* msgPacket)
{
    cout << msgPacket->type << endl;
    cout << msgPacket->priority << endl;
    cout << msgPacket->sender << endl;
    cout << msgPacket->message << endl;
}

Solution 2

You can just do

struct MyStruct {

    int data;
    char* someNullTerminatedName; // Assuming not larger than 1023 chars

    std::ostream& serialize(std::ostream& os) const {
        char null = '\0';
        os.write((char*)&data, sizeof(data));
        os.write(someNullTerminatedName, strlen(someNullTerminatedName));
        os.write(&null, 1);
        return os;
    }
    std::istream& deserialize(std::istream& is) {
        char buffer[1024];
        int i = 0;
        is.read((char*)&data, sizeof(data));
        do { buffer[i] = is.get(); ++i; } while(buffer[i] != '\0');
        if (someNullTerminatedName != NULL) free(someNullTerminatedName);
        someNullTerminatedName = (char*)malloc(i);
        for (i = 0; buffer[i] != '\0'; ++i) {
            someNullTerminatedName[i] = buffer[i];
        }
        return is;
    }
};

It's up to you to take care of endianness and differences in size of ints and whatnot.

Example:

MyStruct foo, bar;
std::stringstream stream;
foo.serialize(stream);
// ... Now stream.str().c_str() contains a char* buffer representation of foo.
// For example it might contain [ 1f 3a 4d 10 h e l l o w o r l d \0 ]
bar.deserialize(stream);
// ... Now bar is a copy, via a serial stream of data, of foo.

If you have a socket library that exposes its interface via C++ iostreams then you don't even need the stringstream.

Solution 3

You can also have a look at Protocol Buffers from Google which is a platform/language independent library for sending data between hosts.

However, the paradigm is shifted towards writing the protocol first and then fitting your data structures into it. The advantage of this though is that it forces your software architecture to fit well with simple data types.

Solution 4

If your struct is POD you can use memcpy:

::memcpy(data, &your_struct, sizeof(YourStruct)));

and vice versa at reception:

::memcpy(&your_struct, data, sizeof(YourStruct)));

Where data is a char*. Don't forget you have to Allocate it, making sure it's big enought and delete it at the end.

Solution 5

Ok I will take the example from the boost web site as I don't understand what you can not understand from it.
I added some comments and changes to it how you can transfer via network. The network code itself is not in here. For this you can take a look at boost::asio.

int main() {
    // create and open a character archive for output
    // we simply use std::strinstream here
    std::stringstream ofs;

    // create class instance
    const gps_position g(35, 59, 24.567f);

    // save data to archive
    {
        boost::archive::text_oarchive oa(ofs);
        // write class instance to archive
        oa << g;
        // archive and stream closed when destructors are called
    }

    // now we have const char* ofs.str().c_str()
    // transfer those bytes via network
    // read them on the other machine

    gps_position newg;
    {
        // create and open an archive for input
        std::stringstream ifs(the_string_we_read_from_the_network);
        boost::archive::text_iarchive ia(ifs);
        // read class state from archive
        ia >> newg;
        // archive and stream closed when destructors are called
    }
    return 0;
}

View more solutions

Comments

[nogard]

What about serialization example on the Boost web-site?

[Vigo]

At the sending end:struct example{ int a; float b; double c; }; example ex; ex.a = 1; ex.b = 1.02; ex.c = 1.040; ::memcpy(Buffer, &ex, sizeof(example)); retval = send(conn_socket, Buffer, sizeof(Buffer), 0); And at the receiving end: example ex; retval = recvfrom(msgsock,Buffer, sizeof(Buffer), 0, (struct sockaddr *)&from, &fromlen); ::memcpy(&ex, Buffer, sizeof(example)); printf("%f", ex.c); The output: -62774385622041925000000000000000000000000000000. Why is this?

[Nick]

You need to be very careful when just mem-copying structs because different architectures and different compilers will apply padding and endianness differently.

[Vigo]

I am using Visual studio 2005 and XP on both systems. But why this difference?

[0x26res]

where does your buffer come from ? Are you sure it's bigger than your struct ?

[Vigo]

yes.. i have malloc-ed 60 bytes for my buffer whereas the the struct is easily less than 60..

[0x26res]

ok but if you malloc 60 bytes in your buffer and your buffer is a char* do you know what, I'm concern that sizeof(buffer) is 4 bytes / 8 bytes depending on your platform...

[Vigo]

now i know why it wasn't working before even though the platforms on both machines are the same.. at the receiving end, winsock command recvfrom() was allocated only 4 bytes.. i did sizeof(Buffer) instead of sizeof(packet), where Buffer was char * and packet was a struct. Hence it was receiving only the first four bytes of data.. good to know about serialization.. really helped.

[PhillyNJ]

Great example of a simple serialization

[Marian Spanik]

Your code has two memory leaks. Actually there is no need to use the new at all. Here are further possible improvements: 1. make the MSG a return type of the deserialize function instead of an output parameter 2. instead of typedef struct MSG{...}MSG; use struct MSG{...};, it's the same in C++ 3. use std::string or std::vector<byte> to hold the serialized data (and dispose of that #define PACKETSIZE altogether) and use it as a return value of the serialize function instead of the output parameter (or if you insist change the parameter to char(&data)[PACKETSIZE]).

[jackmott]

There are no memory leaks in his code. All memory is freed at the end of the main method.

[Victor Savu]

Doesn't his break the strict aliasing rule when dereferencing q in serialize?

[Lightness Races in Orbit]

Yes, this is completely invalid.

[AlphaGoku]

This assumes that the sender and receiver are aware of the members of the struct. If details of each member of the struct is also embedded and sent, the receiver could use this to frame appropriate pointers. A simple change in one end without the other end knowing about it will cause a lot of bugs

[AnthonyD973]

This is a good simple implementation, true. But, there are cases where this implementation would definitely not work: 1) if the sender and receiver do not have the same struct implementation (as @AlphaGoku pointed out), 2) if the compiler or compiler flags are different (e.g. one compiler decides to remove the padding bytes between the struct elements, or, if int is 4 bytes in one compiler and 2 bytes in the other), or, 3) even more insidious, if the sender and receiver do not have the same endianness (e.g. the sender is little-endian and the receiver is big-endian).

[AndrewBloom]

@AnthonyD973 1) could not matter if the versioning is done at protocol level. 2) padding doesn't matter cause struct members are referred directly, only the buffer is advanced. size of ints can be taken into account using types like int32_t or int64_t from <cstdint> 3) can be fixed using endianness functions like boost big_to_native_inplace or endianness macro so buffer is always big endian order for example

[AnthonyD973]

@AndrewBloom I agree with you these problems can be worked around (otherwise different machines wouldn't be able to communicate over a network, a bus or otherwise!). What I meant was that this answer only gives the code and not the logic behind it. As the code currently is, it suffers from the problems I explained.