How do I dump the struct into the byte array without reflection?

Question

I already found encoding/binary package to deal with it, but it depended on reflect package so it didn't work with uncapitalized(that is, unexported) struct fields. However I spent a week to find that problem out, I still have a question: if struct fields should not be exported, how do I dump them easily into binary data?

EDIT: Here's the example. If you capitalize the name of fields of Data struct, that works properly. But Data struct was intended to be an abstract type, so I don't want to export these fields.

package main
import (
    "fmt"
    "encoding/binary"
    "bytes"
)

type Data struct {
    id int32
    name [16]byte
}


func main() {
    d := Data{Id: 1}
    copy(d.Name[:], []byte("tree"))
    buffer := new(bytes.Buffer)
    binary.Write(buffer, binary.LittleEndian, d)
    // d was written properly
    fmt.Println(buffer.Bytes())
    // try to read...
    buffer = bytes.NewBuffer(buffer.Bytes())
    var e = new(Data)
    err := binary.Read(buffer, binary.LittleEndian, e)
    fmt.Println(e, err)
}

Solution

Your best option would probably be to use the gob package and let your struct implement the GobDecoder and GobEncoder interfaces in order to serialize and deserialize private fields.

This would be safe, platform independent, and efficient. And you have to add those GobEncode and GobDecode functions only on structs with unexported fields, which means you don't clutter the rest of your code.

func (d *Data) GobEncode() ([]byte, error) {
    w := new(bytes.Buffer)
    encoder := gob.NewEncoder(w)
    err := encoder.Encode(d.id)
    if err!=nil {
        return nil, err
    }
    err = encoder.Encode(d.name)
    if err!=nil {
        return nil, err
    }
    return w.Bytes(), nil
}

func (d *Data) GobDecode(buf []byte) error {
    r := bytes.NewBuffer(buf)
    decoder := gob.NewDecoder(r)
    err := decoder.Decode(&d.id)
    if err!=nil {
        return err
    }
    return decoder.Decode(&d.name)
}

func main() {
    d := Data{id: 7}
    copy(d.name[:], []byte("tree"))
    buffer := new(bytes.Buffer)
    // writing
    enc := gob.NewEncoder(buffer)
    err := enc.Encode(d)
    if err != nil {
        log.Fatal("encode error:", err)
    }
    // reading
    buffer = bytes.NewBuffer(buffer.Bytes())
    e := new(Data)
    dec := gob.NewDecoder(buffer)
    err = dec.Decode(e)
    fmt.Println(e, err)
}

Comments

[Denys Séguret]

Just to be sure to understand : you want to serialize the structs in order to be able to deserialize them after ? Will your deserializer know into what struct deserialize ?

[minikomi]

Hi! Do you have any example code? And an example of the output you expect?

[ANisus]

If you are just trying to a raw binary dump, the unsafe is capable of doing what you want. It depends on needs.

[kroisse]

@dystroy First time, I tried to deserialize the struct like this: type Data struct { id int32, name [16]byte }

[kroisse]

It seems to name field was the cause for me, so I started to find how to read the array field in the struct. That's a reason why I spent lots of time.

[kroisse]

@ANisus I need the platform-independent conversion, like a network transfer. so unsafe isn't suitable for me.

[ANisus]

@kroisse: As said, it depends on the needs :) . And yes, with platform independance in mind, skip unsafe

[nos]

The traditional way to do this (mostly language agnostic), is to create a separate struct used to put stuff on the network wire from the struct you use for logic in your program. i.e. you create a separate WireData struct, and you convert from your struct Data to WireData before serializing WireData, and the oposite when you receive the data. WireData keeps only the information from Data that needs to be serialized.

[kroisse]

Great solution! I tried like this way to use binary.Read/Write manually for each fields, however this code is closer to standard. But... I should match the layout and size of serialized struct. Gob stream is attached some metadata to self-describe the data itself, so the layout was broken.

[Minty]

@dystroy, how can we approach your solution if the object contains another object and so on?

[Denys Séguret]

@Minty it's automatically recursive, so there's no problem. Just add the two functions when you have unexported fields you want to serialize or when you want to customize the serialization.

[Minty]

Thanks, I need to give it a try.

[faisal_kk]

@dystroy I'm trying write custom GobEncode/decode for struct type which has Channels and function types .Your solution helped me to move but stuck on above type.Could you please extend your advice on that.Thanks!!

[starriet]

I think this should be corrected: in main(), enc.Encode(d) -> enc.Encode(&d).