Encode/Decode Array of Types conforming to protocol with JSONEncoder

Question

I'm trying to find the best way to Encode/Decode an array of structs conforming to a swift protocol using the new JSONDecoder/Encoder in Swift 4.

I made up a little example to illustrate the problem:

First we have a protocol Tag and some Types that conform to this protocol.

protocol Tag: Codable {
    var type: String { get }
    var value: String { get }
}

struct AuthorTag: Tag {
    let type = "author"
    let value: String
}

struct GenreTag: Tag {
    let type = "genre"
    let value: String
}

Then we have a Type Article which has an Array of Tags.

struct Article: Codable {
    let tags: [Tag]
    let title: String
}

Finally we encode or decode the Article

let article = Article(tags: [AuthorTag(value: "Author Tag Value"), GenreTag(value:"Genre Tag Value")], title: "Article Title")


let jsonEncoder = JSONEncoder()
let jsonData = try jsonEncoder.encode(article)
let jsonString = String(data: jsonData, encoding: .utf8)

And this is the JSON structure that I like to have.

{
 "title": "Article Title",
 "tags": [
     {
       "type": "author",
       "value": "Author Tag Value"
     },
     {
       "type": "genre",
       "value": "Genre Tag Value"
     }
 ]
}

The problem is that at some point I have to switch on the type property to decode the Array but to Decode the Array I have to know its type.

EDIT:

It's clear to me why Decodable can not work out of the box but at least Encodable should work. The following modified Article struct compiles but crashes with the following error message.

fatal error: Array<Tag> does not conform to Encodable because Tag does not conform to Encodable.: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-900.0.43/src/swift/stdlib/public/core/Codable.swift, line 3280

struct Article: Encodable {
    let tags: [Tag]
    let title: String

    enum CodingKeys: String, CodingKey {
        case tags
        case title
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(tags, forKey: .tags)
        try container.encode(title, forKey: .title)
    }
}

let article = Article(tags: [AuthorTag(value: "Author Tag"), GenreTag(value:"A Genre Tag")], title: "A Title")

let jsonEncoder = JSONEncoder()
let jsonData = try jsonEncoder.encode(article)
let jsonString = String(data: jsonData, encoding: .utf8)

And this is the relevant part from Codeable.swift

guard Element.self is Encodable.Type else {
    preconditionFailure("\(type(of: self)) does not conform to Encodable because \(Element.self) does not conform to Encodable.")
}

Source: https://github.com/apple/swift/blob/master/stdlib/public/core/Codable.swift

Solution 1

The reason why your first example doesn't compile (and your second crashes) is because protocols don't conform to themselves – Tag is not a type that conforms to Codable, therefore neither is [Tag]. Therefore Article doesn't get an auto-generated Codable conformance, as not all of its properties conform to Codable.

Encoding and decoding only the properties listed in the protocol

If you just want to encode and decode the properties listed in the protocol, one solution would be to simply use an AnyTag type-eraser that just holds those properties, and can then provide the Codable conformance.

You can then have Article hold an array of this type-erased wrapper, rather than of Tag:

struct AnyTag : Tag, Codable {

    let type: String
    let value: String

    init(_ base: Tag) {
        self.type = base.type
        self.value = base.value
    }
}

struct Article: Codable {
    let tags: [AnyTag]
    let title: String
}

let tags: [Tag] = [
    AuthorTag(value: "Author Tag Value"),
    GenreTag(value:"Genre Tag Value")
]

let article = Article(tags: tags.map(AnyTag.init), title: "Article Title")

let jsonEncoder = JSONEncoder()
jsonEncoder.outputFormatting = .prettyPrinted

let jsonData = try jsonEncoder.encode(article)

if let jsonString = String(data: jsonData, encoding: .utf8) {
    print(jsonString)
}

Which outputs the following JSON string:

{
  "title" : "Article Title",
  "tags" : [
    {
      "type" : "author",
      "value" : "Author Tag Value"
    },
    {
      "type" : "genre",
      "value" : "Genre Tag Value"
    }
  ]
}

and can be decoded like so:

let decoded = try JSONDecoder().decode(Article.self, from: jsonData)

print(decoded)

// Article(tags: [
//                 AnyTag(type: "author", value: "Author Tag Value"),
//                 AnyTag(type: "genre", value: "Genre Tag Value")
//               ], title: "Article Title")

---

Encoding and decoding all properties of the conforming type

If however you need to encode and decoded every property of the given Tag conforming type, you'll likely want to store the type information in the JSON somehow.

I would use an enum in order to do this:

enum TagType : String, Codable {

    // be careful not to rename these – the encoding/decoding relies on the string
    // values of the cases. If you want the decoding to be reliant on case
    // position rather than name, then you can change to enum TagType : Int.
    // (the advantage of the String rawValue is that the JSON is more readable)
    case author, genre

    var metatype: Tag.Type {
        switch self {
        case .author:
            return AuthorTag.self
        case .genre:
            return GenreTag.self
        }
    }
}

Which is better than just using plain strings to represent the types, as the compiler can check that we've provided a metatype for each case.

Then you just have to change the Tag protocol such that it requires conforming types to implement a static property that describes their type:

protocol Tag : Codable {
    static var type: TagType { get }
    var value: String { get }
}

struct AuthorTag : Tag {

    static var type = TagType.author
    let value: String

    var foo: Float
}

struct GenreTag : Tag {

    static var type = TagType.genre
    let value: String

    var baz: String
}

Then we need to adapt the implementation of the type-erased wrapper in order to encode and decode the TagType along with the base Tag:

struct AnyTag : Codable {

    var base: Tag

    init(_ base: Tag) {
        self.base = base
    }

    private enum CodingKeys : CodingKey {
        case type, base
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        let type = try container.decode(TagType.self, forKey: .type)
        self.base = try type.metatype.init(from: container.superDecoder(forKey: .base))
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)

        try container.encode(type(of: base).type, forKey: .type)
        try base.encode(to: container.superEncoder(forKey: .base))
    }
}

We're using a super encoder/decoder in order to ensure that the property keys for the given conforming type don't conflict with the key used to encode the type. For example, the encoded JSON will look like this:

{
  "type" : "author",
  "base" : {
    "value" : "Author Tag Value",
    "foo" : 56.7
  }
}

If however you know there won't be a conflict, and want the properties to be encoded/decoded at the same level as the "type" key, such that the JSON looks like this:

{
  "type" : "author",
  "value" : "Author Tag Value",
  "foo" : 56.7
}

You can pass decoder instead of container.superDecoder(forKey: .base) & encoder instead of container.superEncoder(forKey: .base) in the above code.

As an optional step, we could then customise the Codable implementation of Article such that rather than relying on an auto-generated conformance with the tags property being of type [AnyTag], we can provide our own implementation that boxes up a [Tag] into an [AnyTag] before encoding, and then unbox for decoding:

struct Article {

    let tags: [Tag]
    let title: String

    init(tags: [Tag], title: String) {
        self.tags = tags
        self.title = title
    }
}

extension Article : Codable {

    private enum CodingKeys : CodingKey {
        case tags, title
    }

    init(from decoder: Decoder) throws {

        let container = try decoder.container(keyedBy: CodingKeys.self)

        self.tags = try container.decode([AnyTag].self, forKey: .tags).map { $0.base }
        self.title = try container.decode(String.self, forKey: .title)
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)

        try container.encode(tags.map(AnyTag.init), forKey: .tags)
        try container.encode(title, forKey: .title)
    }
}

This then allows us to have the tags property be of type [Tag], rather than [AnyTag].

Now we can encode and decode any Tag conforming type that's listed in our TagType enum:

let tags: [Tag] = [
    AuthorTag(value: "Author Tag Value", foo: 56.7),
    GenreTag(value:"Genre Tag Value", baz: "hello world")
]

let article = Article(tags: tags, title: "Article Title")

let jsonEncoder = JSONEncoder()
jsonEncoder.outputFormatting = .prettyPrinted

let jsonData = try jsonEncoder.encode(article)

if let jsonString = String(data: jsonData, encoding: .utf8) {
    print(jsonString)
}

Which outputs the JSON string:

{
  "title" : "Article Title",
  "tags" : [
    {
      "type" : "author",
      "base" : {
        "value" : "Author Tag Value",
        "foo" : 56.7
      }
    },
    {
      "type" : "genre",
      "base" : {
        "value" : "Genre Tag Value",
        "baz" : "hello world"
      }
    }
  ]
}

and can then be decoded like so:

let decoded = try JSONDecoder().decode(Article.self, from: jsonData)

print(decoded)

// Article(tags: [
//                 AuthorTag(value: "Author Tag Value", foo: 56.7000008),
//                 GenreTag(value: "Genre Tag Value", baz: "hello world")
//               ],
//         title: "Article Title")

Solution 2

Inspired by @Hamish answer. I found his approach reasonable, however few things might be improved:

1. Mapping array [Tag] to and from [AnyTag] in Article leave us without auto-generated Codable conformance
2. It's not possible to have same code for coding/encoding array of base class, since static var type can't be overridden in subclass. (for example if Tag would be super class of AuthorTag & GenreTag)
3. Most importantly this code can't be reused for another Type, you required to create new AnyAnotherType wrapper and it's internal coding/encoding.

I made slightly different solution, instead of wrapping each element of array, it's possible to make wrapper on entire array:

struct MetaArray<M: Meta>: Codable, ExpressibleByArrayLiteral {

    let array: [M.Element]

    init(_ array: [M.Element]) {
        self.array = array
    }

    init(arrayLiteral elements: M.Element...) {
        self.array = elements
    }

    enum CodingKeys: String, CodingKey {
        case metatype
        case object
    }

    init(from decoder: Decoder) throws {
        var container = try decoder.unkeyedContainer()

        var elements: [M.Element] = []
        while !container.isAtEnd {
            let nested = try container.nestedContainer(keyedBy: CodingKeys.self)
            let metatype = try nested.decode(M.self, forKey: .metatype)

            let superDecoder = try nested.superDecoder(forKey: .object)
            let object = try metatype.type.init(from: superDecoder)
            if let element = object as? M.Element {
                elements.append(element)
            }
        }
        array = elements
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.unkeyedContainer()
        try array.forEach { object in
            let metatype = M.metatype(for: object)
            var nested = container.nestedContainer(keyedBy: CodingKeys.self)
            try nested.encode(metatype, forKey: .metatype)
            let superEncoder = nested.superEncoder(forKey: .object)

            let encodable = object as? Encodable
            try encodable?.encode(to: superEncoder)
        }
    }
}

Where Meta is generic protocol:

protocol Meta: Codable {
    associatedtype Element

    static func metatype(for element: Element) -> Self
    var type: Decodable.Type { get }
}

---

Now, storing tags will look like:

enum TagMetatype: String, Meta {

    typealias Element = Tag

    case author
    case genre

    static func metatype(for element: Tag) -> TagMetatype {
        return element.metatype
    }

    var type: Decodable.Type {
        switch self {
        case .author: return AuthorTag.self
        case .genre: return GenreTag.self
        }
    }
}

struct AuthorTag: Tag {
    var metatype: TagMetatype { return .author } // keep computed to prevent auto-encoding
    let value: String
}

struct GenreTag: Tag {
    var metatype: TagMetatype { return .genre } // keep computed to prevent auto-encoding
    let value: String
}

struct Article: Codable {
    let title: String
    let tags: MetaArray<TagMetatype>
}

Result JSON:

let article = Article(title: "Article Title",
                      tags: [AuthorTag(value: "Author Tag Value"),
                             GenreTag(value:"Genre Tag Value")])

{
  "title" : "Article Title",
  "tags" : [
    {
      "metatype" : "author",
      "object" : {
        "value" : "Author Tag Value"
      }
    },
    {
      "metatype" : "genre",
      "object" : {
        "value" : "Genre Tag Value"
      }
    }
  ]
}

And in case you want JSON to look even prettier:

{
  "title" : "Article Title",
  "tags" : [
    {
      "author" : {
        "value" : "Author Tag Value"
      }
    },
    {
      "genre" : {
        "value" : "Genre Tag Value"
      }
    }
  ]
}

Add to Meta protocol

protocol Meta: Codable {
    associatedtype Element
    static func metatype(for element: Element) -> Self
    var type: Decodable.Type { get }

    init?(rawValue: String)
    var rawValue: String { get }
}

And replace CodingKeys with:

struct MetaArray<M: Meta>: Codable, ExpressibleByArrayLiteral {

    let array: [M.Element]

    init(array: [M.Element]) {
        self.array = array
    }

    init(arrayLiteral elements: M.Element...) {
        self.array = elements
    }

    struct ElementKey: CodingKey {
        var stringValue: String
        init?(stringValue: String) {
            self.stringValue = stringValue
        }

        var intValue: Int? { return nil }
        init?(intValue: Int) { return nil }
    }

    init(from decoder: Decoder) throws {
        var container = try decoder.unkeyedContainer()

        var elements: [M.Element] = []
        while !container.isAtEnd {
            let nested = try container.nestedContainer(keyedBy: ElementKey.self)
            guard let key = nested.allKeys.first else { continue }
            let metatype = M(rawValue: key.stringValue)
            let superDecoder = try nested.superDecoder(forKey: key)
            let object = try metatype?.type.init(from: superDecoder)
            if let element = object as? M.Element {
                elements.append(element)
            }
        }
        array = elements
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.unkeyedContainer()
        try array.forEach { object in
            var nested = container.nestedContainer(keyedBy: ElementKey.self)
            let metatype = M.metatype(for: object)
            if let key = ElementKey(stringValue: metatype.rawValue) {
                let superEncoder = nested.superEncoder(forKey: key)
                let encodable = object as? Encodable
                try encodable?.encode(to: superEncoder)
            }
        }
    }
}

Solution 3

Drawn from the accepted answer, I ended up with the following code that can be pasted into an Xcode Playground. I used this base to add a codable protocol to my app.

The output looks like this, without the nesting mentioned in the accepted answer.

ORIGINAL:
▿ __lldb_expr_33.Parent
  - title: "Parent Struct"
  ▿ items: 2 elements
    ▿ __lldb_expr_33.NumberItem
      - commonProtocolString: "common string from protocol"
      - numberUniqueToThisStruct: 42
    ▿ __lldb_expr_33.StringItem
      - commonProtocolString: "protocol member string"
      - stringUniqueToThisStruct: "a random string"

ENCODED TO JSON:
{
  "title" : "Parent Struct",
  "items" : [
    {
      "type" : "numberItem",
      "numberUniqueToThisStruct" : 42,
      "commonProtocolString" : "common string from protocol"
    },
    {
      "type" : "stringItem",
      "stringUniqueToThisStruct" : "a random string",
      "commonProtocolString" : "protocol member string"
    }
  ]
}

DECODED FROM JSON:
▿ __lldb_expr_33.Parent
  - title: "Parent Struct"
  ▿ items: 2 elements
    ▿ __lldb_expr_33.NumberItem
      - commonProtocolString: "common string from protocol"
      - numberUniqueToThisStruct: 42
    ▿ __lldb_expr_33.StringItem
      - commonProtocolString: "protocol member string"
      - stringUniqueToThisStruct: "a random string"

Paste into your Xcode project or Playground and customize to your liking:

import Foundation

struct Parent: Codable {
    let title: String
    let items: [Item]

    init(title: String, items: [Item]) {
        self.title = title
        self.items = items
    }

    enum CodingKeys: String, CodingKey {
        case title
        case items
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)

        try container.encode(title, forKey: .title)
        try container.encode(items.map({ AnyItem($0) }), forKey: .items)
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        title = try container.decode(String.self, forKey: .title)
        items = try container.decode([AnyItem].self, forKey: .items).map { $0.item }
    }

}

protocol Item: Codable {
    static var type: ItemType { get }

    var commonProtocolString: String { get }
}

enum ItemType: String, Codable {

    case numberItem
    case stringItem

    var metatype: Item.Type {
        switch self {
        case .numberItem: return NumberItem.self
        case .stringItem: return StringItem.self
        }
    }
}

struct NumberItem: Item {
    static var type = ItemType.numberItem

    let commonProtocolString = "common string from protocol"
    let numberUniqueToThisStruct = 42
}

struct StringItem: Item {
    static var type = ItemType.stringItem

    let commonProtocolString = "protocol member string"
    let stringUniqueToThisStruct = "a random string"
}

struct AnyItem: Codable {

    var item: Item

    init(_ item: Item) {
        self.item = item
    }

    private enum CodingKeys : CodingKey {
        case type
        case item
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)

        try container.encode(type(of: item).type, forKey: .type)
        try item.encode(to: encoder)
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        let type = try container.decode(ItemType.self, forKey: .type)
        self.item = try type.metatype.init(from: decoder)
    }

}

func testCodableProtocol() {
    var items = [Item]()
    items.append(NumberItem())
    items.append(StringItem())
    let parent = Parent(title: "Parent Struct", items: items)

    print("ORIGINAL:")
    dump(parent)
    print("")

    let jsonEncoder = JSONEncoder()
    jsonEncoder.outputFormatting = .prettyPrinted
    let jsonData = try! jsonEncoder.encode(parent)
    let jsonString = String(data: jsonData, encoding: .utf8)!
    print("ENCODED TO JSON:")
    print(jsonString)
    print("")

    let jsonDecoder = JSONDecoder()
    let decoded = try! jsonDecoder.decode(type(of: parent), from: jsonData)
    print("DECODED FROM JSON:")
    dump(decoded)
    print("")
}
testCodableProtocol()

Solution 4

Why wouldn't you use enums for the type of the tag?

struct Tag: Codable {
  let type: TagType
  let value: String

  enum TagType: String, Codable {
    case author
    case genre
  }
}

Then you can encode like try? JSONEncoder().encode(tag) or decode like let tags = try? JSONDecoder().decode([Tag].self, from: jsonData) and do any sort of processing as filtering the tags by type. You can do the same for the Article struct as well:

struct Tag: Codable {
    let type: TagType
    let value: String

    enum TagType: String, Codable {
        case author
        case genre
    }
}

struct Article: Codable {
    let tags: [Tag]
    let title: String

    enum CodingKeys: String, CodingKey {
        case tags
        case title
    }
}

View more solutions

Comments

[Hamish]

Why make AuthorTag & GenreTag separate types? They both have the exact same interface, and you seem to just be using the type property to differentiate between them anyway (although really that should probably be an enum).

[goesta]

It's just a simplified example. They could have individual properties. I thought about making type an enum too but if type is an enum i can not add new types without modifying the enum.

[Code Different]

Does the code actual work and product the JSON you included? I got Type 'Article' does not conform to protocol 'Decodable' and 'Encodable'

[Anish Parajuli 웃]

I just confirmed in Apple forum ,its a bug.Please file it as a bug

[Hamish]

@ThatlazyiOSGuy웃 I don't see how it's a bug – Tag doesn't conform to Codable (and therefore neither does [Tag]), because protocols don't conform to themselves. Consider if Tag conformed to Codable – what should happen upon a decoder trying to decode into an arbitrary Tag? What concrete type should be created?

[Anish Parajuli 웃]

@Hamish If so compiler should not allow protocol to conform codable

[Hamish]

@ThatlazyiOSGuy웃 Correct, and it doesn't – OP's code doesn't compile.

[Anish Parajuli 웃]

Yes but if you implement your own Decodable and Encodable methods.The error can be easily thrown out..gist.github.com/anonymous/74d2723e18444344f3635d403e8bf‌​6b8 and this looks like a bug

[Hamish]

@ThatlazyiOSGuy웃 Ah I agree that looks like a bug (in that it shouldn't compile) :) It crashes at runtime, as [Tag] doesn't conform to Codable.

[goesta]

I added a Encodable Example to the Question. @ThatlazyiOSGuy웃 can you post the link to the Apple forum thread?

[Anish Parajuli 웃]

@glektrik forums.developer.apple.com/message/234519#234519

[pkamb]

I added a feature request for this behavior on Swift bugs: bugs.swift.org/browse/SR-7566

[zoul]

Wow. Please let me add nothing to the discussion by saying this is a great answer!

[palme]

I tried it and it works great for creating the object but I'm having the problem that the custom attributes of the objects are troubling the decoder. I'm always getting a "Cannot get keyed decoding container -- found null value instead." Do you know any help here? Example: the "foo" var in the AuthorTag, as soon as I add this line, I get the error.

[Hamish]

@palme Ah, you're probably using the decoding logic that expects the base value to be encoded as a separate object (under the "base" key). If you're expecting the properties of the base value to be at the same level as the "type" key, you want to pass decoder instead of container.superDecoder(forKey: .base) & encoder instead of container.superEncoder(forKey: .base) inside the decoding/encoding logic of AnyTag.

[palme]

Again, thanks for the quick and precise answer! It worked.

[matthewfx]

Pure awesomeness!

[kun wang]

but how to handle if Tag with a associatedtype @Hamish

[Hamish]

@kunwang You'll need to do some fairly involved type erasure for that, I'm afraid. Here's a rough example: gist.github.com/hamishknight/5ffe87a43590a1f1fae8e341cf0da41‌​8.

[Freek Sanders]

Thanks ever so much. Was about to post a question, when StackOverflow suggested this awesome post :D

[Bagusflyer]

This helps!!! But I got an error message: Protocol 'Tag' can only be used as a generic constraint because it has Self or associated type requirements if I make the protocol Tag: Codable, Identifiable. Any idea? Thanks.

[Lucas]

Dude, this was awesome. I spent all morning looking for an elegant solution.