Golang concurrency: how to append to the same slice from different goroutines

Question

I have concurrent goroutines which want to append a (pointer to a) struct to the same slice. How do you write that in Go to make it concurrency-safe?

This would be my concurrency-unsafe code, using a wait group:

var wg sync.WaitGroup
MySlice = make([]*MyStruct)
for _, param := range params {
    wg.Add(1)
    go func(param string) {
        defer wg.Done()
        OneOfMyStructs := getMyStruct(param)
        MySlice = append(MySlice, &OneOfMyStructs)
    }(param)
}
wg.Wait()

I guess you would need to use go channels for concurrency-safety. Can anyone contribute with an example?

Solution 1

There is nothing wrong with guarding the MySlice = append(MySlice, &OneOfMyStructs) with a sync.Mutex. But of course you can have a result channel with buffer size len(params) all goroutines send their answers and once your work is finished you collect from this result channel.

If your params has a fixed size:

MySlice = make([]*MyStruct, len(params))
for i, param := range params {
    wg.Add(1)
    go func(i int, param string) {
         defer wg.Done()
         OneOfMyStructs := getMyStruct(param)
         MySlice[i] = &OneOfMyStructs
     }(i, param)
}

As all goroutines write to different memory this isn't racy.

Solution 2

The answer posted by @jimt is not quite right, in that it misses the last value sent in the channel and the last defer wg.Done() is never called. The snippet below has the corrections.

https://play.golang.org/p/7N4sxD-Bai

package main

import "fmt"
import "sync"

type T int

func main() {
    var slice []T
    var wg sync.WaitGroup

    queue := make(chan T, 1)

    // Create our data and send it into the queue.
    wg.Add(100)
    for i := 0; i < 100; i++ {
        go func(i int) {
            // defer wg.Done()  <- will result in the last int to be missed in the receiving channel
            queue <- T(i)
        }(i)
    }

    go func() {
        // defer wg.Done() <- Never gets called since the 100 `Done()` calls are made above, resulting in the `Wait()` to continue on before this is executed
        for t := range queue {
            slice = append(slice, t)
            wg.Done()   // ** move the `Done()` call here
        }
    }()

    wg.Wait()

    // now prints off all 100 int values
    fmt.Println(slice)
}

Comments

[Gustavo Niemeyer]

I believe the response here answers that question well: stackoverflow.com/questions/18467445/…

[Daniele B]

It's very interesting your last consideration: in case the size of the slice is known and you are just dealing with pointers to the objects, you don't need to use a concurrency mechanism at all

[Volker]

This does not depend on "slice of pointers": It would work also for "slice of MyStruct". Again the code never writes to the same memory.

[Daniele B]

I was assuming that the memory allocation for a pointer is fixed, while the memory allocation for a struct is not fixed. I suppose I am wrong then.

[Volker]

Hu? What is "fixed"? Any type in Go has a certain memory layout which is determined completely a compile time. No difference between a pointer and something else.

[Bjarke Ebert]

I've read that you should usually not have buffering in channels, unless there's a specific need. For your suggestion about the goroutines writing to a shared channel, isn't that a case where you would just deliver synchronously (unbuffered)?

[ineiti]

Coming from the future: why does the @jimt solution not work? The wg.Done() is deferred, so it is only called after the value is sent through the channel. What am I missing?

[Pioz]

@chris Can I use an unbuffered channel? queue := make(chan T)

[Abhijit]

@DanieleB Can you elaborate why wouldn't you need a concurrency mechanism at all? How else would I map a slice concurrently if I know the length of the original slice? Looking for a more correct/better approach if it exists.

[Daniele B]

@Abhijit if you create a new slice of fixed length and you iterate on it to write each element, then you don't need a concurrency mechanism, because there is no concurrency involved. Each element already has its allocated memory and don't concur with other elements.

[Abhijit]

@DanieleB If the getMyStruct function in the question is a very slow function and I want to run it for all the elements concurrently instead of sequentially, then I'd need a concurrent mechanism, right?

[Daniele B]

@Abhijit if such slice is fixed size and local to the function until all elements are written, then you shouldn't need a concurrent mechanism. If you instead you are writing to a slice which is not local to the function, then you would need a concurrent mechanism.