Managing a bunch of NSOperation with dependencies

Question

I'm working on an application that create contents and send it to an existing backend. Content is a title, a picture and location. Nothing fancy.

The backend is a bit complicated so here is what I have to do :

• Let the user take a picture, enter a title and authorize the map to use its location
• Generate a unique identifier for the post
• Create the post on the backend
• Upload the picture
• Refresh the UI

I've used a couple of NSOperation subclasses to make this work but I'm not proud of my code, here is a sample.

NSOperation *process = [NSBlockOperation blockOperationWithBlock:^{
    // Process image before upload
}];

NSOperation *filename = [[NSInvocationOperation alloc] initWithTarget: self selector: @selector(generateFilename) object: nil];

NSOperation *generateEntry = [[NSInvocationOperation alloc] initWithTarget: self selector: @selector(createEntry) object: nil];

NSOperation *uploadImage = [[NSInvocationOperation alloc] initWithTarget: self selector: @selector(uploadImageToCreatedEntry) object: nil];

NSOperation *refresh = [NSBlockOperation blockOperationWithBlock:^{
    // Update UI
    [SVProgressHUD showSuccessWithStatus: NSLocalizedString(@"Success!", @"Success HUD message")];
}];

[refresh addDependency: uploadImage];

[uploadImage addDependency: generateEntry];
[generateEntry addDependency: filename];
[generateEntry addDependency: process];

[[NSOperationQueue mainQueue] addOperation: refresh];
[_queue addOperations: @[uploadImage, generateEntry, filename, process] waitUntilFinished: NO];

Here are the things I don't like :

• in my createEntry: for example, I'm storing the generated filename in a property, which mees with the global scope of my class
• in the uploadImageToCreatedEntry: method, I'm using dispatch_async + dispatch_get_main_queue() to update the message in my HUD
• etc.

How would you manage such workflow ? I'd like to avoid embedding multiple completion blocks and I feel like NSOperation really is the way to go but I also feel there is a better implementation somewhere.

Thanks!

Solution 1

You can use ReactiveCocoa to accomplish this pretty easily. One of its big goals is to make this kind of composition trivial.

If you haven't heard of ReactiveCocoa before, or are unfamiliar with it, check out the Introduction for a quick explanation.

I'll avoid duplicating an entire framework overview here, but suffice it to say that RAC actually offers a superset of promises/futures. It allows you to compose and transform events of completely different origins (UI, network, database, KVO, notifications, etc.), which is incredibly powerful.

To get started RACifying this code, the first and easiest thing we can do is put these separate operations into methods, and ensure that each one returns a RACSignal. This isn't strictly necessary (they could all be defined within one scope), but it makes the code more modular and readable.

For example, let's create a couple signals corresponding to process and generateFilename:

- (RACSignal *)processImage:(UIImage *)image {
    return [RACSignal startEagerlyWithScheduler:[RACScheduler scheduler] block:^(id<RACSubscriber> subscriber) {
        // Process image before upload

        UIImage *processedImage = …;
        [subscriber sendNext:processedImage];
        [subscriber sendCompleted];
    }];
}

- (RACSignal *)generateFilename {
    return [RACSignal startEagerlyWithScheduler:[RACScheduler scheduler] block:^(id<RACSubscriber> subscriber) {
        NSString *filename = [self generateFilename];
        [subscriber sendNext:filename];
        [subscriber sendCompleted];
    }];
}

The other operations (createEntry and uploadImageToCreatedEntry) would be very similar.

Once we have these in place, it's very easy to compose them and express their dependencies (though the comments make it look a bit dense):

[[[[[[self
    generateFilename]
    flattenMap:^(NSString *filename) {
        // Returns a signal representing the entry creation.
        // We assume that this will eventually send an `Entry` object.
        return [self createEntryWithFilename:filename];
    }]
    // Combine the value with that returned by `-processImage:`.
    zipWith:[self processImage:startingImage]]
    flattenMap:^(RACTuple *entryAndImage) {
        // Here, we unpack the zipped values then return a single object,
        // which is just a signal representing the upload.
        return [self uploadImage:entryAndImage[1] toCreatedEntry:entryAndImage[0]];
    }]
    // Make sure that the next code runs on the main thread.
    deliverOn:RACScheduler.mainThreadScheduler]
    subscribeError:^(NSError *error) {
        // Any errors will trickle down into this block, where we can
        // display them.
        [self presentError:error];
    } completed:^{
        // Update UI
        [SVProgressHUD showSuccessWithStatus: NSLocalizedString(@"Success!", @"Success HUD message")];
    }];

Note that I renamed some of your methods so that they can accept inputs from their dependencies, giving us a more natural way to feed values from one operation to the next.

There are huge advantages here:

• You can read it top-down, so it's very easy to understand the order that things happen in, and where the dependencies lie.
• It's extremely easy to move work between different threads, as evidenced by the use of -deliverOn:.
• Any errors sent by any of those methods will automatically cancel all the rest of the work, and eventually reach the subscribeError: block for easy handling.
• You can also compose this with other streams of events (i.e., not just operations). For example, you could set this up to trigger only when a UI signal (like a button click) fires.

ReactiveCocoa is a huge framework, and it's unfortunately hard to distill the advantages down into a small code sample. I'd highly recommend checking out the examples for when to use ReactiveCocoa to learn more about how it can help.

Solution 2

A couple of thoughts:

1. I would be inclined to avail myself of completion blocks because you probably only want to initiate the next operation if the previous one succeeded. You want to make sure that you properly handle errors and can easily break out of your chain of operations if one fails.

2. If I wanted to pass data from operation to another and didn't want to use some property of the caller's class, I would probably define my own completion block as a property of my custom operation that had a parameter which included the field that I wanted to pass from one operation to another. This assumes, though, that you're doing NSOperation subclassing.

   For example, I might have a FilenameOperation.h that defines an interface for my operation subclass:

   #import <Foundation/Foundation.h>
   
   typedef void (^FilenameOperationSuccessFailureBlock)(NSString *filename, NSError *error);
   
   @interface FilenameOperation : NSOperation
   
   @property (nonatomic, copy) FilenameOperationSuccessFailureBlock successFailureBlock;
   
   @end
   

   and if it wasn't a concurrent operation, the implementation might look like:

   #import "FilenameOperation.h"
   
   @implementation FilenameOperation
   
   - (void)main
   {
       if (self.isCancelled)
           return;
   
       NSString *filename = ...;
       BOOL failure = ...
   
       if (failure)
       {
           NSError *error = [NSError errorWithDomain:... code:... userInfo:...];
           if (self.successFailureBlock)
               self.successFailureBlock(nil, error);                                                    
       }
       else
       {
           if (self.successFailureBlock)
               self.successFailureBlock(filename, nil);
       }
   }
   
   @end
   

   Clearly, if you have a concurrent operation, you'll implement all of the standard isConcurrent, isFinished and isExecuting logic, but the idea is the same. As an aside, sometimes people will dispatch those success or failures back to the main queue, so you can do that if you want, too.

   Regardless, this illustrates the idea of a custom property with my own completion block that passes the appropriate data. You can repeat this process for each of the relevant types of operations, you can then chain them all together, with something like:

   FilenameOperation *filenameOperation = [[FilenameOperation alloc] init];
   GenerateOperation *generateOperation = [[GenerateOperation alloc] init];
   UploadOperation   *uploadOperation   = [[UploadOperation alloc] init];
   
   filenameOperation.successFailureBlock = ^(NSString *filename, NSError *error) {
       if (error)
       {
           // handle error
           NSLog(@"%s: error: %@", __FUNCTION__, error);
       }
       else
       {
           generateOperation.filename = filename;
           [queue addOperation:generateOperation];
       }
   };
   
   generateOperation.successFailureBlock = ^(NSString *filename, NSData *data, NSError *error) {
       if (error)
       {
           // handle error
           NSLog(@"%s: error: %@", __FUNCTION__, error);
       }
       else
       {
           uploadOperation.filename = filename;
           uploadOperation.data     = data;
           [queue addOperation:uploadOperation];
       }
   };
   
   uploadOperation.successFailureBlock = ^(NSString *result, NSError *error) {
       if (error)
       {
           // handle error
           NSLog(@"%s: error: %@", __FUNCTION__, error);
       }
       else
       {
           [[NSOperationQueue mainQueue] addOperationWithBlock:^{
               // update UI here
               NSLog(@"%@", result);
           }];
       }
   };
   
   [queue addOperation:filenameOperation];
   

3. Another approach in more complicated scenarios is to have your NSOperation subclass employ a technique analogous to how the standard addDependency method works, in which NSOperation sets the isReady state based upon KVO on isFinished on the other operation. This not only allows you to not only establish more complicated dependencies between operations, but also to pass database between them. This is probably beyond the scope of this question (and I'm already suffering from tl:dr), but let me know if you need more here.

4. I wouldn't be too concerned that uploadImageToCreatedEntry is dispatching back to the main thread. In complicated designs, you might have all sorts of different queues dedicated for particular types of operations, and the fact that UI updates are added to the main queue is perfectly consistent with this mode. But instead of dispatch_async, I might be inclined to use the NSOperationQueue equivalent:

   [[NSOperationQueue mainQueue] addOperationWithBlock:^{
       // do my UI update here
   }];
   

5. I wonder if you need all of these operations. For example, I have a hard time imagining that filename is sufficiently complicated to justify its own operation (but if you're getting the filename from some remote source, then a separate operation makes perfect sense). I'll assume that you're doing something sufficiently complicated that justifies it, but the names of those operations make me wonder, though.

6. If you want, you might want to take a look at couchdeveloper's RXPromise class which uses promises to (a) control the logical relationship between separate operations; and (b) simplify the passing of data from one to the next. Mike Ash has a old MAFuture class which does the same thing.

   I'm not sure either of those are mature enough that I'd contemplate using them in my own code, but it's an interesting idea.

Solution 3

I'm probably totally, biased - but for a particular reason - I like @Rob's approach #6 ;)

Assuming you created appropriate wrappers for your asynchronous methods and operations which return a Promise instead of signaling the completion with a completion block, the solution looks like this:

RXPromise* finalResult = [RXPromise all:@[[self filename], [self process]]]
.then(^id(id filenameAndProcessResult){
    return [self generateEntry];
}, nil)
.then(^id(id generateEntryResult){
    return [self uploadImage];
}, nil)
.thenOn(dispatch_get_main_queue() , ^id(id uploadImageResult){
    [self refreshWithResult:uploadImageResult];
    return nil;
}, nil)
.then(nil, ^id(NSError*error){
    // Something went wrong in any of the operations. Log the error:
    NSLog(@"Error: %@", error);
});

And, if you want to cancel the whole asynchronous sequence at any tine, anywhere and no matter how far it has been proceeded:

[finalResult.root cancel];

(A small note: property root is not yet available in the current version of RXPromise, but its basically very simple to implement).

Solution 4

If you still want to use NSOperation, you can rely on ProcedureKit and use the injection properties of the Procedure class.

For each operation, specify which type it produces and inject it to the next dependent operation. You can also at the end wrap the whole process inside a GroupProcedure class.

View more solutions

Comments

[Romain Pouclet]

I think I really like approach #2, even if it feels like homemade dependency management and I'm not sure I'm confortable with that. It does look prettier thought, so thanks for that. About #3, I wouldn't mind a couple more of information, even if my workflow isn't THAT complicated. I may be able to reduce the number of operations AND fix my method names, they are indeed confusing. Thank you so much for your detailled answer !

[CouchDeveloper]

I'm very impressed by RAC. And actually, I would like it. However, it's quite complex. The number of files is huge (roughly 80 source modules). On the other hand, the Promise library is just one class - and one source module. The huge advantages for your RAC example solution do also apply to the Promise solution. Despite its minimalistic API, Promises are surprisingly powerful. I don't think the OPs problem is hard enough to stress RAC, though. It's totally "in the range" of what Promises can do, too. And that's probably 95% of those kind of problems.

[Rob]

@Palleas See this answer for an example of that third approach. But you don't have any compelling need to do anything that complicated (we needed it in that case because we wanted to manually trigger when next operation triggered in the completion block of an operation rather than the operation itself).

[Justin Spahr-Summers]

@CouchDeveloper The problem is that futures don't compose with anything else. Let's say you use them for background operations — then what? What happens when you need information from KVO, NSNotificationCenter, or the UI? I won't deny that ReactiveCocoa is huge (in fact, I stated it outright in my answer), but the size is related to the unification of all these patterns which seem very different at first glance, so I absolutely think it's worth it.

[babygau]

HI @JustinSpahr-Summers, could you care to explain for me what's difference between startEagerlyWithScheduler and deliverOn: messages?

[Justin Spahr-Summers]

@TruongThanhDung +startEagerlyWithScheduler:block: is used when you have some operation that you want to start doing immediately (i.e., as soon as that method is called). -deliverOn: just transforms an existing signal, such that all of its event callbacks happen on a different thread.

[febeling]

@JustinSpahr-Summers What starts processing of this whole pipeline? Is it thought to be in setup code (-viewDidLoad), or in an action? Where does the startingImage used in the zipWith: line come from?

[Justin Spahr-Summers]

@febeling This would go wherever the -addOperation: code (as shown in the OP) would've gone. startingImage is made up for the purposes of demonstration.

[SpaceTrucker]

I've used promises which work for some cases, but operation queues are still useful and offer features promises do not.

[Justin Spahr-Summers]

@SpaceTrucker Streams/signals are not promises. They offer a superset of promises' capabilities.

[sudo]

This framework would be a lot easier to use in Swift if Xcode 7-9's code completion worked properly (e.g. didn't crash, lag, or have memory leaks). You kinda need it for all these block-based operations, unfortunately.