Adding unique objects to Core Data

Question

I'm working on an iPhone app that gets a number of objects from a database. I'd like to store these using Core Data, but I'm having problems with my relationships.

A Detail contains any number of POIs (points of interest). When I fetch a set of POI's from the server, they contain a detail ID. In order to associate the POI with the Detail (by ID), my process is as follows: Query the ManagedObjectContext for the detailID. If that detail exists, add the poi to it. If it doesn't, create the detail (it has other properties that will be populated lazily).

The problem with this is performance. Performing constant queries to Core Data is slow, to the point where adding a list of 150 POI's takes a minute thanks to the multiple relationships involved.

In my old model, before Core Data (various NSDictionary cache objects) this process was super fast (look up a key in a dictionary, then create it if it doesn't exist)

I have more relationships than just this one, but pretty much every one has to do this check (some are many to many, and they have a real problem).

Does anyone have any suggestions for how I can help this? I could perform fewer queries (by searching for a number of different ID's), but I'm not sure how much this will help.

Some code:

        POI *poi = [NSEntityDescription
                insertNewObjectForEntityForName:@"POI"
                inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];

    poi.POIid = [attributeDict objectForKey:kAttributeID];
    poi.detailId = [attributeDict objectForKey:kAttributeDetailID];
    Detail *detail = [self findDetailForID:poi.POIid];
    if(detail == nil)
    {
        detail = [NSEntityDescription
                    insertNewObjectForEntityForName:@"Detail"
                    inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];
        detail.title = poi.POIid;
        detail.subtitle = @"";
        detail.detailType = [attributeDict objectForKey:kAttributeType];
    }





-(Detail*)findDetailForID:(NSString*)detailID {
NSManagedObjectContext *moc = [[UIApplication sharedApplication].delegate managedObjectContext];
NSEntityDescription *entityDescription = [NSEntityDescription
                                          entityForName:@"Detail" inManagedObjectContext:moc];
NSFetchRequest *request = [[[NSFetchRequest alloc] init] autorelease];
[request setEntity:entityDescription];

NSPredicate *predicate = [NSPredicate predicateWithFormat:
                          @"detailid == %@", detailID];
[request setPredicate:predicate];
NSLog(@"%@", [predicate description]);

NSError *error;
NSArray *array = [moc executeFetchRequest:request error:&error];
if (array == nil || [array count] != 1)
{
        // Deal with error...
    return nil;
}
return [array objectAtIndex:0];
}

Solution 1

This page provides some help on optimizing performance: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#//apple_ref/doc/uid/TP40003468-SW1

While not very efficient, why not just build them in-memory with a NSDictionary? Read everything from Core Data into a NSDictionary then merge in your data, replacing everything in Core Data.

Solution 2

Check out the section titled "Batch Faulting" on the page titled "Core Data Performance" in Xcode's Core Data Programming Guide that Norman linked to in his answer.

Only fetching those managedObjects whose ids are IN a collection (NSSet, NSArray, NSDictionary) of ids of the objects returned by the server may be even more efficient.

NSSet *oids = [[NSSet alloc] initWithObjects:@"oid1", @"oid2", ..., nil];
NSPredicate *predicate = [NSPredicate predicateWithFormat:@"oid IN %@", oids];
[oids release];

UPDATE: I worked this tip into a solution for the acani usersView. Basically, after downloading a JSON response of users, the iPhone uses the popular open source JSON framework to parse the response into an NSArray of NSDictionary objects, each representing a user. Then, it makes an NSArray of their uids and does a batch fetch on Core Data to see if any of them already exist on the iPhone. If not, it inserts it. If so, it updates the ones that do exist only if their updated attribute is older than that of the one from the server.

Solution 3

I've gotten all this to work really well, thanks to Norman, who put me on the right path. I'll post my helper class here for others.

Basically, my helper class will look up if an NSManagedObject exists for some ID, and can create it for some ID. This executes quickly enough for me, with 1,000 find/create operations taking around 2 seconds on my iPhone (I also did a few other things there, pure find/create is likely faster).

It does this by caching a dictionary of all the NSManagedObjects, and checking that cache rather than executing a new NSFetchRequest.

A couple of modifications that could help things speed up even further: 1. Get only selected properties for the NSManagedObjects 2. Only get the identifier property for the NSManagedObject into a dictionary, instead of the whole object. In my performance testing, the single query wasn't the slow part (but with only 1,000 items, I'd expect it to be fast). The slow part was the creation of the items.

  #import "CoreDataUniquer.h"


@implementation CoreDataUniquer

    //the identifying property is the field on the NSManagedObject that will be used to look up our custom identifier
-(id)initWithEntityName:(NSString*)newEntityName andIdentifyingProperty:(NSString*)newIdProp
{
    self = [super init];
    if (self != nil) {
        entityName = [newEntityName retain];
        identifyingProperty = [newIdProp retain];
    }
    return self;
}

-(NSManagedObject*)findObjectForID:(NSString*)identifier
{
    if(identifier == nil)
    {
        return nil;
    }
    if(!objectList)
    {   
        NSManagedObjectContext *moc = [(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext];
        NSEntityDescription *entityDescription = [NSEntityDescription
                                                  entityForName:entityName inManagedObjectContext:moc];
        NSFetchRequest *request = [[[NSFetchRequest alloc] init] autorelease];
        [request setEntity:entityDescription];

        NSError *error;
        NSArray *array = [moc executeFetchRequest:request error:&error];
        objectList = [[NSMutableDictionary dictionary] retain];
        for (NSManagedObject* p in array) {
            NSString* itemId = [p valueForKey:identifyingProperty];
            [objectList setObject:p forKey:itemId];
        }
    }
    NSManagedObject* returnedObject = [objectList objectForKey:identifier];
    return returnedObject;
}
-(NSManagedObject*)createObjectForID:(NSString*)identifier
{

    NSManagedObject* returnedObject = [NSEntityDescription
                                       insertNewObjectForEntityForName:entityName
                                       inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];
    [returnedObject setValue:identifier forKey:identifyingProperty];
    [objectList setObject:returnedObject forKey:identifier];
    return returnedObject;
}


- (void) dealloc
{
    DESTROY(entityName);
    DESTROY(identifyingProperty);
    [super dealloc];
}

@end

Comments

[Sneakyness]

Explain your objects in more detail, and post some of the code you use. It's almost 100% likely there is a way to do what you're already doing much faster.

[Jose]

Interesting idea. The pattern would then be, whenever creating objects 1. Fetch the possible range of related objects, store in NSDictionary with key 2. If the id exists, add the relationship to that object 3. If the id doesn't, add the new object to both the dictionary and Core Data. Add the relationship.

[Jose]

Also, make sure you tick the "index" option on your identifier property.

[ma11hew28]

Thanks! Would you please also post the .h file?

[Valerii Pavlov]

You try to find entity with attribute value by checking all entities from store. You access all this entites and it coz change fault state to real query. I think u must use predicate and limitation for NSFetchRequest and execute query. if count > 0, it means that entity with attribute already exists in persistent store. After that u can access this request result to update your entity.