Evolve your Core Data schema

♪ ♪ David Stites: Hi, and welcome to "Evolve Your Core Data Schema." My name is David Stites and I am an engineer on the Core Data team. In this session, I am excited to talk to you about how to update and migrate the Core Data schema in your app. The agenda for this session is to learn what schema migration is and why your app has to perform it after updating its data model, how to migrate the existing schema, and how CloudKit and schema migration interact. First up, what is schema migration and why your app must migrate when you update your data model.

As your application evolves, it may become necessary to change your data model. Updating the data model requires that those changes are materialized in the underlying storage schema. Consider this data model. It has an Aircraft entity with two attributes, type and number of engines. These attributes are reflected in the underlying storage. If I add a number of passengers attribute, I need to add the corresponding storage. After migration, the changes are fully reflected in the underlying storage. Without migrating the changes in the underlying storage, Core Data will refuse to open your persistent store as the freshly-changed model doesn't match the model used for storage. Attempting to open an incompatible store will result in an error with the code NSPersistentStore- IncompatibleVersionHashError. If you receive this error, it should be an indication to you that a migration is required. Now that I've explained what schema migration is and why it's essential to evolving your app, let me tell you how migration is accomplished. Core Data has built-in data migration tools to help keep your app's data storage up-to-date with the current data model. Collectively, these tools are referred to as "lightweight migration." Lightweight migration is the preferred method of migration. Lightweight migration automatically analyzes and infers the migration from the differences between the source and destination managed object models. At runtime, Core Data looks for the models in the bundles returned by .allBundles and .allFrameworks methods of the NSBundle class. Lightweight migration then generates a mapping model to materialize the changes you've made in your app in your database schema.

Using lightweight migration requires the changes to the data model to fit an obvious migration pattern.

Lightweight operations involving attributes include adding an attribute, removing an attribute, making a non-optional attribute optional, making an optional attribute non-optional and defining a default value, and renaming an attribute. If you want to rename an attribute, set the renaming identifier in the destination model to the name of the corresponding attribute in the source model.

The renaming identifier is found in the Xcode Data Model Editor's property inspector. For example, you can rename the Aircraft entity color attribute to paintColor. The renaming identifier creates a canonical name, so set the renaming identifier to the name of the attribute in the source model, unless that attribute already has a renaming identifier. This means you can rename an attribute in version 2 of a model, and then rename it again in version 3. The renaming will work correctly going from version 2 to version 3, or from version 1 to version 3.

Lightweight migration can also handle changes to the relationships without breaking a sweat. You can add a new relationship or delete an existing relationship. You can also rename a relationship by using a renaming identifier, just like an attribute. In addition, you can change relationship cardinality, for example, migrating from a to-one to a to-many, or a non-ordered to-many to an ordered to-many, and vice versa.

If you guessed that entities are also eligible for lightweight migration, you're right. You can add a new entity, remove an existing entity, and rename entities. You can also create a new parent or child entity and move attributes up and down within the entity hierarchy. You can move entities into or out of a hierarchy. You cannot, however, merge entity hierarchies. if two existing entities do not share a common parent in the source, they cannot share a common parent in the destination. Lightweight migration is controlled by two options keys: NSMigratePersistent- StoresAutomaticallyOption and NSInferMappingModelAutomaticallyOption. The presence of these two keys set to a true value when the store is added to the persistent coordinator will cause Core Data to perform lightweight migration automatically if it detects the persistent store no longer matches the current model. If you're using NSPersistentContainer or NSPersistentStoreDescription, these options are set automatically for you and you don't need to do anything. If you're using an alternative API such as NSPersistentStoreCoordinator .addPersistentStore (type:configuration:at:options:), lightweight migration can be requested by setting and passing an options dictionary with the keys set NSMigratePersistent- StoresAutomaticallyOption and NSInferMappingModelAutomaticallyOption to a value of YES. Core Data will perform lightweight migration automatically if it detects the persistent store no longer matches the current model.

Here’s how this works in code. First, I'll import CoreData and create a managed object model. Then, I'll create a persistent store coordinator by using the model I just created. Note the options dictionary I created and that I'll pass when I add the store to the persistent coordinator. Lastly, I'll add the store to the coordinator where migration will occur automatically if necessary. Regardless of whatever API you use, the changes to your data model can be made directly in the same model that is shipping with the application. There is no need to create a new version of the model to make changes. If you want to determine in advance whether Core Data can infer the mapping model between the source and destination models without actually doing the work of migration, you can use NSMappingModel .inferredMappingModel method. The method returns the inferred model if Core Data was able to create it. Otherwise, it returns nil.

Sometimes, combined changes to the schema may exceed the capabilities of lightweight migration. I’m going to describe to you how to deal with that problem and still use lightweight migration.

Returning to our previous example model, suppose that we've previously added an attribute called “flightData” that uses external storage for binary data, indicated by the file path stored in FLIGHT_DATA. Further, suppose there is a need to change that attribute to store data internally and remove the external storage. Checking to see if this migration fits any of the capabilities of lightweight migration, it is discovered that it doesn't. On the face of it, it appears that we're stuck, unable to make this change. However, fear not! Lightweight migration can still be used to perform more complex, non-conforming migrations, albeit in multiple steps.

The goal becomes to decompose the migration tasks that aren't eligible for lightweight migration into a minimum series of migrations that are eligible for lightweight migration. Generally, if the original model is A and the objective model is B, but model B has changes that aren't eligible for lightweight migration, a bridge can be created by introducing one or more model versions that decompose those changes.

Each of the models introduced will have one or more operations that is within the capabilities that compose the non-conforming changes. This results in a series of migrations where each model is now lightweight migrateable but equivalent to the non-conforming migration. Returning to my example that wasn't eligible for lightweight migration, our original model is model A. I will start decomposing the task by introducing a new model version, A prime, and add a new attribute "tmpStorage" that will be used temporarily to store data that is imported from the external files.

Next, I will import the data from the external files into our new attribute. The code to import this data is separate from the functionality provided by Core Data. The execution of this import is interposed between migrations.

Once the data has been safely imported, I'll create another new version of the model A double-prime from A prime. In A double-prime, I will delete the old external storage attribute while simultaneously renaming the new attribute. Each of these steps described is within the capabilities of lightweight migration.

Intuitively, an event loop could be built that opens the persistent store with the lightweight migration options set and iteratively steps through each unprocessed model in a serial order, and Core Data will migrate the store. If you perform app-specific logic during your migrations, such as how I imported data from external files in the previous example, that logic must be "restartable" in the event the migration is interrupted due to the process terminating.

If your app uses Core Data and CloudKit, there are some important points you should keep in mind when designing your data model in Core Data. To pass records between a Core Data store and a CloudKit database, they require a shared understanding of the data model. You define this model in the Core Data model editor. That model is subsequently used to generate the CloudKit schema. The generated schema is created initially in the Development environment, and then promoted to Production. You should be aware that CloudKit doesn't support all the features of a Core Data model. As you design your model, be aware of the following limitations and create a compatible data model. For example, unique constraints on entities aren't supported. Undefined and objectID attribute type aren't supported as attribute types. And relationships must be optional and have an inverse relationship. In addition, CloudKit does not support the deny deletion rule. As you're developing your app, you'll be using the Development environment. The CloudKit schema can be modified freely in this environment. However, after you promote your schema to Production, the record types and their fields are immutable. While lightweight migration handles many different scenarios, CloudKit is more restricted in what it supports. Many of the lightweight operations I described earlier are unsupported. Specifically, what is supported in CloudKit is adding new fields to existing record types and adding new record types. You cannot modify or delete existing record types or fields. Consider these restrictions when modifying the model schema.

When it comes time to update your data model, keep in mind that lightweight migration only materializes schema changes in the local store file. Regardless of whether a particular store is used with CloudKit, migration will only change the store on disk and does not make changes to the CloudKit schema. You still need to materialize those changes in the Development database by running the schema initializer and then promoting those changes in Development to Production using the CloudKit Console. Keep in mind that users of your app will be using old versions as well as new versions. The latest version of the app will of course know about any new additions to the schema. Old versions of the app won't know about the new fields or record types.

Since CloudKit schema is essentially additive, give consideration to the effects of schema migration to devices running older versions of your app. For example, one common pitfall is forgetting to update old fields that the older versions of your app use but newer versions don't. Here are some strategies to migrate CloudKit schema. The first option is to incrementally add new fields to existing record types. If you adopt this approach, older versions of your app will have access to every record a user creates, but not every field.

A second option is to version your entities by including a version attribute, and then use a fetch request to select only the records that are compatible with the current version of the app.

If you adopt this approach, older versions of your app won't fetch records that a user creates with a more recent version, effectively hiding them on that device. The last strategy is to create a completely new container, using NSPersistentCloudKitContainerOptions, to associate the new store with a new container. Be aware that if the user has a large data set, uploading the data set to iCloud could take an extended period of time. Whatever method you use, take care in designing your data model. Be sure to consider cross-version compatibility issues and test different versions of your data model together. Now that we've thoroughly discussed data models, migration, and CloudKit, I am going to demonstrate this in action. As you may have guessed, I'm a pilot. I've created a small app to log my flight time. Here is the data model for that app. I have a single entity called “LogEntry” and have added a number of attributes, such as aircraft type, flight duration, origin, destination, and tail number to allow me to log the required experience information. When I run this application for the first time, Core Data will create the store and materialize the schema in that store. Before I run the application, I am going to turn on the com.apple.CoreData.SQLDebug and the com.apple.CoreData.MigrationDebug environment variables. This will cause Core Data to log the steps it is taking. With these arguments in place, I will run the app.

As the app launches, Core Data is logging the steps that it is taking: creating the file, creating the metadata for the store, and materializing the schema. SQLite created the table ZLOGENTRY with our schema in it. This can also be confirmed by looking at the store file using the sqlite3 command line tool. Here, I have the LogEntry table, and it has the corresponding columns to the attributes I created in the data model. Now I'm going to make some lightweight changes.

I'm adding some new entities, Aircraft, Pilot, and Airport. This will help me normalize the schema. I am changing some of the attributes in the LogEntry entity to be relationships. For example, destination and origin move from being string attributes to being an Airport to-one relationship. The Airport entity also has two new attributes, icaoIdentifier and faaIdentifier. The type attribute is promoted to a new entity; Aircraft and I am adding two new attributes, tailNumber and registrationNumber. On LogEntry, I am creating a to-one relationship to an Aircraft from LogEntry.

Lastly, I added a Pilot entity that has name and certificate ID.

Each log entry will be related to a Pilot entity. Now that I've completed my changes to the data model, I'm going to run the app again.

Oop! I received an error running the app. Inspecting the code, it is NSPersistentStore- IncompatibleVersionHashError. That error means that my current model no longer matches the schema for the model in the store. I need to migrate the store schema. I can do that in one of three ways. Using the first method, I can convert my code to use an NSPersistentContainer as the lightweight migration options are automatically set for me. Using the second method, I can use NSPersistentStoreDescription, as, again, the lightweight migrations options are automatically set for me. Lastly, using the third method, I can manually set the lightweight migration options on an options dictionary and pass that dictionary to the coordinator when opening the store.

I think I'll go with the first option, using an NSPersistentContainer. Now that I have converted the code to use an NSPersistentContainer, I will launch the app and again observe that Core Data is migrating the schema in the store file.

Again, this can be confirmed using the sqlite3 command line tool. Note the new schema was materialized by Core Data automatically, using lightweight migration. What could be easier? Before ending my demo, I wanted to show option number 3. Recall in this option, I am manually setting the lightweight migration options on an options dictionary and then passing that dictionary to the coordinator when opening the store. The end result is the same in that the store is migrated to the new schema. When you make changes to your data model, use lightweight migration to help you. Lightweight migration is very flexible and easy to use for the vast majority of data model changes. If you have more complex data models, break that model down into ones that are composed of lightweight changes. Lastly, if you use CloudKit with your app, carefully consider the implications of the data model changes. Thoroughly test any data model changes. I hope you've found this information useful and that you'll consider updating the model in your project to build some awesome new features. Thanks for flying with me, and have a great WWDC.