Building a custom migration in Drupal 8, Part 2: Tools and Modules

 

In the last post we set the stage to build a custom migration in Drupal 8. We installed Drupal 8 locally, and performed some initial, basic configuration. We didn’t get very far in technical tasks, but we spent a lot of time revisiting and rethinking our previous design choices.

At this point you should have an idea of how you want your new site to work. Work, not look. We’re interested in functional details, content types, organizational strategies, that sort of thing. Visual appearance is a completely different topic (and perhaps one I should write about in the future).

We’ve laid the conceptual foundation, now we can lay the technical one.

Installing required tools

Drupal core provides a lot of migration tools out of the box, but there are still some gaps that make it frustrating to work with. The biggest of which is lack of command line integration.

If you’ve worked with Drupal 7 before, you’re probably familiar with drush. Drush still exists in Drupal 8, and it’s just as useful. At the time of this writing, the recommended way to install drush is with Composer:

$ cd your/drupal/8/site/root
​$ composer require drush/drush

Next you’ll want to get a two additional modules to support your migrations, Migrate Plus and Migrate Tools. Migrate Plus’ key feature is migration groups. This allows migrations to be grouped together and more easily managed. Migrate Plus also provides additional migration plugins that we may need as we develop our migrations, too. Migrate Tools provides drush integration. While you can run all your migrations through the web UI, it can be far easier to troubleshoot problems on the command line.

You can install both using Composer:

$ composer require drupal/migrate_plus
$ composer require drupal/migrate_tools

Note that unlike Drupal 7, the best place to install these modules is in modules/contrib, and not sites/all/modules/contrib. Using Composer to install modules may be a little weird if you’re used to drush dl, but you’ll quickly get accustomed to the workflow.

Next we need to enable migration modules in your Drupal 8 site. Bring up MAMP, Drupal VM, or your Docker containers and navigate to Admin > Extend. You can also use drush en if you prefer the command line.

Enable the following modules:

  • Migrate
  • Migrate Drupal
  • Migrate Drupal UI
  • Migrate Plus
  • Migrate Tools

You technically don’t need the Migrate Drupal UI, but it can be useful for reading migration messages, which is how Migrate will report errors or notices.

Creating a migration group

With the tools in place and enabled, we can start building some migrations. We can start by creating a new module, but it’s actually a lot faster to create the group using the web UI:

  1. Login to your Drupal 8 site.
  2. Navigate to Admin > Structure > Migrations.
  3. Click Add Migration Group.
  4. Enter a Label, and Description of your choice.
  5. Enter the appropriate Source Type for your original site.
  6. Click Create Migration Group.

The Source Type, from what I can tell, appears to just be a label for the convenience of the admin. My source site is Drupal 7, so I entered Drupal 7.

At this point you can go to Admin > Structure > Migrations and select List Migrations for your group under the Operations column. It’s no surprise that there are no migrations listed for your new group, but there also doesn’t seem to be a button to create any! To do that, we’re going to need to break out a text editor and create some YAML.

Before that, however, we need to be able to commit our work to our repository. In Drupal 7, we would have already created a custom module to house our migrations. It’s not a required step in Drupal 8, but we don’t want to have our migration group just floating around in the database either.

Exporting our configuration

Unlike Drupal 7, Drupal 8 has much better configuration management built right in. We don’t need to download and enabled Features module to externalize our configuration. But we do need to change our settings.php to specify a configuration export directory.

While this directory can be anywhere, it’s often useful to place it outside your docroot in the root directory of your repository:

path/to/yoursite
├── src
│   ├── core
│   ├── modules
│   ├── profiles
│   ├── sites
│   ├── themes
│   ├── vendor (git ignored)
│   └── index.php
└── sync

In this series, I’ve used src for the Drupal docroot, and sync for the configuration directory. You may choose other names if you wish. It’s best to create this directory ahead of time, so Drupal doesn’t give you permission problems.

Now we can make Drupal aware of the new configuration directory. Open settings.php in your favorite IDE or text editor and add the following line:

$config_directories[CONFIG_SYNC_DIRECTORY] = '../sync';

The CONFIG_SYNC_DIRECTORY setting can be relative to the Drupal docroot. In this case, we’re pointing it to the sync directory in the same parent directory as our docroot.

Finally, we can export our configuration. While we can do this from the web UI, it’s much faster to use drush config-export, or cex:

$ cd path/to/yoursite
$ drush cex -y

If this is the first time you’ve exported, this produces a lot of files. Drupal 8 has a much more solid separation from configuration and content. All the module settings, content types, fields, theme settings and so on can be exported to a collection of *.yml files. These are the equivalent of feature modules in Drupal 7, and just like feature modules, these *.yml files should be tracked by git.

Examining our work

If we browse around our sync directory, we should find a file named something like migrate_plus.migration_group.yoursite.yml. This is the migration group you created in the web UI. We can now open it in a text editor and see what it’s made of:

uuid: e407154d-72e3-4ca5-a5af-eaaf651fe09f
langcode: en
status: true
dependencies: { }
id: yoursite
label: Yoursite
description: 'Migrations for yoursite'
source_type: 'Drupal 7'
module: null
shared_configuration: null

Yeah, it’s not much. We don’t actually need to change anything here at all, but it is good to know where the configuration was externalized and what it contains.

Importing the legacy database

The migration group alone doesn’t do much. It only serves as a container for later migrations we’ll create. Before we can create any of those, however, we need to get our old site’s data somewhere Drupal 8 can access it.

If you haven’t already, create a database dump of your existing site. You can do this with phpMyAdmin, drush sql-dump, or whatever tools your web host has provided. Next, create a new, empty  database on your local workstation where Drupal 8 is running. Import the dump you created earlier into the new database.

For this series, we have two databases, yoursite_drupal7, and yoursite_drupal8. Our settings.php file already knows about the Drupal 8 database, so we need to update it to make it aware of our Drupal 7 database as well:

$databases['default']['default'] = array(
   'database' => ‘yoursite_drupal8’,
   'username' => ‘myUser’,
   'password' => ‘myPassword’,
   'host' => '127.0.0.1',
   'port' => '',
   'driver' => 'mysql',
   'prefix' => '',
);

$databases['legacy']['default'] = array(
   'database' => 'deninet_drupal7',
   'username' => ‘myUser’,
   'password' => ‘myPassword’,
   'host' => '127.0.0.1',
   'port' => '',
   'driver' => 'mysql',
   'prefix' => '',
);

Both of our $database entries look very similar, and they should! The databases are on the same host, and in our case, have the same access credentials. The two big differences are the name of the database, and the database “key”. Our Drupal 8 database has the key of default, whereas our Drupal 7 database has the key of legacy.

Why is the key important? We need to update our migration group to know what database to look in!

Updating our migration group

When we created the migration group, we didn’t specify any database credentials. In fact, there’s no UI to set this information. Instead we need to do a little sleuthing in order to figure out how to update our configuration *.yml to make Migrate Plus aware of the legacy database.

Fortunately, the Migrate Plus module provides the migrate_example submodule. While there’s a lot of code in there, what we’re interested in is migrate_example’s config/install directory:

path/to/yoursite/modules/contirb/migrate_plus
   └── migrate_example
       ├── config
       │   └── install
       │       ├── migrate_plus.migration.beer_comment.yml
       │       ├── migrate_plus.migration.beer_node.yml
       │       ├── migrate_plus.migration.beer_term.yml
       │       ├── migrate_plus.migration.beer_user.yml
       │       └── migrate_plus.migration_group.beer.yml
       └── migrate_example.info.yml

You’ll quickly notice there’s a migration group *.yml file in there, migrate_plus.migration_group.beer.yml. Examining the file and it’s comments, you’ll quickly find out what we need to do is add the following to our migration group configuration:

shared_configuration:
 source:
​   key: ourDatabaseKey

In our case, our Drupal 7 database key is legacy. So, we can open our migrate_plus.migration_group.yoursite.yml file and edit it like so:

uuid: e407154d-72e3-4ca5-a5af-eaaf651fe09f
langcode: en
status: true
dependencies: { }
id: yoursite
label: Yoursite
description: 'Migrations for yoursite'
source_type: 'Drupal 7'
module: null
shared_configuration:
  source:
    key: legacy

Drupal does not constantly poll our sync directory for changes. Instead, we need to use the drush config-import

$ drush cim

You should see a list of configuration changes to import, including our migration group. Accept the changes. With that done, we can add our sync directory to git and commit it.

Summary

We may not have written any migrations yet, but we certainly have set the stage. Our migration group now acts as a container for all the future migrations we will create. We’ve imported our legacy database and configured Drupal 8 and our migration group how reach it. More importantly, we’ve started using all the tools necessary to update and run our migrations. Next time, we’ll actually create some migrations for the most essential content of our site, users.

Thanks to our sponsors!

This post was created with the support of my wonderful supporters on Patreon:

  • Alina Mackenzie​
  • Chris Weber

If you like this post, consider becoming a supporter at patreon.com/socketwench.

Thank you!!!