[6.x] Feature/import

[i-just]

It imports data from a JSON, CSV or XML files into Craft CMS Elements or opted-in Eloquent models.

Core concepts:

• Data Types => what data sources this feature can work with
• Transformers => mechanism for normalising, manipulating and sometimes even mapping incoming data to existing "slots"
• Importers => they do the heavy lifting, they specialise in actually knowing how to write imported data into a specific kind of destination
• Import Config => stores import configuration and can be used to instantiate an Importer
• Import Runs => are sets of import configs that can be used to perform multi-step imports; it’s also where you can choose to run each step at certain batch sizes

Extensibility:

• you can register extra data types and importer types
• you can manipulate your data any way you want via transformers

Data Types:

• there are 3 built-in data types: JSON, XML and CSV

Transformers:

• there are 3 built-in transformers, all related to Element importing:
  • ElementTransformer => used by default by all elements
  • EntryTransformer => default when importing Entries
  • AssetTransformer => default when importing Assets
• you can create your own transformers and use them to manipulate the data however you wish; e.g. you can choose to simply uppercase all values that should be imported to myPlainTextField, or you can do complex checks to determine what value should be used for myOtherField

Importers:

• ElementImporter — target must be a registered element type whose isImportable() returns true. Adds site (required) and fieldLayout.
• ModelImporter — target must implement the marker interface ImportableModelInterface and must not be an element type.
• Extra types can be registered via the RegisterImporterTypes event.

Import configs:

• there are 2 types of import configs:
  • editable => created via the Control Panel and stored in the database
  • non-editable => file-based, stored in config/craft/import.php and committed to your repository
• it’s where you:
  • choose the Importer type and define the information that it needs to operate; e.g. you choose an element type you wish to import into for the Element Importer or provide a class name for the Model Importer;
  • specify the file that holds the data you want to import
  • provide optional information, like which Transformer you’d like to use
  • define which importable attribute and field maps to which incoming value (map)
  • choose which values should be used to match against existing elements (matchCriteria)
  • choose which values should be cleared out if they’re empty or missing from the incoming data (clearableItems)
  • choose whether you’d like to keep nested content when it is missing from the incoming data (keepMissingNestedElements)
• when using editable (UI-based) config, the field layout provider is defined once per config and can’t be overwritten via the data

Match criteria — three tiers, last wins

1. Config-level — from the UI or file config. Resolved once per run.
2. In the data — a matchCriteria key on the row, [systemHandle => incomingKey].
   Also looks inside $data['fields'].
3. Transformer — additionalMatchCriteria(), already resolved to actual values.

Clearable items

An array similar to match criteria. If a field or attribute is

• marked clearable and the value missing/empty in the incoming data, it’s forced to null and explicitly applied
• not marked clearable and the value is empty in the incoming data, its key is unset() entirely, so the existing value is left untouched.

Import runs:

A run chains ordered steps. Each step consists of Import Config and an optional batch size.
Steps run sequentially, and one bad step doesn’t kill the run. Per-item failures are
logged and skipped, not fatal.

Importing:

• you can trigger import via the UI
  • CP > Import > Runs > Start => for any Import Config
  • CP > Import > Configs => for file-based Import Configs
• or the CLI commands:
  • artisan craft:import:element
  • artisan craft:import:model

Making things importable:

• elements are importable by default and opt out; models are opt-in only
• Elements
  • use CraftCms\Cms\Component\Concerns\Importable and CraftCms\Cms\Component\Contracts\ImportableInterface
  • there’s an isImportable() method, true by default, that elements can use to opt out (like Address element does); NOTE that Addresses are still importable, just not on their own - always as part of importing into their owner
  • CraftCms\Cms\Support\Attributes\Importable is used to mark an element's properties as importable
  • CraftCms\Cms\FieldLayout\Contracts\ImportableFieldLayoutElementInterface - used to make field layout elements importable
  • CraftCms\Cms\Field\Contracts\ImportableElementContainerFieldInterface - used by container fields so that they can be imported correctly
• Models
  • need to implement CraftCms\Cms\Shared\Contracts\ImportableModelInterface to be importable into

Permissions:

There’s a permissions group import, with the following permissions: viewImportConfigs, saveImportConfigs, deleteImportConfigs, viewImportRuns, saveImportRuns, deleteImportRuns, triggerImportRuns.

Related issues

CMS-2390

[linear-code]

CMS-2390

[github-actions]

📚 Storybook previews

@craftcms/ui — open Storybook

No changed components detected in this Storybook.

resources/js — open Storybook

No changed components detected in this Storybook.