M2IF — Powerful and flexible solution for Magento 2 imports

Basically Magento 2 provides the functionality for the import of customers, customer addresses, products, prices and since version 2.3.0 also for the MSI stock. However, numerous projects have shown that the standard functionality often does not meet the requirements of reality, especially in the areas of handling, expandability, stability and performance and in particular with comprehensive projects.

From the point of view of a Magento Solution Partner, but also as a shop operator, there are good reasons for choosing M2IF instead of one of the available extensions for Magento 2 Import.

www.m2if.com

Thus, in no larger project will the shop owner upload and import his products in the backend, if he can create a file in the required CSV format at all. For this, the standard functionality offered by Magento 2 is only suitable to a limited extent, as especially in case of extensive imports, the file size almost always exceeds the maximum upload size, which leads to a termination, on the other hand, the browser almost always runs into a timeout, especially in case of larger files, whereby one does not get any feedback about success or failure of the import. In addition to these issues, the fact that this is a manual process almost always represents an exclusion criterion.

In a professional environment, import processes are almost always automated. If the data volume is small, it is usually transferred directly from the PIM or ERP to Magento 2 via a web service. For larger data volumes, e.g. more than 10,000 SKUs, these are in most cases stored as CSV or XML files on the server and automatically imported from there by a Magento extension, which in turn is triggered by a CRON job. These procedures have been established for a long time, but Magento 2 does not or only partially provide the necessary functionality for this and must be extended by appropriate extensions. This is where M2IF comes into play. The term stands for “Magento 2 Import Framework”, a very powerful import tool for Magento 2, which we have developed based on our extensive Magento 2 experience, which is now successfully used in a variety of Magento 2 projects in practice and which we provide as an open source tool under Github for free download.

In the following, the reasons for the chosen approach are explained in more detail on the basis of frequently recurring import problems known to some readers.

Problem 1: Monolithic approach

In contrast to the known solutions, M2IF deliberately does not use the way of a Magento extension, even if an integration is technically not a problem, but that of a decoupled, lightweight solution based on Symfony. Decoupled because it can also be installed via Composer with Magento, but this is not the preferred way. Instead, M2IF offers two installation options that are optimized for the respective application. Firstly, M2IF can be downloaded directly as a PHAR file and executed via the console, which is particularly suitable for small projects or a test. On the other hand, it can be installed as a standalone application with Composer and called via its own script. The last option is particularly suitable for projects in which M2IF is used as a framework and considerably extended or adapted, since own libraries can be conveniently installed via Composer.

The advantage of a decoupled solution is that it runs without the complete Magento stack on another server, so the web server cannot be loaded and deployed separately during the imports, a kind of micro service. For performance reasons, the database is not accessed via Magento’s RESTFul API, but directly.

Magento itself seems to be thinking about a kind of micro-framework that allows such an approach, even though there is no official statement yet.

Problem 2: Poor use of resources

As a result of the problem of the monolithic approach, in many projects the available resources can only be used poorly or not optimally. Due to the modular architecture not only flexibility can be guaranteed, but M2IF is also suitable for the use in distributed systems and enables an optimal use of the available resources.

Through appropriate modeling, the previously described, independently executable components can be executed either in parallel on one system or, and this is unique, on different systems. The parallel execution on one system allows the optimal use of existing resources such as memory + CPU. If the processes are distributed to different systems, it can be ensured, for example, that a CPU + memory intensive process does not unnecessarily load the web server and thus impair system availability. In addition, the reliability can be increased or even established by distributing the data to several systems.

Problem 3: Inconsistent product data

Another common problem is data inconsistency. The functionality provided by Magento, on which many other solutions are based, does not allow imports to be carried out as part of a transaction. This means that in the event of a termination, e.g. due to a deadlock caused by other processes, the previously imported data is available, but the remaining data is not. The catalog is therefore inconsistent, prices are not current, or warehouse stocks are invalid.

Depending on the application, inconsistent data may be accepted, especially if imports take a long time and a transaction blocks tables that cannot be written to by other components, resulting in deadlocks. In this case it is essential to coordinate the import processes in such a way that competing processes are excluded. A high performance import is therefore inevitable if data consistency plays a role. Although M2IF has already been optimized for performance, the single transaction mode, in addition to data consistency, also increases performance. The Single-Transaction Mode, for example, increases performance by 10–30%, depending on the scenario.

Problem 4: Performance + Memory usage

As mentioned before, M2IF not only extends the standard functionality provided by Magento or uses the API, but goes its own way. The standard functionality and therefore also many of the common extensions read the complete file into the memory, which leads to a not insignificant memory consumption and the associated problems with larger amounts of data. In order to optimize the performance to a maximum and the memory consumption to a minimum, M2IF works on the basis of streams. This ensures that only the data set to be processed and not the complete file is loaded into memory.

In addition, M2IF uses a lightweight database layer instead of one of the common ORMs, which maps entities not as objects but as arrays. This approach also supports performance, optimizes memory consumption, and reduces complexity when developing extensions as much as possible.

Problem 5: Imports are almost always individual

In almost all cases, the shop operator cannot or will not provide the format specified by the extension via the PIM or the ERP, often for cost reasons. The Solution Partner must then ensure that the data is transformed into the desired format.

M2IF supports files in CSV format by default, in case of products, customers, customer addresses + MSI the format is 100% compatible with the Magento standard. In the case of categories, attributes, attribute sets and attribute groups, the format strongly follows the standard. For EAV entities such as products, categories, customers and customer addresses, new columns can be dynamically inserted into the CSV, if the column name matches the attribute code, the values will be imported automatically. If the column name does not match the attribute code, a mapping of column names to the desired attribute code can be carried out via the configuration, so programming is not necessary in many cases.

If the shop operator cannot supply the data in the appropriate format, or if the configuration options are not sufficient to map the data as required, M2IF can be easily extended in the form of plug-ins to meet required requirements and supports the developers with existing functionality in the form of various libraries.

Problem 6: Lack of flexibility of existing solutions

The flexibility of the standard Magento Standard import/export solution has proven to be one of the biggest problems and cost drivers in numerous projects. Especially the high complexity and missing component orientation in combination with a difficult extensibility and adaptability leads to complex and error-prone solutions in many projects.

In contrast, M2IF relies on a completely component-based architecture. For example, the functionality for importing products was not implemented as a self-contained component, but is subdivided into the following components:

• Simples
• Bundles
• Configurables
• Grouped
• Links (Accessories, Up-/Cross-Sell and additional Linktypes)
• Media (Pictures, Videos, etc.)
• URL Rewrites
• MSI Inventory
• Tier Prices

Each of these components can either be used independently, e.g. an import of URL rewrites, or in any combination to import products in all possible constellations. This component-based architecture gives the developer the possibility to implement a customized solution within the requirements of the project. The individual components can be completely configured via the underlying workflow engine and combined in almost any constellation.

If, for example, products are imported, it often happens that the customer only wants to import simple and configurable products. The functionality for bundles, grouped products, images, cross- and upsell, accessories, images can then be deactivated purely configuratively.

For projects based on the standard, M2IF provides an immediately usable tool with a CLI application, with which the automation of the import processes in the project can be implemented quickly and easily. If the shop operator can deliver the data in standard format, the complete import process can be integrated without additional programming effort and at a minimum cost.

Conclusion

The problems described above certainly do not occur in all projects, but in most cases only a part of them and then in almost always different forms.

With the open source solution M2IF, which is freely available on Github, these and other problems can be solved quickly and efficiently. With M2IF, TechDivision GmbH has set itself the goal of making life easier for end customers, shop operators, solution partners and, last but not least, developers. Thus the continuous further development is guaranteed by the employment in own projects. In addition, nothing stands in the way of building a large community that forms the basis for a successful project thanks to the open source approach.

Based on M2IF, TechDivision GmbH has been providing a solution with its new Magento integration platform “Pacemaker” since June 2019 that can be used to solve the above-mentioned problems. Solution partners and shop operators in particular are also supported in the modeling and visualization of processes, such as the import of product data. In addition to functionality, “Pacemaker” also has the know-how, for example how an import process for the catalogue must be structured in order to avoid incorrect product data, stock levels or prices.

By standardizing import processes, as M2IF supports them, time and money can be saved on the one hand, and on the other hand exactly this saved budget can be invested in the functionality required by the shop operator and desired by the end customer. Through the use of M2IF, additional “business value” can be created for the shop operator and the end customer!

Additional links

• Website M2IF — https://m2if.com
• Website Pacemaker — https://www.techdivision.com/en/lp/pacemaker-for-magento