# Native CSV import Goal is to deprecate the old [datainjection](https://plugins.glpi-project.org/#/plugin/datainjection) plugin, that is very old, need adaptation on each major GLPI releases (ex: support of custom assets in 11.0), is not correctly tested and barely maintained. We take the opportunity to rework the UX and UI to have something practical and “pretty”. The new interface will be available under the Tools menu, like the old plugin. I think it’s the best place for it. The menu entry should only appear if the user have a right to add or update a thing in GLPI. We may make a recall in an asset definition dedicated tab. In this case, the mapping step will be initialized on the corresponding itemtype, see in mapping paragraph. :::info Copy and paste existing extra data for searchoptions from datainjection will expose us to copyright issues. Headers of the plugin mentions: > ”@copyright Copyright (C) 2007-2023 by DataInjection plugin team.” We don’t want to add this on all our concerned files. So, it must be preferable for the developer in charge **to not look at the code** while developing the feature in the core. It is still possible to see the plugin in action thought the GLPI interface. ::: ## UI steps **>>> I made a quick mockup here to demo most features I would like to have in this rework:** [**https://orthagh.github.io/glpi-import-csv-mockup/**](https://orthagh.github.io/glpi-import-csv-mockup/) A few basic notes: * We use a wizard style UI. * We **shouldn’t** use numbered step as the total may vary. * We have “Next” and “Prev” buttons when relevant on each step. * Each step should, if relevant, display an information zone to help the users configure its import. * Datainjection plugin use redundant fields for entities_id + is_recursive, I think we should keep the classical way to display these fields (in the header). * To be checked with marketing, if we can have new illustrations for the screen, it could be a great way to enhance the prettiness of the feature ### Start Starting point of the feature, we propose a choice between a fresh setup, or using a saved template. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi81MGE4NWZjNC0zMjBlLTRjN2EtOGYwYy02MzVmY2RiNWNkNTQvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.yRSprsnHIaJbBUR6Vcibsw4cgnBdeljv1NKI2tL3K-o " =1324x750") Even if no template is yet available, we still display the choice to help the discovery of the possibility. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8wMDBkNDhkZi0wNzQ2LTQ3MzEtYmJmYy0wZDgzMjY4MTVmOTgvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.HlAjATVerAclCKL5Suj5T4oaaHovL5d9W007ZtvRpck " =506x211") If at least one exists, we make a list of saved one, and allow deleting them. On selection of one, we continue normally to the next step asking user to upload a file. We limit here with the 5 most recently used templates and make a special page based on Search to see all. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi81ZGRhZWIyYy1lMWJlLTQ4ZTUtYjA3Mi0yMGEzMTFiZGFmZTYvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.Eer0KwE89n7g5CRE-khmimruqzIiJyt_n_H_sybLFQI " =824x517") ### Upload Second step to allow uploading a file ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi9mNzgzZDJlZi1hMmQ1LTRkNjQtOWQyYi1iZDdiYjU1OGU5OTcvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.NHNmvKfqVJ3N4VOIKPUmcJyNcpdxDwNY4rQZtfc8c-g " =1315x633") When a template has been selected, we also provide, above the upload zone, a template for example. This action downloads a CSV file, named as the template name, suffixed as “_example”. We also show an API hint to help CLI usage. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi81MzEzYzczNi02YTE2LTQzN2ItODRjYy1iZTQ2Y2QyNzQxZWUvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.x3fdmEGma0dycO8QlsMURV7kuO8TnYArawHDLi4jfpI " =819x272") When done, we allow the user to adjust a few format options and display a preview zone with the first lines. Any change on formatting options need to be reflected directly on the preview. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8yNjZmZTU3My0wYTgwLTRiYzQtYmZmZi1iOTY5ZmNlMzY5ZmMvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.cFg1mvajg1VMJws_uTKeOOebY9ATk-XNd8BshRn77nc " =864x565") An advanced options zone will be collapsed by default, and propose more controls over the import: ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8zYmEwODU3Mi1iYzhjLTQ5NDItODdhYS1lYzkyNDM5YWJlODcvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.Jf1wjRr0x2brieci6XQCNX2SPMgix9PCzvoTd2b0b1M " =822x493") A few notes about some options: * Date and decimal format, by default re-use values from user’s preferences * “Overwrite fields” is toggle to disable erasing of existing data when “update lines” is enabled. When off, you can only update null fields in the database. * “Append multi-lines” is for comment field, for example, and allow appending the field in the database instead of replacing. * “Import relations” is related to dropdown, datainjection plugin forbid creation of entity without the possibility to change this behavior. I’m pretty ok to continue with that to avoid messing in the entity tree. On the special case when a template has been selected in the previous phase, we make a few changes: * we display a message to alert on the compatibility of the save template with the uploaded file. * If OK, we suggest to directly pass on the import file (directly on parsing + progress bar) * if KO, we make a `.danger` alert to indicate the incompatibility (with details) * The formatting options and the preview are still available if adjustments are need but **collapsed** to orient the UX forward to the most usual process. * If the user continues with the classic “Next” button, we continue the wizard as normal. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8zY2MxMGYwMi1jMjFmLTRhOTAtOTdkZC01ZTkyNDkwNjk2OGQvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.IntgBf9KxHNH7FjwGM2GQQbcXYA5Zl9L3iyj8RshLmQ " =843x383") ### Mapping Next step allows to: * select the itemtype to import, * make the columns mapping, and allowing to define a set of reconciliation columns (how we will match the existing row with data present in database, based on a AND match). * Possibly, let the user add some static fields * and optionally, save the setup as a template for a future usage. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi80OGQ2ZDI0YS1mNDMxLTQ3OTAtOWRhNS1mNmY0YzZlODM4ZGMvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.n6P1Ee89H9eL1kXfSvxgDm4fZaUjqsev2H9YxAxGAvQ " =679x975") The first action is selection of the itemtype where to import the data parsed from the file. We only show itemtypes available to the user. **We should strongly warn the user at this point is the selection of create/update lines doesn’t match its profile’s rights.** Then, we detect columns from the CSV file (the values of the first line, either being columns name or raw values), we display a list of them and show on their side the classic dropdown of search options for the selected item type in the previous step. CF `QueryBuilder::displayCriteria` method. A field should not be selectable for several rows. When one has been selected, we should disable the option in other dropdowns. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi85MjVmZTQ2ZC1jNTcwLTQ5ZmYtYTEwYS1jYmY5ZWVmMjhiMGEvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.-xhRMtZZxeTBV173P8OalPtoA_O9iy5882Sr_LG8I58 " =344x307") Not mandatory, we may try to match automatically fields with columns. At least one column should be selected for reconciliation, otherwise a subtle alert is displayed and the buttons to continue processes (Next and template saving) are disabled. To reduce traction, we may auto-select the first column when configuring a new import (in case of template, reconciliation keys have been saved). ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi84ZWFmNDA2ZS1kMjQ1LTQ5YjktYjU0NC00OWU3ZDkwN2IzN2YvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.kGNrQcpbSiWJTD4QiRg15Uch3T8nyAxdMLnvUskol1w " =694x315") Static field are text inputs the users can set in addition to detected columns. They are filled for each row when the import is processing. We need to not forget to manage them through template’s system. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi83MTMyYTM2Mi0wZGVmLTRiYzEtYWRjNS0zNjI3Yzg3YzJmMjMvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.yVzgQPp55s6UQBbH5G6Y00Jrt2-ytwuqGGid2OIqBsA " =652x175") Fort saving a template, we ask the user a template name, after submit, and we show a confirmation (directly on the button ?). If we come from a template selection in the first step, we allow here to update it. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi82Njk1NmVlNC0wOGYxLTRiOGItODQ4ZS02MWIxMzViNzg4NjEvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.gJaEgFUguxvqUDIiKrZa5kSfA9uJ5QbuzjFc6FoRQ3c " =821x101") ### Import We show a summary before starting really the import. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi80OTA4OTE4Zi01ZTlmLTQzNGUtODAzMy02NzEwOGM3YWU2ODcvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.HD5UHodm-M25eXfjq0E4mGKf_SCX8SeqXj5_ZrK1BM0 " =1310x811") When the “Start import” button is submitted, we move to a progress bar + logs. As done like in installation UI, we should be able to make this working. @[Cédric Anne](mention://40d9d83b-10f4-4dae-808e-7089798d4ed2/user/5e381c46-8015-45ea-a278-ad63f54410a9) should be able to help with this part. The processing should disable PHP timeout to avoid the 30s limit, or at least move to a bigger max. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi9lNTBlZmMzNi0wNjY0LTQwN2QtYjM0Ny00NzlmNGVjZmFhMjkvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.M0naJnMPTiYBUxCMvD_KNurTL0F2hLmO0K8ubFClUkE " =845x461") When the import is done, we display: * a summary of parsed rows with a few cards (a click on them initialize the filter input) * the logs, filterable with an input * a button to export the logs * one for completely restart the import ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi9hZmQyZTEzYS0xNjEzLTQ4YTMtOGFmNC0zYWNiNDVhYTQzNjEvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.EWtrLXQcs_2XvDrxGnpk_g4SsYI7m5kVFS3dQIV_1pg " =870x902") We have the possibility to make a “**dry run**”, it makes at least the reconciliation mechanism in action without importing really the data. We may use a **SQL transaction** to make the thing pertinent. The same summary page is displayed in dry mode, but at the end we propose to make the real import in addition. ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8xYTY2NjJjMi0yY2QwLTQ0MzYtYWVmYy1kMmJhMWQxOTZjYmYvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.ujRy0TrsO8W9LWdJqvpDfg2rR_pH6pqgbdpYuhxB728 " =486x72") ## Tech stuff HLAPI should expose an endpoint to automatize the import. It’s better on this scenario to prepare in advance a template that will be a parameter for the query. No direct CLI feature or command will be proposed, the HLAPI endpoint could be used for that. The UI will hint when using a template: ![](https://outline.teclib.com/api/files.get?sig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJ1cGxvYWRzLzFjOTk2NzhkLTIwNTctNGY5MC04MDY4LWFlOGMwYTU0N2YyZi8zYjMyODc4Yy1mMDZlLTQ1NmEtYmI0MC01NWI3MDUyYjQ0ZjkvaW1hZ2UucG5nIiwidHlwZSI6ImF0dGFjaG1lbnQiLCJpYXQiOjE3OTQ1OTY0MDAsImV4cCI6MTc5NDY4MjgwMH0.5nn3RF9titivI8GDWzpgtj3RQQTHAs-YgXKRd0OaOFM " =858x405") As said earlier in this document, we will use search options as a basis representation of columns. We may consider all to be importable by default. A flag named `noimport` can be set to true to disable its listing in the mapping part. We may also start by only import direct fields and direct foreign key relations (like a ticket category), and then see later for multiple relations, like ticket’s follow-ups. For the moment, no migration from datainjection plugin will be provided. ## Inspirations * * * * *