[!!!][TASK] Flex form data structure refactoring 12/48212/33
authorChristian Kuhn <lolli@schwarzbu.ch>
Wed, 18 May 2016 15:58:06 +0000 (17:58 +0200)
committerSusanne Moog <susanne.moog@typo3.org>
Mon, 7 Nov 2016 17:06:23 +0000 (18:06 +0100)
Method BackendUtility::getFlexFormDS() does two things at once:
It finds a data structure by given data (TCA, row, ...) and then
parses it.
This construct gives tons of headaches, since the methods never
exposes where a specific data structure came from and the lookup
mechanism is complex. Especially if a flex form is used in
combination with ajax requests later, the core has massive issues
since the location can not be found out later again.

To solve that, the patch splits getFlexFormDS() into two methods:
First method "FlexFormTools->getDataStructureIdentifier()" gets
TCA and row and locates a specific structure. It returns an
"identifier" that points to that unique data structure. This
identifier can be later hand around easily.
The second method "FlexFormTools->parseDataStructureByIdentifier()"
then gets this identifier again, fetches the data structure the
identifier points to, and parses it.

Change-Id: I38264e8a4a6f956c12e9e50f6039d3d09af0f03a
Resolves: #78581
Releases: master
Reviewed-on: https://review.typo3.org/48212
Tested-by: TYPO3com <no-reply@typo3.com>
Tested-by: Claus Due <claus@phpmind.net>
Reviewed-by: Wouter Wolters <typo3@wouterwolters.nl>
Reviewed-by: Susanne Moog <susanne.moog@typo3.org>
Tested-by: Susanne Moog <susanne.moog@typo3.org>
47 files changed:
typo3/sysext/backend/Classes/Form/FormDataProvider/AbstractItemProvider.php
typo3/sysext/backend/Classes/Form/FormDataProvider/TcaFlexFetch.php [deleted file]
typo3/sysext/backend/Classes/Form/FormDataProvider/TcaFlexPrepare.php
typo3/sysext/backend/Classes/Form/FormDataProvider/TcaFlexProcess.php
typo3/sysext/backend/Classes/Form/Wizard/SuggestWizard.php
typo3/sysext/backend/Classes/Utility/BackendUtility.php
typo3/sysext/backend/Tests/Unit/Form/FormDataProvider/TcaFlexFetchTest.php [deleted file]
typo3/sysext/backend/Tests/Unit/Form/FormDataProvider/TcaFlexPrepareTest.php
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/AbstractInvalidDataStructureException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidCombinedPointerFieldException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidIdentifierException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowLoopException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowRootException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidPointerFieldValueException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidSinglePointerFieldException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidTcaException.php [new file with mode: 0644]
typo3/sysext/core/Classes/Configuration/FlexForm/FlexFormTools.php
typo3/sysext/core/Classes/DataHandling/DataHandler.php
typo3/sysext/core/Classes/Database/ReferenceIndex.php
typo3/sysext/core/Classes/Utility/GeneralUtility.php
typo3/sysext/core/Configuration/DefaultConfiguration.php
typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FlexFormToolsPublicPropertiesDropped.rst [new file with mode: 0644]
typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FormEngineTcaFlexFetchDataProviderRemoved.rst [new file with mode: 0644]
typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-HookGetFlexFormDSClassNoLongerCalled.rst [new file with mode: 0644]
typo3/sysext/core/Documentation/Changelog/master/Deprecation-78581-FlexFormRelatedParsing.rst [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnArray.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnEmptyArray.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnString.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookThrowException.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnArray.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnEmptyArray.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnString.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookThrowException.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureOfSingleSheet.xml [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnArray.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnString.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookThrowException.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnEmptyString.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnObject.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnString.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookThrowException.php [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureWithSheet.xml [new file with mode: 0644]
typo3/sysext/core/Tests/Unit/Configuration/FlexForm/FlexFormToolsTest.php
typo3/sysext/core/Tests/Unit/DataHandling/DataHandlerTest.php
typo3/sysext/impexp/Classes/Import.php
typo3/sysext/lowlevel/Classes/Command/LostFilesCommand.php

index 353c97d..26f99f6 100644 (file)
@@ -18,6 +18,7 @@ use Doctrine\DBAL\DBALException;
 use TYPO3\CMS\Backend\Module\ModuleLoader;
 use TYPO3\CMS\Backend\Utility\BackendUtility;
 use TYPO3\CMS\Core\Authentication\BackendUserAuthentication;
+use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
 use TYPO3\CMS\Core\Database\ConnectionPool;
 use TYPO3\CMS\Core\Database\Query\QueryBuilder;
 use TYPO3\CMS\Core\Database\Query\QueryHelper;
@@ -715,10 +716,10 @@ abstract class AbstractItemProvider
                 if (!empty($GLOBALS['TCA'][$table]['columns'][$tableField]['label'])) {
                     $labelPrefix = $languageService->sL($GLOBALS['TCA'][$table]['columns'][$tableField]['label']);
                 }
-                // Get all sheets and title
+                // Get all sheets
                 foreach ($flexForms as $extIdent => $extConf) {
                     // Get all fields in sheet
-                    foreach ($extConf['ds']['sheets'] as $sheetName => $sheet) {
+                    foreach ($extConf['sheets'] as $sheetName => $sheet) {
                         if (empty($sheet['ROOT']['el']) || !is_array($sheet['ROOT']['el'])) {
                             continue;
                         }
@@ -763,76 +764,40 @@ abstract class AbstractItemProvider
     }
 
     /**
-     * Returns all registered FlexForm definitions with title and fields
+     * Returns all registered FlexForm definitions
+     *
+     * Note: This only finds flex forms registered in 'ds' config sections.
+     * This does not resolve other sophisticated flex form data structure references.
      *
      * @param string $table Table to handle
-     * @return array Data structures with speaking extension title
+     * @return array Data structures
      */
     protected function getRegisteredFlexForms($table)
     {
         if (empty($table) || empty($GLOBALS['TCA'][$table]['columns'])) {
             return [];
         }
+        $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
         $flexForms = [];
         foreach ($GLOBALS['TCA'][$table]['columns'] as $tableField => $fieldConf) {
             if (!empty($fieldConf['config']['type']) && !empty($fieldConf['config']['ds']) && $fieldConf['config']['type'] == 'flex') {
                 $flexForms[$tableField] = [];
-                // Get pointer fields
-                $pointerFields = !empty($fieldConf['config']['ds_pointerField']) ? $fieldConf['config']['ds_pointerField'] : 'list_type,CType,default';
-                $pointerFields = GeneralUtility::trimExplode(',', $pointerFields);
-                // Get FlexForms
-                foreach ($fieldConf['config']['ds'] as $flexFormKey => $dataStructure) {
+                foreach (array_keys($fieldConf['config']['ds']) as $flexFormKey) {
                     // Get extension identifier (uses second value if it's not empty, "list" or "*", else first one)
                     $identFields = GeneralUtility::trimExplode(',', $flexFormKey);
                     $extIdent = $identFields[0];
+                    // @todo: This approach is limited and doesn't find everything. It works for tt_content plugins, though.
                     if (!empty($identFields[1]) && $identFields[1] !== 'list' && $identFields[1] !== '*') {
                         $extIdent = $identFields[1];
                     }
-                    // Load external file references
-                    if (!is_array($dataStructure)) {
-                        $file = GeneralUtility::getFileAbsFileName(str_ireplace('FILE:', '', $dataStructure));
-                        if ($file && @is_file($file)) {
-                            $dataStructure = file_get_contents($file);
-                        }
-                        $dataStructure = GeneralUtility::xml2array($dataStructure);
-                        if (!is_array($dataStructure)) {
-                            continue;
-                        }
-                    }
-                    // Get flexform content
-                    $dataStructure = GeneralUtility::resolveAllSheetsInDS($dataStructure);
-                    if (empty($dataStructure['sheets']) || !is_array($dataStructure['sheets'])) {
-                        continue;
-                    }
-                    // Use DS pointer to get extension title from TCA
-                    // @todo: I don't understand this code ... does it make sense at all?
-                    $title = $extIdent;
-                    $keyFields = GeneralUtility::trimExplode(',', $flexFormKey);
-                    foreach ($pointerFields as $pointerKey => $pointerName) {
-                        if (empty($keyFields[$pointerKey])
-                            || $keyFields[$pointerKey] === '*'
-                            || $keyFields[$pointerKey] === 'list'
-                            || $keyFields[$pointerKey] === 'default'
-                        ) {
-                            continue;
-                        }
-                        if (!empty($GLOBALS['TCA'][$table]['columns'][$pointerName]['config']['items'])) {
-                            $items = $GLOBALS['TCA'][$table]['columns'][$pointerName]['config']['items'];
-                            if (!is_array($items)) {
-                                continue;
-                            }
-                            foreach ($items as $itemConf) {
-                                if (!empty($itemConf[0]) && !empty($itemConf[1]) && $itemConf[1] == $keyFields[$pointerKey]) {
-                                    $title = $itemConf[0];
-                                    break 2;
-                                }
-                            }
-                        }
-                    }
-                    $flexForms[$tableField][$extIdent] = [
-                        'title' => $title,
-                        'ds' => $dataStructure
-                    ];
+                    $flexFormDataStructureIdentifier = json_encode([
+                        'type' => 'tca',
+                        'tableName' => $table,
+                        'fieldName' => $tableField,
+                        'dataStructureKey' => $flexFormKey,
+                    ]);
+                    $dataStructure = $flexFormTools->parseDataStructureByIdentifier($flexFormDataStructureIdentifier);
+                    $flexForms[$tableField][$extIdent] = $dataStructure;
                 }
             }
         }
diff --git a/typo3/sysext/backend/Classes/Form/FormDataProvider/TcaFlexFetch.php b/typo3/sysext/backend/Classes/Form/FormDataProvider/TcaFlexFetch.php
deleted file mode 100644 (file)
index 9a0107f..0000000
+++ /dev/null
@@ -1,143 +0,0 @@
-<?php
-namespace TYPO3\CMS\Backend\Form\FormDataProvider;
-
-/*
- * This file is part of the TYPO3 CMS project.
- *
- * It is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License, either version 2
- * of the License, or any later version.
- *
- * For the full copyright and license information, please read the
- * LICENSE.txt file that was distributed with this source code.
- *
- * The TYPO3 project - inspiring people to share!
- */
-
-use TYPO3\CMS\Backend\Form\FormDataProviderInterface;
-use TYPO3\CMS\Backend\Utility\BackendUtility;
-use TYPO3\CMS\Core\Utility\GeneralUtility;
-
-/**
- * Resolve and flex data structure and data values.
- *
- * This is the first data provider in the chain of flex form related providers.
- */
-class TcaFlexFetch implements FormDataProviderInterface
-{
-    /**
-     * Resolve ds pointer stuff and parse both ds and dv
-     *
-     * @param array $result
-     * @return array
-     */
-    public function addData(array $result)
-    {
-        foreach ($result['processedTca']['columns'] as $fieldName => $fieldConfig) {
-            if (empty($fieldConfig['config']['type']) || $fieldConfig['config']['type'] !== 'flex') {
-                continue;
-            }
-            $result = $this->initializeDataStructure($result, $fieldName);
-            $result = $this->initializeDataValues($result, $fieldName);
-            $result = $this->resolvePossibleExternalFileInDataStructure($result, $fieldName);
-        }
-
-        return $result;
-    }
-
-    /**
-     * Fetch / initialize data structure.
-     *
-     * The sub array with different possible data structures in ['config']['ds'] is
-     * resolved here, ds array contains only the one resolved data structure after this method.
-     *
-     * @param array $result Result array
-     * @param string $fieldName Currently handled field name
-     * @return array Modified result
-     * @throws \UnexpectedValueException
-     */
-    protected function initializeDataStructure(array $result, $fieldName)
-    {
-        // Fetch / initialize data structure
-        $dataStructureArray = BackendUtility::getFlexFormDS(
-            $result['processedTca']['columns'][$fieldName]['config'],
-            $result['databaseRow'],
-            $result['tableName'],
-            $fieldName
-        );
-        // If data structure can't be parsed, this is a developer error, so throw a non catchable exception
-        if (!is_array($dataStructureArray)) {
-            throw new \UnexpectedValueException(
-                'Data structure error: ' . $dataStructureArray,
-                1440506893
-            );
-        }
-        if (!isset($dataStructureArray['meta']) || !is_array($dataStructureArray['meta'])) {
-            $dataStructureArray['meta'] = [];
-        }
-        // This kicks one array depth:  config['ds']['matchingIdentifier'] becomes config['ds']
-        $result['processedTca']['columns'][$fieldName]['config']['ds'] = $dataStructureArray;
-        return $result;
-    }
-
-    /**
-     * Parse / initialize value from xml string to array
-     *
-     * @param array $result Result array
-     * @param string $fieldName Currently handled field name
-     * @return array Modified result
-     */
-    protected function initializeDataValues(array $result, $fieldName)
-    {
-        if (!array_key_exists($fieldName, $result['databaseRow'])) {
-            $result['databaseRow'][$fieldName] = '';
-        }
-        $valueArray = [];
-        if (isset($result['databaseRow'][$fieldName])) {
-            $valueArray = $result['databaseRow'][$fieldName];
-        }
-        if (!is_array($result['databaseRow'][$fieldName])) {
-            $valueArray = GeneralUtility::xml2array($result['databaseRow'][$fieldName]);
-        }
-        if (!is_array($valueArray)) {
-            $valueArray = [];
-        }
-        if (!isset($valueArray['data'])) {
-            $valueArray['data'] = [];
-        }
-        if (!isset($valueArray['meta'])) {
-            $valueArray['meta'] = [];
-        }
-        $result['databaseRow'][$fieldName] = $valueArray;
-        return $result;
-    }
-
-    /**
-     * Single fields can be extracted to files again. This is resolved and parsed here.
-     *
-     * @todo: Why is this not done in BackendUtility::getFlexFormDS() directly? If done there, the two methods
-     * @todo: GeneralUtility::resolveSheetDefInDS() and GeneralUtility::resolveAllSheetsInDS() could be killed
-     * @todo: since this resolving is basically the only really useful thing they actually do.
-     *
-     * @param array $result Result array
-     * @param string $fieldName Current handle field name
-     * @return array Modified item array
-     */
-    protected function resolvePossibleExternalFileInDataStructure(array $result, $fieldName)
-    {
-        $modifiedDataStructure = $result['processedTca']['columns'][$fieldName]['config']['ds'];
-        if (isset($modifiedDataStructure['sheets']) && is_array($modifiedDataStructure['sheets'])) {
-            foreach ($modifiedDataStructure['sheets'] as $sheetName => $sheetStructure) {
-                if (!is_array($sheetStructure)) {
-                    $file = GeneralUtility::getFileAbsFileName($sheetStructure);
-                    if ($file && @is_file($file)) {
-                        $sheetStructure = GeneralUtility::xml2array(file_get_contents($file));
-                    }
-                }
-                $modifiedDataStructure['sheets'][$sheetName] = $sheetStructure;
-            }
-        }
-        $result['processedTca']['columns'][$fieldName]['config']['ds'] = $modifiedDataStructure;
-        return $result;
-    }
-}
index 4668434..339b811 100644 (file)
@@ -15,13 +15,14 @@ namespace TYPO3\CMS\Backend\Form\FormDataProvider;
  */
 
 use TYPO3\CMS\Backend\Form\FormDataProviderInterface;
+use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
 use TYPO3\CMS\Core\Migrations\TcaMigration;
 use TYPO3\CMS\Core\Utility\GeneralUtility;
 
 /**
- * Prepare flex data structure and data values.
+ * Resolve flex data structure and data values, prepare and normalize.
  *
- * This data provider is typically executed directly after TcaFlexFetch
+ * This is the first data provider in the chain of flex form related providers.
  */
 class TcaFlexPrepare implements FormDataProviderInterface
 {
@@ -40,7 +41,8 @@ class TcaFlexPrepare implements FormDataProviderInterface
             if (empty($fieldConfig['config']['type']) || $fieldConfig['config']['type'] !== 'flex') {
                 continue;
             }
-            $result = $this->createDefaultSheetInDataStructureIfNotGiven($result, $fieldName);
+            $result = $this->initializeDataStructure($result, $fieldName);
+            $result = $this->initializeDataValues($result, $fieldName);
             $result = $this->removeTceFormsArrayKeyFromDataStructureElements($result, $fieldName);
             $result = $this->migrateFlexformTcaDataStructureElements($result, $fieldName);
         }
@@ -49,31 +51,65 @@ class TcaFlexPrepare implements FormDataProviderInterface
     }
 
     /**
-     * Add a sheet structure if data structure has none yet to simplify further handling.
+     * Fetch / initialize data structure.
      *
-     * Example TCA field config:
-     * ['config']['ds']['ROOT'] becomes
-     * ['config']['ds']['sheets']['sDEF']['ROOT']
+     * The sub array with different possible data structures in ['config']['ds'] is
+     * resolved here, ds array contains only the one resolved data structure after this method.
      *
      * @param array $result Result array
      * @param string $fieldName Currently handled field name
      * @return array Modified result
      * @throws \UnexpectedValueException
      */
-    protected function createDefaultSheetInDataStructureIfNotGiven(array $result, $fieldName)
+    protected function initializeDataStructure(array $result, $fieldName)
     {
-        $modifiedDataStructure = $result['processedTca']['columns'][$fieldName]['config']['ds'];
-        if (isset($modifiedDataStructure['ROOT']) && isset($modifiedDataStructure['sheets'])) {
-            throw new \UnexpectedValueException(
-                'Parsed data structure has both ROOT and sheets on top level',
-                1440676540
-            );
+        $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+        $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+            $result['processedTca']['columns'][$fieldName],
+            $result['tableName'],
+            $fieldName,
+            $result['databaseRow']
+        );
+        $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
+        if (!isset($dataStructureArray['meta']) || !is_array($dataStructureArray['meta'])) {
+            $dataStructureArray['meta'] = [];
         }
-        if (isset($modifiedDataStructure['ROOT']) && is_array($modifiedDataStructure['ROOT'])) {
-            $modifiedDataStructure['sheets']['sDEF']['ROOT'] = $modifiedDataStructure['ROOT'];
-            unset($modifiedDataStructure['ROOT']);
+        // This kicks one array depth:  config['ds']['listOfDataStructures'] becomes config['ds']
+        // This also ensures the final ds can be found in 'ds', even if the DS was fetch from
+        // a record, see FlexFormTools->getDataStructureIdentifier() for details.
+        $result['processedTca']['columns'][$fieldName]['config']['ds'] = $dataStructureArray;
+        return $result;
+    }
+
+    /**
+     * Parse / initialize value from xml string to array
+     *
+     * @param array $result Result array
+     * @param string $fieldName Currently handled field name
+     * @return array Modified result
+     */
+    protected function initializeDataValues(array $result, $fieldName)
+    {
+        if (!array_key_exists($fieldName, $result['databaseRow'])) {
+            $result['databaseRow'][$fieldName] = '';
         }
-        $result['processedTca']['columns'][$fieldName]['config']['ds'] = $modifiedDataStructure;
+        $valueArray = [];
+        if (isset($result['databaseRow'][$fieldName])) {
+            $valueArray = $result['databaseRow'][$fieldName];
+        }
+        if (!is_array($result['databaseRow'][$fieldName])) {
+            $valueArray = GeneralUtility::xml2array($result['databaseRow'][$fieldName]);
+        }
+        if (!is_array($valueArray)) {
+            $valueArray = [];
+        }
+        if (!isset($valueArray['data'])) {
+            $valueArray['data'] = [];
+        }
+        if (!isset($valueArray['meta'])) {
+            $valueArray['meta'] = [];
+        }
+        $result['databaseRow'][$fieldName] = $valueArray;
         return $result;
     }
 
index e2f42f1..ada1c46 100644 (file)
@@ -71,6 +71,7 @@ class TcaFlexProcess implements FormDataProviderInterface
      *
      * @todo: This method is only implemented half. It basically should do all the
      * @todo: pointer handling that is done within BackendUtility::getFlexFormDS() to $srcPointer.
+     * @todo: This can be solved now by adding 'identifier' from TcaFlexPrepare to 'config' array
      *
      * @param array $result Result array
      * @param string $fieldName Current handle field name
index 631c973..d40ef87 100644 (file)
@@ -17,6 +17,7 @@ namespace TYPO3\CMS\Backend\Form\Wizard;
 use Psr\Http\Message\ResponseInterface;
 use Psr\Http\Message\ServerRequestInterface;
 use TYPO3\CMS\Backend\Utility\BackendUtility;
+use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
 use TYPO3\CMS\Core\Database\ConnectionPool;
 use TYPO3\CMS\Core\Utility\ArrayUtility;
 use TYPO3\CMS\Core\Utility\GeneralUtility;
@@ -263,8 +264,7 @@ class SuggestWizard
                 return;
             }
 
-            $flexfieldTCAConfig = $GLOBALS['TCA'][$table]['columns'][$parts[0]]['config'];
-            // @todo: should be done via data preparation, resolveAllSheetsInDS() can be deprecated then
+            $flexfieldTCAConfig = $GLOBALS['TCA'][$table]['columns'][$parts[0]];
             if (substr($row['uid'], 0, 3) === 'NEW') {
                 // We have to cleanup record information as they are coming from FormEngines DataProvider
                 $pointerFields = GeneralUtility::trimExplode(',', $flexfieldTCAConfig['ds_pointerField']);
@@ -274,10 +274,17 @@ class SuggestWizard
                     }
                 }
             }
-            $flexformDSArray = BackendUtility::getFlexFormDS($flexfieldTCAConfig, $row, $table, $parts[0]);
-            $flexformDSArray = GeneralUtility::resolveAllSheetsInDS($flexformDSArray);
-
-            $fieldConfig = $this->getFieldConfiguration($parts, $flexformDSArray);
+            // @todo: Better hand around the data structure identifier. This would free us from $row usage
+            // @todo: and getDataStructureIdentifier() would not have to be called anymore at all.
+            $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+            $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                $flexfieldTCAConfig,
+                $table,
+                $parts[0],
+                $row
+            );
+            $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
+            $fieldConfig = $this->getFieldConfiguration($parts, $dataStructureArray);
             // Flexform field name levels are separated with | instead of encapsulation in [];
             // reverse this here to be compatible with regular field names.
             $field = str_replace('|', '][', $field);
index 1402fac..bd930a6 100644 (file)
@@ -954,10 +954,11 @@ class BackendUtility
      * @param bool $WSOL If set, workspace overlay is applied to records. This is correct behaviour for all presentation and export, but NOT if you want a TRUE reflection of how things are in the live workspace.
      * @param int $newRecordPidValue SPECIAL CASES: Use this, if the DataStructure may come from a parent record and the INPUT row doesn't have a uid yet (hence, the pid cannot be looked up). Then it is necessary to supply a PID value to search recursively in for the DS (used from DataHandler)
      * @return mixed If array, the data structure was found and returned as an array. Otherwise (string) it is an error message.
-     * @todo: All those nasty details should be covered with tests, also it is very unfortunate the final $srcPointer is not exposed
+     * @deprecated since TYPO3 v8, will be removed in TYPO3 v9. This is now integrated as FlexFormTools->getDataStructureIdentifier()
      */
     public static function getFlexFormDS($conf, $row, $table, $fieldName = '', $WSOL = true, $newRecordPidValue = 0)
     {
+        GeneralUtility::logDeprecatedFunction();
         // Get pointer field etc from TCA-config:
         $ds_pointerField = $conf['ds_pointerField'];
         $ds_array = $conf['ds'];
@@ -1102,6 +1103,8 @@ class BackendUtility
             $dataStructArray = 'No proper configuration!';
         }
         // Hook for post-processing the Flexform DS. Introduces the possibility to configure Flexforms via TSConfig
+        // This hook isn't called anymore from within the core, the whole method is deprecated.
+        // There are alternative hooks, see FlexFormTools->getDataStructureIdentifier() and ->parseDataStructureByIdentifier()
         if (is_array($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_befunc.php']['getFlexFormDSClass'])) {
             foreach ($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_befunc.php']['getFlexFormDSClass'] as $classRef) {
                 $hookObj = GeneralUtility::getUserObj($classRef);
diff --git a/typo3/sysext/backend/Tests/Unit/Form/FormDataProvider/TcaFlexFetchTest.php b/typo3/sysext/backend/Tests/Unit/Form/FormDataProvider/TcaFlexFetchTest.php
deleted file mode 100644 (file)
index b921d25..0000000
+++ /dev/null
@@ -1,273 +0,0 @@
-<?php
-namespace TYPO3\CMS\Backend\Tests\Unit\Form\FormDataProvider;
-
-/*
- * This file is part of the TYPO3 CMS project.
- *
- * It is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License, either version 2
- * of the License, or any later version.
- *
- * For the full copyright and license information, please read the
- * LICENSE.txt file that was distributed with this source code.
- *
- * The TYPO3 project - inspiring people to share!
- */
-
-use Prophecy\Argument;
-use Prophecy\Prophecy\ObjectProphecy;
-use TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch;
-use TYPO3\CMS\Core\Authentication\BackendUserAuthentication;
-use TYPO3\CMS\Core\Cache\CacheManager;
-use TYPO3\CMS\Core\Cache\Frontend\FrontendInterface;
-use TYPO3\CMS\Core\Tests\UnitTestCase;
-use TYPO3\CMS\Core\Utility\GeneralUtility;
-
-/**
- * Test case
- */
-class TcaFlexFetchTest extends UnitTestCase
-{
-    /**
-     * @var TcaFlexFetch
-     */
-    protected $subject;
-
-    /**
-     * @var BackendUserAuthentication|ObjectProphecy
-     */
-    protected $backendUserProphecy;
-
-    /**
-     * @var array A backup of registered singleton instances
-     */
-    protected $singletonInstances = [];
-
-    protected function setUp()
-    {
-        $this->singletonInstances = GeneralUtility::getSingletonInstances();
-
-        // Suppress cache foo in xml helpers of GeneralUtility
-        /** @var CacheManager|ObjectProphecy $cacheManagerProphecy */
-        $cacheManagerProphecy = $this->prophesize(CacheManager::class);
-        GeneralUtility::setSingletonInstance(CacheManager::class, $cacheManagerProphecy->reveal());
-        $cacheFrontendProphecy = $this->prophesize(FrontendInterface::class);
-        $cacheManagerProphecy->getCache(Argument::cetera())->willReturn($cacheFrontendProphecy->reveal());
-
-        $this->subject = new TcaFlexFetch();
-    }
-
-    protected function tearDown()
-    {
-        GeneralUtility::purgeInstances();
-        GeneralUtility::resetSingletonInstances($this->singletonInstances);
-        parent::tearDown();
-    }
-
-    /**
-     * @test
-     */
-    public function addDataSetsParsedDataStructureArray()
-    {
-        $input = [
-            'systemLanguageRows' => [],
-            'databaseRow' => [
-                'aField' => [
-                    'data' => [],
-                    'meta' => [],
-                ],
-            ],
-            'processedTca' => [
-                'columns' => [
-                    'aField' => [
-                        'config' => [
-                            'type' => 'flex',
-                            'ds' => [
-                                'default' => '
-                                                                       <T3DataStructure>
-                                                                               <ROOT>
-                                                                                       <type>array</type>
-                                                                                       <el>
-                                                                                               <aFlexField>
-                                                                                                       <TCEforms>
-                                                                                                               <label>aFlexFieldLabel</label>
-                                                                                                               <config>
-                                                                                                                       <type>input</type>
-                                                                                                               </config>
-                                                                                                       </TCEforms>
-                                                                                               </aFlexField>
-                                                                                       </el>
-                                                                               </ROOT>
-                                                                       </T3DataStructure>
-                                                               ',
-                            ],
-                        ],
-                    ],
-                ],
-            ],
-        ];
-
-        $expected = $input;
-        $expected['processedTca']['columns']['aField']['config']['ds'] = [
-            'ROOT' => [
-                'type' => 'array',
-                'el' => [
-                    'aFlexField' => [
-                        'TCEforms' => [
-                            'label' => 'aFlexFieldLabel',
-                            'config' => [
-                                'type' => 'input',
-                            ],
-                        ],
-                    ],
-                ],
-            ],
-            'meta' => [],
-        ];
-
-        $this->assertEquals($expected, $this->subject->addData($input));
-    }
-
-    /**
-     * @test
-     */
-    public function addDataSetsParsedDataStructureArrayWithSheets()
-    {
-        $input = [
-            'systemLanguageRows' => [],
-            'databaseRow' => [
-                'aField' => [
-                    'data' => [],
-                    'meta' => [],
-                ],
-            ],
-            'processedTca' => [
-                'columns' => [
-                    'aField' => [
-                        'config' => [
-                            'type' => 'flex',
-                            'ds' => [
-                                'default' => '
-                                                                       <T3DataStructure>
-                                                                               <sheets>
-                                                                                       <sDEF>
-                                                                                               <ROOT>
-                                                                                                       <TCEforms>
-                                                                                                               <sheetTitle>aTitle</sheetTitle>
-                                                                                                       </TCEforms>
-                                                                                                       <type>array</type>
-                                                                                                       <el>
-                                                                                                               <aFlexField>
-                                                                                                                       <TCEforms>
-                                                                                                                               <label>aFlexFieldLabel</label>
-                                                                                                                               <config>
-                                                                                                                                       <type>input</type>
-                                                                                                                               </config>
-                                                                                                                       </TCEforms>
-                                                                                                               </aFlexField>
-                                                                                                       </el>
-                                                                                               </ROOT>
-                                                                                       </sDEF>
-                                                                               </sheets>
-                                                                       </T3DataStructure>
-                                                               ',
-                            ],
-                        ],
-                    ],
-                ],
-            ],
-        ];
-
-        $expected = $input;
-        $expected['processedTca']['columns']['aField']['config']['ds'] = [
-            'sheets' => [
-                'sDEF' => [
-                    'ROOT' => [
-                        'type' => 'array',
-                        'el' => [
-                            'aFlexField' => [
-                                'TCEforms' => [
-                                    'label' => 'aFlexFieldLabel',
-                                    'config' => [
-                                        'type' => 'input',
-                                    ],
-                                ],
-                            ],
-                        ],
-                        'TCEforms' => [
-                            'sheetTitle' => 'aTitle',
-                        ],
-                    ],
-                ],
-            ],
-            'meta' => [],
-        ];
-
-        $this->assertEquals($expected, $this->subject->addData($input));
-    }
-
-    /**
-     * @test
-     */
-    public function addDataThrowsExceptionIfDataStructureCanNotBeParsed()
-    {
-        $input = [
-            'systemLanguageRows' => [],
-            'databaseRow' => [],
-            'processedTca' => [
-                'columns' => [
-                    'aField' => [
-                        'config' => [
-                            'type' => 'flex',
-                            'ds' => ''
-                        ],
-                    ],
-                ],
-            ],
-        ];
-
-        $this->expectException(\UnexpectedValueException::class);
-        $this->expectExceptionCode(1440506893);
-
-        $this->subject->addData($input);
-    }
-
-    /**
-     * @test
-     */
-    public function addDataInitializesDatabaseRowValueIfNoDataStringIsGiven()
-    {
-        $input = [
-            'databaseRow' => [],
-            'systemLanguageRows' => [],
-            'processedTca' => [
-                'columns' => [
-                    'aField' => [
-                        'config' => [
-                            'type' => 'flex',
-                            'ds' => [
-                                'default' => '
-                                                                       <T3DataStructure>
-                                                                               <ROOT></ROOT>
-                                                                       </T3DataStructure>
-                                                               ',
-                            ],
-                        ],
-                    ],
-                ],
-            ],
-        ];
-
-        $expected = $input;
-        $expected['processedTca']['columns']['aField']['config']['ds'] = [
-            'ROOT' => '',
-            'meta' => [],
-        ];
-        $expected['databaseRow']['aField'] = [
-            'data' => [],
-            'meta' => []
-        ];
-
-        $this->assertEquals($expected, $this->subject->addData($input));
-    }
-}
index 94dee29..e613795 100644 (file)
@@ -14,8 +14,14 @@ namespace TYPO3\CMS\Backend\Tests\Unit\Form\FormDataProvider;
  * The TYPO3 project - inspiring people to share!
  */
 
+use Prophecy\Argument;
+use Prophecy\Prophecy\ObjectProphecy;
 use TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare;
+use TYPO3\CMS\Core\Authentication\BackendUserAuthentication;
+use TYPO3\CMS\Core\Cache\CacheManager;
+use TYPO3\CMS\Core\Cache\Frontend\FrontendInterface;
 use TYPO3\CMS\Core\Tests\UnitTestCase;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
 
 /**
  * Test case
@@ -27,18 +33,45 @@ class TcaFlexPrepareTest extends UnitTestCase
      */
     protected $subject;
 
+    /**
+     * @var BackendUserAuthentication|ObjectProphecy
+     */
+    protected $backendUserProphecy;
+
+    /**
+     * @var array A backup of registered singleton instances
+     */
+    protected $singletonInstances = [];
+
     protected function setUp()
     {
+        $this->singletonInstances = GeneralUtility::getSingletonInstances();
+
+        // Suppress cache foo in xml helpers of GeneralUtility
+        /** @var CacheManager|ObjectProphecy $cacheManagerProphecy */
+        $cacheManagerProphecy = $this->prophesize(CacheManager::class);
+        GeneralUtility::setSingletonInstance(CacheManager::class, $cacheManagerProphecy->reveal());
+        $cacheFrontendProphecy = $this->prophesize(FrontendInterface::class);
+        $cacheManagerProphecy->getCache(Argument::cetera())->willReturn($cacheFrontendProphecy->reveal());
+
         $this->subject = new TcaFlexPrepare();
     }
 
+    protected function tearDown()
+    {
+        GeneralUtility::purgeInstances();
+        GeneralUtility::resetSingletonInstances($this->singletonInstances);
+        parent::tearDown();
+    }
+
     /**
      * @test
      */
-    public function addDataThrowsExceptionIfBothSheetsAndRootDefined()
+    public function addDataSetsParsedDataStructureArray()
     {
         $input = [
             'systemLanguageRows' => [],
+            'tableName' => 'aTableName',
             'databaseRow' => [
                 'aField' => [
                     'data' => [],
@@ -51,8 +84,23 @@ class TcaFlexPrepareTest extends UnitTestCase
                         'config' => [
                             'type' => 'flex',
                             'ds' => [
-                                'ROOT' => [],
-                                'sheets' => [],
+                                'default' => '
+                                    <T3DataStructure>
+                                        <ROOT>
+                                            <type>array</type>
+                                            <el>
+                                                <aFlexField>
+                                                    <TCEforms>
+                                                        <label>aFlexFieldLabel</label>
+                                                        <config>
+                                                            <type>input</type>
+                                                        </config>
+                                                    </TCEforms>
+                                                </aFlexField>
+                                            </el>
+                                        </ROOT>
+                                    </T3DataStructure>
+                                ',
                             ],
                         ],
                     ],
@@ -60,63 +108,75 @@ class TcaFlexPrepareTest extends UnitTestCase
             ],
         ];
 
-        $this->expectException(\UnexpectedValueException::class);
-        $this->expectExceptionCode(1440676540);
+        $GLOBALS['TCA']['aTableName']['columns'] = $input['processedTca']['columns'];
 
-        $this->subject->addData($input);
+        $expected = $input;
+        $expected['processedTca']['columns']['aField']['config']['ds'] = [
+            'sheets' => [
+                'sDEF' => [
+                    'ROOT' => [
+                        'type' => 'array',
+                        'el' => [
+                            'aFlexField' => [
+                                'label' => 'aFlexFieldLabel',
+                                'config' => [
+                                    'type' => 'input',
+                                ],
+                            ],
+                        ],
+                    ],
+                ],
+            ],
+            'meta' => [],
+        ];
+
+        $this->assertEquals($expected, $this->subject->addData($input));
     }
 
     /**
      * @test
      */
-    public function addDataRemovesTceFormsFromArrayKeys()
+    public function addDataSetsParsedDataStructureArrayWithSheets()
     {
         $input = [
+            'systemLanguageRows' => [],
+            'tableName' => 'aTableName',
+            'databaseRow' => [
+                'aField' => [
+                    'data' => [],
+                    'meta' => [],
+                ],
+            ],
             'processedTca' => [
                 'columns' => [
                     'aField' => [
                         'config' => [
                             'type' => 'flex',
                             'ds' => [
-                                'sheets' => [
-                                    'sDEF' => [
-                                        'ROOT' => [
-                                            'TCEforms' => [
-                                                'sheetDescription' => 'aDescription',
-                                                'displayCond' => 'aDisplayCond',
-                                            ],
-                                            'type' => 'array',
-                                            'el' => [
-                                                'aFlexField' => [
-                                                    'TCEforms' => [
-                                                        'label' => 'aFlexFieldLabel',
-                                                        'config' => [
-                                                            'type' => 'input',
-                                                        ],
-                                                    ],
-                                                ],
-                                            ],
-                                        ],
-                                    ],
-                                    'sOther' => [
-                                        'ROOT' => [
-                                            'TCEforms' => [
-                                                'sheetTitle' => 'anotherTitle',
-                                            ],
-                                            'type' => 'array',
-                                            'el' => [
-                                                'bFlexField' => [
-                                                    'TCEforms' => [
-                                                        'label' => 'bFlexFieldLabel',
-                                                        'config' => [
-                                                            'type' => 'input',
-                                                        ],
-                                                    ],
-                                                ],
-                                            ],
-                                        ],
-                                    ],
-                                ],
+                                'default' => '
+                                    <T3DataStructure>
+                                        <sheets>
+                                            <sDEF>
+                                                <ROOT>
+                                                    <TCEforms>
+                                                        <sheetTitle>aTitle</sheetTitle>
+                                                    </TCEforms>
+                                                    <type>array</type>
+                                                    <el>
+                                                        <aFlexField>
+                                                            <TCEforms>
+                                                                <label>aFlexFieldLabel</label>
+                                                                <config>
+                                                                    <type>input</type>
+                                                                </config>
+                                                            </TCEforms>
+                                                        </aFlexField>
+                                                    </el>
+                                                </ROOT>
+                                            </sDEF>
+                                        </sheets>
+                                    </T3DataStructure>
+                                ',
                             ],
                         ],
                     ],
@@ -124,6 +184,8 @@ class TcaFlexPrepareTest extends UnitTestCase
             ],
         ];
 
+        $GLOBALS['TCA']['aTableName']['columns'] = $input['processedTca']['columns'];
+
         $expected = $input;
         $expected['processedTca']['columns']['aField']['config']['ds'] = [
             'sheets' => [
@@ -138,27 +200,55 @@ class TcaFlexPrepareTest extends UnitTestCase
                                 ],
                             ],
                         ],
-                        'sheetDescription' => 'aDescription',
-                        'displayCond' => 'aDisplayCond',
+                        'sheetTitle' => 'aTitle',
                     ],
                 ],
-                'sOther' => [
-                    'ROOT' => [
-                        'type' => 'array',
-                        'el' => [
-                            'bFlexField' => [
-                                'label' => 'bFlexFieldLabel',
-                                'config' => [
-                                    'type' => 'input',
-                                ],
+            ],
+            'meta' => [],
+        ];
+
+        $this->assertEquals($expected, $this->subject->addData($input));
+    }
+
+    /**
+     * @test
+     */
+    public function addDataInitializesDatabaseRowValueIfNoDataStringIsGiven()
+    {
+        $input = [
+            'databaseRow' => [],
+            'tableName' => 'aTableName',
+            'systemLanguageRows' => [],
+            'processedTca' => [
+                'columns' => [
+                    'aField' => [
+                        'config' => [
+                            'type' => 'flex',
+                            'ds' => [
+                                'default' => '
+                                    <T3DataStructure>
+                                        <ROOT></ROOT>
+                                    </T3DataStructure>
+                                ',
                             ],
                         ],
-                        'sheetTitle' => 'anotherTitle',
                     ],
                 ],
             ],
         ];
 
+        $GLOBALS['TCA']['aTableName']['columns'] = $input['processedTca']['columns'];
+
+        $expected = $input;
+        $expected['processedTca']['columns']['aField']['config']['ds'] = [
+            'ROOT' => '',
+            'meta' => [],
+        ];
+        $expected['databaseRow']['aField'] = [
+            'data' => [],
+            'meta' => []
+        ];
+
         $this->assertEquals($expected, $this->subject->addData($input));
     }
 
@@ -168,44 +258,56 @@ class TcaFlexPrepareTest extends UnitTestCase
     public function addDataMigratesFlexformTca()
     {
         $input = [
+            'systemLanguageRows' => [],
+            'tableName' => 'aTableName',
+            'databaseRow' => [
+                'aField' => [
+                    'data' => [],
+                    'meta' => [],
+                ],
+            ],
             'processedTca' => [
                 'columns' => [
                     'aField' => [
                         'config' => [
                             'type' => 'flex',
                             'ds' => [
-                                'sheets' => [
-                                    'sDEF' => [
-                                        'ROOT' => [
-                                            'type' => 'array',
-                                            'el' => [
-                                                'aFlexField' => [
-                                                    'TCEforms' => [
-                                                        'label' => 'aFlexFieldLabel',
-                                                        'config' => [
-                                                            'type' => 'text',
-                                                            'default' => 'defaultValue',
-                                                            'wizards' => [
-                                                                't3editor' => [
-                                                                    'type' => 'userFunc',
-                                                                    'userFunc' => 'TYPO3\\CMS\\T3editor\\FormWizard->main',
-                                                                    'title' => 't3editor',
-                                                                    'icon' => 'content-table',
-                                                                    'module' => [
-                                                                        'name' => 'wizard_table',
-                                                                    ],
-                                                                    'params' => [
-                                                                        'format' => 'html',
-                                                                    ],
-                                                                ],
-                                                            ],
-                                                        ],
-                                                    ],
-                                                ],
-                                            ],
-                                        ],
-                                    ],
-                                ],
+                                'default' => '
+                                    <T3DataStructure>
+                                        <sheets>
+                                            <sDEF>
+                                                <ROOT>
+                                                    <type>array</type>
+                                                    <el>
+                                                        <aFlexField>
+                                                            <TCEforms>
+                                                                <label>aFlexFieldLabel</label>
+                                                                <config>
+                                                                    <type>text</type>
+                                                                    <default>defaultValue</default>
+                                                                    <wizards>
+                                                                        <t3editor>
+                                                                            <type>userFunc</type>
+                                                                            <userFunc>TYPO3\\CMS\\T3editor\\FormWizard->main</userFunc>
+                                                                            <title>t3editor</title>
+                                                                            <icon>content-table</icon>
+                                                                            <module>
+                                                                                <name>wizard_table</name>
+                                                                            </module>
+                                                                            <params>
+                                                                                <format>html</format>
+                                                                            </params>
+                                                                        </t3editor>
+                                                                    </wizards>
+                                                                </config>
+                                                            </TCEforms>
+                                                        </aFlexField>
+                                                    </el>
+                                                </ROOT>
+                                            </sDEF>
+                                        </sheets>
+                                    </T3DataStructure>
+                                ',
                             ],
                         ],
                     ],
@@ -213,6 +315,8 @@ class TcaFlexPrepareTest extends UnitTestCase
             ],
         ];
 
+        $GLOBALS['TCA']['aTableName']['columns'] = $input['processedTca']['columns'];
+
         $expected = $input;
         $expected['processedTca']['columns']['aField']['config']['ds'] = [
             'sheets' => [
@@ -233,6 +337,7 @@ class TcaFlexPrepareTest extends UnitTestCase
                     ],
                 ],
             ],
+            'meta' => [],
         ];
 
         $this->assertEquals($expected, $this->subject->addData($input));
@@ -244,58 +349,69 @@ class TcaFlexPrepareTest extends UnitTestCase
     public function addDataMigratesFlexformTcaInContainer()
     {
         $input = [
+            'systemLanguageRows' => [],
+            'tableName' => 'aTableName',
+            'databaseRow' => [
+                'aField' => [
+                    'data' => [],
+                    'meta' => [],
+                ],
+            ],
             'processedTca' => [
                 'columns' => [
                     'aField' => [
                         'config' => [
                             'type' => 'flex',
-                            'ds_pointerField' => 'pointerField',
                             'ds' => [
-                                'sheets' => [
-                                    'sDEF' => [
-                                        'ROOT' => [
-                                            'type' => 'array',
-                                            'el' => [
-                                                'section_1' => [
-                                                    'title' => 'section_1',
-                                                    'type' => 'array',
-                                                    'section' => '1',
-                                                    'el' => [
-                                                        'aFlexContainer' => [
-                                                            'type' => 'array',
-                                                            'title' => 'aFlexContainerLabel',
-                                                            'el' => [
-                                                                'aFlexField' => [
-                                                                    'TCEforms' => [
-                                                                        'label' => 'aFlexFieldLabel',
-                                                                        'config' => [
-                                                                            'type' => 'text',
-                                                                            'default' => 'defaultValue',
-                                                                            'wizards' => [
-                                                                                't3editor' => [
-                                                                                    'type' => 'userFunc',
-                                                                                    'userFunc' => 'TYPO3\CMS\T3editor\FormWizard->main',
-                                                                                    'title' => 't3editor',
-                                                                                    'icon' => 'content-table',
-                                                                                    'module' => [
-                                                                                        'name' => 'wizard_table',
-                                                                                    ],
-                                                                                    'params' => [
-                                                                                        'format' => 'html',
-                                                                                    ],
-                                                                                ],
-                                                                            ],
-                                                                        ],
-                                                                    ],
-                                                                ],
-                                                            ],
-                                                        ],
-                                                    ],
-                                                ],
-                                            ],
-                                        ],
-                                    ],
-                                ],
+                                'default' => '
+                                    <T3DataStructure>
+                                        <sheets>
+                                            <sDEF>
+                                                <ROOT>
+                                                    <type>array</type>
+                                                    <el>
+                                                        <section_1>
+                                                            <title>section_1</title>
+                                                            <type>array</type>
+                                                            <section>1</section>
+                                                            <el>
+                                                                <aFlexContainer>
+                                                                    <type>array</type>
+                                                                    <title>aFlexContainerLabel</title>
+                                                                    <el>
+                                                                        <aFlexField>
+                                                                            <TCEforms>
+                                                                                <label>aFlexFieldLabel</label>
+                                                                                <config>
+                                                                                    <type>text</type>
+                                                                                    <default>defaultValue</default>
+                                                                                    <wizards>
+                                                                                        <t3editor>
+                                                                                            <type>userFunc</type>
+                                                                                            <userFunc>TYPO3\\CMS\\T3editor\\FormWizard->main</userFunc>
+                                                                                            <title>t3editor</title>
+                                                                                            <icon>content-table</icon>
+                                                                                            <module>
+                                                                                                <name>wizard_table</name>
+                                                                                            </module>
+                                                                                            <params>
+                                                                                                <format>html</format>
+                                                                                            </params>
+                                                                                        </t3editor>
+                                                                                    </wizards>
+                                                                                </config>
+                                                                            </TCEforms>
+                                                                        </aFlexField>
+                                                                    </el>
+                                                                </aFlexContainer>
+                                                            </el>
+                                                        </section_1>
+                                                    </el>
+                                                </ROOT>
+                                            </sDEF>
+                                        </sheets>
+                                    </T3DataStructure>
+                                ',
                             ],
                         ],
                     ],
@@ -303,6 +419,8 @@ class TcaFlexPrepareTest extends UnitTestCase
             ],
         ];
 
+        $GLOBALS['TCA']['aTableName']['columns'] = $input['processedTca']['columns'];
+
         $expected = $input;
         $expected['processedTca']['columns']['aField']['config']['ds'] = [
             'sheets' => [
@@ -336,6 +454,7 @@ class TcaFlexPrepareTest extends UnitTestCase
                     ],
                 ],
             ],
+            'meta' => [],
         ];
 
         $this->assertEquals($expected, $this->subject->addData($input));
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/AbstractInvalidDataStructureException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/AbstractInvalidDataStructureException.php
new file mode 100644 (file)
index 0000000..d24ea43
--- /dev/null
@@ -0,0 +1,25 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use TYPO3\CMS\Core\Exception;
+
+/**
+ * Abstract exception thrown if data structures can not be resolved,
+ * found or parsed.
+ */
+abstract class AbstractInvalidDataStructureException extends Exception
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidCombinedPointerFieldException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidCombinedPointerFieldException.php
new file mode 100644 (file)
index 0000000..294240b
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Thrown if a combined ds_pointerField can not be resolved.
+ */
+class InvalidCombinedPointerFieldException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidIdentifierException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidIdentifierException.php
new file mode 100644 (file)
index 0000000..9cd9159
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Thrown if parseFlexFormDataStructureByIdentifier() is given an empty string
+ */
+class InvalidIdentifierException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowException.php
new file mode 100644 (file)
index 0000000..a40bd2e
--- /dev/null
@@ -0,0 +1,23 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Exception thrown if lookup of a parent row in a tree does not return a valid result
+ * during flex from structure lookup.
+ */
+class InvalidParentRowException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowLoopException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowLoopException.php
new file mode 100644 (file)
index 0000000..21b00e8
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Exception thrown if lookup of a parent row in a tree results in a loop.
+ */
+class InvalidParentRowLoopException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowRootException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidParentRowRootException.php
new file mode 100644 (file)
index 0000000..b09d565
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Exception thrown if lookup of a parent row in a tree is root node and still nothing was found.
+ */
+class InvalidParentRowRootException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidPointerFieldValueException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidPointerFieldValueException.php
new file mode 100644 (file)
index 0000000..dea4737
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Exception thrown if ds_pointerField does not point to a valid value.
+ */
+class InvalidPointerFieldValueException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidSinglePointerFieldException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidSinglePointerFieldException.php
new file mode 100644 (file)
index 0000000..a84f1fe
--- /dev/null
@@ -0,0 +1,22 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Thrown if a single ds_pointerField can not be resolved.
+ */
+class InvalidSinglePointerFieldException extends AbstractInvalidDataStructureException
+{
+}
diff --git a/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidTcaException.php b/typo3/sysext/core/Classes/Configuration/FlexForm/Exception/InvalidTcaException.php
new file mode 100644 (file)
index 0000000..79a7160
--- /dev/null
@@ -0,0 +1,24 @@
+<?php
+namespace TYPO3\CMS\Core\Configuration\FlexForm\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Thrown if TCA is invalid.
+ * This may happen if a record is opened that points to a data structure of
+ * a no longer loaded extension
+ */
+class InvalidTcaException extends AbstractInvalidDataStructureException
+{
+}
index ff03ab4..12783d2 100644 (file)
@@ -1,4 +1,5 @@
 <?php
+declare(strict_types=1);
 namespace TYPO3\CMS\Core\Configuration\FlexForm;
 
 /*
@@ -15,7 +16,18 @@ namespace TYPO3\CMS\Core\Configuration\FlexForm;
  */
 
 use TYPO3\CMS\Backend\Utility\BackendUtility;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidCombinedPointerFieldException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidIdentifierException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowLoopException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowRootException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidPointerFieldValueException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidSinglePointerFieldException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidTcaException;
+use TYPO3\CMS\Core\Database\ConnectionPool;
+use TYPO3\CMS\Core\Database\Query\Restriction\DeletedRestriction;
 use TYPO3\CMS\Core\Utility\GeneralUtility;
+use TYPO3\CMS\Core\Utility\MathUtility;
 
 /**
  * Contains functions for manipulating flex form data
@@ -30,20 +42,6 @@ class FlexFormTools
     public $reNumberIndexesOfSectionData = false;
 
     /**
-     * Contains data structure when traversing flexform
-     *
-     * @var array
-     */
-    public $traverseFlexFormXMLData_DS = [];
-
-    /**
-     * Contains data array when traversing flexform
-     *
-     * @var array
-     */
-    public $traverseFlexFormXMLData_Data = [];
-
-    /**
      * Options for array2xml() for flexform.
      * This will map the weird keys from the internal array to tags that could potentially be checked with a DTD/schema
      *
@@ -78,6 +76,682 @@ class FlexFormTools
     public $cleanFlexFormXML = [];
 
     /**
+     * The method locates a specific data structure from given TCA and row combination
+     * and returns an identifier string that can be handed around, and can be resolved
+     * to a single data structure later without giving $row and $tca data again.
+     *
+     * Note: The returned syntax is meant to only specify the target location of the data structure.
+     * It SHOULD NOT be abused and enriched with data from the record that is dealt with. For
+     * instance, it is now allowed to add source record specific date like the uid or the pid!
+     * If that is done, it is up to the hook consumer to take care of possible side effects, eg. if
+     * the data handler copies or moves records around and those references change.
+     *
+     * This method gets: Source data that influences the target location of a data structure
+     * This method returns: Target specification of the data structure
+     *
+     * This method is "paired" with method getFlexFormDataStructureByIdentifier() that
+     * will resolve the returned syntax again and returns the data structure itself.
+     *
+     * Both methods can be extended via hooks to return and accept additional
+     * identifier strings if needed, and to transmit further information within the identifier strings.
+     *
+     * Note that the TCA for data structure definitions MUST NOT be overridden by
+     * 'columnsOverrides' or by parent TCA in an inline relation! This would create a huge mess.
+     *
+     * Note: This method and the resolving methods belowe are well unit tested and document all
+     * nasty details this way.
+     *
+     * @param array $fieldTca Full TCA of the field in question that has type=flex set
+     * @param string $tableName The table name of the TCA field
+     * @param string $fieldName The field name
+     * @param array $row The data row
+     * @return string Identifier string
+     * @throws \RuntimeException If TCA is misconfigured
+     */
+    public function getDataStructureIdentifier(array $fieldTca, string $tableName, string $fieldName, array $row): string
+    {
+        // Hook to inject an own logic to point to a data structure elsewhere.
+        // A hook has to implement method getDataStructureIdentifierPreProcess() to be called here.
+        // All hooks are called in a row, each MUST return an array, and the FIRST one that
+        // returns a non-empty array is used as final identifier.
+        // It is important to restrict hooks as much as possible to give other hooks a chance to kick in.
+        // The returned identifier is later given to parseFlexFormDataStructureByIdentifier() and a hook in there MUST
+        // be used to handle this identifier again.
+        // Warning: If adding source record details like the uid or pid here, this may turn out to be fragile.
+        // Be sure to test scenarios like workspaces and data handler copy/move well, additionally, this may
+        // break in between different core versions.
+        // It is probably a good idea to return at least something like [ 'type' => 'myExtension', ... ], see
+        // the core internal 'tca' and 'record' return values below
+        if (!empty($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+            && is_array($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])) {
+            $hookClasses = $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'];
+            foreach ($hookClasses as $hookClass) {
+                $hookInstance = GeneralUtility::makeInstance($hookClass);
+                if (method_exists($hookClass, 'getDataStructureIdentifierPreProcess')) {
+                    $dataStructureIdentifier = $hookInstance->getDataStructureIdentifierPreProcess(
+                        $fieldTca, $tableName, $fieldName, $row
+                    );
+                    if (!is_array($dataStructureIdentifier)) {
+                        throw new \RuntimeException(
+                            'Hook class ' . $hookClass . ' method getDataStructureIdentifierPreProcess must return an array',
+                            1478096535
+                        );
+                    }
+                    if (!empty($dataStructureIdentifier)) {
+                        // Early break at first hook that returned something!
+                        break;
+                    }
+                }
+            }
+        }
+
+        // If hooks didn't return something, kick in core logic
+        if (empty($dataStructureIdentifier)) {
+            $tcaDataStructureArray = $fieldTca['config']['ds'] ?? null;
+            $tcaDataStructurePointerField = $fieldTca['config']['ds_pointerField'] ?? null;
+            if (!is_array($tcaDataStructureArray) && $tcaDataStructurePointerField) {
+                // "ds" is not an array, but "ds_pointerField" is set -> data structure is found in different table
+                $dataStructureIdentifier = $this->getDataStructureIdentifierFromRecord($fieldTca, $tableName,
+                    $fieldName, $row);
+            } elseif (is_array($tcaDataStructureArray)) {
+                $dataStructureIdentifier = $this->getDataStructureIdentifierFromTcaArray($fieldTca, $tableName,
+                    $fieldName, $row);
+            } else {
+                throw new \RuntimeException(
+                    'TCA misconfiguration in table "' . $tableName . '" field "' . $fieldName . '" config section:'
+                    . ' The field is configured as type="flex" and no "ds_pointerField" is defined and "ds" is not an array.'
+                    . ' Either configure a default data structure in [\'ds\'][\'default\'] or add a "ds_pointerField" lookup mechanism'
+                    . ' that specifies the data structure',
+                    1463826960
+                );
+            }
+        }
+
+        // Second hook to manipulate identifier again. This can be used to add additional data to
+        // identifiers. Be careful here, especially if stuff from the source record like uid or pid
+        // is added! This may easily lead to issues with data handler details like copy or move records,
+        // localization and version overlays. Test this very well!
+        // Multiple hooks may add information to the same identifier here - take care to namespace array keys.
+        // Information added here can be later used in parseDataStructureByIdentifier post process hook again.
+        if (!empty($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+            && is_array($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])) {
+            $hookClasses = $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'];
+            foreach ($hookClasses as $hookClass) {
+                $hookInstance = GeneralUtility::makeInstance($hookClass);
+                if (method_exists($hookClass, 'getDataStructureIdentifierPostProcess')) {
+                    $dataStructureIdentifier = $hookInstance->getDataStructureIdentifierPostProcess(
+                        $fieldTca, $tableName, $fieldName, $row, $dataStructureIdentifier
+                    );
+                    if (!is_array($dataStructureIdentifier) || empty($dataStructureIdentifier)) {
+                        throw new \RuntimeException(
+                            'Hook class ' . $hookClass . ' method getDataStructureIdentifierPostProcess must return a non empty array',
+                            1478350835
+                        );
+                    }
+                }
+            }
+        }
+
+        return json_encode($dataStructureIdentifier);
+    }
+
+    /**
+     * The data structure is located in a record. This method resolves the record and
+     * returns an array to identify that record.
+     *
+     * The example setup below looks in current row for a tx_templavoila_ds value. If not found,
+     * it will search the rootline (the table is a tree, typically pages) until a value in
+     * tx_templavoila_next_ds or tx_templavoila_ds is found. That value should then be an
+     * integer, that points to a record in tx_templavoila_datastructure, and then the data
+     * structure is found in field dataprot:
+     *
+     * fieldTca = [
+     *     'config' => [
+     *         'type' => 'flex',
+     *         'ds_pointerField' => 'tx_templavoila_ds',
+     *         'ds_pointerField_searchParent' => 'pid',
+     *         'ds_pointerField_searchParent_subField' => 'tx_templavoila_next_ds',
+     *         'ds_tableField' => 'tx_templavoila_datastructure:dataprot',
+     *     ]
+     * ]
+     *
+     * More simple scenario without tree traversal and having a valid data structure directly
+     * located in field theFlexDataStructureField.
+     *
+     * fieldTca = [
+     *     'config' => [
+     *         'type' => 'flex',
+     *         'ds_pointerField' => 'theFlexDataStructureField',
+     *     ]
+     * ]
+     *
+     * Example return array:
+     * [
+     *     'type' => 'record',
+     *     'tableName' => 'tx_templavoila_datastructure',
+     *     'uid' => 42,
+     *     'fieldName' => 'dataprot',
+     * ];
+     *
+     * @param array $fieldTca Full TCA of the field in question that has type=flex set
+     * @param string $tableName The table name of the TCA field
+     * @param string $fieldName The field name
+     * @param array $row The data row
+     * @return array Identifier as array, see example above
+     * @throws InvalidParentRowException
+     * @throws InvalidParentRowLoopException
+     * @throws InvalidParentRowRootException
+     * @throws InvalidPointerFieldValueException
+     * @throws InvalidTcaException
+     */
+    protected function getDataStructureIdentifierFromRecord(array $fieldTca, string $tableName, string $fieldName, array $row): array
+    {
+        $pointerFieldName = $finalPointerFieldName = $fieldTca['config']['ds_pointerField'];
+        if (!array_key_exists($pointerFieldName, $row)) {
+            // Pointer field does not exist in row at all -> throw
+            throw new InvalidTcaException(
+                'No data structure for field "' . $fieldName . '" in table "' . $tableName . '" found, no "ds" array'
+                . ' configured and given row does not have a field with ds_pointerField name "' . $pointerFieldName . '".',
+                1464115059
+            );
+        }
+        $pointerValue = $row[$pointerFieldName];
+        // If set, this is typically set to "pid"
+        $parentFieldName = $fieldTca['config']['ds_pointerField_searchParent'] ?? null;
+        $pointerSubFieldName = $fieldTca['config']['ds_pointerField_searchParent_subField'] ?? null;
+        if (!$pointerValue && $parentFieldName) {
+            // Fetch rootline until a valid pointer value is found
+            $handledUids = [];
+            while (!$pointerValue) {
+                $handledUids[$row['uid']] = 1;
+                $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable($tableName);
+                $queryBuilder->getRestrictions()
+                    ->removeAll()
+                    ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
+                $queryBuilder->select('uid', $parentFieldName, $pointerFieldName);
+                if (!empty($pointerSubFieldName)) {
+                    $queryBuilder->addSelect($pointerSubFieldName);
+                }
+                $queryStatement = $queryBuilder->from($tableName)
+                    ->where($queryBuilder->expr()->eq(
+                        'uid',
+                        $queryBuilder->createNamedParameter($row[$parentFieldName], \PDO::PARAM_INT))
+                    )
+                    ->execute();
+                if ($queryStatement->rowCount() !== 1) {
+                    throw new InvalidParentRowException(
+                        'The data structure for field "' . $fieldName . '" in table "' . $tableName . '" has to be looked up'
+                        . ' in field "' . $pointerFieldName . '". That field had no valid value, so a lookup in parent record'
+                        . ' with uid "' . $row[$parentFieldName] . '" was done. This row however does not exist or was deleted.',
+                        1463833794
+                    );
+                }
+                $row = $queryStatement->fetch();
+                if (isset($handledUids[$row[$parentFieldName]])) {
+                    // Row has been fetched before already -> loop detected!
+                    throw new InvalidParentRowLoopException(
+                        'The data structure for field "' . $fieldName . '" in table "' . $tableName . '" has to be looked up'
+                        . ' in field "' . $pointerFieldName . '". That field had no valid value, so a lookup in parent record'
+                        . ' with uid "' . $row[$parentFieldName] . '" was done. A loop of records was detected, the tree is broken.',
+                        1464110956
+                    );
+                }
+                BackendUtility::workspaceOL($tableName, $row);
+                BackendUtility::fixVersioningPid($tableName, $row, true);
+                // New pointer value: This is the "subField" value if given, else the field value
+                // ds_pointerField_searchParent_subField is the "template on next level" structure from templavoila
+                if ($pointerSubFieldName && $row[$pointerSubFieldName]) {
+                    $finalPointerFieldName = $pointerSubFieldName;
+                    $pointerValue = $row[$pointerSubFieldName];
+                } else {
+                    $pointerValue = $row[$pointerFieldName];
+                }
+                if (!$pointerValue && ((int)$row[$parentFieldName] === 0 || $row[$parentFieldName] === null)) {
+                    // If on root level and still no valid pointer found -> exception
+                    throw new InvalidParentRowRootException(
+                        'The data structure for field "' . $fieldName . '" in table "' . $tableName . '" has to be looked up'
+                        . ' in field "' . $pointerFieldName . '". That field had no valid value, so a lookup in parent record'
+                        . ' with uid "' . $row[$parentFieldName] . '" was done. Root node with uid "' . $row['uid'] . '"'
+                        . ' was fetched and still no valid pointer field value was found.',
+                        1464112555
+                    );
+                }
+            }
+        }
+        if (!$pointerValue) {
+            // Still no valid pointer value -> exception, This still can be a data integrity issue, so throw a catchable exception
+            throw new InvalidPointerFieldValueException(
+                'No data structure for field "' . $fieldName . '" in table "' . $tableName . '" found, no "ds" array'
+                . ' configured and data structure could be found by resolving parents. This is probably a TCA misconfiguration.',
+                1464114011
+            );
+        }
+        // Ok, finally we have the field value. This is now either a data structure directly, or a pointer to a file,
+        // or the value can be interpreted as integer (is an uid) and "ds_tableField" is set, so this is the table, uid and field
+        // where the final data structure can be found.
+        if (MathUtility::canBeInterpretedAsInteger($pointerValue)) {
+            if (!isset($fieldTca['config']['ds_tableField'])) {
+                throw new InvalidTcaException(
+                    'Invalid data structure pointer for field "' . $fieldName . '" in table "' . $tableName . '", the value'
+                    . 'resolved to "' . $pointerValue . '" . which is an integer, so "ds_tableField" must be configured',
+                    1464115639
+                );
+            }
+            if (substr_count($fieldTca['config']['ds_tableField'], ':') !== 1) {
+                // ds_tableField must be of the form "table:field"
+                throw new InvalidTcaException(
+                    'Invalid TCA configuration for field "' . $fieldName . '" in table "' . $tableName . '", the setting'
+                    . '"ds_tableField" must be of the form "tableName:fieldName"',
+                    1464116002
+                );
+            }
+            list($foreignTableName, $foreignFieldName) = GeneralUtility::trimExplode(':', $fieldTca['config']['ds_tableField']);
+            $dataStructureIdentifier = [
+                'type' => 'record',
+                'tableName' => $foreignTableName,
+                'uid' => (int)$pointerValue,
+                'fieldName' => $foreignFieldName,
+            ];
+        } else {
+            $dataStructureIdentifier = [
+                'type' => 'record',
+                'tableName' => $tableName,
+                'uid' => (int)$row['uid'],
+                'fieldName' => $finalPointerFieldName,
+            ];
+        }
+        return $dataStructureIdentifier;
+    }
+
+    /**
+     * Find matching data structure in TCA ds array.
+     *
+     * Data structure is defined in 'ds' config array.
+     * Also, there can be a ds_pointerField
+     *
+     * fieldTca = [
+     *     'config' => [
+     *         'type' => 'flex',
+     *         'ds' => [
+     *             'aName' => '<T3DataStructure>...' OR 'FILE:...'
+     *         ],
+     *         'ds_pointerField' => 'optionalSetting,upToTwoCommaSeparatedFieldNames',
+     *     ]
+     * ]
+     *
+     * This method returns an array of the form:
+     * [
+     *     'type' => 'Tca:',
+     *     'tableName' => $tableName,
+     *     'fieldName' => $fieldName,
+     *     'dataStructureKey' => $key,
+     * ];
+     *
+     * Example:
+     * [
+     *     'type' => 'Tca:',
+     *     'tableName' => 'tt_content',
+     *     'fieldName' => 'pi_flexform',
+     *     'dataStructureKey' => 'powermail_pi1,list',
+     * ];
+     *
+     * @param array $fieldTca Full TCA of the field in question that has type=flex set
+     * @param string $tableName The table name of the TCA field
+     * @param string $fieldName The field name
+     * @param array $row The data row
+     * @return array Identifier as array, see example above
+     * @throws InvalidCombinedPointerFieldException
+     * @throws InvalidSinglePointerFieldException
+     * @throws InvalidTcaException
+     */
+    protected function getDataStructureIdentifierFromTcaArray(array $fieldTca, string $tableName, string $fieldName, array $row): array
+    {
+        $dataStructureIdentifier = [
+            'type' => 'tca',
+            'tableName' => $tableName,
+            'fieldName' => $fieldName,
+            'dataStructureKey' => null,
+        ];
+        $tcaDataStructurePointerField = $fieldTca['config']['ds_pointerField'] ?? null;
+        if ($tcaDataStructurePointerField === null) {
+            // No ds_pointerField set -> use 'default' as ds array key if exists.
+            if (isset($fieldTca['config']['ds']['default'])) {
+                $dataStructureIdentifier['dataStructureKey'] = 'default';
+            } else {
+                // A tca is configured as flex without ds_pointerField. A 'default' key must exist, otherwise
+                // this is a configuration error.
+                // May happen with an unloaded extension -> catchable
+                throw new InvalidTcaException(
+                    'TCA misconfiguration in table "' . $tableName . '" field "' . $fieldName . '" config section:'
+                    . ' The field is configured as type="flex" and no "ds_pointerField" is defined. Either configure'
+                    . ' a default data structure in [\'ds\'][\'default\'] or add a "ds_pointerField" lookup mechanism'
+                    . ' that specifies the data structure',
+                    1463652560
+                );
+            }
+        } else {
+            // ds_pointerField is set, it can be a comma separated list of two fields, explode it.
+            $pointerFieldArray = GeneralUtility::trimExplode(',', $tcaDataStructurePointerField, true);
+            // Obvious configuration error, either one or two fields must be declared
+            $pointerFieldsCount = count($pointerFieldArray);
+            if ($pointerFieldsCount !== 1 && $pointerFieldsCount !== 2) {
+                // If it's there, it must be correct -> not catchable
+                throw new \RuntimeException(
+                    'TCA misconfiguration in table "' . $tableName . '" field "' . $fieldName . '" config section:'
+                    . ' ds_pointerField must be either a single field name, or a comma separated list of two fields,'
+                    . ' the invalid configuration string provided was: "' . $tcaDataStructurePointerField . '"',
+                    1463577497
+                );
+            }
+            // Verify first field exists in row array. If not, this is a hard error: Any extension that sets a
+            // ds_pointerField to some field name should take care that field does exist, too. They are a pair,
+            // so there shouldn't be a situation where the field does not exist. Throw an exception if that is violated.
+            if (!isset($row[$pointerFieldArray[0]])) {
+                // If it's declared, it must exist -> not catchable
+                throw new \RuntimeException(
+                    'TCA misconfiguration in table "' . $tableName . '" field "' . $fieldName . '" config section:'
+                    . ' ds_pointerField "' . $pointerFieldArray[0] . '" points to a field name that does not exist.',
+                    1463578899
+                );
+            }
+            // Similar situation for the second field: If it is set, the field must exist.
+            if (isset($pointerFieldArray[1]) && !isset($row[$pointerFieldArray[1]])) {
+                // If it's declared, it must exist -> not catchable
+                throw new \RuntimeException(
+                    'TCA misconfiguration in table "' . $tableName . '" field "' . $fieldName . '" config section:'
+                    . ' Second part "' . $pointerFieldArray[1] . '" of ds_pointerField with full value "'
+                    . $tcaDataStructurePointerField . '" points to a field name that does not exist.',
+                    1463578900
+                );
+            }
+            if ($pointerFieldsCount === 1) {
+                if (isset($fieldTca['config']['ds'][$row[$pointerFieldArray[0]]])) {
+                    // Field value points directly to an existing key in tca ds
+                    $dataStructureIdentifier['dataStructureKey'] = $row[$pointerFieldArray[0]];
+                } elseif (isset($fieldTca['config']['ds']['default'])) {
+                    // Field value does not exit in tca ds, fall back to default key if exists
+                    $dataStructureIdentifier['dataStructureKey'] = 'default';
+                } else {
+                    // The value of the ds_pointerField field points to a key in the ds array that does
+                    // not exists, and there is no fallback either. This can happen if an extension brings
+                    // new flex form definitions and that extension is unloaded later. "Old" records of the
+                    // extension could then still point to the no longer existing key in ds. We throw a
+                    // specific exception here to give controllers an opportunity to catch this case.
+                    throw new InvalidSinglePointerFieldException(
+                        'Field value of field "' . $pointerFieldArray[0] . '" of database record with uid "'
+                        . $row['uid'] . '" from table "' . $tableName . '" points to a "ds" key ' . $row[$pointerFieldArray[0]]
+                        . ' but this key does not exist and there is no "default" fallback.',
+                        1463653197
+                    );
+                }
+            } else {
+                // Two comma separated field names
+                if (isset($fieldTca['config']['ds'][$row[$pointerFieldArray[0]] . ',' . $row[$pointerFieldArray[1]]])) {
+                    // firstValue,secondValue
+                    $dataStructureIdentifier['dataStructureKey'] = $row[$pointerFieldArray[0]] . ',' . $row[$pointerFieldArray[1]];
+                } elseif (isset($fieldTca['config']['ds'][$row[$pointerFieldArray[1]] . ',*'])) {
+                    // secondValue,* ?!
+                    // @deprecated since TYPO3 v8, will be removed in TYPO3 v9 - just remove this elseif together with two unit tests
+                    // This case is a wrong implementation - it matches "secondFieldValue,*", but it
+                    // should match "*,secondFieldValue" only. Since this bug has been in the code for ages, it
+                    // still works in v8 but is deprecated now.
+                    // Try to log a meaningful deprecation message though, so devs can adapt
+                    GeneralUtility::deprecationLog(
+                        'TCA field "' . $fieldName . '" of table "' . $tableName . '" has a registered data structure'
+                        . ' with name "' . $row[$pointerFieldArray[1]] . ',*". The ds_pointerField is set to "'
+                        . $tcaDataStructurePointerField . '", with the matching value "' . $row[$pointerFieldArray[1]] . '"'
+                        . ' for field "' . $pointerFieldArray[1] . '". This should be the other way round, so the name'
+                        . ' should be: "*,' . $row[$pointerFieldArray[1]] . '" in the ds TCA array. Please change that'
+                        . ' until TYPO3 v9, this matching code will be removed then.'
+                    );
+                    $dataStructureIdentifier['dataStructureKey'] = $row[$pointerFieldArray[1]] . ',*';
+                } elseif (isset($fieldTca['config']['ds'][$row[$pointerFieldArray[0]] . ',*'])) {
+                    // firstValue,*
+                    $dataStructureIdentifier['dataStructureKey'] = $row[$pointerFieldArray[0]] . ',*';
+                } elseif (isset($fieldTca['config']['ds']['*,' . $row[$pointerFieldArray[1]]])) {
+                    // *,secondValue
+                    $dataStructureIdentifier['dataStructureKey'] = '*,' . $row[$pointerFieldArray[1]];
+                } elseif (isset($fieldTca['config']['ds'][$row[$pointerFieldArray[0]]])) {
+                    // firstValue
+                    $dataStructureIdentifier['dataStructureKey'] = $row[$pointerFieldArray[0]];
+                } elseif (isset($fieldTca['config']['ds']['default'])) {
+                    // Fall back to default
+                    $dataStructureIdentifier['dataStructureKey'] = 'default';
+                } else {
+                    // No ds_pointerField value could be determined and 'default' does not exist as
+                    // fallback. This is the same case as the above scenario, throw a
+                    // InvalidCombinedPointerFieldException here, too.
+                    throw new InvalidCombinedPointerFieldException(
+                        'Field combination of fields "' . $pointerFieldArray[0] . '" and "' . $pointerFieldArray[1] . '" of database'
+                        . 'record with uid "' . $row['uid'] . '" from table "' . $tableName . '" with values "' . $row[$pointerFieldArray[0]] . '"'
+                        . ' and "' . $row[$pointerFieldArray[1]] . '" could not be resolved to any registered data structure and '
+                        . ' no "default" fallback exists.',
+                        1463678524
+                    );
+                }
+            }
+        }
+        return $dataStructureIdentifier;
+    }
+
+    /**
+     * Parse a data structure identified by $identifier to the final data structure array.
+     * This method is called after getDataStructureIdentifier(), finds the data structure
+     * and returns it.
+     *
+     * Hooks allow to manipulate the find logic and to post process the data structure array.
+     *
+     * Note that the TCA for data structure definitions MUST NOT be overridden by
+     * 'columnsOverrides' or by parent TCA in an inline relation! This would create a huge mess.
+     *
+     * After the data structure definition is found, the method resolves:
+     * * FILE:EXT: prefix of the data structure itself - the ds is in a file
+     * * FILE:EXT: prefix for sheets - if single sheets are in files
+     * * EXT: prefix for sheets - if single sheets are in files (slightly different b/w compat syntax)
+     * * Create an sDEF sheet if the data structure has non, yet.
+     *
+     * After that method is run, the data structure is fully resolved to an array,
+     * and same base normalization is done: If the ds did not contain a sheet,
+     * it will have one afterwards as "sDEF"
+     *
+     * This method gets: Target specification of the data structure.
+     * This method returns: The normalized data structure parsed to an array.
+     *
+     * Read the unit tests for nasty details.
+     *
+     * @param string $identifier String to find the data structure location
+     * @return array Parsed and normalized data structure
+     * @throws InvalidIdentifierException
+     */
+    public function parseDataStructureByIdentifier(string $identifier): array
+    {
+        // Throw an exception for an empty string. This might be a valid use case for new
+        // records in some situations, so this is catchable to give callers a chance to deal with that.
+        if (empty($identifier)) {
+            throw new InvalidIdentifierException(
+                'Empty string given to parseFlexFormDataStructureByIdentifier(). This exception might '
+                . ' be caught to handle some new record situations properly',
+                1478100828
+            );
+        }
+
+        $identifier = json_decode($identifier, true);
+
+        if (!is_array($identifier) || empty($identifier)) {
+            // If there is some identifier and it can't be decoded, programming error -> not catchable
+            throw new \RuntimeException(
+                'Identifier could not be decoded to an array.',
+                1478345642
+            );
+        }
+
+        $dataStructure = '';
+
+        // Hook to fetch data structure by given identifier.
+        // Method parseFlexFormDataStructureByIdentifier() must be implemented and returns either an
+        // empty string "not my business", or a string with the resolved data structure string, or FILE: reference,
+        // or a fully parsed data structure as aray.
+        // Result of the FIRST hook that gives an non-empty string is used, namespace your identifiers in
+        // a way that there is little chance they overlap (eg. prefix with extension name).
+        // If implemented, this hook should be paired with a hook in getDataStructureIdentifier() above.
+        if (!empty($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+            && is_array($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+        ) {
+            $hookClasses = $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'];
+            foreach ($hookClasses as $hookClass) {
+                $hookInstance = GeneralUtility::makeInstance($hookClass);
+                if (method_exists($hookClass, 'parseDataStructureByIdentifierPreProcess')) {
+                    $dataStructure = $hookInstance->parseDataStructureByIdentifierPreProcess($identifier);
+                    if (!is_string($dataStructure) && !is_array($dataStructure)) {
+                        // Programming error -> not catchable
+                        throw new \RuntimeException(
+                            'Hook class ' . $hookClass . ' method parseDataStructureByIdentifierPreProcess must either'
+                            . ' return an empty string or a data structure string or a parsed data structure array.',
+                            1478168512
+                        );
+                    }
+                    if (!empty($dataStructure)) {
+                        // Early break if a hook resolved to something!
+                        break;
+                    }
+                }
+            }
+        }
+
+        // If hooks didn't resolve, try own methods
+        if (empty($dataStructure)) {
+            if ($identifier['type'] === 'record') {
+                // Handle "record" type, see getDataStructureIdentifierFromRecord()
+                if (empty($identifier['tableName']) || empty($identifier['uid']) || empty($identifier['fieldName'])) {
+                    throw new \RuntimeException(
+                        'Incomplete "record" based identifier: ' . json_encode($identifier),
+                        1478113873
+                    );
+                }
+                $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable($identifier['tableName']);
+                $queryBuilder->getRestrictions()->removeAll()->add(GeneralUtility::makeInstance(DeletedRestriction::class));
+                $dataStructure = $queryBuilder
+                    ->select($identifier['fieldName'])
+                    ->from($identifier['tableName'])
+                    ->where(
+                        $queryBuilder->expr()->eq(
+                            'uid',
+                            $queryBuilder->createNamedParameter($identifier['uid'], \PDO::PARAM_INT)
+                        )
+                    )
+                    ->execute()
+                    ->fetchColumn(0);
+            } elseif ($identifier['type'] === 'tca') {
+                // Handle "tca" type, see getDataStructureIdentifierFromTcaArray
+                if (empty($identifier['tableName']) || empty($identifier['fieldName']) || empty($identifier['dataStructureKey'])) {
+                    throw new \RuntimeException(
+                        'Incomplete "tca" based identifier: ' . json_encode($identifier),
+                        1478113471
+                    );
+                }
+                $table = $identifier['tableName'];
+                $field = $identifier['fieldName'];
+                $dataStructureKey = $identifier['dataStructureKey'];
+                if (!isset($GLOBALS['TCA'][$table]['columns'][$field]['config']['ds'][$dataStructureKey])
+                    || !is_string($GLOBALS['TCA'][$table]['columns'][$field]['config']['ds'][$dataStructureKey])
+                ) {
+                    // This may happen for elements pointing to an unloaded extension -> catchable
+                    throw new InvalidIdentifierException(
+                        'Specified identifier ' . json_encode($identifier) . ' does not resolve to a valid'
+                        . ' TCA array value',
+                        1478105491
+                    );
+                }
+                $dataStructure = $GLOBALS['TCA'][$table]['columns'][$field]['config']['ds'][$dataStructureKey];
+            } else {
+                throw new InvalidIdentifierException(
+                    'Identifier ' . json_encode($identifier) . ' could not be resolved',
+                    1478104554
+                );
+            }
+        }
+
+        // Hooks may have parse the data structure already to an array. If that is not the case, parse it now.
+        if (is_string($dataStructure)) {
+            // Resolve FILE: prefix pointing to a DS in a file
+            if (strpos(trim($dataStructure), 'FILE:') === 0) {
+                $file = GeneralUtility::getFileAbsFileName(substr(trim($dataStructure), 5));
+                if (empty($file) || !@is_file($file)) {
+                    throw new \RuntimeException(
+                        'Data structure file ' . $file . ' could not be resolved to an existing file',
+                        1478105826
+                    );
+                }
+                $dataStructure = file_get_contents($file);
+            }
+
+            // Parse main structure
+            $dataStructure = GeneralUtility::xml2array($dataStructure);
+        }
+
+        // Throw if it still is not an array, probably because GeneralUtility::xml2array() failed
+        if (!is_array($dataStructure)) {
+            throw new \RuntimeException(
+                'Parse error: Data structure could not be resolved to a valid structure.',
+                1478106090
+            );
+        }
+
+        // Create default sheet if there is none, yet.
+        if (isset($dataStructure['ROOT']) && isset($dataStructure['sheets'])) {
+            throw new \RuntimeException(
+                'Parsed data structure has both ROOT and sheets on top level. Thats invalid.',
+                1440676540
+            );
+        }
+        if (isset($dataStructure['ROOT']) && is_array($dataStructure['ROOT'])) {
+            $dataStructure['sheets']['sDEF']['ROOT'] = $dataStructure['ROOT'];
+            unset($dataStructure['ROOT']);
+        }
+
+        // Resolve FILE:EXT and EXT: for single sheets
+        if (isset($dataStructure['sheets']) && is_array($dataStructure['sheets'])) {
+            foreach ($dataStructure['sheets'] as $sheetName => $sheetStructure) {
+                if (!is_array($sheetStructure)) {
+                    if (strpos(trim($sheetStructure), 'FILE:') === 0) {
+                        $file = GeneralUtility::getFileAbsFileName(substr(trim($sheetStructure), 5));
+                    } else {
+                        $file = GeneralUtility::getFileAbsFileName(trim($sheetStructure));
+                    }
+                    if ($file && @is_file($file)) {
+                        $sheetStructure = GeneralUtility::xml2array(file_get_contents($file));
+                    }
+                }
+                $dataStructure['sheets'][$sheetName] = $sheetStructure;
+            }
+        }
+
+        // Hook to manipulate data structure further. This can be used to add or remove fields
+        // from given structure. Multiple hooks can be registered, all are called. They
+        // receive the parsed structure and the identifier array.
+        if (!empty($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+            && is_array($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'])
+        ) {
+            $hookClasses = $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][self::class]['flexParsing'];
+            foreach ($hookClasses as $hookClass) {
+                $hookInstance = GeneralUtility::makeInstance($hookClass);
+                if (method_exists($hookClass, 'parseDataStructureByIdentifierPostProcess')) {
+                    $dataStructure = $hookInstance->parseDataStructureByIdentifierPostProcess($dataStructure, $identifier);
+                    if (!is_array($dataStructure)) {
+                        // Programming error -> not catchable
+                        throw new \RuntimeException(
+                            'Hook class ' . $hookClass . ' method parseDataStructureByIdentifierPreProcess must return and array.',
+                            1478350806
+                        );
+                    }
+                }
+            }
+        }
+
+        return $dataStructure;
+    }
+
+    /**
      * Handler for Flex Forms
      *
      * @param string $table The table name of the record
@@ -85,7 +759,7 @@ class FlexFormTools
      * @param array $row The record data array
      * @param object $callBackObj Object in which the call back function is located
      * @param string $callBackMethod_value Method name of call back function in object for values
-     * @return bool|string If TRUE, error happened (error string returned)
+     * @return bool|string true on success, string if error happened (error string returned)
      */
     public function traverseFlexFormXMLData($table, $field, $row, $callBackObj, $callBackMethod_value)
     {
@@ -94,42 +768,30 @@ class FlexFormTools
         }
         $this->callBackObj = $callBackObj;
         // Get Data Structure:
-        $dataStructArray = BackendUtility::getFlexFormDS($GLOBALS['TCA'][$table]['columns'][$field]['config'], $row, $table, $field);
-        // If data structure was ok, proceed:
-        if (is_array($dataStructArray)) {
-            // Get flexform XML data
-            $editData = GeneralUtility::xml2array($row[$field]);
-            if (!is_array($editData)) {
-                return 'Parsing error: ' . $editData;
-            }
-            // Tabs sheets
-            if (is_array($dataStructArray['sheets'])) {
-                $sKeys = array_keys($dataStructArray['sheets']);
+        $dataStructureIdentifier = $this->getDataStructureIdentifier($GLOBALS['TCA'][$table]['columns'][$field], $table, $field, $row);
+        $dataStructureArray = $this->parseDataStructureByIdentifier($dataStructureIdentifier);
+        // Get flexform XML data
+        $editData = GeneralUtility::xml2array($row[$field]);
+        if (!is_array($editData)) {
+            return 'Parsing error: ' . $editData;
+        }
+        // Traverse languages:
+        foreach ($dataStructureArray['sheets'] as $sheetKey => $sheetData) {
+            // Render sheet:
+            if (is_array($sheetData['ROOT']) && is_array($sheetData['ROOT']['el'])) {
+                $PA['vKeys'] = ['DEF'];
+                $PA['lKey'] = 'lDEF';
+                $PA['callBackMethod_value'] = $callBackMethod_value;
+                $PA['table'] = $table;
+                $PA['field'] = $field;
+                $PA['uid'] = $row['uid'];
+                // Render flexform:
+                $this->traverseFlexFormXMLData_recurse($sheetData['ROOT']['el'], $editData['data'][$sheetKey]['lDEF'], $PA, 'data/' . $sheetKey . '/lDEF');
             } else {
-                $sKeys = ['sDEF'];
+                return 'Data Structure ERROR: No ROOT element found for sheet "' . $sheetKey . '".';
             }
-            // Traverse languages:
-            foreach ($sKeys as $sheet) {
-                list($dataStruct, $sheet) = GeneralUtility::resolveSheetDefInDS($dataStructArray, $sheet);
-                // Render sheet:
-                if (is_array($dataStruct['ROOT']) && is_array($dataStruct['ROOT']['el'])) {
-                    $PA['vKeys'] = ['DEF'];
-                    $PA['lKey'] = 'lDEF';
-                    $PA['callBackMethod_value'] = $callBackMethod_value;
-                    $PA['table'] = $table;
-                    $PA['field'] = $field;
-                    $PA['uid'] = $row['uid'];
-                    $this->traverseFlexFormXMLData_DS = &$dataStruct;
-                    $this->traverseFlexFormXMLData_Data = &$editData;
-                    // Render flexform:
-                    $this->traverseFlexFormXMLData_recurse($dataStruct['ROOT']['el'], $editData['data'][$sheet]['lDEF'], $PA, 'data/' . $sheet . '/lDEF');
-                } else {
-                    return 'Data Structure ERROR: No ROOT element found for sheet "' . $sheet . '".';
-                }
-            }
-        } else {
-            return 'Data Structure ERROR: ' . $dataStructArray;
         }
+        return true;
     }
 
     /**
index 302461f..6e7eb49 100644 (file)
@@ -2346,19 +2346,29 @@ class DataHandler
     {
         if (is_array($value)) {
             // This value is necessary for flex form processing to happen on flexform fields in page records when they are copied.
-            // Problem: when copying a page, flexform XML comes along in the array for the new record - but since $this->checkValue_currentRecord does not have a uid or pid for that
-            // sake, the BackendUtility::getFlexFormDS() function returns no good DS. For new records we do know the expected PID so therefore we send that with this special parameter.
-            // Only active when larger than zero.
-            $newRecordPidValue = $status == 'new' ? $realPid : 0;
+            // Problem: when copying a page, flexform XML comes along in the array for the new record - but since $this->checkValue_currentRecord
+            // does not have a uid or pid for that sake, the FlexFormTools->getDataStructureIdentifier() function returns no good DS. For new
+            // records we do know the expected PID so therefore we send that with this special parameter. Only active when larger than zero.
+            $row = $this->checkValue_currentRecord;
+            if ($status === 'new') {
+                $row['pid'] = $realPid;
+            }
             // Get current value array:
-            $dataStructArray = BackendUtility::getFlexFormDS($tcaFieldConf, $this->checkValue_currentRecord, $table, $field, true, $newRecordPidValue);
+            $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+            $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                [ 'config' => $tcaFieldConf ],
+                $table,
+                $field,
+                $row
+            );
+            $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
             $currentValueArray = (string)$curValue !== '' ? GeneralUtility::xml2array($curValue) : [];
             if (!is_array($currentValueArray)) {
                 $currentValueArray = [];
             }
             // Remove all old meta for languages...
             // Evaluation of input values:
-            $value['data'] = $this->checkValue_flex_procInData($value['data'], $currentValueArray['data'], $uploadedFiles['data'], $dataStructArray, [$table, $id, $curValue, $status, $realPid, $recFID, $tscPID]);
+            $value['data'] = $this->checkValue_flex_procInData($value['data'], $currentValueArray['data'], $uploadedFiles['data'], $dataStructureArray, [$table, $id, $curValue, $status, $realPid, $recFID, $tscPID]);
             // Create XML from input value:
             $xmlValue = $this->checkValue_flexArray2Xml($value, true);
 
@@ -2911,21 +2921,28 @@ class DataHandler
      * @param array $dataPart The 'data' part of the INPUT flexform data
      * @param array $dataPart_current The 'data' part of the CURRENT flexform data
      * @param array $uploadedFiles The uploaded files for the 'data' part of the INPUT flexform data
-     * @param array $dataStructArray Data structure for the form (might be sheets or not). Only values in the data array which has a configuration in the data structure will be processed.
+     * @param array $dataStructure Data structure for the form (might be sheets or not). Only values in the data array which has a configuration in the data structure will be processed.
      * @param array $pParams A set of parameters to pass through for the calling of the evaluation functions
      * @param string $callBackFunc Optional call back function, see checkValue_flex_procInData_travDS()  DEPRECATED, use \TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools instead for traversal!
      * @param array $workspaceOptions
      * @return array The modified 'data' part.
      * @see checkValue_flex_procInData_travDS()
      */
-    public function checkValue_flex_procInData($dataPart, $dataPart_current, $uploadedFiles, $dataStructArray, $pParams, $callBackFunc = '', array $workspaceOptions = [])
+    public function checkValue_flex_procInData($dataPart, $dataPart_current, $uploadedFiles, $dataStructure, $pParams, $callBackFunc = '', array $workspaceOptions = [])
     {
         if (is_array($dataPart)) {
             foreach ($dataPart as $sKey => $sheetDef) {
-                list($dataStruct, $actualSheet) = GeneralUtility::resolveSheetDefInDS($dataStructArray, $sKey);
-                if (is_array($dataStruct) && $actualSheet == $sKey && is_array($sheetDef)) {
+                if (isset($dataStructure['sheets'][$sKey]) && is_array($dataStructure['sheets'][$sKey]) && is_array($sheetDef)) {
                     foreach ($sheetDef as $lKey => $lData) {
-                        $this->checkValue_flex_procInData_travDS($dataPart[$sKey][$lKey], $dataPart_current[$sKey][$lKey], $uploadedFiles[$sKey][$lKey], $dataStruct['ROOT']['el'], $pParams, $callBackFunc, $sKey . '/' . $lKey . '/', $workspaceOptions);
+                        $this->checkValue_flex_procInData_travDS(
+                            $dataPart[$sKey][$lKey],
+                            $dataPart_current[$sKey][$lKey],
+                            $uploadedFiles[$sKey][$lKey],
+                            $dataStructure['sheets'][$sKey]['ROOT']['el'],
+                            $pParams,
+                            $callBackFunc,
+                            $sKey . '/' . $lKey . '/', $workspaceOptions
+                        );
                     }
                 }
             }
@@ -3736,11 +3753,18 @@ class DataHandler
         // For "flex" fieldtypes we need to traverse the structure for two reasons: If there are file references they have to be prepended with absolute paths and if there are database reference they MIGHT need to be remapped (still done in remapListedDBRecords())
         if ($conf['type'] == 'flex') {
             // Get current value array:
-            $dataStructArray = BackendUtility::getFlexFormDS($conf, $row, $table, $field);
+            $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+            $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                [ 'config' => $conf ],
+                $table,
+                $field,
+                $row
+            );
+            $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
             $currentValueArray = GeneralUtility::xml2array($value);
             // Traversing the XML structure, processing files:
             if (is_array($currentValueArray)) {
-                $currentValueArray['data'] = $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructArray, [$table, $uid, $field, $realDestPid], 'copyRecord_flexFormCallBack', $workspaceOptions);
+                $currentValueArray['data'] = $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructureArray, [$table, $uid, $field, $realDestPid], 'copyRecord_flexFormCallBack', $workspaceOptions);
                 // Setting value as an array! -> which means the input will be processed according to the 'flex' type when the new copy is created.
                 $value = $currentValueArray;
             }
@@ -5538,6 +5562,7 @@ class DataHandler
         $currentRec = BackendUtility::getRecord($table, $id);
         $swapRec = BackendUtility::getRecord($table, $swapWith);
         $this->version_remapMMForVersionSwap_reg = [];
+        $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
         foreach ($GLOBALS['TCA'][$table]['columns'] as $field => $fConf) {
             $conf = $fConf['config'];
             if ($this->isReferenceField($conf)) {
@@ -5559,16 +5584,28 @@ class DataHandler
                 }
             } elseif ($conf['type'] == 'flex') {
                 // Current record
-                $dataStructArray = BackendUtility::getFlexFormDS($conf, $currentRec, $table, $field);
+                $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                    $fConf,
+                    $table,
+                    $field,
+                    $currentRec
+                );
+                $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
                 $currentValueArray = GeneralUtility::xml2array($currentRec[$field]);
                 if (is_array($currentValueArray)) {
-                    $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructArray, [$table, $id, $field], 'version_remapMMForVersionSwap_flexFormCallBack');
+                    $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructureArray, [$table, $id, $field], 'version_remapMMForVersionSwap_flexFormCallBack');
                 }
                 // Swap record
-                $dataStructArray = BackendUtility::getFlexFormDS($conf, $swapRec, $table, $field);
+                $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                    $fConf,
+                    $table,
+                    $field,
+                    $swapRec
+                );
+                $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
                 $currentValueArray = GeneralUtility::xml2array($swapRec[$field]);
                 if (is_array($currentValueArray)) {
-                    $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructArray, [$table, $swapWith, $field], 'version_remapMMForVersionSwap_flexFormCallBack');
+                    $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructureArray, [$table, $swapWith, $field], 'version_remapMMForVersionSwap_flexFormCallBack');
                 }
             }
         }
@@ -5667,6 +5704,7 @@ class DataHandler
     public function remapListedDBRecords()
     {
         if (!empty($this->registerDBList)) {
+            $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
             foreach ($this->registerDBList as $table => $records) {
                 foreach ($records as $uid => $fields) {
                     $newData = [];
@@ -5690,10 +5728,16 @@ class DataHandler
                                     if (is_array($origRecordRow)) {
                                         BackendUtility::workspaceOL($table, $origRecordRow);
                                         // Get current data structure and value array:
-                                        $dataStructArray = BackendUtility::getFlexFormDS($conf, $origRecordRow, $table, $fieldName);
+                                        $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                                            [ 'config' => $conf ],
+                                            $table,
+                                            $fieldName,
+                                            $origRecordRow
+                                        );
+                                        $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
                                         $currentValueArray = GeneralUtility::xml2array($origRecordRow[$fieldName]);
                                         // Do recursive processing of the XML data:
-                                        $currentValueArray['data'] = $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructArray, [$table, $theUidToUpdate, $fieldName], 'remapListedDBRecords_flexFormCallBack');
+                                        $currentValueArray['data'] = $this->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructureArray, [$table, $theUidToUpdate, $fieldName], 'remapListedDBRecords_flexFormCallBack');
                                         // The return value should be compiled back into XML, ready to insert directly in the field (as we call updateDB() directly later):
                                         if (is_array($currentValueArray['data'])) {
                                             $newData[$fieldName] = $this->checkValue_flexArray2Xml($currentValueArray, true);
index bb74cec..4512c4a 100644 (file)
@@ -315,7 +315,8 @@ class ReferenceIndex
         $deleteField = $GLOBALS['TCA'][$tableName]['ctrl']['delete'];
 
         if ($tableRelationFields === '*') {
-            // If one field of a record is of type flex, all fields have to be fetched to be passed to BackendUtility::getFlexFormDS
+            // If one field of a record is of type flex, all fields have to be fetched
+            // to be passed to FlexFormTools->getDataStructureIdentifier()
             $selectFields = '*';
         } else {
             // otherwise only fields that might contain relations are fetched
@@ -570,7 +571,8 @@ class ReferenceIndex
                 // For "flex" fieldtypes we need to traverse the structure looking for file and db references of course!
                 if ($conf['type'] === 'flex') {
                     // Get current value array:
-                    // NOTICE: failure to resolve Data Structures can lead to integrity problems with the reference index. Please look up the note in the JavaDoc documentation for the function \TYPO3\CMS\Backend\Utility\BackendUtility::getFlexFormDS()
+                    // NOTICE: failure to resolve Data Structures can lead to integrity problems with the reference index. Please look up
+                    // the note in the JavaDoc documentation for the function FlexFormTools->getDataStructureIdentifier()
                     $currentValueArray = GeneralUtility::xml2array($value);
                     // Traversing the XML structure, processing files:
                     if (is_array($currentValueArray)) {
@@ -1122,7 +1124,7 @@ class ReferenceIndex
                 // Check for flex field
                 if (isset($fieldDefinition['config']['type']) && $fieldDefinition['config']['type'] === 'flex') {
                     // Fetch all fields if the is a field of type flex in the table definition because the complete row is passed to
-                    // BackendUtility::getFlexFormDS in the end and might be needed in ds_pointerField or $hookObj->getFlexFormDS_postProcessDS
+                    // FlexFormTools->getDataStructureIdentifier() in the end and might be needed in ds_pointerField or a hook
                     return '*';
                 }
                 // Only fetch this field if it can contain a reference
index 913e979..afba26c 100644 (file)
@@ -3638,9 +3638,11 @@ class GeneralUtility
      * @param array $dataStructArray Input data structure, possibly with a sheet-definition and references to external data source files.
      * @param string $sheet The sheet to return, preferably.
      * @return array An array with two num. keys: key0: The data structure is returned in this key (array) UNLESS an error occurred in which case an error string is returned (string). key1: The used sheet key value!
+     * @deprecated since TYPO3 v8, will be removed in TYPO3 v9. This is now integrated in FlexFormTools->parseDataStructureByIdentifier()
      */
     public static function resolveSheetDefInDS($dataStructArray, $sheet = 'sDEF')
     {
+        self::logDeprecatedFunction();
         if (!is_array($dataStructArray)) {
             return 'Data structure must be an array';
         }
@@ -3676,9 +3678,11 @@ class GeneralUtility
      *
      * @param array $dataStructArray Input data structure, possibly with a sheet-definition and references to external data source files.
      * @return array Output data structure with all sheets resolved as arrays.
+     * @deprecated since TYPO3 v8, will be removed in TYPO3 v9. This is now integrated in FlexFormTools->parseDataStructureByIdentifier()
      */
     public static function resolveAllSheetsInDS(array $dataStructArray)
     {
+        self::logDeprecatedFunction();
         if (is_array($dataStructArray['sheets'])) {
             $out = ['sheets' => []];
             foreach ($dataStructArray['sheets'] as $sheetId => $sDat) {
index 4f46ff6..7c78a92 100644 (file)
@@ -469,7 +469,7 @@ return [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaTypesShowitem::class,
                         ],
                     ],
-                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch::class => [
+                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class => [
                         'depends' => [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\InitializeProcessedTca::class,
                             \TYPO3\CMS\Backend\Form\FormDataProvider\UserTsConfig::class,
@@ -478,11 +478,6 @@ return [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaColumnsProcessFieldLabels::class,
                         ],
                     ],
-                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class => [
-                        'depends' => [
-                            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch::class,
-                        ],
-                    ],
                     \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexProcess::class => [
                         'depends' => [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
@@ -508,8 +503,7 @@ return [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaTypesShowitem::class,
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaColumnsRemoveUnused::class,
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaCheckboxItems::class,
-                            // GeneralUtility::getFlexFormDS() needs unchanged databaseRow values as string
-                            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch::class,
+                            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
                         ],
                     ],
                     \TYPO3\CMS\Backend\Form\FormDataProvider\TcaSelectTreeItems::class => [
@@ -681,18 +675,13 @@ return [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\InlineOverrideChildTca::class,
                         ],
                     ],
-                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch::class => [
+                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class => [
                         'depends' => [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\InitializeProcessedTca::class,
                             \TYPO3\CMS\Backend\Form\FormDataProvider\InlineOverrideChildTca::class,
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaColumnsRemoveUnused::class,
                         ],
                     ],
-                    \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class => [
-                        'depends' => [
-                            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexFetch::class,
-                        ],
-                    ],
                     \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexProcess::class => [
                         'depends' => [
                             \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
diff --git a/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FlexFormToolsPublicPropertiesDropped.rst b/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FlexFormToolsPublicPropertiesDropped.rst
new file mode 100644 (file)
index 0000000..182b71f
--- /dev/null
@@ -0,0 +1,36 @@
+.. include:: ../../Includes.txt
+
+==========================================================
+Breaking: #78581 - FlexFormTools public properties dropped
+==========================================================
+
+See :issue:`78581`
+
+Description
+===========
+
+Two public properties have been dropped from PHP class :php:`FlexFormTools`:
+
+* :php:`FlexFormTools->traverseFlexFormXMLData_DS`
+* :php:`FlexFormTools->traverseFlexFormXMLData_Data`
+
+
+Impact
+======
+
+Accessing those properties will throw a warning.
+
+
+Affected Installations
+======================
+
+Extensions that access these properties. The two were of little use from an extensions point of view,
+it is very unlikely this actually breaks an extension.
+
+
+Migration
+=========
+
+No migration possible.
+
+.. index:: PHP-API, FlexForm, Backend
\ No newline at end of file
diff --git a/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FormEngineTcaFlexFetchDataProviderRemoved.rst b/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-FormEngineTcaFlexFetchDataProviderRemoved.rst
new file mode 100644 (file)
index 0000000..d9381c0
--- /dev/null
@@ -0,0 +1,39 @@
+.. include:: ../../Includes.txt
+
+================================================================
+Breaking: #78581 - FormEngine TcaFlexFetch data provider removed
+================================================================
+
+See :issue:`78581`
+
+Description
+===========
+
+The FormEngine data provider :php:`TcaFlexFetch` has been merged into data provider :php:`TcaFlexPrepare`.
+
+
+Impact
+======
+
+If own registered data providers are declared to "depends" or "before" :php:`TcaFlexFetch`, the
+:php:`DependencyResolver` will be unable to find that and throws an exception or sorts the own data
+provider to an ambiguous place.
+
+
+Affected Installations
+======================
+
+An installation is only affected in the relatively unlikely case that an own data provider declared a
+dependency to :php:`TcaFlexFetch`.
+
+
+Migration
+=========
+
+Move the dependency over to :php:`TcaFlexPrepare`: The two data providers were merged into one, it
+should be save for any data provider to hook before or after :php:`TcaFlexPrepare` instead. There
+is a little additional flex form processing in :php:`TcaFlexPrepare`, so the flex structure might be a
+bit different. have a look at methods :php:`removeTceFormsArrayKeyFromDataStructureElements()`
+and :php:`migrateFlexformTcaDataStructureElements()` for details.
+
+.. index:: PHP-API, FlexForm, Backend
\ No newline at end of file
diff --git a/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-HookGetFlexFormDSClassNoLongerCalled.rst b/typo3/sysext/core/Documentation/Changelog/master/Breaking-78581-HookGetFlexFormDSClassNoLongerCalled.rst
new file mode 100644 (file)
index 0000000..74712a2
--- /dev/null
@@ -0,0 +1,70 @@
+.. include:: ../../Includes.txt
+
+===========================================================
+Breaking: #78581 - Hook getFlexFormDSClass no longer called
+===========================================================
+
+See :issue:`78581`
+
+Description
+===========
+
+With the deprecation of :php:`BackendUtility::getFlexFormDS()` the hook :php:`getFlexFormDSClass` is
+no longer called and there is no one to one substitution available.
+
+
+Impact
+======
+
+The hook is no longer called and flex form field manipulation by extensions does not happen anymore.
+
+
+Affected Installations
+======================
+
+Extensions that extension flex form data structure definitions and use hook :php:`getFlexFormDSClass`
+for that purpose.
+
+
+Migration
+=========
+
+Method :php:`BackendUtility::getFlexFormDS()` has been split into the methods
+:php:`FlexFormTools->getDataStructureIdentifier()` and :php:`FlexFormTools->parseDataStructureByIdentifier()`.
+
+Those two new methods now provide four hooks to allow manipulation of the flex form data structure
+location and parsing. The methods and hooks are documented well, read their description for a deeper
+insight on which combination is the correct one for a specific extension need.
+
+The new hooks are very powerful and must be used with special care to be as future proof as possible.
+
+Since the old hook is used by some often used extension, the core team prepared a transition for some
+of them beforehand:
+
+* Extension news: The extension used the old hook to only remove a couple of fields from the flex
+  form definition. This was moved over to a "FormEngine" data provider: news_
+
+* Extension flux: Flux implements a completely own way of locating and pointing to the flex form
+  data structure that is needed in a specific context. The default core resolving does not match
+  here. Flux now implements the hooks :php:`getDataStructureIdentifierPreProcess` and
+  :php:`parseDataStructureByIdentifierPreProcess` to specify an own "identifier" syntax
+  and to resolve that syntax to a data structure later: flux_
+
+* Extension gridelements: Similar to flux, gridelements has a own logic to choose which specific
+  data structure should be used. However, the data structures are located in database row fields,
+  so the "record" syntax of the core can be re-used to refer to those. gridelements uses the hook
+  :php:`getDataStructureIdentifierPreProcess` together with a small implementation in
+  :php:`parseDataStructureByIdentifierPreProcess` for a fallback scenario: gridelements_
+
+* Extension powermail: Powermail allows extending and changing existing flex form data structure
+  definition depending on page TS. To do that, it now implements hook
+  :php:`getDataStructureIdentifierPostProcess` to add the needed pid to the existing identifier,
+  and then implements hook :php:`parseDataStructureByIdentifierPostProcess` to manipulate the
+  resolved data structure: powermail_
+
+.. _news: https://github.com/georgringer/news/pull/155
+.. _flux: https://github.com/FluidTYPO3/flux/pull/1203
+.. _gridelements: https://review.typo3.org/#/c/50513/
+.. _powermail: https://github.com/einpraegsam/powermail/pull/6
+
+.. index:: PHP-API, FlexForm, Backend
\ No newline at end of file
diff --git a/typo3/sysext/core/Documentation/Changelog/master/Deprecation-78581-FlexFormRelatedParsing.rst b/typo3/sysext/core/Documentation/Changelog/master/Deprecation-78581-FlexFormRelatedParsing.rst
new file mode 100644 (file)
index 0000000..e24ebd4
--- /dev/null
@@ -0,0 +1,42 @@
+.. include:: ../../Includes.txt
+
+===============================================
+Deprecation: #78581 - Flex form related parsing
+===============================================
+
+See :issue:`78581`
+
+Description
+===========
+
+Three flex form data structure related parsing methods have been deprecated:
+
+* :php:`BackendUtility::getFlexFormDS()`
+* :php:`GeneralUtility::resolveSheetDefInDS()`
+* :php:`GeneralUtility::resolveAllSheetsInDS()`
+
+
+Impact
+======
+
+Calling those methods throws a deprecation warning.
+
+
+Affected Installations
+======================
+
+Extensions calling one of the above methods.
+
+
+Migration
+=========
+
+:php:`BackendUtility::getFlexFormDS()` has bee moved to a combination of two methods
+:php:`FlexFormTools->getDataStructureIdentifier()` and :php:`FlexFormTools->parseDataStructureByIdentifier()`.
+The two methods are heavily documented and the combination works in many cases just as before. Read the method
+comments for a detailed description of their purpose.
+
+Warning: The hook :php:`getFlexFormDSClass` within :php:`BackendUtility::getFlexFormDS()` is no longer called
+by the core. Please refer to the according "Breaking" document for details on this topic.
+
+.. index:: PHP-API, FlexForm, Backend
\ No newline at end of file
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnArray.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnArray.php
new file mode 100644 (file)
index 0000000..f5c1b07
--- /dev/null
@@ -0,0 +1,43 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPostProcessHookReturnArray
+{
+    /**
+     * Returns an array
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @param array $identifier
+     * @return array
+     */
+    public function getDataStructureIdentifierPostProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row,
+        array $identifier
+    ): array {
+        $identifier['myExtensionData'] = 'foo';
+        return $identifier;
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnEmptyArray.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnEmptyArray.php
new file mode 100644 (file)
index 0000000..aaf33b9
--- /dev/null
@@ -0,0 +1,42 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPostProcessHookReturnEmptyArray
+{
+    /**
+     * Returns an empty array (no match for this hook)
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @param array $identifier
+     * @return array
+     */
+    public function getDataStructureIdentifierPostProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row,
+        array $identifier
+    ): array {
+        return [];
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnString.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookReturnString.php
new file mode 100644 (file)
index 0000000..ae0c25a
--- /dev/null
@@ -0,0 +1,42 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPostProcessHookReturnString
+{
+    /**
+     * Returns a string (invalid)
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @param array $identifier
+     * @return string
+     */
+    public function getDataStructureIdentifierPostProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row,
+        array $identifier
+    ): string {
+        return 'foo';
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookThrowException.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPostProcessHookThrowException.php
new file mode 100644 (file)
index 0000000..1a766d9
--- /dev/null
@@ -0,0 +1,43 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPostProcessHookThrowException
+{
+    /**
+     * Just throw an exception
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @param array $identifier
+     * @return array
+     * @throws \RuntimeException
+     */
+    public function getDataStructureIdentifierPostProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row,
+        array $identifier
+    ): array {
+        throw new \RuntimeException('testing', 1478342067);
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnArray.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnArray.php
new file mode 100644 (file)
index 0000000..1bccd75
--- /dev/null
@@ -0,0 +1,43 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPreProcessHookReturnArray
+{
+    /**
+     * Returns an array
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @return array
+     */
+    public function getDataStructureIdentifierPreProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row
+    ): array {
+        return [
+            'type' => 'myExtension',
+            'further' => 'data',
+        ];
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnEmptyArray.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnEmptyArray.php
new file mode 100644 (file)
index 0000000..dae06cf
--- /dev/null
@@ -0,0 +1,40 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPreProcessHookReturnEmptyArray
+{
+    /**
+     * Returns a empty array (no match for this hook)
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @return array
+     */
+    public function getDataStructureIdentifierPreProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row
+    ): array {
+        return [];
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnString.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookReturnString.php
new file mode 100644 (file)
index 0000000..fd7497b
--- /dev/null
@@ -0,0 +1,36 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPreProcessHookReturnString
+{
+    /**
+     * Returns a string (invalid)
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @return string
+     */
+    public function getDataStructureIdentifierPreProcess(array $fieldTca, string $tableName, string $fieldName, array $row): string
+    {
+        return 'foo';
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookThrowException.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureIdentifierPreProcessHookThrowException.php
new file mode 100644 (file)
index 0000000..65f0b44
--- /dev/null
@@ -0,0 +1,41 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureIdentifierPreProcessHookThrowException
+{
+    /**
+     * Just throw an exception
+     *
+     * @param array $fieldTca
+     * @param string $tableName
+     * @param string $fieldName
+     * @param array $row
+     * @return array
+     * @throws \RuntimeException
+     */
+    public function getDataStructureIdentifierPreProcess(
+        array $fieldTca,
+        string $tableName,
+        string $fieldName,
+        array $row
+    ): array {
+        throw new \RuntimeException('testing', 1478098527);
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureOfSingleSheet.xml b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureOfSingleSheet.xml
new file mode 100644 (file)
index 0000000..8976ba4
--- /dev/null
@@ -0,0 +1,18 @@
+<T3DataStructure>
+    <ROOT>
+        <TCEforms>
+            <sheetTitle>aTitle</sheetTitle>
+        </TCEforms>
+        <type>array</type>
+        <el>
+            <aFlexField>
+                <TCEforms>
+                    <label>aFlexFieldLabel</label>
+                    <config>
+                        <type>input</type>
+                    </config>
+                </TCEforms>
+            </aFlexField>
+        </el>
+    </ROOT>
+</T3DataStructure>
\ No newline at end of file
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnArray.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnArray.php
new file mode 100644 (file)
index 0000000..c46f124
--- /dev/null
@@ -0,0 +1,37 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePostProcessHookReturnArray
+{
+    /**
+     * Returns a modified array
+     *
+     * @param array $dataStructure
+     * @param array $identifier
+     * @return array
+     */
+    public function parseDataStructureByIdentifierPostProcess(array $dataStructure, array $identifier): array
+    {
+        $dataStructure['sheets'] = [
+            'foo' => 'bar',
+        ];
+        return $dataStructure;
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnString.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookReturnString.php
new file mode 100644 (file)
index 0000000..41210d1
--- /dev/null
@@ -0,0 +1,34 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePostProcessHookReturnString
+{
+    /**
+     * Returns a string (invalid)
+     *
+     * @param array $dataStructure
+     * @param array $identifier
+     * @return string
+     */
+    public function parseDataStructureByIdentifierPostProcess(array $dataStructure, array $identifier): string
+    {
+        return 'foo';
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookThrowException.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePostProcessHookThrowException.php
new file mode 100644 (file)
index 0000000..912d3dc
--- /dev/null
@@ -0,0 +1,35 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePostProcessHookThrowException
+{
+    /**
+     * Just throw an exception
+     *
+     * @param array $dataStructure
+     * @param array $identifier
+     * @return array
+     * @throws \RuntimeException
+     */
+    public function parseDataStructureByIdentifierPostProcess(array $dataStructure, array $identifier): array
+    {
+        throw new \RuntimeException('testing', 1478351691);
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnEmptyString.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnEmptyString.php
new file mode 100644 (file)
index 0000000..4be5b99
--- /dev/null
@@ -0,0 +1,33 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePreProcessHookReturnEmptyString
+{
+    /**
+     * Returns an empty string (no match for this hook)
+     *
+     * @param array $identifier
+     * @return string
+     */
+    public function parseDataStructureByIdentifierPreProcess(array $identifier): string
+    {
+        return '';
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnObject.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnObject.php
new file mode 100644 (file)
index 0000000..e8e076b
--- /dev/null
@@ -0,0 +1,33 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePreProcessHookReturnObject
+{
+    /**
+     * Returns an array (invalid)
+     *
+     * @param array $identifier
+     * @return \stdClass
+     */
+    public function parseDataStructureByIdentifierPreProcess(array $identifier): \stdClass
+    {
+        return new \stdClass();
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnString.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookReturnString.php
new file mode 100644 (file)
index 0000000..9e3c6b1
--- /dev/null
@@ -0,0 +1,41 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePreProcessHookReturnString
+{
+    /**
+     * Returns a valid string
+     *
+     * @param array $identifier
+     * @return string
+     */
+    public function parseDataStructureByIdentifierPreProcess(array $identifier): string
+    {
+        if ($identifier['type'] === 'myExtension') {
+            return '
+                <T3DataStructure>
+                    <sheets></sheets>
+                </T3DataStructure>
+            ';
+        } else {
+            return '';
+        }
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookThrowException.php b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureParsePreProcessHookThrowException.php
new file mode 100644 (file)
index 0000000..8573fc7
--- /dev/null
@@ -0,0 +1,34 @@
+<?php
+declare(strict_types=1);
+namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Fixture to test hooks from FlexFormTools
+ */
+class DataStructureParsePreProcessHookThrowException
+{
+    /**
+     * Just throw an exception
+     *
+     * @param array $identifier
+     * @return string
+     * @throws \RuntimeException
+     */
+    public function parseDataStructureByIdentifierPreProcess(array $identifier): string
+    {
+        throw new \RuntimeException('testing', 1478112411);
+    }
+}
diff --git a/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureWithSheet.xml b/typo3/sysext/core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureWithSheet.xml
new file mode 100644 (file)
index 0000000..0683ecb
--- /dev/null
@@ -0,0 +1,22 @@
+<T3DataStructure>
+    <sheets>
+        <sDEF>
+            <ROOT>
+                <TCEforms>
+                    <sheetTitle>aTitle</sheetTitle>
+                </TCEforms>
+                <type>array</type>
+                <el>
+                    <aFlexField>
+                        <TCEforms>
+                            <label>aFlexFieldLabel</label>
+                            <config>
+                                <type>input</type>
+                            </config>
+                        </TCEforms>
+                    </aFlexField>
+                </el>
+            </ROOT>
+        </sDEF>
+    </sheets>
+</T3DataStructure>
\ No newline at end of file
index 39e6732..47c6df4 100644 (file)
@@ -1,4 +1,5 @@
 <?php
+declare(strict_types=1);
 namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm;
 
 /*
@@ -14,16 +15,1521 @@ namespace TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm;
  * The TYPO3 project - inspiring people to share!
  */
 
+use Doctrine\DBAL\Driver\Statement;
+use Prophecy\Argument;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidCombinedPointerFieldException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidIdentifierException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowLoopException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidParentRowRootException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidPointerFieldValueException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidSinglePointerFieldException;
+use TYPO3\CMS\Core\Configuration\FlexForm\Exception\InvalidTcaException;
 use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
+use TYPO3\CMS\Core\Database\ConnectionPool;
+use TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder;
+use TYPO3\CMS\Core\Database\Query\QueryBuilder;
+use TYPO3\CMS\Core\Database\Query\Restriction\QueryRestrictionContainerInterface;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPostProcessHookReturnArray;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPostProcessHookReturnEmptyArray;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPostProcessHookReturnString;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPostProcessHookThrowException;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPreProcessHookReturnArray;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPreProcessHookReturnEmptyArray;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPreProcessHookReturnString;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureIdentifierPreProcessHookThrowException;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePostProcessHookReturnArray;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePostProcessHookReturnString;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePostProcessHookThrowException;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePreProcessHookReturnEmptyString;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePreProcessHookReturnObject;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePreProcessHookReturnString;
+use TYPO3\CMS\Core\Tests\Unit\Configuration\FlexForm\Fixtures\DataStructureParsePreProcessHookThrowException;
+use TYPO3\CMS\Core\Tests\UnitTestCase;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
 
 /**
  * Test case
  */
-class FlexFormToolsTest extends \TYPO3\CMS\Core\Tests\UnitTestCase
+class FlexFormToolsTest extends UnitTestCase
 {
     /**
      * @test
      */
+    public function getDataStructureIdentifierCallsRegisteredPreProcessHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPreProcessHookThrowException::class,
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478098527);
+        (new FlexFormTools())->getDataStructureIdentifier([], 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPreProcessHookReturnsNoArray()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPreProcessHookReturnString::class
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478096535);
+        (new FlexFormTools())->getDataStructureIdentifier([], 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierUsesCasualLogicIfPreProcessHookReturnsNoIdentifier()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPreProcessHookReturnEmptyArray::class
+        ];
+        $expected = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsStringFromPreProcessHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPreProcessHookReturnArray::class
+        ];
+        $expected = '{"type":"myExtension","further":"data"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier([], 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsStringFromFirstMatchingPreProcessHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPreProcessHookReturnEmptyArray::class,
+            DataStructureIdentifierPreProcessHookReturnArray::class,
+            DataStructureIdentifierPreProcessHookThrowException::class
+        ];
+        $expected = '{"type":"myExtension","further":"data"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier([], 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierCallsRegisteredPostProcessHook()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPostProcessHookThrowException::class,
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478342067);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPostProcessHookReturnsNoArray()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPostProcessHookReturnString::class
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478350835);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPostProcessHookReturnsEmptyArray()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPostProcessHookReturnEmptyArray::class
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478350835);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierPostProcessHookCanEnrichIdentifier()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureIdentifierPostProcessHookReturnArray::class
+        ];
+        $expected = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default","myExtensionData":"foo"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfDsIsNotAnArrayAndNoDsPointerField()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => 'someStringOnly',
+                // no ds_pointerField,
+            ],
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1463826960);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsDefaultIfDsIsSetButNoDsPointerField()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => '<T3DataStructure>...'
+                ],
+            ],
+        ];
+        $expected = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionsIfNoDsPointerFieldIsSetAndDefaultDoesNotExist()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [],
+            ],
+        ];
+        $this->expectException(InvalidTcaException::class);
+        $this->expectExceptionCode(1463652560);
+        $this->assertSame('default', (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPointerFieldStringHasMoreThanTwoFields()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [],
+                'ds_pointerField' => 'first,second,third',
+            ],
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1463577497);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', []);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPointerFieldWithStringSingleFieldDoesNotExist()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [],
+                'ds_pointerField' => 'notExist',
+            ],
+        ];
+        $row = [
+            'foo' => '',
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1463578899);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPointerFieldSWithTwoFieldsFirstDoesNotExist()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [],
+                'ds_pointerField' => 'notExist,second',
+            ],
+        ];
+        $row = [
+            'second' => '',
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1463578899);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPointerFieldSWithTwoFieldsSecondDoesNotExist()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [],
+                'ds_pointerField' => 'first,notExist',
+            ],
+        ];
+        $row = [
+            'first' => '',
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1463578900);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsPointerFieldValueIfDataStructureExists()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'thePointerValue' => 'FILE:...'
+                ],
+                'ds_pointerField' => 'aField'
+            ],
+        ];
+        $row = [
+            'aField' => 'thePointerValue',
+        ];
+        $expected = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"thePointerValue"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsDefaultIfPointerFieldValueDoesNotExist()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'default' => 'theDataStructure'
+                ],
+                'ds_pointerField' => 'aField'
+            ],
+        ];
+        $row = [
+            'aField' => 'thePointerValue',
+        ];
+        $expected = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfPointerFieldValueDoesNotExistAndDefaultToo()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'aDifferentDataStructure' => 'aDataStructure'
+                ],
+                'ds_pointerField' => 'aField'
+            ],
+        ];
+        $row = [
+            'aField' => 'aNotDefinedDataStructure',
+        ];
+        $this->expectException(InvalidSinglePointerFieldException::class);
+        $this->expectExceptionCode(1463653197);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * Data provider for getDataStructureIdentifierReturnsValidNameForTwoFieldCombinations
+     */
+    public function getDataStructureIdentifierReturnsValidNameForTwoFieldCombinationsDataProvider()
+    {
+        return [
+            'direct match of two fields' => [
+                [
+                    // $row
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    // registered data structure names
+                    'firstValue,secondValue' => '',
+                ],
+                // expected name
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"firstValue,secondValue"}'
+            ],
+            'match on first field, * for second' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'firstValue,*' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"firstValue,*"}'
+            ],
+            '@deprecated match on second field, * for first' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'secondValue,*' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"secondValue,*"}'
+            ],
+            'match on second field, * for first' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    '*,secondValue' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"*,secondValue"}'
+            ],
+            'match on first field only' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'firstValue' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"firstValue"}'
+            ],
+            'fallback to default' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'default' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}'
+            ],
+            'chain falls through with no match on second value to *' => [
+                [
+                    'firstField' => 'firstValue',
+                    'secondField' => 'noMatch',
+                ],
+                [
+                    'firstValue,secondValue' => '',
+                    'firstValue,*' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"firstValue,*"}'
+            ],
+            'chain falls through with no match on first value to *' => [
+                [
+                    'firstField' => 'noMatch',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'firstValue,secondValue' => '',
+                    '*,secondValue' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"*,secondValue"}'
+            ],
+            '@deprecated chain falls through with no match on first value to *' => [
+                [
+                    'firstField' => 'noMatch',
+                    'secondField' => 'secondValue',
+                ],
+                [
+                    'firstValue,secondValue' => '',
+                    'secondValue,*' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"secondValue,*"}'
+            ],
+            'chain falls through with no match on any field to default' => [
+                [
+                    'firstField' => 'noMatch',
+                    'secondField' => 'noMatchToo',
+                ],
+                [
+                    'firstValue,secondValue' => '',
+                    'secondValue,*' => '',
+                    'default' => '',
+                ],
+                '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}'
+            ],
+        ];
+    }
+
+    /**
+     * @test
+     * @dataProvider getDataStructureIdentifierReturnsValidNameForTwoFieldCombinationsDataProvider
+     * @param array $row
+     * @param array $ds
+     * @param $expected
+     */
+    public function getDataStructureIdentifierReturnsValidNameForTwoFieldCombinations(array $row, array $ds, string $expected)
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => $ds,
+                'ds_pointerField' => 'firstField,secondField'
+            ],
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionForTwoFieldsWithNoMatchAndNoDefault()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds' => [
+                    'firstValue,secondValue' => '',
+                ],
+                'ds_pointerField' => 'firstField,secondField'
+            ],
+        ];
+        $row = [
+            'firstField' => 'noMatch',
+            'secondField' => 'noMatchToo',
+        ];
+        $this->expectException(InvalidCombinedPointerFieldException::class);
+        $this->expectExceptionCode(1463678524);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfParentRowLookupFails()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+            ]
+        ];
+        $row = [
+            'pid' => 42,
+            'tx_templavoila_ds' => null,
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(42, 1)->willReturn(42);
+        $expressionBuilderProphecy->eq('uid', 42)->shouldBeCalled()->willReturn('uid = 42');
+        $queryBuilderProphecy->where('uid = 42')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+
+        // Error case that is tested here: Do not return a valid parent row from db -> exception should be thrown
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(0);
+        $this->expectException(InvalidParentRowException::class);
+        $this->expectExceptionCode(1463833794);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfParentRowsFormALoop()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => null,
+        ];
+        $thirdRow = [
+            'uid' => 1,
+            'pid' => 3,
+            'tx_templavoila_ds' => null,
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $queryBuilderProphecy->createNamedParameter(1, 1)->willReturn(1);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $expressionBuilderProphecy->eq('uid', 1)->shouldBeCalled()->willReturn('uid = 1');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->where('uid = 1')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow, second returns $thirdRow, which points back to $initialRow -> exception
+        $statementProphecy->fetch()->willReturn($secondRow, $thirdRow);
+
+        $this->expectException(InvalidParentRowLoopException::class);
+        $this->expectExceptionCode(1464110956);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfNoValidPointerFoundUntilRoot()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => null,
+        ];
+        $thirdRow = [
+            'uid' => 1,
+            'pid' => 0,
+            'tx_templavoila_ds' => null,
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $queryBuilderProphecy->createNamedParameter(1, 1)->willReturn(1);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $expressionBuilderProphecy->eq('uid', 1)->shouldBeCalled()->willReturn('uid = 1');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->where('uid = 1')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow, second returns $thirdRow. $thirdRow has pid 0 and still no ds -> exception
+        $statementProphecy->fetch()->willReturn($secondRow, $thirdRow);
+
+        $this->expectException(InvalidParentRowRootException::class);
+        $this->expectExceptionCode(1464112555);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfNoValidPointerValueFound()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'aPointerField',
+            ]
+        ];
+        $row = [
+            'aPointerField' => null,
+        ];
+        $this->expectException(InvalidPointerFieldValueException::class);
+        $this->expectExceptionCode(1464114011);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfResorvedPointerValueIsIntegerButDsFieldNameIsNotConfigured()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'aPointerField',
+            ]
+        ];
+        $row = [
+            'aPointerField' => 3,
+        ];
+        $this->expectException(InvalidTcaException::class);
+        $this->expectExceptionCode(1464115639);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierThrowsExceptionIfDsTableFieldIsMisconfigured()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'aPointerField',
+                'ds_tableField' => 'misconfigured',
+            ]
+        ];
+        $row = [
+            'aPointerField' => 3,
+        ];
+        $this->expectException(InvalidTcaException::class);
+        $this->expectExceptionCode(1464116002);
+        (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row);
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForPointerField()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'aPointerField',
+            ]
+        ];
+        $row = [
+            'uid' => 42,
+            'aPointerField' => '<T3DataStructure>...',
+        ];
+        $expected = '{"type":"record","tableName":"aTableName","uid":42,"fieldName":"aPointerField"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForParentLookup()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => 0,
+        ];
+        $thirdRow = [
+            'uid' => 1,
+            'pid' => 0,
+            'tx_templavoila_ds' => '<T3DataStructure>...',
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $queryBuilderProphecy->createNamedParameter(1, 1)->willReturn(1);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $expressionBuilderProphecy->eq('uid', 1)->shouldBeCalled()->willReturn('uid = 1');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->where('uid = 1')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow, second returns $thirdRow. $thirdRow resolves ds
+        $statementProphecy->fetch()->willReturn($secondRow, $thirdRow);
+
+        $expected = '{"type":"record","tableName":"aTableName","uid":1,"fieldName":"tx_templavoila_ds"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForParentLookupAndBreaksLoop()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => '<T3DataStructure>...',
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow. $secendRow resolves DS and does not look further up
+        $statementProphecy->fetch()->willReturn($secondRow);
+
+        $expected = '{"type":"record","tableName":"aTableName","uid":2,"fieldName":"tx_templavoila_ds"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForParentLookupAndPrefersSubField()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+                'ds_pointerField_searchParent_subField' => 'tx_templavoila_next_ds',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+            'tx_templavoila_next_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => '<T3DataStructure>...',
+            'tx_templavoila_next_ds' => 'anotherDataStructure',
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->addSelect('tx_templavoila_next_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow. $secendRow resolves DS and does not look further up
+        $statementProphecy->fetch()->willReturn($secondRow);
+
+        $expected = '{"type":"record","tableName":"aTableName","uid":2,"fieldName":"tx_templavoila_next_ds"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForTableAndFieldPointer()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'aPointerField',
+                'ds_tableField' => 'foreignTableName:foreignTableField',
+            ]
+        ];
+        $row = [
+            'uid' => 3,
+            'pid' => 2,
+            'aPointerField' => 42,
+        ];
+        $expected = '{"type":"record","tableName":"foreignTableName","uid":42,"fieldName":"foreignTableField"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $row));
+    }
+
+    /**
+     * @test
+     */
+    public function getDataStructureIdentifierReturnsValidIdentifierForTableAndFieldPointerWithParentLookup()
+    {
+        $fieldTca = [
+            'config' => [
+                'ds_pointerField' => 'tx_templavoila_ds',
+                'ds_pointerField_searchParent' => 'pid',
+                'ds_pointerField_searchParent_subField' => 'tx_templavoila_next_ds',
+                'ds_tableField' => 'foreignTableName:foreignTableField',
+            ]
+        ];
+        $initialRow = [
+            'uid' => 3,
+            'pid' => 2,
+            'tx_templavoila_ds' => null,
+            'tx_templavoila_next_ds' => null,
+        ];
+        $secondRow = [
+            'uid' => 2,
+            'pid' => 1,
+            'tx_templavoila_ds' => '<T3DataStructure>...',
+            'tx_templavoila_next_ds' => '42',
+        ];
+
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        // Two queries are done, so we need two instances
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('uid', 'pid', 'tx_templavoila_ds')->shouldBeCalled();
+        $queryBuilderProphecy->addSelect('tx_templavoila_next_ds')->shouldBeCalled();
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(2, 1)->willReturn(2);
+        $expressionBuilderProphecy->eq('uid', 2)->shouldBeCalled()->willReturn('uid = 2');
+        $queryBuilderProphecy->where('uid = 2')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->rowCount()->shouldBeCalled()->willReturn(1);
+
+        // First db call returns $secondRow. $secendRow resolves DS and does not look further up
+        $statementProphecy->fetch()->willReturn($secondRow);
+
+        $expected = '{"type":"record","tableName":"foreignTableName","uid":42,"fieldName":"foreignTableField"}';
+        $this->assertSame($expected, (new FlexFormTools())->getDataStructureIdentifier($fieldTca, 'aTableName', 'aFieldName', $initialRow));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionWithEmptyString()
+    {
+        $this->expectException(InvalidIdentifierException::class);
+        $this->expectExceptionCode(1478100828);
+        (new FlexFormTools())->parseDataStructureByIdentifier('');
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierIfIdentifierDoesNotResolveToArray()
+    {
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478345642);
+        (new FlexFormTools())->parseDataStructureByIdentifier('egon');
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierCallsRegisteredHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePreProcessHookThrowException::class,
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478112411);
+        (new FlexFormTools())->parseDataStructureByIdentifier('{"some":"input"}');
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionIfHookReturnsNoString()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePreProcessHookReturnObject::class
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478168512);
+        (new FlexFormTools())->parseDataStructureByIdentifier('{"some":"input"}');
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierUsesCasualLogicIfHookReturnsNoIdentifier()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePreProcessHookReturnEmptyString::class
+        ];
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => '',
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierParsesDataStructureReturnedByHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePreProcessHookReturnString::class
+        ];
+        $identifier = '{"type":"myExtension"}';
+        $expected = [
+            'sheets' => '',
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierParsesDataStructureFromFirstMatchingHook()
+    {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePreProcessHookReturnEmptyString::class,
+            DataStructureParsePreProcessHookReturnString::class,
+            DataStructureParsePreProcessHookThrowException::class
+        ];
+        $identifier = '{"type":"myExtension"}';
+        $expected = [
+            'sheets' => '',
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionForInvalidSyntax()
+    {
+        $this->expectException(InvalidIdentifierException::class);
+        $this->expectExceptionCode(1478104554);
+        (new FlexFormTools())->parseDataStructureByIdentifier('{"type":"bernd"}');
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionForIncompleteTcaSyntax()
+    {
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478113471);
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName"}';
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionForInvalidTcaSyntaxPointer()
+    {
+        $this->expectException(InvalidIdentifierException::class);
+        $this->expectExceptionCode(1478105491);
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierResolvesTcaSyntaxPointer()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => '',
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionForIncompleteRecordSyntax()
+    {
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478113873);
+        $identifier = '{"type":"record","tableName":"foreignTableName","uid":42}';
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierResolvesRecordSyntaxPointer()
+    {
+        // Prophecies and revelations for a lot of the database stack classes
+        $queryBuilderProphecy = $this->prophesize(QueryBuilder::class);
+        $queryBuilderRevelation = $queryBuilderProphecy->reveal();
+        $connectionPoolProphecy = $this->prophesize(ConnectionPool::class);
+        $queryRestrictionContainerProphecy = $this->prophesize(QueryRestrictionContainerInterface::class);
+        $queryRestrictionContainerRevelation = $queryRestrictionContainerProphecy->reveal();
+        $expressionBuilderProphecy = $this->prophesize(ExpressionBuilder::class);
+        $statementProphecy = $this->prophesize(Statement::class);
+
+        // Register connection pool revelation in framework, this is the entry point used by system under test
+        GeneralUtility::addInstance(ConnectionPool::class, $connectionPoolProphecy->reveal());
+
+        // Simulate method call flow on database objects and verify correct query is built
+        $connectionPoolProphecy->getQueryBuilderForTable('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryRestrictionContainerProphecy->removeAll()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryRestrictionContainerProphecy->add(Argument::cetera())->shouldBeCalled();
+        $queryBuilderProphecy->getRestrictions()->shouldBeCalled()->willReturn($queryRestrictionContainerRevelation);
+        $queryBuilderProphecy->select('dataprot')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->from('aTableName')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->expr()->shouldBeCalled()->willReturn($expressionBuilderProphecy->reveal());
+        $queryBuilderProphecy->createNamedParameter(42, 1)->willReturn(42);
+        $expressionBuilderProphecy->eq('uid', 42)->shouldBeCalled()->willReturn('uid = 42');
+        $queryBuilderProphecy->where('uid = 42')->shouldBeCalled()->willReturn($queryBuilderRevelation);
+        $queryBuilderProphecy->execute()->shouldBeCalled()->willReturn($statementProphecy->reveal());
+        $statementProphecy->fetchColumn(0)->willReturn('
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ');
+        $identifier = '{"type":"record","tableName":"aTableName","uid":42,"fieldName":"dataprot"}';
+        $expected = [
+            'sheets' => '',
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionIfDataStructureFileDoesNotExist()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default']
+            = 'FILE:EXT:core/Does/Not/Exist.xml';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478105826);
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierFetchesFromFile()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default']
+            = ' FILE:EXT:core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureWithSheet.xml ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => [
+                'sDEF' => [
+                    'ROOT' => [
+                        'type' => 'array',
+                        'el' => [
+                            'aFlexField' => [
+                                'TCEforms' => [
+                                    'label' => 'aFlexFieldLabel',
+                                    'config' => [
+                                        'type' => 'input',
+                                    ],
+                                ],
+                            ],
+                        ],
+                        'TCEforms' => [
+                            'sheetTitle' => 'aTitle',
+                        ],
+                    ],
+                ],
+            ]
+        ];
+        $this->assertEquals($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionForInvalidXmlStructure()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets>
+                    <bar>
+                </sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478106090);
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionIfStructureHasBothSheetAndRoot()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <ROOT></ROOT>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1440676540);
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierCreatesDefaultSheet()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <ROOT>
+                    <TCEforms>
+                        <sheetTitle>aTitle</sheetTitle>
+                    </TCEforms>
+                    <type>array</type>
+                    <el>
+                        <aFlexField>
+                            <TCEforms>
+                                <label>aFlexFieldLabel</label>
+                                <config>
+                                    <type>input</type>
+                                </config>
+                            </TCEforms>
+                        </aFlexField>
+                    </el>
+                </ROOT>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => [
+                'sDEF' => [
+                    'ROOT' => [
+                        'type' => 'array',
+                        'el' => [
+                            'aFlexField' => [
+                                'TCEforms' => [
+                                    'label' => 'aFlexFieldLabel',
+                                    'config' => [
+                                        'type' => 'input',
+                                    ],
+                                ],
+                            ],
+                        ],
+                        'TCEforms' => [
+                            'sheetTitle' => 'aTitle',
+                        ],
+                    ],
+                ],
+            ]
+        ];
+        $this->assertEquals($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierResolvesExtReferenceForSingleSheets()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets>
+                    <aSheet>
+                        EXT:core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureOfSingleSheet.xml
+                    </aSheet>
+                </sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => [
+                'aSheet' => [
+                    'ROOT' => [
+                        'type' => 'array',
+                        'el' => [
+                            'aFlexField' => [
+                                'TCEforms' => [
+                                    'label' => 'aFlexFieldLabel',
+                                    'config' => [
+                                        'type' => 'input',
+                                    ],
+                                ],
+                            ],
+                        ],
+                        'TCEforms' => [
+                            'sheetTitle' => 'aTitle',
+                        ],
+                    ],
+                ],
+            ]
+        ];
+        $this->assertEquals($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierResolvesExtReferenceForSingleSheetsWithFilePrefix()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets>
+                    <aSheet>
+                        FILE:EXT:core/Tests/Unit/Configuration/FlexForm/Fixtures/DataStructureOfSingleSheet.xml
+                    </aSheet>
+                </sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $expected = [
+            'sheets' => [
+                'aSheet' => [
+                    'ROOT' => [
+                        'type' => 'array',
+                        'el' => [
+                            'aFlexField' => [
+                                'TCEforms' => [
+                                    'label' => 'aFlexFieldLabel',
+                                    'config' => [
+                                        'type' => 'input',
+                                    ],
+                                ],
+                            ],
+                        ],
+                        'TCEforms' => [
+                            'sheetTitle' => 'aTitle',
+                        ],
+                    ],
+                ],
+            ]
+        ];
+        $this->assertEquals($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierCallsPostProcessHook()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePostProcessHookThrowException::class,
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478351691);
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierThrowsExceptionIfPostProcessHookReturnsNoArray()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePostProcessHookReturnString::class,
+        ];
+        $this->expectException(\RuntimeException::class);
+        $this->expectExceptionCode(1478350806);
+        (new FlexFormTools())->parseDataStructureByIdentifier($identifier);
+    }
+
+    /**
+     * @test
+     */
+    public function parseDataStructureByIdentifierPostProcessHookManipulatesDataStructure()
+    {
+        $GLOBALS['TCA']['aTableName']['columns']['aFieldName']['config']['ds']['default'] = '
+            <T3DataStructure>
+                <sheets></sheets>
+            </T3DataStructure>
+        ';
+        $identifier = '{"type":"tca","tableName":"aTableName","fieldName":"aFieldName","dataStructureKey":"default"}';
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS'][FlexFormTools::class]['flexParsing'] = [
+            DataStructureParsePostProcessHookReturnArray::class,
+        ];
+        $expected = [
+            'sheets' => [
+                'foo' => 'bar'
+            ]
+        ];
+        $this->assertSame($expected, (new FlexFormTools())->parseDataStructureByIdentifier($identifier));
+    }
+
+    /**
+     * @test
+     */
     public function traverseFlexFormXmlDataRecurseDoesNotFailOnNotExistingField()
     {
         $dataStruct = [
index 94f225b..209b72c 100644 (file)
@@ -14,7 +14,9 @@ namespace TYPO3\CMS\Core\Tests\Unit\DataHandler;
  * The TYPO3 project - inspiring people to share!
  */
 
+use Prophecy\Argument;
 use TYPO3\CMS\Core\Authentication\BackendUserAuthentication;
+use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
 use TYPO3\CMS\Core\DataHandling\DataHandler;
 use TYPO3\CMS\Core\Tests\AccessibleObjectInterface;
 use TYPO3\CMS\Core\Tests\Unit\DataHandling\Fixtures\AllowAccessHookFixture;
@@ -400,6 +402,10 @@ class DataHandlerTest extends \TYPO3\CMS\Core\Tests\UnitTestCase
         $hookMock->expects($this->once())->method('checkFlexFormValue_beforeMerge');
         $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['checkFlexFormValue'][] = $hookClass;
         GeneralUtility::addInstance($hookClass, $hookMock);
+        $flexFormToolsProphecy = $this->prophesize(FlexFormTools::class);
+        $flexFormToolsProphecy->getDataStructureIdentifier(Argument::cetera())->willReturn('anIdentifier');
+        $flexFormToolsProphecy->parseDataStructureByIdentifier('anIdentifier')->willReturn([]);
+        GeneralUtility::addInstance(FlexFormTools::class, $flexFormToolsProphecy->reveal());
         $this->subject->_call('checkValueForFlex', [], [], [], '', 0, '', '', 0, 0, 0, [], '');
     }
 
index 7088dcc..1e0af8d 100644 (file)
@@ -15,6 +15,7 @@ namespace TYPO3\CMS\Impexp;
  */
 
 use TYPO3\CMS\Backend\Utility\BackendUtility;
+use TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools;
 use TYPO3\CMS\Core\Database\ConnectionPool;
 use TYPO3\CMS\Core\DataHandling\DataHandler;
 use TYPO3\CMS\Core\Exception;
@@ -1186,10 +1187,17 @@ class Import extends ImportExport
                         if (!empty($config['flexFormRels']['db']) || !empty($config['flexFormRels']['file'])) {
                             $origRecordRow = BackendUtility::getRecord($table, $thisNewUid, '*');
                             // This will fetch the new row for the element (which should be updated with any references to data structures etc.)
-                            $conf = $GLOBALS['TCA'][$table]['columns'][$field]['config'];
-                            if (is_array($origRecordRow) && is_array($conf) && $conf['type'] === 'flex') {
+                            $fieldTca = $GLOBALS['TCA'][$table]['columns'][$field];
+                            if (is_array($origRecordRow) && is_array($fieldTca['config']) && $fieldTca['config']['type'] === 'flex') {
                                 // Get current data structure and value array:
-                                $dataStructArray = BackendUtility::getFlexFormDS($conf, $origRecordRow, $table, $field);
+                                $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+                                $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                                    $fieldTca,
+                                    $table,
+                                    $field,
+                                    $origRecordRow
+                                );
+                                $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
                                 $currentValueArray = GeneralUtility::xml2array($updateData[$table][$thisNewUid][$field]);
                                 // Do recursive processing of the XML data:
                                 $iteratorObj = GeneralUtility::makeInstance(DataHandler::class);
@@ -1198,7 +1206,7 @@ class Import extends ImportExport
                                     $currentValueArray['data'],
                                     [],
                                     [],
-                                    $dataStructArray,
+                                    $dataStructureArray,
                                     [$table, $thisNewUid, $field, $config],
                                     'remapListedDBRecords_flexFormCallBack'
                                 );
@@ -1294,19 +1302,25 @@ class Import extends ImportExport
                         // Now, if there are any fields that require substitution to be done, lets go for that:
                         foreach ($fieldsIndex as $field => $softRefCfgs) {
                             if (is_array($GLOBALS['TCA'][$table]['columns'][$field])) {
-                                $conf = $GLOBALS['TCA'][$table]['columns'][$field]['config'];
-                                if ($conf['type'] === 'flex') {
+                                if ($GLOBALS['TCA'][$table]['columns'][$field]['config']['type'] === 'flex') {
                                     // This will fetch the new row for the element (which should be updated with any references to data structures etc.)
                                     $origRecordRow = BackendUtility::getRecord($table, $thisNewUid, '*');
                                     if (is_array($origRecordRow)) {
                                         // Get current data structure and value array:
-                                        $dataStructArray = BackendUtility::getFlexFormDS($conf, $origRecordRow, $table, $field);
+                                        $flexFormTools = GeneralUtility::makeInstance(FlexFormTools::class);
+                                        $dataStructureIdentifier = $flexFormTools->getDataStructureIdentifier(
+                                            $GLOBALS['TCA'][$table]['columns'][$field],
+                                            $table,
+                                            $field,
+                                            $origRecordRow
+                                        );
+                                        $dataStructureArray = $flexFormTools->parseDataStructureByIdentifier($dataStructureIdentifier);
                                         $currentValueArray = GeneralUtility::xml2array($origRecordRow[$field]);
                                         // Do recursive processing of the XML data:
                                         /** @var $iteratorObj DataHandler */
                                         $iteratorObj = GeneralUtility::makeInstance(DataHandler::class);
                                         $iteratorObj->callBackObj = $this;
-                                        $currentValueArray['data'] = $iteratorObj->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructArray, [$table, $uid, $field, $softRefCfgs], 'processSoftReferences_flexFormCallBack');
+                                        $currentValueArray['data'] = $iteratorObj->checkValue_flex_procInData($currentValueArray['data'], [], [], $dataStructureArray, [$table, $uid, $field, $softRefCfgs], 'processSoftReferences_flexFormCallBack');
                                         // The return value is set as an array which means it will be processed by DataHandler for file and DB references!
                                         if (is_array($currentValueArray['data'])) {
                                             $inData[$table][$thisNewUid][$field] = $currentValueArray;
index c107cad..3f876f1 100644 (file)
@@ -47,7 +47,7 @@ Assumptions:
 The assumptions are not requirements by the TYPO3 API but reflects the de facto implementation of most TYPO3 installations and therefore a practical approach to cleaning up the uploads/ folder.
 Therefore, if all "group" type fields in TCA and flexforms are positioned inside the uploads/ folder and if no files inside are managed manually it should be safe to clean out files with no relations found in the system.
 Under such circumstances there should theoretically be no lost files in the uploads/ folder since DataHandler should have managed relations automatically including adding and deleting files.
-However, there is at least one reason known to why files might be found lost and that is when FlexForms are used. In such a case a change of/in the Data Structure XML (or the ability of the system to find the Data Structure definition!) used for the flexform could leave lost files behind. This is not unlikely to happen when records are deleted. More details can be found in a note to the function TYPO3\\CMS\\Backend\\Utility\\BackendUtility::getFlexFormDS()
+However, there is at least one reason known to why files might be found lost and that is when FlexForms are used. In such a case a change of/in the Data Structure XML (or the ability of the system to find the Data Structure definition!) used for the flexform could leave lost files behind. This is not unlikely to happen when records are deleted. More details can be found in a note to the function FlexFormTools->getDataStructureIdentifier()
 Another scenario could of course be de-installation of extensions which managed files in the uploads/ folders.
 
 If the option "--dry-run" is not set, the files are then deleted automatically.