[TASK] Compatibility with TYPO3 CMS 7 28/40128/2
authorFrancois Suter <francois@typo3.org>
Tue, 9 Jun 2015 12:02:07 +0000 (14:02 +0200)
committerFrancois Suter <francois@typo3.org>
Tue, 9 Jun 2015 12:08:26 +0000 (14:08 +0200)
Move to namespaces.
Ensure compatibility with TYPO3 CMS 7.
Raise requirements to TYPO3 CMS 6.2.

Resolves: #67370
Releases: 2.0
Change-Id: Ia0543889f76242cf0b229ae9f0964d41cfe981a5
Reviewed-on: http://review.typo3.org/40128
Reviewed-by: Francois Suter <francois@typo3.org>
Tested-by: Francois Suter <francois@typo3.org>
69 files changed:
ChangeLog
Classes/Ajax/AjaxHandler.php [new file with mode: 0644]
Classes/Cache/CacheHandler.php [new file with mode: 0755]
Classes/Cache/CacheParametersProcessorInterface.php [new file with mode: 0644]
Classes/Component/DataProvider.php [new file with mode: 0644]
Classes/Exception/InvalidQueryException.php [new file with mode: 0644]
Classes/Hook/DataFilterHook.php [new file with mode: 0644]
Classes/Parser/Fulltext.php [deleted file]
Classes/Parser/FulltextParser.php [new file with mode: 0644]
Classes/Parser/QueryParser.php [new file with mode: 0644]
Classes/Parser/SqlParser.php [new file with mode: 0644]
Classes/Sample/DataQueryHook.php [new file with mode: 0644]
Classes/UserFunction/FormEngine.php [new file with mode: 0644]
Classes/Userfunc/FormEngine.php [deleted file]
Classes/Utility/DatabaseAnalyser.php
Classes/Utility/QueryObject.php [new file with mode: 0644]
Classes/Utility/SqlUtility.php [new file with mode: 0644]
Classes/Wizard/QueryCheckWizard.php [new file with mode: 0644]
Configuration/TCA/Overrides/tx_datafilter_filters.php [new file with mode: 0644]
Configuration/TCA/tx_dataquery_queries.php [new file with mode: 0644]
Documentation/.gitignore
Documentation/BehindTheScenes/AdvancedAliases/Index.rst
Documentation/BehindTheScenes/DataFilters/Index.rst
Documentation/BehindTheScenes/Output/Index.rst
Documentation/Developers/Index.rst
Documentation/Introduction/Index.rst
Documentation/Queries/Comments/Index.rst
Documentation/Queries/Expressions/Index.rst
Documentation/Queries/Joins/Index.rst
Documentation/Settings.yml
Resources/Public/Icons/AddDataQueryWizard.png [new file with mode: 0644]
Resources/Public/Icons/DataQuery.png [new file with mode: 0755]
Resources/Public/JavaScript/CheckWizard.js [new file with mode: 0644]
Resources/Public/Styles/CheckWizard.css [new file with mode: 0644]
Tests/Unit/DataProviderTest.php [new file with mode: 0755]
Tests/Unit/QueryParserTest.php [new file with mode: 0755]
Tests/Unit/SqlBuilderDefaultTest.php [new file with mode: 0644]
Tests/Unit/SqlBuilderLanguageTest.php [new file with mode: 0644]
Tests/Unit/SqlBuilderTest.php [new file with mode: 0644]
Tests/Unit/SqlBuilderWorkspaceTest.php [new file with mode: 0644]
Tests/Unit/SqlParserTest.php [new file with mode: 0644]
class.tx_dataquery_ajax.php [deleted file]
class.tx_dataquery_cache.php [deleted file]
class.tx_dataquery_parser.php [deleted file]
class.tx_dataquery_queryobject.php [deleted file]
class.tx_dataquery_sqlparser.php [deleted file]
class.tx_dataquery_sqlutility.php [deleted file]
class.tx_dataquery_wrapper.php [deleted file]
ext_autoload.php [deleted file]
ext_emconf.php
ext_icon.gif [deleted file]
ext_icon.png [new file with mode: 0644]
ext_localconf.php
ext_tables.php
hooks/class.tx_dataquery_datafilterhook.php [deleted file]
interface.tx_dataquery_cacheParametersProcessor.php [deleted file]
res/icons/add_dataquery_wizard.gif [deleted file]
res/icons/icon_tx_dataquery_queries.gif [deleted file]
res/js/check_wizard.js [deleted file]
samples/class.tx_dataquery_sample_hook.php [deleted file]
tca.php [deleted file]
tests/tx_dataquery_parser_Test.php [deleted file]
tests/tx_dataquery_sqlbuilder_Test.php [deleted file]
tests/tx_dataquery_sqlbuilder_default_Test.php [deleted file]
tests/tx_dataquery_sqlbuilder_language_Test.php [deleted file]
tests/tx_dataquery_sqlbuilder_workspace_Test.php [deleted file]
tests/tx_dataquery_sqlparser_Test.php [deleted file]
tests/tx_dataquery_wrapper_Test.php [deleted file]
wizards/class.tx_dataquery_wizards_check.php [deleted file]

index 7d14f9a..37e7309 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,7 @@
+2015-06-09 Francois Suter (Cobweb)  <typo3@cobweb.ch>
+
+       * Moved to namespaces, verified compatibility with TYPO3 CMS 7, resolves #67370
+
 2015-02-27 Francois Suter (Cobweb)  <typo3@cobweb.ch>
 
        * Fixed wrong output in custom fulltext field, resolves #65409
diff --git a/Classes/Ajax/AjaxHandler.php b/Classes/Ajax/AjaxHandler.php
new file mode 100644 (file)
index 0000000..2b00786
--- /dev/null
@@ -0,0 +1,123 @@
+<?php
+namespace Tesseract\Dataquery\Ajax;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use TYPO3\CMS\Core\Http\AjaxRequestHandler;
+use TYPO3\CMS\Core\Messaging\FlashMessage;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * This class answers to AJAX calls from the 'dataquery' extension.
+ *
+ * @author Fabien Udriot <fabien.udriot@ecodev.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class AjaxHandler {
+
+       /**
+        * Returns the parsed query through dataquery parser
+        * or error messages from exceptions should any have been thrown
+        * during query parsing.
+        *
+        * @param array $parameters Empty array (yes, that's weird but true)
+        * @param AjaxRequestHandler $ajaxObj Back-reference to the calling object
+        * @return      void
+        */
+       public function validate($parameters, AjaxRequestHandler $ajaxObj) {
+               $parsingSeverity = FlashMessage::OK;
+               $executionSeverity = FlashMessage::OK;
+               $executionMessage = '';
+               $warningMessage = '';
+               /** @var \TYPO3\CMS\Lang\LanguageService $languageService */
+               $languageService = $GLOBALS['LANG'];
+
+               // Try parsing and building the query
+               try {
+                       // Get the query to parse from the GET/POST parameters
+                       $query = GeneralUtility::_GP('query');
+                       // Create an instance of the parser
+                       /** @var $parser \Tesseract\Dataquery\Parser\QueryParser */
+                       $parser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Parser\\QueryParser');
+                       // Clean up and prepare the query string
+                       $query = $parser->prepareQueryString($query);
+                       // Parse the query
+                       // NOTE: if the parsing fails, an exception will be received, which is handled further down
+                       // The parser may return a warning, though
+                       $warningMessage = $parser->parseQuery($query);
+                       // Build the query
+                       $parsedQuery = $parser->buildQuery();
+                       // The query building completed, issue success message
+                       $parsingTitle = $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.success');
+                       $parsingMessage = $parsedQuery;
+
+                       // Force a LIMIT to 1 and try executing the query
+                       $parser->getSQLObject()->structure['LIMIT'] = 1;
+                       // Rebuild the query with the new limit
+                       $executionQuery = $parser->buildQuery();
+                       // Execute query and report outcome
+                       /** @var \TYPO3\CMS\Core\Database\DatabaseConnection $databaseConnection */
+                       $databaseConnection = $GLOBALS['TYPO3_DB'];
+                       $res = $databaseConnection->sql_query($executionQuery);
+                       if ($res === FALSE) {
+                               $executionSeverity = FlashMessage::ERROR;
+                               $errorMessage = $databaseConnection->sql_error();
+                               $executionMessage = sprintf(
+                                       $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.executionFailed'),
+                                       $errorMessage
+                               );
+                       } else {
+                               $executionMessage = $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.executionSuccessful');
+                       }
+               }
+               catch(\Exception $e) {
+                       // The query parsing failed, issue error message
+                       $parsingSeverity = FlashMessage::ERROR;
+                       $parsingTitle = $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.failure');
+                       $exceptionCode = $e->getCode();
+                       $parsingMessage = $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.exception-' . $exceptionCode);
+               }
+               // Render parsing result as flash message
+               /** @var $flashMessage FlashMessage */
+               $flashMessage = GeneralUtility::makeInstance(
+                       'TYPO3\\CMS\\Core\\Messaging\\FlashMessage',
+                       $parsingMessage,
+                       $parsingTitle,
+                       $parsingSeverity
+               );
+               $content = $flashMessage->render();
+               // If a warning was returned by the query parser, display it here
+               if (!empty($warningMessage)) {
+                       $flashMessage = GeneralUtility::makeInstance(
+                               'TYPO3\\CMS\\Core\\Messaging\\FlashMessage',
+                               $warningMessage,
+                                       $languageService->sL('LLL:EXT:dataquery/locallang.xml:query.warning'),
+                               FlashMessage::WARNING
+                       );
+                       $content .= $flashMessage->render();
+               }
+               // If the query was also executed, render execution result
+               if (!empty($executionMessage)) {
+                       $flashMessage = GeneralUtility::makeInstance(
+                               'TYPO3\\CMS\\Core\\Messaging\\FlashMessage',
+                               $executionMessage,
+                               '',
+                               $executionSeverity
+                       );
+                       $content .= $flashMessage->render();
+               }
+               $ajaxObj->addContent('dataquery', $content);
+       }
+}
diff --git a/Classes/Cache/CacheHandler.php b/Classes/Cache/CacheHandler.php
new file mode 100755 (executable)
index 0000000..3bdd168
--- /dev/null
@@ -0,0 +1,42 @@
+<?php
+namespace Tesseract\Dataquery\Cache;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Cache management class for extension "dataquery".
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class CacheHandler {
+
+       /**
+        * Clears the dataquery for selected pages only.
+        *
+        * @param array $parameters Parameters passed by DataHandler, including the pages to clear the cache for
+        * @param \TYPO3\CMS\Core\DataHandling\DataHandler $parentObject Reference to the calling DataHandler object
+        * @return void
+        */
+       public function clearCache($parameters, $parentObject) {
+               // Clear the dataquery cache for all the pages passed to this method
+               if (isset($parameters['pageIdArray']) && count($parameters['pageIdArray']) > 0) {
+                       $GLOBALS['TYPO3_DB']->exec_DELETEquery(
+                               'tx_dataquery_cache',
+                               'page_id IN (' . implode(',', $parameters['pageIdArray']) . ')'
+                       );
+               }
+       }
+}
diff --git a/Classes/Cache/CacheParametersProcessorInterface.php b/Classes/Cache/CacheParametersProcessorInterface.php
new file mode 100644 (file)
index 0000000..07d5700
--- /dev/null
@@ -0,0 +1,36 @@
+<?php
+namespace Tesseract\Dataquery\Cache;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Interface which defines the method to implement when creating a hook to process cache parameters.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+interface CacheParametersProcessorInterface {
+       /**
+        * This method must be implemented for processing cache parameters.
+        *
+        * It receives a reference to the current cache parameters and a back-reference to the calling object.
+        * It is expected to return the cache parameters, modified or not.
+        *
+        * @param array $cacheParameters Current cache parameters
+        * @param \Tesseract\Dataquery\Component\DataProvider $parentObject Back-reference to the calling object
+        * @return array Modified cache parameters
+        */
+       public function processCacheParameters($cacheParameters, $parentObject);
+}
diff --git a/Classes/Component/DataProvider.php b/Classes/Component/DataProvider.php
new file mode 100644 (file)
index 0000000..d882c22
--- /dev/null
@@ -0,0 +1,1199 @@
+<?php
+namespace Tesseract\Dataquery\Component;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use Cobweb\Overlays\OverlayEngine;
+use Tesseract\Dataquery\Cache\CacheParametersProcessorInterface;
+use Tesseract\Tesseract\Service\ProviderBase;
+use Tesseract\Tesseract\Tesseract;
+use Tesseract\Tesseract\Utility\Utilities;
+use TYPO3\CMS\Core\Messaging\FlashMessage;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * Wrapper for data query
+ * This class is used to get the results of a specific data query
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class DataProvider extends ProviderBase {
+       public $extKey = 'dataquery';
+
+       /**
+        * @var array Extension configuration
+        */
+       protected $configuration;
+
+       /**
+        * @var string Name of the main table of the query
+        */
+       protected $mainTable;
+
+       /**
+        * Local instance of the SQL parser class
+        *
+        * @var \Tesseract\Dataquery\Parser\QueryParser $sqlParser
+        */
+       protected $sqlParser;
+
+       /**
+        * @var array List of fields used for sorting recordset
+        */
+       static public $sortingFields = array();
+
+       /**
+        * @var int Current level of sorting
+        */
+       static public $sortingLevel = 0;
+
+       /**
+        * @var string Type of data structure to provide (default is idList)
+        */
+       protected $dataStructureType = Tesseract::IDLIST_STRUCTURE_TYPE;
+
+       public function __construct() {
+               $this->initialise();
+       }
+
+       /**
+        * Performs various initializations that are shared between the constructor
+        * and the reset() method inherited from the service interface.
+        *
+        * NOTE: this method is NOT called init() to avoid conflicts with the init() method of the service interface.
+        *
+        * @return      void
+        */
+       public function initialise() {
+               $this->configuration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf'][$this->extKey]);
+               $this->mainTable = '';
+               $this->sqlParser = GeneralUtility::makeInstance(
+                       'Tesseract\\Dataquery\\Parser\\QueryParser',
+                       $this
+               );
+       }
+
+       /**
+        * Gets the data for the current query and return it in a standardised format.
+        *
+        * @return array Resulting data structure
+        */
+       protected function getQueryData() {
+               $dataStructure = array();
+               $returnStructure = array();
+
+               // If the cache duration is not set to 0, try to find a cached query
+               // Avoid that if global no_cache flag is set or if in a workspace
+               $hasStructure = FALSE;
+               if (!empty($this->providerData['cache_duration']) && empty($GLOBALS['TSFE']->no_cache) && empty($GLOBALS['TSFE']->sys_page->versioningPreview)) {
+                       try {
+                               $dataStructure = $this->getCachedStructure();
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($dataStructure);
+                               $hasStructure = TRUE;
+                       }
+                       // No structure was found, set flag that there's no structure yet
+                       // Add a message for debugging
+                       catch (\Exception $e) {
+                               $hasStructure = FALSE;
+                               $this->controller->addMessage(
+                                       $this->extKey,
+                                       'No cached structure found',
+                                       '',
+                                       FlashMessage::NOTICE
+                               );
+                       }
+               }
+
+               // If there's no structure yet, assemble it
+               if (!$hasStructure) {
+                       try {
+                               // Parse the query and log any warning that may have been raised
+                               $warning = $this->sqlParser->parseQuery($this->providerData['sql_query']);
+                               if (!empty($warning)) {
+                                       $this->controller->addMessage(
+                                               $this->extKey,
+                                               $warning,
+                                               'Problems in the query',
+                                               FlashMessage::WARNING
+                                       );
+                               }
+
+                               // Pass provider data to the parser
+                               $this->sqlParser->setProviderData($this->providerData);
+
+                               // Add the SQL conditions for the selected TYPO3 mechanisms
+                               $this->sqlParser->addTypo3Mechanisms();
+
+                               // Assemble filters, if defined
+                               if (is_array($this->filter) && count($this->filter) > 0) {
+                                       $this->sqlParser->addFilter($this->filter);
+                               }
+
+                               // Use idList from input SDS, if defined
+                               if (is_array($this->structure) && !empty($this->structure['count'])) {
+                                       $this->sqlParser->addIdList($this->structure['uidListWithTable']);
+                               }
+
+                               // Build the complete query and execute it
+                               try {
+                                       $query = $this->sqlParser->buildQuery();
+                                       $this->controller->addMessage(
+                                               $this->extKey,
+                                               sprintf('Query parsed and rebuilt successfully (%s)', $this->providerData['title']),
+                                               '',
+                                               FlashMessage::OK,
+                                               $query
+                                       );
+
+                                       // Execute the query
+                                       $res = $this->getDatabaseConnection()->sql_query($query);
+                                       // If something went wrong, issue an error
+                                       if ($res === FALSE) {
+                                               $dataStructure = array();
+                                               $this->controller->addMessage(
+                                                       $this->extKey,
+                                                               $this->getDatabaseConnection()->sql_error(),
+                                                       'Query execution failed',
+                                                       FlashMessage::ERROR,
+                                                       $query
+                                               );
+
+                                       // Otherwise prepare the full data structure
+                                       } elseif ($this->dataStructureType == Tesseract::IDLIST_STRUCTURE_TYPE) {
+                                               $dataStructure = $this->assembleIdListStructure($res);
+                                       } else {
+                                               $dataStructure = $this->assembleRecordsetStructure($res);
+                                       }
+                               }
+                               catch (\Exception $e) {
+                                       $this->controller->addMessage($this->extKey, $e->getMessage() . ' (' . $e->getCode() . ')', 'Query building failed', FlashMessage::ERROR);
+                               }
+                       }
+                       catch (\Exception $e) {
+                               $this->controller->addMessage($this->extKey, $e->getMessage() . ' (' . $e->getCode() . ')', 'Query parsing error', FlashMessage::ERROR, $this->providerData);
+                       }
+               }
+
+               // Continue only if there were some results
+               if (count($dataStructure) > 0) {
+                       // Apply limit and offset constraints
+                       $returnStructure = $this->applyLimit($dataStructure);
+                       // Free some memory
+                       unset($dataStructure);
+                       // As a last step add the filter to the data structure
+                       // NOTE: not all Data Consumers may be able to handle this data, but at least it's available
+                       $returnStructure['filter'] = $this->filter;
+
+                       // Hook for post-processing the data structure
+                       if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructure'])) {
+                               foreach($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructure'] as $className) {
+                                       $postProcessor = GeneralUtility::getUserObj($className);
+                                       $returnStructure = $postProcessor->postProcessDataStructure($returnStructure, $this);
+                               }
+                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($returnStructure);
+               }
+               return $returnStructure;
+       }
+
+       /**
+        * Applies limit and offset constraints, if any and if relevant.
+        *
+        * @param array $dataStructure Full data structure
+        * @return array Data structure with limits applied (if relevant)
+        */
+       protected function applyLimit($dataStructure) {
+               $returnStructure = $dataStructure;
+               // Prepare the limit and offset parameters
+               $limit = (isset($this->filter['limit']['max'])) ? $this->filter['limit']['max'] : 0;
+               $offset = 0;
+               if ($limit > 0) {
+                       // If there's a direct pointer, it takes precedence over the offset
+                       if (isset($this->filter['limit']['pointer']) && $this->filter['limit']['pointer'] > 0) {
+                               $offset = $this->filter['limit']['pointer'];
+
+                       } else {
+                               $offset = $limit * ((isset($this->filter['limit']['offset'])) ? $this->filter['limit']['offset'] : 0);
+                               if ($offset < 0) {
+                                       $offset = 0;
+                               }
+                       }
+               }
+
+               // Take the structure and apply limit and offset, if defined
+               if ($limit > 0 || $offset > 0) {
+                       // Reset offset if beyond total number of records
+                       if ($offset > $dataStructure['totalCount']) {
+                               $offset = 0;
+                       }
+                       // Initialise final structure with data that won't change
+                       $returnStructure = array(
+                                                                       'name' => $dataStructure['name'],
+                                                                       'trueName' => $dataStructure['trueName'],
+                                                                       'totalCount' => $dataStructure['totalCount'],
+                                                                       'header' => $dataStructure['header'],
+                                                                       'records' => array()
+                                                                        );
+                       $counter = 0;
+                       $uidList = array();
+                       foreach ($dataStructure['records'] as $record) {
+                               // Get only those records that are after the offset and within the limit
+                               if ($counter >= $offset && ($limit == 0 || ($limit > 0 && $counter - $offset < $limit))) {
+                                       $counter++;
+                                       $returnStructure['records'][] = $record;
+                                       $uidList[] = $record['uid'];
+                               }
+                               // If the offset has not been reached yet, just increase the counter
+                               elseif ($counter < $offset) {
+                                       $counter++;
+                               }
+                               else {
+                                       break;
+                               }
+                       }
+                       $returnStructure['count'] = count($returnStructure['records']);
+                       $returnStructure['uidList'] = implode(',', $uidList);
+               }
+               return $returnStructure;
+       }
+
+       /**
+        * Prepares a full data structure with overlays if needed but without limits and offset.
+        *
+        * This is the structure that will be cached (at the end of method) to be called again from the cache when appropriate.
+        *
+        * @param mysqli_result $res Database resource from the executed query
+        * @return array The full data structure
+        */
+       protected function prepareFullStructure($res) {
+               // Initialise some variables
+               $this->mainTable = $this->sqlParser->getMainTableName();
+               $subtables = $this->sqlParser->getSubtablesNames();
+               $numSubtables = count($subtables);
+               $allTables = $subtables;
+               array_push($allTables, $this->mainTable);
+               $tableAndFieldLabels = $this->sqlParser->getLocalizedLabels();
+               $uidList = array();
+               $fullRecords = array();
+
+               // Get true table names for all tables
+               $allTablesTrueNames = array();
+               foreach ($allTables as $alias) {
+                       $allTablesTrueNames[$alias] = $this->sqlParser->getTrueTableName($alias);
+               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($allTablesTrueNames, 'True table names');
+
+               // Prepare the header parts for all tables
+               $headers = array();
+               foreach ($allTables as $table) {
+                       if (isset($tableAndFieldLabels[$table]['fields'])) {
+                               $headers[$table] = array();
+                               foreach ($tableAndFieldLabels[$table]['fields'] as $key => $label) {
+                                       $headers[$table][$key] = array('label' => $label);
+                       }
+                       }
+               }
+
+               // Act only if there are records
+               $databaseConnection = $this->getDatabaseConnection();
+               if ($databaseConnection->sql_num_rows($res) > 0) {
+                       // Initialise array for storing records
+                       $rows = array($this->mainTable => array(0 => array()));
+                       if ($numSubtables > 0) {
+                               foreach ($subtables as $table) {
+                                       $rows[$table] = array();
+                               }
+                       }
+
+                       // Loop on all records to assemble the raw recordset
+                       $rawRecordset = array();
+                       while ($row = $databaseConnection->sql_fetch_assoc($res)) {
+                               $rawRecordset[] = $row;
+
+                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($rawRecordset, 'Raw result');
+
+                       // Analyze the first row of the raw recordset to get which column belongs to which table
+                       // and which aliases are used, if any
+                       $testRow = $rawRecordset[0];
+                       $columnsMappings = array();
+                       $reverseColumnsMappings = array();
+                       foreach ($testRow as $columnName => $value) {
+                               $info = $this->sqlParser->getTrueFieldName($columnName);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($info, 'Field info');
+                               $columnsMappings[$columnName] = $info;
+                               $reverseColumnsMappings[$info['aliasTable']][$info['field']] = $columnName;
+                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($columnsMappings, 'Columns mappings');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($reverseColumnsMappings, 'Reversed columns mappings');
+
+                       // Split each row in the recordset into parts for each table in the query
+                       $splitRecordset = $this->splitResultIntoSubparts($rawRecordset, $columnsMappings);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($splitRecordset, 'Split result');
+
+                       // If workspace preview is on, records must be overlaid with appropriate version
+                       $versionedRecordset = array();
+                       if ($GLOBALS['TSFE']->sys_page->versioningPreview) {
+                               // Loop on all recordset rows to overlay them
+                               foreach ($splitRecordset as $index => $subParts) {
+                                       $versionedRecord = array();
+                                       $skipRecord = FALSE;
+                                       foreach ($subParts as $alias => $subRow) {
+                                               $table = $allTablesTrueNames[$alias];
+                                               $result = $subRow;
+                                               if ($GLOBALS['TSFE']->sys_page->versioningPreview && $this->sqlParser->mustHandleVersioningOverlay($table)) {
+                                                       $GLOBALS['TSFE']->sys_page->versionOL($table, $result);
+                                               }
+                                               // Include result only if it was not unset during overlaying
+                                               if ($result !== FALSE) {
+                                                       $versionedRecord[$alias] = $result;
+                                               } else {
+                                                       if ($alias == $this->mainTable) {
+                                                               $skipRecord = TRUE;
+                                                       }
+                                               }
+                                       }
+                                       if (!$skipRecord) {
+                                               $versionedRecordset[$index] = $versionedRecord;
+                                       }
+                               }
+                       } else {
+                               $versionedRecordset = $splitRecordset;
+                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($versionedRecordset, 'Versioned split result');
+
+                       // Get overlays for each table, if language is not default
+                       // Set a general flag about having been through this process or not
+                       $hasBeenThroughOverlayProcess = FALSE;
+                       $finalRecordset = array();
+                       if ($GLOBALS['TSFE']->sys_language_content == 0) {
+                               $finalRecordset = array();
+                               foreach ($versionedRecordset as $subParts) {
+                                               // Reassemble the full record
+                                       $overlaidRecord = array();
+                                       foreach ($subParts as $alias => $subRow) {
+                                               if (isset($subRow)) {
+                                                       foreach ($subRow as $field => $value) {
+                                                               $overlaidRecord[$reverseColumnsMappings[$alias][$field]] = $value;
+                                                       }
+                                               }
+                                       }
+                                       $finalRecordset[] = $overlaidRecord;
+                               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Reassembled versioned result');
+
+                               // If no sorting is defined at all, perform fixed order sorting, if defined
+                               // Note this will work only if the input structure's id list refers to a single table
+                               if (!$this->sqlParser->hasOrdering() && !empty($this->structure['count'])) {
+                                               // Add fixed order to recordset
+                                       $uidList = GeneralUtility::trimExplode(',', $this->structure['uidList']);
+                                       $fixedOrder = array_flip($uidList);
+                                       foreach ($finalRecordset as $index => $record) {
+                                               $finalRecordset[$index]['tx_dataquery:fixed_order'] = $fixedOrder[$record['uid']];
+                                       }
+                                       unset($fixedOrder);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Recordset with fixed order');
+
+                                       // Sort recordset according to fixed order
+                                       usort($finalRecordset, array('tx_dataquery_wrapper', 'sortUsingFixedOrder'));
+                               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Recordset after sorting (no overlays)');
+
+                       } else {
+                               // First collect all the uid's for each table
+                               $allUIDs = array();
+                               foreach ($versionedRecordset as $subParts) {
+                                       foreach ($subParts as $alias => $subRow) {
+                                               $table = $allTablesTrueNames[$alias];
+                                               foreach ($subRow as $fieldName => $fieldValue) {
+                                                       if ($fieldName == 'uid' && isset($fieldValue)) {
+                                                               if (!isset($allUIDs[$table])) {
+                                                                       $allUIDs[$table] = array();
+                                                               }
+                                                               $allUIDs[$table][] = $fieldValue;
+                                                       }
+                                               }
+                                       }
+                               }
+                               // Get overlays for all tables
+                               $overlays = array();
+                               $doOverlays = array();
+                               foreach ($allUIDs as $table => $uidList) {
+                                       // Make sure the uid's are unique
+                                       $allUIDs[$table] = array_unique($uidList);
+                                       $doOverlays[$table] = $this->sqlParser->mustHandleLanguageOverlay($table);
+                                       // Get overlays only if needed/possible
+                                       if ($doOverlays[$table] && count($allUIDs[$table]) > 0) {
+                                               $overlays[$table] = OverlayEngine::getOverlayRecords($table, $allUIDs[$table], $GLOBALS['TSFE']->sys_language_content, $this->sqlParser->mustHandleVersioningOverlay($table));
+                                               // Set global overlay process flag to true
+                                               $hasBeenThroughOverlayProcess |= TRUE;
+                                       }
+                               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($allUIDs, 'Unique IDs per table');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($doOverlays, 'Do overlays?');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($overlays, 'Overlays');
+
+                               // Loop on all recordset rows to overlay them
+                               foreach ($versionedRecordset as $subParts) {
+                                               // Overlay each part
+                                       foreach ($subParts as $alias => $subRow) {
+                                               $table = $allTablesTrueNames[$alias];
+                                               $tableCtrl = $GLOBALS['TCA'][$table]['ctrl'];
+                                               $result = $subRow;
+                                               if ($GLOBALS['TSFE']->sys_page->versioningPreview && $this->sqlParser->mustHandleVersioningOverlay($table)) {
+                                                       $GLOBALS['TSFE']->sys_page->versionOL($table, $result);
+                                               }
+                                               if ($doOverlays[$table] && $subRow[$tableCtrl['languageField']] != $GLOBALS['TSFE']->sys_language_content) {
+                                                       // Overlay with record from foreign table
+                                                       if (isset($tableCtrl['transForeignTable']) && isset($overlays[$table][$subRow['uid']])) {
+                                                               $result = OverlayEngine::overlaySingleRecord($table, $subRow, $overlays[$table][$subRow['uid']]);
+
+                                                       // Overlay with record from same table
+                                                       } elseif (isset($overlays[$table][$subRow['uid']][$subRow['pid']])) {
+                                                               $result = OverlayEngine::overlaySingleRecord($table, $subRow, $overlays[$table][$subRow['uid']][$subRow['pid']]);
+
+                                                       // No overlay exists
+                                                       } else {
+                                                               // Take original record, only if non-translated are not hidden, or if language is [All]
+                                                               if ($GLOBALS['TSFE']->sys_language_contentOL == 'hideNonTranslated' && $subRow[$tableCtrl['languageField']] != -1) {
+                                                                       // Skip record
+                                                                       unset($result);
+                                                               } else {
+                                                                       $result = $subRow;
+                                                               }
+                                                       }
+                                               }
+                                               // Include result only if it was not unset during overlaying
+                                               if (isset($result)) {
+                                                       $subParts[$alias] = $result;
+                                               } else {
+                                                       unset($subParts[$alias]);
+                                               }
+                                       }
+                                       // Reassemble the full record
+                                       $overlaidRecord = array();
+                                       foreach ($subParts as $alias => $subRow) {
+                                               if (isset($subRow)) {
+                                                       foreach ($subRow as $field => $value) {
+                                                               // Remap fields, except for special key "_LOCALIZED_UID"
+                                                               if ($field === '_LOCALIZED_UID') {
+                                                                       // Differentiate main table and sub-tables
+                                                                       if ($alias == $this->mainTable) {
+                                                                               $key = '_LOCALIZED_UID';
+                                                                       } else {
+                                                                               $key = $alias . '$_LOCALIZED_UID';
+                                                                       }
+                                                                       $overlaidRecord[$key] = $value;
+                                                               } else {
+                                                                       $overlaidRecord[$reverseColumnsMappings[$alias][$field]] = $value;
+                                                               }
+                                                       }
+                                               // If the record for main table does not exist anymore, remove it entirely
+                                               } else {
+                                                       if ($alias == $this->mainTable) {
+                                                               unset($overlaidRecord);
+                                                               break;
+                                                       }
+                                               }
+                                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($overlaidRecord, 'Overlaid record');
+                                       if (isset($overlaidRecord['uid'])) {
+                                               $finalRecordset[] = $overlaidRecord;
+                                       }
+                               }
+                               // Clean up (potentially large) arrays that are not used anymore
+                               unset($rawRecordset);
+                               unset($overlays);
+                               unset($subParts);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Overlaid recordset');
+
+                               // If the dataquery was provided with a structure,
+                               // use the list of uid's to define a fixed order of records
+                               if (isset($this->structure['uidList'])) {
+                                       $uidList = GeneralUtility::trimExplode(',', $this->structure['uidList']);
+                                       $fixedOrder = array_flip($uidList);
+                                       foreach ($finalRecordset as $index => $record) {
+                                               $finalRecordset[$index]['tx_dataquery:fixed_order'] = $fixedOrder[$record['uid']];
+                                       }
+                                       unset($fixedOrder);
+                               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Final recordset before sorting');
+
+                               // Perform sorting if not handled by SQL
+                               if (!$this->sqlParser->isSqlUsedForOrdering()) {
+                                       self::$sortingFields = $this->sqlParser->getOrderByFields();
+                                       // The names of the fields as stored in the sorting fields configuration
+                                       // match the names used in the SQL query, but not the aliases
+                                       // So they will not match the column names in the full recordset
+                                       // Use the reverse mapping information (if available) to get the aliases
+                                       // (if defined, otherwise stick to field name)
+                                       foreach (self::$sortingFields as $index => $orderInfo) {
+                                               $alias = $this->mainTable;
+                                               // Field may have a special alias which is also not the colum name
+                                               // found in the recordset, but should be used to find that name
+                                               $fieldName = (isset($orderInfo['alias'])) ? $orderInfo['alias'] : $orderInfo['field'];
+                                               $fieldParts = explode('.', $fieldName);
+                                               if (count($fieldParts) == 1) {
+                                                       $field = $fieldParts[0];
+                                               } else {
+                                                       $alias = $fieldParts[0];
+                                                       $field = $fieldParts[1];
+                                               }
+                                               if (isset($reverseColumnsMappings[$alias][$field])) {
+                                                       self::$sortingFields[$index]['field'] = $reverseColumnsMappings[$alias][$field];
+                                               } else {
+                                                       self::$sortingFields[$index]['field'] = $field;
+                                               }
+                                       }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug(self::$sortingFields, 'Sorting fields');
+                                       self::$sortingLevel = 0;
+                                       usort($finalRecordset, array('tx_dataquery_wrapper', 'sortRecordset'));
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($finalRecordset, 'Sorted, overlaid recordset');
+
+                               // If no sorting is defined at all, perform fixed order sorting, if defined
+                               } elseif (!$this->sqlParser->hasOrdering() && isset($this->structure['uidList'])) {
+                                       usort($finalRecordset, array('tx_dataquery_wrapper', 'sortUsingFixedOrder'));
+                               }
+                       } // End of translation handling
+
+                       // Loop on all records to sort them by table. This can be seen as "de-JOINing" the tables.
+                       // This is necessary for such operations as overlays. When overlays are done, tables will be joined again
+                       // but within the format of Standardised Data Structure
+                       $handledUids = array();
+                       foreach ($finalRecordset as $row) {
+                               $currentUID = $row['uid'];
+                               // If we're handling a record we haven't handled before, perform some initialisations
+                               if (!array_key_exists($currentUID, $handledUids)) {
+                                       if ($numSubtables > 0) {
+                                               foreach ($subtables as $table) {
+                                                       $rows[$table][$currentUID] = array();
+                                               }
+                                       }
+                               }
+                               $recordsPerTable = array();
+                               foreach ($row as $fieldName => $fieldValue) {
+                                       // The query contains no joined table
+                                       // All fields belong to the main table
+                                       if ($numSubtables == 0) {
+                                               // If field is special field "_LOCALIZED_UID", keep its name as is
+                                               if ($fieldName === '_LOCALIZED_UID') {
+                                                       $finalFieldName = '_LOCALIZED_UID';
+
+                                               // Otherwise, use mapped name
+                                               } else {
+                                                       $finalFieldName = $columnsMappings[$fieldName]['mapping']['field'];
+                                               }
+                                               $recordsPerTable[$this->mainTable][$finalFieldName] = $fieldValue;
+
+                                       // There are multiple tables
+                                       } else {
+                                               // If field is special field "_LOCALIZED_UID" (possibly prepended with table name),
+                                               // keep it as is
+                                               if (strpos($fieldName, '_LOCALIZED_UID') !== FALSE) {
+                                                       // Handle main table and sub-tables separately
+                                                       if ($fieldName === '_LOCALIZED_UID') {
+                                                               $recordsPerTable[$this->mainTable]['_LOCALIZED_UID'] = $fieldValue;
+                                                       } else {
+                                                               list($table, $finalFieldName) = explode('$', $fieldName);
+                                                               $recordsPerTable[$table]['_LOCALIZED_UID'] = $fieldValue;
+                                                       }
+                                               // For other fields, get the field's true name
+                                               } else {
+                                                       $finalFieldName = $columnsMappings[$fieldName]['mapping']['field'];
+                                                       // However, if the field had an explicit alias, use that alias
+                                                       if (isset($columnsMappings[$fieldName]['mapping']['alias'])) {
+                                                               $finalFieldName = $columnsMappings[$fieldName]['mapping']['alias'];
+                                                       }
+
+                                                       // Field belongs to a subtable
+                                                       if (in_array($columnsMappings[$fieldName]['mapping']['table'], $subtables)) {
+                                                               $subtableName = $columnsMappings[$fieldName]['mapping']['table'];
+                                                               if (isset($fieldValue)) {
+                                                                       $recordsPerTable[$subtableName][$finalFieldName] = $fieldValue;
+                                                               }
+                                                       // Else assume the field belongs to the main table
+                                                       } else {
+                                                               $recordsPerTable[$this->mainTable][$finalFieldName] = $fieldValue;
+                                                       }
+                                               }
+                                       }
+                               }
+                               // If we're handling a record we haven't handled before, store the current information for the main table
+                               if (!array_key_exists($currentUID, $handledUids)) {
+                                       $rows[$this->mainTable][0][] = $recordsPerTable[$this->mainTable];
+                                       $handledUids[$currentUID] = $currentUID;
+                               }
+                               // Store information for each subtable
+                               if ($numSubtables > 0) {
+                                       foreach ($subtables as $table) {
+                                               if (isset($recordsPerTable[$table]) && count($recordsPerTable[$table]) > 0) {
+                                                       $rows[$table][$currentUID][] = $recordsPerTable[$table];
+                                               }
+                                       }
+                               }
+                       }
+                       // Clean up a potentially large array that is not used anymore
+                       unset($finalRecordset);
+                       unset($recordsPerTable);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($rows, 'De-JOINed tables');
+
+                               // Now loop on all the records of the main table and join them to their subtables
+                       $hasInnerJoin = $this->sqlParser->hasInnerJoinOnFirstSubtable();
+                       $uidList = array();
+                       foreach ($rows[$this->mainTable][0] as $aRecord) {
+                               $uidList[] = $aRecord['uid'];
+                               $theFullRecord = $aRecord;
+                               $theFullRecord['__substructure'] = array();
+                               // Check if there are any subtables in the query
+                               $recordsPerSubtable = array();
+                               if ($numSubtables > 0) {
+                                       foreach ($subtables as $table) {
+                                               // Check if there are any subrecords for this record
+                                               if (isset($rows[$table][$aRecord['uid']])) {
+                                                       $numSubrecords = count($rows[$table][$aRecord['uid']]);
+                                                       if ($numSubrecords > 0) {
+                                                               $sublimit = $this->sqlParser->getSubTableLimit($table);
+                                                               $subcounter = 0;
+                                                               // Perform overlays only if language is not default and if necessary for table
+                                                               $subRecords = array();
+                                                               $subUidList = array();
+                                                               // Loop on all subrecords and apply limit, if any
+                                                               foreach ($rows[$table][$aRecord['uid']] as $subRow) {
+                                                                       if (!isset($subRow['uid'])) continue;
+                                                                       // Add the subrecord to the subtable only if it hasn't been included yet
+                                                                       // Multiple identical subrecords may happen when joining several tables together
+                                                                       // Take into account any limit that may have been placed on the number of subrecords in the query
+                                                                       // (using the non-SQL standard keyword MAX)
+                                                                       if (!in_array($subRow['uid'], $subUidList)) {
+                                                                               if ($sublimit == 0 || $subcounter < $sublimit) {
+                                                                                       $subRecords[] = $subRow;
+                                                                                       $subUidList[] = $subRow['uid'];
+                                                                               } elseif ($sublimit != 0 || $subcounter >= $sublimit) {
+                                                                                       break;
+                                                                               }
+                                                                               $subcounter++;
+                                                                       }
+                                                               }
+                                                               // If there are indeed items, add the subtable to the record
+                                                               $numItems = count($subUidList);
+                                                               $recordsPerSubtable[$table] = $numItems;
+                                                               if ($numItems > 0) {
+                                                                       $theFullRecord['__substructure'][$table] = array(
+                                                                                                                                                       'name' => $table,
+                                                                                                                                                       'trueName' => $allTablesTrueNames[$table],
+                                                                                                                                                       'count' => $numItems,
+                                                                                                                                                       'uidList' => implode(',' , $subUidList),
+                                                                                                                                                       'header' => $headers[$table],
+                                                                                                                                                       'records' => $subRecords
+                                                                                                                                               );
+                                                               }
+                                                       }
+                                               }
+                                       }
+                               }
+                               // If the query used INNER JOINs and went through the overlay process,
+                               // preform additional checks
+                               if ($numSubtables > 0 && !empty($hasInnerJoin) && $hasBeenThroughOverlayProcess) {
+                                       // If there are no subrecords after the overlay process, but the query
+                                       // used an INNER JOIN, the record must be removed, so that the end result
+                                       // will look like what it would have been in the default language
+                                       if (!empty($recordsPerSubtable[$hasInnerJoin])) {
+                                               $fullRecords[] = $theFullRecord;
+                                       }
+
+                               // Otherwise just take the record as is
+                               } else {
+                                       $fullRecords[] = $theFullRecord;
+                               }
+                       }
+                       // Clean up a potentially large array that is not used anymore
+                       unset($rows);
+               }
+               $databaseConnection->sql_free_result($res);
+
+                       // Assemble the full structure
+               $numRecords = count($fullRecords);
+               $dataStructure = array(
+                                                       'name' => $this->mainTable,
+                                                       'trueName' => $allTablesTrueNames[$this->mainTable],
+                                                       'count' => $numRecords,
+                                                       'totalCount' => $numRecords,
+                                                       'uidList' => implode(',', $uidList),
+                                                       'header' => $headers[$this->mainTable],
+                                                       'records' => $fullRecords
+                                               );
+               // Clean up a potentially large array that is not used anymore
+               unset($fullRecords);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($dataStructure, 'Finished data structure');
+
+               // Hook for post-processing the data structure before it is stored into cache
+               if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructureBeforeCache'])) {
+                       foreach($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructureBeforeCache'] as $className) {
+                               $postProcessor = GeneralUtility::getUserObj($className);
+                               $dataStructure = $postProcessor->postProcessDataStructureBeforeCache($dataStructure, $this);
+                       }
+               }
+
+               // Store the structure in the cache table,
+               $this->writeStructureToCache($dataStructure);
+
+               // Finally return the assembled structure
+               return $dataStructure;
+       }
+
+       /**
+        * Wrapper around prepareFullStructure() for harmonizing naming between data providers
+        *
+        * @param mysqli_result $res Database resource from the executed query
+        * @return array A recordset-type data structure
+        * @see tx_dataquery_wrapper::prepareFullStructure()
+        */
+       protected function assembleRecordsetStructure($res) {
+               return $this->prepareFullStructure($res);
+       }
+
+       /**
+        * Prepares an id-list type data structure based on the results of the SQL query
+        *
+        * @param mysqli_result $res Database resource from the executed query
+        * @return array An id-list type data structure
+        */
+       protected function assembleIdListStructure($res) {
+               $databaseConnection = $this->getDatabaseConnection();
+               $uidList = array();
+               // Act only if there are records
+               if ($databaseConnection->sql_num_rows($res) > 0) {
+                       $this->mainTable = $this->sqlParser->getMainTableName();
+
+                       // Loop on all records to assemble the list of all uid's
+                       // Note that all overlay mechanisms (language and version) are ignored when assembling
+                       // an id-list type structure. Indeed the uid's returned by such a structure will always
+                       // be fed into some other mechanism that will query the database for a more complete structure
+                       // and overlays will be taken into account at that point
+                       $uidColumns = array();
+                       $firstPass = TRUE;
+                       while ($row = $databaseConnection->sql_fetch_assoc($res)) {
+                               // On the first pass, assemble a list of all the columns containing uid's
+                               // and their relative tables
+                               if ($firstPass) {
+                                       foreach ($row as $columnName => $value) {
+                                               if ($columnName == 'uid' || substr($columnName, -4) == '$uid') {
+                                                       if ($columnName == 'uid') {
+                                                               $table = $this->mainTable;
+                                                       } else {
+                                                               $table = substr($columnName, 0, -4);
+                                                       }
+                                                       $uidColumns[$table] = $columnName;
+                                                       $uidList[$table] = array();
+                                               }
+                                       }
+                                       $firstPass = FALSE;
+                               }
+                               // Loop on all uid columns and store them relative to their related table
+                               foreach ($uidColumns as $table => $columnName) {
+                                       if (isset($row[$columnName])) {
+                                               $uidList[$table][] = $row[$columnName];
+                                       }
+                               }
+                       }
+                       $databaseConnection->sql_free_result($res);
+
+                       // Remove duplicates for every table
+                       foreach ($uidList as $table => $uidListForTable) {
+                               $uidList[$table] = array_unique($uidListForTable);
+                       }
+               }
+
+               // Assemble the full structure
+               $numRecords = (isset($uidList[$this->mainTable])) ? count($uidList[$this->mainTable]) : 0;
+               $uidListString = (isset($uidList[$this->mainTable])) ? implode(',', $uidList[$this->mainTable]) : '';
+               $uidListWithTable = '';
+               foreach ($uidList as $table => $uidListForTable) {
+                       if (!empty($uidListWithTable)) {
+                               $uidListWithTable .= ',';
+                       }
+                       $uidListWithTable .= $table . '_' . implode(',' . $table . '_', $uidListForTable);
+               }
+               $dataStructure = array (
+                       'uniqueTable' => (count($uidList) == 1) ? $this->mainTable : '',
+                       'uidList' => $uidListString,
+                       'uidListWithTable' => $uidListWithTable,
+                       'count' => $numRecords,
+                       'totalCount' => $numRecords
+               );
+
+               // Hook for post-processing the data structure before it is stored into cache
+               if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructureBeforeCache'])) {
+                       foreach ($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessDataStructureBeforeCache'] as $className) {
+                               $postProcessor = GeneralUtility::getUserObj($className);
+                               $dataStructure = $postProcessor->postProcessDataStructureBeforeCache($dataStructure, $this);
+                       }
+               }
+
+               // Store the structure in the cache table,
+               $this->writeStructureToCache($dataStructure);
+
+               // Finally return the assembled structure
+               return $dataStructure;
+       }
+
+       /**
+        * Splits the rows in the recordset returned by the database query into several parts,
+        * one for each (aliased) table in the row.
+        *
+        * Example:
+        * Consider the following query:
+        *
+        * SELECT pages.title, tt_content.header FROM pages,tt_content WHERE tt_content.pid = pages.uid
+        *
+        * A typical result row might look like:
+        *
+        * array('title' => 'foo', 'header' => 'bar');
+        *
+        * This method will turn such an entry into:
+        *
+        * array(
+        *              'pages' => array('title' => 'foo'),
+        *              'tt_content' => array('header' => 'bar'),
+        * );
+        *
+        * This is necessary so that each of these sub-parts can be overlaid (for language or version) properly.
+        *
+        * @param array $result Raw entry from the DB recordset
+        * @param array $columnsMappings Information about each column in the recordset
+        * @return array The result, split into its constituent tables
+        */
+       protected function splitResultIntoSubparts(array $result, array $columnsMappings) {
+               $splitResults = array();
+               foreach ($result as $row) {
+                       $subParts = array();
+                       // Split record into parts related to a single table
+                       foreach ($columnsMappings as $columnName => $columnInfo) {
+                               if (!isset($subParts[$columnInfo['aliasTable']])) {
+                                       $subParts[$columnInfo['aliasTable']] = array();
+                               }
+                               $subParts[$columnInfo['aliasTable']][$columnInfo['field']] = $row[$columnName];
+                       }
+                       $splitResults[] = $subParts;
+               }
+               return $splitResults;
+       }
+
+       /**
+        * Retrieves a data structure stored in cache provided it fits all parameters.
+        *
+        * If no appropriate cache is found, it throws an exception.
+        *
+        * @throws \Tesseract\Tesseract\Exception\Exception
+        * @return array
+        */
+       protected function getCachedStructure() {
+               // Assemble condition for finding correct cache
+               // This means matching the dataquery's primary key, the current language, the filter's hash (without the limit)
+               // and that it has not expired
+               $where = 'query_id = ' . intval($this->providerData['uid']) . ' AND page_id = ' . intval($GLOBALS['TSFE']->id);
+               $where .= ' AND cache_hash = \'' . $this->calculateCacheHash(array()) . '\'';
+               $where .= ' AND expires > \'' . time() . '\'';
+               $row = $this->getDatabaseConnection()->exec_SELECTgetSingleRow(
+                       'structure_cache',
+                       'tx_dataquery_cache',
+                       $where
+               );
+               if (empty($row)) {
+                       throw new \Tesseract\Tesseract\Exception\Exception('No cached structure');
+               } else {
+                       return unserialize($row['structure_cache']);
+               }
+       }
+
+       /**
+        * This method write the standard data structure to cache,
+        * provided some conditions are met
+        *
+        * @param array $structure A standard data structure
+        * @return void
+        */
+       protected function writeStructureToCache($structure) {
+               // Write only if cache is active, i.e.
+               // if cache duration is not empty, and we're not in no_cache mode or in a workspace
+               if (!empty($this->providerData['cache_duration']) && empty($GLOBALS['TSFE']->no_cache) && empty($GLOBALS['TSFE']->sys_page->versioningPreview)) {
+                       $cacheHash = $this->calculateCacheHash(array());
+                       $serializedStructure = serialize($structure);
+                       // Write only if serialized data is not too large
+                       if (empty($this->configuration['cacheLimit']) || strlen($serializedStructure) <= $this->configuration['cacheLimit']) {
+                               $fields = array(
+                                       'query_id' => $this->providerData['uid'],
+                                       'page_id' => $GLOBALS['TSFE']->id,
+                                       'cache_hash' => $cacheHash,
+                                       'structure_cache' => $serializedStructure,
+                                       'expires' => time() + $this->providerData['cache_duration']
+                               );
+                               $this->getDatabaseConnection()->exec_INSERTquery('tx_dataquery_cache', $fields);
+                       }
+                       // If data is too large for caching, make sure no other cache is left over
+                       else {
+                               $where = 'query_id = ' . intval($this->providerData['uid']);
+                               $where .= ' AND page_id = ' . intval($GLOBALS['TSFE']->id);
+                               $where .= ' AND cache_hash = \'' . $cacheHash . '\'';
+                               $this->getDatabaseConnection()->exec_DELETEquery('tx_dataquery_cache', $where);
+                       }
+               }
+       }
+
+       /**
+        * Assembles a hash parameter depending on a variety of parameters, including
+        * the current FE language and the groups of the current FE user, if any.
+        *
+        * @param array $parameters Additional parameters to add to the hash calculation
+        * @return string A md5 hash
+        */
+       protected function calculateCacheHash(array $parameters) {
+               // The base of the hash parameters is the current filter
+               // To this we add the uidList (if it exists)
+               // This makes it possible to vary the cache as a function of the idList provided by the input structure
+               $filterForCache = $this->filter;
+               if (is_array($this->structure) && isset($this->structure['uidListWithTable'])) {
+                       $filterForCache['uidListWithTable'] = $this->structure['uidListWithTable'];
+               }
+               // If some parameters were given, add them to the base cache parameters
+               $cacheParameters = $filterForCache;
+               if (is_array($parameters) && count($parameters) > 0) {
+                       $cacheParameters = array_merge($cacheParameters, $parameters);
+               }
+               // Finally we add other parameters of uniqueness:
+               //      - the current FE language
+               //      - the groups of the currently logged in FE user (if any)
+               //      - the type of structure
+               $cacheParameters['sys_language_uid'] = $GLOBALS['TSFE']->sys_language_content;
+               if (is_array($GLOBALS['TSFE']->fe_user->user) && count($GLOBALS['TSFE']->fe_user->groupData['uid']) > 0) {
+                       $cacheParameters['fe_groups'] = $GLOBALS['TSFE']->fe_user->groupData['uid'];
+               }
+               $cacheParameters['structure_type'] = $this->dataStructureType;
+               // Go through a hook for manipulating cache parameters
+               if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['dataquery']['processCacheHashParameters'])) {
+                       foreach($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['dataquery']['processCacheHashParameters'] as $className) {
+                               /** @var $processor CacheParametersProcessorInterface */
+                               $processor = GeneralUtility::getUserObj($className);
+                               if ($processor instanceof CacheParametersProcessorInterface) {
+                                       $cacheParameters = $processor->processCacheParameters($cacheParameters, $this);
+                               }
+                       }
+               }
+               // Calculate the hash using the method provided by the base controller,
+               // which filters out the "limit" part of the filter
+               return Utilities::calculateFilterCacheHash($cacheParameters);
+       }
+
+    /**
+        * Returns the name of the main table of the query,
+        * which is the table name that appears in the FROM clause, or the alias, if any.
+        *
+        * @return string Main table name
+        */
+       public function getMainTableName() {
+               return $this->mainTable;
+       }
+
+       /**
+        * Performs a special sorting of the recordset.
+        *
+        * @param mixed $a First element to sort
+        * @param mixed $b Second element to sort
+        * @return integer -1 if first argument is smaller than second argument, 1 if first is greater than second and 0 if both are equal
+        *
+        * @see \Tesseract\Dataquery\Component\DataProvider::prepareFullStructure()
+        */
+       static public function sortRecordset($a, $b) {
+               // Get the sorting information from static variables
+               // The level is a pointer to the current field being used for sorting
+               $level = self::$sortingLevel;
+               $field = self::$sortingFields[$level]['field'];
+               $order = (empty(self::$sortingFields[$level]['order'])) ? 'ASC' : strtoupper(self::$sortingFields[$level]['order']);
+               $result = strnatcasecmp($a[$field], $b[$field]);
+               if ($result == 0) {
+                       // If results are equal on the current level, check if there's a next level of sorting
+                       // for differentiating the records
+                       // If yes, call sorting method recursively
+                       if (isset(self::$sortingFields[$level + 1])) {
+                               self::$sortingLevel++;
+                               $result = self::sortRecordset($a, $b);
+                               self::$sortingLevel--;
+                       }
+               }
+               else {
+                       if ($order == 'DESC') {
+                               $result = -$result;
+                       }
+               }
+               return $result;
+       }
+
+       /**
+        * Sorts records using a special fixed order value.
+        *
+        * @param mixed $a First element to sort
+        * @param mixed $b Second element to sort
+        * @return integer -1 if first argument is smaller than second argument, 1 if first is greater than second and 0 if both are equal
+        *
+        * @see \Tesseract\Dataquery\Component\DataProvider::prepareFullStructure()
+        */
+       static public function sortUsingFixedOrder($a, $b) {
+               $result = 1;
+               if ($a['tx_dataquery:fixed_order'] == $b['tx_dataquery:fixed_order']) {
+                       $result = 0;
+               } elseif ($a['tx_dataquery:fixed_order'] < $b['tx_dataquery:fixed_order']) {
+                       $result = -1;
+               }
+               return $result;
+       }
+
+// Data Provider interface methods
+
+       /**
+        * Returns the type of data structure that the Data Provider can prepare.
+        *
+        * @return string Type of the provided data structure
+        */
+       public function getProvidedDataStructure() {
+               return $this->dataStructureType;
+       }
+
+       /**
+        * Indicates whether the Data Provider can create the type of data structure requested or not.
+        *
+        * @param string $type: Type of data structure
+        * @return boolean TRUE if it can handle the requested type, FALSE otherwise
+        */
+       public function providesDataStructure($type) {
+
+               // Check which type was requested and return true if type can be provided
+               // Store requested type internally for later processing
+               if ($type == Tesseract::IDLIST_STRUCTURE_TYPE) {
+                       $this->dataStructureType = Tesseract::IDLIST_STRUCTURE_TYPE;
+                       $result = TRUE;
+               } elseif ($type == Tesseract::RECORDSET_STRUCTURE_TYPE) {
+                       $this->dataStructureType = Tesseract::RECORDSET_STRUCTURE_TYPE;
+                       $result = TRUE;
+               } else {
+                       $result = FALSE;
+               }
+               return $result;
+       }
+
+       /**
+        * Returns the type of data structure that the Data Provider can receive as input.
+        *
+        * @return string Type of used data structures
+        */
+       public function getAcceptedDataStructure() {
+               return Tesseract::IDLIST_STRUCTURE_TYPE;
+       }
+
+       /**
+        * Indicates whether the Data Provider can use as input the type of data structure requested or not.
+        *
+        * @param string $type Type of data structure
+        * @return boolean True if it can use the requested type, false otherwise
+        */
+       public function acceptsDataStructure($type) {
+               return $type == Tesseract::IDLIST_STRUCTURE_TYPE;
+       }
+
+       /**
+        * Assembles the data structure and returns it.
+        *
+        * If the empty structure flag has been set, a dummy empty structure is returned instead.
+        *
+        * @return array Standardised data structure
+        */
+       public function getDataStructure() {
+               // If the empty output structure flag was raised, prepare a proper structure devoid of data
+               if ($this->hasEmptyOutputStructure) {
+                       try {
+                               // Parse the query to get the main table's name
+                               $this->sqlParser->parseQuery($this->providerData['sql_query']);
+                               $this->initEmptyDataStructure(
+                                       $this->sqlParser->getMainTableName(),
+                                       $this->dataStructureType
+                               );
+                       }
+                       catch (\Exception $e) {
+                               $this->controller->addMessage($this->extKey, $e->getMessage() . ' (' . $e->getCode() . ')', 'Query parsing error', FlashMessage::ERROR, $this->providerData);
+                       }
+               } else {
+                       return $this->getQueryData();
+               }
+               return $this->outputStructure;
+       }
+
+       /**
+        * Passes a data structure to the Data Provider.
+        *
+        * @param array $structure Standardised data structure
+        * @return void
+        */
+       public function setDataStructure($structure) {
+               if (is_array($structure)) {
+                       $this->structure = $structure;
+               }
+       }
+
+       /**
+        * Loads the query and gets the list of tables and fields,
+        * complete with localized labels.
+        *
+        * @param string $language 2-letter iso code for language
+        * @return array List of tables and fields
+        */
+       public function getTablesAndFields($language = '') {
+               $this->sqlParser->parseQuery($this->providerData['sql_query']);
+               $tablesAndFields = $this->sqlParser->getLocalizedLabels($language);
+
+               // Hook for post-processing the tables and fields information
+               if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessFieldInformation'])) {
+                       foreach($GLOBALS['TYPO3_CONF_VARS']['EXTCONF'][$this->extKey]['postProcessFieldInformation'] as $className) {
+                               $postProcessor = GeneralUtility::getUserObj($className);
+                               $tablesAndFields = $postProcessor->postProcessFieldInformation($tablesAndFields, $this);
+                       }
+               }
+               return $tablesAndFields;
+    }
+
+// t3lib_svbase methods
+
+       /**
+        * Resets values for a number of properties.
+        *
+        * This is necessary because services are managed as singletons
+        *
+        * NOTE: If you make your own implementation of reset in your DataProvider class, don't forget to call parent::reset()
+        *
+        * @return      void
+        */
+       public function reset() {
+               parent::reset();
+               $this->initialise();
+       }
+
+       /**
+        * Returns the global database object.
+        *
+        * @return \TYPO3\CMS\Core\Database\DatabaseConnection
+        */
+       protected function getDatabaseConnection() {
+               return $GLOBALS['TYPO3_DB'];
+       }
+}
diff --git a/Classes/Exception/InvalidQueryException.php b/Classes/Exception/InvalidQueryException.php
new file mode 100644 (file)
index 0000000..deb19b7
--- /dev/null
@@ -0,0 +1,25 @@
+<?php
+namespace Tesseract\Dataquery\Exception;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Exception to be thrown when a given query contains syntax errors.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_tesseract
+ */
+class InvalidQueryException extends \Exception {
+}
diff --git a/Classes/Hook/DataFilterHook.php b/Classes/Hook/DataFilterHook.php
new file mode 100644 (file)
index 0000000..3986e10
--- /dev/null
@@ -0,0 +1,60 @@
+<?php
+namespace Tesseract\Dataquery\Hook;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use Cobweb\Expressions\ExpressionParser;
+use Tesseract\Datafilter\Component\DataFilter;
+use Tesseract\Datafilter\PostprocessFilterInterface;
+use Tesseract\Datafilter\PostprocessEmptyFilterCheckInterface;
+
+/**
+ * Class for hooking into datafilter to handle the tx_dataquery_sql field.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class DataFilterHook implements PostprocessFilterInterface, PostprocessEmptyFilterCheckInterface {
+
+       /**
+        * Handles the tx_dataquery_sql field and adds it
+        * to the filter itself.
+        *
+        * @param DataFilter $filter A datafilter object
+        * @return void
+        */
+       public function postprocessFilter(DataFilter $filter) {
+               $filterData = $filter->getData();
+               if (!empty($filterData['tx_dataquery_sql'])) {
+                       // Parse any expressions inside the additional sql field
+                       $additionalSQL = ExpressionParser::evaluateString($filterData['tx_dataquery_sql'], FALSE);
+                       $filterArray = $filter->getFilter();
+                       $filterArray['rawSQL'] = $additionalSQL;
+                       $filter->setFilter($filterArray);
+               }
+       }
+
+       /**
+        * Modified the empty filter check to take the tx_dataquery_sql field into account.
+        *
+        * @param boolean $isEmpty Current value of the is filter empty flag
+        * @param DataFilter $filter The calling filter object
+        * @return boolean
+        */
+       public function postprocessEmptyFilterCheck($isEmpty, DataFilter $filter) {
+               $filterStructure = $filter->getFilter();
+               return $isEmpty && empty($filterStructure['rawSQL']);
+       }
+}
diff --git a/Classes/Parser/Fulltext.php b/Classes/Parser/Fulltext.php
deleted file mode 100644 (file)
index 842c214..0000000
+++ /dev/null
@@ -1,195 +0,0 @@
-<?php
-/***************************************************************
-*  Copyright notice
-*
-*  (c) 2012-2015 Fabien Udriot (Cobweb) <fudriot@cobweb.ch>
-*  All rights reserved
-*
-*  This script is part of the TYPO3 project. The TYPO3 project is
-*  free software; you can redistribute it and/or modify
-*  it under the terms of the GNU General Public License as published by
-*  the Free Software Foundation; either version 2 of the License, or
-*  (at your option) any later version.
-*
-*  The GNU General Public License can be found at
-*  http://www.gnu.org/copyleft/gpl.html.
-*
-*  This script is distributed in the hope that it will be useful,
-*  but WITHOUT ANY WARRANTY; without even the implied warranty of
-*  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-*  GNU General Public License for more details.
-*
-*  This copyright notice MUST APPEAR in all copies of the script!
-***************************************************************/
-
-/**
- * Provides routine to manipulate a query adding a fulltext segment in the query.
- *
- * @author Fabien Udriot (Cobweb) <support@cobweb.ch>
- * @author Francois Suter (Cobweb) <support@cobweb.ch>
- * @package TYPO3
- * @subpackage dataquery
- */
-class Tx_Dataquery_Parser_Fulltext {
-
-       /**
-        * @var string
-        */
-       protected $searchTerms = array();
-
-       /**
-        * @var Tx_Dataquery_Utility_DatabaseAnalyser
-        */
-       protected $analyser;
-
-       /**
-        * @var array
-        */
-       protected $indexedFields = array();
-
-       /**
-        * Unserialized extension configuration
-        * @var array
-        */
-       protected $configuration;
-
-       /**
-        * Constructor
-        *
-        * @param string $tableName: the main table name
-        * @return Tx_Dataquery_Parser_Fulltext
-        */
-       public function __construct() {
-
-               /** @var $analyser Tx_Dataquery_Utility_DatabaseAnalyser */
-               $analyser = t3lib_div::makeInstance('Tx_Dataquery_Utility_DatabaseAnalyser');
-               $this->setAnalyser($analyser);
-               $this->configuration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['dataquery']);
-       }
-
-       /**
-        * Retrieves full-text index fields for a given table.
-        *
-        * @param string $tableName
-        */
-       protected function retrieveIndexedFields($tableName) {
-               $this->indexedFields = $this->analyser->getFields($tableName);
-       }
-
-       /**
-        * Sets the analyser.
-        *
-        * Useful for unit tests.
-        *
-        * @param Tx_Dataquery_Utility_DatabaseAnalyser $analyser
-        */
-       public function setAnalyser($analyser) {
-               $this->analyser = $analyser;
-       }
-
-       /**
-        * Sets the extension configuration.
-        *
-        * Useful for unit tests.
-        *
-        * @param array $configuration Extension configuration
-        * @return void
-        */
-       public function setConfiguration($configuration) {
-               $this->configuration = $configuration;
-       }
-
-       /**
-        * Parses the query. If a placeholder "fulltext:foo" is found, then replace with a MATCH / AGAINST expression.
-        *
-        * @param string $table Name of the table to search
-        * @param string $index Name of the fulltext index to use
-        * @param string $search Search string
-        * @param boolean $isNaturalSearch TRUE if fulltext search should be in natural mode
-        * @param boolean $isNegated TRUE if condition should be negated
-        * @return string SQL MATCH() condition
-        * @throws tx_tesseract_exception
-        */
-       public function parse($table, $index, $search, $isNaturalSearch, $isNegated) {
-               $this->retrieveIndexedFields($table);
-               if (isset($this->indexedFields[$index])) {
-                       $indexFields = $this->indexedFields[$index];
-               } else {
-                       throw new tx_tesseract_exception(
-                               sprintf('Table %s has no index "%s"', $table, $index),
-                               1421769189
-                       );
-               }
-               // Search terms from a query string will be urlencode'd
-               $processedSearchTerms = urldecode($search);
-               $booleanMode = '';
-               if (!$isNaturalSearch) {
-                       $processedSearchTerms = $this->processSearchTerm($processedSearchTerms);
-                       $booleanMode = ' IN BOOLEAN MODE';
-               }
-               if (empty($processedSearchTerms)) {
-                       throw new tx_tesseract_exception(
-                               'Empty fulltext search condition',
-                               1423068811
-                       );
-               }
-               $baseCondition = "MATCH(%s) AGAINST('%s'%s)";
-               if ($isNegated) {
-                       $baseCondition = 'NOT ' . $baseCondition;
-               }
-               $condition = sprintf($baseCondition, $indexFields, $processedSearchTerms, $booleanMode);
-               return $condition;
-       }
-
-       /**
-        * Processes the search term.
-        *
-        * @param string $term Search term
-        * @return string
-        */
-       public function processSearchTerm($term) {
-
-               $termsProcessed = array();
-
-               // Handle double quote wrapping
-               if (preg_match_all('/".+"/isU', $term, $matches)) {
-
-                       foreach ($matches as $match) {
-                               $searchedCharacters = array(
-                                       '"',
-                                       ' '
-                               );
-                               $replacedCharacters = array(
-                                       '',
-                                       '###'
-                               );
-                               $search = $match;
-                               $replace = str_replace($searchedCharacters, $replacedCharacters, $match);
-                               $term = str_replace($search, $replace, $term);
-                       }
-               }
-
-               $terms = explode(' ', $term);
-               foreach ($terms as $term) {
-                       if (!empty($term)) {
-                               // Handle exclusion of term
-                               $logic = '+';
-                               if (substr($term, 0, 1) == '-') {
-                                       $term = substr($term, 1);
-                                       $logic = '-';
-                               }
-                               if (strlen($term) >= $this->configuration['fullTextMinimumWordLength']) {
-                                       $termProcessed = str_replace('###', ' ', addslashes($term));
-                                       $termsProcessed[] = sprintf('%s"%s"', $logic, $termProcessed);
-                               }
-                       }
-               }
-               return implode(' ', $termsProcessed);
-       }
-}
-
-
-if (defined('TYPO3_MODE') && $TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Parser/Fulltext.php'])  {
-       include_once($TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Parser/Fulltext.php']);
-}
-?>
\ No newline at end of file
diff --git a/Classes/Parser/FulltextParser.php b/Classes/Parser/FulltextParser.php
new file mode 100644 (file)
index 0000000..7bc7415
--- /dev/null
@@ -0,0 +1,183 @@
+<?php
+namespace Tesseract\Dataquery\Parser;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use Tesseract\Dataquery\Exception\InvalidQueryException;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * Provides routine to manipulate a query adding a fulltext segment in the query.
+ *
+ * @author Fabien Udriot (Cobweb) <support@cobweb.ch>
+ * @author Francois Suter (Cobweb) <support@cobweb.ch>
+ * @package TYPO3
+ * @subpackage dataquery
+ */
+class FulltextParser {
+
+       /**
+        * @var string
+        */
+       protected $searchTerms = array();
+
+       /**
+        * @var \Tesseract\Dataquery\Utility\DatabaseAnalyser
+        */
+       protected $analyser;
+
+       /**
+        * @var array
+        */
+       protected $indexedFields = array();
+
+       /**
+        * Unserialized extension configuration
+        * @var array
+        */
+       protected $configuration;
+
+       /**
+        * Constructor
+        *
+        * @return FulltextParser
+        */
+       public function __construct() {
+
+               /** @var $analyser \Tesseract\Dataquery\Utility\DatabaseAnalyser */
+               $analyser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Utility\\DatabaseAnalyser');
+               $this->setAnalyser($analyser);
+               $this->configuration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['dataquery']);
+       }
+
+       /**
+        * Retrieves full-text index fields for a given table.
+        *
+        * @param string $tableName
+        */
+       protected function retrieveIndexedFields($tableName) {
+               $this->indexedFields = $this->analyser->getFields($tableName);
+       }
+
+       /**
+        * Sets the analyser.
+        *
+        * Useful for unit tests.
+        *
+        * @param \Tesseract\Dataquery\Utility\DatabaseAnalyser $analyser
+        */
+       public function setAnalyser($analyser) {
+               $this->analyser = $analyser;
+       }
+
+       /**
+        * Sets the extension configuration.
+        *
+        * Useful for unit tests.
+        *
+        * @param array $configuration Extension configuration
+        * @return void
+        */
+       public function setConfiguration($configuration) {
+               $this->configuration = $configuration;
+       }
+
+       /**
+        * Parses the query. If a placeholder "fulltext:foo" is found, then replace with a MATCH / AGAINST expression.
+        *
+        * @param string $table Name of the table to search
+        * @param string $index Name of the fulltext index to use
+        * @param string $search Search string
+        * @param boolean $isNaturalSearch TRUE if fulltext search should be in natural mode
+        * @param boolean $isNegated TRUE if condition should be negated
+        * @return string SQL MATCH() condition
+        * @throws InvalidQueryException
+        */
+       public function parse($table, $index, $search, $isNaturalSearch, $isNegated) {
+               $this->retrieveIndexedFields($table);
+               if (isset($this->indexedFields[$index])) {
+                       $indexFields = $this->indexedFields[$index];
+               } else {
+                       throw new InvalidQueryException(
+                               sprintf('Table %s has no index "%s"', $table, $index),
+                               1421769189
+                       );
+               }
+               // Search terms from a query string will be urlencode'd
+               $processedSearchTerms = urldecode($search);
+               $booleanMode = '';
+               if (!$isNaturalSearch) {
+                       $processedSearchTerms = $this->processSearchTerm($processedSearchTerms);
+                       $booleanMode = ' IN BOOLEAN MODE';
+               }
+               if (empty($processedSearchTerms)) {
+                       throw new InvalidQueryException(
+                               'Empty fulltext search condition',
+                               1423068811
+                       );
+               }
+               $baseCondition = "MATCH(%s) AGAINST('%s'%s)";
+               if ($isNegated) {
+                       $baseCondition = 'NOT ' . $baseCondition;
+               }
+               $condition = sprintf($baseCondition, $indexFields, $processedSearchTerms, $booleanMode);
+               return $condition;
+       }
+
+       /**
+        * Processes the search term.
+        *
+        * @param string $term Search term
+        * @return string
+        */
+       public function processSearchTerm($term) {
+
+               $termsProcessed = array();
+
+               // Handle double quote wrapping
+               if (preg_match_all('/".+"/isU', $term, $matches)) {
+
+                       foreach ($matches as $match) {
+                               $searchedCharacters = array(
+                                       '"',
+                                       ' '
+                               );
+                               $replacedCharacters = array(
+                                       '',
+                                       '###'
+                               );
+                               $search = $match;
+                               $replace = str_replace($searchedCharacters, $replacedCharacters, $match);
+                               $term = str_replace($search, $replace, $term);
+                       }
+               }
+
+               $terms = explode(' ', $term);
+               foreach ($terms as $term) {
+                       if (!empty($term)) {
+                               // Handle exclusion of term
+                               $logic = '+';
+                               if (substr($term, 0, 1) == '-') {
+                                       $term = substr($term, 1);
+                                       $logic = '-';
+                               }
+                               if (strlen($term) >= $this->configuration['fullTextMinimumWordLength']) {
+                                       $termProcessed = str_replace('###', ' ', addslashes($term));
+                                       $termsProcessed[] = sprintf('%s"%s"', $logic, $termProcessed);
+                               }
+                       }
+               }
+               return implode(' ', $termsProcessed);
+       }
+}
diff --git a/Classes/Parser/QueryParser.php b/Classes/Parser/QueryParser.php
new file mode 100644 (file)
index 0000000..dc748ad
--- /dev/null
@@ -0,0 +1,1459 @@
+<?php
+namespace Tesseract\Dataquery\Parser;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use Cobweb\Expressions\ExpressionParser;
+use Cobweb\Overlays\OverlayEngine;
+use Tesseract\Dataquery\Exception\InvalidQueryException;
+use Tesseract\Dataquery\Utility\SqlUtility;
+use Tesseract\Tesseract\Utility\Utilities;
+use TYPO3\CMS\Core\Messaging\FlashMessage;
+use TYPO3\CMS\Core\Utility\ArrayUtility;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * This class is used to parse a SELECT SQL query into a structured array.
+ *
+ * It rebuilds the query afterwards, automatically handling a number of TYPO3 constructs,
+ * like enable fields and language overlays.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class QueryParser {
+       static public $extKey = 'dataquery';
+
+       /**
+        * List of eval types which indicate non-text fields
+        * @var array   $notTextTypes
+        */
+       static protected $notTextTypes = array('date', 'datetime', 'time', 'timesec', 'year', 'num', 'md5', 'int', 'double2');
+
+       /**
+        * Reference to the calling object
+        * @var \Tesseract\Dataquery\Component\DataProvider $parentObject
+        */
+       protected $parentObject;
+
+       /**
+        * Unserialized extension configuration
+        * @var array $configuration
+        */
+       protected $configuration;
+
+       /**
+        * Structured type containing the parts of the parsed query
+        * @var \Tesseract\Dataquery\Utility\QueryObject $queryObject
+        */
+       protected $queryObject;
+
+       /**
+        * True names for all the fields. The key is the actual alias used in the query.
+        * @var array $fieldTrueNames
+        */
+       protected $fieldTrueNames = array();
+
+       /**
+        * List of all fields being queried, arranged per table (aliased)
+        * @var array $queryFields
+        */
+       protected $queryFields = array();
+
+       /**
+        * Flag for each table whether to perform overlays or not
+        * @var array
+        */
+       protected $doOverlays = array();
+
+       /**
+        * Flag for each table whether to perform versioning overlays or not
+        * @var array
+        */
+       protected $doVersioning = array();
+
+       /**
+        * True if order by is processed using SQL, false otherwise (see preprocessOrderByFields())
+        * @var boolean
+        */
+       protected $processOrderBy = TRUE;
+
+       /**
+        * Cache array to store table name matches
+        * @var array
+        * @see matchAliasOrTableNeme()
+        */
+       protected $tableMatches = array();
+
+       /**
+        * @var array Database record corresponding to the current Data Query
+        */
+       protected $providerData;
+
+       public function  __construct($parentObject) {
+               $this->parentObject = $parentObject;
+               $this->configuration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['dataquery']);
+       }
+
+       /**
+        * This method is used to parse a SELECT SQL query.
+        * It is a simple parser and no way generic. It expects queries to be written a certain way.
+        *
+        * @param string $query The query to be parsed
+        * @throws InvalidQueryException
+        * @return string A warning message, if any (fatal errors throw exceptions)
+        */
+       public function parseQuery($query) {
+               $warning = '';
+               // Clean up and prepare the query string
+               $query = $this->prepareQueryString($query);
+
+               // Parse the SQL query
+               /** @var $sqlParser \Tesseract\Dataquery\Parser\SqlParser */
+               $sqlParser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Parser\\SqlParser');
+               // NOTE: the following call may throw exceptions,
+               // but we let them bubble up
+               $this->queryObject = $sqlParser->parseSQL($query);
+               // Perform some further analysis on the query components
+               $this->analyzeQuery();
+               // Make sure the list of selected fields contains base fields
+               // like uid and pid (if available)
+               // Don't do this for queries using the DISTINCT keyword, as it may mess it up
+               if (!$this->queryObject->structure['DISTINCT']) {
+                       $this->addBaseFields();
+
+               // If the query uses the DISTINCT keyword, check if a "uid" field has been defined manually
+               // If not, issue warning
+               } else {
+                       if (!$this->checkUidForDistinctUsage()) {
+                               throw new InvalidQueryException('"uid" field missing with DISTINCT usage', 1313354033);
+                       }
+               }
+               // Check if the query selects the same field multiple times
+               // Issue a warning if yes, since the results may be unpredictable
+               $duplicates = $this->checkForDuplicateFields();
+               if (count($duplicates) > 0) {
+                       $warning = 'Duplicate fields in query: ' . implode(' / ', $duplicates);
+               }
+
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->aliases, 'Table aliases');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->fieldAliases, 'Field aliases');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->fieldTrueNames, 'Field true names');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryFields, 'Query fields');
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->structure, 'Structure');
+               return $warning;
+       }
+
+       /**
+        * This method performs a number of operations on a given string,
+        * supposed to be a SQL query
+        * It is meant to be called before the query is actually parsed
+        *
+        * @param       string  $string: a SQL query
+        * @return      string  Cleaned up SQL query
+        */
+       public function prepareQueryString($string) {
+               // Put the query through the field parser to filter out commented lines
+               $queryLines = Utilities::parseConfigurationField($string);
+               // Put the query into a single string
+               $query = implode(' ', $queryLines);
+               // Strip backquotes
+               $query = str_replace('`', '', $query);
+               // Strip trailing semi-colon if any
+               if (strrpos($query, ';') == strlen($query) - 1) {
+                       $query = substr($query, 0, -1);
+               }
+               // Parse query for subexpressions
+               $query = ExpressionParser::evaluateString($query, FALSE);
+               return $query;
+       }
+
+       /**
+        * Analyzes the query in more depth.
+        *
+        * In particular, it loops on all SELECT field and makes sure every field
+        * has a proper alias.
+        *
+        * @return void
+        */
+       protected function analyzeQuery() {
+               // Loop on all query fields to assemble additional information structures
+               foreach ($this->queryObject->structure['SELECT'] as $index => $fieldInfo) {
+                       // Assemble list of fields per table
+                       // The name of the field is used both as key and value, but the value will be replaced by the fields' labels in getLocalizedLabels()
+                       if (!isset($this->queryFields[$fieldInfo['tableAlias']])) {
+                               $this->queryFields[$fieldInfo['tableAlias']] = array('name' => $fieldInfo['table'], 'table' => $fieldInfo['tableAlias'], 'fields' => array());
+                       }
+                       $this->queryFields[$fieldInfo['tableAlias']]['fields'][] = array('name' => $fieldInfo['field'], 'function' => $fieldInfo['function']);
+
+                       // Assemble full names for each field
+                       // The full name is:
+                       //      1) the name of the table or its alias
+                       //      2) a dot
+                       //      3) the name of the field
+                       //
+                       // => If it's the main table and there's an alias for the field
+                       //
+                       //      4a) AS and the field alias
+                       //
+                       //      4a-2)   if the alias contains a dot (.) it means it contains a table name (or alias)
+                       //                      and a field name. So we use this information
+                       //
+                       // This means something like foo.bar AS hey.you will get transformed into foo.bar AS hey$you
+                       //
+                       // In effect this means that you can "reassign" a value from one table (foo) to another (hey)
+                       //
+                       // => If it's not the main table, all fields get an alias using either their own name or the given field alias
+                       //
+                       //      4b) AS and $ and the field or its alias
+                       //
+                       // So something like foo.bar AS boink will get transformed into foo.bar AS foo$boink
+                       //
+                       //      4b-2)   like 4a-2) above, but for subtables
+                       //
+                       // The $ sign is used in class tx_dataquery_wrapper for building the data structure
+                       // Initialize values
+                       $mappedField = '';
+                       $mappedTable = '';
+                       $fullField = $fieldInfo['tableAlias'] . '.' . $fieldInfo['field'];
+                       if ($fieldInfo['function']) {
+                               $fullField = $fieldInfo['field'];
+                       }
+                       $theField = $fieldInfo['field'];
+                       // Case 4a
+                       if ($fieldInfo['tableAlias'] == $this->queryObject->mainTable) {
+                               if (empty($fieldInfo['fieldAlias'])) {
+                                       $theAlias = $theField;
+                               } else {
+                                       $fullField .= ' AS ';
+                                       if (strpos($fieldInfo['fieldAlias'], '.') === FALSE) {
+                                               $theAlias = $fieldInfo['fieldAlias'];
+                                               $mappedTable = $fieldInfo['tableAlias'];
+                                               $mappedField = $fieldInfo['fieldAlias'];
+                                       }
+                                       // Case 4a-2
+                                       else {
+                                               list($mappedTable, $mappedField) = explode('.', $fieldInfo['fieldAlias']);
+                                               $theAlias = str_replace('.', '$', $fieldInfo['fieldAlias']);
+                                       }
+                                       $fullField .= $theAlias;
+                               }
+                       } else {
+                               $fullField .= ' AS ';
+                               if (empty($fieldInfo['fieldAlias'])) {
+                                       $theAlias = $fieldInfo['tableAlias'] . '$' . $fieldInfo['field'];
+                               }
+                               else {
+                                       // Case 4b
+                                       if (strpos($fieldInfo['fieldAlias'], '.') === FALSE) {
+                                               $theAlias = $fieldInfo['tableAlias'] . '$' . $fieldInfo['fieldAlias'];
+                                       }
+                                       // Case 4b-2
+                                       else {
+                                               list($mappedTable, $mappedField) = explode('.', $fieldInfo['fieldAlias']);
+                                               $theAlias = str_replace('.', '$', $fieldInfo['fieldAlias']);
+                                       }
+                               }
+                               $fullField .= $theAlias;
+                       }
+                       if (empty($mappedTable)) {
+                               $mappedTable = $fieldInfo['tableAlias'];
+                               $mappedField = $theField;
+                       }
+                       $this->fieldTrueNames[$theAlias] = array(
+                               'table' => $fieldInfo['table'],
+                               'aliasTable' => $fieldInfo['tableAlias'],
+                               'field' => $theField,
+                               'mapping' => array('table' => $mappedTable, 'field' => $mappedField)
+                       );
+                       $this->queryObject->structure['SELECT'][$index] = $fullField;
+        }
+       }
+
+       /**
+        * Checks every table that doesn't have a uid or pid field and tries to add it
+        * to the list of fields to select.
+        *
+        * @return void
+        */
+       protected function addBaseFields() {
+               // Loop on the tables that don't have a uid field
+        foreach ($this->queryObject->hasBaseFields as $alias => $listOfFields) {
+                       // Get all fields for the given table
+                       $fieldsInfo = OverlayEngine::getAllFieldsForTable($this->queryObject->aliases[$alias]);
+                       foreach ($listOfFields as $baseField => $flag) {
+                               if (!$flag) {
+                                       // Add the uid field only if it exists
+                                       if (isset($fieldsInfo[$baseField])) {
+                                               $this->addExtraField($baseField, $alias, $this->getTrueTableName($alias));
+                                       }
+                               }
+                       }
+        }
+       }
+
+       /**
+        * Gets the localized labels for all tables and fields in the query in the given language.
+        *
+        * @param string $language Two-letter ISO code of a language
+        * @return array List of all localized labels
+        */
+       public function getLocalizedLabels($language = '') {
+               $lang = Utilities::getLanguageObject($language);
+
+               // Now that we have a properly initialised language object,
+               // loop on all labels and get any existing localised string
+               $localizedStructure = array();
+               foreach ($this->queryFields as $alias => $tableData) {
+                       $table = $tableData['name'];
+                               // Initialize structure for table, if not already done
+                       if (!isset($localizedStructure[$alias])) {
+                               $localizedStructure[$alias] = array('table' => $table, 'fields' => array());
+                       }
+                       // Get the labels for the tables
+                       if (isset($GLOBALS['TCA'][$table]['ctrl']['title'])) {
+                               $tableName = $lang->sL($GLOBALS['TCA'][$table]['ctrl']['title']);
+                               $localizedStructure[$alias]['name'] = $tableName;
+                       }
+                       else {
+                               $localizedStructure[$alias]['name'] = $table;
+                       }
+                       // Get the labels for the fields
+                       foreach ($tableData['fields'] as $fieldData) {
+                               // Set default values
+                               $tableAlias = $alias;
+                               $field = $fieldData['name'];
+                               // Get the localized label, if it exists, otherwise use field name
+                               // Skip if it's a function (it will have no TCA definition anyway)
+                               $fieldName = $field;
+                               if (!$fieldData['function'] && isset($GLOBALS['TCA'][$table]['columns'][$fieldData['name']]['label'])) {
+                                       $fieldName = $lang->sL($GLOBALS['TCA'][$table]['columns'][$fieldData['name']]['label']);
+                               }
+                               // Check if the field has an alias, if yes use it
+                               // Otherwise use the field name itself as an alias
+                               $fieldAlias = $field;
+                               if (isset($this->queryObject->fieldAliases[$alias][$field])) {
+                                       $fieldAlias = $this->queryObject->fieldAliases[$alias][$field];
+                                       // If the alias contains a dot (.), it means it contains the alias of a table name
+                                       // Explode the name on the dot and use the parts as a new table alias and field name
+                                       if (strpos($fieldAlias, '.') !== false) {
+                                               list($tableAlias, $fieldAlias) = GeneralUtility::trimExplode('.', $fieldAlias);
+                                               // Initialize structure for table, if not already done
+                                               if (!isset($localizedStructure[$tableAlias])) $localizedStructure[$tableAlias] = array('table' => $tableAlias, 'fields' => array());
+                                       }
+                               }
+                               // Store the localized label
+                               $localizedStructure[$tableAlias]['fields'][$fieldAlias] = $fieldName;
+            }
+        }
+//             \TYPO3\CMS\Core\Utility\DebugUtility::debug($localizedStructure, 'Localized structure');
+               return $localizedStructure;
+    }
+
+       /**
+        * Sets the data coming from the Data Provider class.
+        *
+        * @param array $providerData Database record corresponding to the current Data Query record
+        * @return void
+        */
+       public function setProviderData($providerData) {
+               $this->providerData = $providerData;
+               // Perform some processing on some fields
+               // Mostly this is about turning into arrays the fields containing comma-separated values
+               $this->providerData['ignore_time_for_tables_exploded'] = GeneralUtility::trimExplode(',', $this->providerData['ignore_time_for_tables']);
+               $this->providerData['ignore_disabled_for_tables_exploded'] = GeneralUtility::trimExplode(',', $this->providerData['ignore_disabled_for_tables']);
+               $this->providerData['ignore_fegroup_for_tables_exploded'] = GeneralUtility::trimExplode(',', $this->providerData['ignore_fegroup_for_tables']);
+               $this->providerData['get_versions_directly_exploded'] = GeneralUtility::trimExplode(',', $this->providerData['get_versions_directly']);
+       }
+
+       /**
+        * Returns an associative array containing information for method enableFields.
+        *
+        * enableFields() will skip each enable field condition from the returned array.
+        *
+        * @param string $tableName The name of the table
+        * @return array The array containing the keys to be ignored
+        */
+       protected function getIgnoreArray($tableName) {
+               $ignoreArray = array();
+               // Handle case when some fields should be partially excluded from enableFields()
+               if ($this->providerData['ignore_enable_fields'] == '2') {
+
+                       // starttime / endtime field
+                       if (in_array($tableName, $this->providerData['ignore_time_for_tables_exploded']) ||
+                                       $this->providerData['ignore_time_for_tables'] == '*') {
+                               $ignoreArray['starttime'] = TRUE;
+                               $ignoreArray['endtime'] = TRUE;
+                       }
+
+                       // disabled field
+                       if (in_array($tableName, $this->providerData['ignore_disabled_for_tables_exploded']) ||
+                                       $this->providerData['ignore_disabled_for_tables'] == '*') {
+                               $ignoreArray['disabled'] = TRUE;
+                       }
+
+                       // fe_group field
+                       if (in_array($tableName, $this->providerData['ignore_fegroup_for_tables_exploded']) ||
+                                       $this->providerData['ignore_fegroup_for_tables'] == '*') {
+                               $ignoreArray['fe_group'] = TRUE;
+                       }
+               }
+               return $ignoreArray;
+       }
+
+       /**
+        * Adds where clause elements related to typical TYPO3 control parameters.
+        *
+        * The parameters are:
+        *
+        *      - the enable fields
+        *      - the language handling
+        *      - the versioning system
+        *
+        * @return      void
+        */
+       public function addTypo3Mechanisms() {
+               // Add enable fields conditions
+               $this->addEnableFieldsCondition();
+               // Assemble a list of all currently selected fields for each table,
+               // skipping function calls (which can't be overlayed anyway)
+               // This is used by the next two methods, which may add some necessary fields,
+               // if not present already
+               $fieldsPerTable = array();
+               foreach ($this->queryFields as $alias => $tableData) {
+                       $fieldsPerTable[$alias] = array();
+                       foreach ($tableData['fields'] as $fieldData) {
+                               if (!$fieldData['function']) {
+                                       $fieldsPerTable[$alias][] = $fieldData['name'];
+                               }
+                       }
+               }
+               // Add language-related conditions
+               $this->addLanguageCondition($fieldsPerTable);
+               // Add versioning-related conditions
+               $this->addVersioningCondition($fieldsPerTable);
+       }
+
+       /**
+        * This method adds all SQL conditions needed to enforce the enable fields for
+        * all tables involved
+        *
+        * @return void
+        */
+       protected function addEnableFieldsCondition() {
+               // First check if enable fields must really be added or should be ignored
+               if ($this->providerData['ignore_enable_fields'] == '0' || $this->providerData['ignore_enable_fields'] == '2') {
+
+                       // Start with main table
+                       // Define parameters for enable fields condition
+                       $trueTableName = $this->queryObject->aliases[$this->queryObject->mainTable];
+                       $showHidden = ($trueTableName == 'pages') ? $GLOBALS['TSFE']->showHiddenPage : $GLOBALS['TSFE']->showHiddenRecords;
+                       $ignoreArray = $this->getIgnoreArray($this->queryObject->mainTable);
+
+                       $enableClause = OverlayEngine::getEnableFieldsCondition(
+                               $trueTableName,
+                               $showHidden,
+                               $ignoreArray
+                       );
+                       // Replace the true table name by its alias if necessary
+                       // NOTE: there's a risk that a field containing the table name might be modified abusively
+                       // There's no real way around it except changing tx_overlays::getEnableFieldsCondition()
+                       // to re-implement a better t3lib_page::enableFields()
+                       // Adding the "." in the replacement reduces the risks
+                       if ($this->queryObject->mainTable != $trueTableName) {
+                               $enableClause = str_replace($trueTableName . '.', $this->queryObject->mainTable . '.', $enableClause);
+                       }
+                       $this->addWhereClause($enableClause);
+
+                       // Add enable fields to JOINed tables
+                       if (isset($this->queryObject->structure['JOIN']) && is_array($this->queryObject->structure['JOIN'])) {
+                               foreach ($this->queryObject->structure['JOIN'] as $joinData) {
+
+                                       // Define parameters for enable fields condition
+                                       $table = $joinData['table'];
+                                       $showHidden = ($table == 'pages') ? $GLOBALS['TSFE']->showHiddenPage : $GLOBALS['TSFE']->showHiddenRecords;
+                                       $ignoreArray = $this->getIgnoreArray($joinData['alias']);
+
+                                       $enableClause = OverlayEngine::getEnableFieldsCondition(
+                                               $table,
+                                               $showHidden,
+                                               $ignoreArray
+                                       );
+                                       if (!empty($enableClause)) {
+                                               if ($table != $joinData['alias']) {
+                                                       $enableClause = str_replace($table . '.', $joinData['alias'] . '.', $enableClause);
+                                               }
+                                               $this->addOnClause($enableClause, $joinData['alias']);
+                                       }
+                               }
+                       }
+               }
+       }
+
+       /**
+        * Adds SQL conditions related to language handling.
+        *
+        * Also adds the necessary fields to the list of SELECTed fields
+        *
+        * @param array $fieldsPerTable List of all fields already SELECTed, per table
+        *
+        * @return void
+        */
+       protected function addLanguageCondition($fieldsPerTable) {
+               $skippedTablesForLanguageOverlays = GeneralUtility::trimExplode(',', $this->providerData['skip_overlays_for_tables'], TRUE);
+
+               // Add the language condition, if necessary
+               if (empty($this->providerData['ignore_language_handling']) && !$this->queryObject->structure['DISTINCT']) {
+
+                       // Add the DB fields and the SQL conditions necessary for having everything ready to handle overlays
+                       // as per the standard TYPO3 mechanism
+                       // Loop on all tables involved
+                       foreach ($this->queryFields as $alias => $tableData) {
+                               $table = $tableData['name'];
+
+                               // First entirely skip tables which are defined in the skip list
+                               if (in_array($table, $skippedTablesForLanguageOverlays)) {
+                                       $this->doOverlays[$table] = FALSE;
+
+                               // Check which handling applies, based on existing TCA structure
+                               // The table must at least have a language field or point to a foreign table for translation
+                               } elseif (isset($GLOBALS['TCA'][$table]['ctrl']['languageField']) || isset($GLOBALS['TCA'][$table]['ctrl']['transForeignTable'])) {
+
+                                       // The table uses translations in the same table (transOrigPointerField) or in a foreign table (transForeignTable)
+                                       // Prepare for overlays
+                                       if (isset($GLOBALS['TCA'][$table]['ctrl']['transOrigPointerField']) || isset($GLOBALS['TCA'][$table]['ctrl']['transForeignTable'])) {
+                                               // For each table, make sure that the fields necessary for handling the language overlay are included in the list of selected fields
+                                               try {
+                                                       $fieldsForOverlayArray = OverlayEngine::selectOverlayFieldsArray($table, implode(',', $fieldsPerTable[$alias]));
+                                                       // Extract which fields were added and add them to the list of fields to select
+                                                       $addedFields = array_diff($fieldsForOverlayArray, $fieldsPerTable[$alias]);
+                                                       if (count($addedFields) > 0) {
+                                                               foreach ($addedFields as $aField) {
+                                                                       $this->addExtraField($aField, $alias, $table);
+                                                               }
+                                                       }
+                                                       $this->doOverlays[$table] = TRUE;
+                                                       // Add the language condition for the given table (only for tables containing their own translations)
+                                                       if (isset($GLOBALS['TCA'][$table]['ctrl']['transOrigPointerField'])) {
+                                                               $languageCondition = OverlayEngine::getLanguageCondition($table, $alias);
+                                                               if ($alias == $this->queryObject->mainTable) {
+                                                                       $this->addWhereClause($languageCondition);
+                                                               } else {
+                                                                       $this->addOnClause($languageCondition, $alias);
+                                                               }
+                                                       }
+                                               }
+                                               catch (\Exception $e) {
+                                                       $this->doOverlays[$table] = FALSE;
+                                               }
+                                       }
+
+                               // The table simply contains a language flag.
+                               // This is just about adding the proper condition on the language field and nothing more
+                               // No overlays will be handled at a later time
+                               } else {
+                                       if (isset($GLOBALS['TCA'][$table]['ctrl']['languageField'])) {
+                                               // Take language that corresponds to current language or [All]
+                                               $languageCondition = $alias . '.' . $GLOBALS['TCA'][$table]['ctrl']['languageField'] . ' IN (' . $GLOBALS['TSFE']->sys_language_content . ', -1)';
+                                               if ($alias == $this->queryObject->mainTable) {
+                                                       $this->addWhereClause($languageCondition);
+                                               } else {
+                                                       $this->addOnClause($languageCondition, $alias);
+                                               }
+                                       }
+                               }
+                       }
+               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->doOverlays);
+       }
+
+       /**
+        * Adds SQL conditions related version handling.
+        *
+        * Also add the necessary fields to the list of SELECTed fields.
+        * Contrary to the other conditions, versioning conditions are always added,
+        * if only to make sure that only LIVE records are selected.
+        *
+        * @param array $fieldsPerTable List of all fields already SELECTed, per table
+        *
+        * @return void
+        */
+       protected function addVersioningCondition($fieldsPerTable) {
+               foreach ($this->queryFields as $alias => $tableData) {
+                       $table = $tableData['name'];
+                       $this->doVersioning[$table] = FALSE;
+
+                       // Continue if table indeed supports versioning
+                       if (!empty($GLOBALS['TCA'][$table]['ctrl']['versioningWS'])) {
+                               // By default make sure to take only LIVE version
+                               $workspaceCondition = $alias . ".t3ver_oid = '0'";
+                               // If in preview mode, assemble condition according to current workspace
+                               if ($GLOBALS['TSFE']->sys_page->versioningPreview) {
+                                       // For each table, make sure that the fields necessary for handling the language overlay are included in the list of selected fields
+                                       try {
+                                               $fieldsForOverlayArray = OverlayEngine::selectVersioningFieldsArray(
+                                                       $table,
+                                                       implode(',', $fieldsPerTable[$alias])
+                                               );
+                                               // Extract which fields were added and add them to the list of fields to select
+                                               $addedFields = array_diff($fieldsForOverlayArray, $fieldsPerTable[$alias]);
+                                               if (count($addedFields) > 0) {
+                                                       foreach ($addedFields as $aField) {
+                                                               $this->addExtraField($aField, $alias, $table);
+                                                       }
+                                               }
+                                               $this->doVersioning[$table] = TRUE;
+                                               $getVersionsDirectly = FALSE;
+                                               if ($this->providerData['get_versions_directly'] == '*' || in_array($alias, $this->providerData['get_versions_directly_exploded'])) {
+                                                       $getVersionsDirectly = TRUE;
+                                               }
+                                               $workspaceCondition = OverlayEngine::getVersioningCondition(
+                                                       $table,
+                                                       $alias,
+                                                       $getVersionsDirectly
+                                               );
+                                       }
+                                       catch (\Exception $e) {
+                                               $this->doVersioning[$table] = FALSE;
+                                               $this->parentObject->getController()->addMessage(
+                                                       self::$extKey,
+                                                       'A problem happened with versioning: ' . $e->getMessage() . ' (' . $e->getCode() . ')',
+                                                       'Falling back to LIVE records for table ' . $table,
+                                                       FlashMessage::WARNING
+                                               );
+                                       }
+                               }
+                               if ($alias == $this->queryObject->mainTable) {
+                                       $this->addWhereClause($workspaceCondition);
+                               } else {
+                                       $this->addOnClause($workspaceCondition, $alias);
+                               }
+                       }
+               }
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->doVersioning);
+       }
+
+       /**
+        * Takes a Data Filter structure and processes its instructions.
+        *
+        * @param array $filter Data Filter structure
+        * @return void
+        */
+       public function addFilter($filter) {
+               // First handle the "filter" part, which will be turned into part of a SQL WHERE clause
+               $completeFilters = array();
+               $logicalOperator = (empty($filter['logicalOperator'])) ? 'AND' : $filter['logicalOperator'];
+               if (isset($filter['filters']) && is_array($filter['filters'])) {
+                       foreach ($filter['filters'] as $index => $filterData) {
+                               $table = '';
+                               // Check if the condition must be explicitly ignored
+                               // (i.e. it is transmitted by the filter only for information)
+                               // If not, resolve the table name, if possible
+                               if ($filterData['void']) {
+                                       $ignoreCondition = TRUE;
+                               } else {
+                                       $ignoreCondition = FALSE;
+                                       $table = (empty($filterData['table'])) ? $this->queryObject->mainTable : $filterData['table'];
+                                       // Check if the table is available in the query
+                                       try {
+                                               $table = $this->matchAliasOrTableName($table, 'Filter - ' . ((empty($filterData['string'])) ? $index : $filterData['string']));
+                                       }
+                                       catch (InvalidQueryException $e) {
+                                               $ignoreCondition = TRUE;
+                                               $this->parentObject->getController()->addMessage(
+                                                       self::$extKey,
+                                                       'The condition did not apply to a table used in the query.',
+                                                       'Condition ignored',
+                                                       FlashMessage::NOTICE,
+                                                       $filterData
+                                               );
+                                       }
+                               }
+                               // If the table is not in the query, ignore the condition
+                               if (!$ignoreCondition) {
+                                       $field = $filterData['field'];
+                                       $fullField = $table . '.' . $field;
+                                       // If the field is an alias, override full field definition
+                                       // to whatever the alias is mapped to
+                                       if (isset($this->queryObject->fieldAliasMappings[$field])) {
+                                               $fullField = $this->queryObject->fieldAliasMappings[$field];
+                                       }
+                                       $condition = '';
+                                       // Define table on which to apply the condition
+                                       // Conditions will normally be applied in the WHERE clause
+                                       // if the table is the main one, otherwise it is applied
+                                       // in the ON clause of the relevant JOIN statement
+                                       // However the application of the condition may be forced to be in the WHERE clause,
+                                       // no matter which table it targets
+                                       $tableForApplication = $table;
+                                       if ($filterData['main']) {
+                                               $tableForApplication = $this->queryObject->mainTable;
+                                       }
+                                       foreach ($filterData['conditions'] as $conditionData) {
+                                               // If the value is special value "\all", all values must be taken,
+                                               // so the condition is simply ignored
+                                               if ($conditionData['value'] != '\all') {
+                                                       try {
+                                                               $parsedCondition = SqlUtility::conditionToSql($fullField, $table, $conditionData);
+                                                               if (!empty($condition)) {
+                                                                       $condition .= ' AND ';
+                                                               }
+                                                               $condition .= '(' . $parsedCondition . ')';
+                                                               // If the operator was a full text search, store resulting condition which will be used
+                                                               // later to replace the placeholder in the SELECT part of the statement
+                                                               if ($conditionData['operator'] == 'fulltext' || $conditionData['operator'] == 'fulltext_natural') {
+                                                                       $fullFieldParts = explode('.', $fullField);
+                                                                       $placeholderKey = $table . '.fulltext.' . $fullFieldParts[2];
+                                                                       if (isset($this->queryObject->fulltextSearchPlaceholders[$placeholderKey])) {
+                                                                               $this->queryObject->fulltextSearchPlaceholders[$placeholderKey] = $condition;
+                                                                       }
+                                                               }
+                                                       }
+                                                       catch (InvalidQueryException $e) {
+                                                               $this->parentObject->getController()->addMessage(
+                                                                       self::$extKey,
+                                                                       $e->getMessage(),
+                                                                       'Condition ignored',
+                                                                       FlashMessage::WARNING,
+                                                                       $filterData
+                                                               );
+                                                       }
+                                               }
+                                       }
+                                       // Add the condition only if it wasn't empty
+                                       if (!empty($condition)) {
+                                               if (empty($completeFilters[$tableForApplication])) {
+                                                       $completeFilters[$tableForApplication] = '';
+                                               } else {
+                                                       $completeFilters[$tableForApplication] .= ' ' . $logicalOperator . ' ';
+                                               }
+                                               $completeFilters[$tableForApplication] .= '(' . $condition . ')';
+                                       }
+                               }
+                       }
+                       foreach ($completeFilters as $table => $whereClause) {
+                               if ($table == $this->queryObject->mainTable) {
+                                       $this->addWhereClause($whereClause);
+                               } elseif (in_array($table, $this->queryObject->subtables)) {
+                                       $this->addOnClause($whereClause, $table);
+                               }
+                       }
+                       // Free some memory
+                       unset($completeFilters);
+               }
+               // Add the eventual raw SQL in the filter
+               // Raw SQL is always added to the main where clause
+               if (!empty($filter['rawSQL'])) {
+                       $this->addWhereClause($filter['rawSQL']);
+               }
+               // Handle the order by clauses
+               if (count($filter['orderby']) > 0) {
+                       foreach ($filter['orderby'] as $orderData) {
+                               // Special case if ordering is random
+                               if ($orderData['order'] == 'RAND') {
+                                       $this->queryObject->structure['ORDER BY'][] = 'RAND()';
+
+                               // Handle normal configuration
+                               } else {
+                                       $table = ((empty($orderData['table'])) ? $this->queryObject->mainTable : $orderData['table']);
+                                       // Try applying the order clause to an existing table
+                                       try {
+                                               $table = $this->matchAliasOrTableName($table, 'Order clause - ' . $table . ' - ' . $orderData['field'] . ' - ' . $orderData['order']);
+                                               $completeField = $table . '.' . $orderData['field'];
+                                               $orderbyClause = $completeField . ' ' . $orderData['order'];
+                                               $this->queryObject->structure['ORDER BY'][] = $orderbyClause;
+                                               $this->queryObject->orderFields[] = array(
+                                                       'field' => $completeField,
+                                                       'order' => $orderData['order'],
+                                                       'engine' => isset($orderData['engine']) ? $orderData['engine'] : ''
+                                               );
+                                       }
+                                       // Table was not matched
+                                       catch (InvalidQueryException $e) {
+                                               $this->parentObject->getController()->addMessage(
+                                                       self::$extKey,
+                                                       'The ordering clause did not apply to a table used in the query.',
+                                                       'Ordering ignored',
+                                                       FlashMessage::NOTICE,
+                                                       $orderData
+                                               );
+                                       }
+                               }
+                       }
+               }
+       }
+
+       /**
+        * Takes a list of uid's prepended by their table name,
+        * as returned in the "uidListWithTable" property of a idList-type SDS,
+        * and makes it into appropriate SQL IN conditions for every table that matches those used in the query.
+        *
+        * @param string $idList Comma-separated list of uid's prepended by their table name
+        * @return void
+        */
+       public function addIdList($idList) {
+               if (!empty($idList)) {
+                       $idArray = GeneralUtility::trimExplode(',', $idList);
+                       $idlistsPerTable = array();
+                       // First assemble a list of all uid's for each table
+                       foreach ($idArray as $item) {
+                               // Code inspired from \TYPO3\CMS\Core\Database\RelationHandler
+                               // String is reversed before exploding, to get uid first
+                               list($uid, $table) = explode('_', strrev($item), 2);
+                               // Exploded parts are reversed back
+                               $uid = strrev($uid);
+                               // If table is not defined, assume it's the main table
+                               if (empty($table)) {
+                                       $table = $this->queryObject->mainTable;
+                                       if (!isset($idlistsPerTable[$table])) {
+                                               $idlistsPerTable[$table] = array();
+                                       }
+                                       $idlistsPerTable[$table][] = $uid;
+                               } else {
+                                       $table = strrev($table);
+                                       // Make sure the table name matches one used in the query
+                                       try {
+                                               $table = $this->matchAliasOrTableName($table, 'Id list - ' . $item);
+                                               if (!isset($idlistsPerTable[$table])) {
+                                                       $idlistsPerTable[$table] = array();
+                                               }
+                                               $idlistsPerTable[$table][] = $uid;
+                                       }
+                                       catch (InvalidQueryException $e) {
+                                               $this->parentObject->getController()->addMessage(
+                                                       self::$extKey,
+                                                       'An item from the id list did not apply, because table ' . $table . ' is not used in the query.',
+                                                       'Id ignored',
+                                                       FlashMessage::NOTICE,
+                                                       $item
+                                               );
+                                       }
+                               }
+                       }
+                       // Loop on all tables and add test on list of uid's, if table is indeed in query
+                       foreach ($idlistsPerTable as $table => $uidArray) {
+                               $condition = $table . '.uid IN (' . implode(',', $uidArray) . ')';
+                               if ($table == $this->queryObject->mainTable) {
+                                       $this->addWhereClause($condition);
+                               } elseif (in_array($table, $this->queryObject->subtables)) {
+                                       if (!empty($this->queryObject->structure['JOIN'][$table]['on'])) {
+                                               $this->queryObject->structure['JOIN'][$table]['on'] .= ' AND ';
+                                       }
+                                       $this->queryObject->structure['JOIN'][$table]['on'] .= $condition;
+                               }
+                       }
+                       // Free some memory
+                       unset($idlistsPerTable);
+               }
+       }
+
+       /**
+        * Builds up the query with all the data stored in the structure.
+        *
+        * @return string The assembled SQL query
+        */
+       public function buildQuery() {
+               // First check what to do with ORDER BY fields
+               $this->preprocessOrderByFields();
+               // Start assembling the query
+               $query  = 'SELECT ';
+               if ($this->queryObject->structure['DISTINCT']) {
+                       $query .= 'DISTINCT ';
+               }
+               $query .= implode(', ', $this->queryObject->structure['SELECT']) . ' ';
+
+               // Process fulltext replacements, if any
+               foreach ($this->queryObject->fulltextSearchPlaceholders as $placeholder => $replacement) {
+                       $query = str_replace($placeholder, $replacement, $query);
+               }
+
+               $query .= 'FROM ' . $this->queryObject->structure['FROM']['table'];
+               if (!empty($this->queryObject->structure['FROM']['alias'])) {
+                       $query .= ' AS ' . $this->queryObject->structure['FROM']['alias'];
+               }
+               $query .= ' ';
+               if (isset($this->queryObject->structure['JOIN'])) {
+                       foreach ($this->queryObject->structure['JOIN'] as $theJoin) {
+                               $query .= strtoupper($theJoin['type']) . ' JOIN ' . $theJoin['table'];
+                               if (!empty($theJoin['alias'])) {
+                                       $query .= ' AS ' . $theJoin['alias'];
+                               }
+                               if (!empty($theJoin['on'])) {
+                                       $query .= ' ON ' . $theJoin['on'];
+                               }
+                               $query .= ' ';
+                       }
+               }
+               if (count($this->queryObject->structure['WHERE']) > 0) {
+                       $whereClause = '';
+                       foreach ($this->queryObject->structure['WHERE'] as $clause) {
+                               if (!empty($whereClause)) {
+                                       $whereClause .= ' AND ';
+                               }
+                               $whereClause .= '(' . $clause . ')';
+                       }
+                       $query .= 'WHERE ' . $whereClause . ' ';
+               }
+               if (count($this->queryObject->structure['GROUP BY']) > 0) {
+                       $query .= 'GROUP BY ' . implode(', ', $this->queryObject->structure['GROUP BY']) . ' ';
+               }
+                       // Add order by clause if defined and if applicable (see preprocessOrderByFields())
+               if ($this->processOrderBy && count($this->queryObject->structure['ORDER BY']) > 0) {
+                       $query .= 'ORDER BY ' . implode(', ', $this->queryObject->structure['ORDER BY']) . ' ';
+               }
+               if (isset($this->queryObject->structure['LIMIT'])) {
+                       $query .= 'LIMIT ' . $this->queryObject->structure['LIMIT'];
+                       if (isset($this->queryObject->structure['OFFSET'])) {
+                               $query .= ' OFFSET ' . $this->queryObject->structure['OFFSET'];
+                       }
+               }
+
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($query);
+               return $query;
+       }
+
+       /**
+        * Performs some operations on the fields used for ordering the query, if any.
+        *
+        * If the language is not the default one, order may not be desirable in SQL
+        * As translations are handled using overlays in TYPO3, it is not possible
+        * to sort the records alphabetically in the SQL statement, because the SQL
+        * statement gets only the records in original language.
+        *
+        * @return boolean True if order by must be processed by the SQL query, false otherwise
+        */
+       protected function preprocessOrderByFields() {
+/*
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->orderFields, 'Order fields');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->fieldAliases, 'Field aliases');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->fieldTrueNames, 'Field true names');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryFields, 'Query fields');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->structure['SELECT'], 'Select structure');
+*/
+               if (count($this->queryObject->orderFields) > 0) {
+                       // If in the FE context and not the default language, start checking for possible use of SQL or not
+                       if (isset($GLOBALS['TSFE']) && $GLOBALS['TSFE']->sys_language_content > 0) {
+                               // Initialise sorting mode flag
+                               $cannotUseSQLForSorting = FALSE;
+                               // Initialise various arrays
+                               $newQueryFields = array();
+                               $newSelectFields = array();
+                               $newTrueNames = array();
+                               $countNewFields = 0;
+                               foreach ($this->queryObject->orderFields as $index => $orderInfo) {
+                                       // Define the table and field names
+                                       $fieldParts = explode('.', $orderInfo['field']);
+                                       if (count($fieldParts) == 1) {
+                                               $alias = $this->queryObject->mainTable;
+                                               $field = $fieldParts[0];
+                                       } else {
+                                               $alias = $fieldParts[0];
+                                               $field = $fieldParts[1];
+                                       }
+                                       // Skip all the rest of the logic for some special values
+                                       if ($field !== 'RAND()' && $field !== 'NULL') {
+                                               // If the field has an alias, change the order fields list to use it
+                                               if (isset($this->queryObject->fieldAliases[$alias][$field])) {
+                                                       $this->queryObject->orderFields[$index]['alias'] = $this->queryObject->orderFields[$index]['field'];
+                                                       $this->queryObject->orderFields[$index]['field'] = $this->queryObject->fieldAliases[$alias][$field];
+                                               }
+                                               // Get the field's true table and field name, if defined, in case an alias is used in the ORDER BY statement
+                                               if (isset($this->fieldTrueNames[$field])) {
+                                                       $alias = $this->fieldTrueNames[$field]['aliasTable'];
+                                                       $field = $this->fieldTrueNames[$field]['field'];
+                                               }
+                                               // Get the true table name and initialize new field array, if necessary
+                                               $table = $this->getTrueTableName($alias);
+                                               if (!isset($newQueryFields[$alias])) {
+                                                       $newQueryFields[$alias] = array(
+                                                               'name' => $alias,
+                                                               'table' => $table,
+                                                               'fields' => array()
+                                                       );
+                                               }
+                                               // Check if there's some explicit engine information
+                                               if (!empty($orderInfo['engine'])) {
+                                                       // If at least one field must be handled by the provider, set the flag to true
+                                                       if ($orderInfo['engine'] == 'provider') {
+                                                               $cannotUseSQLForSorting |= TRUE;
+                                                       } else {
+                                                       // Nothing to do here. If the field was forced to be applied to the source,
+                                                       // it does not need to be checked further
+                                                       }
+                                               } else {
+
+                                                       // Check the type of the field in the TCA
+                                                       // If the field is of some text type and that the table uses overlays,
+                                                       // ordering cannot happen in SQL.
+                                                       if (isset($GLOBALS['TCA'][$table])) {
+                                                               // Check if table uses overlays
+                                                               $usesOverlay = isset($GLOBALS['TCA'][$table]['ctrl']['languageField']) || isset($GLOBALS['TCA'][$table]['ctrl']['transForeignTable']);
+                                                               // Check the field type (load full TCA first)
+                                                               $isTextField = $this->isATextField($table, $field);
+                                                               $cannotUseSQLForSorting |= ($usesOverlay && $isTextField);
+                                                       }
+                                               }
+                                               // Check if the field is already part of the SELECTed fields (under its true name or an alias)
+                                               // If not, get ready to add it by defining all necessary info in temporary arrays
+                                               // (it will be added only if necessary, i.e. if at least one field needs to be ordered later)
+                                               if (!$this->isAQueryField($alias, $field) && !isset($this->queryObject->fieldAliases[$alias][$field])) {
+                                                       $fieldAlias = $alias . '$' . $field;
+                                                       $newQueryFields[$alias]['fields'][] = array('name' => $field, 'function' => FALSE);
+                                                       $newSelectFields[] = $alias . '.' . $field . ' AS ' . $fieldAlias;
+                                                       $newTrueNames[$fieldAlias] = array(
+                                                               'table' => $table,
+                                                               'aliasTable' => $alias,
+                                                               'field' => $field,
+                                                               'mapping' => array('table' => $alias, 'field' => $field)
+                                                       );
+                                                       $countNewFields++;
+                                               }
+                                       }
+                               }
+                               // If sorting cannot be left simply to SQL, prepare to return false
+                               // and add the necessary fields to the SELECT statement
+                               if ($cannotUseSQLForSorting) {
+                                       if ($countNewFields > 0) {
+                                               ArrayUtility::mergeRecursiveWithOverrule($this->queryFields, $newQueryFields);
+                                               $this->queryObject->structure['SELECT'] = array_merge($this->queryObject->structure['SELECT'], $newSelectFields);
+                                               ArrayUtility::mergeRecursiveWithOverrule($this->fieldTrueNames, $newTrueNames);
+/*
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($newQueryFields, 'New query fields');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryFields, 'Updated query fields');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($newTrueNames, 'New field true names');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->fieldTrueNames, 'Updated field true names');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($newSelectFields, 'New select fields');
+\TYPO3\CMS\Core\Utility\DebugUtility::debug($this->queryObject->structure['SELECT'], 'Updated select structure');
+ *
+ */
+                                               // Free some memory
+                                               unset($newQueryFields);
+                                               unset($newSelectFields);
+                                               unset($newTrueNames);
+                                       }
+                                       $this->processOrderBy = FALSE;
+                               } else {
+                                       $this->processOrderBy = TRUE;
+                               }
+                       } else {
+                               $this->processOrderBy = TRUE;
+                       }
+               } else {
+                       $this->processOrderBy = TRUE;
+               }
+       }
+
+       /**
+        * Tries to figure out if a given field of a given table is a text field, based on its TCA definition
+        *
+        * @param string $table Name of the table
+        * @param string $field Name of the field
+        * @return bool TRUE is the field can be considered to be text
+        */
+       public function isATextField($table, $field) {
+               $isTextField = TRUE;
+               // We can guess only if there's a TCA definition
+               if (isset($GLOBALS['TCA'][$table]['columns'][$field])) {
+                       $fieldConfig = $GLOBALS['TCA'][$table]['columns'][$field]['config'];
+                       // It's text, easy :-)
+                       if ($fieldConfig['type'] == 'text') {
+                               $isTextField = TRUE;
+
+                       // It's input, further check the "eval" property
+                       } elseif ($fieldConfig['type'] == 'input') {
+                               // If the field has no eval property, assume it's just text
+                               if (empty($fieldConfig['eval'])) {
+                                       $isTextField = TRUE;
+                               } else {
+                                       $evaluations = explode(',', $fieldConfig['eval']);
+                                       // Check if some eval types are common to both array. If yes, it's not a text field.
+                                       $foundTypes = array_intersect($evaluations, self::$notTextTypes);
+                                       $isTextField = (count($foundTypes) > 0) ? FALSE : TRUE;
+                               }
+
+                       // It's another type, it's definitely not text
+                       } else {
+                               $isTextField = FALSE;
+                       }
+               }
+               return $isTextField;
+       }
+
+       /**
+        * Internal utility method that checks whether a given field
+        * can be found in the fields reference list (i.e. $this->queryFields) for
+        * a given table.
+        *
+        * @param string $table Name of the table inside which to look up
+        * @param string $field Name of the field to search for
+        * @return boolean True if the field was found, false otherwise
+        */
+       protected function isAQueryField($table, $field) {
+               $isAQueryField = FALSE;
+               if (isset($this->queryFields[$table]['fields'])) {
+                       foreach ($this->queryFields[$table]['fields'] as $fieldData) {
+                               if ($fieldData['name'] == $field) {
+                                       $isAQueryField = TRUE;
+                                       break;
+                               }
+                       }
+               }
+               return $isAQueryField;
+       }
+
+       /**
+        * This method tries to match a name to the name or alias of a table used in the query
+        * If no alias or straight table name is found, it looks for a true table name instead
+        * If nothing is found, an exception is thrown
+        *
+        * Explanations: a table name may come from an outside source, a Data Filter or another provider.
+        * In order to apply the condition from that other element to the query,
+        * the table(s) referenced in that other element must match tables used in the query.
+        * If the query uses aliases and the other element not, dataquery tries
+        * (using this method) to match the tables from the other element to aliases used
+        * in the query. This may lead to some kind of guess work in which case a warning is logged.
+        *
+        * @param string $name Name to match
+        * @param string $identifier Some key identifying the circumstances in which the call was made (used for logging)
+        * @throws InvalidQueryException
+        * @return string Alias or table name
+        */
+       protected function matchAliasOrTableName($name, $identifier) {
+               $returnedName = $name;
+
+               // If the name was already match, reuse result
+               if (isset($this->tableMatches[$name])) {
+                       $returnedName = $this->tableMatches[$name];
+
+               // If not, perform matching
+               } else {
+                       // If the name matches an existing alias, use it as is
+                       if (isset($this->queryObject->aliases[$name])) {
+                               $this->tableMatches[$name] = $name;
+
+                       // If the name is not in the list of aliases, try to match it
+                       // to a true table name
+                       } else {
+                               // Get the relation of true table names to aliases
+                               // NOTE: true table names are not necessarily unique
+                               $reversedAliasTable = array_flip($this->queryObject->aliases);
+                               if (isset($reversedAliasTable[$name])) {
+                                       $returnedName = $reversedAliasTable[$name];
+                                       $this->tableMatches[$name] = $reversedAliasTable[$name];
+                                               // Write a notice to the message queue
+                                       $message = sprintf('Potentially unreliable match of table %1$s from component %2$s', $name, $identifier);
+                                       $this->parentObject->getController()->addMessage(
+                                               self::$extKey,
+                                               $message,
+                                               'Unreliable alias match',
+                                               FlashMessage::WARNING
+                                       );
+
+                               // No match found, throw exception
+                               } else {
+                                       $message = sprintf('No match found for table %1$s from component %2$s', $name, $identifier);
+                                       throw new InvalidQueryException($message, 1291753564);
+                               }
+                       }
+               }
+               return $returnedName;
+       }
+
+       /**
+        * Adds an extra field to be SELECTed.
+        *
+        * It must be added to the SELECT list, to the list of fields being queried
+        * and to the registry of true names.
+        *
+        * @param string $field Name of the field to add
+        * @param string $tableAlias Alias of the table to add the field to
+        * @param string $table True name of the table to add the field to
+        */
+       protected function addExtraField($field, $tableAlias, $table) {
+               $newFieldName = $tableAlias . '.' . $field;
+               $newFieldAlias = $field;
+               if ($tableAlias != $this->queryObject->mainTable) {
+                       $newFieldAlias = $tableAlias . '$' . $field;
+                       $newFieldName .= ' AS ' . $newFieldAlias;
+               }
+               $this->queryObject->structure['SELECT'][] = $newFieldName;
+               $this->queryFields[$tableAlias]['fields'][] = array('name' => $field, 'function' => FALSE);
+               $this->fieldTrueNames[$newFieldAlias] = array(
+                       'table' => $table,
+                       'aliasTable' => $tableAlias,
+                       'field' => $field,
+                       'mapping' => array('table' => $tableAlias, 'field' => $field)
+               );
+       }
+
+// Setters and getters
+
+       /**
+        * Adds a condition for the WHERE clause.
+        *
+        * @param string $clause SQL WHERE clause (without WHERE)
+        * @return void
+        */
+       public function addWhereClause($clause) {
+               if (!empty($clause)) {
+                       $this->queryObject->structure['WHERE'][] = $clause;
+               }
+       }
+
+       /**
+        * Adds a condition to the ON clause of a given table.
+        *
+        * @param string $clause SQL to add to the ON clause
+        * @param string $alias Alias of the table to the statement to
+        * @return void
+        */
+       public function addOnClause($clause, $alias) {
+               if (!empty($this->queryObject->structure['JOIN'][$alias]['on'])) {
+                       $this->queryObject->structure['JOIN'][$alias]['on'] .= ' AND ';
+               }
+               $this->queryObject->structure['JOIN'][$alias]['on'] .= '(' . $clause . ')';
+       }
+
+       /**
+        * Returns the structure of the parsed query.
+        *
+        * There should be little real-life uses for this, but it is used by the
+        * test case to get the parsed structure.
+        *
+        * @return array The parsed query
+        */
+       public function getQueryStructure() {
+               return $this->queryObject->structure;
+       }
+
+       /**
+        * Returns the name (alias) of the main table of the query,
+        * which is the table name that appears in the FROM clause, or the alias, if any.
+        *
+        * @return string Main table name (alias)
+        */
+       public function getMainTableName() {
+               return $this->queryObject->mainTable;
+       }
+
+       /**
+        * Returns an array containing the list of all subtables in the query,
+        * i.e. the tables that appear in any of the JOIN statements.
+        *
+        * @return array Names of all the joined tables
+        */
+       public function getSubtablesNames() {
+               return $this->queryObject->subtables;
+       }
+
+       /**
+        * Takes an alias and returns the true table name
+        *
+        * @param string $alias Alias of a table
+        * @return string True name of the corresponding table
+        */
+       public function getTrueTableName($alias) {
+               return $this->queryObject->aliases[$alias];
+       }
+
+       /**
+        * Takes the alias and returns it's true name.
+        *
+        * The alias is the full alias as used in the query (e.g. table$field).
+        *
+        * @param string $alias Alias of a field
+        * @return array Array with the true name of the corresponding field
+        *               and the true name of the table it belongs and the alias of that table
+        */
+       public function getTrueFieldName($alias) {
+               $trueNameInformation = $this->fieldTrueNames[$alias];
+                       // Assemble field key (possibly disambiguated with function name)
+               $fieldKey = $trueNameInformation['field'];
+//             if (!empty($trueNameInformation['function'])) {
+//                     $fieldKey .= '_' . $trueNameInformation['function'];
+//             }
+                       // If the field has an explicit alias, we must also pass back that information
+               if (isset($this->queryObject->fieldAliases[$trueNameInformation['aliasTable']][$fieldKey])) {
+                       $alias = $this->queryObject->fieldAliases[$trueNameInformation['aliasTable']][$fieldKey];
+                               // Check if the alias contains a table name
+                               // If yes, strip it, as this information is already handled
+                       if (strpos($alias, '.') !== FALSE) {
+                               list(, $field) = explode('.', $alias);
+                               $alias = $field;
+                       }
+                       $trueNameInformation['mapping']['alias'] = $alias;
+               }
+               return $trueNameInformation;
+       }
+
+       /**
+        * Returns the list of fields defined for ordering the data.
+        *
+        * @return array Fields for ordering (and sort order)
+        */
+       public function getOrderByFields() {
+               return $this->queryObject->orderFields;
+       }
+
+       /**
+        * Returns the query object.
+        *
+        * @return \Tesseract\Dataquery\Utility\QueryObject
+        */
+       public function getSQLObject() {
+               return $this->queryObject;
+       }
+
+       /**
+        * Indicates whether the language overlay mechanism must/can be handled for a given table.
+        *
+        * @param string $table True name of the table to handle
+        * @return boolean True if language overlay must and can be performed, false otherwise
+        * @see \Tesseract\Dataquery\Parser\QueryParser::addTypo3Mechanisms()
+        */
+       public function mustHandleLanguageOverlay($table) {
+               return (isset($this->doOverlays[$table])) ? $this->doOverlays[$table] : FALSE;
+       }
+
+       /**
+        * Indicates whether the language overlay mechanism must/can be handled for a given table.
+        *
+        * @param string $table True name of the table to handle
+        * @return boolean True if language overlay must and can be performed, false otherwise
+        * @see \Tesseract\Dataquery\Parser\QueryParser::addTypo3Mechanisms()
+        */
+       public function mustHandleVersioningOverlay($table) {
+               return (isset($this->doVersioning[$table])) ? $this->doVersioning[$table] : FALSE;
+       }
+
+       /**
+        * Returns whether the ordering of the records was done in the SQL query or not.
+        *
+        * @return boolean True if SQL was used, false otherwise
+        */
+       public function isSqlUsedForOrdering() {
+               return $this->processOrderBy;
+       }
+
+       /**
+        * Returns true if any ordering has been defined at all. False otherwise.
+        *
+        * @return boolean True if there's at least one ordering criterion, false otherwise
+        */
+       public function hasOrdering() {
+               return count($this->queryObject->orderFields) > 0;
+       }
+
+       /**
+        * Returns the name of the first significant table to be INNER JOINed.
+        *
+        * A "significant table" is a table that has a least one field SELECTed
+        * If the first significant table is not INNER JOINed or if there are no JOINs
+        * or no INNER JOINs, an empty string is returned
+        *
+        * @return string Alias of the first significant table, if INNER JOINed, empty string otherwise
+        */
+       public function hasInnerJoinOnFirstSubtable() {
+               $returnValue = '';
+               if (count($this->queryObject->structure['JOIN']) > 0) {
+                       foreach ($this->queryObject->structure['JOIN'] as $alias => $joinInfo) {
+                               if (isset($this->queryFields[$alias])) {
+                                       if ($joinInfo['type'] == 'inner') {
+                                               $returnValue = $alias;
+                                       }
+                                       break;
+                               }
+                       }
+               }
+               return $returnValue;
+       }
+
+       /**
+        * Gets the limit that was defined for a given sub-table
+        * (i.e. a JOINed table). If no limit exists, 0 is returned.
+        *
+        * @param string $table Name of the table to find the limit for
+        * @return integer Value of the limit, or 0 if not defined
+        */
+       public function getSubTableLimit($table) {
+               return isset($this->queryObject->structure['JOIN'][$table]['limit']) ? $this->queryObject->structure['JOIN'][$table]['limit'] : 0;
+       }
+
+       /**
+        * Checks for the existence of a field (possibly with alias) called "uid"
+        * for the query's main table.
+        *
+        * @return bool TRUE if a "uid" field is present, FALSE otherwise
+        */
+       protected function checkUidForDistinctUsage() {
+               $hasUid = FALSE;
+               // There should be either an alias called "uid" or "table$uid"
+               // (where "table" is the name of the main table)
+               $possibleKeys = array('uid', $this->queryObject->mainTable . '$uid');
+               // Also add the possible aliases of the main table
+               // NOTE: this may be wrong when there are more than 1 alias for the main table,
+               // as the uid may actually belong to another table
+               $reversedAliases = array_flip($this->queryObject->aliases);
+               foreach ($reversedAliases as $table => $alias) {
+                       if ($table == $this->queryObject->mainTable) {
+                               $possibleKeys[] = $alias . '$uid';
+                       }
+               }
+               // Loop on all possible keys and exit successfully if one matches a field mapped to "uid"
+               foreach ($possibleKeys as $key) {
+                       if (isset($this->fieldTrueNames[$key]) && $this->fieldTrueNames[$key]['mapping']['field'] == 'uid') {
+                               $hasUid = TRUE;
+                               break;
+                       }
+               }
+               return $hasUid;
+       }
+
+       /**
+        * Returns information about each field that appears more than once in the current query.
+        *
+        * @return array
+        */
+       protected function checkForDuplicateFields() {
+               $duplicates = array();
+               $fieldCountPerTable = array();
+               // Loop on all included fields and make a list of aliases per table and field
+               // Note that these are the fields explicitly entered in the SELECT statement
+               // plus all base fields added for the needs of dataquery
+               foreach ($this->fieldTrueNames as $alias => $aliasInformation) {
+                       $table = $aliasInformation['aliasTable'];
+                       $field = $aliasInformation['field'];
+                       if (!isset($fieldCountPerTable[$table])) {
+                               $fieldCountPerTable[$table] = array();
+                       }
+                       if (!isset($fieldCountPerTable[$table][$field])) {
+                               $fieldCountPerTable[$table][$field] = array($alias);
+                       } else {
+                               $fieldCountPerTable[$table][$field][] = $alias;
+                       }
+               }
+               // Loop on the aliases list found in the first loop
+               // List a warning for each field (in a given table) with multiple aliases
+               foreach ($fieldCountPerTable as $table => $countPerField) {
+                       foreach ($countPerField as $field => $aliases) {
+                               if (count($aliases) > 1) {
+                                       $duplicates[] = 'In table ' . $table . ', duplicates for ' . $field . ' as: ' . implode(',', $aliases);
+                               }
+                       }
+
+               }
+               return $duplicates;
+       }
+}
diff --git a/Classes/Parser/SqlParser.php b/Classes/Parser/SqlParser.php
new file mode 100644 (file)
index 0000000..49fd3e8
--- /dev/null
@@ -0,0 +1,469 @@
+<?php
+namespace Tesseract\Dataquery\Parser;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use Tesseract\Dataquery\Exception\InvalidQueryException;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+use Cobweb\Overlays\OverlayEngine;
+
+/**
+ * SQL parser class for extension "dataquery"
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class SqlParser {
+       /**
+        * @var array   List of all the main keywords accepted in the query
+        */
+       static protected $tokens = array('INNER JOIN', 'LEFT JOIN', 'RIGHT JOIN', 'WHERE', 'GROUP BY', 'ORDER BY', 'LIMIT', 'OFFSET', 'MERGED');
+
+       /**
+        * @var \Tesseract\Dataquery\Utility\QueryObject Structured type containing the parts of the parsed query
+        */
+       protected $queryObject;
+
+       /**
+        *
+        * @var integer Number of SQL function calls inside SELECT statement
+        */
+       protected $numFunctions = 0;
+
+       /**
+        * Parses a SQL query and extract structured information about it.
+        *
+        * @param string $query The SQL to parse
+        * @throws InvalidQueryException
+        * @return \Tesseract\Dataquery\Utility\QueryObject An object containing the parsed query information
+        */
+       public function parseSQL($query) {
+               $this->queryObject = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Utility\\QueryObject');
+
+               // First find the start of the SELECT statement
+               $selectPosition = stripos($query, 'SELECT');
+               if ($selectPosition === FALSE) {
+                       throw new InvalidQueryException('Missing SELECT keyword', 1272556228);
+               }
+               // Next find the position of the last FROM keyword
+               // There may be more than one FROM keyword when some functions are used
+               // (example: EXTRACT(YEAR FROM tstamp))
+               // NOTE: sub-selects are not supported, but these could be a source
+               // of additional FROMs
+               $queryParts = preg_split('/\bFROM\b/', $query);
+               // If the query was not split, FROM keyword is missing
+               if (count($queryParts) == 1) {
+                       throw new InvalidQueryException('Missing FROM keyword', 1272556601);
+               }
+               $afterLastFrom = array_pop($queryParts);
+
+               // Everything before the last FROM is the SELECT part
+               // This is parsed last as we need information about any table aliases used in the query first
+               $selectPart = implode(' FROM ', $queryParts);
+               $selectedFields = trim(substr($selectPart, $selectPosition + 6));
+
+               // Get all parts of the query after SELECT ... FROM, using the SQL keywords as tokens
+               // The returned matches array contains the keywords matched (in position 2) and the string after each keyword (in position 3)
+               $regexp = '/(' . implode('|', self::$tokens) . ')/';
+               $matches = preg_split($regexp, $afterLastFrom, -1, PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($regexp);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($query);
+//\TYPO3\CMS\Core\Utility\DebugUtility::debug($matches, 'Matches');
+
+               // The first position is the string that followed the main FROM keyword
+               // Parse that information. It's important to do this first,
+               // as we need to know the query' main table for later
+               $fromPart = array_shift($matches);
+               // NOTE: this may throw an Exception, but we let it bubble up
+               $this->parseFromStatement($fromPart);
+
+               // Fill the structure array, as suited for each keyword
+               $i = 0;
+               $numMatches = count($matches);
+               while ($i < $numMatches) {
+                       $keyword = $matches[$i];
+                       $i++;
+                       $value = $matches[$i];
+                       $i++;
+                       switch ($keyword) {
+                               case 'INNER JOIN':
+                               case 'LEFT JOIN':
+                               case 'RIGHT JOIN':
+                                       // Extract the JOIN type (INNER, LEFT or RIGHT)
+                                       $joinType = strtolower(substr($keyword, 0, strpos($keyword,'JOIN') - 1));
+                                       $theJoin = array();
+                                       $theJoin['type'] = $joinType;
+                                       // Separate the table from the join condition
+                                       $parts = preg_split('/\bON\b/', $value);
+                                       // Separate an alias from the table name
+                                       $moreParts = GeneralUtility::trimExplode('AS', $parts[0]);
+                                       $theJoin['table'] = trim($moreParts[0]);
+                                       if (count($moreParts) > 1) {
+                                               $theJoin['alias'] = trim($moreParts[1]);
+                                       }
+                                       else {
+                                               $theJoin['alias'] = $theJoin['table'];
+                                       }
+                                       $this->queryObject->subtables[] = $theJoin['alias'];
+                                       $this->queryObject->aliases[$theJoin['alias']] = $theJoin['table'];
+                                       // Handle the "ON" part which may contain the non-SQL keyword "MAX"
+                                       // This keyword is not used in the SQL query, but is an indication to the wrapper that
+                                       // we want only a single record from this join
+                                       if (count($parts) > 1) {
+                                               $moreParts = GeneralUtility::trimExplode('MAX', $parts[1]);
+                                               $theJoin['on'] = trim($moreParts[0]);
+                                               if (count($moreParts) > 1) {
+                                                       $theJoin['limit'] = $moreParts[1];
+                                               }
+                                       }
+                                       else {
+                                               $theJoin['on'] = '';
+                                       }
+                                       if (!isset($this->queryObject->structure['JOIN'])) $this->queryObject->structure['JOIN'] = array();
+                                       $this->queryObject->structure['JOIN'][$theJoin['alias']] = $theJoin;
+                                       break;
+                               case 'WHERE':
+                                       $this->queryObject->structure[$keyword][] = trim($value);
+                                       break;
+                               case 'ORDER BY':
+                               case 'GROUP BY':
+                                       $orderParts = explode(',', $value);
+                                       foreach ($orderParts as $part) {
+                                               $thePart = trim($part);
+                                               $this->queryObject->structure[$keyword][] = $thePart;
+                                                       // In case of ORDER BY, perform additional operation to get field name and sort order separately
+                                               if ($keyword == 'ORDER BY') {
+                                                       $finerParts = preg_split('/\s/', $thePart, -1, PREG_SPLIT_NO_EMPTY);
+                                                       $orderField = $finerParts[0];
+                                                       $orderSort = (isset($finerParts[1])) ? $finerParts[1] : 'ASC';
+                                                       $this->queryObject->orderFields[] = array('field' => $orderField, 'order' => $orderSort);
+                                               }
+
+                                       }
+                                       break;
+                               case 'LIMIT':
+                                       if (strpos($value, ',') !== FALSE) {
+                                               $limitParts = GeneralUtility::trimExplode(',', $value, TRUE);
+                                               $this->queryObject->structure['OFFSET'] = intval($limitParts[0]);
+                                               $this->queryObject->structure[$keyword] = intval($limitParts[1]);
+                                       } else {
+                                               $this->queryObject->structure[$keyword] = intval($value);
+                                       }
+                                       break;
+                               case 'OFFSET':
+                                       $this->queryObject->structure[$keyword] = intval($value);
+                                       break;
+                       }
+               }
+               // Free some memory
+               unset($matches);
+
+               // Parse the SELECT part
+               $this->parseSelectStatement($selectedFields);
+
+               // Return the object containing the parsed query
+               return $this->queryObject;
+       }
+
+       /**
+        * Parses the SELECT part of the statement and isolates each field in the selection.
+        *
+        * @param string $select The beginning of the SQL statement, between SELECT and FROM (both excluded)
+        * @throws InvalidQueryException
+        * @return void
+        */
+       public function parseSelectStatement($select) {
+               if (empty($select)) {
+                       throw new InvalidQueryException('Nothing SELECTed', 1280323976);
+               }
+
+               // Parse the SELECT part
+               // First, check if the select string starts with "DISTINCT"
+               // If yes, remove that and set the distinct flag to true
+               $distinctPosition = strpos($select, 'DISTINCT');
+               if ($distinctPosition === 0) {
+                       $this->queryObject->structure['DISTINCT'] = TRUE;
+                       $croppedString = substr($select, 8);
+                       $select = trim($croppedString);
+               }
+               // Next, parse the rest of the string character by character
+               $stringLenth = strlen($select);
+               $openBrackets = 0;
+               $lastBracketPosition = 0;
+               $currentField = '';
+               $currentPosition = 0;
+               $hasFunctionCall = FALSE;
+               $hasWildcard = FALSE;
+               for ($i = 0; $i < $stringLenth; $i++) {
+                       // Get the current character
+                       $character = $select[$i];
+                       // Count the position inside the current field
+                       // This is reset for each new field found
+                       $currentPosition++;
+                       switch ($character) {
+                               // An open bracket is the sign of a function call
+                               // Functions may be nested, so we count the number of open brackets
+                               case '(':
+                                       $currentField .= $character;
+                                       $openBrackets++;
+                                       $hasFunctionCall = TRUE;
+                                       break;
+
+                               // Decrease the open bracket count
+                               case ')':
+                                       $currentField .= $character;
+                                       $openBrackets--;
+                                       // Store position of closing bracket (minus one), as we need the position
+                                       // of the last one later for further processing
+                                       $lastBracketPosition = $currentPosition - 1;
+                                       break;
+
+                               // If the wildcard character appears outside of function calls,
+                               // take it into consideration. Otherwise not (it might be COUNT(*) for example)
+                               case '*':
+                                       $currentField .= $character;
+                                       if (!$hasFunctionCall) {
+                                               $hasWildcard = TRUE;
+                                       }
+                                       break;
+
+                               // A comma indicates that we have reached the end of a field,
+                               // unless there are open brackets, in which case the comma is
+                               // a separator of function arguments
+                               case ',':
+                                       // We are at the end of a field: add it to the list of fields
+                                       // and reset some values
+                                       if ($openBrackets == 0) {
+                                               $this->parseSelectField(trim($currentField), $lastBracketPosition, $hasFunctionCall, $hasWildcard);
+                                               $currentField = '';
+                                               $hasFunctionCall = FALSE;
+                                               $hasWildcard = FALSE;
+                                               $currentPosition = 0;
+                                               $lastBracketPosition = 0;
+
+                                       // We're inside a function, keep the comma and keep the current character
+                                       } else {
+                                               $currentField .= $character;
+                                       }
+                                       break;
+
+                               // Nothing special, just add the current character to the current field's name
+                               default:
+                                       $currentField .= $character;
+                                       break;
+                       }
+               }
+               // Upon exit from the loop, save the last field found,
+               // except if there's still an open bracket, in which case we have a syntax error
+               if ($openBrackets > 0) {
+                       throw new InvalidQueryException('Bad SQL syntax, opening and closing brackets are not balanced', 1272954424);
+               } else {
+                       $this->parseSelectField(trim($currentField), $lastBracketPosition, $hasFunctionCall, $hasWildcard);
+               }
+       }
+
+       /**
+        * Parses one field from the SELECT part of the SQL query and analyzes its content.
+        *
+        * In particular it will expand the "*" wildcard to include
+        * all fields. It also keeps tracks of field aliases.
+        *
+        * @param string $fieldString The string to parse
+        * @param integer $lastBracketPosition The position of the last closing bracket in the string, if any
+        * @param boolean $hasFunctionCall True if a SQL function call was detected in the string
+        * @param boolean $hasWildcard Ttrue if the wildcard character (*) was detected in the string
+        * @return void
+        */
+       protected function parseSelectField($fieldString, $lastBracketPosition = 0, $hasFunctionCall = FALSE, $hasWildcard = FALSE) {
+               // Exit early if field string is empty
+               if (empty($fieldString)) {
+                       return;
+               }
+
+               // If the string is just * (or possibly table.*), get all the fields for the table
+               if ($hasWildcard) {
+                       // It's only *, set table as main table
+                       if ($fieldString === '*') {
+                               $table = $this->queryObject->mainTable;
+                               $alias = $table;
+
+                       // It's table.*, extract table name
+                       } else {
+                               $fieldParts = GeneralUtility::trimExplode('.', $fieldString, 1);
+                               $table = (isset($this->queryObject->aliases[$fieldParts[0]]) ? $this->queryObject->aliases[$fieldParts[0]] : $fieldParts[0]);
+                               $alias = $fieldParts[0];
+                       }
+                       if (!isset($this->queryObject->hasBaseFields[$alias])) {
+                               $this->queryObject->hasBaseFields[$alias] = array('uid' => FALSE, 'pid' => FALSE);
+                       }
+                       // Get all fields for the given table
+                       $fieldInfo = OverlayEngine::getAllFieldsForTable($table);
+                       $fields = array_keys($fieldInfo);
+                       // Add all fields to the query structure
+                       foreach ($fields as $aField) {
+                               if ($aField == 'uid') {
+                                       $this->queryObject->hasBaseFields[$alias]['uid'] = TRUE;
+                               } elseif ($aField == 'pid') {
+                                       $this->queryObject->hasBaseFields[$alias]['pid'] = TRUE;
+                               }
+                               $this->queryObject->structure['SELECT'][] = array(
+                                       'table' => $table,
+                                       'tableAlias' => $alias,
+                                       'field' => $aField,
+                                       'fieldAlias' => '',
+                                       'function' => FALSE
+                               );
+                       }
+
+               // Else, the field is some string, analyse it
+               } else {
+
+                       // If there's an alias, extract it and continue parsing
+                       // An alias is indicated by a "AS" keyword after the last closing bracket if any
+                       // (brackets indicate a function call and there might be "AS" keywords inside them)
+                       $fieldAlias = '';
+                       if ($lastBracketPosition > strlen($fieldString)) {
+                           $asPosition = FALSE;
+                       } else {
+                           $asPosition = strpos($fieldString, ' AS ', $lastBracketPosition);
+                       }
+                       if ($asPosition !== FALSE) {
+                               $fieldAlias = trim(substr($fieldString, $asPosition + 4));
+                               $fieldString = trim(substr($fieldString, 0, $asPosition));
+                       }
+                       if ($hasFunctionCall) {
+                               $this->numFunctions++;
+                               $alias = $this->queryObject->mainTable;
+                               $table = (isset($this->queryObject->aliases[$alias]) ? $this->queryObject->aliases[$alias] : $alias);
+                               $field = $fieldString;
+                               // Function calls need aliases
+                               // If none was given, define one
+                               if (empty($fieldAlias)) {
+                                       $fieldAlias = 'function_' . $this->numFunctions;
+                               }
+
+                       // There's no function call
+                       } else {
+
+                               // If there's a dot, get table name
+                               if (stristr($fieldString, '.')) {
+                                       $fieldParts = GeneralUtility::trimExplode('.', $fieldString, 1);
+                                       $table = (isset($this->queryObject->aliases[$fieldParts[0]]) ? $this->queryObject->aliases[$fieldParts[0]] : $fieldParts[0]);
+                                       $alias = $fieldParts[0];
+                                       $field = $fieldParts[1];
+
+                               // No dot, the table is the main one
+                               } else {
+                                       $alias = $this->queryObject->mainTable;
+                                       $table = (isset($this->queryObject->aliases[$alias]) ? $this->queryObject->aliases[$alias] : $alias);
+                                       $field = $fieldString;
+                               }
+                       }
+
+                       // For fulltext search, create placeholder which is replaced later with the full MATCH() statement
+                       // (if necessary)
+                       if (strpos($field, 'fulltext:') !== FALSE || strpos($field, 'fulltext_natural:') !== FALSE) {
+                               $fulltextSearchParts = explode(':', $field);
+                               $field = 'fulltext.' . $fulltextSearchParts[1];
+                               // Create placeholder entry (to be filled later)
+                               // If no fulltext value is entered or the table has no fulltext index, the dummy value "1" will be used,
+                               // which is neutral to the query.
+                               $this->queryObject->fulltextSearchPlaceholders[$table . '.' . $field] = '1';
+                       }
+
+                       // Set the appropriate flag if the field is uid or pid
+                       // Initialize first, if not yet done
+                       if (!isset($this->queryObject->hasBaseFields[$alias])) {
+                               $this->queryObject->hasBaseFields[$alias] = array('uid' => FALSE, 'pid' => FALSE);
+                       }
+                       if ((empty($fieldAlias) && $field == 'uid') || (!empty($fieldAlias) && $fieldAlias == 'uid')) {
+                               $this->queryObject->hasBaseFields[$alias]['uid'] = TRUE;
+                       } elseif ((empty($fieldAlias) && $field == 'pid') || (!empty($fieldAlias) && $fieldAlias == 'pid')) {
+                               $this->queryObject->hasBaseFields[$alias]['pid'] = TRUE;
+                       }
+                       // Add field's information to query structure
+                       $this->queryObject->structure['SELECT'][] = array(
+                               'table' => $table,
+                               'tableAlias' => $alias,
+                               'field' => $field,
+                               'fieldAlias' => $fieldAlias,
+                               'function' => $hasFunctionCall
+                       );
+
+                       // If there's an alias for the field, store it in a separate array, for later use
+                       if (!empty($fieldAlias)) {
+                               if (!isset($this->queryObject->fieldAliases[$alias])) {
+                                       $this->queryObject->fieldAliases[$alias] = array();
+                               }
+                               $this->queryObject->fieldAliases[$alias][$field] = $fieldAlias;
+                               // Keep track of which field the alias is related to
+                               // (this is used by the parser to map alias used in filters)
+                               // If the alias is related to a function, we store the function syntax as is,
+                               // otherwise we map the alias to the syntax table.field
+                               if ($hasFunctionCall) {
+                                       $this->queryObject->fieldAliasMappings[$fieldAlias] = $field;
+                               } else {
+                                       $this->queryObject->fieldAliasMappings[$fieldAlias] = $table . '.' . $field;
+                               }
+                       }
+               }
+       }
+
+       /**
+        * Parses the FROM statement of the query,
+        * which may be comprised of a comma-separated list of tables.
+        *
+        * @param string $from The FROM statement
+        * @throws InvalidQueryException
+        * @return void
+        */
+       public function parseFromStatement($from) {
+               $fromTables = GeneralUtility::trimExplode(',', $from, TRUE);
+               $numTables = count($fromTables);
+               // If there's nothing in the string, thrown an exception
+               if ($numTables == 0) {
+                       throw new InvalidQueryException('No table defined in query (FROM).', 1280323639);
+               }
+
+               for ($i = 0; $i < $numTables; $i++) {
+                       $tableName = $fromTables[$i];
+                       $tableAlias = $tableName;
+                       if (strpos($fromTables[$i], ' AS ') !== FALSE) {
+                               $tableParts = GeneralUtility::trimExplode(' AS ', $fromTables[$i], TRUE);
+                               $tableName = $tableParts[0];
+                               $tableAlias = $tableParts[1];
+                       }
+                       // Consider the first table to be the main table of the query,
+                       // i.e. the table to which all others are JOINed
+                       if ($i == 0) {
+                               $this->queryObject->structure['FROM']['table'] = $tableName;
+                               $this->queryObject->structure['FROM']['alias'] = $tableAlias;
+                               $this->queryObject->mainTable = $tableAlias;
+
+                       // Each further table in the FROM statement is registered
+                       // as being INNER JOINed
+                       } else {
+                               $this->queryObject->structure['JOIN'][$tableAlias] = array(
+                                       'type' => 'inner',
+                                       'table' => $tableName,
+                                       'alias' => $tableAlias,
+                                       'on' => ''
+                               );
+                               $this->queryObject->subtables[] = $tableAlias;
+                       }
+                       $this->queryObject->aliases[$tableAlias] = $tableName;
+               }
+       }
+}
diff --git a/Classes/Sample/DataQueryHook.php b/Classes/Sample/DataQueryHook.php
new file mode 100644 (file)
index 0000000..cf7da45
--- /dev/null
@@ -0,0 +1 @@
+<?php\rnamespace Tesseract\Dataquery\Hook;\r\r/*\r * This file is part of the TYPO3 CMS project.\r *\r * It is free software; you can redistribute it and/or modify it under\r * the terms of the GNU General Public License, either version 2\r * of the License, or any later version.\r *\r * For the full copyright and license information, please read the\r * LICENSE.txt file that was distributed with this source code.\r *\r * The TYPO3 project - inspiring people to share!\r */\r\r/**\r * This is a sample file for adding a hook to dataquery. Add the line below in an ext_localconf.php file and *adapt* it:\r *\r * $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['dataquery']['postProcessDataStructureBeforeCache'][] = 'Tesseract\\Dataquery\\Hook\\DataQueryHook';\r *\r * @author Your Name <your.name@domain.tld>\r */\rclass DataQueryHook {\r\r     /**\r     * Modifies the data structure for special needs.\r       * $pObj is a reference to the parent Object. Have a look at the object definition.\r     *\r      * @param array $structure Data structure\r       * @param \Tesseract\Dataquery\Component\DataProvider $dataProvider Parent object\r       * @return void\r         */\r//  public function postProcessDataStructure($structure, $dataProvider) {\r//\r//             if ($structure['name'] == 'structureName') {\r//                 foreach ($structure['records'] as &$record) {\r//\r//                     }\r//            }\r//            return $structure;\r//   }\r\r     /**\r     * Modifies the data structure for special needs before the structure is cached. Optimal for performance!\r       * $pObj is a reference to the parent Object. Have a look at the object definition.\r     *\r      * @param array $structure Data structure\r       * @param \Tesseract\Dataquery\Component\DataProvider $dataProvider Parent object\r       * @return void\r         */\r//  public function postProcessDataStructureBeforeCache($structure, $dataProvider) {\r//             if ($structure['name'] == 'structureName') {\r//                 foreach ($structure['records'] as &$record) {\r//\r//                     }\r//            }\r//            return $structure;\r//   }\r\r}\r
\ No newline at end of file
diff --git a/Classes/UserFunction/FormEngine.php b/Classes/UserFunction/FormEngine.php
new file mode 100644 (file)
index 0000000..3b1e7ff
--- /dev/null
@@ -0,0 +1,139 @@
+<?php
+namespace Tesseract\Dataquery\UserFunction;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * Displays a custom field in the BE to check for fulltext indices and
+ * provide useful hints.
+ *
+ * @author Fabien Udriot (Cobweb) <support@cobweb.ch>
+ * @author Francois Suter (Cobweb) <support@cobweb.ch>
+ * @package TYPO3
+ * @subpackage dataquery
+ */
+class FormEngine {
+
+       /**
+        * @var \Tesseract\Dataquery\Utility\DatabaseAnalyser
+        */
+       protected $analyser;
+
+       /**
+        * @var \Tesseract\Dataquery\Parser\SqlParser
+        */
+       protected $sqlParser;
+
+       /**
+        * @var \TYPO3\CMS\Lang\LanguageService
+        */
+       protected $language;
+
+       /**
+        * Stores the main table of the SQL query
+        * @var string
+        */
+       protected $table;
+
+       /**
+        * Constructor
+        */
+       public function __construct() {
+               $this->language = $GLOBALS['LANG'];
+               $this->analyser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Utility\\DatabaseAnalyser');
+               $this->sqlParser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Parser\\SqlParser');
+       }
+
+       /**
+        * This method format a message regarding FULLTEXT indexes in the database towards a BE user.
+        *
+        * @param array $parameters Properties of the field being modified
+        * @param \TYPO3\CMS\Backend\Form\FormEngine $parentObject Back-reference to the calling object
+        * @return string
+        */
+       public function renderFulltextIndices($parameters, \TYPO3\CMS\Backend\Form\FormEngine $parentObject) {
+               $output = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_index_or_missing_table');
+
+               if (empty($parameters['row']['sql_query'])) {
+                       $output = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_query');
+               } else {
+
+                       // Fetch the query parts
+                       try {
+                               $query = $this->sqlParser->parseSQL($parameters['row']['sql_query']);
+
+                               if (!empty($query->structure['FROM']['table'])) {
+                                       $this->table = $query->structure['FROM']['table'];
+                                       if ($this->analyser->hasIndex($this->table)) {
+                                               $output = $this->getMessageOk();
+                                       } else {
+                                               $output = $this->getMessageNoIndexFound();
+                                       }
+                               }
+                       }
+                       catch (\Exception $e) {
+                               // Nothing to do, the default message will do fine
+                       }
+               }
+               return $output;
+       }
+
+       /**
+        * Formats a message for the BE displaying all possible FULLTEXT index to the BE User.
+        *
+        * @return string
+        */
+       protected function getMessageOk() {
+               $fields = $this->analyser->getFields($this->table);
+               $output = '';
+               foreach ($fields as $index => $indexedFields) {
+                       $output .= sprintf(
+                               '%s <strong>fulltext:%s AS foo</strong>',
+                               $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.syntax_for_query'),
+                               $index
+                       );
+                       $output .= sprintf(
+                               '<br/>' . $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.indexed_fields') . '<br/>',
+                               $index,
+                               $indexedFields
+                       );
+                       $output = '<div style="margin-bottom: 10px">' . $output . '</div>';
+               }
+               return '<div>' . $output . '</div>';
+       }
+
+       /**
+        * Formats a message for the BE when no FULLTEXT index is found against a table.
+        *
+        * @return string
+        */
+       protected function getMessageNoIndexFound() {
+               $outputs = array();
+
+               $string = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_index');
+               $outputs[] = sprintf($string, $this->table);
+
+               $tables = $this->analyser->getTables();
+               if (!empty($tables)) {
+                       $listOfTables = implode(', ', $tables);
+                       $string = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.tables_list');
+                       $outputs[] = sprintf($string, $listOfTables);
+               } else {
+                       $outputs[] = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_table_found');
+               }
+               return implode('<br>', $outputs);
+       }
+}
diff --git a/Classes/Userfunc/FormEngine.php b/Classes/Userfunc/FormEngine.php
deleted file mode 100644 (file)
index 18ce1a7..0000000
+++ /dev/null
@@ -1,151 +0,0 @@
-<?php
-/***************************************************************
-*  Copyright notice
-*
-*  (c) 2012-2015 Fabien Udriot (Cobweb) <fudriot@cobweb.ch>
-*  All rights reserved
-*
-*  This script is part of the TYPO3 project. The TYPO3 project is
-*  free software; you can redistribute it and/or modify
-*  it under the terms of the GNU General Public License as published by
-*  the Free Software Foundation; either version 2 of the License, or
-*  (at your option) any later version.
-*
-*  The GNU General Public License can be found at
-*  http://www.gnu.org/copyleft/gpl.html.
-*
-*  This script is distributed in the hope that it will be useful,
-*  but WITHOUT ANY WARRANTY; without even the implied warranty of
-*  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-*  GNU General Public License for more details.
-*
-*  This copyright notice MUST APPEAR in all copies of the script!
-***************************************************************/
-
-/**
- * Displays a custom field in the BE to check for fulltext indices and
- * provide useful hints.
- *
- * @author Fabien Udriot (Cobweb) <support@cobweb.ch>
- * @author Francois Suter (Cobweb) <support@cobweb.ch>
- * @package TYPO3
- * @subpackage dataquery
- */
-class Tx_Dataquery_Userfunc_FormEngine {
-
-       /**
-        * @var Tx_Dataquery_Utility_DatabaseAnalyser
-        */
-       protected $analyser;
-
-       /**
-        * @var tx_dataquery_sqlparser
-        */
-       protected $sqlParser;
-
-       /**
-        * @var language
-        */
-       protected $language;
-
-       /**
-        * Stores the main table of the SQL query
-        * @var string
-        */
-       protected $table;
-
-       /**
-        * Constructor
-        */
-       public function __construct() {
-               $this->language = $GLOBALS['LANG'];
-               $this->analyser = t3lib_div::makeInstance('Tx_Dataquery_Utility_DatabaseAnalyser');
-               $this->sqlParser = t3lib_div::makeInstance('tx_dataquery_sqlparser');
-       }
-
-       /**
-        * This method format a message regarding FULLTEXT indexes in the database towards a BE user.
-        *
-        * @param array $parameters Properties of the field being modified
-        * @param t3lib_TCEforms $parentObject Back-reference to the calling object
-        * @return string
-        */
-       public function renderFulltextIndices($parameters, t3lib_TCEforms $parentObject) {
-               $output = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_index_or_missing_table');
-
-               if (empty($parameters['row']['sql_query'])) {
-                       $output = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_query');
-               } else {
-
-                       // Fetch the query parts
-                       try {
-                               $query = $this->sqlParser->parseSQL($parameters['row']['sql_query']);
-
-                               if (!empty($query->structure['FROM']['table'])) {
-                                       $this->table = $query->structure['FROM']['table'];
-                                       if ($this->analyser->hasIndex($this->table)) {
-                                               $output = $this->getMessageOk();
-                                       } else {
-                                               $output = $this->getMessageNoIndexFound();
-                                       }
-                               }
-                       }
-                       catch (Exception $e) {
-                               // Nothing to do, the default message will do fine
-                       }
-               }
-               return $output;
-       }
-
-       /**
-        * Formats a message for the BE displaying all possible FULLTEXT index to the BE User.
-        *
-        * @return string
-        */
-       protected function getMessageOk() {
-               $fields = $this->analyser->getFields($this->table);
-               $output = '';
-               foreach ($fields as $index => $indexedFields) {
-                       $output .= sprintf(
-                               '%s <strong>fulltext:%s AS foo</strong>',
-                               $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.syntax_for_query'),
-                               $index
-                       );
-                       $output .= sprintf(
-                               '<br/>' . $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.indexed_fields') . '<br/>',
-                               $index,
-                               $indexedFields
-                       );
-                       $output = '<div style="margin-bottom: 10px">' . $output . '</div>';
-               }
-               return '<div>' . $output . '</div>';
-       }
-
-       /**
-        * Formats a message for the BE when no FULLTEXT index is found against a table.
-        *
-        * @return string
-        */
-       protected function getMessageNoIndexFound() {
-               $outputs = array();
-
-               $string = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_index');
-               $outputs[] = sprintf($string, $this->table);
-
-               $tables = $this->analyser->getTables();
-               if (!empty($tables)) {
-                       $listOfTables = implode(', ', $tables);
-                       $string = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.tables_list');
-                       $outputs[] = sprintf($string, $listOfTables);
-               } else {
-                       $outputs[] = $this->language->sL('LLL:EXT:dataquery/locallang_db.xlf:fulltext.no_table_found');
-               }
-               return implode('<br>', $outputs);
-       }
-}
-
-
-if (defined('TYPO3_MODE') && $TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Userfunc/FormEngine.php'])      {
-       include_once($TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Userfunc/FormEngine.php']);
-}
-?>
\ No newline at end of file
index cf3833a..bb7b835 100644 (file)
@@ -1,26 +1,18 @@
 <?php
-/***************************************************************
-*  Copyright notice
-*
-*  (c) 2012-2015 Fabien Udriot (Cobweb) <fudriot@cobweb.ch>
-*  All rights reserved
-*
-*  This script is part of the TYPO3 project. The TYPO3 project is
-*  free software; you can redistribute it and/or modify
-*  it under the terms of the GNU General Public License as published by
-*  the Free Software Foundation; either version 2 of the License, or
-*  (at your option) any later version.
-*
-*  The GNU General Public License can be found at
-*  http://www.gnu.org/copyleft/gpl.html.
-*
-*  This script is distributed in the hope that it will be useful,
-*  but WITHOUT ANY WARRANTY; without even the implied warranty of
-*  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-*  GNU General Public License for more details.
-*
-*  This copyright notice MUST APPEAR in all copies of the script!
-***************************************************************/
+namespace Tesseract\Dataquery\Utility;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
 
 /**
  * This class provides some API methods related to FULLTEXT indexes
@@ -30,9 +22,9 @@
  * @package TYPO3
  * @subpackage dataquery
  */
-class Tx_Dataquery_Utility_DatabaseAnalyser {
+class DatabaseAnalyser {
        /**
-        * @var t3lib_DB
+        * @var \TYPO3\CMS\Core\Database\DatabaseConnection
         */
        protected $databaseHandle;
 
@@ -41,7 +33,7 @@ class Tx_Dataquery_Utility_DatabaseAnalyser {
        /**
         * Constructor
         *
-        * @return Tx_Dataquery_Utility_DatabaseAnalyser
+        * @return DatabaseAnalyser
         */
        public function __construct() {
                $this->databaseHandle = $GLOBALS['TYPO3_DB'];
@@ -126,8 +118,3 @@ class Tx_Dataquery_Utility_DatabaseAnalyser {
                );
        }
 }
-
-if (defined('TYPO3_MODE') && $TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Utility/DatabaseAnalyser.php']) {
-       include_once($TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['dataquery/Classes/Utility/DatabaseAnalyser.php']);
-}
-?>
\ No newline at end of file
diff --git a/Classes/Utility/QueryObject.php b/Classes/Utility/QueryObject.php
new file mode 100644 (file)
index 0000000..49b51ae
--- /dev/null
@@ -0,0 +1,92 @@
+<?php
+namespace Tesseract\Dataquery\Utility;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+/**
+ * Object containing information about the structure of a parsed query.
+ *
+ * NOTE: this object has no method. In some languages, it would be called a
+ * structured type.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class QueryObject {
+       /**
+        * Contains all components of the parsed query
+        * @var array $structure
+        */
+       public $structure = array();
+
+       /**
+        * Name (or alias if defined) of the main query table, i.e. the first one in the FROM part of the query
+        * @var string $mainTable
+        */
+       public $mainTable;
+
+       /**
+        * List of all subtables, i.e. tables in the JOIN statements
+        * @var array $subtables
+        */
+       public $subtables = array();
+
+       /**
+        * The keys to this array are the aliases of the tables used in the query and they point to the true table names
+        * @var array $aliases
+        */
+       public $aliases = array();
+
+       /**
+        * For each table, record with boolean values whether it has some base fields or not
+        * @var array $hasBaseFields
+        */
+       public $hasBaseFields = array();
+
+       /**
+        * Array with all information of the fields used to order data
+        * @var array $orderFields
+        */
+       public $orderFields = array();
+
+       /**
+        * List of aliases for all fields that have one, per table
+        * @var array $fieldAliases
+        */
+       public $fieldAliases = array();
+
+       /**
+        * List of what field aliases map to (table, field and whether it's a function or not)
+        * @var array $fieldAliasMappings
+        */
+       public $fieldAliasMappings = array();
+
+       /**
+        * List of placeholders for replacement by MATCH() statements
+        * @var array
+        */
+       public $fulltextSearchPlaceholders = array();
+
+       public function __construct() {
+                       // Initialize some values
+               $this->structure['DISTINCT'] = FALSE;
+               $this->structure['SELECT'] = array();
+               $this->structure['FROM'] = array();
+               $this->structure['JOIN'] = array();
+               $this->structure['WHERE'] = array();
+               $this->structure['ORDER BY'] = array();
+               $this->structure['GROUP BY'] = array();
+       }
+}
diff --git a/Classes/Utility/SqlUtility.php b/Classes/Utility/SqlUtility.php
new file mode 100644 (file)
index 0000000..d790fd0
--- /dev/null
@@ -0,0 +1,193 @@
+<?php
+namespace Tesseract\Dataquery\Utility;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+
+/**
+ * Class containing some utility SQL methods.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+final class SqlUtility {
+       /**
+        * @var \Tesseract\Dataquery\Parser\FulltextParser Local instance of full text parser utility
+        */
+       static protected $fulltextParser = NULL;
+
+       /**
+        * Transforms a condition transmitted by data-filter to a real SQL segment.
+        *
+        * @throws \Tesseract\Tesseract\Exception\Exception
+        * @param string $field
+        * @param string $table
+        * @param array $conditionData
+        *              + operator: andgroup, orgroup, like, start, fulltext
+        *              + value: the value given as input
+        *              + negate: negate the expression
+        * @return string
+        */
+       static public function conditionToSql($field, $table, $conditionData) {
+               /** @var \TYPO3\CMS\Core\Database\DatabaseConnection $databaseConnection */
+               $databaseConnection = $GLOBALS['TYPO3_DB'];
+
+               $condition = '';
+               // If the value is special value "\all", all values must be taken,
+               // so the condition is simply ignored
+               if ($conditionData['value'] != '\all') {
+                       // Some operators require a bit more handling
+                       // "in" values just need to be put within brackets
+                       if ($conditionData['operator'] == 'in') {
+                               // If the condition value is an array, use it as is
+                               // Otherwise assume a comma-separated list of values and explode it
+                               $conditionParts = $conditionData['value'];
+                               if (!is_array($conditionParts)) {
+                                       $conditionParts = GeneralUtility::trimExplode(',', $conditionData['value'], TRUE);
+                               }
+                               $escapedParts = array();
+                               foreach ($conditionParts as $value) {
+                                       $escapedParts[] = $databaseConnection->fullQuoteStr($value, $table);
+                               }
+                               $condition = $field . (($conditionData['negate']) ? ' NOT' : '') . ' IN (' . implode(',', $escapedParts) . ')';
+
+                       // "andgroup" and "orgroup" require more handling
+                       // The associated value is a list of comma-separated values and each of these values must be handled separately
+                       // Furthermore each value will be tested against a comma-separated list of values too, so the test is not so simple
+                       } elseif ($conditionData['operator'] == 'andgroup' || $conditionData['operator'] == 'orgroup') {
+                               // If the condition value is an array, use it as is
+                               // Otherwise assume a comma-separated list of values and explode it
+                               $values = $conditionData['value'];
+                               if (!is_array($values)) {
+                                       $values = GeneralUtility::trimExplode(',', $conditionData['value'], TRUE);
+                               }
+                               $condition = '';
+                               $localOperator = 'OR';
+                               if ($conditionData['operator'] == 'andgroup') {
+                                       $localOperator = 'AND';
+                               }
+                               foreach ($values as $aValue) {
+                                       if (!empty($condition)) {
+                                               $condition .= ' ' . $localOperator . ' ';
+                                       }
+                                       $condition .= $databaseConnection->listQuery($field, $aValue, $table);
+                               }
+                               if ($conditionData['negate']) {
+                                       $condition = 'NOT (' . $condition . ')';
+                               }
+
+                       // If the operator is "like", "start" or "end", the SQL operator is always LIKE, but different wildcards are used
+                       } elseif ($conditionData['operator'] == 'like' || $conditionData['operator'] == 'start' || $conditionData['operator'] == 'end') {
+                               // Make sure values are an array
+                               $values = $conditionData['value'];
+                               if (!is_array($values)) {
+                                       $values = array($conditionData['value']);
+                               }
+                               // Loop on each value and assemble condition
+                               $condition = '';
+                               foreach ($values as $aValue) {
+                                       $aValue = $databaseConnection->escapeStrForLike($aValue, $table);
+                                       if (!empty($condition)) {
+                                               $condition .= ' OR ';
+                                       }
+                                       if ($conditionData['operator'] == 'start') {
+                                               $value = $aValue . '%';
+                                       } elseif ($conditionData['operator'] == 'end') {
+                                               $value = '%' . $aValue;
+                                       } else {
+                                               $value = '%' . $aValue . '%';
+                                       }
+                                       $condition .= $field . ' LIKE ' . $databaseConnection->fullQuoteStr($value, $table);
+                               }
+                               if ($conditionData['negate']) {
+                                       $condition = 'NOT (' . $condition . ')';
+                               }
+
+                       // Operator "fulltext" requires some special care, as a full MATCH() condition must be assembled
+                       } elseif ($conditionData['operator'] == 'fulltext' || $conditionData['operator'] == 'fulltext_natural') {
+                               $fulltextParser = self::getFulltextParserInstance();
+                               $fulltextParts = explode('.', $field);
+                               $condition = $fulltextParser->parse(
+                                       $table,
+                                       $fulltextParts[2],
+                                       $conditionData['value'],
+                                       ($conditionData['operator'] == 'fulltext_natural'),
+                                       $conditionData['negate']
+                               );
+
+                       // Other operators are handled simply
+                       // We just need to take care of special values: "\empty" and "\null"
+                       } else {
+                               $operator = $conditionData['operator'];
+                               // Make sure values are an array
+                               $values = $conditionData['value'];
+                               if (!is_array($values)) {
+                                       $values = array($conditionData['value']);
+                               }
+                               // Loop on each value and assemble condition
+                               $condition = '';
+                               foreach ($values as $aValue) {
+                                       if (!empty($condition)) {
+                                               $condition .= ' OR ';
+                                       }
+                                       // Special value "\empty" means evaluation against empty string
+                                       if ($conditionData['value'] == '\empty') {
+                                               $quotedValue = "''";
+
+                                       // Special value "\null" means evaluation against IS NULL or IS NOT NULL
+                                       } elseif ($conditionData['value'] == '\null') {
+                                               if ($operator == '=') {
+                                                       $operator = 'IS';
+                                               }
+                                               $quotedValue = 'NULL';
+
+                                       // Normal value
+                                       } else {
+                                               $quotedValue = $databaseConnection->fullQuoteStr($aValue, $table);
+                                       }
+                                       $condition .= $field . ' ' . $operator . ' ' . $quotedValue;
+                               }
+                               if ($conditionData['negate']) {
+                                       $condition = 'NOT (' . $condition . ')';
+                               }
+                       }
+               }
+               return $condition;
+       }
+
+       /**
+        * Returns an instance of Tx_Dataquery_Parser_Fulltext, which is created on demand.
+        *
+        * @return \Tesseract\Dataquery\Parser\FulltextParser
+        */
+       static public function getFulltextParserInstance() {
+               if (self::$fulltextParser === NULL) {
+                       self::$fulltextParser = GeneralUtility::makeInstance('Tesseract\\Dataquery\\Parser\\FulltextParser');
+               }
+               return self::$fulltextParser;
+       }
+
+       /**
+        * Sets the fulltext parser instance.
+        *
+        * This is used for unit tests.
+        *
+        * @param \Tesseract\Dataquery\Parser\FulltextParser $fulltextParser
+        */
+       static public function setFulltextParserInstance($fulltextParser) {
+               self::$fulltextParser = $fulltextParser;
+       }
+}
diff --git a/Classes/Wizard/QueryCheckWizard.php b/Classes/Wizard/QueryCheckWizard.php
new file mode 100644 (file)
index 0000000..e99014e
--- /dev/null
@@ -0,0 +1,68 @@
+<?php
+namespace Tesseract\Dataquery\Wizard;
+
+/*
+ * This file is part of the TYPO3 CMS project.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ *
+ * For the full copyright and license information, please read the
+ * LICENSE.txt file that was distributed with this source code.
+ *
+ * The TYPO3 project - inspiring people to share!
+ */
+
+use TYPO3\CMS\Backend\Form\FormEngine;
+use TYPO3\CMS\Core\Utility\ExtensionManagementUtility;
+
+/**
+ * Wizard for checking the validity and the results of a SQL query.
+ *
+ * @author Francois Suter (Cobweb) <typo3@cobweb.ch>
+ * @package TYPO3
+ * @subpackage tx_dataquery
+ */
+class QueryCheckWizard {
+
+       /**
+        * Renders the wizard itself.
+        *
+        * @param array $fieldParameters Parameters of the field
+        * @param FormEngine $formObject Calling object
+        * @return string HTML for the wizard
+        */
+       public function render($fieldParameters, FormEngine $formObject) {
+               // Get the id attribute of the field tag
+               preg_match('/id="(.+?)"/', $fieldParameters['item'], $matches);
+
+               /** @var \TYPO3\CMS\Core\Page\PageRenderer $pageRenderer */
+               $pageRenderer = $GLOBALS['SOBE']->doc->getPageRenderer();
+               // Add specific CSS
+               $pageRenderer->addCssFile(
+                       ExtensionManagementUtility::extRelPath('dataquery') . 'Resources/Public/Styles/CheckWizard.css'
+               );
+               // Load the necessary JavaScript
+               $pageRenderer->addJsFile(
+                       ExtensionManagementUtility::extRelPath('dataquery') . 'Resources/Public/JavaScript/CheckWizard.js'
+               );
+               // Load some localized labels, plus the field's id
+               $pageRenderer->addJsInlineCode(
+                       'tx_dataquery_wizard',
+                       'var TX_DATAQUERY = {
+                               fieldId : "' . $matches[1] . '",
+                               labels : {
+                                       "debugTab" : "' . $GLOBALS['LANG']->sL('LLL:EXT:dataquery/locallang.xml:wizard.check.debugTab') . '",
+                                       "previewTab" : "' . $GLOBALS['LANG']->sL('LLL:EXT:dataquery/locallang.xml:wizard.check.previewTab') . '",
+                                       "validateButton" : "' . $GLOBALS['LANG']->sL('LLL:EXT:dataquery/locallang.xml:wizard.check.validateButton') . '"
+                               }
+                       };'
+               );
+               // First of all render the button that will show/hide the rest of the wizard
+               $wizard = '';
+               // Assemble the base HTML for the wizard
+               $wizard .= '<div id="tx_dataquery_wizardContainer"></div>';
+               return $wizard;
+       }
+}
diff --git a/Configuration/TCA/Overrides/tx_datafilter_filters.php b/Configuration/TCA/Overrides/tx_datafilter_filters.php
new file mode 100644 (file)
index 0000000..a2dc2db
--- /dev/null
@@ -0,0 +1,23 @@
+<?php
+if (!defined ('TYPO3_MODE'))   die ('Access denied.');
+
+// Add SQL field to datafilter
+$tempColumns = array(
+       'tx_dataquery_sql' => array(
+               'exclude' => TRUE,
+               'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_datafilter_filters.tx_dataquery_sql',
+               'config' => array(
+                       'type' => 'text',
+                       'cols' => '30',
+                       'rows' => '8',
+               )
+       )
+);
+\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
+       'tx_datafilter_filters',
+       $tempColumns
+);
+\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(
+       'tx_datafilter_filters',
+       '--div--;LLL:EXT:dataquery/locallang_db.xml:sql, tx_dataquery_sql'
+);
diff --git a/Configuration/TCA/tx_dataquery_queries.php b/Configuration/TCA/tx_dataquery_queries.php
new file mode 100644 (file)
index 0000000..e0de765
--- /dev/null
@@ -0,0 +1,173 @@
+<?php
+if (!defined ('TYPO3_MODE'))   die ('Access denied.');
+
+return array(
+       'ctrl' => array (
+               'title'     => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries',
+               'label'     => 'title',
+               'tstamp'    => 'tstamp',
+               'crdate'    => 'crdate',
+               'cruser_id' => 'cruser_id',
+               'versioningWS' => TRUE,
+               'origUid' => 't3_origuid',
+               'default_sortby' => 'ORDER BY title',
+               'delete' => 'deleted',
+               'enablecolumns' => array (
+                       'disabled' => 'hidden',
+               ),
+               'searchFields' => 'title,description,sql_query',
+               'typeicon_classes' => array(
+                       'default' => 'extensions-dataquery-query'
+               ),
+               'dividers2tabs' => 1,
+       ),
+       'interface' => array(
+               'showRecordFieldList' => 'hidden,title,description,sql_query,t3_mechanisms'
+       ),
+       'columns' => array(
+               't3ver_label' => array(
+                       'label'  => 'LLL:EXT:lang/locallang_general.xml:LGL.versionLabel',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => '30',
+                               'max'  => '30',
+                       )
+               ),
+               'hidden' => array(
+                       'exclude' => 1,
+                       'label'   => 'LLL:EXT:lang/locallang_general.xml:LGL.hidden',
+                       'config'  => array(
+                               'type'    => 'check',
+                               'default' => '0'
+                       )
+               ),
+               'title' => array(
+                       'exclude' => 0,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.title',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => '30',
+                               'eval' => 'required,trim',
+                       )
+               ),
+               'description' => array(
+                       'exclude' => 0,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.description',
+                       'config' => array(
+                               'type' => 'text',
+                               'cols' => '30',
+                               'rows' => '4',
+                       )
+               ),
+               'sql_query' => array(
+                       'exclude' => 0,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.sql_query',
+                       'config' => array(
+                               'type' => 'text',
+                               'cols' => '30',
+                               'rows' => '8',
+                               'wizards' => array(
+                                       '_PADDING' => 2,
+                                       'check' => array(
+                                               'type' => 'userFunc',
+                                               'userFunc' => 'Tesseract\Dataquery\Wizard\QueryCheckWizard->render'
+                                       )
+                               )
+                       )
+               ),
+               'fulltext_indices' => array(
+                       'exclude' => 0,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xlf:tx_dataquery_queries.fulltext_indices',
+                       'config' => array(
+                               'type' => 'user',
+                               'userFunc' => 'Tesseract\Dataquery\UserFunction\FormEngine->renderFulltextIndices',
+                       )
+               ),
+               'cache_duration' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.cache_duration',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 20,
+                               'default' => 86400,
+                               'eval' => 'int',
+                       )
+               ),
+               'ignore_enable_fields' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_enable_fields',
+                       'config' => array(
+                               'type' => 'radio',
+                               'default' => 0,
+                               'items' => array(
+                                       array('LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_enable_fields.I.0', '0'), # don't ignore
+                                       array('LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_enable_fields.I.1', '1'), # ignore everything
+                                       array('LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_enable_fields.I.2', '2'), # ignore partially
+                               ),
+                       )
+               ),
+               'ignore_time_for_tables' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_time_for_tables',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 255,
+                               'default' => '*',
+                       )
+               ),
+               'ignore_disabled_for_tables' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_disabled_for_tables',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 255,
+                               'default' => '*',
+                       )
+               ),
+               'ignore_fegroup_for_tables' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_fegroup_for_tables',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 255,
+                               'default' => '*',
+                       )
+               ),
+               'ignore_language_handling' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_language_handling',
+                       'config' => array(
+                               'type' => 'check',
+                               'default' => 0,
+                               'items' => array(
+                                       array('LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.ignore_language_handling.I.0', ''),
+                               ),
+                       )
+               ),
+               'skip_overlays_for_tables' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.skip_overlays_for_tables',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 255,
+                       )
+               ),
+               'get_versions_directly' => array(
+                       'exclude' => 1,
+                       'label' => 'LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.get_versions_directly',
+                       'config' => array(
+                               'type' => 'input',
+                               'size' => 255,
+                       )
+               ),
+       ),
+       'types' => array(
+               '0' => array('showitem' => 'hidden;;;;1-1-1, title;;1;;2-2-2, sql_query;;;;3-3-3, fulltext_indices,
+                                                                       --div--;LLL:EXT:dataquery/locallang_db.xml:tx_dataquery_queries.tab.advanced, cache_duration;;;;1-1-1, ignore_enable_fields;;2;;2-2-2 , ignore_language_handling;;3;;3-3-3, get_versions_directly')
+       ),
+       'palettes' => array(
+               '1' => array('showitem' => 'description'),
+               '2' => array('showitem' => 'ignore_time_for_tables, --linebreak--, ignore_disabled_for_tables, --linebreak--, ignore_fegroup_for_tables'),
+               '3' => array('showitem' => 'skip_overlays_for_tables')
+       )
+);
index d35112a..4940b84 100644 (file)
@@ -20,4 +20,5 @@
 .settings
 .TemporaryItems
 .webprj
+_make
 nbproject
index 7323fe9..9fcd44e 100644 (file)
@@ -18,10 +18,10 @@ query that appears in the introductory screenshot:
 
 .. code-block:: sql
 
-   SELECT uid, title, COUNT(children.uid) AS pages.children FROM pages
-   LEFT JOIN pages AS children ON children.pid = pages.uid
-   WHERE children.uid IS NOT NULL AND pages.pid = 1
-   ORDER BY pages.title ASC GROUP BY pages.uid
+       SELECT uid, title, COUNT(children.uid) AS pages.children FROM pages
+       LEFT JOIN pages AS children ON children.pid = pages.uid
+       WHERE children.uid IS NOT NULL AND pages.pid = 1
+       ORDER BY pages.title ASC GROUP BY pages.uid
 
 If you try to execute it as is you will get several SQL errors. It is
 indeed not correct, but will be by the time Data Query has rewritten
index 3a3477e..b459377 100644 (file)
@@ -20,7 +20,7 @@ Data Query and change it dynamically via the filters.
 
 Technically the filter structure created by the Data Filter is passed
 to the Data Query by the controller using the
-:code:`tx\_tesseract\_dataprovider::setDataFilter()` method from the
+:code:`\Tesseract\Tesseract\Service\ConsumerBase::setDataFilter::setDataFilter()` method from the
 base provider interface. The filter structure is then translated into
 SQL by Data Query and added to the base query from the "SQL query"
 field.
@@ -38,26 +38,26 @@ query:
 
 .. code-block:: sql
 
-   SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content
+       SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content
 
 with the following Data Filter:
 
-.. code-block:: sql
+.. code-block:: text
 
-   year = date:Y
+       year = date:Y
 
 (which would select all content element edited during the current
 year). This will be (correctly) interpreted as:
 
 .. code-block:: sql
 
-   SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content WHERE (FROM_UNIXTIME(tstamp, '%Y') = 2010)
+       SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content WHERE (FROM_UNIXTIME(tstamp, '%Y') = 2010)
 
 (assuming the current year is 2010), instead of:
 
 .. code-block:: sql
 
-   SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content WHERE (year = 2010)
+       SELECT FROM_UNIXTIME(tstamp, '%Y') AS year FROM tt_content WHERE (year = 2010)
 
 which would cause a SQL syntax error.
 
@@ -71,14 +71,14 @@ Imagine setting up a group of checkboxes like:
 
 .. code-block:: html
 
-   <input type="checkbox" name="tx_myext[foo][]" value="bob" />
-   <input type="checkbox" name="tx_myext[foo][]" value="alice" />
+       <input type="checkbox" name="tx_myext[foo][]" value="bob" />
+       <input type="checkbox" name="tx_myext[foo][]" value="alice" />
 
 Next imagine a filter like:
 
 .. code-block:: text
 
-   fe_users.name like gp:tx_myext|foo
+       fe_users.name like gp:tx_myext|foo
 
 The value returned will be an array. This is handled by Data Query by
 creating a LIKE condition for each value and concatenating all these
@@ -88,7 +88,7 @@ checked):
 
 .. code-block:: sql
 
-   (fe_users.name LIKE '%bob%' OR fe_users.name LIKE '%alice%')
+       (fe_users.name LIKE '%bob%' OR fe_users.name LIKE '%alice%')
 
 It's not possible to change the logical operator to "AND" (this didn't
 seem useful after thinking quite a bit about it; the whole reasoning
index 1007230..2b280d0 100644 (file)
@@ -12,6 +12,5 @@ Data Query output
 ^^^^^^^^^^^^^^^^^
 
 "dataquery" returns a standardized data structure of type "recordset".
-Refer to the `"tesseract" manual <http://docs.typo3.org/typo3cms/extensions/tesseract/>`_
+Refer to the :ref:`"tesseract" manual <tesseract:start>`
 for more details on such data structure.
-
index e3f983f..58600d1 100644 (file)
@@ -30,19 +30,19 @@ postProcessDataStructureBeforeCache
   cache)
 
 Both hooks receive as arguments the full Data Structure as well as a
-back-reference to the calling :code:`tx\_dataquery\_wrapper` object.
+back-reference to the calling :code:`\Tesseract\Dataquery\Component\DataProvider` object.
 They are expected to return a complete Data Structure even if they did
 not perform any change.
 
 Skeleton code for both hooks can be found in
-:code:`samples/class.tx\_dataquery\_sample\_hook.php`.
+:file:`Classes/Sample/DataQueryHook.php`.
 
 Another hook is available for manipulating the tables and fields
 information:
 
 postProcessFieldInformation
   This hook is inside
-  :code:`tx\_dataquery\_wrapper::getTablesAndFields()`, a method which
+  :code:`\Tesseract\Dataquery\Component\DataProvider::getTablesAndFields()`, a method which
   is called when "dataquery" provides Data Consumers with a list of
   available tables and fields while working within the TYPO3 backend
   (this is how, for example, "templatedisplay" knows which fields to
@@ -57,11 +57,11 @@ manipulating the parameters used to calculate the hash:
 
 processCacheHashParameters
   This hook is called inside
-  :code:`tx\_dataquery\_wrapper::calculateCacheHash()`. It receives as
+  :code:`\Tesseract\Dataquery\Component\DataProvider::calculateCacheHash()`. It receives as
   arguments the current cache parameters (an associative array) and a
   back-reference to the calling object (an instance of
-  :code:`tx\_dataquery\_wrapper`). It is expected to return the full
+  :code:`\Tesseract\Dataquery\Component\DataProvider`). It is expected to return the full
   array of cache parameters, whether it modified them or not. Classes
   using this hook must implement interface
-  :code:`tx\_dataquery\_cacheParametersProcessor`.
+  :code:`\Tesseract\Dataquery\Cache\CacheParametersProcessorInterface`.
 
index 81b88d9..e956dea 100644 (file)
@@ -40,8 +40,8 @@ Questions?
 ^^^^^^^^^^
 
 If you have any questions about this extension, you may want to refer
-to the Tesseract Project web site (`http://www.typo3-tesseract.com/
-<http://www.typo3-tesseract.com/>`_) for support and tutorials. You
+to the Tesseract Project web site http://www.typo3-tesseract.com/
+for support and tutorials. You
 may also ask questions in the TYPO3 English mailing list
 (typo3.english).
 
index c3074da..7f179e5 100644 (file)
@@ -12,7 +12,7 @@ Comments
 ^^^^^^^^
 
 It is possible to comment lines in the query by starting them with a
-"#" or "//" marker. Example:
+:code:`#` or :code:`//` marker. Example:
 
 .. code-block:: sql
 
index f721af5..f52ceb3 100644 (file)
@@ -11,7 +11,7 @@
 Expressions in queries
 ^^^^^^^^^^^^^^^^^^^^^^
 
-It is possible to use `expressions <http://typo3.org/extensions/repository/view/expressions>`_
+It is possible to use :ref:`expressions <expressions:start>`
 inside a query. Consider the following:
 
 .. code-block:: sql
@@ -29,5 +29,4 @@ be used in the WHERE clause. Instead Data Filters should be used in
 this case. This is not a limitation per se, just a best practice.
 
 For more information on expressions, please refer to the
-`extension manual <http://docs.typo3.org/typo3cms/extensions/expressions/>`_.
-
+:ref:`extension manual <expressions:start>`.