1. 28 Nov, 2021 2 commits
    • Christian Kuhn's avatar
      [!!!][TASK] Remove QueryView and QueryGenerator in ext:core · 784729b8
      Christian Kuhn authored and Benni Mack's avatar Benni Mack committed
      Change-Id: I417f0f896e07b05d4577c865195724955fa7bc45
      Resolves: #96113
      Related: #92080
      Releases: master
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72334
      
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      784729b8
    • Sybille Peters's avatar
      [BUGFIX] Handle large number of pages in linkvalidator · 7fecd98c
      Sybille Peters authored and Christian Kuhn's avatar Christian Kuhn committed
      This patch handles problems which occured with a large number of pages
      and also for a large number of records to be checked for broken links.
      The problems previously resulted in "Prepared statement contains too
      many placeholders" exceptions and consumption of too much RAM and
      (possibly) out of memory errors.
      
      Technical details about the implementation follows because this is
      handled slightly differently than in other areas of the core.
      
      LinkValidator used named placeholders to build database queries,
      and the EditableRestriction class also used a number of placeholder.
      
      Technically 'doctrine/dbal' explodes named placeholders for array
      values with quote type 'PARAM_INT_ARRAY' or 'PARAM_STR_ARRAY' to
      single unnamed placeholders for each value, using one placeholder
      for each array value. If multiple array value placeholder are used
      with a large number of values this could quickly emit the "Prepared
      statement contains too many placeholders" error as the limit is
      reached. The limit differs beetween the various database systems.
      One way to work around this issue is to do database operations in
      chunks, but it could become quite difficult to calculate the chunk
      size beforehand.
      
      Array values are mainly used for 'in()' or 'notIn()' query expressions,
      which naturally do not not have direct limits for the value count in it
      through all supported dbms systems. This opens the possibility to use
      direct value lists for these expressions. There are other dbms
      limitations depending on the used dbms (for example on mariadb/mysql
      'max_allowed_packets', max query size setting for SQLite), which may be
      adjusted in some cases. However we could not safely expect that these
      values are monitored and adjusted, so we still need to do database
      operations in chunks, but make it a little easier for 'chunksize'
      calculations.
      
      During testing on an instance with a large number of pages, it was
      found that LinkValidator used up a lot of memory, reaching the
      configured memory limit of 2G on that system, which itself was very
      high already (and far beyond the recommended 256MB limit).
      Instead of collecting unlimited results in an array before checking
      links against this array, doing this for each record directly avoids
      excessive memory consumption during link checking.
      
      It was decided to do this in this patch directly instead of a separate
      patch, as this patch otherwise would not be testable in real
      instances.
      
      This patch does the following:
      
      * introduce two method to implode value arrays in QueryHelper
      * replace placeholder in EditableRestriction with quoted values
      * replace placeholder in BrokenLinksRepository with quoted values
      * introduce chunk database operations in BrokenLinkRepository
      * introduce chunk database operations in LinkAnalyzer
      * move $results array and call checkLinks() on it to the inner
        loop to process this for each record (instead of accumulating the
        results in a huge array.
      
      In the future we need to investigate and find a more generic way to
      handle such scenarios for larger instances. This may take some time
      and investigation to do this properly and in a reusable way. To not
      stall the bugfix further, we are doing the workarounds in this patch
      to solve these issues in the meantime.
      
      Resolves: #92493
      Releases: master, 11.5
      Change-Id: I9a596df6749d07afee9c5ac37266b876ddc6e297
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/66896
      
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      7fecd98c
  2. 27 Nov, 2021 21 commits
  3. 26 Nov, 2021 8 commits
  4. 25 Nov, 2021 7 commits
  5. 24 Nov, 2021 2 commits
    • Oliver Bartsch's avatar
      [BUGFIX] Make selectCheckBox work with readOnly · 8a455d39
      Oliver Bartsch authored
      Using "readOnly" for a TCA field with renderType
      "selectCheckBox" previously led to the fact, that
      nothing was rendered at all. This can be seen with
      `tx_styleguide_elements_select.select_checkbox_4`.
      
      This is now fixed, while the whole element got a
      light code cleanup. This will increase performance
      a bit, since unneeded JavaScript is no longer loaded.
      
      Additionally, this also fixes the "expandAll" option,
      which was broken due to the bootstrap 5 upgrade.
      
      Resolves: #96058
      Resolves: #96068
      Releases: master, 11.5
      Change-Id: I762d328c9f4c83558dd8fd98683f8ccfe6c4b3f0
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72279
      
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Jochen's avatarJochen <rothjochen@gmail.com>
      Tested-by: Nikita Hovratov's avatarNikita Hovratov <nikita.h@live.de>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Jochen's avatarJochen <rothjochen@gmail.com>
      Reviewed-by: Nikita Hovratov's avatarNikita Hovratov <nikita.h@live.de>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      8a455d39
    • Christian Kuhn's avatar
      [BUGFIX] Better sys_refindex with workspace mm · 3801adce
      Christian Kuhn authored
      This fixes issues regarding sys_refindex handling when dealing
      with workspace mm relations. Various DataHandler tests run with
      disabled refindex check after performing mm related operations.
      All those are enabled now, so all known issues in this area
      are fixed.
      
      The patch touches various areas in DataHandler, ReferenceIndex
      and RelationHandler to achieve this.
      
      The key changes are triggered by a database scenario that is
      unique for mm relations. Usually, when creating a workspace record
      or overlay, child record overlays (for instance inline children)
      are also created in this workspace.
      
      For mm records, this can be different: When the local or foreign
      side of a record is created in a workspace, it does not necessarily
      mean that a connected opposite side is also created as workspace
      overlay.
      
      This would otherwise create the need for a recursive operation that
      would basically end up with "everything" being created as workspace
      overlay. However, mm table entries are created for these relations,
      leading to a situation that a workspace record can point to a
      non-workspace record through the mm table.
      
      Reference index entries for mm table connections are always handled
      from the 'local' side. For instance, with categories, 'sys_category'
      table is the local side, with 'pages', 'tt_content' and others
      being the foreign side. If now one of the records on the foreign
      side gets a workspace overlay record, sys_refindex rows of all
      connected records of the affected local side need to be created.
      This can lead to the funny situation that we end up with refindex
      rows that point from a non-workspace to a non-workspace record,
      but have a non-0 workspace entry. Those additional rows are needed
      to have "the full set" and proper sorting when looking at refindex
      of such a relation from the local side.
      
      The patch basically handles especially these 'foreign side has
      a workspace overlay' scenarios, plus the side effects that
      have to be taken care of when discarding and publishing these
      records.
      
      Additionally, a couple of side effects are tackled: First, the
      ReferenceIndex->updateIndex() - that's the main logic when
      running cli referenceindex:update command - is tuned to drop
      reference index entries related to meanwhile deleted workspaces.
      This is covered with additional tests and is done now since
      the code needs to iterate over existing workspaces for local-side
      mm records that may have workspace overlays on the foreign side.
      So most of the code had to be created now anyways.
      
      RelationHandler->readMM() receives a fix for a long standing bug
      (Issue #83582, introduced by #57169), to no longer show workspace
      relations in live when looking at local-side records, for instance
      when looking at category items in live when one of the connected
      items has a workspace overlay.
      
      Next to lots of comments explaining details in place and
      referencing the test cases that nail specific scenarios, a
      couple of @todo's are added to point out things we may
      want to tackle in the future.
      
      All in all, the patch resolves a series of issues that will
      especially lead to 'categories' being much more reliable in
      workspaces.
      
      Change-Id: I24407f93665852fa761f6fbe6c5ab249473468d2
      Related: #57169
      Resolves: #83582
      Resolves: #96067
      Releases: master, 11.5
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72250
      
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Tested-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      3801adce