1. 02 Dec, 2021 1 commit
  2. 24 Nov, 2021 1 commit
    • Christian Kuhn's avatar
      [BUGFIX] Better sys_refindex with workspace mm · 3801adce
      Christian Kuhn authored
      This fixes issues regarding sys_refindex handling when dealing
      with workspace mm relations. Various DataHandler tests run with
      disabled refindex check after performing mm related operations.
      All those are enabled now, so all known issues in this area
      are fixed.
      
      The patch touches various areas in DataHandler, ReferenceIndex
      and RelationHandler to achieve this.
      
      The key changes are triggered by a database scenario that is
      unique for mm relations. Usually, when creating a workspace record
      or overlay, child record overlays (for instance inline children)
      are also created in this workspace.
      
      For mm records, this can be different: When the local or foreign
      side of a record is created in a workspace, it does not necessarily
      mean that a connected opposite side is also created as workspace
      overlay.
      
      This would otherwise create the need for a recursive operation that
      would basically end up with "everything" being created as workspace
      overlay. However, mm table entries are created for these relations,
      leading to a situation that a workspace record can point to a
      non-workspace record through the mm table.
      
      Reference index entries for mm table connections are always handled
      from the 'local' side. For instance, with categories, 'sys_category'
      table is the local side, with 'pages', 'tt_content' and others
      being the foreign side. If now one of the records on the foreign
      side gets a workspace overlay record, sys_refindex rows of all
      connected records of the affected local side need to be created.
      This can lead to the funny situation that we end up with refindex
      rows that point from a non-workspace to a non-workspace record,
      but have a non-0 workspace entry. Those additional rows are needed
      to have "the full set" and proper sorting when looking at refindex
      of such a relation from the local side.
      
      The patch basically handles especially these 'foreign side has
      a workspace overlay' scenarios, plus the side effects that
      have to be taken care of when discarding and publishing these
      records.
      
      Additionally, a couple of side effects are tackled: First, the
      ReferenceIndex->updateIndex() - that's the main logic when
      running cli referenceindex:update command - is tuned to drop
      reference index entries related to meanwhile deleted workspaces.
      This is covered with additional tests and is done now since
      the code needs to iterate over existing workspaces for local-side
      mm records that may have workspace overlays on the foreign side.
      So most of the code had to be created now anyways.
      
      RelationHandler->readMM() receives a fix for a long standing bug
      (Issue #83582, introduced by #57169), to no longer show workspace
      relations in live when looking at local-side records, for instance
      when looking at category items in live when one of the connected
      items has a workspace overlay.
      
      Next to lots of comments explaining details in place and
      referencing the test cases that nail specific scenarios, a
      couple of @todo's are added to point out things we may
      want to tackle in the future.
      
      All in all, the patch resolves a series of issues that will
      especially lead to 'categories' being much more reliable in
      workspaces.
      
      Change-Id: I24407f93665852fa761f6fbe6c5ab249473468d2
      Related: #57169
      Resolves: #83582
      Resolves: #96067
      Releases: master, 11.5
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72250
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Tested-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      3801adce
  3. 12 Oct, 2021 1 commit
  4. 30 Sep, 2021 1 commit
  5. 24 Sep, 2021 1 commit
  6. 17 Sep, 2021 1 commit
  7. 12 Aug, 2021 1 commit
    • Nikita Hovratov's avatar
      [FEATURE] Register SoftReference parsers via DI · 48810cb7
      Nikita Hovratov authored and Benni Mack's avatar Benni Mack committed
      The concept for registration and usage of soft reference parsers
      received a complete overhaul.
      
      Starting with the registration, it is now possible to register soft
      reference parsers by dependency injection in the extension's
      Services.(yaml|php) file. For this, the new tag name
      "softreference.parser" has been introduced. One has to provide the
      additional attribute "parserKey" to identify the parser. This
      replaces the old way of registering these parsers in the $GLOBALS
      array. If a parser is registered with the same key in both ways,
      the old way takes precedence for b/w compatibility.
      
      This comes with a completely new factory service class
      TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserFactory.
      This classes' responsibilities are collecting all registered soft
      reference parsers and serving them to the consumer by calling the
      method "getSoftReferenceParser" with the desired parser key as the
      only argument. There is a compatibility layer for the old way of
      registration and for classes not implementing the new interface.
      
      Soft reference parsers now have to implement
      TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserInterface.
      The interface defines the implementation of the "parse" method.
      The first 4 and the last parameter stay the same as in the old method
      "findRef". The remaining two parameters "spKey" and "spParams" have to
      be set with the "setParserKey" method, if they are needed. The key can
      be retrieved by using the "getParserKey" method.
      
      The different parser implementations in the old class
      TYPO3\CMS\Core\Database\SoftReferenceIndex have been extracted and
      moved into dedicated classes in the
      TYPO3\CMS\Core\DataHandling\SoftReference namespace. Missing tests
      for parsers other than "typolink" and "typolink_tag" are added.
      
      The method makeTokenID of SoftReferenceIndex has been moved into
      TYPO3\CMS\Core\DataHandling\SoftReference\AbstractSoftReferenceParser.
      A parser can extend this abstract class, if this method is needed.
      
      The methods of BackendUtility "softRefParserObj" and
      "explodeSoftRefParserList" are now deprecated and the replacement
      TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserFactory
      should be used instead.
      
      Resolves: #94687
      Resolves: #94741
      Releases: master
      Change-Id: I460bfdd4478194fa4b4111fc44871f7225c6c084
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/70177
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      48810cb7
  8. 02 Aug, 2021 1 commit
    • Oliver Bartsch's avatar
      [FEATURE] Introduce TCA type "category" · 1e7653ce
      Oliver Bartsch authored and Benni Mack's avatar Benni Mack committed
      A new TCA type "category" is introduced, which
      allows to simplify the configuration of category
      TCA columns. Besides the benefit for integrators,
      this allows to deprecate the CategoryRegistry
      in the next step.
      
      The new type can also be used for other use cases.
      Therefore, the TCA option "relationship" is available
      for this TCA type. Besides "manyToMany" (default), this
      can also be set to "oneToOne" or "oneToMany".
      
      Using the new type, FormEngine will always render a
      category tree. This means, no additional `renderType`
      is defined. In such case, TCA type "select" can
      still be used as before, without any limitation. However,
      all relevant places in core are adjusted in this patch.
      
      The category element is rendered through a custom element
      (web component), reducing inline javascript.
      
      Resolves: #94622
      Releases: master
      Change-Id: I1b95c42288b070fa6bac114266f5ad246a045b21
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/69899
      
      Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      1e7653ce
  9. 29 Jul, 2021 1 commit
  10. 22 Jul, 2021 1 commit
  11. 08 Jul, 2021 1 commit
  12. 16 Jun, 2021 1 commit
  13. 14 May, 2021 1 commit
  14. 08 Mar, 2021 1 commit
  15. 02 Mar, 2021 1 commit
  16. 22 Jan, 2021 1 commit
  17. 12 Jan, 2021 1 commit
  18. 14 Dec, 2020 2 commits
    • Christian Kuhn's avatar
      [BUGFIX] More deterministic mm sorting · ff1a76ed
      Christian Kuhn authored
      A 'true' MM relation (select / group with MM, but not inline
      with foreign_field - used in core for pages/tt_content table
      to sys_category via sys_category_record_mm) has two sorting
      fields: 'sorting' for the local (sys_category) side, and
      'sorting_foreign' for the foreign (pages / tt_content) side.
      
      When categories are added to a tt_content element,
      'sorting_foreign' is set to the given order of sys_category
      elements. 'sorting' is set to 0. When then opening a
      sys_category record that has multiple records pointing to
      it with 0 as sorting, the returned order of records is
      non-deterministic and depends on implicit DB engine fallbacks.
      
      This also confuses the reference index, which checks
      refindex integrity always from the 'local' side.
      
      The patch adds 'uid_foreign' as second order-by field to
      force deterministic ordering. There is still a possible
      collision in multi table relations (more than one foreign
      table uses the mm table like pages AND tt_content to
      sys_category with foreign records having the same uid) and
      if the mm table allows "multi" relations. Those scenarios
      however have no explicit TCA configuration (only implicit
      via MM_match_fields), need a deeper investigation and
      possibly further detail patches later.
      
      The patch should for now fix a functional test that is
      flaky with postgres.
      
      Change-Id: I3c89d0e67f8a4065354f9df173020ca0080e0d57
      Resolves: #93075
      Releases: master, 10.4
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67110
      
      Reviewed-by: Andreas Fernandez's avatarAndreas Fernandez <a.fernandez@scripting-base.de>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Tested-by: default avatarTYPO3com <noreply@typo3.com>
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      ff1a76ed
    • Benni Mack's avatar
      [BUGFIX] DBAL: Do not use deprecated classes · b2a22815
      Benni Mack authored and Christian Kuhn's avatar Christian Kuhn committed
      In preparation for Doctrine DBAL 3.0,
      
      1. all usages of Doctrine\DBAL\DBALException have
      been migrated to Doctrine\DBAL\Exception,
      because DBAL Exception does not exist in
      Doctrine 3.0 anymore.
      
      2. Doctrine\DBAL\Platforms\PostgreSqlPlatform
      has been migrated to Doctrine\DBAL\Platforms\PostgreSQL94Platform
      because this class does not exist anymore in Doctrine DBAL 3.0,
      same goes for Doctrine\DBAL\Platforms\SQLServerPlatform
      which has been replaced by Doctrine\DBAL\Platforms\SQLServer2012Platform
      
      3. Doctrine\DBAL\Driver\PDOException has been
      renamed to Doctrine\DBAL\Driver\PDO\Exception
      
      4. Doctrine\DBAL\Driver\PDOStatement has been
      renamed to Doctrine\DBAL\Driver\PDO\Statement
      
      Resolves: #93071
      Releases: master
      Change-Id: I05e82f2fca09eb7718a90c09f95e503980ae10ae
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67109
      
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Tested-by: default avatarTYPO3com <noreply@typo3.com>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      b2a22815
  19. 09 Dec, 2020 1 commit
  20. 08 Dec, 2020 1 commit
  21. 20 Nov, 2020 1 commit
  22. 12 Nov, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Do not create new version placeholders in workspaces anymore · 74899ecf
      Benni Mack authored
      Creating a new record in a workspace adds two database rows.
      
      One that is the "placeholder", which - since v10.4 - contains
      the same metadata as the other record:
      
      * t3ver_wsid = workspaceID
      * t3ver_oid = 0 (simulating behavior of an "online pendant record")
      * t3ver_state = 1
      
      And the "versionized" record, identified by:
      
      * t3ver_wsid = workspaceID
      * t3ver_oid = uid of the new placeholder record
      * t3ver_state = -1
      
      As of TYPO3 v10, the first record is not needed anymore,
      the versioned record can be queried directly, however, since
      the relations (except MM) point to the placeholder record,
      this one is kept.
      
      As result, only one record is created from now on:
      
      * t3ver_wsid = workspaceID
      * t3ver_oid = 0 (no online counterpart)
      * t3ver_state = 1
      
      On reading, the record is queried directly (no overlay needed anymore!)
      with the existing Database Doctrine Restrictions. On publishing, the
      record just gets the state/stage/wsid set and is "live".
      
      This brings fundamental benefits:
      
      * No overlays needed when querying
      * Fewer database records (placeholders are not helpful)
      * Conceptual problems with placeholder and shadowed fields are removed
      
      Resolves: #92791
      Releases: master
      Change-Id: I0288cc63fe72d8442d586f309bd4054ac44e829b
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/65587
      
      Tested-by: default avatarTYPO3com <noreply@typo3.com>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      74899ecf
  23. 29 Oct, 2020 1 commit
  24. 12 Oct, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Remove move placeholders · 27c7de8a
      Benni Mack authored and Christian Kuhn's avatar Christian Kuhn committed
      Workspaces ("Element-based versioning") previously had - due to
      the "pid=-1" logic until TYPO3 v10 - a so-called "MOVE PLACEHOLDER".
      
      This was indicated by t3ver_state = 3, all relevant fields:
      * t3ver_state = 3 (move placeholder)
      * t3ver_oid = 0 no connected live record, it allowed fetching these records
        with one query together with live records as db restrictions t3ver_oid > 0
      * t3ver_wsid = workspace UID
      * t3ver_move_id = UID of the live record
      * pid = new PID the version was moved to
      * sorting - when a record was moved within page with activated sorting
      
      Other record fields were not important. However, when moving a record, the
      value from TCA ctrl section "shadowColumnsForMovePlaceholders" was used to
      fill in gaps from the live record.
      
      The ACTUAL versioned record was indicated by t3ver_state = 4, the so-called
      "MOVE POINTER". In previous version until TYPO3 v10, it's PID field was set
      to -1, but since TYPO3 v10, it has the same PID as the "MOVE PLACEHOLDER".
      
      Characteristics of the move pointer as of TYPO3 v10:
      * t3ver_state = 4 (move pointer)
      * t3ver_oid = UID of the live record
      * t3ver_wsid = workspace UID
      * t3ver_move_id = 0
      * pid = PID the version was moved to
      * sorting - same value as the live record (not evaluated until now)
      * All other fields with optionally modified content
      
      Both move placeholder and move pointer did not know each other directly.
      Fetching the move pointer for a move placeheldor (or the other way around)
      involved the live record, leading to many queries.
      
      The patch obsoletes the move placeholder records, moving necessary
      information to the move pointer: It now contains the updated sorting
      and is considered in the Database Restrictions to be fetched.
      
      In general, when publishing, the moved record now
      behaves identical to the other versioned types.
      
      This makes the internal code much easier, creates less DB queries
      on read + write and leads to less DB records in the database.
      
      The change removes creation of move placeholders, and considers the
      move pointers when evaluating sorting and PID in DataHandler.
      
      Read functionality from BackendUtility and PageRepository don't need an
      additional step to fetch the live version of a move placeholder anymore.
      
      An upgrade wizard takes existing move placeholders (state=3), updates
      pid+sorting (PID generally not needed, just to be sure) of the move
      pointer (state=4) and then deletes the move placeholder.
      
      TCA definition $TCA[my-table][ctrl][shadowColumnsForMovePlaceholders]
      is not needed anymore and removed by an auto TCA migration.
      
      Finally, workspace enabled tables do not need the t3ver_move_id field
      anymore: The live record UID is already in t3ver_oid field for state=4
      records, just like with all other versioned records. The field will
      be fully removed with a separate patch in order to keep the actual CSV
      tests readable for this patch.
      
      Resolves: #92497
      Releases: master
      Change-Id: I206336aec8be8a324fefdfd69f648f5a298c6ad1
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/65797
      
      Tested-by: default avatarTYPO3com <noreply@typo3.com>
      Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
      Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      27c7de8a
  25. 20 Sep, 2020 1 commit
  26. 19 Sep, 2020 1 commit
  27. 04 Sep, 2020 1 commit
  28. 09 Jul, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Remove various deprecated arguments and methods · eb0d4528
      Benni Mack authored and Daniel Goerz's avatar Daniel Goerz committed
      This change removes deprecated functionality:
      
      - ReferenceIndex->updateIndex() is now either ProgressListenerInterface or null
      - ExtensionManagementUtility->findService() expects an array as third argument
      - BasicFileUtility->setFileExtensionPermissions() removed
      - GeneralUtility->callUserFunction now expects object or null as third argument
      - DataMapper->__construct does not expect $query to be set anymore
      - ObjectAccess->setProperty - fourth argument removed
      
      Resolves: #91613
      Related: #91473
      Releases: master
      Change-Id: I39fa30f84201b0ed837f6ac0e6d27e1ddb15376d
      eb0d4528
  29. 16 Apr, 2020 1 commit
  30. 15 Apr, 2020 1 commit
  31. 14 Apr, 2020 1 commit
  32. 29 Feb, 2020 1 commit
  33. 14 Feb, 2020 1 commit
  34. 13 Feb, 2020 1 commit
  35. 27 Jan, 2020 1 commit
  36. 30 Dec, 2019 1 commit
  37. 22 Nov, 2019 1 commit
    • Benni Mack's avatar
      [FEATURE] Migrate various Signal Slots to PSR-14 events · b9d67bc7
      Benni Mack authored and Daniel Goerz's avatar Daniel Goerz committed
      This change migrates existing Extbase Signal Slots in EXT:core
      to the new PSR-14 events, which allow to define a proper API
      for each event fired.
      
      The following new Events are in place:
      - TYPO3\CMS\Core\Imaging\Event\ModifyIconForResourcePropertiesEvent
      - TYPO3\CMS\Core\DataHandling\Event\IsTableExcludedFromReferenceIndexEvent
      - TYPO3\CMS\Core\DataHandling\Event\AppendLinkHandlerElementsEvent
      - TYPO3\CMS\Core\Configuration\Event\AfterTcaCompilationEvent
      - TYPO3\CMS\Core\Database\Event\AlterTableDefinitionStatementsEvent
      - TYPO3\CMS\Core\Tree\Event\ModifyTreeDataEvent
      
      The following signals are now deprecated:
      - TYPO3\CMS\Core\Imaging\IconFactory::buildIconForResourceSignal
      - TYPO3\CMS\Core\Database\SoftReferenceIndex::setTypoLinkPartsElement
      - TYPO3\CMS\Core\Database\ReferenceIndex::shouldExcludeTableFromReferenceIndex
      - TYPO3\CMS\Core\Utility\ExtensionManagementUtility::tcaIsBeingBuilt
      - TYPO3\CMS\Install\Service\SqlExpectedSchemaService::tablesDefinitionIsBeingBuilt
      - TYPO3\CMS\Core\Tree\TableConfiguration\DatabaseTreeDataProvider::PostProcessTreeData
      
      Resolves: #89733
      Releases: master
      Change-Id: I0747c1de3b77a6be2870d87a054522a7df2fdb18
      Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/62331
      
      Tested-by: default avatarTYPO3com <noreply@typo3.com>
      Tested-by: Susanne Moog's avatarSusanne Moog <look@susi.dev>
      Tested-by: Daniel Goerz's avatarDaniel Goerz <daniel.goerz@posteo.de>
      Reviewed-by: Susanne Moog's avatarSusanne Moog <look@susi.dev>
      Reviewed-by: Daniel Goerz's avatarDaniel Goerz <daniel.goerz@posteo.de>
      b9d67bc7
  38. 04 Nov, 2019 1 commit
  39. 07 Oct, 2019 1 commit