Skip to content

GitLab

Switch branch/tag
  1. 02 Dec, 2021 1 commit
  3. 24 Nov, 2021 1 commit
    • Christian Kuhn's avatar
      [BUGFIX] Better sys_refindex with workspace mm · 3801adce
      Christian Kuhn authored 
      This fixes issues regarding sys_refindex handling when dealing
with workspace mm relations. Various DataHandler tests run with
disabled refindex check after performing mm related operations.
All those are enabled now, so all known issues in this area
are fixed.

The patch touches various areas in DataHandler, ReferenceIndex
and RelationHandler to achieve this.

The key changes are triggered by a database scenario that is
unique for mm relations. Usually, when creating a workspace record
or overlay, child record overlays (for instance inline children)
are also created in this workspace.

For mm records, this can be different: When the local or foreign
side of a record is created in a workspace, it does not necessarily
mean that a connected opposite side is also created as workspace
overlay.

This would otherwise create the need for a recursive operation that
would basically end up with "everything" being created as workspace
overlay. However, mm table entries are created for these relations,
leading to a situation that a workspace record can point to a
non-workspace record through the mm table.

Reference index entries for mm table connections are always handled
from the 'local' side. For instance, with categories, 'sys_category'
table is the local side, with 'pages', 'tt_content' and others
being the foreign side. If now one of the records on the foreign
side gets a workspace overlay record, sys_refindex rows of all
connected records of the affected local side need to be created.
This can lead to the funny situation that we end up with refindex
rows that point from a non-workspace to a non-workspace record,
but have a non-0 workspace entry. Those additional rows are needed
to have "the full set" and proper sorting when looking at refindex
of such a relation from the local side.

The patch basically handles especially these 'foreign side has
a workspace overlay' scenarios, plus the side effects that
have to be taken care of when discarding and publishing these
records.

Additionally, a couple of side effects are tackled: First, the
ReferenceIndex->updateIndex() - that's the main logic when
running cli referenceindex:update command - is tuned to drop
reference index entries related to meanwhile deleted workspaces.
This is covered with additional tests and is done now since
the code needs to iterate over existing workspaces for local-side
mm records that may have workspace overlays on the foreign side.
So most of the code had to be created now anyways.

RelationHandler->readMM() receives a fix for a long standing bug
(Issue #83582, introduced by #57169), to no longer show workspace
relations in live when looking at local-side records, for instance
when looking at category items in live when one of the connected
items has a workspace overlay.

Next to lots of comments explaining details in place and
referencing the test cases that nail specific scenarios, a
couple of @todo's are added to point out things we may
want to tackle in the future.

All in all, the patch resolves a series of issues that will
especially lead to 'categories' being much more reliable in
workspaces.

Change-Id: I24407f93665852fa761f6fbe6c5ab249473468d2
Related: #57169
Resolves: #83582
Resolves: #96067
Releases: master, 11.5
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/72250

Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Tested-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Reviewed-by: Stefan Bürk's avatarStefan Bürk <stefan@buerk.tech>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      3801adce
  5. 12 Oct, 2021 1 commit
  7. 30 Sep, 2021 1 commit
  9. 24 Sep, 2021 1 commit
  11. 17 Sep, 2021 1 commit
  13. 12 Aug, 2021 1 commit
    • Nikita Hovratov's avatar
      [FEATURE] Register SoftReference parsers via DI · 48810cb7
      Nikita Hovratov authored and Benni Mack's avatar Benni Mack committed 
      The concept for registration and usage of soft reference parsers
received a complete overhaul.

Starting with the registration, it is now possible to register soft
reference parsers by dependency injection in the extension's
Services.(yaml|php) file. For this, the new tag name
"softreference.parser" has been introduced. One has to provide the
additional attribute "parserKey" to identify the parser. This
replaces the old way of registering these parsers in the $GLOBALS
array. If a parser is registered with the same key in both ways,
the old way takes precedence for b/w compatibility.

This comes with a completely new factory service class
TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserFactory.
This classes' responsibilities are collecting all registered soft
reference parsers and serving them to the consumer by calling the
method "getSoftReferenceParser" with the desired parser key as the
only argument. There is a compatibility layer for the old way of
registration and for classes not implementing the new interface.

Soft reference parsers now have to implement
TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserInterface.
The interface defines the implementation of the "parse" method.
The first 4 and the last parameter stay the same as in the old method
"findRef". The remaining two parameters "spKey" and "spParams" have to
be set with the "setParserKey" method, if they are needed. The key can
be retrieved by using the "getParserKey" method.

The different parser implementations in the old class
TYPO3\CMS\Core\Database\SoftReferenceIndex have been extracted and
moved into dedicated classes in the
TYPO3\CMS\Core\DataHandling\SoftReference namespace. Missing tests
for parsers other than "typolink" and "typolink_tag" are added.

The method makeTokenID of SoftReferenceIndex has been moved into
TYPO3\CMS\Core\DataHandling\SoftReference\AbstractSoftReferenceParser.
A parser can extend this abstract class, if this method is needed.

The methods of BackendUtility "softRefParserObj" and
"explodeSoftRefParserList" are now deprecated and the replacement
TYPO3\CMS\Core\DataHandling\SoftReference\SoftReferenceParserFactory
should be used instead.

Resolves: #94687
Resolves: #94741
Releases: master
Change-Id: I460bfdd4478194fa4b4111fc44871f7225c6c084
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/70177

Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      48810cb7
  15. 02 Aug, 2021 1 commit
    • Oliver Bartsch's avatar
      [FEATURE] Introduce TCA type "category" · 1e7653ce
      Oliver Bartsch authored and Benni Mack's avatar Benni Mack committed 
      A new TCA type "category" is introduced, which
allows to simplify the configuration of category
TCA columns. Besides the benefit for integrators,
this allows to deprecate the CategoryRegistry
in the next step.

The new type can also be used for other use cases.
Therefore, the TCA option "relationship" is available
for this TCA type. Besides "manyToMany" (default), this
can also be set to "oneToOne" or "oneToMany".

Using the new type, FormEngine will always render a
category tree. This means, no additional `renderType`
is defined. In such case, TCA type "select" can
still be used as before, without any limitation. However,
all relevant places in core are adjusted in this patch.

The category element is rendered through a custom element
(web component), reducing inline javascript.

Resolves: #94622
Releases: master
Change-Id: I1b95c42288b070fa6bac114266f5ad246a045b21
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/69899

Tested-by: core-ci's avatarcore-ci <typo3@b13.com>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      1e7653ce
  17. 29 Jul, 2021 1 commit
  19. 22 Jul, 2021 1 commit
  21. 08 Jul, 2021 1 commit
  23. 16 Jun, 2021 1 commit
  25. 14 May, 2021 1 commit
  27. 08 Mar, 2021 1 commit
  29. 02 Mar, 2021 1 commit
  31. 22 Jan, 2021 1 commit
  33. 12 Jan, 2021 1 commit
  35. 14 Dec, 2020 2 commits
    • Christian Kuhn's avatar
      [BUGFIX] More deterministic mm sorting · ff1a76ed
      Christian Kuhn authored 
      A 'true' MM relation (select / group with MM, but not inline
with foreign_field - used in core for pages/tt_content table
to sys_category via sys_category_record_mm) has two sorting
fields: 'sorting' for the local (sys_category) side, and
'sorting_foreign' for the foreign (pages / tt_content) side.

When categories are added to a tt_content element,
'sorting_foreign' is set to the given order of sys_category
elements. 'sorting' is set to 0. When then opening a
sys_category record that has multiple records pointing to
it with 0 as sorting, the returned order of records is
non-deterministic and depends on implicit DB engine fallbacks.

This also confuses the reference index, which checks
refindex integrity always from the 'local' side.

The patch adds 'uid_foreign' as second order-by field to
force deterministic ordering. There is still a possible
collision in multi table relations (more than one foreign
table uses the mm table like pages AND tt_content to
sys_category with foreign records having the same uid) and
if the mm table allows "multi" relations. Those scenarios
however have no explicit TCA configuration (only implicit
via MM_match_fields), need a deeper investigation and
possibly further detail patches later.

The patch should for now fix a functional test that is
flaky with postgres.

Change-Id: I3c89d0e67f8a4065354f9df173020ca0080e0d57
Resolves: #93075
Releases: master, 10.4
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67110

Reviewed-by: Andreas Fernandez's avatarAndreas Fernandez <a.fernandez@scripting-base.de>
Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: default avatarTYPO3com <noreply@typo3.com>
Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      ff1a76ed
    • Benni Mack's avatar
      [BUGFIX] DBAL: Do not use deprecated classes · b2a22815
      Benni Mack authored and Christian Kuhn's avatar Christian Kuhn committed 
      In preparation for Doctrine DBAL 3.0,

1. all usages of Doctrine\DBAL\DBALException have
been migrated to Doctrine\DBAL\Exception,
because DBAL Exception does not exist in
Doctrine 3.0 anymore.

2. Doctrine\DBAL\Platforms\PostgreSqlPlatform
has been migrated to Doctrine\DBAL\Platforms\PostgreSQL94Platform
because this class does not exist anymore in Doctrine DBAL 3.0,
same goes for Doctrine\DBAL\Platforms\SQLServerPlatform
which has been replaced by Doctrine\DBAL\Platforms\SQLServer2012Platform

3. Doctrine\DBAL\Driver\PDOException has been
renamed to Doctrine\DBAL\Driver\PDO\Exception

4. Doctrine\DBAL\Driver\PDOStatement has been
renamed to Doctrine\DBAL\Driver\PDO\Statement

Resolves: #93071
Releases: master
Change-Id: I05e82f2fca09eb7718a90c09f95e503980ae10ae
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/67109

Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Tested-by: default avatarTYPO3com <noreply@typo3.com>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      b2a22815
  37. 09 Dec, 2020 1 commit
  39. 08 Dec, 2020 1 commit
  41. 20 Nov, 2020 1 commit
  43. 12 Nov, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Do not create new version placeholders in workspaces anymore · 74899ecf
      Benni Mack authored 
      Creating a new record in a workspace adds two database rows.

One that is the "placeholder", which - since v10.4 - contains
the same metadata as the other record:

* t3ver_wsid = workspaceID
* t3ver_oid = 0 (simulating behavior of an "online pendant record")
* t3ver_state = 1

And the "versionized" record, identified by:

* t3ver_wsid = workspaceID
* t3ver_oid = uid of the new placeholder record
* t3ver_state = -1

As of TYPO3 v10, the first record is not needed anymore,
the versioned record can be queried directly, however, since
the relations (except MM) point to the placeholder record,
this one is kept.

As result, only one record is created from now on:

* t3ver_wsid = workspaceID
* t3ver_oid = 0 (no online counterpart)
* t3ver_state = 1

On reading, the record is queried directly (no overlay needed anymore!)
with the existing Database Doctrine Restrictions. On publishing, the
record just gets the state/stage/wsid set and is "live".

This brings fundamental benefits:

* No overlays needed when querying
* Fewer database records (placeholders are not helpful)
* Conceptual problems with placeholder and shadowed fields are removed

Resolves: #92791
Releases: master
Change-Id: I0288cc63fe72d8442d586f309bd4054ac44e829b
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/65587

Tested-by: default avatarTYPO3com <noreply@typo3.com>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Tested-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Reviewed-by: Benni Mack's avatarBenni Mack <benni@typo3.org>
      74899ecf
  45. 29 Oct, 2020 1 commit
  47. 12 Oct, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Remove move placeholders · 27c7de8a
      Benni Mack authored and Christian Kuhn's avatar Christian Kuhn committed 
      Workspaces ("Element-based versioning") previously had - due to
the "pid=-1" logic until TYPO3 v10 - a so-called "MOVE PLACEHOLDER".

This was indicated by t3ver_state = 3, all relevant fields:
* t3ver_state = 3 (move placeholder)
* t3ver_oid = 0 no connected live record, it allowed fetching these records
  with one query together with live records as db restrictions t3ver_oid > 0
* t3ver_wsid = workspace UID
* t3ver_move_id = UID of the live record
* pid = new PID the version was moved to
* sorting - when a record was moved within page with activated sorting

Other record fields were not important. However, when moving a record, the
value from TCA ctrl section "shadowColumnsForMovePlaceholders" was used to
fill in gaps from the live record.

The ACTUAL versioned record was indicated by t3ver_state = 4, the so-called
"MOVE POINTER". In previous version until TYPO3 v10, it's PID field was set
to -1, but since TYPO3 v10, it has the same PID as the "MOVE PLACEHOLDER".

Characteristics of the move pointer as of TYPO3 v10:
* t3ver_state = 4 (move pointer)
* t3ver_oid = UID of the live record
* t3ver_wsid = workspace UID
* t3ver_move_id = 0
* pid = PID the version was moved to
* sorting - same value as the live record (not evaluated until now)
* All other fields with optionally modified content

Both move placeholder and move pointer did not know each other directly.
Fetching the move pointer for a move placeheldor (or the other way around)
involved the live record, leading to many queries.

The patch obsoletes the move placeholder records, moving necessary
information to the move pointer: It now contains the updated sorting
and is considered in the Database Restrictions to be fetched.

In general, when publishing, the moved record now
behaves identical to the other versioned types.

This makes the internal code much easier, creates less DB queries
on read + write and leads to less DB records in the database.

The change removes creation of move placeholders, and considers the
move pointers when evaluating sorting and PID in DataHandler.

Read functionality from BackendUtility and PageRepository don't need an
additional step to fetch the live version of a move placeholder anymore.

An upgrade wizard takes existing move placeholders (state=3), updates
pid+sorting (PID generally not needed, just to be sure) of the move
pointer (state=4) and then deletes the move placeholder.

TCA definition $TCA[my-table][ctrl][shadowColumnsForMovePlaceholders]
is not needed anymore and removed by an auto TCA migration.

Finally, workspace enabled tables do not need the t3ver_move_id field
anymore: The live record UID is already in t3ver_oid field for state=4
records, just like with all other versioned records. The field will
be fully removed with a separate patch in order to keep the actual CSV
tests readable for this patch.

Resolves: #92497
Releases: master
Change-Id: I206336aec8be8a324fefdfd69f648f5a298c6ad1
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/65797

Tested-by: default avatarTYPO3com <noreply@typo3.com>
Tested-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Tested-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
Reviewed-by: Oliver Bartsch's avatarOliver Bartsch <bo@cedev.de>
Reviewed-by: Christian Kuhn's avatarChristian Kuhn <lolli@schwarzbu.ch>
      27c7de8a
  49. 20 Sep, 2020 1 commit
  51. 19 Sep, 2020 1 commit
  53. 04 Sep, 2020 1 commit
  55. 09 Jul, 2020 1 commit
    • Benni Mack's avatar
      [!!!][TASK] Remove various deprecated arguments and methods · eb0d4528
      Benni Mack authored and Daniel Goerz's avatar Daniel Goerz committed 
      This change removes deprecated functionality:

- ReferenceIndex->updateIndex() is now either ProgressListenerInterface or null
- ExtensionManagementUtility->findService() expects an array as third argument
- BasicFileUtility->setFileExtensionPermissions() removed
- GeneralUtility->callUserFunction now expects object or null as third argument
- DataMapper->__construct does not expect $query to be set anymore
- ObjectAccess->setProperty - fourth argument removed

Resolves: #91613
Related: #91473
Releases: master
Change-Id: I39fa30f84201b0ed837f6ac0e6d27e1ddb15376d
      eb0d4528
  57. 16 Apr, 2020 1 commit
  59. 15 Apr, 2020 1 commit
  61. 14 Apr, 2020 1 commit
  63. 29 Feb, 2020 1 commit
  65. 14 Feb, 2020 1 commit
  67. 13 Feb, 2020 1 commit
  69. 27 Jan, 2020 1 commit
  71. 30 Dec, 2019 1 commit
  73. 22 Nov, 2019 1 commit
    • Benni Mack's avatar
      [FEATURE] Migrate various Signal Slots to PSR-14 events · b9d67bc7
      Benni Mack authored and Daniel Goerz's avatar Daniel Goerz committed 
      This change migrates existing Extbase Signal Slots in EXT:core
to the new PSR-14 events, which allow to define a proper API
for each event fired.

The following new Events are in place:
- TYPO3\CMS\Core\Imaging\Event\ModifyIconForResourcePropertiesEvent
- TYPO3\CMS\Core\DataHandling\Event\IsTableExcludedFromReferenceIndexEvent
- TYPO3\CMS\Core\DataHandling\Event\AppendLinkHandlerElementsEvent
- TYPO3\CMS\Core\Configuration\Event\AfterTcaCompilationEvent
- TYPO3\CMS\Core\Database\Event\AlterTableDefinitionStatementsEvent
- TYPO3\CMS\Core\Tree\Event\ModifyTreeDataEvent

The following signals are now deprecated:
- TYPO3\CMS\Core\Imaging\IconFactory::buildIconForResourceSignal
- TYPO3\CMS\Core\Database\SoftReferenceIndex::setTypoLinkPartsElement
- TYPO3\CMS\Core\Database\ReferenceIndex::shouldExcludeTableFromReferenceIndex
- TYPO3\CMS\Core\Utility\ExtensionManagementUtility::tcaIsBeingBuilt
- TYPO3\CMS\Install\Service\SqlExpectedSchemaService::tablesDefinitionIsBeingBuilt
- TYPO3\CMS\Core\Tree\TableConfiguration\DatabaseTreeDataProvider::PostProcessTreeData

Resolves: #89733
Releases: master
Change-Id: I0747c1de3b77a6be2870d87a054522a7df2fdb18
Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/62331

Tested-by: default avatarTYPO3com <noreply@typo3.com>
Tested-by: Susanne Moog's avatarSusanne Moog <look@susi.dev>
Tested-by: Daniel Goerz's avatarDaniel Goerz <daniel.goerz@posteo.de>
Reviewed-by: Susanne Moog's avatarSusanne Moog <look@susi.dev>
Reviewed-by: Daniel Goerz's avatarDaniel Goerz <daniel.goerz@posteo.de>
      b9d67bc7
  75. 04 Nov, 2019 1 commit
  77. 07 Oct, 2019 1 commit