[FEATURE] Install tool: Extension scanner 50/53450/68
authorChristian Kuhn <lolli@schwarzbu.ch>
Wed, 5 Jul 2017 11:27:04 +0000 (13:27 +0200)
committerChristian Kuhn <lolli@schwarzbu.ch>
Tue, 1 Aug 2017 13:36:43 +0000 (15:36 +0200)
commit081ce36ba9a3110faf463aa9f54a8236612589b1
treec033fd9f815df63faea676c218d56ff73a914442
parent68228c73be2c49a6969aa03e9f3d74ebf41dd75d
[FEATURE] Install tool: Extension scanner

The patch adds a new feature - the "extension scanner" - to the
install tool. It is meant as a helper script for extension
and project developers to quickly spot places in extension code
that may need adaptions if upgrading to younger core versions.

The scanner could later work with different languages, for now
only scanning PHP files is supported. This is based on the great
library nikic/php-parser which creates an AST from php source files
and allows simple traversing.
A series of matchers hooks into the traverser and scans for
configured core deprecation or removals. Some additional matchers
could be added if a core example evolves (for instance usage of
dropped core constants).

From now on, whenever the core deprecates or breaks existing core
PHP API, an entry in one of the matcher configurations in
ext:install/Configuration/ExtensionScanner/Php should be done.
Since these API changes are documented via ReST files already, the
configuration links to according ReST files. A bamboo integrity
test script verifies those can be found.
Furthermore, all Breaking-* and Deprecation-* ReST files since
core version 9.x / master now need an additional Tag "FullyScanned",
"PartiallyScanned" or "NotScanned". This is also checked by an
integration test script in bamboo.

The matcher configuration files are pretty straight forward and
should be obvious if looking at
ext:install/Configuration/ExtensionScanner/Php.

This patch contains:
* Add nikic/php-parser as dependency
* New install tool gui to scan extensions in typo3conf/ext
* Scanner infrastructure for PHP
* Preparation and configuration for all current "master" / v9
  ReST files
* Mark ReST files which are tagged with "FullyScanned" as "not affected"
  in install tool "Upgrade analysis" if no match was found
* Integration script testing referenced ReST files are found
* Integration script verifying Breaking- and Deprecation- ReST file
  contain "FullyScanned", "PartiallyScanned" or "NotSCanned" keyword.

More information on this feature can be found at
https://docs.typo3.org/typo3cms/CoreApiReference/ExtensionScanner/Index.html

Change-Id: I760e88f9d929fdd710655fa0d25cb26291a7ed2b
Resolves: #82014
Releases: master
Reviewed-on: https://review.typo3.org/53450
Tested-by: TYPO3com <no-reply@typo3.com>
Reviewed-by: Jan Helke <typo3@helke.de>
Tested-by: Jan Helke <typo3@helke.de>
Reviewed-by: Wouter Wolters <typo3@wouterwolters.nl>
Reviewed-by: Frank Naegler <frank.naegler@typo3.org>
Tested-by: Frank Naegler <frank.naegler@typo3.org>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
114 files changed:
Build/Resources/Public/Sass/component/_panel.scss
Build/Resources/Public/Sass/install.scss
Build/Scripts/extensionScannerRstFileReferences.php [new file with mode: 0755]
Build/Scripts/validateRstFiles.sh
Build/bamboo/src/main/java/core/AbstractCoreSpec.java
composer.json
composer.lock
typo3/sysext/backend/Resources/Public/Css/backend.css
typo3/sysext/core/Documentation/Changelog/8.4/Deprecation-77987-DeprecatedRecordListingInPageModule.rst
typo3/sysext/core/Documentation/Changelog/8.5/Deprecation-57385-DeprecateParameterCaseSensitiveOfExtbaseLikeComparison.rst
typo3/sysext/core/Documentation/Changelog/Howto.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-74533-ThrowExceptionIfUserFunctionDoesNotExist.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-80700-DeprecatedFunctionalityRemoved.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-80876-RemoveSystemExtensionCss_styled_content.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-80929-TYPO3_DBMovedToExtension.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81171-EditAbilityOfTypoScriptTemplateInEXTtstemplateRemoved.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81225-MergedEXTcontext_helpToEXTbackend.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81534-DatabaseFieldBe_groupshide_in_listsDropped.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81536-MoveOfServicesListReportFromSvToReports.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81735-GetRidOfSysextsv.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81763-HookParametersOfTypo3file_editphppreOutputProcessingHookChanged.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81787-DropEXTfunc.rst
typo3/sysext/core/Documentation/Changelog/master/Breaking-81847-RemoveJSMENU.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-78410-DeprecatePopoverMemberInstanceInTYPO3GlobalObject.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-80993-GeneralUtilitygetUserObj.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81201-EidUtilityinitTCA.rst [new file with mode: 0644]
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81201-EidUtilityloadTCA.rst [deleted file]
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81213-RenderMethodArgumentOnViewHelpersDeprecated.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81218-NoWSOLArgumentInPageRepository-getRawRecord.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81534-BackendUtilitygetListGroupNamesDeprecated.r [deleted file]
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81534-BackendUtilitygetListGroupNamesDeprecated.rst [new file with mode: 0644]
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81540-DeprecateDocumentTemplateformWidth.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81600-UnusedExtbaseExceptions.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81651-ArgumentParametersInListModuleHook.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81763-DeprecatedLanguageLabelForFileRename.rst
typo3/sysext/core/Documentation/Changelog/master/Deprecation-81951-InstallToolEntryPointMoved.rst
typo3/sysext/core/Documentation/Changelog/master/Feature-82014-ExtensionScanner.rst [new file with mode: 0644]
typo3/sysext/install/Classes/Controller/Action/Ajax/ExtensionScannerFiles.php [new file with mode: 0644]
typo3/sysext/install/Classes/Controller/Action/Ajax/ExtensionScannerMarkFullyScannedRestFiles.php [new file with mode: 0644]
typo3/sysext/install/Classes/Controller/Action/Ajax/ExtensionScannerScanFile.php [new file with mode: 0644]
typo3/sysext/install/Classes/Controller/Action/Tool/ExtensionScanner.php [new file with mode: 0644]
typo3/sysext/install/Classes/Controller/Action/Tool/UpgradeAnalysis.php
typo3/sysext/install/Classes/Controller/AjaxController.php
typo3/sysext/install/Classes/Controller/ToolController.php
typo3/sysext/install/Classes/ExtensionScanner/CodeScannerInterface.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/CodeStatistics.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/GeneratorClassesResolver.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/AbstractCoreMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/ArrayDimensionMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/ArrayGlobalMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/ClassConstantMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/ClassNameMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/InterfaceMethodChangedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodArgumentDroppedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodArgumentDroppedStaticMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodArgumentRequiredMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodArgumentUnusedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodCallMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/MethodCallStaticMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/PropertyProtectedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/Matcher/PropertyPublicMatcher.php [new file with mode: 0644]
typo3/sysext/install/Classes/ExtensionScanner/Php/MatcherFactory.php [new file with mode: 0644]
typo3/sysext/install/Classes/UpgradeAnalysis/DocumentationFile.php
typo3/sysext/install/Configuration/ExtensionScanner/Php/ArrayDimensionMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/ArrayGlobalMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/ClassConstantMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/ClassNameMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/InterfaceMethodChangedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodArgumentDroppedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodArgumentDroppedStaticMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodArgumentRequiredMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodArgumentUnusedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodCallMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodCallStaticMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/PropertyProtectedMatcher.php [new file with mode: 0644]
typo3/sysext/install/Configuration/ExtensionScanner/Php/PropertyPublicMatcher.php [new file with mode: 0644]
typo3/sysext/install/Resources/Private/Partials/Action/Common/Left.html
typo3/sysext/install/Resources/Private/Partials/Action/Tool/UpgradeAnalysis/ListDocumentation.html
typo3/sysext/install/Resources/Private/Templates/Action/Tool/ExtensionScanner.html [new file with mode: 0644]
typo3/sysext/install/Resources/Public/Css/install.css
typo3/sysext/install/Resources/Public/JavaScript/Install.js
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/CodeStatisticsTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Fixtures/CodeStatisticsFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Fixtures/MatcherFactoryInvalidConfigurationFileFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/GeneratorClassesResolverTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/AbstractCoreMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/ArrayDimensionMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/ArrayGlobalMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/ClassConstantMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/ClassNameMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/ArrayDimensionMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/ArrayGlobalMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/ClassConstantMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/ClassNameMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/InterfaceMethodChangedMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodArgumentDroppedMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodArgumentDroppedStaticMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodArgumentRequiredMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodArgumentUnusedMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodCallMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/MethodCallStaticMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/PropertyProtectedMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/Fixtures/PropertyPublicMatcherFixture.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/InterfaceMethodChangedMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodArgumentDroppedMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodArgumentDroppedStaticMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodArgumentRequiredMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodArgumentUnusedMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodCallMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/MethodCallStaticMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/PropertyProtectedMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/Matcher/PropertyPublicMatcherTest.php [new file with mode: 0644]
typo3/sysext/install/Tests/Unit/ExtensionScanner/Php/MatcherFactoryTest.php [new file with mode: 0644]
typo3/sysext/install/composer.json