“”

Site Crawler Site Crawler

17 likes

crawler / stable

TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.

This version supports TYPO3

12 LTS 13 LTS

Older versions also support TYPO3

10 LTS 11 LTS 12 LTS 13 LTS


Tags

Get started

Download 12.0.11

Details

Author
Tomas Norre Mikkelsen
Last update
11. May 2026
First upload
22. Dec 2005
Downloads
537,176
Category
Backend Modules
Dependencies
  • PHP (>=8.1.0 <=8.99.99)
  • TYPO3 (>=12.4.0 <=13.4.99)

Last upload comment

[RELEASE] Crawler 12.0.11 - Security Please Update ASAP

Security
* Switch from serialize to json_encode [@tomasnorre](https://github.com/tomasnorre)


Downloads by month

Installation

  1. Download ZIP file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the ZIP file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.
  1. Go to your folder where the root composer.json file is located
  2. Type: composer require tomasnorre/crawler to get the latest version that runs on your TYPO3 version.
  1. Download T3X file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the T3X file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.

Version history

Filter for TYPO3 versions

12.0.11 stable Latest
May 11, 2026
TYPO3: 12 LTS 13 LTS (>=12.4.0 <=13.4.99)
[RELEASE] Crawler 12.0.11 - Security Please Update ASAP

Security
* Switch from serialize to json_encode [@tomasnorre](https://github.com/tomasnorre)
Download
12.0.10 stable Security
October 28, 2025
TYPO3: (>=12.4.0 <=13.4.99)
[RELEASE] Crawler version 12.0.10

Added
* Extract error information in SubProcess crawl strategy [@cweiske](https://github.com/cweiske)
* Expose error when crawl response has not X-T3Crawler-Meta header [@cweiske](https://github.com/cweiske)
* RequestHelper to more some of the logic for fetching value from request [@tomasnorre](https://github.com/tomasnorre)
* Automatic reload process status without page reload [@tomasnorre](https://github.com/tomasnorre)

Changed
* ProcessCleanUpHook is splitted into more classes to strive for seperations of concerns [@tomasnorre](https://github.com/tomasnorre)

Fixed
* Set guzzle connect timeout to 5 seconds [@cweiske](https://github.com/cweiske)
* Prevent warning when guzzle crawl requests return errors [@cweiske](https://github.com/cweiske)
N/A
12.0.9 stable Security
August 21, 2025
TYPO3: (>=12.4.0 <=13.4.99)
[RELEASE] Crawler version 12.0.9

Added
* Document setup for indexed_search + crawler [@cweiske](https://github.com/cweiske)

Fixed
* Fix usage of no_cache in CrawlerInitialization [@luisbecker-gh](https://github.com/luisbecker-gh)
* Ensure that pages are added to queue on clear page cache [@tomasnorre](https://github.com/tomasnorre)
* Ensure that selections are kept in Crawler Start module, when Updating URLs [@tomasnorre](https://github.com/tomasnorre)
* Ensure that pageRow['uid'] is set or return with skipMessage [@tomasnorre](https://github.com/tomasnorre)
* Back link in Crawler Log after visiting details about Queue Item [@tomasnorre](https://github.com/tomasnorre)
N/A
12.0.8 stable Security
May 23, 2025
TYPO3: (>=12.4.0 <=13.4.99)
[RELEASE] Crawler version 12.0.8

Fixed
* Ensure that correct typo3 binary is returned in non-composer installations
* Adjust path in cli/bootstrap.php to match new folder structure of where extensions are installed
* Don't wrap empty HTML in table-tag to ensure valid HTML
* Fix crawling of mount pages
* Fix indexing of cached pages when "disableFrontendIndexing" is active [@cweiske](https://github.com/cweiske)
N/A
12.0.7 stable Security
January 27, 2025
TYPO3: (>=12.4.0 <=13.4.99)
[RELEASE] Crawler version 12.0.7

Added
* Support for PHP 8.4
N/A
12.0.6 stable Security
January 24, 2025
TYPO3: (>=12.4.0 <=13.4.99)
[RELEASE] Crawler version 12.0.6
Added
* Support for TYPO3 13 LTS

Changed
* ProcessRepository doesn't extend from TYPO3\CMS\Extbase\Persistence\Repository anymore
* QueueRepository doesn't extend from TYPO3\CMS\Extbase\Persistence\Repository anymore
* PageRepository::DOKTYPE_RECYCLER is removed from disallowedDoctypes in PageService for TYPO3 13 LTS
N/A
12.0.5 stable Security
January 20, 2025
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.5

Added
* Possibility to flush all processes from the backend module in process view
* Information when page does not exists when crawler:buildQueue command is called from cli

Changed
* Disallow DOKTYPE_BE_USER_SECTION to be crawled
* CrawlerCsvWriter::CARRIAGE_RETURN constant is now private

Fixed
* Prevent exception `Truncated incorrect DECIMAL value` in `crawler:processQueue` [@xyng](https://github.com/xyng)
* Typecast PageTS Crawler Configuration `force_ssl` to `int`
* Use proper pid's in EXT:news docs code example
N/A
12.0.4 stable Security
June 13, 2024
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.4

Changed
* Update composer dependencies

Fixed
* Check queueRec before usages to avoid php-errors
N/A
12.0.3 stable Security
May 30, 2024
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.3

Fixed:
* Regression, enables installed on non-composer projects again
N/A
12.0.2 stable Security
May 29, 2024
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.2
N/A
12.0.1 stable Security
May 29, 2024
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.1
N/A
12.0.0 stable Security
May 21, 2024
TYPO3: (>=12.4.0 <=12.9.99)
[RELEASE] Crawler version 12.0.0

Add support for TYPO3 12

N/A
11.0.15 stable
May 20, 2026
TYPO3: 10 LTS 11 LTS (>=10.4.11 <=11.5.99)
[RELEASE] Crawler 11.0.15

Changed
* Disabled automatic deployment to TYPO3 Extension Repository
Download
11.0.14 stable
May 20, 2026
TYPO3: 10 LTS 11 LTS (>=10.4.11 <=11.5.99)
[RELEASE] Crawler 11.0.14

Fixed
* Installation for non-composer projects
Download
11.0.13 stable
May 18, 2026
TYPO3: 10 LTS 11 LTS (>=10.4.11 <=11.5.99)
[RELEASE] Crawler 11.0.13 - Security Please Update ASAP

Security
* Switch from serialize to json_encode
Download
11.0.12 stable Security
March 23, 2026
TYPO3: (>=10.4.11 <=11.5.99)
[RELASE] Crawler version 11.0.12

Added
* Extract error information in SubProcess crawl strategy [@cweiske](https://github.com/cweiske)

Fixed
* "Scheduled" and "Run-time" column values in backend log [cweiske](https://github.com/cweiske)
* Clear page cache before crawling [cweiske](https://github.com/cweiske)
N/A
11.0.11 stable Security
January 20, 2025
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.11

Backport of #1009 by @Hawkeye1909 in #1089
[CI] Update PHPStan baseline by @tomasnorre in #1092
BUGFIX: Typecast PageTS Crawler Configuration force_ssl to int by @tomasnorre in #1112
[BUGFIX] Catch E_WARNING when unserialize() fails by @cweiske in #1117
N/A
11.0.10 stable Security
July 04, 2024
TYPO3: (>=10.4.11 <=11.5.99)
[RELASE] Crawler version 11.0.10

Fixes:
* Check queueRec before usages to avoid php-errors
N/A
11.0.9 stable Security
May 30, 2024
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.9

Fixes:
* Updated composer.json to have prepare-release to enable installations on non-composer projects
N/A
11.0.8 stable Security
May 22, 2024
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.8
N/A
11.0.7 stable Security
November 18, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELASE] Crawler version 11.0.7

* Ensure that array key is set before used in BuildQueueCommand (PHP 8.1)
N/A
11.0.6 stable Security
November 12, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.6

Fixed
* Zip extension with PHP 7.4 builds instead of PHP 8.1

N/A
11.0.5 stable Security
November 11, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.5

Fixed:
* If indexed_search is installed activate procInstructions for indexed_search
* Ensure undefined indexes is not throwing a warning in PHP 8
N/A
11.0.4 stable Security
February 11, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.4

* Crawler could not index frontend because of trustedHostPattern mismatch
* Fatal PHP error is thrown in the backend crawler log
* Prevent buildQueue from creating entries for non-localized pages
* CrawlerController checking if URL is correct ObjectType fixed

* ConfigurationRepository->getCrawlerConfigurationRecords()
* ProcessRepository->findByProcessId()
* QueueRepository->countAllUnassignedPendingItems()
* QueueRepository->countPendingItemsGroupedByConfigurationKey()
* QueueRepository->getSetIdWithUnprocessedEntries()
* QueueRepository->getTotalQueueEntriesByConfiguration()
* QueueRepository->getLastProcessedEntriesTimestamps()
* QueueRepository->getLastProcessedEntries()
* QueueRepository->getPerformanceData()
* QueueRepository->isPageInQueueTimed()
* QueueRepository->getAvailableSets()
* QueueRepository->findByQueueId()
N/A
11.0.3 stable Security
January 28, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.3

Added
* PHP 8.1 compatibility

Fixed
* PHP8 undefined array key when creating/updating/saving configuration record
* RestrictedPages is getting crawled when Crawler Configuration contains the belonging UserGroup
N/A
11.0.2 stable Security
January 11, 2022
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.2

Added
* Docs: Troubleshooting - Site based baseVariants not used
* Docs: Example on how to exclude pages recursively from being crawled
* Docs: Respecting Categories in News to prevent double indexing
* Docs: Set correct main branch name

Fixed
* DateTime parse error in backend module when displaying log
* PHP8 undefined array keys in CrawlerController, ConfigurationService and middlewares
* Text not displayed in confirm dialog in backend module
N/A
11.0.1 stable Security
November 01, 2021
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.1

Changed
* Use more expressive message for excluded doktypes when building queue

Fixed
* Set `result_data` back to `json_encode($result)` to ensure correct result handling in backend
* Removes QueueRepository from FrontendUserAuthenticator Middleware, to ensure FE plugins can be rendered
* PHP 8.0 compatibility problems. Undefined indexes resolved
* Change requests to write to Header to ensure StaticFileCache is working
* Switched back to fetch() instead of fetchAssociative() to keep support for legacy TYPO3 installations
N/A
11.0.0 stable Security
October 22, 2021
TYPO3: (>=10.4.11 <=11.5.99)
[RELEASE] Crawler version 11.0.0

Only added functionality is added here, please see the
https://github.com/tomasnorre/crawler/blob/main/CHANGELOG.md
for more changes.

Added
* Support for TYPO3 11 LTS
* PHP 8.0 Support
* Display error messages in buildQueue command in CLI
* PSR-14 ModifySkipPageEvent
* PSR-14 AfterUrlCrawledEvent
* PSR-14 InvokeQueueChangeEvent
* PSR-14 AfterUrlCrawledEvent
* PSR-14 BeforeQueueItemAddedEvent
* PSR-14 AfterQueueItemAddedEvent
N/A
10.0.3 stable Security
September 29, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 10.0.3

Fixed
* Avoid empty mountpoint parameter in URL when building the queue
* Add description to crawler:commands for TYPO3 10+
N/A
10.0.2 stable Security
September 16, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 10.0.2

Added
* Replace in composer for aoepeople/crawler

Changed
* Changed composer package name to tomasnorre/crawler
N/A
10.0.1 stable Security
August 27, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 10.0.1

* Fixed
Exclude Pages in Crawler configs accepts e.g. `6+3` again, page id 6 & depth 3
N/A
10.0.0 stable Security
August 03, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 10.0.0

* Added
Added information about indexed_search money pool

* Fixed
Add processId to finished QueueEntries to ensure the process will been seen as successful instead of cancelled.
Fix type error when exclude subcfg is not defined
Sites/URLs get crawled but not indexed [@twestner](https://github.com/twestner)

* Removed Properties
!!! CrawlerController->tableName
!!! ConfigurationRepository->tableName
!!! ProcessRepository->tableName
!!! QueueRepository->tableName
N/A
9.2.6 stable Security
July 14, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.6

Fixed:
* Added missing symfony/polyfill-php80 to non-composer versions
N/A
9.2.5 stable Security
May 28, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.5

Added
* Added Troubleshooting information about Crawler Module
* Added PageService

Fixed:
* Date parsing for immediate queue entries

Changed:
* A lot of Classes are marked as internal (See CHANGELOG)
* Functions are marked as deprecated (See CHANGELOG)

https://github.com/AOEpeople/crawler/blob/main/CHANGELOG.md
N/A
9.2.4 stable Security
April 09, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.4

Fixes:
* Date parsing for immediate queue entries
N/A
9.2.3 stable Security
March 29, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.3
N/A
9.2.2 stable Security
January 26, 2021
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.2

Added
* AccessService::hasGroupAccess()

Deprecated Functions & Properties
* CrawlerController->CLI_releaseProcesses()
* CrawlerController->hasGroupAccess()

Fixed
* buildQueue with `--mode exec` resulted in a 503
* Set content in `fetchUrlContents()` to false if null, to prevent serializing from failing
N/A
9.2.1 stable Security
December 27, 2020
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.1

Changed
* Moved HTML from the LogRequestForm->drawLog_addRows() to Fluid-template
* Extended the Example for PageTS Configuration

Deprecated Functions
* SignalSlotUtility->emitSignal()

Deprecated Classes
* SignalSlotUtility

Fixed
* PageTS Crawler configurations (broken since 9.1.3)
N/A
9.2.0 stable Security
December 22, 2020
TYPO3: (>=9.5.14 <=11.0.99)
[RELEASE] Crawler version 9.2.0

Added
* Add support for TYPO3 11.0

Deprecated Classes
* CrawlerApi
N/A
9.1.5 stable Security
December 19, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.5

Added
* Error-Handling when Page 0 is used for crawler:buildQueue

Changed
* Removed all references to cHash as it is not used anymore.

Deprecated Functions & Properties
* CrawlerController->getLogEntriesForPageId()
* CrawlerController->CLI_runHooks()
* QueueRepository->countAllByProcessId()
* QueueRepository->countUnprocessedItems()
* FlushQueueCommand --page option

Fixed
* Typecasting port number in cli/bootstrap to have processQueue working with non-standard ports
* Flush entire queue works again
* Check if PageRow['uid'] is integer before handing it on to next function
N/A
9.1.4 stable Security
November 27, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.4

Changed
* IndexedSearchCrawlerHook::class is marked as deprecated and will be removed when dropping support for TYPO3 9LTS and 10LTS

Deprecated Classes
* IndexedSearchCrawlerHook

Deprecated Functions & Properties
* CrawlerController->getDuplicateRowsIfExist()

Fixed
* Auto-loading for non-composer projects fixed
N/A
9.1.3 stable Security
November 20, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.3

Added
* QueueFilter to operate with Object instead of string/arrays

Deprecated
Functions & Properties

* BackendModule->modMenu()
* CrawlerController->CLI_debug()
* CrawlerController->getAccessMode()
* CrawlerController->setAccessMode()
* CrawlerController->getDisabled()
* CrawlerController->setDisabled()
* CrawlerController->getProcessFilename()
* CrawlerController->setProcessFilename()
* CrawlerController->accessMode
* CrawlerController->processFilename

Changed
* Updated Rector, PHPStan, ECS for better static code analysis
* BackendModule is splittet into smaller classes to improve readability and maintainability
* Crawler Class added, this can be used to check whether the crawler is enabled or disabled

Fixed
* aoe/crawler/initialization (middleware) is moved to before typo3/cms-core/normalizedParams to have crawler being "last" in middleware chain
N/A
9.1.2 stable Security
November 07, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.2

Added
* Progress bar to the crawler:buildQueue command output when using with --mode exec
* Improve documentation about proper crawler configuration for user with Website Usergroups

Fixed
* Detailed process views is callable again
* Makes sure the QueueRepository is always set when needed in Domain/Model/Process
* Crawling with FE-Groups is correct initialized with both TYPO3 9 & 10
N/A
9.1.1 stable Security
October 17, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.1

Added
* Documentation example for ext:news
* CrawlStrategyFactory to move login out of the QueueExecutor

Deprecated
Functions & Properties
* ProcessRepository->countActive()
* ProcessRepository->getLimitFromItemCountAndOffset()
* CrawlerController->getUrlFromPageAndQueryParameters()

Changed
* UrlService->getUrlFromPageAndQueryParameters() moved from CrawlerController

Fixed
* Frontend User initialization with UserGroups for crawling protected pages
* Making sure PageUid added with ExcludeString is kept as integers
* Instantiation of ProcessRepository and QueueRepository change to GeneralUtility::makeInstance
* Ensure that DataHandlerHook will not add pages to queue that does not exist
N/A
9.1.0 stable Security
August 02, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.1.0

Added:
* Adds light red background color + icons to crawler log rows with errors
* Crawler processing by page priority
* Automatically adding pages being edited or a page caches is cleared to the crawler queue
N/A
9.0.3 stable Security
July 20, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.0.3

Added:
* More information into CONTRIBUTING.md about development using container

Fixed:
* Load crawler initialization before TSFE rendering preparation
N/A
9.0.2 stable Security
July 13, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.0.2

Added:
* Update specific doktypes for skipping page
* Configure commands in Services.yaml for TYPO3 v10
* Tests for BackendModule
* Add index for tx_crawler_queue.scheduled table field
* Support for Symfony console 5
* Add example for pageVeto hook in documentation

Changed:
* Group TCA in tabs and add descriptions to fields

Fixed:
* Handle empty response when fetching URL
* Set default value when procInstrFilter is not defined
N/A
9.0.1 stable Security
May 11, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.0.1

* Changed result handling for Crawling in case of 500 errors on directRequests
* Set correct tableName for mountPoint queries
* Fallback to alturl is added to ensure crawling of pages including PDF files
* Get fe_user from $GLOBALS['TSFE'] instead of $request->getAttribute()
* Have getTimeForFirstItem() and getTimeForLastItem return 0 is no process found

Please read
https://github.com/AOEpeople/crawler/blob/master/CHANGELOG.md for more
information.
N/A
9.0.0 stable Security
April 14, 2020
TYPO3: (>=9.5.14 <=10.4.99)
[RELEASE] Crawler version 9.0.0

* Adds support for TYPO3 9 LTS
* Adds support for TYPO3 ^10.3
* Drops support for TYPO3 7.6
* Drops support for TYPO3 8.7
* Drops support for < PHP 7.2

For more details check:
https://github.com/AOEpeople/crawler/blob/master/CHANGELOG.md#version-900
N/A
6.7.3 stable Security
November 26, 2020
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Crawler version 6.7.3

Adjust SQL to support MySQL Strict-Mode
N/A
6.7.1 stable Security
March 28, 2020
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Crawler version 6.7.1

[BUGFIX] Switches $sleepAfter and $sleeptime
[BUGFIX] Match force ssl property in model with tca (#494)
[FEATURE] add _RECURSIVE option to the subpartParams Configuration of the lookup _TABLE instructions (#476)
N/A
6.7.0 stable Security
November 08, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.7.0

[BUGFIX] Populate Queue object before returning it from repository (#473)
[DOC] Adds documentation for Command Controllers
N/A
6.6.5 stable Security
October 16, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.5

[BUGFIX] Remove obsolete HTML and adjust tables
[BUGFIX] Adjust table styling to match both TYPO3 7 & 8
[BUGFIX] Resolve typo3cms non-composer path for typo3_console < v5 (#469)
[DOC] Update issue template for security issues (#468)
[BUGFIX] Resolving typo3cms path if composer not used (#446) (#449)
[BUGFIX] Change the SignalSlots to call by reference
N/A
6.6.4 stable Security
August 23, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.4

[BUGFIX] PHP Warning in CrawlerController in CLI mode (#307)
[BUGFIX] Make sure that Process is added and mapped correctly (#413)
[BUGFIX] Change the calculation of composer-bin-dir (#416)
N/A
6.6.3 stable Security
August 21, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.3

[BUGFIX] Changed findByUid() to findByProcessId() (#408)
[BUGFIX] Change type declaration for pidsonly
[BUGFIX] Changing the getters/setters for configuration model (#407)
N/A
6.6.2 stable Security
August 08, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.2

[BUGFIX] Change incorrect mocking and function calls (#404)
N/A
6.6.1 stable Security
August 07, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.1

[BUGFIX] Use correct method names
N/A
6.6.0 stable Security
August 06, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.6.0

[TASK] Add extension-release to composer scripts
[TASK] Setting up Travis CI with Deployment to TER on tags (#400)
[BUGFIX] Change check for CLI request or backend request in CrawlerCommandController
[BUGFIX] Resolves the bin-dir from composer.json (#357)
[TASK] Add tests for expandParameters (#391)
[TASK] Deprecated functions not longer needed (#390)
[TASK] Update documentation rendering to match new setup from docs-team (#389)
[TASK] Deprecated functions (#387)
[BUGFIX] Composer: Move system dependency (#386)
[TASK] Add tests for CrawlerController (#385)
[TASK] Update PHP Requirements (#382)
[FEATURE] Use Extbase Domain/Model + Repository #323 (#328)
N/A
6.5.0 stable Security
July 08, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.5.0

[BUGFIX] Error 'processing_instruction_filter has no default value' (#381)
[TASK] Update composer dependencies
N/A
6.4.1 stable Security
May 08, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.4.1

[BUGFIX] Changed TCA Label for hidden checkbox (#321)
[BUGFIX] Adds version contraint to resolve missing (#352)
N/A
6.4.0 stable Security
May 02, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.4.0

[FEATURE] Deprecates EventDispatcher + Adds Signal Slots (#252)
[BUGFIX] Removes deprecated functions
[CLEANUP] Fix typo in SIGNAL constants
[CLEANUP] Fix problems with PHP 7.2
N/A
6.3.2 stable Security
March 29, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.3.2

[BUGFIX] Adjust getConfigurationKeys to match old usages (#341)
N/A
6.3.1 stable Security
March 25, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.3.1

[TASK] Updates helhum/typo3-console dependencies - Thanks @stucki
N/A
6.3.0 stable Security
March 08, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.3.0

[FEATURE] Implements CommandController, deprecates the CommandLineControllers #322
N/A
6.2.0 stable Security
January 23, 2019
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.2.0

Changelog:

[BUGFIX] Prevents a potential SQL injection vulnerability (#333)
[TASK] Re-adds type hint (#10) (#303)
[TASK] Change NOT NULL for pidsonly field (#318)
[BUGFIX] Avoid warning if $q_entry['result_data'] is em
N/A
6.1.2 stable Security
May 09, 2018
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.1.2

Changelog:
[BUGFIX] Only execute getSelectedState if $task != null (#279) (#296)
[BUGFIX] Fixes $task is null (#287)
[BUGFIX] Create file which is expected by TYPO3 CMS (#262)
[BUGFIX] Switched Refresh-icon to use Icon
N/A
6.1.1 stable Security
February 07, 2018
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Crawler 6.1.1
N/A
6.1.0 stable Security
January 08, 2018
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.1.0

Changelog:
[BUGFIX] Crawler only crawls first level (#261)
[TASK] Add Warn when no configuration is selected (#191)
[CLEANUP] Adjust variable names to reflect renamed Classes (#250)
[FEATURE] Configuration_hash for carw
N/A
6.0.0 stable Security
December 21, 2017
TYPO3: (>=7.6.0 <=8.7.99)
[RELEASE] Release of crawler 6.0.0

Changelog:
[FEATURE] Introduce namespaces (#226)
[BUGFIX] Changed Url-generation for pagination (#89) (#246)
[CLEANUP] Remove legacy TYPO3 CMS 4.x XCLASS registration (#243)
[BUGFIX] Makes starting a new cli process in
N/A
5.2.1 stable Security
December 04, 2017
TYPO3: (>=6.2.0 <=7.6.99)
[RELEASE] Release of crawler 5.2.1

Changelog:
[CLEANUP] Change PHP requirements + updated testing (#228)
[CLEANUP] Remove function call deprecated in TYPO3 7.6+ (#222)
[BUGFIX] Keeps Crawler Info on current page on refresh (#198)
[CLEANUP] Remove hhvm fr
N/A
5.2.0 stable Security
November 06, 2017
TYPO3: (>=6.2.0 <=7.6.99)
[RELEASE] Release of crawler 5.2.0

* [TASK] add post process signal slot also for use with crawler_im (#211)
* [FEATURE] TypoScript Root Template must be configured in crawler configuration
* [TASK] Add CSS class for AdditionalFieldProviders for TYPO3 7.
N/A
5.1.4 stable Security
October 09, 2017
TYPO3: (>=6.2.0 <=7.6.99)
[RELEASE] Release of crawler 5.1.4

** Release Notes **

* [CLEANUP] Adjust version constraints
* [FEATURE] Use actual page ID for TSFE initialization (#214)
* [TASK] Set more realistic timeouts (30 min should be sufficient) (#213)
* Cleanup/codestyle (#2
N/A
5.1.3 stable Security
July 21, 2017
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.1.3

[TASK] Added Cache to TypoScriptUtility
N/A
5.1.2 stable Security
November 24, 2016
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.1.2

[BUGFIX] Scheduler Tasks broken after upgrade to 5.1.1 (composer) (#154)
[BUGFIX] Fixed incorrect class name in ClassAliasMap (#153)
N/A
5.1.1 stable Security
November 21, 2016
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.1.1

[BUGFIX] It's not possible to "add process" after upgrade to 5.1.0 (#145)
[BUGFIX] #1442236317: MenuItem "" is not valid (More information) (#148)
[BUGFIX] Not possible to add new scheduler tasks (not only Crawler tasks
N/A
5.1.0 stable Security
November 15, 2016
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.1.0

2016-11-08 28961a0 [FEATURE] Remove assignment of queue items to process if process gets killed by cleanup hook
2016-11-13 4d82941 [FEATURE] Adjust namespace of ProcessCleanupTask (#138)
2016-11-08 714a222 [CLEANU
N/A
5.0.9 stable Security
September 16, 2016
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.0.9
N/A
5.0.8 stable Security
September 16, 2016
TYPO3: (>=6.2.0 <=7.99.99)
[RELEASE] Release of crawler 5.0.8
N/A
5.0.7 stable Security
September 06, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.6 stable Security
August 22, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.5 stable Security
August 15, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.4 stable Security
March 18, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.3 stable Security
March 10, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.2 stable Security
January 26, 2016
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.1 stable Security
December 08, 2015
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
5.0.0 stable Security
October 27, 2015
TYPO3: (>=6.2.0 <=7.99.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
4.1.1 stable Security
March 10, 2016
TYPO3: (>=4.5.0 <=6.2.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
4.1.0 stable Security
October 27, 2015
TYPO3: (>=4.5.0 <=6.2.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
4.0.0 stable Security
August 11, 2015
TYPO3: (>=4.5.0 <=6.2.99)
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
N/A
3.5.0 stable Security
October 22, 2012
TYPO3:
- Update Documentation
- Add Multiprocessing CLI
- Add Feature to use sys_domain record for BaseURL
- Added UnitTests
- Enable Crawling with HTTPS
- Improve Scheduler Tasks
- Bug-Fixes
N/A
3.2.0 stable Security
June 29, 2010
TYPO3:
Improve scheduler integration, minor bug fixes.
N/A
3.1.2 stable Security
April 19, 2010
TYPO3:
Improve handling with "t3lib_FlashMessage", Update documentation, Reimplement "Result Log" column.
N/A
3.1.1 stable Security
April 01, 2010
TYPO3:
Fix documentation
N/A
3.1.0 stable Security
March 31, 2010
TYPO3:
Update documentation
Improved scheduler and TYPO3 4.3 integration
Fixed several bugs
Add more tests
N/A
3.0.5 beta Security
February 27, 2010
TYPO3:
Release notes:

* TYPO3 4.3 integration
* Scheduler integration
* CLI API cleanup
* Bugfixes
N/A
3.0.0 alpha Security
September 10, 2009
TYPO3:
- multiprocessing
- new way to configure
- see manual for details
N/A
2.1.0 alpha Security
July 24, 2008
TYPO3:
diffrent options in em to control sleeptimes / change "keep-alive" header / add debugmode / readurl fixes
N/A
2.0.0 alpha Security
March 23, 2007
TYPO3:
Updated various features, see manual
N/A
1.1.0 alpha Security
March 21, 2006
TYPO3:
N/A
1.0.0 alpha Security
February 16, 2006
TYPO3:
N/A
0.0.1 alpha Security
December 22, 2005
TYPO3:
[Dont download yet - its only for beta testing before 4.0]
N/A

Get started

Download 12.0.11

Details

Author
Tomas Norre Mikkelsen
Last update
11. May 2026
First upload
22. Dec 2005
Downloads
537,176
Category
Backend Modules
Dependencies
  • PHP (>=8.1.0 <=8.99.99)
  • TYPO3 (>=12.4.0 <=13.4.99)