“”

Site Crawler Site Crawler

crawler / stable / verified logoverified

Libraries and scripts for crawling the TYPO3 page tree.

This version supports TYPO3

Older versions also support TYPO3

Composer support

composer req tomasnorre/crawler

Tags

#cache #realurl #crawling #warmup

Last upload comment

[RELEASE] Crawler version 12.0.4

Changed
* Update composer dependencies

Fixed
* Check queueRec before usages to avoid php-errors

Downloads by month

Download 12.0.4 Extension Manual Found an Issue? Code Insights Packagist.org Donate and Give Kudos Crowdin translations
Author
Tomas Norre Mikkelsen
Last update
13. Jun 2024
First upload
22. Dec 2005
Downloads
362,948
Category
Backend Modules
Dependencies
  • PHP (8.1.0 - 8.99.99)
  • TYPO3 (12.4.0 - 12.9.99)

Installation

  1. Download ZIP file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the ZIP file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.
  1. Go to your folder where the root composer.json file is located
  2. Type: composer req tomasnorre/crawler to get the latest version that runs on your TYPO3 version.
  1. Download T3X file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the T3X file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.

Other extensions by this user (1)

TYPO3 Demo Extension for testing GitHub Actions

Supported TYPO3 versions:

Version history

Version Upload comment Works with TYPO3 Download
12.0.4 / stable / verified logo
June 13, 2024
[RELEASE] Crawler version 12.0.4

Changed
* Update composer dependencies

Fixed
* Check queueRec before usages to avoid php-errors
12.4.0 - 12.9.99 Download
12.0.3 / stable / verified logo
May 30, 2024
[RELEASE] Crawler version 12.0.3

Fixed:
* Regression, enables installed on non-composer projects again
12.4.0 - 12.9.99 Download
12.0.2 / stable / verified logo
May 29, 2024
[RELEASE] Crawler version 12.0.2
12.4.0 - 12.9.99 Download
12.0.1 / stable / verified logo
May 29, 2024
[RELEASE] Crawler version 12.0.1
12.4.0 - 12.9.99 Download
12.0.0 / stable / verified logo
May 21, 2024
[RELEASE] Crawler version 12.0.0

Add support for TYPO3 12


12.4.0 - 12.9.99 Download
11.0.10 / stable / verified logo
July 04, 2024
[RELASE] Crawler version 11.0.10

Fixes:
* Check queueRec before usages to avoid php-errors
10.4.11 - 11.5.99 Download
11.0.9 / stable / verified logo
May 30, 2024
[RELEASE] Crawler version 11.0.9

Fixes:
* Updated composer.json to have prepare-release to enable installations on non-composer projects
10.4.11 - 11.5.99 Download
11.0.8 / stable / verified logo
May 22, 2024
[RELEASE] Crawler version 11.0.8
10.4.11 - 11.5.99 Download
11.0.7 / stable / verified logo
November 18, 2022
[RELASE] Crawler version 11.0.7

* Ensure that array key is set before used in BuildQueueCommand (PHP 8.1)
10.4.11 - 11.5.99 Download
11.0.6 / stable / verified logo
November 12, 2022
[RELEASE] Crawler version 11.0.6

Fixed
* Zip extension with PHP 7.4 builds instead of PHP 8.1


10.4.11 - 11.5.99 Download
11.0.5 / stable / verified logo
November 11, 2022
[RELEASE] Crawler version 11.0.5

Fixed:
* If indexed_search is installed activate procInstructions for indexed_search
* Ensure undefined indexes is not throwing a warning in PHP 8
10.4.11 - 11.5.99 Download
11.0.4 / stable / verified logo
February 11, 2022
[RELEASE] Crawler version 11.0.4

* Crawler could not index frontend because of trustedHostPattern mismatch
* Fatal PHP error is thrown in the backend crawler log
* Prevent buildQueue from creating entries for non-localized pages
* CrawlerController checking if URL is correct ObjectType fixed

* ConfigurationRepository->getCrawlerConfigurationRecords()
* ProcessRepository->findByProcessId()
* QueueRepository->countAllUnassignedPendingItems()
* QueueRepository->countPendingItemsGroupedByConfigurationKey()
* QueueRepository->getSetIdWithUnprocessedEntries()
* QueueRepository->getTotalQueueEntriesByConfiguration()
* QueueRepository->getLastProcessedEntriesTimestamps()
* QueueRepository->getLastProcessedEntries()
* QueueRepository->getPerformanceData()
* QueueRepository->isPageInQueueTimed()
* QueueRepository->getAvailableSets()
* QueueRepository->findByQueueId()
10.4.11 - 11.5.99 Download
11.0.3 / stable / verified logo
January 28, 2022
[RELEASE] Crawler version 11.0.3

Added
* PHP 8.1 compatibility

Fixed
* PHP8 undefined array key when creating/updating/saving configuration record
* RestrictedPages is getting crawled when Crawler Configuration contains the belonging UserGroup
10.4.11 - 11.5.99 Download
11.0.2 / stable / verified logo
January 11, 2022
[RELEASE] Crawler version 11.0.2

Added
* Docs: Troubleshooting - Site based baseVariants not used
* Docs: Example on how to exclude pages recursively from being crawled
* Docs: Respecting Categories in News to prevent double indexing
* Docs: Set correct main branch name

Fixed
* DateTime parse error in backend module when displaying log
* PHP8 undefined array keys in CrawlerController, ConfigurationService and middlewares
* Text not displayed in confirm dialog in backend module
10.4.11 - 11.5.99 Download
11.0.1 / stable / verified logo
November 01, 2021
[RELEASE] Crawler version 11.0.1

Changed
* Use more expressive message for excluded doktypes when building queue

Fixed
* Set `result_data` back to `json_encode($result)` to ensure correct result handling in backend
* Removes QueueRepository from FrontendUserAuthenticator Middleware, to ensure FE plugins can be rendered
* PHP 8.0 compatibility problems. Undefined indexes resolved
* Change requests to write to Header to ensure StaticFileCache is working
* Switched back to fetch() instead of fetchAssociative() to keep support for legacy TYPO3 installations
10.4.11 - 11.5.99 Download
11.0.0 / stable / verified logo
October 22, 2021
[RELEASE] Crawler version 11.0.0

Only added functionality is added here, please see the
https://github.com/tomasnorre/crawler/blob/main/CHANGELOG.md
for more changes.

Added
* Support for TYPO3 11 LTS
* PHP 8.0 Support
* Display error messages in buildQueue command in CLI
* PSR-14 ModifySkipPageEvent
* PSR-14 AfterUrlCrawledEvent
* PSR-14 InvokeQueueChangeEvent
* PSR-14 AfterUrlCrawledEvent
* PSR-14 BeforeQueueItemAddedEvent
* PSR-14 AfterQueueItemAddedEvent
10.4.11 - 11.5.99 Download
10.0.3 / stable
September 29, 2021
[RELEASE] Crawler version 10.0.3

Fixed
* Avoid empty mountpoint parameter in URL when building the queue
* Add description to crawler:commands for TYPO3 10+
9.5.14 - 11.0.99 Download
10.0.2 / stable
September 16, 2021
[RELEASE] Crawler version 10.0.2

Added
* Replace in composer for aoepeople/crawler

Changed
* Changed composer package name to tomasnorre/crawler
9.5.14 - 11.0.99 Download
10.0.1 / stable
August 27, 2021
[RELEASE] Crawler version 10.0.1

* Fixed
Exclude Pages in Crawler configs accepts e.g. `6+3` again, page id 6 & depth 3
9.5.14 - 11.0.99 Download
10.0.0 / stable
August 03, 2021
[RELEASE] Crawler version 10.0.0

* Added
Added information about indexed_search money pool

* Fixed
Add processId to finished QueueEntries to ensure the process will been seen as successful instead of cancelled.
Fix type error when exclude subcfg is not defined
Sites/URLs get crawled but not indexed [@twestner](https://github.com/twestner)

* Removed Properties
!!! CrawlerController->tableName
!!! ConfigurationRepository->tableName
!!! ProcessRepository->tableName
!!! QueueRepository->tableName
9.5.14 - 11.0.99 Download
9.2.6 / stable
July 14, 2021
[RELEASE] Crawler version 9.2.6

Fixed:
* Added missing symfony/polyfill-php80 to non-composer versions
9.5.14 - 11.0.99 Download
9.2.5 / stable
May 28, 2021
[RELEASE] Crawler version 9.2.5

Added
* Added Troubleshooting information about Crawler Module
* Added PageService

Fixed:
* Date parsing for immediate queue entries

Changed:
* A lot of Classes are marked as internal (See CHANGELOG)
* Functions are marked as deprecated (See CHANGELOG)

https://github.com/AOEpeople/crawler/blob/main/CHANGELOG.md
9.5.14 - 11.0.99 Download
9.2.4 / stable
April 09, 2021
[RELEASE] Crawler version 9.2.4

Fixes:
* Date parsing for immediate queue entries
9.5.14 - 11.0.99 Download
9.2.3 / stable
March 29, 2021
[RELEASE] Crawler version 9.2.3
9.5.14 - 11.0.99 Download
9.2.2 / stable
January 26, 2021
[RELEASE] Crawler version 9.2.2

Added
* AccessService::hasGroupAccess()

Deprecated Functions & Properties
* CrawlerController->CLI_releaseProcesses()
* CrawlerController->hasGroupAccess()

Fixed
* buildQueue with `--mode exec` resulted in a 503
* Set content in `fetchUrlContents()` to false if null, to prevent serializing from failing
9.5.14 - 11.0.99 Download
9.2.1 / stable
December 27, 2020
[RELEASE] Crawler version 9.2.1

Changed
* Moved HTML from the LogRequestForm->drawLog_addRows() to Fluid-template
* Extended the Example for PageTS Configuration

Deprecated Functions
* SignalSlotUtility->emitSignal()

Deprecated Classes
* SignalSlotUtility

Fixed
* PageTS Crawler configurations (broken since 9.1.3)
9.5.14 - 11.0.99 Download
9.2.0 / stable
December 22, 2020
[RELEASE] Crawler version 9.2.0

Added
* Add support for TYPO3 11.0

Deprecated Classes
* CrawlerApi
9.5.14 - 11.0.99 Download
9.1.5 / stable
December 19, 2020
[RELEASE] Crawler version 9.1.5

Added
* Error-Handling when Page 0 is used for crawler:buildQueue

Changed
* Removed all references to cHash as it is not used anymore.

Deprecated Functions & Properties
* CrawlerController->getLogEntriesForPageId()
* CrawlerController->CLI_runHooks()
* QueueRepository->countAllByProcessId()
* QueueRepository->countUnprocessedItems()
* FlushQueueCommand --page option

Fixed
* Typecasting port number in cli/bootstrap to have processQueue working with non-standard ports
* Flush entire queue works again
* Check if PageRow['uid'] is integer before handing it on to next function
9.5.14 - 10.4.99 Download
9.1.4 / stable
November 27, 2020
[RELEASE] Crawler version 9.1.4

Changed
* IndexedSearchCrawlerHook::class is marked as deprecated and will be removed when dropping support for TYPO3 9LTS and 10LTS

Deprecated Classes
* IndexedSearchCrawlerHook

Deprecated Functions & Properties
* CrawlerController->getDuplicateRowsIfExist()

Fixed
* Auto-loading for non-composer projects fixed
9.5.14 - 10.4.99 Download
9.1.3 / stable
November 20, 2020
[RELEASE] Crawler version 9.1.3

Added
* QueueFilter to operate with Object instead of string/arrays

Deprecated
Functions & Properties

* BackendModule->modMenu()
* CrawlerController->CLI_debug()
* CrawlerController->getAccessMode()
* CrawlerController->setAccessMode()
* CrawlerController->getDisabled()
* CrawlerController->setDisabled()
* CrawlerController->getProcessFilename()
* CrawlerController->setProcessFilename()
* CrawlerController->accessMode
* CrawlerController->processFilename

Changed
* Updated Rector, PHPStan, ECS for better static code analysis
* BackendModule is splittet into smaller classes to improve readability and maintainability
* Crawler Class added, this can be used to check whether the crawler is enabled or disabled

Fixed
* aoe/crawler/initialization (middleware) is moved to before typo3/cms-core/normalizedParams to have crawler being "last" in middleware chain
9.5.14 - 10.4.99 Download
9.1.2 / stable
November 07, 2020
[RELEASE] Crawler version 9.1.2

Added
* Progress bar to the crawler:buildQueue command output when using with --mode exec
* Improve documentation about proper crawler configuration for user with Website Usergroups

Fixed
* Detailed process views is callable again
* Makes sure the QueueRepository is always set when needed in Domain/Model/Process
* Crawling with FE-Groups is correct initialized with both TYPO3 9 & 10
9.5.14 - 10.4.99 Download
9.1.1 / stable
October 17, 2020
[RELEASE] Crawler version 9.1.1

Added
* Documentation example for ext:news
* CrawlStrategyFactory to move login out of the QueueExecutor

Deprecated
Functions & Properties
* ProcessRepository->countActive()
* ProcessRepository->getLimitFromItemCountAndOffset()
* CrawlerController->getUrlFromPageAndQueryParameters()

Changed
* UrlService->getUrlFromPageAndQueryParameters() moved from CrawlerController

Fixed
* Frontend User initialization with UserGroups for crawling protected pages
* Making sure PageUid added with ExcludeString is kept as integers
* Instantiation of ProcessRepository and QueueRepository change to GeneralUtility::makeInstance
* Ensure that DataHandlerHook will not add pages to queue that does not exist
9.5.14 - 10.4.99 Download
9.1.0 / stable
August 02, 2020
[RELEASE] Crawler version 9.1.0

Added:
* Adds light red background color + icons to crawler log rows with errors
* Crawler processing by page priority
* Automatically adding pages being edited or a page caches is cleared to the crawler queue
9.5.14 - 10.4.99 Download
9.0.3 / stable
July 20, 2020
[RELEASE] Crawler version 9.0.3

Added:
* More information into CONTRIBUTING.md about development using container

Fixed:
* Load crawler initialization before TSFE rendering preparation
9.5.14 - 10.4.99 Download
9.0.2 / stable
July 13, 2020
[RELEASE] Crawler version 9.0.2

Added:
* Update specific doktypes for skipping page
* Configure commands in Services.yaml for TYPO3 v10
* Tests for BackendModule
* Add index for tx_crawler_queue.scheduled table field
* Support for Symfony console 5
* Add example for pageVeto hook in documentation

Changed:
* Group TCA in tabs and add descriptions to fields

Fixed:
* Handle empty response when fetching URL
* Set default value when procInstrFilter is not defined
9.5.14 - 10.4.99 Download
9.0.1 / stable
May 11, 2020
[RELEASE] Crawler version 9.0.1

* Changed result handling for Crawling in case of 500 errors on directRequests
* Set correct tableName for mountPoint queries
* Fallback to alturl is added to ensure crawling of pages including PDF files
* Get fe_user from $GLOBALS['TSFE'] instead of $request->getAttribute()
* Have getTimeForFirstItem() and getTimeForLastItem return 0 is no process found

Please read
https://github.com/AOEpeople/crawler/blob/master/CHANGELOG.md for more
information.
9.5.14 - 10.4.99 Download
9.0.0 / stable
April 14, 2020
[RELEASE] Crawler version 9.0.0

* Adds support for TYPO3 9 LTS
* Adds support for TYPO3 ^10.3
* Drops support for TYPO3 7.6
* Drops support for TYPO3 8.7
* Drops support for < PHP 7.2

For more details check:
https://github.com/AOEpeople/crawler/blob/master/CHANGELOG.md#version-900
9.5.14 - 10.4.99 Download
6.7.3 / stable
November 26, 2020
[RELEASE] Crawler version 6.7.3

Adjust SQL to support MySQL Strict-Mode
7.6.0 - 8.7.99 Download
6.7.1 / stable
March 28, 2020
[RELEASE] Crawler version 6.7.1

[BUGFIX] Switches $sleepAfter and $sleeptime
[BUGFIX] Match force ssl property in model with tca (#494)
[FEATURE] add _RECURSIVE option to the subpartParams Configuration of the lookup _TABLE instructions (#476)
7.6.0 - 8.7.99 Download
6.7.0 / stable
November 08, 2019
[RELEASE] Release of crawler 6.7.0

[BUGFIX] Populate Queue object before returning it from repository (#473)
[DOC] Adds documentation for Command Controllers
7.6.0 - 8.7.99 Download
6.6.5 / stable
October 16, 2019
[RELEASE] Release of crawler 6.6.5

[BUGFIX] Remove obsolete HTML and adjust tables
[BUGFIX] Adjust table styling to match both TYPO3 7 & 8
[BUGFIX] Resolve typo3cms non-composer path for typo3_console < v5 (#469)
[DOC] Update issue template for security issues (#468)
[BUGFIX] Resolving typo3cms path if composer not used (#446) (#449)
[BUGFIX] Change the SignalSlots to call by reference
7.6.0 - 8.7.99 Download
6.6.4 / stable
August 23, 2019
[RELEASE] Release of crawler 6.6.4

[BUGFIX] PHP Warning in CrawlerController in CLI mode (#307)
[BUGFIX] Make sure that Process is added and mapped correctly (#413)
[BUGFIX] Change the calculation of composer-bin-dir (#416)
7.6.0 - 8.7.99 Download
6.6.3 / stable
August 21, 2019
[RELEASE] Release of crawler 6.6.3

[BUGFIX] Changed findByUid() to findByProcessId() (#408)
[BUGFIX] Change type declaration for pidsonly
[BUGFIX] Changing the getters/setters for configuration model (#407)
7.6.0 - 8.7.99 Download
6.6.2 / stable
August 08, 2019
[RELEASE] Release of crawler 6.6.2

[BUGFIX] Change incorrect mocking and function calls (#404)
7.6.0 - 8.7.99 Download
6.6.1 / stable
August 07, 2019
[RELEASE] Release of crawler 6.6.1

[BUGFIX] Use correct method names
7.6.0 - 8.7.99 Download
6.6.0 / stable
August 06, 2019
[RELEASE] Release of crawler 6.6.0

[TASK] Add extension-release to composer scripts
[TASK] Setting up Travis CI with Deployment to TER on tags (#400)
[BUGFIX] Change check for CLI request or backend request in CrawlerCommandController
[BUGFIX] Resolves the bin-dir from composer.json (#357)
[TASK] Add tests for expandParameters (#391)
[TASK] Deprecated functions not longer needed (#390)
[TASK] Update documentation rendering to match new setup from docs-team (#389)
[TASK] Deprecated functions (#387)
[BUGFIX] Composer: Move system dependency (#386)
[TASK] Add tests for CrawlerController (#385)
[TASK] Update PHP Requirements (#382)
[FEATURE] Use Extbase Domain/Model + Repository #323 (#328)
7.6.0 - 8.7.99 Download
6.5.0 / stable
July 08, 2019
[RELEASE] Release of crawler 6.5.0

[BUGFIX] Error 'processing_instruction_filter has no default value' (#381)
[TASK] Update composer dependencies

7.6.0 - 8.7.99 Download
6.4.1 / stable
May 08, 2019
[RELEASE] Release of crawler 6.4.1

[BUGFIX] Changed TCA Label for hidden checkbox (#321)
[BUGFIX] Adds version contraint to resolve missing (#352)
7.6.0 - 8.7.99 Download
6.4.0 / stable
May 02, 2019
[RELEASE] Release of crawler 6.4.0

[FEATURE] Deprecates EventDispatcher + Adds Signal Slots (#252)
[BUGFIX] Removes deprecated functions
[CLEANUP] Fix typo in SIGNAL constants
[CLEANUP] Fix problems with PHP 7.2
7.6.0 - 8.7.99 Download
6.3.2 / stable
March 29, 2019
[RELEASE] Release of crawler 6.3.2

[BUGFIX] Adjust getConfigurationKeys to match old usages (#341)
7.6.0 - 8.7.99 Download
6.3.1 / stable
March 25, 2019
[RELEASE] Release of crawler 6.3.1

[TASK] Updates helhum/typo3-console dependencies - Thanks @stucki
7.6.0 - 8.7.99 Download
6.3.0 / stable
March 08, 2019
[RELEASE] Release of crawler 6.3.0

[FEATURE] Implements CommandController, deprecates the CommandLineControllers #322
7.6.0 - 8.7.99 Download
6.2.0 / stable
January 23, 2019
[RELEASE] Release of crawler 6.2.0

Changelog:

[BUGFIX] Prevents a potential SQL injection vulnerability (#333)
[TASK] Re-adds type hint (#10) (#303)
[TASK] Change NOT NULL for pidsonly field (#318)
[BUGFIX] Avoid warning if $q_entry['result_data'] is em
7.6.0 - 8.7.99 Download
6.1.2 / stable
May 09, 2018
[RELEASE] Release of crawler 6.1.2

Changelog:
[BUGFIX] Only execute getSelectedState if $task != null (#279) (#296)
[BUGFIX] Fixes $task is null (#287)
[BUGFIX] Create file which is expected by TYPO3 CMS (#262)
[BUGFIX] Switched Refresh-icon to use Icon
7.6.0 - 8.7.99 Download
6.1.1 / stable
February 07, 2018
[RELEASE] Crawler 6.1.1
7.6.0 - 8.7.99 Download
6.1.0 / stable
January 08, 2018
[RELEASE] Release of crawler 6.1.0

Changelog:
[BUGFIX] Crawler only crawls first level (#261)
[TASK] Add Warn when no configuration is selected (#191)
[CLEANUP] Adjust variable names to reflect renamed Classes (#250)
[FEATURE] Configuration_hash for carw
7.6.0 - 8.7.99 Download
6.0.0 / stable
December 21, 2017
[RELEASE] Release of crawler 6.0.0

Changelog:
[FEATURE] Introduce namespaces (#226)
[BUGFIX] Changed Url-generation for pagination (#89) (#246)
[CLEANUP] Remove legacy TYPO3 CMS 4.x XCLASS registration (#243)
[BUGFIX] Makes starting a new cli process in
7.6.0 - 8.7.99 Download
5.2.1 / stable
December 04, 2017
[RELEASE] Release of crawler 5.2.1

Changelog:
[CLEANUP] Change PHP requirements + updated testing (#228)
[CLEANUP] Remove function call deprecated in TYPO3 7.6+ (#222)
[BUGFIX] Keeps Crawler Info on current page on refresh (#198)
[CLEANUP] Remove hhvm fr
6.2.0 - 7.6.99 Download
5.2.0 / stable
November 06, 2017
[RELEASE] Release of crawler 5.2.0

* [TASK] add post process signal slot also for use with crawler_im (#211)
* [FEATURE] TypoScript Root Template must be configured in crawler configuration
* [TASK] Add CSS class for AdditionalFieldProviders for TYPO3 7.
6.2.0 - 7.6.99 Download
5.1.4 / stable
October 09, 2017
[RELEASE] Release of crawler 5.1.4

** Release Notes **

* [CLEANUP] Adjust version constraints
* [FEATURE] Use actual page ID for TSFE initialization (#214)
* [TASK] Set more realistic timeouts (30 min should be sufficient) (#213)
* Cleanup/codestyle (#2
6.2.0 - 7.6.99 Download
5.1.3 / stable
July 21, 2017
[RELEASE] Release of crawler 5.1.3

[TASK] Added Cache to TypoScriptUtility
6.2.0 - 7.99.99 Download
5.1.2 / stable
November 24, 2016
[RELEASE] Release of crawler 5.1.2

[BUGFIX] Scheduler Tasks broken after upgrade to 5.1.1 (composer) (#154)
[BUGFIX] Fixed incorrect class name in ClassAliasMap (#153)
6.2.0 - 7.99.99 Download
5.1.1 / stable
November 21, 2016
[RELEASE] Release of crawler 5.1.1

[BUGFIX] It's not possible to "add process" after upgrade to 5.1.0 (#145)
[BUGFIX] #1442236317: MenuItem "" is not valid (More information) (#148)
[BUGFIX] Not possible to add new scheduler tasks (not only Crawler tasks
6.2.0 - 7.99.99 Download
5.1.0 / stable
November 15, 2016
[RELEASE] Release of crawler 5.1.0

2016-11-08 28961a0 [FEATURE] Remove assignment of queue items to process if process gets killed by cleanup hook
2016-11-13 4d82941 [FEATURE] Adjust namespace of ProcessCleanupTask (#138)
2016-11-08 714a222 [CLEANU
6.2.0 - 7.99.99 Download
5.0.9 / stable
September 16, 2016
[RELEASE] Release of crawler 5.0.9
6.2.0 - 7.99.99 Download
5.0.8 / stable
September 16, 2016
[RELEASE] Release of crawler 5.0.8
6.2.0 - 7.99.99 Download
5.0.7 / stable
September 06, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.6 / stable
August 22, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.5 / stable
August 15, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.4 / stable
March 18, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.3 / stable
March 10, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.2 / stable
January 26, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.1 / stable
December 08, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
5.0.0 / stable
October 27, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download
4.1.1 / stable
March 10, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download
4.1.0 / stable
October 27, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download
4.0.0 / stable
August 11, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download
3.5.0 / stable
October 22, 2012
- Update Documentation
- Add Multiprocessing CLI
- Add Feature to use sys_domain record for BaseURL
- Added UnitTests
- Enable Crawling with HTTPS
- Improve Scheduler Tasks
- Bug-Fixes
Download
3.2.0 / stable
June 29, 2010
Improve scheduler integration, minor bug fixes.
Download
3.1.2 / stable
April 19, 2010
Improve handling with "t3lib_FlashMessage", Update documentation, Reimplement "Result Log" column.
Download
3.1.1 / stable
April 01, 2010
Fix documentation
Download
3.1.0 / stable
March 31, 2010
Update documentation
Improved scheduler and TYPO3 4.3 integration
Fixed several bugs
Add more tests

Download
3.0.5 / beta
February 27, 2010
Release notes:

* TYPO3 4.3 integration
* Scheduler integration
* CLI API cleanup
* Bugfixes

Download
3.0.0 / alpha
September 10, 2009
- multiprocessing
- new way to configure
- see manual for details
Download
2.1.0 / alpha
July 24, 2008
diffrent options in em to control sleeptimes / change "keep-alive" header / add debugmode / readurl fixes

Download
2.0.0 / alpha
March 23, 2007
Updated various features, see manual
Download
1.1.0 / alpha
March 21, 2006

Download
1.0.0 / alpha
February 16, 2006

Download
0.0.1 / alpha
December 22, 2005
[Dont download yet - its only for beta testing before 4.0]
Download