“”

Site Crawler Site Crawler

crawler / stable

Libraries and scripts for crawling the TYPO3 page tree.

This version supports TYPO3

Composer support

composer req aoepeople/crawler

Tags

#crawling #cache #warmup #realurl

Last upload comment

[RELEASE] Release of crawler 6.7.0

[BUGFIX] Populate Queue object before returning it from repository (#473)
[DOC] Adds documentation for Command Controllers

Downloads by version

Download 6.7.0 Extension Manual Found an Issue? Code Insights
Author
Kasper Skaarhoj, Daniel Poetzinger, Fabrizio Branca, Tolleiv Nietsch, Timo Schmidt, Michael Klapper, Stefan Rotsch, Tomas Norre Mikkelsen
Company
AOE GmbH
Last update
08. Nov 2019
First upload
22. Dec 2005
Downloads
53,591
Category
Backend Modules
Dependencies
  • TYPO3 (7.6.0 - 8.7.99)

Installation

  1. Download ZIP file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the ZIP file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.
  1. Go to your folder where the root composer.json file is located
  2. Type: composer req aoepeople/crawler to get the latest version that runs on your TYPO3 version.
  1. Download T3X file
  2. Log into your TYPO3 backend
  3. Go to Extension Manager module
  4. Press the upload button on the top bar
  5. Select the T3X file and upload it. If you want to overwrite an existing extension installation, activate the checkbox.

Version history

Version Upload comment Works with TYPO3 Download
6.7.0 / stable
November 08, 2019
[RELEASE] Release of crawler 6.7.0

[BUGFIX] Populate Queue object before returning it from repository (#473)
[DOC] Adds documentation for Command Controllers
7.6.0 - 8.7.99 Download ZIP Archive
6.6.5 / stable
October 16, 2019
[RELEASE] Release of crawler 6.6.5

[BUGFIX] Remove obsolete HTML and adjust tables
[BUGFIX] Adjust table styling to match both TYPO3 7 & 8
[BUGFIX] Resolve typo3cms non-composer path for typo3_console < v5 (#469)
[DOC] Update issue template for security issues (#468)
[BUGFIX] Resolving typo3cms path if composer not used (#446) (#449)
[BUGFIX] Change the SignalSlots to call by reference
7.6.0 - 8.7.99 Download ZIP Archive
6.6.4 / stable
August 23, 2019
[RELEASE] Release of crawler 6.6.4

[BUGFIX] PHP Warning in CrawlerController in CLI mode (#307)
[BUGFIX] Make sure that Process is added and mapped correctly (#413)
[BUGFIX] Change the calculation of composer-bin-dir (#416)
7.6.0 - 8.7.99 Download ZIP Archive
6.6.3 / stable
August 21, 2019
[RELEASE] Release of crawler 6.6.3

[BUGFIX] Changed findByUid() to findByProcessId() (#408)
[BUGFIX] Change type declaration for pidsonly
[BUGFIX] Changing the getters/setters for configuration model (#407)
7.6.0 - 8.7.99 Download ZIP Archive
6.6.2 / stable
August 08, 2019
[RELEASE] Release of crawler 6.6.2

[BUGFIX] Change incorrect mocking and function calls (#404)
7.6.0 - 8.7.99 Download ZIP Archive
6.6.1 / stable
August 07, 2019
[RELEASE] Release of crawler 6.6.1

[BUGFIX] Use correct method names
7.6.0 - 8.7.99 Download ZIP Archive
6.6.0 / stable
August 06, 2019
[RELEASE] Release of crawler 6.6.0

[TASK] Add extension-release to composer scripts
[TASK] Setting up Travis CI with Deployment to TER on tags (#400)
[BUGFIX] Change check for CLI request or backend request in CrawlerCommandController
[BUGFIX] Resolves the bin-dir from composer.json (#357)
[TASK] Add tests for expandParameters (#391)
[TASK] Deprecated functions not longer needed (#390)
[TASK] Update documentation rendering to match new setup from docs-team (#389)
[TASK] Deprecated functions (#387)
[BUGFIX] Composer: Move system dependency (#386)
[TASK] Add tests for CrawlerController (#385)
[TASK] Update PHP Requirements (#382)
[FEATURE] Use Extbase Domain/Model + Repository #323 (#328)
7.6.0 - 8.7.99 Download ZIP Archive
6.5.0 / stable
July 08, 2019
[RELEASE] Release of crawler 6.5.0

[BUGFIX] Error 'processing_instruction_filter has no default value' (#381)
[TASK] Update composer dependencies
7.6.0 - 8.7.99 Download ZIP Archive
6.4.1 / stable
May 08, 2019
[RELEASE] Release of crawler 6.4.1

[BUGFIX] Changed TCA Label for hidden checkbox (#321)
[BUGFIX] Adds version contraint to resolve missing (#352)
7.6.0 - 8.7.99 Download ZIP Archive
6.4.0 / stable
May 02, 2019
[RELEASE] Release of crawler 6.4.0

[FEATURE] Deprecates EventDispatcher + Adds Signal Slots (#252)
[BUGFIX] Removes deprecated functions
[CLEANUP] Fix typo in SIGNAL constants
[CLEANUP] Fix problems with PHP 7.2
7.6.0 - 8.7.99 Download ZIP Archive
6.3.2 / stable
March 29, 2019
[RELEASE] Release of crawler 6.3.2

[BUGFIX] Adjust getConfigurationKeys to match old usages (#341)
7.6.0 - 8.7.99 Download ZIP Archive
6.3.1 / stable
March 25, 2019
[RELEASE] Release of crawler 6.3.1

[TASK] Updates helhum/typo3-console dependencies - Thanks @stucki
7.6.0 - 8.7.99 Download ZIP Archive
6.3.0 / stable
March 08, 2019
[RELEASE] Release of crawler 6.3.0

[FEATURE] Implements CommandController, deprecates the CommandLineControllers #322
7.6.0 - 8.7.99 Download ZIP Archive
6.2.0 / stable
January 23, 2019
[RELEASE] Release of crawler 6.2.0

Changelog:

[BUGFIX] Prevents a potential SQL injection vulnerability (#333)
[TASK] Re-adds type hint (#10) (#303)
[TASK] Change NOT NULL for pidsonly field (#318)
[BUGFIX] Avoid warning if $q_entry['result_data'] is em
7.6.0 - 8.7.99 Download ZIP Archive
6.1.2 / stable
May 09, 2018
[RELEASE] Release of crawler 6.1.2

Changelog:
[BUGFIX] Only execute getSelectedState if $task != null (#279) (#296)
[BUGFIX] Fixes $task is null (#287)
[BUGFIX] Create file which is expected by TYPO3 CMS (#262)
[BUGFIX] Switched Refresh-icon to use Icon
7.6.0 - 8.7.99 Download ZIP Archive
6.1.1 / stable
February 07, 2018
[RELEASE] Crawler 6.1.1 7.6.0 - 8.7.99 Download ZIP Archive
6.1.0 / stable
January 08, 2018
[RELEASE] Release of crawler 6.1.0

Changelog:
[BUGFIX] Crawler only crawls first level (#261)
[TASK] Add Warn when no configuration is selected (#191)
[CLEANUP] Adjust variable names to reflect renamed Classes (#250)
[FEATURE] Configuration_hash for carw
7.6.0 - 8.7.99 Download ZIP Archive
6.0.0 / stable
December 21, 2017
[RELEASE] Release of crawler 6.0.0

Changelog:
[FEATURE] Introduce namespaces (#226)
[BUGFIX] Changed Url-generation for pagination (#89) (#246)
[CLEANUP] Remove legacy TYPO3 CMS 4.x XCLASS registration (#243)
[BUGFIX] Makes starting a new cli process in
7.6.0 - 8.7.99 Download ZIP Archive
5.2.1 / stable
December 04, 2017
[RELEASE] Release of crawler 5.2.1

Changelog:
[CLEANUP] Change PHP requirements + updated testing (#228)
[CLEANUP] Remove function call deprecated in TYPO3 7.6+ (#222)
[BUGFIX] Keeps Crawler Info on current page on refresh (#198)
[CLEANUP] Remove hhvm fr
6.2.0 - 7.6.99 Download ZIP Archive
5.2.0 / stable
November 06, 2017
[RELEASE] Release of crawler 5.2.0

* [TASK] add post process signal slot also for use with crawler_im (#211)
* [FEATURE] TypoScript Root Template must be configured in crawler configuration
* [TASK] Add CSS class for AdditionalFieldProviders for TYPO3 7.
6.2.0 - 7.6.99 Download ZIP Archive
5.1.4 / stable
October 09, 2017
[RELEASE] Release of crawler 5.1.4

** Release Notes **

* [CLEANUP] Adjust version constraints
* [FEATURE] Use actual page ID for TSFE initialization (#214)
* [TASK] Set more realistic timeouts (30 min should be sufficient) (#213)
* Cleanup/codestyle (#2
6.2.0 - 7.6.99 Download ZIP Archive
5.1.3 / stable
July 21, 2017
[RELEASE] Release of crawler 5.1.3

[TASK] Added Cache to TypoScriptUtility
6.2.0 - 7.99.99 Download ZIP Archive
5.1.2 / stable
November 24, 2016
[RELEASE] Release of crawler 5.1.2

[BUGFIX] Scheduler Tasks broken after upgrade to 5.1.1 (composer) (#154)
[BUGFIX] Fixed incorrect class name in ClassAliasMap (#153)
6.2.0 - 7.99.99 Download ZIP Archive
5.1.1 / stable
November 21, 2016
[RELEASE] Release of crawler 5.1.1

[BUGFIX] It's not possible to "add process" after upgrade to 5.1.0 (#145)
[BUGFIX] #1442236317: MenuItem "" is not valid (More information) (#148)
[BUGFIX] Not possible to add new scheduler tasks (not only Crawler tasks
6.2.0 - 7.99.99 Download ZIP Archive
5.1.0 / stable
November 15, 2016
[RELEASE] Release of crawler 5.1.0

2016-11-08 28961a0 [FEATURE] Remove assignment of queue items to process if process gets killed by cleanup hook
2016-11-13 4d82941 [FEATURE] Adjust namespace of ProcessCleanupTask (#138)
2016-11-08 714a222 [CLEANU
6.2.0 - 7.99.99 Download ZIP Archive
5.0.9 / stable
September 16, 2016
[RELEASE] Release of crawler 5.0.9 6.2.0 - 7.99.99 Download ZIP Archive
5.0.8 / stable
September 16, 2016
[RELEASE] Release of crawler 5.0.8 6.2.0 - 7.99.99 Download ZIP Archive
5.0.7 / stable
September 06, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.6 / stable
August 22, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.5 / stable
August 15, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.4 / stable
March 18, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.3 / stable
March 10, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.2 / stable
January 26, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.1 / stable
December 08, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
5.0.0 / stable
October 27, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
6.2.0 - 7.99.99 Download ZIP Archive
4.1.1 / stable
March 10, 2016
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download ZIP Archive
4.1.0 / stable
October 27, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download ZIP Archive
4.0.0 / stable
August 11, 2015
New TER Release

git-svn-id: https://svn.typo3.org/TYPO3v4/Extensions/crawler/tags/V_3-5-0@67349 735d13b6-9817-0410-8766-e36946ffe9aa
4.5.0 - 6.2.99 Download ZIP Archive
3.5.0 / stable
October 22, 2012
- Update Documentation
- Add Multiprocessing CLI
- Add Feature to use sys_domain record for BaseURL
- Added UnitTests
- Enable Crawling with HTTPS
- Improve Scheduler Tasks
- Bug-Fixes
Download ZIP Archive
3.2.0 / stable
June 29, 2010
Improve scheduler integration, minor bug fixes. Download ZIP Archive
3.1.2 / stable
April 19, 2010
Improve handling with "t3lib_FlashMessage", Update documentation, Reimplement "Result Log" column. Download ZIP Archive
3.1.1 / stable
April 01, 2010
Fix documentation Download ZIP Archive
3.1.0 / stable
March 31, 2010
Update documentation
Improved scheduler and TYPO3 4.3 integration
Fixed several bugs
Add more tests
Download ZIP Archive
3.0.5 / beta
February 27, 2010
Release notes:

* TYPO3 4.3 integration
* Scheduler integration
* CLI API cleanup
* Bugfixes
Download ZIP Archive
3.0.0 / alpha
September 10, 2009
- multiprocessing
- new way to configure
- see manual for details
Download ZIP Archive
2.1.0 / alpha
July 24, 2008
diffrent options in em to control sleeptimes / change "keep-alive" header / add debugmode / readurl fixes
Download ZIP Archive
2.0.0 / alpha
March 23, 2007
Updated various features, see manual Download ZIP Archive
1.1.0 / alpha
March 21, 2006
Download ZIP Archive
1.0.0 / alpha
February 16, 2006
Download ZIP Archive
0.0.1 / alpha
December 22, 2005
[Dont download yet - its only for beta testing before 4.0] Download ZIP Archive