Spider: Version 0.1.0

Spider is a complete standalone Java application designed to easily integrate varied datasources.

  • XML driven framework for data retrieval from network accessible sources
  • Scheduled pulling
  • Highly extensible
  • Provides hooks for custom post-processing and configuration
  • Implemented as a Avalon/Keel framework datafeed service

Included Core Connectors:

  • Files and Zip Archives via HTTP/FTP/HTTPS/FileSystem
  • Supports access via links described as literals or regular expressions
  • Supports sessions/cookies/form parameters
Included Optional Connectors:
  • Axis (SOAP webservices)