The Aspider Web Crawler Connector can be configured using the Rest API. It requires the following entities to be created:

Credential
Connection
Connector
Seed

Below are the examples of how to create the Connection and the Seed. For the Connector please check this page.

Easy Heading Free

navigationTitle	On this Page
wrapNavigationText	true
navigationExpandOption	expand-all-by-default

Create Credential (Common)

Field	Required	Default	Multiple	Notes	Example
type	Yes	-	No	The value must be "aspider".	"aspider"
description	Yes	-	No	Name of the credential object.	"AspiderCredential"
properties	Yes	-	No	Configuration object
useSelenium	No	false	No	Flag to let aspider know if it has to set up Selenium	true / false
webDriverImplementation	Yes	-	No	Browser used by selenium. Possible values: CHROME FIREFOX Note: Only used if useSelenium is set to true.	"CHROME" "FIREFOX"
webDriverPath	Yes	-	No	Path to the selenium web driver executable. Note: The driver must have execution permission. Note: Only used if useSelenium is set to true.	"lib\\chromedriver.exe"
headlessMode	Yes	-	No	Flag to start the browser on headless mode (no GUI). Note: Only used if useSelenium is set to true.	true / false
authMech	No	[]	Yes	Array containing the authentication mechanisms	[]
host	No	""	No	Hostname where the authentication mechanism should be used. Note: If empty, the authentication mechanism will be used against any host.	"example.com"
port	Yes	-1	No	Port where the authentication mechanism should be used. Note: -1 means the URL can have any port.	8000
scheme	Yes	-	No	Scheme to use during the authentication. Possible values: Basic Digest NTLM Negotiate Forms Selenium	"Basic" "Digest" "NTLM" "Negotiate" "Forms" "Selenium"
user	Yes	-	No	Name of the account to authenticate with.	"Administrator"
password	Yes	-	No	Password to authenticate with. See Encryption API for more information.	"123456abC"
domain	No	""	No	Domain of the account used to authenticate with.	"EXAMPLE"
realm	No	""	No	Realm of the account to authenticate with.	"my-realm"

Create Credential - NTLM specific fields

Field	Required	Default	Multiple	Notes	Example
adfs	Yes	false	No	Flag to indicate if ADFS should be used, only required when *scheme* is "NTLM".	true / false

Create Credential - Negotiate specific fields

Field	Required	Default	Multiple	Notes	Example
useDefaultKrb5	No	true	No	Flag to indicate if Aspider should use the system settings for Kerberos.	true / false
kdc	Yes	-	No	Hostname of the key distribution center to get the Kerberos tickets.	"kdc.example.com"
verbose	Yes	false	No	Flag to indicate if the entire negotiation process should be logged.	true / false

Create Credential - Forms specific fields

Field	Required	Default	Multiple	Notes	Example
loginUrl	Yes	-	No	URL of the login page	"https://example.com/login"
formPath	Yes	-	No	CSS Selector for getting the login form.	"#content > form"
userField	Yes	-	No	Id of the username field	"txtUser"
passwordField	Yes	-	No	Id of the password field	"txtPass"
adfs	No	false	No	Flag to enable the ADFS flow of requests during authentication.	true / false
saml	No	false	No	Flag to enable the SAML flow of requests during authentication.	true / false
retries	Yes	-	No	Number of retries to do if the authentication fails.	5
customField	No	[]	Yes	Array of other fields in the form	[]
name	Yes	-	No	Name of the field in the form	"myField"
value	Yes	-	No	Value of the field in the form	"myValue"

Create Credential - Selenium specific fields

Field	Required	Default	Multiple	Notes	Example
loginUrl	Yes	-	No	URL of the login page	"https://example.com/login"
user	Yes	-	No	username to authenticate with	"user"
password	Yes	-	No	password	"123456abC"
loginScript	No	-	No	Script with the instructions to log in. Avalable variables: seedId: String, Id of the current seed logger: ALogger, logger implementation driver: WebDriver, Selenium web driver, used for interacting with the browser loginUrl: String, URL of the log in page user: String, username for authentication password: String, password for authentication
sessionScript	No	-	No	Script with the instructions to validate the session, must return true if the session is valid. Avalable variables see above.

Example

Code Block

theme	RDark
title	POST aspire/_api/credentials

{
    "type": "Aspider",
    "description": "AspiderCredential",
    "properties": {
      "useSelenium": true,
      "webDriverImplementation": "CHROME",
      "webDriverPath": "/dev/chromedriver.exe",
      "headlessMode": false,
      "authMech": [
       {
         "host": "chessbase.com",
         "port": -1,
         "scheme": "Basic",
         "user": "userP",
         "password": "passwd",
         "domain": "",
         "realm": ""
       }
     ]
   }
 }

Update Credential

Field	Required	Default	Multiple	Notes	Example
id	Yes	-	No	Id of the credential to update.	"2f287669-d163-4e35-ad17-6bbfe9df3778"
description	Yes	-	No	Name of the credential object.	"Aspider Credential"
properties	Yes	-	No	Configuration object
(see create credential)

Example

Code Block

theme	RDark
title	PUT aspire/_api/credentials/2a5ca234-e328-4d40-bb2a-2df3e550b065

{
    "id": "2a5ca234-e328-4d40-bb2a-2df3e550b065",
    "description": "AspiderCredential",
     "properties": {
      "useSelenium": true,
      "webDriverImplementation": "CHROME",
      "webDriverPath": "/dev/chromedriver.exe",
      "headlessMode": false,
      "authMech": [
       {
         "host": "chessbase.com",
         "port": -1,
         "scheme": "Basic",
         "user": "userP",
         "password": "passwd",
         "domain": "",
         "realm": ""
       }
     ]
   }
 }

Create Connection

extractJavaScript

Field	Required	Default	Multiple	Notes	Example
type	Yes	-	No	The value must be "aspider".	"aspider"
description	Yes	-	No	Name of the connection object.	"My Aspider connection"
credential	Yes	-	No	Id of the credential assigned to this object.	"2a5ca234-e328-4d40-bb2a-2df3e550b065"
throttlePolicy	No	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	Yes	-	No	Configuration object
				Scope
crawlScope	yes	HOST	No	Determines the scope of the crawl. The values are HOST, EVERYTHING, CUSTOM. Everything will allow all URLs to be crawled. Host only will allow only URLs like 'abc.domain.com' when your seeds contain 'abc.domain.com'. Custom scope allows you to add one or more regular expression patterns to match the host name. These scopes can be extended by the Include patterns.	"HOST"
scopePattern	yes for crawl scope "CUSTOM"	-	No	Custom scope patterns. Any URL matching the following patterns will be included as part of the scope. Pattern evaluated against the document URL.	".*\\.example.com"
userAgent	yes	Aspider - The Aspire Web Crawler	No	User Agent request header to identify the web crawler.	"Aspider - The Aspire Web Crawler"
obeyRobots	yesno	true	No	If checked, the crawler will obey the robots.txt restrictions of each site.	true
obeyMetaRobots	yesno	true	No	If checked, the crawler will obey the HTML robots meta tags.	true
caseSensitiveUrls	yesno	true	No	If unchecked all URLs will be transformed into lower-case before processing.	true
maxHops	yes	5	No	Crawl Depth. How many hops from the seed is the crawler allowed to follow links.	5
maxOutLinks	yes	6000	No	Maximum number of links to be reported by a single page.	6000
extractValuePropsfollowRedirects	no	false	includes	excludes	No	Extract value attributes. If checked, the crawler will extract links from value attributes (i.e. input tags).	yes
followRedirects	no	false	No	If checked, redirects will be followed and content will be set as it was from original URL. Otherwise the redirects will enqueue new document.	yes
extractJavaScript	no	true	No	Crawl JavaScript URIs. If checked, in-page Javascript is scanned for strings that appear likely to be URIs. This typically finds both valid and invalid URIs.	yes
includes	no	-	Yes	The document will be processed by the connector if it matches one of the following patterns. Pattern evaluated against the document URL.	".*\\.pdf"
excludes	no	-	Yes	The document will not be processed by the connector if it matches one of the following patterns. Pattern evaluated against the document URL.	".*\\.xml"
scanExcludedItems	no	false	No	Scan excluded pages". If checked, the crawler will scan the links of pages that have been excluded by a pattern (because it matches an exclude pattern or because it doesn't match an include pattern).	true
absoluteExclude	yes for scanExcludedItems = true	-	Yes	Do not follow patterns. URL patterns that the crawler must not scan (follow). This only applies to items marked as excluded by the include/exclude rules.	".*\\.xml$"scanExcludedItems
				Document processing
cleanupRule
				Crawler
maxContentSize
showNon200AsErrors
stopOnScanError
logCrawledUrls
debugContentOutput
incrementalUrlCleanupRegex
excludeMultimedia
				Connection
requestHeader
trustAllCertificates
connectionTimeout
connectionRequestTimeout
socketTimeout
useProxy
				Security
staticAcl

Example

Code Block

theme	RDark
title	POST aspire/_api/connections

{
    "type": "Aspider",
    "description": "Aspider Test Connector",
	"credential": "2a5ca234-e328-4d40-bb2a-2df3e550b065",	
    "properties": {
      "crawlScope": "CUSTOM",
      "scopePattern": ".*\\.example.com",
      "userAgent": "Aspider - The Aspire Web Crawler",
      "obeyRobots": true,
      "obeyMetaRobots": true,
      "caseSensitiveUrls": true,
      "maxHops": 5,
      "maxOutLinks": 6000,
      "extractValueProps": true,
      "followRedirects": true,
      "extractJavaScript": true,
      "includes": ".*\\.pdf",
      "excludes": ".*\\.xml",
      "scanExcludedItems": true,
      "absoluteExclude": ".*\\.xml$",
      "cleanupRule": [
          {
            "urlPattern": ".*\\xml",
            "contentTypes": "text/html\\.*",
            "noIndexClassnames": "nofollow",
            "cleanupPattern": "<!--noindex-->",
            "cleanupBeforeExtraction": true
           }
         ],
      "maxContentSize": "10mb",
      "showNon200AsErrors": true,
      "stopOnScanError": true,
      "logCrawledUrls": true,
      "debugContentOutput": true,
      "incrementalUrlCleanupRegex": "\\?.*",
      "excludeMultimedia": true,
      "requestHeader": [
        {
          "header": "customHeader",
          "value": "val"
         }
       ],
      "trustAllCertificates": false,
      "connectTimeout": "10s",
      "connectionRequestTimeout": "10s",
      "socketTimeout": "10s",
      "useProxy": true,
      "proxyHost": "proxy.domain",
      "proxyPort": 8080,
      "proxyAuthentication": "NTLM",
      "proxyDomain": "sss",
      "proxyUser": "sss",
      "proxyPassword": "sss",
      "staticAcl": [
         {
           "name": "acl",
           "domain": "domain",
           "entity": "group",
           "access": "allow"
          }
        ]
     }
}

Update Connection

Field	Required	Default	Multiple	Notes	Example
id	Yes	-	No	Id of the connection to update	"89d6632a-a296-426c-adb0-d442adcab4b0",
description	No	-	No	Name of the connection object.	"MyAspiderConnection"
credential	No	-	No	Id of the credential assigned to this object.	"2a5ca234-e328-4d40-bb2a-2df3e550b065"
throttlePolicy	No	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	Yes	-	No	Configuration object

Example

Code Block

theme	RDark
title	PUT aspire/_api/connections/89d6632a-a296-426c-adb0-d442adcab4b0

{
    "id": "89d6632a-a296-426c-adb0-d442adcab4b0",
    "description": "Aspider Test Connector",
	"credential": "2a5ca234-e328-4d40-bb2a-2df3e550b065",
    "properties": {
        
    }
}

Create Connector Instance

For the creation of the Connector object using the Rest API check this page

Update Connector Instance

For the update of the Connector object using the Rest API check this page

Create Seed

Field	Required	Default	Multiple	Notes	Example
seed	Yes	-	No	<seed description>
type	Yes	-	No	The value must be "<connector>".	"<connector>"
description	Yes	-	No	Name of the seed object.	"My<connector>Seed"
connector	Yes	-	No	The id of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection	Yes	-	No	The id of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	No	[ ]	Yes	The ids of the workflows that will be executed for the documents crawled.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	No	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids of the routing policies that this seed will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
tags	No	[ ]	Yes	The tags of the seed. These can be used to filter the seed	["tag1", "tag2"]
properties	Yes	-	No	Configuration object

Example

Code Block

theme	RDark
title	POST aspire/_api/seeds

{
    "type": "Aspider",
    "seed": "directory",
    "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
    "description": "Aspider_Test_Seed",
    "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
    "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
    "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
    "workflows": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
    "tags": ["tag1", "tag2"],
    "properties": {
        
    }
}

Update Seed

Field	Required	Default	Multiple	Notes	Example
id	Yes	-	No	Id of the seed to update.	"2f287669-d163-4e35-ad17-6bbfe9df3778"
seed	No	-	No	<seed description>
description	No	-	No	Name of the seed object.	"My<connector>Seed"
connector	No	-	No	The id of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection	No	-	No	The id of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	No	[ ]	Yes	The ids of the workflows that will be executed for the documents crawled.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
workflows.add	No	[ ]	Yes	The ids of the workflows to add.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
workflows.remove	No	[ ]	Yes	The ids of the workflows to remove.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	No	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	No	[ ]	Yes	The ids of the routing policies that this seed will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
routingPolicies.add	No	[ ]	Yes	The ids of the routingPolicies to add.	["b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
routingPolicies.remove	No	[ ]	Yes	The ids of the routingPolicies to remove.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7"]
tags	No	[ ]	Yes	The tags of the seed. These can be used to filter the seed	["tag1", "tag3"]
tags.add	No	[ ]	Yes	The tags to add	["tag4"]
tags.remove	No	[ ]	Yes	The tags to remove	["tag2"]
properties	Yes	-	No	Configuration object

Example

Code Block

theme	RDark
title	PUT aspire/_api/seeds/2f287669-d163-4e35-ad17-6bbfe9df3778

{
    "id": "2f287669-d163-4e35-ad17-6bbfe9df3778",
    "seed": "<seed example>",
    "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
    "description": "Aspider_Test_Seed",
    "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
    "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
    "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
    "workflows": ["b255e950-1dac-46dc-8f86-1238b2fbdf27", "f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
    "tags": ["tag", "tag2"],
    "properties": {
        
    }
}

Page tree

Versions Compared

Old Version 20

New Version 21

Key

Create Credential (Common)

Create Credential - NTLM specific fields

Create Credential - Negotiate specific fields

Create Credential - Forms specific fields

Create Credential - Selenium specific fields

Example

Update Credential

Create Connection

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example

Page tree

Page History

Versions Compared

Old Version 20

New Version 21

Key

Create Credential (Common)

Create Credential - NTLM specific fields

Create Credential - Negotiate specific fields

Create Credential - Forms specific fields

Create Credential - Selenium specific fields

Example

Update Credential

Create Connection

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example