The Aspider Web Crawler Connector can be configured using the Rest API. It requires the following entities to be created:

Credential
Connection
Connector
Seed

Below are the examples of how to create the Connection and the Seed. For the Connector please check this page.

Easy Heading Free

navigationTitle	On this Page
wrapNavigationText	true
navigationExpandOption	expand-all-by-default

Create Credential (Common)

Field	Optional	Default	Multiple	Notes	Example
type	No	-	No	The value must be "<connector>aspider".	"<connector>aspider"
description	No	-	No	Name of the credential object.	"<connector>CredentialAspiderCredential"
properties	No	-	No	Configuration object
useSelenium	Yes	false	No	Flag to let aspider know if it has to set up Selenium	true / false
webDriverImplementation	No	-	No	Browser used by selenium. Possible values: CHROME FIREFOX Note: Only used if useSelenium is set to true.	"CHROME" "FIREFOX"
webDriverPath	No	-	No	Path to the selenium web driver executable. Note: The driver must have execution permission. Note: Only used if useSelenium is set to true.	"lib\\chromedriver.exe"
headlessMode	No	-	No	Flag to start the browser on headless mode (no GUI). Note: Only used if useSelenium is set to true.	true / false
authMech	No	[]	Yes	Array containing the authentication mechanisms	[]
host	Yes	""
port	Yes	-1
scheme	No	-	No	Scheme to use during the authentication. Possible values: Basic Digest NTLM Negotiate Forms Selenium	"Basic" "Digest" "NTLM" "Negotiate" "Forms" "Selenium"
user	No	-
password	No	-
domain	Yes	""
realm	No	""

Create Credential - NTLM specific fields

Field	Optional	Default	Multiple	Notes	Example
adfs	No	false	No	Flag to indicate if ADFS should be used, only required when *scheme* is "NTLM".	true / false

Create Credential - Negotiate specific fields

Field	Optional	Default	Multiple	Notes	Example
useDefaultKrb5	No	true	No	Flag to indicate if Aspider should use the system settings for Kerberos.	true / false
kdc	Yes	-	No	Hostname of the key distribution center to get the Kerberos tickets.	"kdc.example.com"
verbose	No	false	No	Flag to indicate if the entire negotiation process should be logged.	true / false

Create Credential - Forms specific fields

Field	Optional	Default	Multiple	Notes	Example
loginUrl	Yes	-	No	URL of the login page	"https://example.com/login"
formPath	Yes	-	No	CSS Selector for getting the login form.	"#content > form"
userField	Yes	-	No	Id of the username field	"txtUser"
passwordField	Yes	-	No	Id of the password field	"txtPass"
adfs	No	false	No		true / false
saml	No	false	No		true / false
retries	Yes	-	No	Number of retries to do if the authentication fails.	5
customField	No	[]	Yes	Array of other fields in the form	[]
name	Yes	-	No	Name of the field in the form	"myField"
value	Yes	-	No	Value of the field in the form	"myValue"

Example

Code Block

theme	RDark
title	POST aspire/_api/credentials

{
    "type": "<connector>",
    "description": "<connector>Credential",
    "properties": {
		
    }
}

Update Credential

Field	Optional	Default	Multiple	Notes	Example
id	No	-	No	Id of the credential to update.	"2f287669-d163-4e35-ad17-6bbfe9df3778"
description	No	-	No	Name of the credential object.	"<connector>Credential"
properties	No	-	No	Configuration object

Example

Code Block

theme	RDark
title	PUT aspire/_api/credentials/2a5ca234-e328-4d40-bb2a-2df3e550b065

{
    "id": "2a5ca234-e328-4d40-bb2a-2df3e550b065",
    "description": "<connector>Credential",
    "properties": {
		
    }
}

Create Connection

Field	Optional	Default	Multiple	Notes	Example
type	No	-	No	The value must be "<connector>".	"<connector>"
description	No	-	No	Name of the connection object.	"My<connector>Connection"
credential	No	-	No	Id of the credential assigned to this object.	"2a5ca234-e328-4d40-bb2a-2df3e550b065"
throttlePolicy	Yes	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	Yes	[ ]	Yes	The ids of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	No	-	No	Configuration object

Example

Code Block

theme	RDark
title	POST aspire/_api/connections

{
    "type": "<connector>",
    "description": "<connector> Test Connector",
	"credential": "2a5ca234-e328-4d40-bb2a-2df3e550b065",	
    "properties": {
        
    }
}

Update Connection

Field	Optional	Default	Multiple	Notes	Example
id	No	-	No	Id of the connection to update	"89d6632a-a296-426c-adb0-d442adcab4b0",
description	Yes	-	No	Name of the connection object.	"MyAspiderConnection"
credential	Yes	-	No	Id of the credential assigned to this object.	"2a5ca234-e328-4d40-bb2a-2df3e550b065"
throttlePolicy	Yes	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	Yes	[ ]	Yes	The ids of the routing policies that this connection will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
properties	No	-	No	Configuration object

Example

Code Block

theme	RDark
title	PUT aspire/_api/connections/89d6632a-a296-426c-adb0-d442adcab4b0

{
    "id": "89d6632a-a296-426c-adb0-d442adcab4b0",
    "description": "<connector> Test Connector",
	"credential": "2a5ca234-e328-4d40-bb2a-2df3e550b065",
    "properties": {
        
    }
}

Create Connector Instance

For the creation of the Connector object using the Rest API check this page

Update Connector Instance

For the update of the Connector object using the Rest API check this page

Create Seed

Field	Optional	Default	Multiple	Notes	Example
seed	No	-	No	<seed description>
type	No	-	No	The value must be "<connector>".	"<connector>"
description	No	-	No	Name of the seed object.	"My<connector>Seed"
connector	No	-	No	The id of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection	No	-	No	The id of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	Yes	[ ]	Yes	The ids of the workflows that will be executed for the documents crawled.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	Yes	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	Yes	[ ]	Yes	The ids of the routing policies that this seed will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
tags	Yes	[ ]	Yes	The tags of the seed. These can be used to filter the seed	["tag1", "tag2"]
properties	No	-	No	Configuration object

Example

Code Block

theme	RDark
title	POST aspire/_api/seeds

{
    "type": "<connector>",
    "seed": "directory",
    "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
    "description": "<connector>_Test_Seed",
    "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
    "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
    "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
    "workflows": ["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
    "tags": ["tag1", "tag2"],
    "properties": {
        
    }
}

Update Seed

Field	Optional	Default	Multiple	Notes	Example
id	No	-	No	Id of the seed to update.	"2f287669-d163-4e35-ad17-6bbfe9df3778"
seed	Yes	-	No	<seed description>
description	Yes	-	No	Name of the seed object.	"My<connector>Seed"
connector	Yes	-	No	The id of the connector to be used with this seed. The connector type must match the seed type.	"82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31"
connection	Yes	-	No	The id of the connection to be used with this seed. The connection type must match the seed type.	"602d3700-28dd-4a6a-8b51-e4a663fe9ee6"
workflows	Yes	[ ]	Yes	The ids of the workflows that will be executed for the documents crawled.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
workflows.add	Yes	[ ]	Yes	The ids of the workflows to add.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
workflows.remove	Yes	[ ]	Yes	The ids of the workflows to remove.	["f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"]
throttlePolicy	Yes	-	No	Id of the throttle policy that applies to this connection object.	"f5587cee-9116-4011-b3a9-6b235b333a1b"
routingPolicies	Yes	[ ]	Yes	The ids of the routing policies that this seed will use.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
routingPolicies.add	Yes	[ ]	Yes	The ids of the routingPolicies to add.	["b4d2579f-1a0a-4a8b-9fd4-d42780003b36"]
routingPolicies.remove	Yes	[ ]	Yes	The ids of the routingPolicies to remove.	["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7"]
tags	Yes	[ ]	Yes	The tags of the seed. These can be used to filter the seed	["tag1", "tag3"]
tags.add	Yes	[ ]	Yes	The tags to add	["tag4"]
tags.remove	Yes	[ ]	Yes	The tags to remove	["tag2"]
properties	No	-	No	Configuration object

Example

Code Block

theme	RDark
title	PUT aspire/_api/seeds/2f287669-d163-4e35-ad17-6bbfe9df3778

{
    "id": "2f287669-d163-4e35-ad17-6bbfe9df3778",
    "seed": "<seed example>",
    "connector": "82f7f0a4-8d28-47ce-8c9d-e3ca414b0d31",
    "description": "<connector>_Test_Seed",
    "throttlePolicy": "6b8b5f23-fc77-47a1-9b58-106577162e7b",
    "routingPolicies": ["313de87c-3cb9-4fe0-a2cb-17f75ce7d0c7", "b4d2579f-1a0a-4a8b-9fd4-d42780003b36"],
    "connection": "602d3700-28dd-4a6a-8b51-e4a663fe9ee6",
    "workflows": ["b255e950-1dac-46dc-8f86-1238b2fbdf27", "f8c414cb-1f5d-42ef-9cc9-5696c3f0bda4"],
    "tags": ["tag", "tag2"],
    "properties": {
        
    }
}

Page tree

Versions Compared

Old Version 6

New Version 7

Key

Create Credential (Common)

Create Credential - NTLM specific fields

Create Credential - Negotiate specific fields

Create Credential - Forms specific fields

Example

Update Credential

Create Connection

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example

Page tree

Page History

Versions Compared

Old Version 6

New Version 7

Key

Create Credential (Common)

Create Credential - NTLM specific fields

Create Credential - Negotiate specific fields

Create Credential - Forms specific fields

Example

Update Credential

Create Connection

Example

Update Connection

Example

Create Connector Instance

Update Connector Instance

Create Seed

Example

Update Seed

Example