Create Credential
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| type | Yes | - | No | The value must be "rdb-snapshot". | "rdb-snapshot" |
| description | Yes | - | No | Name of the credential object. | "My RDB Credential" |
| properties | Yes | - | No | Configuration object | |
| authenticationHandler | Yes | - | Yes | Authentication handlers | [] |
| host | No | - | No | Hostname of the urls that will apply this handler, if no hostname is set, it will be used for all. | "domain.com" |
| port | No | -1 | No | Port of the url, if set to -1, any port will be accepted | 8080 |
| loginUrl | Yes | - | No | Url to the login page | "http://yoursite/login.php" |
| user | Yes | - | No | User name | "admin" |
| password | Yes | - | No | User password | "password" |
| authenticationType | Yes | - | No | Authentication implementation: The crawler will use this configuration to log in the page (SIMPLE/SCRIPT) | "SIMPLE" |
| Simple Authentication | |||||
| userSelectorType | Yes | Id | No | Username field selector type (Class, Css, Id, Name, XPath) | "Id" |
| userSelector | yes | - | No | Username field: Field on the login form where the username should be set | "txtUsername" |
| passwordSelectorType | Yes | Id | No | Password field selector type (Class, Css, Id, Name, XPath) | "Id" |
| passwordSelector | yes | - | No | Password field: Field on the login form where the password should be set | "txtPassword" |
| customField | no | - | Yes | Custom field | [] |
| selectorType | yes | Id | No | Type of selector used to locate the field within the page. (Class, Css, Id, Name, XPath) | "Id" |
| selector | yes | - | no | Value of the selector used to locate the field within the page. | "myField" |
| fieldType | no | Text | no | Type of field to locate within the page. (Button, Checkbox, Select, RadioButton, Text) | "Text" |
| fieldValue | yes | - | no | Value of the field | "myFieldValue" |
| submitSelectorType | yes | Id | No | Type of selector used to locate the field within the page. (Class, Css, Id, Name, XPath) | "Id" |
| submitSelector | yes | - | - | Value of the selector used to locate the field within the page. | "btnSubmit" |
| Scripted authentication | |||||
| authenticationScript | yes | - | No | Groovy Code with instructions to fill the login form (see code block below) |
Authentication Script
// Write the instructions to fill the login form // n// Available variables: // // - seedId, String. Id of the current seed. // - logger, ALogger. Logger instance. // - driver, WebDriver. Selenium WebDriver instance for controlling a browser. // - loginUrl, String. Login URL. // - username, String. Username // - password, String. Password (decrypted). IMPORTANT: DO NOT LOG THIS VALUE
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| verificationType | Yes | - | no | Verification style (SIMPLE, SCRIPT) | "SIMPLE" |
| Simple verification | |||||
| verificationField | Yes | - | yes | Fields to verify | [] |
| fieldSelectorType | yes | Id | no | Type of selector used to locate the field within the page. (Class, Css, Id, Name, XPath) | "Id" |
| fieldSelector | yes | - | no | Value of the selector used to locate the field within the page. | "myField" |
| Scripted verification | |||||
| verificationScript | yes | - | no | Groovy Code with instructions to validate whether or not the login was successful or not. Must return a boolean (see code block below) |
Verification Script
// Check if the session is still valid. // Must return a boolean. // If the session is invalid, a login shall be attempted // // Available variables: // // - seedId, String. Id of the current seed. // - logger, ALogger. Logger instance. // - driver, WebDriver. Selenium WebDriver instance for controlling a browser. // - loginUrl, String. Login URL. return false;
Example
POST aspire/_api/credentials
{
"type": "selenium",
"description": "Selenium Desc",
"properties": {
"authenticationHandler": [
{
"host": "domain.com",
"port": 80,
"loginUrl": "http://yoursite/login.php",
"user": "user",
"password": "password",
"authenticationType": "SIMPLE",
"userSelectorType": "Id",
"userSelector": "txtUsername",
"passwordSelectorType": "Id",
"passwordSelector": "txtPassword",
"customField": [],
"submitSelectorType": "Id",
"submitSelector": "btnSubmit",
"verificationType": "SIMPLE",
"verificationField": [
{
"fieldSelectorType": "Id",
"fieldSelector": "myField"
}
]
}
]
}
}
Create Connection
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| type | Yes | - | No | The value must be "rdb-snapshot". | "rdb-snapshot" |
| description | Yes | - | No | Name of the connection object. | "My RDB Connection" |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
| credential | Yes | - | No | Id of the credential | "6b235b333a1b" |
| properties | Yes | - | No | Configuration object | |
| jdbcUrl | Yes | - | No | The JDBC url for your RDBMS server and database. Use database marker {DATABASE} to denote the database | "jdbc:mysql://db:3306/{DATABASE}" |
| jdbcDriverJar | Yes | - | No | Path to the JDBC driver jar file for your RDBMS. | "/lib/myjdbcdriver.jar" |
| jdbcDriverClass | No | see notes | No | The name of the default JDBC driver class (if the class name from the META-INF/services/java.sql.Driver file in the driver Jar file should not be used), or if that file does not exist in the driver jar file (Oracle) | "java.sql.Driver" |
| jdbcDriverClasspath | No | the driver jar file | No | The class path for external jars required by the jdbc driver | |
| stopOnError | No | true | No | When selected, the scan stops if the JDBC driver throws an error while getting a row, and the crawl halts. When unselected, the connector attempts to get subsequent rows | false |
| useSlices | No | false | No | Select this option to divide the full SQL into multiple slices. For example,if you have a 10 Million row table to scan, executing the 10 M query will take a while. After it completes, the connector starts sending items. By activating 10 slices, the scan is split into 10 1 Million scans, which takes less time and you can view results faster. This only works when the idColumn contains an integer. | true |
| numSlices | No | - | No | The number of SQL slices to split fullSQL. Slicing the full SQL should improve the performance significantly if a big database is to be crawled. Only works when the id column is an integer | 10 |
| percentAsMod | No | false | No | Use % Operator for Modulo. Check this option if you want to specify the MODULO operator to use for a particular Database system that doesn't recognize the MOD() function. "MOD()" is available for MySQL, PostgreSQL and Oracle. But systems like Microsoft SQL Server you must check this option | true |
Example
POST aspire/_api/connections
{
"type": rdb-snapshot,
"description": "RDB_TEST",
"properties": {
"jdbcUrl":"jdbc:mysql://localhost:3307/{DATABASE}",
"jdbcDriverJar" : "/lib/myjdbcdriver.jar",
"jdbcDriverClass": null,
"jdbcDriverClasspath": null,
"stopOnError": true,
"useSlices": false,
"numSlices": 2,
"percentAsMod": false
}
Update Connection
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | Id of the connection to update | "d442adcab4b0", |
| description | No | - | No | Name of the connection object. | "My RDB Connection" |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this connection object. | "b3a9-6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this connection will use. | ["17f75ce7d0c7", "d42780003b36"] |
| credential | No | - | No | Id of the credential | "6b235b333a1b" |
| properties | No | - | No | Configuration object | |
| (see create connection) |
Example
PUT aspire/_api/connections/89d6632a-a296-426c-adb0-d442adcab4b0
{
"id": "89d6632a-a296-426c-adb0-d442adcab4b0",
"description": "RDB_TEST",
"properties": {
"jdbcUrl":"jdbc:mysql://localhost:3307/{DATABASE}",
"jdbcDriverJar" : "/lib/myjdbcdriver.jar",
"jdbcDriverClass": null,
"jdbcDriverClasspath": null,
"stopOnError": true,
"useSlices": false,
"numSlices": 2,
"percentAsMod": false
}
Create Seed
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| seed | Yes | - | No | The name of the database. It will replace the marker {DATABASE} used in the field jdbcUrl of connection object | "test_db" |
| type | Yes | - | No | The value must be "rdb-snapshot". | "rdb-snapshot" |
| description | Yes | - | No | Name of the seed object. | "My RDB Seed" |
| connector | Yes | - | No | The id of the connector to be used with this seed. The connector type must match the seed type. | "e3ca414b0d31" |
| connection | Yes | - | No | The id of the connection to be used with this seed. The connection type must match the seed type. | "e4a663fe9ee6" |
| workflows | No | [ ] | Yes | The ids of the workflows that will be executed for the documents crawled. | ["5696c3f0bda4"] |
| throttlePolicy | No | - | No | Id of the throttle policy that applies to this seed object. | "6b235b333a1b" |
| routingPolicies | No | [ ] | Yes | The ids of the routing policies that this seed will use. | ["17f75ce7d0c7", "d42780003b36"] |
| tags | No | [ ] | Yes | The tags of the seed. These can be used to filter the seed | ["tag1", "tag2"] |
| properties | Yes | - | No | Configuration object | |
| fullSQL | Yes (this or discoverySQL + extractionSQL) | - | No | The "SELECT" query to be run to retrieve all documents. This query is used for full or incremental scans. The "WHERE" clause can be used to specify any required condition for crawling the desired documents. Any change to any column selected in this SQL will cause the document to be re-indexed. For example "SELECT idCol, col1, col2, col3 FROM data_table" When slicing is enabled, add a "WHERE" clause containing "{SLICES}". For example "SELECT idCol, col1, col2, col3 FROM data_table WHERE {SLICES}" . | "SELECT * FROM table" |
| discoverySQL | Yes (this or fullSQL) | - | No | The "SELECT" query to run for discovering documents. This query is used for full or incremental scans. A "WHERE" clause can be used to specify any required condition for crawling the desired documents. A change to any column selected in this SQL will cause the document to be re-indexed. For example: "SELECT idCol, lastModifiedDate FROM data_table". When slicing is enabled, add a "WHERE" clause containing "{SLICES}". For example: "SELECT idCol, col1 FROM data_table WHERE {SLICES} | "SELECT id, lastModified FROM table" |
| extractionSQL | Yes (this or fullSQL) | - | No | "SELECT" query for extracting all data for each document found in the Discovery SQL. At the least, you MUST include a "WHERE" clause containing the expression "idColumnName IN {IDS}", where idColumnName corresponds to a unique key field name. {IDS} is replaced automatically by the connector with the corresponding unique key values. For example: "SELECT col1, col2, col3 FROM data_table WHERE idCol in {IDS}" You must not include the {SLICES} condition here. | "SELECT * FROM table WHERE id IN {IDS}" |
| idColumn | Yes | - | No | The column name that holds the unique key. The default name of the column which holds the value to use as the document id. This column must be present in both discoverySQL and extractionSQL. SQL aliases are NOT supported. | "id" |
| stringIdColumn | No | false | No | Check if the unique key is a string value | true |
| quoteId | No | doNotQuote | No | Quote id column - use if you have a name clashing with RDBMS keywords. You can use one of the values: doNotQuote, `, " | doNotQuote |
| ACL | |||||
| aclColumn | Yes (aclColumn or aclSQL) | - | No | The column name that holds the ACLs. Each ACL must be separated by semi-colons and must follow this format: my-domain\userOrGroup@NT | "acl" |
| aclSQL | Yes (aclColumn or aclSQL) | - | No | The query to use for extracting and building ACLs. This query depends of the Database engine, so the syntax could vary. For example on Oracle: SELECT 'my-domain\\' || user || '@NT;' FROM myTable | "SELECT * FROM table_acl" |
Example
POST aspire/_api/seeds
{
"seed":"test_db",
"type":"rdb-snapshot",
"description" : "RDB_TEST",
"properties" : {
"idColumn" : "film_id",
"stringIdColumn" : false,
"aclSQL" : null,
"aclColumn" : "acl",
"quoteId" : "doNotQuote",
"discoverySQL" : "SELECT film_id, title FROM film",
"extractionSQL" : "SELECT * FROM film WHERE film_id IN {IDS}",
"fullSQL" : null"
}
}
Update Seed
| Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | Id of the seed to update | "2f287669-d163-4e35-ad17-6bbfe9df3778" |
| (see the "Create seed" for other fields) |
Example
PUT aspire/_api/seeds/2f287669-d163-4e35-ad17-6bbfe9df3778
{
"id": "2f287669-d163-4e35-ad17-6bbfe9df3778",
"seed":"test_db",
"description" : "RDB_TEST",
"properties" : {
"idColumn" : "film_id",
"stringIdColumn" : false,
"aclSQL" : null,
"aclColumn" : "acl",
"quoteId" : "doNotQuote",
"discoverySQL" : "SELECT film_id, title FROM film",
"extractionSQL" : "SELECT * FROM film WHERE film_id IN {IDS}",
"fullSQL" : null"
}
}
Overview
Content Tools
Copyright © 2024 Accenture
