Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Warning
titleIn Progress

The RDB via Snapshot The Rest API Connector can be configured using the Aspire Admin UI. It requires the following entities to be created:

  • Credential
  • Connection
  • Connector
  • Seed

Bellow Below are the examples of how to create the Credential, Connection, and the Seed. For the Connector please check general connectors' configuration, please refer to this page.

Easy Heading Free
navigationTitleOn this Page
navigationExpandOptionexpand-all-by-default

Create Credential


  1. On the Aspire Admin UI, go to the credentials tab tab
  2. All existing credential credentials will be listlisted. Click on the new button button
  3. Enter the new credential description. 
  4. Select Rdbrest-Snapshot api from the Type list.
  5. Enter User and Password. Note: The password will be automatically encrypted by Aspire.
  6. Select required authentication method: basic, bearer, api-token, azure, none
  7. The description of all "credential" fields can be found here.
    1. Basic:

      1. Login Account

      2. Password
    2. API Token:

      1. Header Name: The name of the header in which the API Token should be included.

      2. Header Token Value: The value of the token to use, should include any value header, such as Bearer in case of bearer tokens.
    3. Bearer:

      1. Login URL: Login URL template

      2. Login Account: Username to authenticate with. Replaces ${loginAccount} within POST login body
      3. Password: Password or token to authenticate with. Replaces ${password} within POST login body
      4. Request Body: The request body of the authentication query
      5. Access token: The result field with access token
      6. Is Form Body: If the request body is a Form Body
      7. Specify the expires field name
        1. Expiration type: "Expires at" or "Expires in"
          1. Expires at:
            1. "Expires at" field name: field name related to the token generated
            2. "Expires at" field date format: field date format of the token
          2. Expires in:
            1. "Expires in" Field name
            2. Time unit: the unit of the timestamp (milliseconds, seconds, minutes or hours)
            3. Expires in specific time? 
              1. Expires in "x" millis, seconds, minutes or hours
              2. Time unit: the unit of the timestamp (milliseconds, seconds, minutes or hours)
      8. Specify Authentication Header Name: if you want to specify the authentication header name. Otherwise, it will use the bearer default format.
        1. Header name: Header name related to the token generated
      9. Other Authentication Headers configuration: Specify other headers for the authentication request
        1. Header Name
        2. Header Value
    4. Azure Credentials:

      1. Client ID: The Application (client) ID that the Azure portal - App registrations page assigned to your app

      2. Client Secret: Needed if the app is a confidential client
      3. Scope: A space-separated list of scopes, or permissions, that the app requires
      4. Tenant: The directory tenant that you want to log the user into. This can be in GUID or friendly name format




Image Added


Image Added


Image Added



Image AddedImage Removed

Create Connection

  1. On the Aspire Admin UI, go to the connections tab tab
  2. All existing connection connections will be listlisted. Click on the new button button
  3. Enter the new connection description. 
  4. Select Rdbrest-Snapshot api from the Type list.
  5. In the "JDBC UrlAPI Base URL:" field in the Properties section, enter the JDBC URL for the database to be crawled.
    1. RDBMS URLs

      An RDBMS "URL" is needed to tell the connector application what database to crawl. The exact form of this URL is dictated by the JDBC driver and therefore the database vendor, but will be of the form

      jdbc:<vendor>://<server>:<port>/{DATABASE}

      For example:  jdbc:mysql://192.168.40.27/{DATABASE} - the marker {DATABASE} will be replaced with the value of Database name field (see Create Seed section on this page)

      See your database vendor's documentation for more information on JDBC URLs.

  6. In the "JDBC Driver Jar:" field, enter the name of the JAR file containing the driver for your database.
  7. In the "JDBC Driver Class" field, enter the Java class name for the driver (optional)
  8. If your JDBC Driver jar does not contain all the necessary classes, you may add additional jar files in to the class path. To do this, select "Specify classpath" and enter the classpath to the jars containing the classes. The classpath may include jar files, directories or individual classes.
  9. etc.

The description of all "connection" fields  can be found here.

  1. your REST service API URL
  2. Connection Timeout: The period of time (ms) after which the connection will be closed if a client connection does not receive a response from the server.
  3. Socket Timeout: The period of time (ms) after which the socket will be closed if a client connection does not receive a response from the server
  4. Max Retries: The number of connections retries
  5. Use Throttling:
    1. API Calls Limit Size: The number of calls that your API can receive within a fixed period of time
    2. API Limit Time: The period of fixed time in which the number of API calls can be received (in milliseconds)
  6. Use Proxy:
    1. Select Protocol: the protocol of the proxy connection
    2. Proxy Host: Your proxy host
    3. Proxy Port: Your proxy port
    4. Use Proxy Authentication:
      1. Proxy Username
      2. Proxy Password
  7. Trust all certificates: If selected, no HTTPS certificate validation will be done

Image AddedImage Removed

Create Connector


For the creation of the Connector object using the Admin UI check , please refer to this page


Create Seed


  1. On the Aspire Admin UI, go to the seeds tab tab
  2. All existing seeds will be listlisted. Click on the new button button
  3. Enter the new seed description. 
  4. Select Rdbrest-Snapshot api from the Type list.
  5. Specify crawl rule.
  6. Database name field: The name of the database. It will replace the marker {DATABASE} used in the field jdbcUrl of connection object
  7. etc.

The description of all "seed" fields can be found here.

  1. The description of all "seed" fields can be found here.
  2. Conditional Script:
    1. If true:
      1. Rule condition: Groovy condition to determine which items should execute this set of queries
    2. If false:
      1. Entity: Entity type to match
  3. Stop querying: If selected, the no other queries will be executed for the given item
  4. Should index: If selected, the item matching this crawl rule will be indexed
  5. Use custom scripted query:
    1. If true:
      1. Is scan query?: Select if the current script should be able to extract new items
      2. Scripted query: Custom query executed with groovy
    2. If false:
      1. Query: The query to execute
      2. HTTP Method (GET, POST, PUT):
        1. If POST or PUT:
          1. Request Body: The body of the request.
      3. JSON Request: Check this option, the request body should be JSON.
        1. If false:
          1. Content Type: The content type to use for the request body
      4. Headers configuration:
        1. Header Name: The name of the header
        2. Value: The value of the header
      5. Query type:
        1. Scan:
          1. Extraction path: The path to the response array that contains the children to extract
          2. Child ID field: Field within each child holding its ID
          3. Entity type: Type given to the items discovered by the scan query, this can be used to match a rule condition by type, if left empty 'entity' will be used
          4. Set Metadata Field: Select if you want to specify the metadata parent field, otherwise all the metadata will be at document root level
            1. Metadata Field Name: Specify the metadata parent field name
          5. Incremental configuration:
            1. Signature JSON Path: JSON path to extract fields to use as signature. Check out https://github.com/json-path/JsonPath for JsonPath documentation
          6. Extended incremental signature: Check this option if extra requests must be executed to obtain metadata needed to calculate modifications properly. Use this option carefully, as this decreases linearly the incremental crawl performance.
            1. Query: The query to execute
            2. HTTP Method (GET, POST, PUT):
              1. If POST or PUT:
                1. Request Body: The body of the request.
            3. Signature JSON Path: JSON path to extract fields to use as signature. Check out https://github.com/json-path/JsonPath for JsonPath documentation
            4. Result field: Internal name of metadata where the results will be extracted into
          7. Enable pagination:
            1. Elastic Scroll: Use Elasticsearch scroll ID for pagination
              1. If false:
                1. Use the next page link from the response: Select if the next page link should be taken from a field in the response
                  1. if true:
                    1. Next page link field: Path to the next page link field in the response, use dots to use fields in hierarchical paths
                    2. Next page from header: Check if the next page link is on the header. Otherwise, the response will be checked
                    3. Next page link is relative: Check if the next page link is relative, i.e: /api/data/9/children/?skip=1&limit=1
                  2. If false:
                    1. Page size: The maximum number of entries the query retrieves per page
                    2. Response contains total count: Check this option if the response has as a response the total number of entities in all pages
                      1. Total field: Path to the total field in the response
                    3. Request params: Check this option if pagination is controlled by request parameters. Properties that you can use: ${pagination.offset} ${pagination.pageNumber} ${pagination.pageSize}
                      1. Parameters template: Parameters template for pagination
                      2. Pagination start index: Select the start index of the pagination






Image AddedImage AddedImage AddedImage AddedImage Removed