The Push Updates Listener is an Aspire Service that API allows external systems to push update requests into an Aspire Content Sourceseed. Each request will have one or more documents, each with one of the following actions: add, update and delete. The content source seed will then start a new incremental crawl and only process the documents that were sent on the request.

All requests will be stacked in a separate queue called pushRequestQueue, and each request will be processed sequentially using "first-in, first-out" policy.

Easy Heading Free

navigationTitle	On this Page
navigationExpandOption	expand-all-by-default

Response

Section

Ingest Push Update

Creates basic crawl metrics report from the audit index. This endpoint needs the OPERATOR role to be executed.

Panel

borderColor	black
bgColor	#fafafb

GET POST /aspire/_api/metrics/:seedId/:crawlIdpushUpdates

Path and Request Body Parameters

Name	Type	Required	Description
seedId	string	Required	The seed

id

Id

crawlId

description

string

Required

The crawl id

	An optional description of the server
credential	string	Optional	Credential id can be specified for the server
deleteIncrementalPolicy	string	Optional	Delete policy id
throttlePolicy	string	Optional	Throttle policy id
routingPolicies	string array	Optional	Routing policies ids
properties	JSON	Required	Dynamic JSON object with server properties

Example

Code Block

theme	RDark

{
    "seedId": "b6c3c23a-2e4d-415a-8499-afd23815a5ad",
    "documents": [

Code Block

language	js
theme	RDark

{
    "crawl-metrics": {
        "seedId": "0dda9332-ca52-44e1-ba63-95e51b46f951",
        "crawlId": "1658154843524",
        "metrics": {
            "add": 3,
            "enqueued": 4,
            "crawlBegin": 1,{
            "scannedurl": 1"file://C:tmp/test.txt",
            "crawlEndaction": 1"add",
            "scannedNotIndexedrepItemType": 1"FilesystemItemType@file",
            "controlItemmetadata": 4{
        },
        "workerDistributiondisplayUrl": [
            {"C:\\tmp\\test.txt",
                "workerIdfetchUrl": "worker_192.168.112.1:41443//AAABghGLnak=file:///C:/tmp/test.txt",
                "metricsconnectorSpecific": {
                    "addfield": 3,[
                      "enqueued": 4,
  {
                      "crawlBegin      "@name": 1"bigBlb",
                     "scanned": 1,
      "$": "PePo"
             "crawlEnd": 1,
          }
          "scannedNotIndexed": 1,
         ]
           "controlItem": 4
    }
            }
        }
    ]
}

Response

Code Block

language	js
theme	RDark

{
  "documents":{
    "url": "file://C:/tmp/test.txt",
   ]
 "pushQueueId": "f6c3c23a-2e4d-415a-8499-afd23815a599"
  }
}

Status: 200

Section

Jobs performance statistics

Creates jobs performance statistics report from the audit index. This endpoint needs the ADMINISTRATOR role to be executed.

Panel

borderColor	black
bgColor	#fafafb

POST /aspire/_api/metrics/getJobsStatistics

Query String Parameters

Name	Type	Required	Description
size	number	Optional	How many audit items should be included in the report in addition to the aggregation part. This does not affect the aggregation part of the report. (default = 0)
aggSource	string	Optional	The audit item "metrics" part selected for aggregation. Can be a workflow stage, a workflow event or a workflow itself. Internal id is required here. Please check the audit index items for the idea which "id" should be used. (default = root)
aggSortField	string	Optional	How to sort audit items requested by the "size" parameter. Descending order is used. (default = sumExecutionTime). Options: maxStageExecutionTime, minStageExecutionTime, avgStageExecutionTime, sumExecutionTime, countStages

Filter body parameters

How to select audit items for aggregation. All fields are optional but we suggest at least one field is used (typically crawlId)

Code Block

theme	RDark

{
  "filter" : {
    "seedId" : "seedId"
	"connectionId" : "connectionId",
	"connectorId" : "connectorId",
	"docId" : "docId",
    "crawlId": "crawlId",
	"timestampFrom" : millis from
	"timestampTo" : millis to
  }
}

Response

Code Block

language	js
theme	RDark

{
  "jobsStatisticsReport": {
    "configuration": {
      "JobStatisticsFilter": {
        "seedId": "0dda9332-ca52-44e1-ba63-95e51b46f951",
        "connectorId": null,
        "connectionId": null,
        "crawlId": null,
        "docId": null,
        "timestampFrom": null,
        "timestampTo": null
      },
      "size": 1,
      "aggSource": "root",
      "aggSortField": "sumExecutionTime"
    },
    "aggregations": {
      "documentsCount": {
        "value": 3
      },
      "metrics.root.countStages": {
        "count": 3,
        "min": 4.0,
        "max": 4.0,
        "avg": 4.0,
        "sum": 12.0
      },
      "metrics.root.minStageExecutionTime": {
        "count": 3,
        "min": 0.0,
        "max": 3.0,
        "avg": 1.3333333333333333,
        "sum": 4.0
      },
      "metrics.root.sumExecutionTime": {
        "count": 3,
        "min": 1065.0,
        "max": 2044.0,
        "avg": 1579.3333333333333,
        "sum": 4738.0
      },
      "metrics.root.avgStageExecutionTime": {
        "count": 3,
        "min": 266.0,
        "max": 511.0,
        "avg": 394.6666666666667,
        "sum": 1184.0
      },
      "metrics.root.maxStageExecutionTime": {
        "count": 3,
        "min": 518.0,
        "max": 1507.0,
        "avg": 1014.0,
        "sum": 3042.0
      }
    },
    "audit": {
      "itemId": "c:\\tmp\\ach1\\Directory4\\test2.txt",
      "seedId": "0dda9332-ca52-44e1-ba63-95e51b46f951",
      "connectionId": "0b069957-0f95-40b3-b036-f4f9c93614e1",
      "crawlId": "1658154843524",
      "crawlType": "full",
      "timestamp": 1658154891061,
      "type": "add",
      "worker": "worker_192.168.112.1:41443//AAABghGLnak=",
      "status": "success",
      "retry": false,
      "statistics": {
        "jobId": "192.168.112.1:41443/2022-07-18T13:42:18Z/21",
        "startTimestamp": 1658154882983,
        "stage": [
          {
            "source": "onAddUpdate",
            "sourceName": "onAddUpdate",
            "startTimestamp": 1658154883014,
            "stage": {
              "source": "627a709e-74ee-456c-a263-056a63a389f0",
              "sourceName": "fs2it",
              "startTimestamp": 1658154883014,
              "stage": {
                "source": "f0d17635-3fcc-475b-918b-86e157448df0",
                "sourceName": "onAdik",
                "startTimestamp": 1658154883014,
                "endTimestamp": 1658154883530
              },
              "endTimestamp": 1658154883530
            },
            "endTimestamp": 1658154883530
          },
          {
            "source": "onPublish",
            "sourceName": "onPublish",
            "startTimestamp": 1658154883530,
            "stage": {
              "source": "627a709e-74ee-456c-a263-056a63a389f0",
              "sourceName": "fs2it",
              "startTimestamp": 1658154883531,
              "stage": [
                {
                  "source": "58be59a2-c88d-437e-abc5-6cf5ab804fb6",
                  "sourceName": "pepoScr1",
                  "startTimestamp": 1658154883531,
                  "endTimestamp": 1658154885038
                },
                {
                  "source": "9da23484-1523-4ab7-8e16-455390866c59",
                  "sourceName": "fsPubPep",
                  "startTimestamp": 1658154885038,
                  "endTimestamp": 1658154885039
                },
                {
                  "source": "8f07f20b-3934-4944-8046-583aa49e84ed",
                  "sourceName": "pubikE",
                  "startTimestamp": 1658154885039,
                  "endTimestamp": 1658154885059
                }
              ],
              "endTimestamp": 1658154885059
            },
            "endTimestamp": 1658154885059
          }
        ],
        "endTimestamp": 1658154885059,
        "crawlId": "1658154843524",
        "seedId": "0dda9332-ca52-44e1-ba63-95e51b46f951",
        "id": "c:\\tmp\\ach1\\Directory4\\test2.txt",
        "connectorId": "4048af9e-55ae-4ee0-a730-66f1d87f14e1",
        "connectionId": "0b069957-0f95-40b3-b036-f4f9c93614e1",
        "metrics": {
          "root": {
            "maxStageExecutionTime": 1507,
            "minStageExecutionTime": 1,
            "avgStageExecutionTime": 511,
            "sumExecutionTime": 2044,
            "countStages": 4
          },
          "f0d17635-3fcc-475b-918b-86e157448df0": {
            "maxStageExecutionTime": 516,
            "minStageExecutionTime": 516,
            "avgStageExecutionTime": 516,
            "sumExecutionTime": 516,
            "countStages": 1
          },
          "58be59a2-c88d-437e-abc5-6cf5ab804fb6": {
            "maxStageExecutionTime": 1507,
            "minStageExecutionTime": 1507,
            "avgStageExecutionTime": 1507,
            "sumExecutionTime": 1507,
            "countStages": 1
          },
          "627a709e-74ee-456c-a263-056a63a389f0": {
            "maxStageExecutionTime": 1507,
            "minStageExecutionTime": 1,
            "avgStageExecutionTime": 511,
            "sumExecutionTime": 2044,
            "countStages": 4
          },
          "onPublish": {
            "maxStageExecutionTime": 1507,
            "minStageExecutionTime": 1,
            "avgStageExecutionTime": 509,
            "sumExecutionTime": 1528,
            "countStages": 3
          },
          "8f07f20b-3934-4944-8046-583aa49e84ed": {
            "maxStageExecutionTime": 20,
            "minStageExecutionTime": 20,
            "avgStageExecutionTime": 20,
            "sumExecutionTime": 20,
            "countStages": 1
          },
          "onAddUpdate": {
            "maxStageExecutionTime": 516,
            "minStageExecutionTime": 516,
            "avgStageExecutionTime": 516,
            "sumExecutionTime": 516,
            "countStages": 1
          },
          "9da23484-1523-4ab7-8e16-455390866c59": {
            "maxStageExecutionTime": 1,
            "minStageExecutionTime": 1,
            "avgStageExecutionTime": 1,
            "sumExecutionTime": 1,
            "countStages": 1
          }
        }
      }
    }
  }
}

Status: 200

Page tree

Versions Compared

Old Version 1

New Version 2

Key

Ingest Push Update

Jobs performance statistics

Page tree

Page History

Versions Compared

Old Version 1

New Version 2

Key

Ingest Push Update

Jobs performance statistics