Table of Contents

maxLevel	1

Provides the functionality to manage the staging repository storage units (create, edit, delete, report, etc).

Create Storage Unit

Creates a new storage unit in the staging repository. A storage unit

Request

The create storage unit GET/PUT/POST request requires the storage unit name.

Code Block

language	text

PUT admin/create/<storage-unit-name>

Response

If the storage unit was created successfully, a 200 response code with an OK message is returned.

Code Block

language	js
theme	RDark

{"message": "OK"}

If a storage unit with the given name already exists, a 400 response code with a STORAGE_UNIT_EXISTS message is returned.

Code Block

language	js
theme	RDark

{"message": "STORAGE_UNIT_EXISTS"}

Drop Storage Unit

Deletes a storage unit from the staging repository server. All content and transactions for the storage unit are deleted.

Request

The drop storage unit GET/DELETE/POST request requires the name of the storage unit to delete.

Code Block

language	text

DELETE admin/drop/<storage-unit-name>

Response

If the storage unit was deleted successfully, a 200 response code with an OK message is returned.

Code Block

language	js
theme	RDark

 {"message": "OK"}

If the storage unit doesn't exist, a 400 response code with a STORAGE_UNIT_DOESNT_EXIST message is returned.

Code Block

language	js
theme	RDark

{"message": "STORAGE_UNIT_DOESNT_EXIST"}

List Storage Units

Lists all available storage units with the option of returning content and transaction statistics.

Request

The list storage units GET/POST request can receive a withStats parameter to return content and transaction statistics for each storage unit.

Code Block

language	text

GET admin/list?withStats

Response

A 200 response code with the list of storage unit names is returned.

Code Block

language	js
theme	RDark

 [
    {"name": "Management"},
    {"name": "Management10"},
    {"name": "Management4"},
    {"name": "Management5"},
    {"name": "testStorageUnit"}
]

A 200 response code with the list of storage unit names and statistics is return when using the withStats parameter.

Code Block

language	js
theme	RDark

[
    {
        "name": "Sales",
        "records": 0,
        "transactions": 0
    },
    {
        "name": "Products",
        "records": 32,
        "transactions": 32,
        "latestTransactionTime": "2015-11-12T20:13:13.737Z",
        "latestTransactionNum": "5644f2d997cc11144081d14f"
    }
]

Get Storage Unit Statistics

Gets the statistics for a storage unit.

Request

The get storage unit statistics GET/POST request requires the name of the storage unit.

Code Block

language	js

GET admin/getStatistics/<storage-unit-name>

Response

If the storage unit exists, a 200 response code and a JSON with the storage unit statistics is returned.

Code Block

language	js
theme	RDark

{
	"name": "Management5",
	"records": 32,
	"transactions": 32,
	"latestTransactionTime": "2015-11-12T20:13:13.737Z",
	"latestTransactionNum": "5644f2d997cc11144081d14f"
}

If the storage unit doesn't exist, a 400 response code with a STORAGE_UNIT_DOESNT_EXIST message is returned.

Code Block

language	js
theme	RDark

{"message": "STORAGE_UNIT_DOESNT_EXIST"}

Dump Storage Unit

Dumps and returns the content and transaction collections of a storage unit in a zip file.

Request

The dump storage unit GET/POST request requires the name of the storage unit to dump.

Code Block

language	text

GET admin/dump/<storage-unit-name>

Response

If the storage unit exists, a 200 response code and a zip file are returned.

If the storage unit doesn't exist, a 400 response code with a STORAGE_UNIT_DOESNT_EXIST message is returned.

Code Block

language	js
theme	RDark

 {"message": "STORAGE_UNIT_DOESNT_EXIST"}

Anchor
Configure Storage Unit
Configure Storage Unit
Configure Storage Unit

A storage unit has a set of events that are triggered during different actions performed on the storage unit. There are two different types of events: per document events and general events.

Per document events are triggered for each document when it is added, update, deleted or fetched.

PreAdd: this event is triggered before the record is stored (added or updated) in the storage unit.
PostAdd: this event is triggered after the record is stored (added or updated) in the storage unit.
PreDelete: this event is triggered before the record is deleted from the storage unit.
PostDelete: this event is triggered after the record is deleted from the storage unit.
Fetch: this event is triggered after the record is fetched from the storage unit.
User Defined Document Events:

General events are triggered when specific transactions are submitted to the storage unit:

BatchStart: this event is triggered when a new batch is created during content ingestion. When a batch is created, a batch variable is added to the execution context which can be accessed by records on per document events.
BatchEnd: this event is triggered when a batch is completed during content ingestion.
User Defined General Events:
StartFullScan: this event is triggered when a start full scan transaction is received. This transaction can be received through execute or batch calls of the transaction API
EndFullScan: this event is triggered when an end full scan transaction is received. This transaction can be received through execute or batch calls of the transaction API

Event actions are configured on external javascript modules placed inside the processing_modules folder of the staging repository server. Each module can implement one or more of the event functions. The name of the function needs to be the name of the implemented event. A module can have functions of both general and per document events.

Per document events receive four parameters:

key: The id of the record.
content: A Javascript Object with the content of the scope of the record that is being processed.
context: A Javascript Object with configuration variables and access to utility functions.
settings: List of available configuration properties for the module.

Code Block

language	js
theme	RDark

 exports.PostAdd = function(key, content, context, settings){
    initialize(settings);
    if (context.isBatch !== undefined && context.isBatch === true){
        if (content){
            context.batchArray.push({ index:  { _index: index, _type: type, _id: key }});
            context.batchArray.push(content);
        }
        return content;
    } else {
        client.index({
            index: index,
            type: type,
            id: key,
            body: content
        }, function (err, resp) {
            if (err) console.log('Error publishing' + JSON.stringify(err));
            if (resp) console.log('Publish successful');
            return content;
        });
    }
};

General events receive two parameters: context: A Javascript Object with configuration variables and access to utility functions. settings: List of available configuration properties for the module.

Code Block

language	js
theme	RDark

 exports.BatchStart = function(context, settings){
    initialize(settings);
    if (context.batchArray === undefined){
        context.batchArray =[];
    }
};

The configure API call configures content processing modules and module settings for a storage unit. Content processing modules are configured per scope. A default list of modules can be configured for scopes that are not explicitly defined. Each module can define it's own collection of settings. When executing the events for each module, these will be executed in the order in which they appear in the configuration array.

Storage unit configuration consists of lists of modules for each scope and general settings. Each module may contain specific settings, as well.

Code Block

language	js
theme	RDark

 {
    "modules" : {
        "connector": [ 
            {
                "module" : "AspireFieldMapping"
            },
            {
                "settings" : {
                    "elasticsearch-index" : "aspiredocs",
                    "elasticsearch-type" : "aspiredoc"
                },
                "module" : "ESPublisher"
            }
        ], 
        "index" : [ 
            {
                "module" : "FieldMapping"
            }, 
            {
                "settings" : {
                    "elasticsearch-index" : "researchdocs",
                    "elasticsearch-type" : "researchdoc"
                },
                "module" : "ESPublisher"
            }
        ],
        "research" : [ 
            {
                "module" : "NormalizeCategory"
            }
        ]
    },
    "settings" : {
        "elasticsearch-port" : 9200,
        "elasticsearch-server" : "localhost"
    }
}

Request

The storage unit PUT/POST set content processing modules request requires the name of the storage unit in the URL and receives a JSON in the body with the content processing modules configuration. This will replace any previous stored configuration for the storage unit.

Code Block

language	text

POST admin/setContentProcessingModules/<storage-unit-name>

Code Block

language	js
theme	RDark

{
    "modules" : {
        "connector": [ 
            {
                "module" : "FieldMapping"
            },
            {
                "settings" : {
                    "elasticsearch-index" : "aspiredocs",
                    "elasticsearch-type" : "aspiredoc"
                },
                "module" : "ESPublisher"
            }
        ], 
        ...
    },
    "settings" : {
        "elasticsearch-port" : 9200,
        "elasticsearch-server" : "localhost"
    }
}

Response

If the operation is successful, a 200 response code and an OK message are returned.

Code Block

language	js
theme	RDark

{"message":"OK"}

Enable Content Processing

Enables or disables content processing for a storage unit.

Request

The storage unit GET/POST enable content processing request requires the name of the storage unit and a boolean value to indicate whether to enable (true) or disable (false) content processing. When a storage unit is created, content processing is enabled by default.

Enable

Code Block

language	text

POST admin/enableContentProcessing/<storage-unit-name>/true

Disable

Code Block

language	text

POST admin/enableContentProcessing/<storage-unit-name>/false

Response

If the operation is successful, a 200 response code and an OK message are returned.

Code Block

language	js
theme	RDark

{"message":"OK"}

Enable Content Compression

Enables or disables content compression for a storage unit. When enabled, the JSON documents are compressed before encryption happens for storage.

Request

The storage unit GET/POST enable content compression request requires the name of the storage unit and a boolean value to indicate whether to enable (true) or disable (false) content processing. By default compression is disabled.

Enable

Code Block

language	text

POST admin/enableContentCompression/<storage-unit-name>/true

Disable

Code Block

language	js

POST admin/enableContentCompression/<storage-unit-name>/false

Response

If the operation is successful, a 200 response code and an OK message are returned.

Code Block

language	js
theme	RDark

{"message":"OK"}

Enable Reprocessing Queue

Enables or disables the background reprocessing queue of a storage unit. When enabled, automatic and manual reprocess requests will execute the Process document events configured for documents being reprocessed.

Request

The storage unit GET/POST enable reprocessing queue request requires the name of the storage unit and a boolean value to indicate whether to enable (true) or disable (false) content processing. By default the reprocessing queue is enabled.

Enable

Code Block

language	text

POST admin/enableContentCompression/<storage-unit-name>/true

Disable

Code Block

language	text

POST admin/enableContentCompression/<storage-unit-name>/false

Response

If the operation is successful, a 200 response code and an OK message are returned.

Code Block

language	js
theme	RDark

 {"message":"OK"}

Get Configuration

Get the current storage unit configuration (content processing).

Request

The storage unit GET/POST get configuration request requires the name of the storage unit in the URL.

Code Block

language	text

 POST admin/getConfiguration/<storage-unit-name>

Response

If the operation is successful, a 200 response code and the storage unit JSON configuration.

Code Block

language	js
theme	RDark

{
    "settings":{
        "modules" : {
            "connector": [ 
                {
                    "module" : "FieldMapping"
                },
                {
                    "settings" : {
                        "elasticsearch-index" : "aspiredocs",
                        "elasticsearch-type" : "aspiredoc"
                    },
                    "module" : "ESPublisher"
                }
            ], 
            ...
        },
        "settings" : {
            "elasticsearch-port" : 9200,
            "elasticsearch-server" : "localhost"
        }
    },
    "contentProcessing" : true,
    "compressContent" : true,
    "reprocessingQueue" : true
}

Page tree

Page History

Versions Compared

Old Version 8

New Version 9

Key

Create Storage Unit

Request

Response

Drop Storage Unit

Request

Response

List Storage Units

Request

Response

Get Storage Unit Statistics

Request

Response

Dump Storage Unit

Request

Response

AnchorConfigure Storage UnitConfigure Storage UnitConfigure Storage Unit

Request

Response

Enable Content Processing

Request

Enable

Disable

Response

Enable Content Compression

Request

Enable

Disable

Response

Enable Reprocessing Queue

Request

Enable

Disable

Response

Get Configuration

Request

Response

Anchor
Configure Storage Unit
Configure Storage Unit
Configure Storage Unit