You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Current »

General

Why is does an incremental crawl last as long as a full crawl?

The RDB via Snapshots Connector performs incremental crawls based on snapshot files, which are meant to match the exact documents that have been indexed by the connector to the search engine. On an incremental crawl the connector fully crawls the RDBMS content the same way as a full crawl, but it only indexes the modified, new or deleted documents during that crawl.

Common Errors

Connection errors

If you can't get a connection to the database, you'll see an exception similar to this:

Couldn't get an RDB connection

com.searchtechnologies.aspire.services.AspireException: Unable to open connection to jdbc:postgresql://localhost/RDBMS_Test_DB (aspire_user) (component='/My_RDB_Source/RDBMSRDB', componentFactory='aspire-multiple-rdb')
	at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.createConnection(MultiRDBConnectionPool.java:535)
	at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.getConnection(MultiRDBConnectionPool.java:477)
	at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.access$000(MultiRDBConnectionPool.java:387)
	at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool.getConnection(MultiRDBConnectionPool.java:131)
	at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.doCrawl(RDBScanner.java:420)
	at com.searchtechnologies.aspire.scanner.AbstractPushScanner.performScan(AbstractPushScanner.java:30)
	at com.searchtechnologies.aspire.scanner.AbstractScanner.scanProcess(AbstractScanner.java:701)
	at com.searchtechnologies.aspire.scanner.AbstractScanner.process(AbstractScanner.java:323)
	at com.searchtechnologies.aspire.application.JobHandlerImpl.runNested(JobHandlerImpl.java:158)
	at com.searchtechnologies.aspire.application.JobHandlerImpl.run(JobHandlerImpl.java:80)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:410)
	at org.postgresql.Driver.connect(Driver.java:280)
	at com.searchtechnologies.aspire.components.rdb.MultiRDBConnectionPool$ConnectionPool.createConnection(MultiRDBConnectionPool.java:525)
	... 12 more
Caused by: java.net.ConnectException: Connection refused: connect
	at java.net.DualStackPlainSocketImpl.connect0(Native Method)
	at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.net.PlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at org.postgresql.core.PGStream.<init>(PGStream.java:60)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101)
	... 21 more

SQL Errors

Errors in the SQL configured for the job will produce exceptions such as:

2014-02-10T19:54:07Z ERROR: Error executing select sql for full crawl (SELECT "id", "name", "company", "email",  birthday FROM Test_table WHERE "id" <= 1000 AND  MOD(id,10) = 9 )
org.postgresql.util.PSQLException: ERROR: relation "test_table" does not exist
  Position: 57
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:560)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
	at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.executeSelectSQL(RDBScanner.java:57)
	at com.searchtechnologies.aspire.components.rdbconnector.RDBScanner.doCrawl(RDBScanner.java:466)
	at com.searchtechnologies.aspire.scanner.AbstractPushScanner.performScan(AbstractPushScanner.java:30)
	at com.searchtechnologies.aspire.scanner.AbstractScanner.scanProcess(AbstractScanner.java:701)
	at com.searchtechnologies.aspire.scanner.AbstractScanner.process(AbstractScanner.java:323)
	at com.searchtechnologies.aspire.application.JobHandlerImpl.runNested(JobHandlerImpl.java:158)
	at com.searchtechnologies.aspire.application.JobHandlerImpl.run(JobHandlerImpl.java:80)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
  • No labels