InternalError Exception

[Joachim Worringen]

Greetings,

my recent multithreading problems have been solved (multiple connections
scale well); now the next issue, not related to multithreading as it
shows up in single-threaded mode as well.

I perform a query to determine whether a table contains a certain column
(more elegant ways to detect this welcome). In the case it does not
exist, I expect (and normally get) a ProgrammingError exception, which I
catch.

However, with a certain query, I always get an InternalError exception -
which I could handle in the same way as an ProgrammingError, but I think
ProgrammingError would be the better exception.

psycopg2 is version 2.2.1; the last lines of the debug output is below.
Full dump on request.

thanks, Joachim

[25507] psyco_conn_cursor: parameters: name = (null)
[25507] cursor_setup: init cursor object at 0x25c3c18
[25507] cursor_setup: parameters: name = (null), conn = 0x264f320
[25507] cursor_setup: good cursor object at 0x25c3c18, refcnt = 1
[25507] psyco_conn_cursor: new cursor at 0x25c3c18: refcnt = 1
[25507] psyco_conn_cursor: new cursor for connection at 0x264f320
[25507] psyco_conn_cursor: parameters: name = (null)
[25507] cursor_setup: init cursor object at 0x25c3d00
[25507] cursor_setup: parameters: name = (null), conn = 0x264f320
[25507] cursor_setup: good cursor object at 0x25c3d00, refcnt = 1
[25507] psyco_conn_cursor: new cursor at 0x25c3d00: refcnt = 1
[25507] psyco_curs_execute: starting execution of new query
[25507] curs_execute: pg connection at 0x26d1250 OK
[25507] pq_begin_locked: pgconn = 0x26d1250, isolevel = 1, status = 1
[25507] pq_execute_command_locked: pgconn = 0x26d1250, query = BEGIN;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
[25507] cursor_dealloc: deleted cursor object at 0x2555d00, refcnt = 0
[25507] pq_execute: executing SYNC query:
[25507] SELECT wtime_A FROM qt_1574 WHERE _pb_data_idx_=0
[25507] pq_execute: entering syncronous DBAPI compatibility mode
[25507] pq_fetch: pgstatus = PGRES_TUPLES_OK
[25507] pq_fetch: data from a SELECT (got tuples)
[25507] _pq_fetch_tuples: looking for cast 701:
[25507] _pq_fetch_tuples: per-connection dict: (nil)
[25507] _pq_fetch_tuples: global dict: 0x2552fc8
[25507] _pq_fetch_tuples: using cast at 0x2552fc8 (FLOAT) for type 701
[25507] pq_fetch: fetching done; check for critical errors
[25507] psyco_curs_execute: res = 1, pgres = 0x7f4b8c001640
[25507] psyco_curs_execute: starting execution of new query
[25507] curs_execute: pg connection at 0x26d1250 OK
[25507] pq_begin_locked: pgconn = 0x26d1250, isolevel = 1, status = 2
[25507] pq_begin_locked: transaction in progress
[25507] pq_execute: executing SYNC query:
[25507] SELECT wtime_A FROM qt_1577 WHERE _pb_data_idx_=0
[25507] pq_execute: entering syncronous DBAPI compatibility mode
[25507] pq_fetch: pgstatus = PGRES_FATAL_ERROR
[25507] pq_fetch: uh-oh, something FAILED
[25507] pq_fetch: fetching done; check for critical errors
[25507] psyco_curs_execute: res = -1, pgres = (nil)
[25507] psyco_curs_execute: starting execution of new query
[25507] curs_execute: pg connection at 0x26d1250 OK
[25507] pq_begin_locked: pgconn = 0x26d1250, isolevel = 1, status = 2
[25507] pq_begin_locked: transaction in progress
[25507] pq_execute: executing SYNC query:
[25507] SELECT wtime_B FROM qt_1574 WHERE _pb_data_idx_=0
[25507] pq_execute: entering syncronous DBAPI compatibility mode
[25507] pq_fetch: pgstatus = PGRES_FATAL_ERROR
[25507] pq_fetch: uh-oh, something FAILED
[25507] pq_fetch: fetching done; check for critical errors
[25507] psyco_curs_execute: res = -1, pgres = (nil)
#* Internal database error.
current transaction is aborted, commands ignored until end of
transaction block

[Joachim Worringen]

Joachim,

[..]

Thanks, Harald - but in this case, this is a more complex workaround
than just catching InternalError in addtion to ProgrammingError.

I feel that InternalError should not be raised in the first place at
this point (and normally, it isn't).

Joachim
--
Joachim Worringen
Senior Performance Architect

International Algorithmic Trading GmbH
fon: [phone]
fax: [phone]

Am Knick 8 - D-22113 Oststeinbek

Geschäfsführer: Yorck Schumacher - Mirko Stamm - Timo Wagner
Gesellschaftssitz: Oststeinbek
Registergericht: Lübeck | Registernummer: HRB 7454HL

[Jeremy Mason]

Post by Joachim Worringen

Joachim,

[..]
Thanks, Harald - but in this case, this is a more complex workaround
than just catching InternalError in addtion to ProgrammingError.
I feel that InternalError should not be raised in the first place at
this point (and normally, it isn't).
Joachim

I use a query like:

SELECT count(column_name)
FROM information_schema.columns
WHERE table_name = 'mytable'
AND column_name = 'missing_column'

It returns 0 if the column is missing. Although, information_schema is
only available in Postgres 7.4 or later.

Jeremy

[Joachim Worringen]

Post by Joachim Worringen

Joachim,

[..]
Thanks, Harald - but in this case, this is a more complex workaround
than just catching InternalError in addtion to ProgrammingError.
I feel that InternalError should not be raised in the first place at
this point (and normally, it isn't).

Think I found the reason: I need to close() (or somehow reset?) the
cursor before proceeding after a ProgrammingError exception.

Things work well now.

Joachim

[Joachim Worringen]

Post by Joachim Worringen

Post by Joachim Worringen

Joachim,

[..]
Thanks, Harald - but in this case, this is a more complex workaround
than just catching InternalError in addtion to ProgrammingError.
I feel that InternalError should not be raised in the first place at
this point (and normally, it isn't).

Think I found the reason: I need to close() (or somehow reset?) the
cursor before proceeding after a ProgrammingError exception.

Correction: commit the transaction is the relevant operation.

Joachim

[Jan Urbański]

Post by Joachim Worringen

Post by Joachim Worringen

Joachim,

[..]
Thanks, Harald - but in this case, this is a more complex workaround
than just catching InternalError in addtion to ProgrammingError.
I feel that InternalError should not be raised in the first place at
this point (and normally, it isn't).

Think I found the reason: I need to close() (or somehow reset?) the
cursor before proceeding after a ProgrammingError exception.

Actually, you need to rollback the transaction (call .rollback() on the
connection object).

PostgreSQL will not allow you to execute anything in a transaction that
had an error until you rollback, and psycopg2 cursors automatically
create a transaction the first time you use them.

Cheers,
Jan