Grokbase
Topics Posts Groups | in
x
[ help ]

David Monarchi (david.e.mon...@gmail.com)

Profile | Posts (17)

User Information

Display Name:David Monarchi
Partial Email Address:david.e.mon...@gmail.com
Posts:
17 total
17 in PostgreSQL - Novice

5 Most Recent

All Posts
1) David Monarchi [NOVICE] ERROR: relation with OID XXXX does not exist
| +1 vote
Hello - I'm using PG 8.2.4. I am executing a function which includes the code fragment shown below....
PostgreSQL - Novice
[ Profile | Reply to group ] [ Flat  Thread  Threaded ]
Hello -

I'm using PG 8.2.4.

I am executing a function which includes the code fragment shown below.  It
goes through the loop and then fails with the error message
     ERROR: relation with OID 591161 does not exist

I'm using the serial data type as a way to record the order of the records
based on a specific sort.  I thought that by dropping the table and
recreating it that I would start off with a fresh counter each time.  I
suspect that I do not understand something about how the SQL is compiled and
stored, and that the OID is related to this.

I have included the output below the code.

Any guidance would be appreciated.

Thank you.

david
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
FETCH classIDCursor INTO rec; -- get first record;
WHILE FOUND LOOP
  numClassesRead = numClassesRead + 1;
  classIdArray = rec.class_id_dom;
  currentClassId = classIdArray[1];
  -- drop & recreate table for sorting
  drop table if exists dom1_classid_sorted_temp;
  create table dom1_classid_sorted_temp (id_dom integer, class_rank_dom
serial) without oids;
RAISE NOTICE 'Table created for class id: %', CAST(currentClassId AS TEXT);
  insert into dom1_classid_sorted_temp (id_dom)
       select id_dom from dom1_temp
        WHERE class_id_dom && classIdArray
              order by fast_score_dom desc, link_score_dom desc;
RAISE NOTICE 'Data inserted for class id: %', CAST(currentClassId AS TEXT);
  -- write the class rank order back into the temporary table
  update dom1_temp
  SET class_rank_dom = dom1_classid_sorted_temp.class_rank_dom
   from dom1_classid_sorted_temp where dom1_temp.id_dom =
dom1_classid_sorted_temp.id_dom;
RAISE NOTICE 'Data updated for class id: %', CAST(currentClassId AS TEXT);

   elapsedTime = clock_timestamp() - startTime;
   RAISE NOTICE 'Elapsed time: %   for class id: %', CAST(elapsedTime AS
TEXT), CAST(currentClassId AS TEXT);
   FETCH classIDCursor INTO rec; -- get next record;
   END LOOP;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
NOTICE:  table "dom1_classid_sorted_temp" does not exist, skipping
CONTEXT:  SQL statement "drop table if exists dom1_classid_sorted_temp"
PL/pgSQL function "deletethisversioncreateclassandsyntryxranksfordomain1"
line 29 at SQL statement
NOTICE:  CREATE TABLE will create implicit sequence
"dom1_classid_sorted_temp_class_rank_dom_seq" for serial column
"dom1_classid_sorted_temp.class_rank_dom"
CONTEXT:  SQL statement "create table dom1_classid_sorted_temp (id_dom
integer, class_rank_dom serial) without oids"
PL/pgSQL function "deletethisversioncreateclassandsyntryxranksfordomain1"
line 31 at SQL statement
NOTICE:  Table created for class id: 1
NOTICE:  Data inserted for class id: 1
NOTICE:  Data updated for class id: 1
NOTICE:  Elapsed time: 00:06:34.315307   for class id: 1
NOTICE:  Table created for class id: 2

ERROR: relation with OID 591161 does not exist
SQL state: 42P01
Context: SQL statement "INSERT INTO dom1_classid_sorted_temp (id_dom) select
id_dom from dom1_temp WHERE class_id_dom &&  $1  order by fast_score_dom
desc, link_score_dom desc"
PL/pgSQL function "deletethisversioncreateclassandsyntryxranksfordomain1"
line 33 at SQL statement
2) David Monarchi COPY and index updating
| +1 vote
PG 8.2.4 Hello - I need to append rows to an existing table that is in a production db. There are...
PostgreSQL - Novice
[ Profile | Reply to group ] [ Flat  Thread  Threaded ]
PG 8.2.4

Hello -

I need to append rows to an existing table that is in a production db.
There are 60 fields in the table and 17 indexes, including the primary key.
I cannot drop the indexes because the data is being actively accessed by
users.  I can't create a duplicate table because we do not have enough disk
storage to do so at this time.

I believe that the SQL COPY statement will delay updating the indexes until
after it has inserted the rows because (again, I believe) the copy is
treated as a single transaction that either succeeds or fails.

I would appreciate knowing if I am correct about this.

Thank you,

David
3) David Monarchi unreasonable run time for vacuum analyze?
| +1 vote
Hello - I'm running 8.2.3 on an 8-processor Linux box with a fiber-optic SAN. One table in the...
PostgreSQL - Novice
[ Profile | Reply to group ] [ Flat  Thread  Threaded ]
Hello -

I'm running 8.2.3 on an 8-processor Linux box with a fiber-optic SAN.

One table in the database contains 22.6M rows of 71 fields with an average
length of about 500 bytes.  The table is in BCNF.  The table is queried in a
wide variety of ways.  There are 37 btree and Gin indexes defined on the
table.  The GIN indexes are on array fields.  The rest are btree.

I started running vacuum analyze on this table over 44 hours ago, and it is
still running.  The load on the machine has been modest.  There have been no
times when all eight processors were being used.

1) is this amount of time reasonable?
2) if I abort the vacuum analyze, will I corrupt the table, its indexes, or
the database?
3) are there parameters that I can set to improve the performance of the
vacuum analyze process?

Thank you,
David
4) David Monarchi query plan and the use of indexes
| +1 vote
Hello - I'm trying to understand why the planner sometimes chooses to use an index and sometimes...
PostgreSQL - Novice
[ Profile | Reply to group ] [ Flat  Thread  Threaded ]
Hello -

I'm trying to understand why the planner sometimes chooses to use an index
and sometimes not.  In the query plans below, the part of the query being
varied is the where clause involving fast_score_dom at the end of the select
statement.  fast_score_dom is indexed with a b-tree.  The distribution of
values in the field is highly skewed. The range is 1-100, with about 75% of
the occurrences between 1 and 10.  There are about 22.6M rows in the table
with an average length of about 500 bytes.

I don't understand why the use of a single inequality ( > or < ) causes the
planner to filter on the condition, but the use of an interval (> and <)
causes the planner to use the index.  I thought that a b-tree could always
use an inequality, even if it is only one.  I altered the ranges on the
inequalities, but that didn't seem to affect the decision.

My underlying problem is sluggish response times, and I hope that
understanding the planner better will let me organize the queries better.

Thanks for your help.

david

===================================


stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and fast_score_dom =
50;
                                                                      QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=2008.49..2008.50 rows=1 width=0) (actual time=
80.158..80.160 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=2000.16..2008.18 rows=121
width=0) (actual time=78.959..79.841 rows=162 loops=1)
         Recheck Cond: ((alexa_contactinfo___physicaladdress___state_dom =
'tx'::text) AND (fast_score_dom = 50))
         ->  BitmapAnd  (cost=2000.16..2000.16 rows=2 width=0) (actual time=
78.429..78.429 rows=0 loops=1)
               ->  Bitmap Index Scan on domain_alexa_state_dom  (cost=
0.00..353.09 rows=1078 width=0) (actual time=47.775..47.775 rows=52236
loops=1)
                     Index Cond:
(alexa_contactinfo___physicaladdress___state_dom = 'tx'::text)
               ->  Bitmap Index Scan on domain_fast_score_dom  (cost=
0.00..1646.76 rows=37818 width=0) (actual time=4.554..4.554 rows=7086
loops=1)
                     Index Cond: (fast_score_dom = 50)
Total runtime: 81.096 ms
(9 rows)

Time: 94.000 ms
stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and fast_score_dom >
50;
                                                                   QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=4644.83..4644.84 rows=1 width=0) (actual time=
306.703..306.706 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=355.10..4624.72 rows=8044
width=0) (actual time=89.808..304.484 rows=1079 loops=1)
         Recheck Cond: (alexa_contactinfo___physicaladdress___state_dom =
'tx'::text)
         Filter: (fast_score_dom > 50)
         ->  Bitmap Index Scan on domain_alexa_state_dom
(cost=0.00..353.09rows=1078 width=0) (actual time=
47.899..47.899 rows=52236 loops=1)
               Index Cond: (alexa_contactinfo___physicaladdress___state_dom
= 'tx'::text)
Total runtime: 307.760 ms
(7 rows)

Time: 328.000 ms
stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and fast_score_dom >
90;
                                                                   QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=4644.83..4644.84 rows=1 width=0) (actual time=
302.594..302.596 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=355.10..4624.72 rows=8044
width=0) (actual time=269.649..302.513 rows=2 loops=1)
         Recheck Cond: (alexa_contactinfo___physicaladdress___state_dom =
'tx'::text)
         Filter: (fast_score_dom > 90)
         ->  Bitmap Index Scan on domain_alexa_state_dom
(cost=0.00..353.09rows=1078 width=0) (actual time=
47.918..47.918 rows=52236 loops=1)
               Index Cond: (alexa_contactinfo___physicaladdress___state_dom
= 'tx'::text)
Total runtime: 303.658 ms
(7 rows)

Time: 329.000 ms
stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and fast_score_dom >
90 and fast_score_dom < 100;
                                                                      QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=2304.17..2304.18 rows=1 width=0) (actual time=
71.894..71.896 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=2283.79..2303.86 rows=121
width=0) (actual time=71.843..71.853 rows=2 loops=1)
         Recheck Cond: ((alexa_contactinfo___physicaladdress___state_dom =
'tx'::text) AND (fast_score_dom > 90) AND (fast_score_dom < 100))
         ->  BitmapAnd  (cost=2283.79..2283.79 rows=5 width=0) (actual time=
71.415..71.415 rows=0 loops=1)
               ->  Bitmap Index Scan on domain_alexa_state_dom  (cost=
0.00..353.09 rows=1078 width=0) (actual time=48.054..48.054 rows=52236
loops=1)
                     Index Cond:
(alexa_contactinfo___physicaladdress___state_dom = 'tx'::text)
               ->  Bitmap Index Scan on domain_fast_score_dom  (cost=
0.00..1930.39 rows=113455 width=0) (actual time=0.140..0.140 rows=131
loops=1)
                     Index Cond: ((fast_score_dom > 90) AND (fast_score_dom
< 100))
Total runtime: 72.904 ms
(9 rows)

Time: 94.000 ms
stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and fast_score_dom >
10 and fast_score_dom < 100;

QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=2304.17..2304.18 rows=1 width=0) (actual time=
286851.236..286851.238 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=2283.79..2303.86 rows=121
width=0) (actual time=286266.662..286769.573 rows=48021 loops=1)
         Recheck Cond: ((alexa_contactinfo___physicaladdress___state_dom =
'tx'::text) AND (fast_score_dom > 10) AND (fast_score_dom < 100))
         ->  BitmapAnd  (cost=2283.79..2283.79 rows=5 width=0) (actual time=
286264.337..286264.337 rows=0 loops=1)
               ->  Bitmap Index Scan on domain_alexa_state_dom  (cost=
0.00..353.09 rows=1078 width=0) (actual time=47.946..47.946 rows=52236
loops=1)
                     Index Cond:
(alexa_contactinfo___physicaladdress___state_dom = 'tx'::text)
               ->  Bitmap Index Scan on domain_fast_score_dom  (cost=
0.00..1930.39 rows=113455 width=0) (actual
time=286172.307..286172.307rows=7251484 loops=1)
                     Index Cond: ((fast_score_dom > 10) AND (fast_score_dom
< 100))
Total runtime: 286852.339 ms
(9 rows)

Time: 286844.000 ms
stx=# explain analyze select count(*) from domain_dom where
alexa_ContactInfo___PhysicalAddress___State_dom = 'tx' and  fast_score_dom <
10;
                                                                   QUERY
PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=4644.83..4644.84 rows=1 width=0) (actual time=
325.753..325.755 rows=1 loops=1)
   ->  Bitmap Heap Scan on domain_dom  (cost=355.10..4624.72 rows=8044
width=0) (actual time=95.283..319.123 rows=3329 loops=1)
         Recheck Cond: (alexa_contactinfo___physicaladdress___state_dom =
'tx'::text)
         Filter: (fast_score_dom < 10)
         ->  Bitmap Index Scan on domain_alexa_state_dom
(cost=0.00..353.09rows=1078 width=0) (actual time=
50.832..50.832 rows=52236 loops=1)
               Index Cond: (alexa_contactinfo___physicaladdress___state_dom
= 'tx'::text)
Total runtime: 327.018 ms
(7 rows)

Time: 344.000 ms
5) David Monarchi forcing pg to use an index
| +1 vote
Hello - I'm using PG 8.2.3 on an 8-processor server with 32GB of memory. The table in question has...
PostgreSQL - Novice
[ Profile | Reply to group ] [ Flat  Thread  Threaded ]
Hello -

I'm using PG 8.2.3 on an 8-processor server with 32GB of memory.  The table
in question has about 22.6M rows.

One of the fields in the table is mostly null.  Only about 9K rows have
values.  Of those 9K, about half of of them occur only once.  The rest occur
between 2 and 76 times, so the distribution is highly skewed.

I've defined a partial b-tree index on the field where the value is not
null, but the planner doesn't seem to use it.

CREATE INDEX domain_alexa_stock_ticker_dom ON domain_dom
  USING btree (alexa_contactinfo___companystockticker___symbol_dom)
  WHERE NOT alexa_contactinfo___companystockticker___symbol_dom IS NULL;

I've examined the plans, and the planner keeps using a filter.  For example

explain analyse select alexa_contactinfo___companystockticker___symbol_dom
from domain_dom
  where alexa_contactinfo___companystockticker___symbol_dom = 'AAA';

 "Seq Scan on domain_dom  (cost=0.00..1422366.60 rows=1 width=32) (actual
time=223139.526..223340.822 rows=1 loops=1)"
"  Filter: (alexa_contactinfo___companystockticker___symbol_dom =
'AAA'::text)"
"Total runtime: 223340.892 ms"

It seems to me that the query would run a lot faster if the system would
just use the index to go to the rows (there's only one in this case) for
which the value is 'AAA'.

I tried turning seqscan off with set enable_seqscan = off, but that didn't
help.

Any suggestions/help would be appreciated.

Thank you,
David

spacer
Profile | Posts (17)
Home > People > David Monarchi