I have to agree with Hernan.

The simplest saving I can see is an extra column to save on the "run time" calculation. This does not have to be dangerous (ie potentially different to the date column you have already)Just setup a trigger for whenever the existing date column gets updated(/inserted) to update the extra column.

Another alternative is to use a materialised view. On Postgres this is a manual process involving setting up a view and rules, and ig you want it always up to date, triggers again on the source table to update the materialised view whenever the data changes.

This latter idea would be the fastest - faster than your original view by quite a margin, in fact. You may then be able to get rid of some (all?) indexes you have already on the existing table - because they would no longer be needed.

One thing I think would be handy that I have on my list of "when I get time" is to write some reasonably automatic code to do materialised views without the manual coding (ala Oracle). People may use these things more if they were a bit more simple to do.

Link to Need Help Reducing View Calculations page: http://www.justatheory.com/computers/databases/postgresql/reducing_view_calculations.html

The argument that I have been unable to win in putting together a warehouse on PostgreSQL comes down to tool maturity - ie the risk involved in something "not proven" so it is almost certainly going to be built using Oracle Warehouse Builder and a BI tool tbd.

I have a long history with databases, and experience with PostgreSQL that dates to the beginning of version 7, however I've never been involved in formal datawarehousing or ETL - so I can't speak with the authority I would prefer ;-)

From my research (mostly involving asking Liam at Fujitsu) Bizgres which seems to be still at the "not finished" stage, and Jasper. I have experience in Jasper products from some time back - they looked very good then - so I can't wait to try out JasperETL.

In any case I am likely to do a parallel run of the work with OWB using Jasper/Postgres to see how they go in my own time. I would welcome any comments on all this - particularly if they can be backed up with real-world datawarehousing projects.

For those wondering about this blog, family illness has kept me from this for some time, unfortunately... something that is now coming to an end in a good way.

Cheers,

Mathew Frank (http://frakkle.com)

The second part is how do we do operations on these arrays - there is no SUM/COUNT/AVG etc for array elements. And how do we work with different time-slices of data. (ie if we have a moving period of 30 days how to we return a moving average of 21 days or similar)

If we remember from the first post - 41 seconds is the time to beat (btw - I forget to mention before: I am dealing with about 10 years of data for NAB (National Australia Bank - listed on the ASX)). The following function - running the same resultant rows - takes 3.4 seconds.

Since building the original function, I have actually changed it to give me arrays for every column (trading day, open, closed, low, high) and the final runtime is 4.1 seconds. It should be noted that this is a specific function, however for different tables/column sets it is very easy to change for your own use. I could have made this more generic at the cost of performance - but there did not seem to be much point for my use - certainly not to provide this example.

So this is the function - at least the new version:

CREATE TYPE dt_stock_set AS
(stock varchar(10),
day_id int4,
day_ids int4[],
closing numeric[],
open numeric[],
low numeric[],
high numeric[],
volume numeric[],
period int4);

CREATE OR REPLACE FUNCTION last_set(code "varchar", period int4, day_max int4, day_min int4)
RETURNS SETOF dt_stock_set AS
$BODY$
declare
rtn dt_stock_set%rowtype;
rs record;
last_stock varchar(10);
ar_day_id int4[];
ar_close numeric[];
ar_open numeric[];
ar_high numeric[];
ar_low numeric[];
ar_volume numeric[];
ar_pointer int4;
BEGIN
last_stock = '';
FOR rs IN
SELECT s.stock, s.day_id, s.closing, s.open, s.high, s.low, s.volume
FROM stocks AS s
WHERE (code IS NULL OR s.stock=code)
AND (day_max IS NULL OR s.day_id <= day_max)
AND (day_min IS NULL OR s.day_id >= day_min)
ORDER BY s.stock ASC, day_id ASC
LOOP
IF last_stock != rs.stock THEN
last_stock := rs.stock;
ar_day_id := Array[rs.day_id];
ar_close := Array[rs.closing];
ar_open := Array[rs.open];
ar_low := Array[rs.low];
ar_high := Array[rs.high];
ar_volume := Array[rs.volume];
ELSE
--ar_close := rs.closing || ar_close[1:29] ; --NOTE this does not work in PL hence the loop
FOR ar_pointer IN REVERSE period .. 2 LOOP
ar_day_id[ar_pointer] := ar_day_id[ar_pointer-1];
ar_close[ar_pointer] := ar_close[ar_pointer-1];
ar_open[ar_pointer] := ar_open[ar_pointer-1];
ar_low[ar_pointer] := ar_low[ar_pointer-1];
ar_high[ar_pointer] := ar_high[ar_pointer-1];
ar_volume[ar_pointer] := ar_volume[ar_pointer-1];
END LOOP;
ar_day_id[1] := rs.day_id;
ar_close[1] := rs.closing;
ar_open[1] := rs.open;
ar_low[1] := rs.low;
ar_high[1] := rs.high;
ar_volume[1] := rs.volume;
END IF;
rtn := (rs.stock, rs.day_id, ar_day_id, ar_close, ar_open, ar_low, ar_high, ar_volume , period);
return next rtn;
END LOOP;
END
$BODY$
LANGUAGE 'plpgsql' STABLE;

CREATE OR REPLACE FUNCTION array_to_set(ar_in numeric[])
RETURNS SETOF numeric AS
$BODY$
declare
rtn numeric;
ar_pointer int4 = 1;
BEGIN
--ar_upper := array_upper( ar_in );
WHILE ( ar_in[ar_pointer] IS NOT NULL ) LOOP
return next ar_in[ar_pointer];
ar_pointer := ar_pointer + 1;
END LOOP;
END
$BODY$
LANGUAGE 'plpgsql' STRICT IMMUTABLE;

CREATE OR REPLACE FUNCTION sum(numeric[])
RETURNS numeric AS
$BODY$
SELECT sum( array_to_set )
FROM array_to_set( $1 )
$BODY$
LANGUAGE 'sql' STRICT IMMUTABLE;

CREATE FUNCTION count_array(numeric[])
RETURNS bigint AS
$BODY$
SELECT count( array_to_set )
FROM array_to_set( $1 )
$BODY$
LANGUAGE 'sql' STRICT IMMUTABLE;

CREATE OR REPLACE FUNCTION avg(numeric[])
RETURNS numeric AS
$BODY$
SELECT avg( array_to_set )
FROM array_to_set( $1 )
$BODY$
LANGUAGE 'sql' STRICT IMMUTABLE;

CREATE OR REPLACE FUNCTION stddev(numeric[])
RETURNS numeric AS
$BODY$
SELECT stddev( array_to_set )
FROM array_to_set( $1 )
$BODY$
LANGUAGE 'sql' STRICT IMMUTABLE;

So how do I work with smaller moving sets (ie smaller array sizes). The moving arrays are indexed such that index 1 is the newest value (it corrosponds to the current row value). So slicing the array to the first 10 slices will give the last 10 rows of data. This is done by this notation:

colname[1:10] --example of returning the array itself
avg( last_set.closing[1:10] ) --example of using the avg function

select last_set.day_ids[1] AS trading_day, last_set.closing[1] AS closing,
avg( last_set.closing[1:10] ) moving_average_10,
avg( last_set.closing[1:10] ) + 2*stddev( last_set.closing[1:10] ) AS bollinger_high_10,
avg( last_set.closing[1:10] ) - 2*stddev( last_set.closing[1:10] ) AS bollinger_low_10,
avg( last_set.closing[1:21] ) AS moving_average_21,
last_set.closing[1]-last_set.open[1] AS movement,
extract( dow from date(last_set.day_ids[1]) ) AS day_of_week
from last_set('NAB', 30, NULL, NULL)
order by last_set.day_ids[1]

http://frakkle.com

Stocks table definition:

CREATE TABLE stocks(
stock varchar(10) NOT NULL,
day_id int4 NOT NULL,
open numeric,
high numeric,
low numeric,
closing numeric,
volume int4,
stock_index int4,
CONSTRAINT pk PRIMARY KEY (stock, day_id)
) WITHOUT OIDS;

A picture of what I mean when I say "moving data set":

 |------------15 row set (row 15)-----------|
 |  |------------15 row set (row 16)-----------|
 |  |  |------------15 row set (row 17)-----------|
| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|
 |----10 row set (row 10)----|  |  |  |  |  |  |  |
    |----10 row set (row 11)----|  |  |  |  |  |  |
       |----10 row set (row 12)----|  |  |  |  |  |
          |----10 row set (row 13)----|  |  |  |  |
             |----10 row set (row 14)----|  |  |  |
                |----10 row set (row 15)----|  |  |
                   |----10 row set (row 16)----|  |
                      |----10 row set (row 17)----|

Instead of linking 15 rows to every row, then linking 10 rows to every row, how about a function that returns the last n (15 above) rows in a way that can be sliced up into smaller chunks (like the last 10 rows). A function that simply adds the current row values to the beginning of the set and drops the last value off the end? The latter would mean the data itself is only looked up (stocks table row-count) once and not -

stocks table row-count * set1 row-count * set2 row-count * setn row-count... ( = a lot of work for the db! )

This post is getting long again, so I'll tell you exactly how tomorrow - with code.

http://frakkle.com

The following is an example with those two moving data sets - 21 and 10. Even though indexes are being used, it takes 40 seconds to run!

SELECT s.stock, s.day_id, s.closing, avg(s10.closing), avg(s10.closing) + 2*stddev(s10.closing) AS bollinger_high,
avg(s10.closing) - 2*stddev(s10.closing) AS bollinger_low,
avg(s21.closing) AS ma_21
FROM stocks AS s
INNER JOIN stocks s10
ON s10.stock = s.stock
AND s10.stock_index <= s.stock_index
AND s10.stock_index > s.stock_index - 10
INNER JOIN stocks s21
ON s21.stock = s.stock
AND s21.stock_index <= s.stock_index
AND s21.stock_index > s.stock_index - 21
GROUP BY s.stock, s.day_id, s.closing

GroupAggregate (cost=76987947.62..80822192.41 rows=2316 width=44) (actual time=31699.194..38996.966 rows=2316 loops=1)
-> Sort (cost=76987947.62..77371364.00 rows=153366549 width=44) (actual time=31672.742..32523.684 rows=484095 loops=1)
Sort Key: s.stock, s.day_id, s.closing
-> Nested Loop (cost=0.00..18700393.29 rows=153366549 width=44) (actual time=82.308..16989.105 rows=484095 loops=1)
Join Filter: (("inner".stock)::text = ("outer".stock)::text)
-> Nested Loop (cost=0.00..194391.82 rows=595984 width=44) (actual time=82.277..13726.831 rows=48426 loops=1)
Join Filter: ((("outer".stock)::text = ("inner".stock)::text) AND ("outer".stock_index > ("inner".stock_index - 21)))
-> Index Scan using pk on stocks s21 (cost=0.00..439.98 rows=2316 width=22) (actual time=38.341..2949.632 rows=2316 loops=1)
-> Index Scan using idx_stock_index on stocks s (cost=0.00..68.30 rows=772 width=26) (actual time=0.034..2.765 rows=1159 loops=2316)
Index Cond: ("outer".stock_index <= s.stock_index)
-> Index Scan using idx_stock_index on stocks s10 (cost=0.00..25.91 rows=257 width=22) (actual time=0.007..0.033 rows=10 loops=48426)
Index Cond: ((s10.stock_index <= "outer".stock_index) AND (s10.stock_index > ("outer".stock_index - 10)))
Total runtime: 39006.705 ms

Does not a 21 day moving average include the figures required to do a 10 day moving average?

For the details as to how, you will need to read the next installment. ;-)

http://frakkle.com