Quote Originally Posted by Arnoutdv View Post
A lot my arrays have only sparse values in them, ZLib does stunning compression on this data.
There's "Sparse-Matrix"-approaches, to hold such data efficiently InMemory
(no need for "blown-up, normal Arrays" in such cases).

Quote Originally Posted by Arnoutdv View Post
I also have a method which on my data works well for having additional compression.
It's just simply reordering the data in chunks, based on a parameter:
And that hints at, that (along to the way you stored your Data) - you might
have destroyed 'context' (Values which better be kept tightly together in a
sequence were moved "out of location").

On a more general note...

I don't have experience with "panel-research" - but I could imagine that
the "true raw data" which daily comes in from a kind of "survey-program",
might be much less "voluminous" than your current ~450MB.

With "true raw data" I mean that this data should not contain anything which
was derived by some math - or AND/OR-combined extra-queries - not sure
how much (or if at all) of such "artificially generated" data is currently contained
in your (uncompressed) daily 450MB.

Isn't there a simple (2D-DataTable-like) scheme behind such surveys -
with simple "Field-Header-Columns" like:

RespondentID, ActivityID, LocationID, BrandID, SatisfactionID, etc.

I mean, there's only so much "raw-data" a willing respondent (who's part of a panel),
is willing to provide in a daily fashion (the above table perhaps not having more
than 20 Columns or so).

Then the amount of daily raw-data (the Rows in above table) is only dependent
on the count of the respondents...

Simply cannot imagine, that this Raw-Table amounts to a daily volume of
450MB (uncompressed).

So my suspicion is, that you're trying to precalculate kind of like an
"OLAP-cube", based on a "Star-Schema" or something - to be able to
deliver "fast answers in any context" - and that these pre-calculations
are responsible for the 450MB?

Olaf