6rtury
Let's post here for the time being. It's a long story,
but I'll explain more once you post here.
Spoo
Printable View
6rtury
Let's post here for the time being. It's a long story,
but I'll explain more once you post here.
Spoo
OK, I'm here. Wow -- a PowerPoster! I'm impressed, but really shouldn't be. Have to sign off for the night (and tomorrow) as I leave early in the AM. Will check my email on Sunday (I have a really DumbPhone. I could, but don't subscribe to its web capabilities. It only makes and receives calls. I enjoy being a phone Luddite.)
J
Hi Spoo:
Guess I'm at the point -- based on your PM's, of comparing my approach to yours. The biggest issue with mine
is locating a specific date within an array without looping the entire array. Didn't have time to work on "ClosestDate" this weekend as still tied up making sure virus did No damage. Finished late Sunday Night and kinda pooped today.
Just wanted to post to let you know I found the thread.
David
DW
OK, glad you found it.
Just to establish a baseline, here is a segment of an earlier discussion ...
I also have a "master" RecordSet that has 1 record per day, and always has
5 records per week. Some key fields are:
- DayNumber
- Date
- DOW ... (Mon thru Fri),
- Stat ... where possible values are
- h .. holiday
- s .. std trading day
- e .. early (ie, if a "1/2" day, wherein the mkt closes at 1:00 PM Eastern)
- c .. closed (to distinguish from "h") .. ie, the several days after 9-11.
An index on Date enables a quick Seek, and then depending on value of Stat,
I can determine the appropriate prior day wherein trading occurred.
This part of yours has me a little confused ..
Could you elaborate a bit on the characteristics of the array ...Quote:
The biggest issue with mine is locating a specific date within an array without looping the entire array.
Spoo
- What is the granularity?
- Daily OHLC
- Minute bars .. but covering several days
- something else
- What are the "fields" in the array? .. I assume it is a 2-D array
- Date
- Time
- Price
- others
- How is the array populated?
- 2-steps .. you "pre-load" the array with dates (and/or times), then "in-fill" with data from a RecordSet
- 1-step .. simply extracted from a RecordSet in reverse chronological order
- What is the context?
- backtesting
- real-time
- both
Any period -- from ticks to whateverQuote:
What is the granularity?
For data arrays onlyQuote:
What are the "fields" in the array?
DateTime, Open, High, Low, Close, Vol, OpnInt
"pre-load" the array (datetime order), then "in-fill" (realtime only)Quote:
How is the array populated?
Array size usually stays constant (can adjust if needed), but use
a FIFO concept on the array when filling.
Design is such that doesn't make a difference.Quote:
What is the context?
DW
OK, that helps.
We've established that you "pre-load" the array with date/time.
Here is a simplistic example
Here again is your issue:Code:Rec Date Time O H L C Vol OpnInt
0 3/26/12 09:30
1 3/26/12 09:31
2 3/26/12 09:32
3 3/26/12 09:33
4 3/26/12 09:34
...
390 3/26/12 16:00
I'm struggling with how needing to loop to find a specific date/time "slot"Quote:
The biggest issue with mine is locating a specific date within an array without looping the entire array.
comes into play.
Is this what you mean ...
- You create the array at 9:00 am, say, before the mkt opens.
- It is now 4:00 pm
- You want to find which "Rec" corresponds to the current time "slot"
If not, could you correct me where I'm wrong.
If I'm right, is the issue needing to "quickly" find "slot" 390 (for 4:00 pm)?
If it is the latter, could you perhaps use DateDiff to determine that
the offset from "slot" 0 time is = 390, so populate array element 390,
as in ..
SpooCode:t1 = #9:30:00 AM#
t2 = #4:00:00 PM#
oo = DateDiff("n", t1, t2) ' = 390
Spoo
The answer is No -- NOT the current time slot, but some previous time slot.Quote:
Is this what you mean ...
You create the array at 9:00 am, say, before the mkt opens.
It is now 4:00 pm
You want to find which "Rec" corresponds to the current time "slot"
Using your example:
Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)
So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
Using the above, a binary search "might" work since the value may be in the array.
However, if the datetime has been offloaded from the array, then a binarysearch will return Not Found.
So if the array is dimensioned at 1500 and using the FIFO concept that particular datetime has
been forced out of the array, then I need to increase the array size and bring in the missing records.
This poses both a potential memory issue (huge arrays) or figure out some way to make this seamless so all data records are available.
DW
Let me try again, using the reverse chronologicalQuote:
Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)
So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
nature of your array.
A 780 element example, assuming for simplicity
only 390 slots per day ...
Does that more closely illustrate how the array looksCode:Rec Date DOW Time O H L C Vol OpnInt
0 3/26/12 Mon 16:00
1 3/26/12 Mon 15:59
2 3/26/12 Mon 15:58
3 3/26/12 Mon 15:57
4 3/26/12 Mon 15:56
...
390 3/26/12 Mon 09:30
391 3/23/12 Fri 16:00
392 3/23/12 Fri 15:59
393 3/23/12 Fri 15:58
394 3/23/12 Fri 15:57
395 3/23/12 Fri 15:56
...
780 3/23/12 Fri 09:30
and the element number you would be trying to find?
Let's get this bit nailed down before proceeding.
Spoo
Your code example is a good representation of the array contents.
Re: the element I'm looking for, it could be:
Code:395 (which a binary search will find);
780 (which a binary search will also find;
some date > element 780 which is Not loaded into the array and which a binary search will fail to find
////////////////
The one > 780 is the one of concern.
DW
OK, some progress
Big question ...
Why would you ever be looking for one > 780, which
loosely translates into "a time slot larger than the array".
If, indeed, a "binary search" will find 395 and 780, then
why wouldn't you always "pre-load" the array to a number
of elements greater than the number you would ever
"search" for?
What am I missing?
Spoo
Spoo:
First -- I'm seen some programs that somehow have the ability to scroll backward for quite a lengthy period.Quote:
What am I missing?
Based on evaluation, the array size would be enormous. They are somehow buffering a certain amount of data
within the arrays, and when they reach a certain point (probably before the max array dimension) they are
reloading the array with new data. All of this is seamless -- no delay in loading data, calculating indicators, etc.
Only thing I can think of is maybe a MemoryMapped File -- but not sure how one interfaces this to an array, or
whether the MemoryMapped File would be used in Place of the array?
In regard to:
I prefer not to discuss this in open forum.Quote:
Why would you ever be looking for one > 780,
=============================
Second -- Regarding "ClosestDate" which I have yet to explain.
Since you now understand my data arrays and how they are loaded, the problem that arises is that since ONLY active dates are loaded to the array, the array can in fact have date holes. That is:
Index {0 to 300} may be datetime of date 3/26/12, and index {301 to 780} may be 3/22/12 as 3/25/12 might be a holiday,
and 3/24 and 3/23 would be a weekend. If the user requests a date and enters either the holiday or weekend date,
rather than erroring with "Invalid Date", I would prefer that the next closest date (here 3/22) would be automatically entered.
This is where you master date table would be a "great" help", and may be even better than a closest date. But heres the problem with a table as I see it.
Different exchanges may have different holidays, so with closest date, then the data would be in control
instead of trying to monitor multiple exchanges and symbols that trade on those exchanges. This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
DW
Let's just deal with your 2nd matter for now
I see 2 possible solutions:Quote:
This is where you master date table would be a "great" help", and may be even better than a closest date.
But heres the problem with a table as I see it.
Different exchanges may have different holidays, so with closest date, then the data would be in control instead of trying to monitor multiple exchanges and symbols that trade on those exchanges.
This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
1. Master date table approach
To deal with the "problem", perhaps you could just expand
the number of fields. As an example, I added a set of fields
for each possible instrument of interest.
Natch, pick the exchanges and instruments that apply.
- CME_SP .. pit-traded
- CME_SPmini .. e-mini's
- CME_PorkBellies
Further, to deal with the cutoff times issue for each instrument,
I have 3 fields, generically ..
- Stat
- SessionBegin
- SessionEnd
This is just a concept, and not necessarily the most efficient
database design approach. Indeed, it is probably better to have
a separate table with one record per instrument, and fields listing
session begin/end times. But it may give you some ideas.
So, put together, it might look something like this
- DayNumber
- Date
- DOW ... (Mon thru Fri),
- Stats_SP
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_SP
- SessionEnd_SP
- Stats_SPmini
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_SPmini
- SessionEnd_SPmini
- Stats_PorkBellies
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_PorkBellies
- SessionEnd_Porkbellies
2. Stored data approach
If creating the master table suggested above is deemed too
taxing, perhaps you could "construct" such a table by just
reading all of your stored data files. This assumes that you have
such files.
I envision this as a 1-time operation. Once you have "caught up",
you could devise an automated way to update said master on a
daily basis (or manually intervene as necessary).
Do either of these approaches appear acceptable?
Spoo
Hi Spoo:
Expanding the MasterTable is one Approach --or-- better yet a MasterTable for Each Symbol probably is preferable.
However seems redundant to replicate all dates within each symbols table. Have to think on a workaround for this one.
================
Out of curosity, what timezone are you in. I'm PST.
DW
Yes, a MasterTable for each Symbol seems reasonable.
Regarding reduncancy issue, I guess it depends on the scope
of what you trade. Currencies (which I don't trade) probably
have different holiday dates than do equitiy index futures, just
to name one.
As for time zone, the Right Coast .. EST .. ;)
Spoo
Spoo:
Need some time to decide what / how I want to implement as well as deal with ClosestDate.
David
DW
Sounds fine.
Cogitation is a good thing.
Holler when you're ready.
Spoo
Hi Spoo:
After looking this over, I've guess I've decided to leave things as programmed.
It's been working for a number of years and the amount of effort involved versus benefit gained is questionable.
If I migrate to another language (M$ should be drawn an quartered for dropping VB5/6 support and forcing everyone to NET), I'll consider dealing with it then.
As long as I stay with Vista and below, I think no issues, but if I upgrade OSes at some date future I'm sure this is going to break.
Thanks for all your input.
Will consider thread closed unless you have objections.
David
DW
Re: leaving things as programmed .. fair enough.
Re: will consider thread closed .. fair enough .. regarding your "closest date" issue.
However, I think I'll actually leave the thread open, as others (including 6rtury)
may, from time to time, pop in.
Spoo
Hadn't forgotten about ClosetDate, just been toooooooo busy to put something together yet.
Will post here since thread open when done.
David