-
Mar 16th, 2012, 03:23 PM
#1
Trading System Development
6rtury
Let's post here for the time being. It's a long story,
but I'll explain more once you post here.
Spoo
-
Mar 16th, 2012, 05:46 PM
#2
New Member
Re: Trading System Development
OK, I'm here. Wow -- a PowerPoster! I'm impressed, but really shouldn't be. Have to sign off for the night (and tomorrow) as I leave early in the AM. Will check my email on Sunday (I have a really DumbPhone. I could, but don't subscribe to its web capabilities. It only makes and receives calls. I enjoy being a phone Luddite.)
J
-
Mar 26th, 2012, 02:45 PM
#3
PowerPoster
Re: Trading System Development
Hi Spoo:
Guess I'm at the point -- based on your PM's, of comparing my approach to yours. The biggest issue with mine
is locating a specific date within an array without looping the entire array. Didn't have time to work on "ClosestDate" this weekend as still tied up making sure virus did No damage. Finished late Sunday Night and kinda pooped today.
Just wanted to post to let you know I found the thread.
David
-
Mar 26th, 2012, 08:47 PM
#4
Re: Trading System Development
DW
OK, glad you found it.
Just to establish a baseline, here is a segment of an earlier discussion ...
I also have a "master" RecordSet that has 1 record per day, and always has
5 records per week. Some key fields are:
- DayNumber
- Date
- DOW ... (Mon thru Fri),
- Stat ... where possible values are
- h .. holiday
- s .. std trading day
- e .. early (ie, if a "1/2" day, wherein the mkt closes at 1:00 PM Eastern)
- c .. closed (to distinguish from "h") .. ie, the several days after 9-11.
An index on Date enables a quick Seek, and then depending on value of Stat,
I can determine the appropriate prior day wherein trading occurred.
This part of yours has me a little confused ..
The biggest issue with mine is locating a specific date within an array without looping the entire array.
Could you elaborate a bit on the characteristics of the array ...
- What is the granularity?
- Daily OHLC
- Minute bars .. but covering several days
- something else
- What are the "fields" in the array? .. I assume it is a 2-D array
- How is the array populated?
- 2-steps .. you "pre-load" the array with dates (and/or times), then "in-fill" with data from a RecordSet
- 1-step .. simply extracted from a RecordSet in reverse chronological order
- What is the context?
Spoo
-
Mar 27th, 2012, 08:08 AM
#5
PowerPoster
Re: Trading System Development
Any period -- from ticks to whatever
What are the "fields" in the array?
For data arrays only
DateTime, Open, High, Low, Close, Vol, OpnInt
How is the array populated?
"pre-load" the array (datetime order), then "in-fill" (realtime only)
Array size usually stays constant (can adjust if needed), but use
a FIFO concept on the array when filling.
Design is such that doesn't make a difference.
Last edited by dw85745; Mar 27th, 2012 at 08:13 AM.
-
Mar 27th, 2012, 09:41 AM
#6
Re: Trading System Development
DW
OK, that helps.
We've established that you "pre-load" the array with date/time.
Here is a simplistic example
Code:
Rec Date Time O H L C Vol OpnInt
0 3/26/12 09:30
1 3/26/12 09:31
2 3/26/12 09:32
3 3/26/12 09:33
4 3/26/12 09:34
...
390 3/26/12 16:00
Here again is your issue:
The biggest issue with mine is locating a specific date within an array without looping the entire array.
I'm struggling with how needing to loop to find a specific date/time "slot"
comes into play.
Is this what you mean ...
- You create the array at 9:00 am, say, before the mkt opens.
- It is now 4:00 pm
- You want to find which "Rec" corresponds to the current time "slot"
If not, could you correct me where I'm wrong.
If I'm right, is the issue needing to "quickly" find "slot" 390 (for 4:00 pm)?
If it is the latter, could you perhaps use DateDiff to determine that
the offset from "slot" 0 time is = 390, so populate array element 390,
as in ..
Code:
t1 = #9:30:00 AM#
t2 = #4:00:00 PM#
oo = DateDiff("n", t1, t2) ' = 390
Spoo
Last edited by Spoo; Mar 27th, 2012 at 09:53 AM.
-
Mar 27th, 2012, 12:26 PM
#7
PowerPoster
Re: Trading System Development
Spoo
Is this what you mean ...
You create the array at 9:00 am, say, before the mkt opens.
It is now 4:00 pm
You want to find which "Rec" corresponds to the current time "slot"
The answer is No -- NOT the current time slot, but some previous time slot.
Using your example:
Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)
So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
Using the above, a binary search "might" work since the value may be in the array.
However, if the datetime has been offloaded from the array, then a binarysearch will return Not Found.
So if the array is dimensioned at 1500 and using the FIFO concept that particular datetime has
been forced out of the array, then I need to increase the array size and bring in the missing records.
This poses both a potential memory issue (huge arrays) or figure out some way to make this seamless so all data records are available.
-
Mar 27th, 2012, 01:54 PM
#8
Re: Trading System Development
DW
Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)
So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
Let me try again, using the reverse chronological
nature of your array.
A 780 element example, assuming for simplicity
only 390 slots per day ...
Code:
Rec Date DOW Time O H L C Vol OpnInt
0 3/26/12 Mon 16:00
1 3/26/12 Mon 15:59
2 3/26/12 Mon 15:58
3 3/26/12 Mon 15:57
4 3/26/12 Mon 15:56
...
390 3/26/12 Mon 09:30
391 3/23/12 Fri 16:00
392 3/23/12 Fri 15:59
393 3/23/12 Fri 15:58
394 3/23/12 Fri 15:57
395 3/23/12 Fri 15:56
...
780 3/23/12 Fri 09:30
Does that more closely illustrate how the array looks
and the element number you would be trying to find?
Let's get this bit nailed down before proceeding.
Spoo
-
Mar 27th, 2012, 03:09 PM
#9
PowerPoster
Re: Trading System Development
Your code example is a good representation of the array contents.
Re: the element I'm looking for, it could be:
Code:
395 (which a binary search will find);
780 (which a binary search will also find;
some date > element 780 which is Not loaded into the array and which a binary search will fail to find
////////////////
The one > 780 is the one of concern.
-
Mar 27th, 2012, 03:49 PM
#10
Re: Trading System Development
DW
OK, some progress
Big question ...
Why would you ever be looking for one > 780, which
loosely translates into "a time slot larger than the array".
If, indeed, a "binary search" will find 395 and 780, then
why wouldn't you always "pre-load" the array to a number
of elements greater than the number you would ever
"search" for?
What am I missing?
Spoo
Last edited by Spoo; Mar 27th, 2012 at 03:53 PM.
-
Mar 27th, 2012, 08:46 PM
#11
PowerPoster
Re: Trading System Development
Spoo:
First -- I'm seen some programs that somehow have the ability to scroll backward for quite a lengthy period.
Based on evaluation, the array size would be enormous. They are somehow buffering a certain amount of data
within the arrays, and when they reach a certain point (probably before the max array dimension) they are
reloading the array with new data. All of this is seamless -- no delay in loading data, calculating indicators, etc.
Only thing I can think of is maybe a MemoryMapped File -- but not sure how one interfaces this to an array, or
whether the MemoryMapped File would be used in Place of the array?
In regard to:
Why would you ever be looking for one > 780,
I prefer not to discuss this in open forum.
=============================
Second -- Regarding "ClosestDate" which I have yet to explain.
Since you now understand my data arrays and how they are loaded, the problem that arises is that since ONLY active dates are loaded to the array, the array can in fact have date holes. That is:
Index {0 to 300} may be datetime of date 3/26/12, and index {301 to 780} may be 3/22/12 as 3/25/12 might be a holiday,
and 3/24 and 3/23 would be a weekend. If the user requests a date and enters either the holiday or weekend date,
rather than erroring with "Invalid Date", I would prefer that the next closest date (here 3/22) would be automatically entered.
This is where you master date table would be a "great" help", and may be even better than a closest date. But heres the problem with a table as I see it.
Different exchanges may have different holidays, so with closest date, then the data would be in control
instead of trying to monitor multiple exchanges and symbols that trade on those exchanges. This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
Last edited by dw85745; Mar 27th, 2012 at 08:51 PM.
-
Mar 28th, 2012, 09:49 AM
#12
Re: Trading System Development
DW
Let's just deal with your 2nd matter for now
This is where you master date table would be a "great" help", and may be even better than a closest date.
But heres the problem with a table as I see it.
Different exchanges may have different holidays, so with closest date, then the data would be in control instead of trying to monitor multiple exchanges and symbols that trade on those exchanges.
This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
I see 2 possible solutions:
1. Master date table approach
To deal with the "problem", perhaps you could just expand
the number of fields. As an example, I added a set of fields
for each possible instrument of interest.
- CME_SP .. pit-traded
- CME_SPmini .. e-mini's
- CME_PorkBellies
Natch, pick the exchanges and instruments that apply.
Further, to deal with the cutoff times issue for each instrument,
I have 3 fields, generically ..
- Stat
- SessionBegin
- SessionEnd
This is just a concept, and not necessarily the most efficient
database design approach. Indeed, it is probably better to have
a separate table with one record per instrument, and fields listing
session begin/end times. But it may give you some ideas.
So, put together, it might look something like this
- DayNumber
- Date
- DOW ... (Mon thru Fri),
- Stats_SP
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_SP
- SessionEnd_SP
- Stats_SPmini
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_SPmini
- SessionEnd_SPmini
- Stats_PorkBellies
- h .. holiday
- s .. std trading day
- e .. early
- c .. closed
- SessionBeg_PorkBellies
- SessionEnd_Porkbellies
2. Stored data approach
If creating the master table suggested above is deemed too
taxing, perhaps you could "construct" such a table by just
reading all of your stored data files. This assumes that you have
such files.
I envision this as a 1-time operation. Once you have "caught up",
you could devise an automated way to update said master on a
daily basis (or manually intervene as necessary).
Do either of these approaches appear acceptable?
Spoo
-
Mar 28th, 2012, 10:13 AM
#13
PowerPoster
Re: Trading System Development
Hi Spoo:
Expanding the MasterTable is one Approach --or-- better yet a MasterTable for Each Symbol probably is preferable.
However seems redundant to replicate all dates within each symbols table. Have to think on a workaround for this one.
================
Out of curosity, what timezone are you in. I'm PST.
-
Mar 28th, 2012, 11:33 AM
#14
Re: Trading System Development
DW
Yes, a MasterTable for each Symbol seems reasonable.
Regarding reduncancy issue, I guess it depends on the scope
of what you trade. Currencies (which I don't trade) probably
have different holiday dates than do equitiy index futures, just
to name one.
As for time zone, the Right Coast .. EST ..
Spoo
-
Mar 29th, 2012, 01:06 PM
#15
PowerPoster
Re: Trading System Development
Spoo:
Need some time to decide what / how I want to implement as well as deal with ClosestDate.
David
-
Mar 29th, 2012, 04:56 PM
#16
Re: Trading System Development
DW
Sounds fine.
Cogitation is a good thing.
Holler when you're ready.
Spoo
-
Apr 4th, 2012, 12:36 PM
#17
PowerPoster
Re: Trading System Development
Hi Spoo:
After looking this over, I've guess I've decided to leave things as programmed.
It's been working for a number of years and the amount of effort involved versus benefit gained is questionable.
If I migrate to another language (M$ should be drawn an quartered for dropping VB5/6 support and forcing everyone to NET), I'll consider dealing with it then.
As long as I stay with Vista and below, I think no issues, but if I upgrade OSes at some date future I'm sure this is going to break.
Thanks for all your input.
Will consider thread closed unless you have objections.
David
-
Apr 4th, 2012, 02:39 PM
#18
Re: Trading System Development
DW
Re: leaving things as programmed .. fair enough.
Re: will consider thread closed .. fair enough .. regarding your "closest date" issue.
However, I think I'll actually leave the thread open, as others (including 6rtury)
may, from time to time, pop in.
Spoo
-
Apr 4th, 2012, 03:14 PM
#19
PowerPoster
Re: Trading System Development
Hadn't forgotten about ClosetDate, just been toooooooo busy to put something together yet.
Will post here since thread open when done.
David
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|