Trading System Development

[Spoo]

6rtury

Let's post here for the time being. It's a long story,
but I'll explain more once you post here.

Spoo

[6rtury]

OK, I'm here. Wow -- a PowerPoster! I'm impressed, but really shouldn't be. Have to sign off for the night (and tomorrow) as I leave early in the AM. Will check my email on Sunday (I have a really DumbPhone. I could, but don't subscribe to its web capabilities. It only makes and receives calls. I enjoy being a phone Luddite.)

J

[dw85745]

Hi Spoo:

Guess I'm at the point -- based on your PM's, of comparing my approach to yours. The biggest issue with mine
is locating a specific date within an array without looping the entire array. Didn't have time to work on "ClosestDate" this weekend as still tied up making sure virus did No damage. Finished late Sunday Night and kinda pooped today.
Just wanted to post to let you know I found the thread.

David

[Spoo]

DW

OK, glad you found it.

Just to establish a baseline, here is a segment of an earlier discussion ...

I also have a "master" RecordSet that has 1 record per day, and always has
5 records per week. Some key fields are:

• DayNumber
• Date
• DOW ... (Mon thru Fri),
• Stat ... where possible values are
  
  • h .. holiday
  • s .. std trading day
  • e .. early (ie, if a "1/2" day, wherein the mkt closes at 1:00 PM Eastern)
  • c .. closed (to distinguish from "h") .. ie, the several days after 9-11.

An index on Date enables a quick Seek, and then depending on value of Stat,
I can determine the appropriate prior day wherein trading occurred.

This part of yours has me a little confused ..

The biggest issue with mine is locating a specific date within an array without looping the entire array.

Could you elaborate a bit on the characteristics of the array ...

• What is the granularity?
  
  • Daily OHLC
  • Minute bars .. but covering several days
  • something else
• What are the "fields" in the array? .. I assume it is a 2-D array
  
  • Date
  • Time
  • Price
  • others
• How is the array populated?
  
  • 2-steps .. you "pre-load" the array with dates (and/or times), then "in-fill" with data from a RecordSet
  • 1-step .. simply extracted from a RecordSet in reverse chronological order
• What is the context?
  
  • backtesting
  • real-time
  • both

Spoo

[dw85745]

What is the granularity?

Any period -- from ticks to whatever

What are the "fields" in the array?

For data arrays only
DateTime, Open, High, Low, Close, Vol, OpnInt

How is the array populated?

"pre-load" the array (datetime order), then "in-fill" (realtime only)
Array size usually stays constant (can adjust if needed), but use
a FIFO concept on the array when filling.

What is the context?

Design is such that doesn't make a difference.

[Spoo]

DW

OK, that helps.
We've established that you "pre-load" the array with date/time.

Here is a simplistic example

Rec  Date      Time   O   H   L   C  Vol  OpnInt
0    3/26/12   09:30
1    3/26/12   09:31
2    3/26/12   09:32
3    3/26/12   09:33
4    3/26/12   09:34
...
390  3/26/12   16:00

Here again is your issue:

The biggest issue with mine is locating a specific date within an array without looping the entire array.

I'm struggling with how needing to loop to find a specific date/time "slot"
comes into play.

Is this what you mean ...

• You create the array at 9:00 am, say, before the mkt opens.
• It is now 4:00 pm
• You want to find which "Rec" corresponds to the current time "slot"

If not, could you correct me where I'm wrong.
If I'm right, is the issue needing to "quickly" find "slot" 390 (for 4:00 pm)?

If it is the latter, could you perhaps use DateDiff to determine that
the offset from "slot" 0 time is = 390, so populate array element 390,
as in ..

t1 = #9:30:00 AM#
t2 = #4:00:00 PM#
oo = DateDiff("n", t1, t2)    ' = 390

Spoo

[dw85745]

Spoo

Is this what you mean ...

You create the array at 9:00 am, say, before the mkt opens.
It is now 4:00 pm
You want to find which "Rec" corresponds to the current time "slot"

The answer is No -- NOT the current time slot, but some previous time slot.

Using your example:

Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)

So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
Using the above, a binary search "might" work since the value may be in the array.

However, if the datetime has been offloaded from the array, then a binarysearch will return Not Found.
So if the array is dimensioned at 1500 and using the FIFO concept that particular datetime has
been forced out of the array, then I need to increase the array size and bring in the missing records.
This poses both a potential memory issue (huge arrays) or figure out some way to make this seamless so all data records are available.

[Spoo]

DW

Lets say the current datetime is: 3/26/12 16:00
Rather than the array index being 390, mine is zero (I load the most current into array index 0)

So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.

Let me try again, using the reverse chronological
nature of your array.

A 780 element example, assuming for simplicity
only 390 slots per day ...

Rec  Date      DOW   Time   O   H   L   C  Vol  OpnInt
0    3/26/12   Mon   16:00
1    3/26/12   Mon   15:59
2    3/26/12   Mon   15:58
3    3/26/12   Mon   15:57
4    3/26/12   Mon   15:56
...
390  3/26/12   Mon   09:30
391  3/23/12   Fri   16:00
392  3/23/12   Fri   15:59
393  3/23/12   Fri   15:58
394  3/23/12   Fri   15:57
395  3/23/12   Fri   15:56
...
780  3/23/12   Fri   09:30

Does that more closely illustrate how the array looks
and the element number you would be trying to find?

Let's get this bit nailed down before proceeding.

Spoo

[dw85745]

Your code example is a good representation of the array contents.

Re: the element I'm looking for, it could be:

395 (which a binary search will find);
780 (which a binary search will also find;
some date > element 780 which is Not loaded into the array and which a binary search will fail to find
////////////////
The one > 780 is the one of concern.

[Spoo]

DW

OK, some progress

Big question ...

Why would you ever be looking for one > 780, which
loosely translates into "a time slot larger than the array".

If, indeed, a "binary search" will find 395 and 780, then
why wouldn't you always "pre-load" the array to a number
of elements greater than the number you would ever
"search" for?

What am I missing?

Spoo

[dw85745]

Spoo:

What am I missing?

First -- I'm seen some programs that somehow have the ability to scroll backward for quite a lengthy period.
Based on evaluation, the array size would be enormous. They are somehow buffering a certain amount of data
within the arrays, and when they reach a certain point (probably before the max array dimension) they are
reloading the array with new data. All of this is seamless -- no delay in loading data, calculating indicators, etc.

Only thing I can think of is maybe a MemoryMapped File -- but not sure how one interfaces this to an array, or
whether the MemoryMapped File would be used in Place of the array?

In regard to:

Why would you ever be looking for one > 780,

I prefer not to discuss this in open forum.

=============================

Second -- Regarding "ClosestDate" which I have yet to explain.
Since you now understand my data arrays and how they are loaded, the problem that arises is that since ONLY active dates are loaded to the array, the array can in fact have date holes. That is:

Index {0 to 300} may be datetime of date 3/26/12, and index {301 to 780} may be 3/22/12 as 3/25/12 might be a holiday,
and 3/24 and 3/23 would be a weekend. If the user requests a date and enters either the holiday or weekend date,
rather than erroring with "Invalid Date", I would prefer that the next closest date (here 3/22) would be automatically entered.

This is where you master date table would be a "great" help", and may be even better than a closest date. But heres the problem with a table as I see it.

Different exchanges may have different holidays, so with closest date, then the data would be in control
instead of trying to monitor multiple exchanges and symbols that trade on those exchanges. This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.

[Spoo]

DW

Let's just deal with your 2nd matter for now

This is where you master date table would be a "great" help", and may be even better than a closest date.
But heres the problem with a table as I see it.

Different exchanges may have different holidays, so with closest date, then the data would be in control instead of trying to monitor multiple exchanges and symbols that trade on those exchanges.

This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.

I see 2 possible solutions:

1. Master date table approach

To deal with the "problem", perhaps you could just expand
the number of fields. As an example, I added a set of fields
for each possible instrument of interest.

• CME_SP .. pit-traded
• CME_SPmini .. e-mini's
• CME_PorkBellies

Natch, pick the exchanges and instruments that apply.

Further, to deal with the cutoff times issue for each instrument,
I have 3 fields, generically ..

• Stat
• SessionBegin
• SessionEnd

This is just a concept, and not necessarily the most efficient
database design approach. Indeed, it is probably better to have
a separate table with one record per instrument, and fields listing
session begin/end times. But it may give you some ideas.

So, put together, it might look something like this

1. DayNumber
2. Date
3. DOW ... (Mon thru Fri),
4. Stats_SP
   
   • h .. holiday
   • s .. std trading day
   • e .. early
   • c .. closed
5. SessionBeg_SP
6. SessionEnd_SP
7. Stats_SPmini
   
   • h .. holiday
   • s .. std trading day
   • e .. early
   • c .. closed
8. SessionBeg_SPmini
9. SessionEnd_SPmini
10. Stats_PorkBellies
    
    • h .. holiday
    • s .. std trading day
    • e .. early
    • c .. closed
11. SessionBeg_PorkBellies
12. SessionEnd_Porkbellies

2. Stored data approach

If creating the master table suggested above is deemed too
taxing, perhaps you could "construct" such a table by just
reading all of your stored data files. This assumes that you have
such files.

I envision this as a 1-time operation. Once you have "caught up",
you could devise an automated way to update said master on a
daily basis (or manually intervene as necessary).

Do either of these approaches appear acceptable?

Spoo

[dw85745]

Hi Spoo:

Expanding the MasterTable is one Approach --or-- better yet a MasterTable for Each Symbol probably is preferable.
However seems redundant to replicate all dates within each symbols table. Have to think on a workaround for this one.

================

Out of curosity, what timezone are you in. I'm PST.

[Spoo]

DW

Yes, a MasterTable for each Symbol seems reasonable.

Regarding reduncancy issue, I guess it depends on the scope
of what you trade. Currencies (which I don't trade) probably
have different holiday dates than do equitiy index futures, just
to name one.

As for time zone, the Right Coast .. EST ..

Spoo

[dw85745]

Spoo:

Need some time to decide what / how I want to implement as well as deal with ClosestDate.

David

[Spoo]

DW

Sounds fine.
Cogitation is a good thing.
Holler when you're ready.

Spoo

[dw85745]

Hi Spoo:

After looking this over, I've guess I've decided to leave things as programmed.
It's been working for a number of years and the amount of effort involved versus benefit gained is questionable.

If I migrate to another language (M$ should be drawn an quartered for dropping VB5/6 support and forcing everyone to NET), I'll consider dealing with it then.
As long as I stay with Vista and below, I think no issues, but if I upgrade OSes at some date future I'm sure this is going to break.

Thanks for all your input.
Will consider thread closed unless you have objections.

David

[Spoo]

DW

Re: leaving things as programmed .. fair enough.
Re: will consider thread closed .. fair enough .. regarding your "closest date" issue.

However, I think I'll actually leave the thread open, as others (including 6rtury)
may, from time to time, pop in.

Spoo

[dw85745]

Hadn't forgotten about ClosetDate, just been toooooooo busy to put something together yet.
Will post here since thread open when done.

David