Results 1 to 19 of 19

Thread: Trading System Development

  1. #1

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Trading System Development

    6rtury

    Let's post here for the time being. It's a long story,
    but I'll explain more once you post here.

    Spoo

  2. #2
    New Member
    Join Date
    Mar 2007
    Posts
    13

    Re: Trading System Development

    OK, I'm here. Wow -- a PowerPoster! I'm impressed, but really shouldn't be. Have to sign off for the night (and tomorrow) as I leave early in the AM. Will check my email on Sunday (I have a really DumbPhone. I could, but don't subscribe to its web capabilities. It only makes and receives calls. I enjoy being a phone Luddite.)

    J

  3. #3
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Hi Spoo:

    Guess I'm at the point -- based on your PM's, of comparing my approach to yours. The biggest issue with mine
    is locating a specific date within an array without looping the entire array. Didn't have time to work on "ClosestDate" this weekend as still tied up making sure virus did No damage. Finished late Sunday Night and kinda pooped today.
    Just wanted to post to let you know I found the thread.

    David

  4. #4

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    OK, glad you found it.

    Just to establish a baseline, here is a segment of an earlier discussion ...

    I also have a "master" RecordSet that has 1 record per day, and always has
    5 records per week. Some key fields are:
    • DayNumber
    • Date
    • DOW ... (Mon thru Fri),
    • Stat ... where possible values are
      • h .. holiday
      • s .. std trading day
      • e .. early (ie, if a "1/2" day, wherein the mkt closes at 1:00 PM Eastern)
      • c .. closed (to distinguish from "h") .. ie, the several days after 9-11.

    An index on Date enables a quick Seek, and then depending on value of Stat,
    I can determine the appropriate prior day wherein trading occurred.

    This part of yours has me a little confused ..
    The biggest issue with mine is locating a specific date within an array without looping the entire array.
    Could you elaborate a bit on the characteristics of the array ...
    • What is the granularity?
      • Daily OHLC
      • Minute bars .. but covering several days
      • something else
    • What are the "fields" in the array? .. I assume it is a 2-D array
      • Date
      • Time
      • Price
      • others
    • How is the array populated?
      • 2-steps .. you "pre-load" the array with dates (and/or times), then "in-fill" with data from a RecordSet
      • 1-step .. simply extracted from a RecordSet in reverse chronological order
    • What is the context?
      • backtesting
      • real-time
      • both
    Spoo

  5. #5
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    What is the granularity?
    Any period -- from ticks to whatever

    What are the "fields" in the array?
    For data arrays only
    DateTime, Open, High, Low, Close, Vol, OpnInt

    How is the array populated?
    "pre-load" the array (datetime order), then "in-fill" (realtime only)
    Array size usually stays constant (can adjust if needed), but use
    a FIFO concept on the array when filling.

    What is the context?
    Design is such that doesn't make a difference.
    Last edited by dw85745; Mar 27th, 2012 at 08:13 AM.

  6. #6

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    OK, that helps.
    We've established that you "pre-load" the array with date/time.

    Here is a simplistic example
    Code:
    Rec  Date      Time   O   H   L   C  Vol  OpnInt
    0    3/26/12   09:30
    1    3/26/12   09:31
    2    3/26/12   09:32
    3    3/26/12   09:33
    4    3/26/12   09:34
    ...
    390  3/26/12   16:00
    Here again is your issue:
    The biggest issue with mine is locating a specific date within an array without looping the entire array.
    I'm struggling with how needing to loop to find a specific date/time "slot"
    comes into play.

    Is this what you mean ...
    • You create the array at 9:00 am, say, before the mkt opens.
    • It is now 4:00 pm
    • You want to find which "Rec" corresponds to the current time "slot"

    If not, could you correct me where I'm wrong.
    If I'm right, is the issue needing to "quickly" find "slot" 390 (for 4:00 pm)?

    If it is the latter, could you perhaps use DateDiff to determine that
    the offset from "slot" 0 time is = 390, so populate array element 390,
    as in ..
    Code:
    t1 = #9:30:00 AM#
    t2 = #4:00:00 PM#
    oo = DateDiff("n", t1, t2)    ' = 390
    Spoo
    Last edited by Spoo; Mar 27th, 2012 at 09:53 AM.

  7. #7
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Spoo

    Is this what you mean ...

    You create the array at 9:00 am, say, before the mkt opens.
    It is now 4:00 pm
    You want to find which "Rec" corresponds to the current time "slot"
    The answer is No -- NOT the current time slot, but some previous time slot.

    Using your example:

    Lets say the current datetime is: 3/26/12 16:00
    Rather than the array index being 390, mine is zero (I load the most current into array index 0)

    So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
    Using the above, a binary search "might" work since the value may be in the array.

    However, if the datetime has been offloaded from the array, then a binarysearch will return Not Found.
    So if the array is dimensioned at 1500 and using the FIFO concept that particular datetime has
    been forced out of the array, then I need to increase the array size and bring in the missing records.
    This poses both a potential memory issue (huge arrays) or figure out some way to make this seamless so all data records are available.

  8. #8

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    Lets say the current datetime is: 3/26/12 16:00
    Rather than the array index being 390, mine is zero (I load the most current into array index 0)

    So where my problem lies is finding the array index for say: 3/23/12 9:30 AM.
    Let me try again, using the reverse chronological
    nature of your array.

    A 780 element example, assuming for simplicity
    only 390 slots per day ...

    Code:
    Rec  Date      DOW   Time   O   H   L   C  Vol  OpnInt
    0    3/26/12   Mon   16:00
    1    3/26/12   Mon   15:59
    2    3/26/12   Mon   15:58
    3    3/26/12   Mon   15:57
    4    3/26/12   Mon   15:56
    ...
    390  3/26/12   Mon   09:30
    391  3/23/12   Fri   16:00
    392  3/23/12   Fri   15:59
    393  3/23/12   Fri   15:58
    394  3/23/12   Fri   15:57
    395  3/23/12   Fri   15:56
    ...
    780  3/23/12   Fri   09:30
    Does that more closely illustrate how the array looks
    and the element number you would be trying to find?

    Let's get this bit nailed down before proceeding.

    Spoo

  9. #9
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Your code example is a good representation of the array contents.

    Re: the element I'm looking for, it could be:

    Code:
    395 (which a binary search will find);
    780 (which a binary search will also find;
    some date > element 780 which is Not loaded into the array and which a binary search will fail to find
    ////////////////
    The one > 780 is the one of concern.

  10. #10

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    OK, some progress

    Big question ...

    Why would you ever be looking for one > 780, which
    loosely translates into "a time slot larger than the array".

    If, indeed, a "binary search" will find 395 and 780, then
    why wouldn't you always "pre-load" the array to a number
    of elements greater than the number you would ever
    "search" for?

    What am I missing?

    Spoo
    Last edited by Spoo; Mar 27th, 2012 at 03:53 PM.

  11. #11
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Spoo:

    What am I missing?
    First -- I'm seen some programs that somehow have the ability to scroll backward for quite a lengthy period.
    Based on evaluation, the array size would be enormous. They are somehow buffering a certain amount of data
    within the arrays, and when they reach a certain point (probably before the max array dimension) they are
    reloading the array with new data. All of this is seamless -- no delay in loading data, calculating indicators, etc.

    Only thing I can think of is maybe a MemoryMapped File -- but not sure how one interfaces this to an array, or
    whether the MemoryMapped File would be used in Place of the array?

    In regard to:

    Why would you ever be looking for one > 780,
    I prefer not to discuss this in open forum.

    =============================

    Second -- Regarding "ClosestDate" which I have yet to explain.
    Since you now understand my data arrays and how they are loaded, the problem that arises is that since ONLY active dates are loaded to the array, the array can in fact have date holes. That is:

    Index {0 to 300} may be datetime of date 3/26/12, and index {301 to 780} may be 3/22/12 as 3/25/12 might be a holiday,
    and 3/24 and 3/23 would be a weekend. If the user requests a date and enters either the holiday or weekend date,
    rather than erroring with "Invalid Date", I would prefer that the next closest date (here 3/22) would be automatically entered.

    This is where you master date table would be a "great" help", and may be even better than a closest date. But heres the problem with a table as I see it.

    Different exchanges may have different holidays, so with closest date, then the data would be in control
    instead of trying to monitor multiple exchanges and symbols that trade on those exchanges. This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
    Last edited by dw85745; Mar 27th, 2012 at 08:51 PM.

  12. #12

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    Let's just deal with your 2nd matter for now

    This is where you master date table would be a "great" help", and may be even better than a closest date.
    But heres the problem with a table as I see it.

    Different exchanges may have different holidays, so with closest date, then the data would be in control instead of trying to monitor multiple exchanges and symbols that trade on those exchanges.

    This is especially true even with the CME, as each futures contract may have different holidays and/or time cutoffs.
    I see 2 possible solutions:

    1. Master date table approach

    To deal with the "problem", perhaps you could just expand
    the number of fields. As an example, I added a set of fields
    for each possible instrument of interest.
    • CME_SP .. pit-traded
    • CME_SPmini .. e-mini's
    • CME_PorkBellies
    Natch, pick the exchanges and instruments that apply.

    Further, to deal with the cutoff times issue for each instrument,
    I have 3 fields, generically ..
    • Stat
    • SessionBegin
    • SessionEnd

    This is just a concept, and not necessarily the most efficient
    database design approach. Indeed, it is probably better to have
    a separate table with one record per instrument, and fields listing
    session begin/end times. But it may give you some ideas.

    So, put together, it might look something like this
    1. DayNumber
    2. Date
    3. DOW ... (Mon thru Fri),
    4. Stats_SP
      • h .. holiday
      • s .. std trading day
      • e .. early
      • c .. closed
    5. SessionBeg_SP
    6. SessionEnd_SP
    7. Stats_SPmini
      • h .. holiday
      • s .. std trading day
      • e .. early
      • c .. closed
    8. SessionBeg_SPmini
    9. SessionEnd_SPmini
    10. Stats_PorkBellies
      • h .. holiday
      • s .. std trading day
      • e .. early
      • c .. closed
    11. SessionBeg_PorkBellies
    12. SessionEnd_Porkbellies

    2. Stored data approach

    If creating the master table suggested above is deemed too
    taxing, perhaps you could "construct" such a table by just
    reading all of your stored data files. This assumes that you have
    such files.

    I envision this as a 1-time operation. Once you have "caught up",
    you could devise an automated way to update said master on a
    daily basis (or manually intervene as necessary).

    Do either of these approaches appear acceptable?

    Spoo

  13. #13
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Hi Spoo:

    Expanding the MasterTable is one Approach --or-- better yet a MasterTable for Each Symbol probably is preferable.
    However seems redundant to replicate all dates within each symbols table. Have to think on a workaround for this one.

    ================

    Out of curosity, what timezone are you in. I'm PST.

  14. #14

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    Yes, a MasterTable for each Symbol seems reasonable.

    Regarding reduncancy issue, I guess it depends on the scope
    of what you trade. Currencies (which I don't trade) probably
    have different holiday dates than do equitiy index futures, just
    to name one.

    As for time zone, the Right Coast .. EST ..

    Spoo

  15. #15
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Spoo:

    Need some time to decide what / how I want to implement as well as deal with ClosestDate.

    David

  16. #16

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    Sounds fine.
    Cogitation is a good thing.
    Holler when you're ready.

    Spoo

  17. #17
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Hi Spoo:

    After looking this over, I've guess I've decided to leave things as programmed.
    It's been working for a number of years and the amount of effort involved versus benefit gained is questionable.

    If I migrate to another language (M$ should be drawn an quartered for dropping VB5/6 support and forcing everyone to NET), I'll consider dealing with it then.
    As long as I stay with Vista and below, I think no issues, but if I upgrade OSes at some date future I'm sure this is going to break.

    Thanks for all your input.
    Will consider thread closed unless you have objections.

    David

  18. #18

    Thread Starter
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: Trading System Development

    DW

    Re: leaving things as programmed .. fair enough.
    Re: will consider thread closed .. fair enough .. regarding your "closest date" issue.

    However, I think I'll actually leave the thread open, as others (including 6rtury)
    may, from time to time, pop in.

    Spoo

  19. #19
    PowerPoster
    Join Date
    Jul 2001
    Location
    Tucson, AZ
    Posts
    2,166

    Re: Trading System Development

    Hadn't forgotten about ClosetDate, just been toooooooo busy to put something together yet.
    Will post here since thread open when done.

    David

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width