You mention an error, but I'm assuming you mean unexpected results rather than an error message.

Based on your textual description of the intended order of events, and your example formulas that you report are working properly, the error appears to be an order of operations error in your vba code. In your code, "SQRD Mod 400" takes place before "/ decx".

So, this should do things in the order you describe.

Code:
For Count = 1 To looptime
    SQRD = CDec(SQRD ^ 2)  '+ looptime
    SQRD = CDec(SQRD / decx)  Mod 400
Next Count