That's just a tiny part of the problem though.

There's a rule programmers have. Whatever you THINK your program spends time on is likely wrong. Use a profiler to find out where the program spends time.

This means that in several places in your loop you should measure tick counts and see where you waste all the time.