I guess I would consider at least limiting the number of controls.
Perhaps create four panels, each with the same set of controls.
You position the four panels vertically inside your scrolling panel.
You are displaying the top two panels.
As they scroll through the records, you get to the point where the two middle panels are displayed, if they continue scrolling and the fourth one is beginning to come into view, reposition the first panel to the end of the stack, and load it with the fifth record data.
Each time you have 1.x panels off the top, you reposition the top one to the bottom and load the appropriate record.
If they start scrolling the other way, when you have 1.x panels off the bottom, move the bottom panel to the top, load the appropriate data into it.