Keep Scale while Zooming

I have a set of drawing tools (e.g. pencil) where I draw on a graph.

I also display the endpoint values of the line.

When I zoom, the base Number of the Y axis is recalculated which
affects all other plotted numbers as the Y Axis scale distance changes and all numbers are calculated off this scale.

Any suggestions on how to plot the numbers in relation to the scale yet keep the endpoint values correct without fooling the scale and plotting the original calculated endpoint value?

Thanks
David