It's in the very nature of raster images not to scale well. You could design the buttons entirely as Flash vector components, then they'd scale. You could use larger images as source, then the loss of quality won't be as noticeable.
GIF (or rather PNG) or JPG has little to do with it, although for most buttons, PNG is the better choice because of the nature of the image.