Try showing all the files in the Solution Explorer, expand the references and delete it there