MsgBox returns an Integer and not a string and that may be your problem, but this should work in any case:
Code:
If vbNo = MsgBox("Really quit?", vbYesNo) Then
    Cancel = True 
Else 
    Unload Me 
End If
You also need to make sure that you have the code in a routine like the QueryUnload event that has a Cancel value.