You can't just delete a record in a binary file. The file is the size it is and you can't simply remove bytes from the middle. Appending is different because that's adding bytes to the end. If you want to delete a record then you can do one of two things:
1. Logically delete the file, which would involve setting a flag in the record to indicate that it was no longer a valid record. In that case no records would ever actually be removed from the file, which might be a problem if you add and delete a lot of records. The file might end up being large and primarily contain obsolete data.
2. Physically delete the record, which would involve reading in all the data, removing the record to delete, then writing out all the data again.
This is an example of why, for all but the simplest scenarios, databases are preferred to structured data files.