You can use EARLY binding and LATE binding 'together' using the "Conditional Compilation Arguments" so no need to remove any reference from the project.
See the Project Properties window
Options-> Make tab
for Conditional Arguments setting.
This applies to any application that supports Office Automation.
See notes included on VB6 project Excel sample attached.