Deleting Primary credit card

I'm trying to remove a credit card in a customer from being Primary however the Primary field is greyed out. The user has permissions to perform this task and even as an admin I see the same. Any suggestion