We too needed to move some invoice balances from the AR code to the Retention code in GL. The method I have found that appears to work:

Create a credit memo applied to the original invoice for just the retention amount, with $0 in the Retention field.

Create a debit memo also applied against the original invoice for the same amount: enter the retention amount in the Retention field.

You can do both these entries in the same batch, and the billing code you use doesn't matter as long as it is not taxable (why complicate things) since it washes out between them. The original invoice will now show a Retention Amount, and that amount will have been moved from AR to Retention in GL.

BTW this process can be reversed if you need to clear an amount billed to retention in error: simply enter the retention amount on the credit memo and leave it at $0 on the debit memo to clear retention from the invoice.