Just give them as much notice as you can. I'd give them the economical truth: I have an appointment I can't shift.
When you see Agent B, you can ask about when shifts / posts might become available and when and scale back Agent A work accordingly, with as much notice as possible, to take shifts not in conflict with Agent B, if that makes sense.
Also depends on your personal financial situation. Have you spoken with Mr GG? His thoughts?