My understanding is that as your daughter would be entitled to US citizenship, she must enter the US on a US passport. So, she may be refused entry anyhow ... but in reality I don't know how they would know she has a claim to US citizenship if she is on a UK passport stating her birthplace as the UK, with a UK parent. I would suggest calling the US Embassy, they may be prepared to set up an alert for her on their system.
I'd recommend getting your custody issues formalised asap though.
Also, if your ex does go aahead with this trip and you are truly worried for your child's safety, I'd get in contact with the child protection people in the state he's visiting.