I'm assuming your wife is NatCol, since she also posted about this, but I didn't see it until after I had replied to you so wasn't aware of your situation.
To enter as a visitor you would normally need to show a return ticket to leave the UK and evidence of a life to return to in the US (home, job etc.).
However, you have previous refused permit/residence card applications so it's unlikely you would be allowed into the UK without applying for a visitor visa in advance, but you already have the family permit, so that wouldn't work.
Honestly, I don't know what will happen if you try to enter on the 5th - you have only been given permission to enter the UK on the permit after August 30th and you are not supposed to enter before that date.
You can't get the permit stamped once you are already in the UK because it can only be stamped on initial entry, and I'm assuming it can't be stamped before the 30th... Which means you would need to do a channel hop to France or something similar in order to get it stamped when it becomes valid.
Personally, my advice would be to change your flight and not try to enter the UK until the 30th. That way, you can guarantee no issues with getting into the UK.
You could try flying on the 5th, but I really don't know whether they would let you in or not... You could end up making things worse for yourself and being sent back to the US if they won't let you in. I really don't know what will happen though.
Sent from my iPhone using Tapatalk