My main point will the uk border agency be okay with her and the children on a one way ticket into the uk with of course our intention to buy her return tickets after her couple of months stay to return back to the uk
Short answer: no, they won't be okay with it.
The most important thing to have when you enter the UK as a visitor is a return ticket. No return ticket means almost an automatic refusal to enter the country, or at the very least, extensive questioning from UK immigration (especially if you want to stay for 2 months and are travelling with a UK citizen fiance and 2 kids).
On top of the return ticket, your fiance will also need to prove that she and the kids will definitely leave after two months - she will need evidence of enough money to support the whole family for the whole 2 months without her working, a home, job and life in the US to return to, by means of rental leases/mortgage statements, a letter from her employer giving her time off and specifying when she must return to work, evidence of when the kids must be back in school (letter from the school), permission from the kids' father giving her permission to take them out of the US etc.
Once she is back I will then apply for our visas.
Just to clarify: she will need to apply for the visas, not you. The application process must be done by the applicant from inside the US. You will need to provide her with your supporting documents for the visas, but she is the one who has to apply.
Also remember that the kids will need a visa each, so you're looking at £2,478 in visa fees (£826 each - although this will probably have gone up by the time she applies, as the fees usually increase in April each year), plus if she wants to pay for priority visa processing (15 working days), that's another $900 ($300 per visa) on top.
Are the children school age? If so, with or without return tickets that's going to raise questions .
Very good point - as visitors, it will be illegal for them to attend school in the UK, unless you pay for them to go to a private school (they will not be entitled to free state education), so what will you do about their schooling while they are in the UK?
Why can't you buy round trip tickets? One way tickets are often more expensive, so I can't understand why you wouldn't buy round trip tickets in the first place, particularly given the sensitivity of the family traveling without visas.
Exactly - I've always bought return tickets, even when I only needed a one-way, because it was cheaper. When I went to the US for a holiday in June 2011, if I remember correctly, my return ticket was about £580... and a one-way ticket was over £600!
