FIL paying can we visit other family?

My FIL is ill with prostate cancer and has been told that he shouldn't travel, so him coming here to the states to see us as we planned for September is out. He wants to pay for us to see him in Hampshire for 2 weeks later this year, put us in a hotel and rent us a car. The trouble comes as Davids parents are divorced, really bad divorce at that. His mum is in the Lake District with her hubby and his brother is about 2 hours away from his dad but thay haven't talked in over 15 years. Is is right for us to take a couple of days to see them when his dad will be paying? His mum came to the states in 2000 and she met our daughter, but since than we had Chris who is now 2. We have not seen his brother since 1996 and none of their 3 kids. Everytime we get ready to go back something has happened and the money ends up going for emergency surgery for David, I got pregnant or buying a house. We really want to go back and have the kids see all the family, but not sure if it is right to do so. Any advice?

I forgot to say he has some type of heart trouble and this is why he is not to travel, not the cancer.