I have been asked twice by immigration my teenaged son & once with my teenaged son's friend if they have permission to travel with me
First time I did not have a letter of permission from my son's father, the second time I brought along my custody papers. When we were traveling with my son & his friend, I made sure that his parent's gave me a notarized letter, just in case
So it does kind of make sense to have the letter of permission, or custody papers, if you are not asked, its no big deal, but if you are asked and you don't have it, it can be a minor hassle, or potentially a very big deal!
So as KFdancer says, it is better to err on the side of caution
good luck with your situation Greenapple