I had a similar problem when I lived in Brighton, my friend and I had a nice flat and lived above a man and his family, a 2 bed flat there were 4 of them living in there, they complained about the noise CONSTANTLY, slamming doors, walking, the TV anything, we didn't play music around or have people over, they told my landlord that lots of different men came and went (not true) and implied we were on the game.
Because he kept complaining, eventually environmental health were dispatched and they came and did a noise study and found that we were NOT causing excessive noise, but the guy still wouldn't stop, it got frightening because he had "anger problems" and was off from work to deal with them, he'd call our landlord every time we ran the washing machine or walked up the stairs and our landlord would pretty much ignore it because of the environmental health ruling. It did worry me though, one time we were watching eastenders (about 7:30) not loudly and he was banging on the door and effing and blinding, we just shouted at him out the window that we'd call the police and his wife dragged him inside - the ironic thing is he and his wife would have these blazing rows where he'd be screaming at her for hours, and she'd be sobbing loudly, and the kids would cry and cry - eventually we contacted the council about that but I don't know what if anything social services did because it was when we moved out, we'd been too scared to do it before.
This may not be what you want to hear sweetpeach, but eventually I moved out as it was causing me so much stress but on the plus side, unless you're making noise which is audible from the street by environmental health, you're pretty much fine.