So an update!
I wrote a very long letter in legal speak to my landlord, about why things weren't justified and also to show me the receipts for his work- he was really pissed off at me, because we had previously been very cordial.
He wrote me an email trying to be nice again, but still not justifying any of my claims against him, just reiterating what he said before.
I went away on holiday and had absolutely no chance to write a response to him.
I have now gotten a letter which was really cheeky and rude with a cheque for my whole deposit back. He closed it along the lines of "I'll leave it up to you if your concious allows you to deposit the whole thing"
Yes, my concious does allow, thank you.
So on one hand, I am quite chuffed that he's given me all £650 back, but on the other hand, I am upset that I have made someone pissed off at me. Still, not upset enough to not take my whole deposit back! I will write him a thank you letter and say no more!