The reason people from other countries do not have to take the test and can just exchange their license is because the U.K. has a reciprocal agreement with each of those countries to allow exchange of licences both ways.
However, in order for people from the US to be able to exchange their licences, the U.K. government would need to form 50 separate reciprocal agreements, one with each US state... but that just isn’t possible as not all the US states will agree to it. Therefore US residents/citizens have to go through the driving test process from scratch instead of being able to simply exchange their licence.
Edited to add:
Just realised you are already aware of this... as far as I know, the lack of reciprocal agreements is the only reason you have to take the tests.
Sent from my iPhone using Tapatalk