Most philosophers who reject that Barry and Gary have swapped bodies are animalists. Animalists, such as Eric Olson, believe that we are fundamentally organisms or animals, Homo Sapiens, and as such, just like other organisms, we normally acquire psychological capacities as we develop, and may lose those capacities through injury or illness. So, although having psychological states is a normal part of what we are, they are not essential for our continued existence.
But now consider the teletransporters found in Star Trek. These work by destroying the original organism on the space ship and assembling a duplicate organism, psychologically continuous but made from different pieces of matter, on the planet below. For the animalist, teletransporting represents death, since the organism we are is destroyed and a new organism created. But if you were being attacked by Klingons, wouldn’t you use a teletransporter to escape?