That does sort of make sense because he and Phil wound up looking like two dumb jerks from the city who didn't know what they were doing and feeling the burn of having the locals deride them for their epic failure. As far as I know, neither of them go back there because the idea of being laughed at for being two dumb motherhubbards who should have hired a competent guide instead of witlessly risking their lives makes returning there untenable.
This talk of 'guides' is, of course, not the only instance in which their fragile little egos make the whole thing into a humiliation they can never live down. This is because before things started, someone who wasn't supposed to know what she was talking about did while at the same time revealing that someone who was didn't. Since one of the things he wanted to get away from was Elly being right about his heading for a fall, John simply couldn't face the neighbors for weeks because he fears and hates being laughed at when he makes a fool of himself. Eventually, he finds a way of getting away from it all without going anywhere when he gets that damned train fixation. For now, though, he's going to be "Panicky Idiot Number Two."