What he tends to lose sight of is the fact that his older sister tends to take the normal human tendency to focus on what's gone wrong in her life and dialed it up to eleven. While he sees the two of them as having had the very normal childhood that they actually seem to have had, he doesn't want to admit that Elly sees things as having gone quite differently. He remembers himself as not having gotten away with a Hell of a lot so doesn't get why Elly tells everyone that he was a little pasha who was allowed to do whatever he wanted because he was a boy while she was practically a slave. It wasn't personal, he and their parents weren't setting out to be cruel to her and did what they thought that they had to so it annoys him that she wants to him to grovel pathetically for forgiveness for laughing at her misery when he can't remember a day when he didn't live in fear of her rage and malice.
This sort of thing comes to a head eight years from now when Jim and Marian downsize. Elly clearly wants the old pump organ as a sort of symbol that as oldest child, she should have the right of first refusal over everything. She sees Phil's wanting to have the damned thing because unlike her, he can actually play it as a sign that Phil gets everything ever because he's a boy and because she happens to be a woman, she is not allowed to win anything ever. The look of horror on their faces when they aren't allowed to see who the favourite child really is because Jim gave the bloody thing away is sort of priceless. I mean, all that build-up to settling a pointless argument wasted.
It makes me wonder what sort of mud-slinging went on when Jim died. Given that Elly never really lets anything go, it makes a certain amount of sense that while Mike was delivering an overwrought eulogy loaded with factual errors, Lizardbreath huffing about how unreasonable April was about her doing something that made her feel good and John standing in the background with the usual nothing on his alleged mind, Phil and Elly were still bitching about who was more loved. Also, Phil was probably headed down to the drug store to stock up on nicotine patches.