The reason that I say this is that he still has yet to realize even now that he's in his early sixties that Elly has never been what you'd call an education mother. He doesn't have the advantage of reading the strip so he has no idea that most of her life has been spent shooing the children away because she can't think straight when she has to keep track of what other people happen to be doing and can't admit that children might actually be interested in what she's doing let alone admitting that they'd like to be part of the process. As I've said before, the poor dope doesn't seem to realize that Elly learned the wrong lesson from watching their parents parent. I can see a world in which Jim and Marian actually did try to include the kids in the process when they had the time and I can see Phil remembering that and assuming that that's what Elly is doing. Unlike him, I can see a world in which Elly came to the conclusion "Don't let children get underfoot or you'll never get anything done. Mom shooed us away and we turned out great so she must know what she's doing."
What this means is that he doesn't realize that there's never been a moment wherein Elly could communicate with her kids that she couldn't somehow or other dodge in the unswerving belief that doing so is the best thing for all concerned. On the rare occasions she does try, the kids are so unused to the idea of being able to talk to her, they don't know what to do.