So I've experienced this same thing twice. Once was in my fiances 1999 Honda Accord. Then it was most noticeable at highway speeds whenever you would go to slow down, and it didn't matter if it was a casual slowing down or a more sudden slow down. It would always seem to pull the same way as well, and I forget which. She was running 3 different types of tires(don't ask me!

) and they all had different speed ratings except for the two that were the same(and they weren't both at the front). One day I was a bit fed up with it, and as they were worn down significantly, and landed some decent tires on sale. The problem went away after that.
The other time I had something like this happen was with a mid-90's MR2 I had. It too only did it when slowing down, and it would sway in either direction, sometimes stronger than others, and sometimes none at all. I never solved this issue, but it had 200k on it, and I started thinking after the fact that perhaps it had some wear between the teeth and the rack at that point. When slowing, something with the R&P would shift forward and with the worn gap, it would then slide a bit to one side or the other. I never solved it and sold it on at some point, and that idea is a bit speculative, but maybe a possibility. The Pathfinder is relatively heavy, and could wear faster here that usual, depending on driving style/surfaces.