I totally agree with you, that's why I think the swap was so suspicious:

The swap in All-Stars was at random. They showed the buffs being pulled. The odds of Amber staying put and EVERYONE else switching? Same odds as any other tribe combination. It could have been Big Tom going to the other tribe. It could have put each tribe at a 3-2 split, which seems likelier - and seals Mogo Mogo's fate. The problem with that situation wasn't that Amber was left hanging. It was that Rob used a pre-existing friendship with Lex to get what he wanted - two people who didn't know each other couldn't have pulled that off.

The odds of that swap? What were the odds of a men vs. women draw in that challenge in Nicaragua? What were the odds that Paschal draws the purple rock? What are the odds that David Tyree catches a ball with one hand and his helmet in Super Bowl XLII? What are the odds that my lottery numbers come up? We can hash out billions of scenarios for what happened if this or that took place. But just because it turned out a certain way doesn't necessarily mean it's rigged. Give the producers a little more credit than that - there may not be laws against rigging these shows, but I guarantee that producer influence is a lot more subtle than a flat-out game rigging.

The producers can do a lot with just about anyone. A crotchety old Navy SEAL. A goat farmer. A guy sitting and sharpening a machete becomes the ultimate underdog. A chunky guy with a crazy beard. A man in a red tank top manages to outsmart six women bent on wiping him out. A gray-haired firefighter dominates wire-to-wire. Four underdogs embarrass a dominant tribe. A guy reneges on a promise made over a car. Another gambles with two immunity idols - and loses. Another takes his own life in the game - metaphorically, of course - by handing over immunity. Someone plays a perfect game. Someone else destroys his tribe's things. Someone wins the game twice. Two people quit. It's not like EPMB doesn't have much to work with. Sure, we see an edited version, but that doesn't mean it's rigged.