Dravyk...short of hiring gentlemen in black and white striped shirts to bonk people in the head with yellow flags when the topic drifts, I think this is going to have to be accepted as the nature of "group conversations." If two people sit down and attempt to talk "about a subject," there's probably a 50% chance they can stay on that subject for any length of time. Add a third person, and it drops to 30%. Add a fourth person...and any chance of it staying to topic flies out the window. (This is from research that I have absolutely no intention of ever doing someday.)
Scientifically, this is called the "Oh...by the way..." Factor.