The Gemara in Sanhedrin 27b (and Brachos 7a) asks, why does our passuk need to tell us that a son cannot be put to death for his father's sin, when we already were told "Ish b'cheto yumas," a man will die for his own sin? The Gemara answers that our passuk is talking about court testimony- that close relatives cannot testify against each other. But then the Gemara asks, is our inference that a son is not punished for a father's sin correct? But it says ""Pokeid avon avos al banim," that Hashem remembers a father's sin upon his children! The Gemara answers that "Pokeid" is only when the children embrace their father's sin. This is why that passuk says pokeid...l'sone'oi, to those that hate me. Then the Gemara asks, but in Vayikra it says וְכָשְׁלוּ אִישׁ-בְּאָחִיו, that a man will fall because of his brother's sin? The Gemara answers that this is because he failed to protest and prevent the sin; his failure to protest makes him an enabler, and Hashem punishes for aiding and abetting sinful behavior.
Interestingly, the Gemara in Makkos 24a says that Yechezkel "nullified" Moshe Rabbinu's declaration. Moshe said Pokeid avon avos al banim, and Yechezkel said
So whether from the passuk in our parsha, or from Yechezkel's words, the bottom line is that a righteous son does not suffer from the sins of his father.
There are exceptions to this rule, though. The Gemara in Shevuos 39a says that when a person swears falsely, his family and the entire world suffer as a result, and the Gemara in Kiddushin 13a says the same about Arayos.
It's interesting that the expression used for punishment is the same as that used for collecting debts- nifra mimenu. You have to wonder whether the laws of inherited debt can shed any light on this discussion.
The basic rule is that personal debt is not heritable, but land that is encumbered by a lien remains encumbered in the hands of the heir. If one designates movable property as collateral, then even in the hands of the heir the movable property will be subject to that lien. This is the difference between shibud nechasim and shibud haguf. Shibud nechasim, the encumbrance of property, runs with the land. The shibud haguf, the personal liability, is specific to the original party.
(Even though no heir has any legal obligation to pay unsecured debts of the legator, a son has a mitzva of Kibbud Av to pay those debts, in order to remove the stigma of default, of the father's failure to pay his debt.)
Reb Chaim Volozhiner, in his introduction to his Ruach Chaim on Avos and there in perek 5 says that most of the time, what we call "punishment" is not sent by Hashem. Punishment stems from two things: sometimes, it is the natural consequence of the flaw in the soul that was generated by the sin, and sometimes the sin itself causes suffering.
Perhaps we can say that the same applies here. A flaw in a soul might be heritable. The sin itself is not heritable. The flaw in the soul runs with the soul. The sin itself is exclusive to the sinner. Only when the heir reinforces that ancestral sin does it take residence within him- it is machzeres ahl achsanya shelo. The descendants of the sons of Eli suffered from the spiritual flaw engendered by their progenitor, and the consequence of that flaw was a truncated life. Beyond that, an ancestor's sin that is repeated by a descendant becomes especially entrenched in his personna, and thereby causes even more trouble than usual.
Pursuant to a discussion in the comments, there is an interesting discussion in Shulchan Aruch along the same lines.
The Rama in YD 334:6 says that a court may punish a person who has been placed in Nidui/Cherem by expelling his children from school and his wife from the shul until the person accepts the authority of the court. His source is the Nimukei Yosef Bava Kama in Hagozeil Umaachil, 39b in the Rif pages, quoting Rav Paltui Gaon. The Taz brings from the Yam Shel Shlomo that "Chalila to do such a thing! The whole world stands on the Hevel Pihem shel Tinokos shel beis rabban! And what right do we have to expel his wife? If he sinned, what did she do wrong?" The Taz says that the Gaon only was referring to minor children, and the Gaon neer would apply his rule to children above bar mitzva, or a spouse. The Aruch Hashulchan says that the rule of the Gaon applies only where Beis Din sees that the additional pressure on the father will make him bend to the court to protect his family, but not as a punishment on the others per se.
So, the bottom line is this; the consequences of your behavior are not limited to their effect on you. What you do ripples through your descendants in a Lamarckian cascade. Even if you've given up on yourself, remember that your failures hurt your children almost as much as they do you.
I just saw that Reb Meir Simcha in this parsha, in 21:22, talks about the concept of avon avos al banim. It goes without saying that it's kedai to see inside, but here's what I got with a weak OCR, so use your imagination.