No human, no Malach, can understand Hashem's judgment. Avraham Avinu, Moshe Rabbeinu, the Malachei Hashareis, nobody can even begin to understand why and how. Even so, Chazal and our baalei hashkafa discuss such things. What follows is only in the spirit of Limud Hatorah, and is not an Icarus-like quest for the resolution of unanswerable questions (Chagiga 14b).
This first part is from Harav Meir Bergman, Rav Shach's son in law, in his sefer on Chumash, Shaarei Ora.
The discussion involves psukim, 32:3-4:
The Talmidei Rabbeinu Yona there mentions that the Rif also redacted it out of his iteration of the Gemara. Talmidei Rabbeinu Yona are not at all happy about this. They end by saying אע"פ שיש מיתה בלא חטא... כל דרכיו משפט והוא יודע למה עושה כן, even though death can be without mishpat, He knows why He does so.
The Chidushei Anshei Shem there discusses this, and says other surprising things, not directly in line with this discussion but worth reading.
Rav Bergman asks, how did these rishonim- the Bahag, the Rambam, and the Rif- understand the beginning of the bracha, which they leave intact- dayan emes, Keil emes, and so forth? And what do they do with our pesukim, that talk of ein avel and yashrus?
He answers that we have personal identities and a national identity. Heavenly judgment can take national identity as the predominant field of Din and Schar V'Onesh, even when it results in a judgment incompatible with certain individuals in the klal. Our dominant identity is as parts of an organism rather than as individuals; just as we judge a person as a whole, and not his individual parts, Hashem might judge Klal Yisrael as a whole, and not its individual parts.
Rav Bergman reads these two approaches to Din into the various interpretations of כל באי עולם עוברין לפניו כבני מרון in the Gemara in Rosh Hashanna 18a. According to one pshat, this means judgment of each individual; according to another, it means judgment of the nation as a whole.
He ends by quoting the Rambam in Pirush Hamishnayos there in Rosh Hashanna who ends the Mishna with the words "this statement is self explanatory, but its hidden aspect is extremely difficult." Rav Bergman says that the Rambam is alluding to the difficulty of correlating the judgment of individuals qua individual with judgment of Klal Yisrael as a whole.
The gist of this idea is that an individual might experience things that are not the result of his own behavior. If he were ajudicated on his own merits, his judgment would be X. But since he is a member of the klal, and the judgment on the klal was Y, he will (might? always? sometimes?) share in that judgment. So from the individual perspective, the decree is not Mishpat. But from the Klal perspective, it is perfectly just.
Rav Bergman's dvar Torah ends here. Now lets think about how this comports with Reb Yisrael Salanter's words about Din Shamayim.
Reb Yisrael Salanter is widely quoted as having said that אין עול teaches an essential difference between Hashem's judgment and an earthly court's judgment. If a man is guilty and deserves punishment, the consequences that his punishment will have on others are legally irrelevant. That executing him or incarcerating him will result in grief or financial distress to innocent people matters not at all to an earthly court. But Hashem takes all of this into consideration. With this, the Sefer Chakal Tapuchin explains the Gemara in Shabbos 106a that if a member of a group dies, all the members of the group should worry; he explains that if their friend died, then it must be that the Beis Din shel Maalah decided that all the members of the group deserved to experience the grief of losing a friend, which shows that they need to do teshuva. Reb Chaim Shmuelevitz in his Sichos here says that Reb Yisrael used this approach to explain the passuk in Tehillim (19:10) "משפטי ה' אמת צדקו יחדיו" Hashem's judgment is true, righteous together: this means that Hashem only punishes an individual when all the people affected by the repercussions of the judgment deserve those consequences. In fact, Reb Elia Lopian used to say that this is a good reason to develop a group of friends that love you and sympathize with you. Even if you deserve punishment, you might be saved because your good friends don't deserve the emotional distress that would result from seeing you suffer. (see, e.g., Mishulchan Gavoah and Lekach Tov on this passuk in Haazinu.)
It appears that Rav Bergman's and Reb Yisrael's ideas are fundamentally incompatible. Do you agree that they are inconsistent? Yes? I initially thought so too. You, as I, have just been taught a lesson: Instead of jumping to conclusions that there is a kashe, instead we should pay more careful attention. There is an enormous difference between the two, and there is no contradiction at all.
Reb Yisrael Salanter said that Reuven cannot suffer an undeserved repercussion of Shimon's punishment. Rav Bergman is talking about Reuven suffering as an undeserved consequence of a Tzibbur's sin. Reuven is by no means a part of Shimon. Reuven is, however, a part of the Tzibbur. Reuven's personal identity is not necessarily as important a consideration as his "part of tzibbur" identity. In fact, Reb Yisrael's pshat is based on the word יחדיו. He interpreted that to mean that Shimon cannot be punished when the punishment will cause undeserved suffering to Reuven. But the same word יחדיו tells us that we are not only individuals, we are also parts of a tzibur. When a member of a tzibur sins, the tzibur as a whole might be punished, including innocent Shimon.
There is certainly no contradiction between Rav Bergman and Reb Yisrael Salanter. But there is no evidence that Reb Yisrael would agree with Rav Bergman. Having said this, there is one very important point that everyone would agree to. There is one time when a person might be punished because his generation sinned, even when he himself did not share in their sin, and that is when he could have protested and he did not. The concept embodied in Arvus, in Lo Saamod ahl dahm rei'echa, in Arur asher lo Yakim, creates an ethical and halachic parity between Noninterference and Aiding and Abetting. Shabbos 54b-55a: