When we began this website, many people were drawn here by divrei Torah, but then they were put off by the humor. Since they could only recognize serious divrei Torah when they're bound in a black book, they never came back. Others were drawn by the name Havolim, looking for letizanus, for jokes, and were put off by the serious divrei Torah. What can I do. The Beis Halevi first printed his teshuvos together with his divrei torah on the parsha, but in the later editions he separated them into two volumes, saying that the audience that was interested in the one was totally uninterested in the other. Lehavdil, I've done something similar, and have begun posting the serious Divrei Torah on my other site, Beis Vaad.
But the truth is that there is no contradiction between humor and learning Gemara.
Stimulating the mind through humor is mentioned in the Gemara (Shabbos 30b) where it says that before he would begin the shiur, Rabba would say something that would make the students smile, and then his tone and demeanor would change, the mood in the room would shift from light-hearted to an extremely tense focus, and he would begin the shiur, expecting absolute attention, unforgiving of even the smallest lapse.
A 1976 study by the late professor emeritus at Tel Aviv University, Avner Ziv, (the author of the entry on Humor in the Encyclopedia Judaica) found that those who listened to a comedy album before taking a creativity test scored 20% better than a control group that had not heard the routine.
A 1987 study by Isen, Daubman and Nowicki asked participants to solve a creative logic problem. In the control group, two of fifteen solved the problem. The other group was shown a comedy film before being presented the problem. In the latter group, nine of the fifteen solved the problem.
Remarkably, there are very highly regarded studies that show that laughter and humor increase the immune response significantly- not by some meaningless scale, but by increased lymphocyte blastogenesis and killer-cell activity (a good thing.)
So in that light, when you read the Gemara about Rav's method of teaching, you realize that no, it's not just another quaint habit of those ancients. Chazal's life was the pursuit of wisdom and they lived to discern the truth; they were well aware of the benefit of humor in opening the mind, and they used it as a very serious means to enhance their students' learning.
First we have our famous Gemara, as we brought above, where a humorous word opened their minds.
So a simple reading of the Gemara would be that there are two things that can benefit from בדיחותא; Hashra'as HaShechina and learning. Rashi explained that בדיחותא opens the mind. We have a Yalkut (ילקוט שמעוני מלכים ב׳ רכ״ו באד״ה כי לולי פני) that says exactly like Rashi-
But the Ran (ד״ה לא קשיא) seems to connect those two things, the Hashra'as HaShechina and learning.
And third, you have the Rambam, (at least we had this Rambam before Kapach said it's not in the original, but it's in the Meiri anyway,) that says that when the students are exhausted, you can give them strength with בדיחותא:
כמו שילאה הגוף בעשותו המלאכות הכבדות עד שינוח וינפש, ואז ישוב למזגו השוה - כן צריכה הנפש גם כן להתעסק במנוחת החושים, בעיון לפיתוחים ולענינים הנאים, עד שתסור ממנה הַ ּלֵ אּות. כמו שאמרו. כי הוו חלשי רבנן מגרסיהו, הוו אמרי מלתא דבדיחותא
As to what the בדיחותא consists of, I'm going to leave that to you. Obviously, it is the type of בדיחותא that a Rebbi would tell his talmidim before a shiur, and involves chochma and torah. Any other type, as the Pnei Yehoshua says in his introduction to Kesuvos, is מוקצה מהדעת. And the truth is, it's hard to learn the Gemara any other way, because the Gemara comes to the idea of בדיחותא as a means of attaining שמחה של מצוה. What kind of בדיחותא is שמחה של מצוה? So the Pnei Yehoshua says that it either means a farfetched but seemingly logical dvar torah, or an interesting aggadeta.