Knowing Is Not Enough

Today, a senior asked me, “You say you read many papers, but how do you know whether a paper has taken effect after you read it?” I was tongue-tied.

I have run into this kind of problem a lot lately. In fact, I have felt stuck in a kind of stagnation since this summer. I seemed to have done and experienced many things, but in the end I had only “known” them. This discouraged me a lot: my meta-abilities and my understanding had barely improved, despite my apparent diligence.

That is a serious problem. After all, I devoted my time, my attention and my energy. Ideally, whether an experiment or a study succeeds or fails, I should come away with some understanding from it.

There is a common temptation. New techniques, trends and other fancy things are attractive, and there is some bar even to “knowing” them. So we spend a certain amount of time and energy getting to know them, and then we are easily satisfied. That is terrible.

Reading papers has a similar trap: we are awed by novel ideas and forget to think critically. The figures and graphs may be impressive, but a series of questions should still be asked:

What is the baseline or benchmark? The authors claim strong results, but strong compared with what? If I have read the related papers, I should know the common baselines. Is a common baseline missing? Does the benchmark truly match the problem?
Why this method? If they use PPO, was that forced by a constraint (e.g. a continuous action space rules out plain Q-learning), or were there real alternatives? Did they justify the choice? If they say an abandoned method gave unstable training losses, did they actually find and explain the reason?
Where does the gain come from? How do the authors explain their edge over other methods? The ablation experiments may prove something. But does the reasoning part, such as arguments about the manifold or about multiple modes, really convince me?
What obstacles did I run into while reading? If I ran into some, I deal with them. But what if I have no questions at all? There are only two possibilities: either I truly understand it, or I am trusting my instinct, which says everything is fine. Then I always need to ask one more question: is my instinct trustworthy? Many pitfalls hide beneath instinct.
What am I reading this paper for? Is it to push back the frontier of my knowledge, or just to learn a technique more deeply?

This approach is not only for reading papers; I believe I can apply it to many other things. As a freshman, I might have been proud of “having dabbled in many areas.” Now that I am a sophomore, things should change. Harder, deeper and more critical thinking should become part of my daily study.

Besides, I need to output. Output forces me to understand and then regenerate. When I explain something again, I automatically rethink the problem. Chats, seminars, talks: I will try them all.

So here is the answer I did not have that day: a paper has taken effect if I can rebuild its argument without looking, say where it would fail, and use it in my own work. If all I can say is “I’ve read it,” then I only know it.