Nihar B Shah is Associate Professor at Carnegie Mellon University, USA, where he leads a research group exploring how human-AI collaboration can lead to more trustworthy research publications and fairer evaluation systems. He completed his ME from the Department of Electrical Communication Engineering (ECE) at IISc in 2010, and also received the Young Alumnus Medal in 2024. Nihar believes that AI can improve the way we do science, provided its vulnerabilities are strengthened. His work has had real-world impact, with his algorithms adopted by academic publishers and grant-funding agencies for peer review, college admissions, and competitions. He is now extending his research interests to new fields, such as large-scale telescope programmes and biomedical sciences. Nihar speaks to CONNECT about tackling scientific fraud, building more reliable AI systems, and the Institute’s influence on shaping his research and teaching.
Can you tell us about your childhood?
I grew up in Vadodara, Gujarat. I don’t have siblings, but there were many friends in my locality. We would play a lot, flying kites on 14 January during the festival of Uttarayan. Luckily, as I’m on sabbatical this semester, I got to go back again during the festival this January. It was a lot of fun. I taught my kids how to fly and fight with kites.
How did you get into research?
I liked programming, so I wanted to go for a software job; at that time, the software industry was booming. I joined Adobe in Bengaluru. The learning curve was steep initially, but soon became saturated, and I realised that it was not my cup of tea.
I understood what research is only when I joined IISc for a Master’s degree. The Institute provided an incredibly supportive atmosphere. I was so fortunate to have P Vijay Kumar as my thesis advisor. He taught me broader aspects of research that I continue to use: how to select meaningful problems, how to approach a problem from various perspectives, how to effectively communicate your research, and how to identify opportunities for real-world impact. He also demonstrated, by example, how to mentor students effectively, and have a fun and collegial atmosphere among research groups.
You thereafter decided to venture into improvising processes in the “science of science”. How did that happen?
Near the end of my PhD at UC Berkeley, I was unhappy with the way the conference peer review process worked. So, I approached conference organisers to collaborate and analyse the review data, and make recommendations for improvement. One of the top venues adopted the recommendations, and since then, many others also have.
Many of our scientific processes involve humans evaluating items. For example, peer review involves writing a paper, submitting it to a conference or a journal, and having other researchers evaluate it. This part of the science of science – human evaluations – needs better ways to process complex human data, and also better incentives. If I’m a reviewer, what motivates me to do a good job? There are also different types of fraud that occur, so we need to design incentives to prevent or reduce those.
Is your focus now on finding fraud in scientific publishing?
Fraud is one key focus. A major problem in scientific paper publications is fraud. We work with various top venues to identify and mitigate this. We have seen extremely creative kinds of fraud – so creative that if the researcher had used this intelligence to do their research, they would have been better off (laughs). Then, there are also problems of bias [of the reviewer], arbitrariness and subjectivity. With the advent of Gen AI, there are now lots of opportunities, but also challenges. Gen AI can improve efficiency, but it has some vulnerabilities and failure modes. A malicious person can game these AI systems in a way they would not be able to game humans.
‘A major problem in scientific paper publications is fraud’
How exactly is AI involved in the paper review process?
It is involved behind the scenes. For example, when people submit papers to a conference, algorithms are used to assign topically relevant reviewers to each paper. Algorithms also check for problems of arbitrariness. If there are significant problems, it’ll either flag or automatically correct them.
So, humans and AI work together through the entire review process.
That’s correct. Currently, there is some exploration on using Gen AI itself to make the full decision. But our recommendation is to do a hybrid approach, because the failure modes of Gen AI are very different as compared to the failure modes of a human.
Do you think there’s still a lot of scepticism or resistance toward using AI?
Yes, and that’s fair enough. Both the excitement and the concern are fair and valid. There are places where AI has been deployed for many years, such as in computer science. Top publication venues have been using machine learning algorithms to assign reviewers to papers for at least a decade. They receive about 10,000 to 20,000 papers; it’s too large a task to do manually. But there are cases where people can rightly be concerned about AI. It has its vulnerabilities.
Can you share an example?
We conducted an experiment a few months ago to test some fully autonomous AI reviewers. We took three different papers from the same field, took different parts of them, and combined them into a completely nonsensical paper, and then gave it to the AI to review. None of the AI reviewers recognised that it was a problem. This was a clear failure mode.
Another example of an AI failure mode is that it is prone to adversarial attacks from what’s called a collusion ring – researchers who make a deal to try to get assigned each other’s papers and give good reviews. They also do this to win awards and other recognitions.
How does that work, exactly?
As AI is being used to assign reviewers to papers, it can be tricked. The AI looks at the text of the submitted paper, and then looks at the text of a list of potential reviewers’ previous papers and assesses how similar they are and matches based on that.
If an author and a reviewer are colluding, the author can slightly change the paper they are submitting so that the AI thinks it’s very similar to the reviewer’s past work. At the same time, the reviewer can selectively show the AI only a few of their papers that seem more aligned with the submitted paper. This way, the reviewer looks like a perfect match.
In our work, we identify such vulnerabilities in AI and devise ways to prevent them. We have also instituted whistleblower reporting mechanisms, where people can report incidents of fraud.
‘In our work, we identify vulnerabilities in AI and devise ways to prevent them’
Was there any incident of fraud that really shocked you?
A person wanted their paper to get published in a top conference. But they knew that their paper was terrible. So they went to arXiv [a free, open-access archive of papers yet to be peer-reviewed], where you can download the LaTeX source of papers. They downloaded the LaTeX source of a good paper, made sufficient changes to avoid the plagiarism detectors, and then submitted it in their own name.
When the paper got accepted at the conference, the person switched out the arXiv paper and replaced the PDF with their own paper. But they were not able to change the title on the conference management system because that needed the approval of the conference programme chairs.
This person was brave enough to actually write an email to the conference programme chair asking to change their paper’s title. But since the new title had no relation to the old one, they were caught.
Wow, this is crazy …
This is nothing. There is this entire market where actual companies generate fake papers and sell them to researchers. The researchers can buy a paper and publish it in their own name.
Just to get a higher number of publications?
Yes. In fact, there are various journals which allow you to change the author list, during and even after the review process. Some of these fake paper companies will even submit a paper to a journal with some random author name, get it reviewed, and once it is accepted, then they will sell it to researchers, saying: “Do you want a paper already accepted to the Journal of XYZ?”
How common is this fraud?
It’s hard to give a number because we don’t know what we don’t know. Even if the numbers are small, the reason it is a big problem is that science builds on top of prior science. So, if there’s something fraudulent and then 10 genuine papers build on it, all of them go down the drain.
It looks like you’re pleased about shifting from working in software to a research career.
Yes, I am very, very happy with my current job. I think academia allows you to pursue things that you think are important in the long run. There’s a lot of room for creativity. You choose a problem, and you can throw whatever tools at solving it.
‘Academia allows you to pursue things that you think are important in the long run’
What is an average day in your life like? You teach classes as well?
It is a lot of multitasking. There will be a one-hour meeting with student one, and then teaching a class, then meeting student two, then student three – who are all working on very different problems – and then going to a committee meeting, and so on. It can be challenging, but then it’s all very rewarding.
Being a teacher now, how do you reflect on your own student days and your relationship with teachers at IISc?
I remember IISc very well, and there’s so much that I learned from IISc that I now use in my own teaching. Every single class I took at IISc was amazing. I took a class in my first semester with Neelesh Mehta, who’s currently the department head for ECE. He was teaching a class called Digital called Digital Communications. The midterm exam he set was wonderful. When I came out of this midterm, the thought I had in my mind was: “Wow. I learned so much when writing the exam.”
He had created a very nice exam where you would use the concepts that you had learned in class and derive new concepts. I even remember one example. There’s something called the Box-Muller transform. In the exam, he set us a task to mathematically prove it. He broke down the proof into smaller components. Each of them could be proved by the concepts that we had learnt in class. And eventually we end up with this amazing result, which is very useful, and you yourself derive it in the exam. It’s so amazing, and I still use it [this technique]. Every exam that I create, for every question, I explicitly write down a learning objective and create an exam where the students learn something new that can be useful.
That sounds like a very different kind of exam than what I’m used to!
Yes. I also enjoyed classes with Sunil Chandran in the Department of Computer Science and Automation (CSA). I remember once he was teaching graph theory. He would never prove anything by himself on the board. He would break down the proof into smaller steps and then ask the students to guess how to do each step, and then all the students would discuss, debate, and throw out ideas, and then he would compile all the ideas and we would do it together. Again, I try to do this in my classes.
What are some of your other favourite memories from your time at IISc?
I have lots of great memories. My hostel was in the PD block. It is far away, and it is kind of secluded from campus. The playground was right next door, and we used to play a lot of football. I remember the food was so good at A Mess at that time. We practised and performed a drama called ‘Mera Beta Aayega’ for our seniors at their farewell party. I was of the Kung Fu classes during my second year. I also remember this one game that I had created where I had photoshopped pictures of various professors in all kinds of situations, and then the professors had to guess which professor was in the image. The professors were friendly, approachable and super cool people.
Going back to your research, what are you excited to explore moving forward?
With Gen AI progressing, there are more opportunities and challenges. There are a few AI scientist systems now, which will come up with a question or hypothesis, collect data, do the analysis and then write a paper for you. So, how do we make use of these opportunities to improve science, but also understand and guard against the numerous problems that could arise?
The second thing is that we’ve started recent collaborations with funding agencies, and we’re working with them to see how we can make better funding decisions.
The third interesting project we’re working on is with astronomy agencies, helping them make decisions about how to allocate telescope time. For example, someone might be studying a celestial phenomenon that’s going to happen only on 21 April; or, say, since the European Space Agency funded 20% of the telescope, they want 20% of the time on the telescope for European researchers; or some researchers can’t work when there are clouds in the sky. So, we’re collaborating with these agencies on how to allocate telescope time based on all of these diverse considerations.
Since I’m now on sabbatical, I’m also using this opportunity to work with a lot of other domains, like biomedicine and life sciences, on their science of science related problems.
What would you advise upcoming research students on how creatively they can use AI in their research?
If you’re already in a specific field and you’re excited about certain problems, then AI can help significantly improve your efficiency. It can help in writing code, collecting data, and even writing papers. That said, students should be aware of the various failure modes of AI. So, I encourage students to take a software engineering approach – use Gen AI as a developer, and the students can be the testers. They can do it in 5% of the time that it would have taken to write the whole code. I strongly encourage all my students to use AI a lot.
(Edited by Sandeep Menon, Ranjini Raghunath)
