Evaluating Long-Form Text
How reliable are the metrics and model judges we use to score generated text? I study self-inconsistency in LLM-as-a-judge frameworks and what it means for benchmarks built on top of them.
PhD Candidate · Computer Science
Rajarshi Haldar
University of Illinois Urbana-Champaign
01 About
I am a PhD candidate in Computer Science at the University of Illinois Urbana-Champaign, advised by Prof. Julia Hockenmaier in the Hockenmaier Lab. My research sits at the intersection of natural language processing and software — I study how language models understand and describe code, and how we should (and should not) measure the quality of the long-form text they generate.
Most recently I have been working on the reliability of LLM-as-a-judge evaluation, showing how much of a model's verdict is signal and how much is noise it invents on rerun. Earlier work spans semantic code search, code summarization, and event extraction — some of it done during research internships at IBM Research, where it also became three granted US patents.
Before Illinois, I earned my B.Tech. in Computer Science and Engineering from the Indian Institute of Technology Kharagpur.
02 Research Interests
Code Intelligence
Semantic code search, code summarization, and matching natural language to programs — including what large language models actually capture when they explain a function.
Structure & Knowledge in NLP
Event extraction, schema induction, and knowledge graphs — recovering the structure implicit in unstructured text so it can be searched, compared, and reasoned over.
03 Publications
Full and continuously updated lists are on Google Scholar, DBLP, and the ACL Anthology.
Findings of EMNLP
2025
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Findings of the Association for Computational Linguistics: EMNLP 2025
LREC-COLING
2024
Analyzing the Performance of Large Language Models on Code Summarization
Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)
ACL
2020
A Multi-Perspective Architecture for Semantic Code Search
58th Annual Meeting of the Association for Computational Linguistics (ACL)
IEEE TKDE
2020
Deep Learning Driven Venue Recommender for Event-Based Social Networks
IEEE Transactions on Knowledge and Data Engineering, 32(11)
SIGCSE
2020
A Validated Scoring Rubric for Explain-in-Plain-English Questions
51st ACM Technical Symposium on Computer Science Education (SIGCSE)
IWCMC
2019
Cluster Aware Mobility Encounter Dataset Enlargement
15th International Wireless Communications & Mobile Computing Conference (IWCMC)
NAACL-HLT Demo
2018
CL Scholar: The ACL Anthology Knowledge Graph Miner
2018 Conference of the North American Chapter of the ACL: Demonstrations (NAACL-HLT)
04 Patents
Granted US patents, all assigned to International Business Machines Corporation.
-
US 12,001,896 B2 Granted Jun 2024
Unsupervised Event Extraction
-
US 11,615,152 B2 Granted Mar 2023
Graph-Based Event Schema Induction for Information Retrieval
-
US 11,132,512 B2 Granted Sep 2021
Multi-Perspective, Multi-Task Neural Network Model for Matching Text to Program Code
05 Experience & Education
-
Aug 2018 — Present
PhD Candidate & Graduate Research Assistant
University of Illinois Urbana-Champaign · Urbana, IL
Advised by Prof. Julia Hockenmaier
- Quantified how much of an LLM-as-a-judge benchmark ranking is evaluator noise rather than model quality (EMNLP 2025).
- Characterized where large language models succeed and fail at code summarization (LREC-COLING 2024).
- Introduced a multi-perspective neural architecture for semantic code search (ACL 2020; granted US patent).
- Built the dataset, training, and evaluation pipelines in PyTorch and Hugging Face Transformers.
- Designed a system to auto-grade code-reading assignments, collaborating with four researchers to devise a validated scoring rubric and to produce a dataset of code snippets, student responses, and quality scores (SIGCSE 2020).
-
Summer 2022
Digital Intern
Schlumberger · Menlo Park, CA
- Built a knowledge-graph embedding model that learns from limited labeled data via labeling functions and Snorkel AI.
- Shipped a query-answering model on those embeddings that replaced the team's system at lower latency and overhead.
-
Summers 2019 & 2020
Graduate Research Intern
IBM Research · Yorktown Heights, NY
- Led an automated tech-support agent that resolves IT queries by extracting events from unstructured incident text.
- Designed an unsupervised event-representation model and graph-based schema induction — both granted as US patents.
- Derived a novel event schema for tech support by analyzing an IT corpus with statistical modeling, then extracted entities, their states, and their relationships via event extraction.
-
Jul 2014 — May 2018
B.Tech. in Computer Science and Engineering
Indian Institute of Technology Kharagpur · Kharagpur, India
- Undergraduate research on knowledge graphs over the ACL Anthology (NAACL-HLT 2018 demo) and on venue recommendation for event-based social networks (IEEE TKDE).
06 Teaching
-
2022 — 2025
Teaching Assistant, CS 447 — Natural Language Processing
University of Illinois Urbana-Champaign · Urbana, IL
- Designed a Google Colab coding assignment on machine translation and wrote its auto-grader on Gradescope.
- Held office hours twice a week and answered student questions on Campuswire.
-
Summers 2024 & 2025
Teaching Assistant, CS 416 — Data Visualization
University of Illinois Urbana-Champaign · Urbana, IL
- Graded a midterm and a final project in which students built dashboards in Tableau and D3.js.
- Held weekly office hours and answered student questions on Campuswire.
Get in touch
Happy to talk about LLM evaluation, code intelligence, or anything adjacent.
[email protected]