This study demonstrates the potential of large language models (LLMs) to analyze environmental court rulings from India. Using a novel dataset of 12,615 environmental court orders spanning three decades, we evaluate the performance of two LLMs - GPT-4 API and Claude 3.5 Sonnet - in coding and interpreting judicial decisions. The LLMs are tasked with identifying pro-environmental rulings and extracting key case attributes, with their performance benchmarked against human coders who analyzed 1,910 rulings. Both models achieve approximately 70% accuracy compared to human coding, with the GPT-4 API showing slightly better performance in various sub-samples. These findings suggest promising applications for AI to improve access to and analysis of legal data, particularly in jurisdictions where administrative records lack standardization.
Environmental courts issue thousands of complex rulings, collectively shaping policy and regulatory
frameworks across jurisdictions [
The analysis of environmental rulings faces some fundamental limitations. Manual review of
thousands of unstructured legal documents is cumbersome and requires specialized expertise [
This study examines whether LLMs perform as well as human experts in categorizing rulings as environmental and assessing whether judicial decisions produce positive environmental results in the context of India. Assessment of the feasibility of using LLMs to complete these tasks is important to determine whether they can be used to expand the quantitative analysis of environmental jurisprudence to inform policy and improve access to environmental justice.
We examine a novel data set of 12,615 environmental court rulings from India spanning three decades, evaluating two state-of-the-art LLMs (GPT-4 and Claude 3.5 Sonnet) against human expert coding of 1,910 rulings. Our central task, determining whether a judicial decision is "pro-environment", is a complex judgment requiring understanding of legal reasoning, environmental science, and implementation realities.
Our work makes three primary contributions. First, we develop and validate a methodology for
AI-assisted environmental law analysis that achieves approximately 70% agreement with human experts,
which is comparable to studies of the US Supreme Court[
Our analysis begins with India’s three foundational environmental acts: the Water (Prevention and Control of Pollution) Act 1974, the Air (Prevention and Control of Pollution) Act 1981, and the Environment (Protection) Act 1986.1 We conducted a comprehensive search of the Indian Kanoon.org database, identifying 2,996 judicial rulings that explicitly cited at least one of these acts2 To ensure complete coverage, we systematically expanded our data set by analyzing all additional legislative acts cited within this initial corpus, identifying 23 additional environmental statutes frequently referenced in environmental litigation. The most cited acts in our data are presented in Table 1. Number of rulings citing Act 3499 2566 2219 1667 1547 1374 1304 1150 1059 892 667 495
Our final dataset encompasses all judicial rulings from 1974 onward citing any identified
environmental statute, resulting in 12,615 court rulings spanning through 2024. The raw data consisted of
1This identification was based on extensive desk research, personal interviews with environmental law experts, and consultation
of leading environmental law textbooks ([
Table 2 presents the distribution of rulings at diferent levels of the Indian judiciary system. Most of the rulings (69%) originated in the High Courts, which serve as the primary forums for environmental litigation in the Indian federal system. The National Green Tribunal (NGT), established in 2010 as a specialized environmental court, represents 23.1% of rulings despite its relatively recent creation. The Supreme Court of India, as the apex court, contributed 3.29% of the rulings, usually involving appeals or matters of national importance.
Our data set covers all Indian states and union territories, with rulings concentrated in industrialized regions and major metropolitan areas. The rulings cover the full spectrum of environmental issues, from industrial pollution and waste management to forest conservation and wildlife protection. Table 1 presents the most frequently cited environmental statutes in our dataset, providing insight into the main areas of environmental litigation.
However, despite its size and geographic coverage, our data set has several limitations. First, coverage of lower court decisions may be incomplete, particularly for earlier periods. Second, we only include rulings explicitly citing identified environmental statutes, potentially missing those that rely on other legal provisions. Third, the quality of case documentation has improved with time due to better digital practices. Finally, our data set reflects only rulings that reach formal adjudication, excluding disputes resolved through alternative mechanisms.
Despite these limitations, our dataset represents the most comprehensive compilation of Indian environmental court rulings available for research, providing unprecedented scope for analyzing judicial approaches to environmental protection over decades.
Our methodology involves four distinct phases: constructing the complete data set, establishing humancoding benchmarks, implementing the Large Language Model (LLM) analysis, and then analyzing model performance.
Our data set contains 12,615 environmental court rulings spanning 1974-2024, representing the full
universe of litigation citing our identified statutes in the courts for which IndianKanoon has data. For
computational eficiency and validation, we selected a subset of 1,910 rulings directly citing the Air
(Prevention and Control of Pollution) Act 1981, chosen because air pollution rulings represent a
significant category of environmental litigation and this act is particularly salient in Indian environmental
jurisprudence [
During summer 2021, we recruited 14 law students from the National Law School of India in Bangalore to manually analyze the 1,910-case subset. All coders underwent comprehensive training through a detailed video guide and codebook. A senior research assistant supervised the entire process of allocating rulings to students, collecting responses, and monitoring the quality of the coding.
Each case was assigned to at least one coder, with 746 rulings (39%) receiving independent review by two coders to assess inter-rater reliability. When coders disagreed on the primary classification (pro-environment vs. not pro-environment), a third coder reviewed the case to determine the final classification. This occurred in only three rulings, indicating high inter-rater agreement.
The central question posed to human coders was: "Is this judgment likely to have a positive impact on the environment (or not)?" To answer the question, we provided additional guidance in the training manual.3 Reflecting a conservative approach that prioritizes direct, observable environmental interventions over potential indirect efects, this guidance directed coders to classify dismissed cases as having "no environmental impact."
Next we deployed two state-of-the-art Large Language Models (LLM) for further analysis: GPT-4 (via OpenAI API) and Claude 3.5 Sonnet (via Anthropic API). Implementation involved two distinct prompts, reflecting an evolution in our methodological approach.
Phase 1: Replication prompt Initially, we attempted to replicate the human coding process using the identical prompt given to human coders: Is this judgment likely to have a positive impact on the environment (or not)?" Phase 2: Improved prompt After analyzing preliminary results and recognizing limitations in the original prompt, we developed an improved and more specific prompt: Extract the result of the order. Respond 1 if the case likely has a near-term or immediate positive environmental impact that would reduce air pollution, otherwise respond 0 and do not write anything else.
We modified the prompt for several methodological reasons. The first rationale was specificity. The improved prompt focuses on "near-term or immediate" impacts rather than general environmental efects, providing clearer evaluation criteria. The second was measurability. By specifying “reduce air pollution"," the prompt targets a concrete, observable outcome rather than an abstract environmental benefit. Our third rationale was bias reduction. The revised prompt eliminates explicit instructions on dismissed cases, allowing the LLM to make more nuanced interpretations of the outcomes of the case. Finally, we note that the improved prompt provides more objective criteria, reducing subjective interpretation variability.
Both GPT-4 and Claude 3.5 Sonnet processed the same 1,910 rulings using both prompt versions with systematic quality controls including standardized API calls, error handling, and response validation. Due to minor technical issues (API timeouts, formatting errors), our final analytical sample contains 1,906 rulings with complete data from all three coding approaches, representing 99.8% of our intended sample. 3The training manual instructed coders: "If you think that the judgment is likely to have a positive impact, select ‘Yes’ from the drop-down menu. For example, if the court orders that a polluting factory be shut down or imposes fines on the polluter, such a judgment is likely to have a positive impact on the environment. If, on the other hand, you believe that the judgment will have no impact or a negative impact on the environment, select ‘No’ from the dropdown menu. This may include judgments where the petition is dismissed without passing any further orders. Judgments, where the case is sent back to a lower court to be heard afresh without passing any orders on the merits of the case, will also fall into this category."
We evaluated the models using standard classification metrics (accuracy, precision, recall, F1 score, and Krippendorf’s alpha) in multiple dimensions. We also performed robustness analysis on multiple subsamples.
When we compare the two prompts across LLM models, we see that GPT-4 classified fewer rulings as environmentally favorable when using the human prompt (35.4%) compared to the improved prompt (48.6%). However, Claude shows minimal sensitivity to prompt variation, classifying a similar proportion of rulings as green with both the improved prompt (42.9%) and the human prompt (43.1%). In the case of the GPT-4 model, this pattern initially appears counterintuitive, given that the improved prompt specifically asked whether a ruling would have “near-term or immediate positive environmental impact that would reduce air pollution" - a more restrictive criterion than the broader question of whether it would “have a positive impact on the environment (or not)".
Upon examining the rulings driving this discrepancy, we found that the diference is explained by procedural rulings with ambiguous outcomes. For example, in rulings involving multiple polluter defendants, where only some parties were ordered to implement abatement measures while others were exempted, determining the overall environmental impact proved challenging under either prompt formulation. We note that such procedural rulings are more likely to be interim court orders than final judgments, accounting for 31% of our sample. Our results remain robust when excluding this entire category of rulings.
In general, these findings highlight that prompt engineering efects vary significantly between diferent LLM architectures, suggesting that optimal prompting strategies may need to be model-specific rather than universally applicable.
Table 5 presents detailed confusion matrices showing classification agreement and disagreement patterns between human coders and each LLM model. As noted earlier, both LLM models systematically identify more rulings as environmentally favorable compared to human coders. GPT-4 with the improved prompt shows 541 false positives (rulings humans coded as "not green" but GPT-4 coded as "green") versus only 95 false negatives. This pattern persists in both models and prompts, suggesting fundamental diferences in how AI systems and human experts evaluate the environmental impact.
Analysis of 25 randomly selected disagreement rulings reveals that LLMs and humans use fundamentally diferent evaluation frameworks. In all examined rulings, humans classified rulings as "not green" while LLMs classified them as "green." Human coders appear to interpret the rulings pessimistically based on their experience with India’s environmental policy implementation challenges, while LLMs
Total 1,081 801 1,882 Total 1,079 817 1,896 Notes: Rows represent human classifications, columns represent LLM classifications. Diagonal elements show agreement, of-diagonal elements show disagreement. displayed systematic optimism about formal legal outcomes, perhaps due to a lack of contextual understanding of enforcement realities. For example, when a court prevented illegal thresher machine use (Kanoon ID 20982084), human coders anticipated continued unauthorized use despite the ruling, while GPT-4 focused on the formal legal barrier established by the court decision.
Table 6 examines the performance of the model in various sub-samples. All analyses use the human prompt for consistency. Here we see that GPT-4 consistently outperforms Claude in all sub-samples, with accuracy ranging from 70.43% to 83.23%. Both models perform best in rulings that do not involve a Pollution Control Board (PCB) action, suggesting that procedural enforcement rulings present particular challenges for AI interpretation. We also note that the LLM models perform less well in Supreme Court and NGT rulings, possibly because these pertain to more complex cases.
When we applied GPT-4 to the complete data set of 12,615 rulings (using the improved prompt), it classified 35.0% of the rulings as environmentally favorable. This estimate aligns closely with the 35.4% rate in our validation subset, suggesting consistency in AI classification patterns throughout the entire data set.
Our analysis reveals both promising opportunities and important limitations for AI-assisted
environmental law analysis [
The most notable pattern in our results is that LLMs consistently identified more rulings as environmentally favorable compared to human experts. GPT-4 classified 35.4% of the rulings as "proenvironment" versus 25.2% by human coders. However, our findings reveal systematic diferences in the way AI and human experts assess environmental impact. Although LLMs excel at identifying formal legal outcomes, human experts incorporate a contextual understanding of enforcement challenges that LLMs lack. The systematic patterns of disagreement, rather than random errors, suggest that these groups access fundamentally diferent types of information. This reveals that human judgment remains essential for evaluating implementation prospects, particularly in environmental law, where the gaps between judicial declarations and enforcement significantly afect real-world outcomes.
These results have several implications for legal research and policy analysis. AI tools ofer unprecedented eficiency for systematic analysis of large legal datasets, enabling researchers to identify patterns and track judicial trends at previously impossible scales. For practitioners, AI could streamline legal research by helping to identify relevant precedents and litigation strategies. However, efective AI-assisted legal analysis requires acknowledging these limitations and developing hybrid approaches that combine computational eficiency with human expertise.
These insights inform future studies that seek to improve methodology and policy. Our approach demonstrates how human expert validation can be systematically integrated into AI-assisted legal research. Our documented disagreement patterns could guide the development of more sophisticated legal AI systems that incorporate enforcement probability models alongside formal legal analysis.
Finally, this research opens a new path for the analysis of the broad impacts of environmental
policy in India. Our data set of 12,165 rulings provides a foundation for examining legal arguments
and connecting judicial decisions to measurable environmental outcomes, as has been extensively
conducted elsewhere ([
This study demonstrates the potential of AI to improve the analysis of environmental court rulings, achieving 73% agreement with human coders on our comprehensive dataset of 12,615 Indian environmental rulings. Although AI efectively catalogs formal legal interventions and tracks doctrinal developments, human judgment remains essential for evaluating implementation prospects and policy efectiveness. These findings suggest that hybrid approaches combining computational eficiency with human expertise can significantly improve the scalability of legal research, particularly where administrative data are not standardized, opening new avenues for revolutionizing the analysis of large-scale legal datasets across jurisdictions and policy domains.
During the preparation of this work, the authors used ChatGPT-4 and Claude Sonnet 4 for grammar and spelling checks, as well as paraphrasing and rewording assistance. After using these services, the authors reviewed and edited the content as needed and assume full responsibility for the content of the publication. and database developments for European works Councils and law. In Field Guide to Researching Employment and Industrial Relations (pp. 141-160). Edward Elgar Publishing. [19] Bhupatiraju, S., Chen, D.L., Joshi, S., Neis, P. and Singh, S.: Environmental Litigation as Scrutiny: A Four Decade Analysis of Justice, Firms, and Pollution in India (October 31, 2024). Available at SSRN. Paper 5013333.