Update demo and README

README.md

@@ -13,11 +13,10 @@ short_description: EMNLP 2024
 
 # AgentReview
 
-Official implementation for the 🔗[EMNLP 2024](https://2024.emnlp.org/) (main) paper: [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://arxiv.org/abs/2406.12708)
+Official implementation for the 🔗[EMNLP 2024](https://2024.emnlp.org/) main track (Oral) paper -- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://arxiv.org/abs/2406.12708)
 
 * 🌐 Website: [https://agentreview.github.io/](https://agentreview.github.io/)
 * 📄 Paper: [https://arxiv.org/abs/2406.12708](https://arxiv.org/abs/2406.12708)
-* **🚀 Note: This repository is under construction development. Please stay tuned!!**
 
 
 
@@ -34,49 +33,79 @@ Official implementation for the 🔗[EMNLP 2024](https://2024.emnlp.org/) (main)
 
 ---
 
-### Introduction
+## Introduction
 
-Agent4Review is a simulation framework for systematic analysis of the peer review process based on large language models (LLMs). This framework aims to understand decision-making patterns, reviewer behavior, and the dynamics of paper acceptance and rejection.
+AgentReview is a pioneering large language model (LLM)-based framework for simulating peer review processes, developed to analyze and address the complex, multivariate factors influencing review outcomes. Unlike traditional statistical methods, AgentReview captures latent variables while respecting the privacy of sensitive peer review data. 
 
 ### Academic Abstract
 
-Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms
+Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation 
+framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms.
 
 
 ![Review Stage Design](static/img/ReviewPipeline.png)
 
+## Getting Started
 
 ### Installation
 
-1. Download and unzip the [data](https://www.dropbox.com/scl/fi/mydblhx8yxk8kbz8b7zmr/AgentReview_data.zip?rlkey=se16p9gonclw5t8t3vn9p0o6n&st=6988u8lx&dl=0) under `data/`, which contains the PDF versions of the paper as well as the forum discussions for ICLR 2020 - 2023
-2. **Install Required Packages**:
+**Download the data**
+
+Download both zip files in the [Dropbox](https://www.dropbox.com/scl/fo/etzu5h8kwrx8vrcaep9tt/ALCnxFt2cT9aF477d-h1-E8?rlkey=9r5ep9psp8u4yaxxo9caf5nnc&st=k946oui5&dl=0):
+
+Unzip [AgentReview_Paper_Data.zip](https://www.dropbox.com/scl/fi/l17brtbzsy3xwflqd58ja/AgentReview_Paper_Data.zip?rlkey=vldiexmgzi7zycmz7pumgbooc&st=b6g3nkry&dl=0) under `data/`, which contains:
+   1. The PDF versions of the paper
+   2. The real-world peer review for ICLR 2020 - 2023
+   
+   ```bash
+   unzip AgentReview_Paper_Data.zip -d data/
+   ```
+
+(Optional) Unzip [AgentReview_LLM_Reviews.zip](https://www.dropbox.com/scl/fi/ckr0hpxyedx8u9s6235y6/AgentReview_LLM_Reviews.zip?rlkey=cgexir5xu38tm79eiph8ulbkq&st=q23x2trr&dl=0) under `outputs/`, which contains:
+   1. The LLM-generated reviews, (our LLM-generated dataset)
+    
+   ```bash
+   unzip AgentReview_LLM_Review.zip -d outputs/
+   ```
+   
+**Install Required Packages**:
    ```
-   cd AgentReview
+   cd AgentReview/
    pip install -r requirements.txt
    ```
+
+3. Set environment variables
+
+If you use OpenAI API, set OPENAI_API_KEY.
+
+```bash
+export OPENAI_API_KEY=... # Format: sk-...
+```
+
+If you use AzureOpenAI API, set the following
+
+```bash
+export AZURE_ENDPOINT=...  # Format: https://<your-endpoint>.openai.azure.com/
+export AZURE_DEPLOYMENT=...  # Your Azure OpenAI deployment here
+export AZURE_OPENAI_KEY=... # Your Azure OpenAI key here
+```
    
-3. **Run the Project**:
+**Running the Project**
 
-   **Note: all project files should be run from the `AgentReview` directory.**
-    
+   Set the environment variables in `run.sh` and run it:
+   
+   ```bash
+   bash run.sh
+   ```  
 
-## Data
+   **Note: all project files should be run from the `AgentReview` directory.**
 
-#### Project Structure
-- `app.py`: The main application file for running the framework.
-- `analysis/`: Contains Python scripts for various statistical analyses of review data.
-- `chatarena/`: Core module for simulating different review environments and integrating LLM backends.
-- `dataset/`: Scripts for handling dataset operations, such as downloading and processing submissions.
-- `demo/`: Demonstrative scripts showcasing the functionality of different components.
-- `docs/`: Documentation files and markdown guides for using and extending the framework.
-- `examples/`: Configuration files and examples to demonstrate the capabilities and setup of simulations.
-- `experiments/`: Experimental scripts to test new ideas or improvements on the framework.
-- `visual/`: Visualization scripts for generating insightful plots and charts from the simulation data.
+**Demo**
 
-#### Usage
+A demo can be found in `notebooks/demo.ipynb`
 
-**[UNDER CONSTRUCTION]**
 
+## Framework Overview
 
 ### Stage Design
 
@@ -94,7 +123,6 @@ Our simulation adopts a structured, 5-phase pipeline
 - Sometimes the API can apply strict filtering to the request. You may need to adjust the content filtering to get the desired results.  
 
 
-
 ## License
 
 This project is licensed under the Apache-2.0 License.

agentreview/const.py

@@ -100,3 +100,13 @@ year2paper_ids = {
     "ICLR2024": [39, 247, 289, 400, 489, 742, 749] + [62, 78, 159, 161, 170, 192, 198, 215, 219, 335, 344, 386, 427, 432, 448, 451, 461, 472, 485, 536, 546, 559, 573, 577, 597] + [5, 9, 11, 19, 20, 30, 31, 32, 40, 49, 52, 53, 54, 56, 61, 66, 67, 73, 74, 77, 85, 87, 100, 104, 114, 116, 124, 130, 133, 138, 145, 151, 153, 156, 165, 166, 172, 181, 183, 187, 195, 204, 212, 221, 224, 230, 237, 243, 248, 257, 258, 259, 263, 272, 278, 287, 288, 291, 292, 298, 300, 302, 304, 306, 308, 318, 320, 321, 324, 325, 326, 327, 331, 332, 334, 336, 338, 340, 345, 349, 350, 356, 357, 358, 360] + [1, 2, 12, 14, 24, 26, 33, 35, 36, 41, 42, 44, 50, 51, 55, 57, 59, 70, 72, 75, 76, 81, 89, 90, 93,
                   94, 97, 99, 101, 105, 110, 111, 112, 117, 119, 120, 125, 128, 129, 131, 134, 135, 140, 148, 150, 157, 158, 163, 167, 173, 175, 177, 182, 185, 186, 188, 189, 197, 202, 207, 209, 210, 214, 216, 226, 231, 234, 236, 238, 239, 241, 244, 245, 249, 260, 262, 264, 265, 271, 276, 277, 279, 281, 282, 284, 286, 290, 294, 295, 301, 303, 307, 309, 313, 315, 319, 322, 333, 337, 339, 342, 354, 363, 364, 369, 373, 374, 375, 377, 378, 381, 382, 385, 388, 398, 399, 401, 407, 412, 413, 415, 416, 417, 420, 421, 422, 426, 428, 436, 437, 444, 446, 449, 453, 454, 463, 464, 469, 478, 480, 487, 490, 496, 498, 501, 502, 504, 506, 513, 516, 517, 518, 520, 521, 523, 524, 525, 537, 541, 545, 551, 552, 554, 555, 558, 562, 563, 574, 575, 579, 581, 584, 588, 595, 596, 598, 607, 608, 615, 622, 624, 625, 627, 629, 630, 634, 636, 641, 645, 647, 648, 651, 652, 654, 655, 662, 667, 668, 671, 672, 673, 681, 682, 685, 689, 690, 691, 697, 698, 701]
 }
+AGENTREVIEW_LOGO = r"""
+                           _   _____            _               
+     /\                   | | |  __ \          (_)              
+    /  \   __ _  ___ _ __ | |_| |__) |_____   ___  _____      __
+   / /\ \ / _` |/ _ \ '_ \| __|  _  // _ \ \ / / |/ _ \ \ /\ / /
+  / ____ \ (_| |  __/ | | | |_| | \ \  __/\ V /| |  __/\ V  V / 
+ /_/    \_\__, |\___|_| |_|\__|_|  \_\___| \_/ |_|\___| \_/\_/  
+           __/ |                                                
+          |___/                                                                                   
+"""

agentreview/ui/cli.py

@@ -14,19 +14,10 @@ from agentreview.utility.utils import get_rebuttal_dir, load_llm_ac_decisions, \
     save_llm_ac_decisions
 from ..arena import Arena, TooManyInvalidActions
 from ..backends.human import HumanBackendError
+from ..const import AGENTREVIEW_LOGO
 from ..environments import PaperReview, PaperDecision
 
 # Get the ASCII art from https://patorjk.com/software/taag/#p=display&f=Big&t=Chat%20Arena
-ASCII_ART = r"""
-                           _   _____            _               
-     /\                   | | |  __ \          (_)              
-    /  \   __ _  ___ _ __ | |_| |__) |_____   ___  _____      __
-   / /\ \ / _` |/ _ \ '_ \| __|  _  // _ \ \ / / |/ _ \ \ /\ / /
-  / ____ \ (_| |  __/ | | | |_| | \ \  __/\ V /| |  __/\ V  V / 
- /_/    \_\__, |\___|_| |_|\__|_|  \_\___| \_/ |_|\___| \_/\_/  
-           __/ |                                                
-          |___/                                                                                   
-"""
 
 color_dict = {
     "red": Fore.RED,
@@ -87,7 +78,6 @@ class ArenaCLI:
 
         console = Console()
         # Print ascii art
-        console.print(ASCII_ART, style="bold dark_orange3")
         timestep = self.arena.reset()
         console.print("🎓AgentReview Initialized!", style="bold green")
 

demo.py

@@ -1,217 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-# # AgentReview
-# 
-# 
-# 
-# In this tutorial, you will explore customizing the AgentReview experiment.
-# 
-# 📑 Venue: EMNLP 2024 (Oral)
-# 
-# 🔗 arXiv: [https://arxiv.org/abs/2406.12708](https://arxiv.org/abs/2406.12708)
-# 
-# 🌐 Website: [https://agentreview.github.io/](https://agentreview.github.io/)
-# 
-# ```bibtex
-# @inproceedings{jin2024agentreview,
-#   title={AgentReview: Exploring Peer Review Dynamics with LLM Agents},
-#   author={Jin, Yiqiao and Zhao, Qinlin and Wang, Yiyang and Chen, Hao and Zhu, Kaijie and Xiao, Yijia and Wang, Jindong},
-#   booktitle={EMNLP},
-#   year={2024}
-# }
-# ```
-# 
-
-# In[2]:
-
-
-import os
-
-import numpy as np
-
-from agentreview import const
-
-os.environ["OPENAI_API_VERSION"] = "2024-06-01-preview"
-
-
-# ## Overview
-# 
-# AgentReview features a range of customizable variables, such as characteristics of reviewers, authors, area chairs (ACs), as well as the reviewing mechanisms 
-
-# In[3]:
-
-
-
-# ## Review Pipeline
-# 
-# The simulation adopts a structured, 5-phase pipeline (Section 2 in the [paper](https://arxiv.org/abs/2406.12708)):
-# 
-# * **I. Reviewer Assessment.** Each manuscript is evaluated by three reviewers independently.
-# * **II. Author-Reviewer Discussion.** Authors submit rebuttals to address reviewers' concerns;
-# * **III. Reviewer-AC Discussion.** The AC facilitates discussions among reviewers, prompting updates to their initial assessments.
-# * **IV. Meta-Review Compilation.** The AC synthesizes the discussions into a meta-review.
-# * **V. Paper Decision.** The AC makes the final decision on whether to accept or reject the paper, based on all gathered inputs.
-
-# In[2]:
-
-
-
-# In[4]:
-
-
-import os
-
-if os.path.basename(os.getcwd()) == "notebooks":
-    os.chdir("..")
-# Change the working directory to AgentReview
-print(f"Changing the current working directory to {os.path.basename(os.getcwd())}")
-
-
-# In[5]:
-
-
-from argparse import Namespace
-
-args = Namespace(openai_key=None, 
-          deployment=None, 
-          openai_client_type='azure_openai', 
-          endpoint=None, 
-          api_version='2023-05-15',
-          ac_scoring_method='ranking', 
-          conference='ICLR2024', 
-          num_reviewers_per_paper=3,  
-          ignore_missing_metareviews=False, 
-          overwrite=False, 
-          num_papers_per_area_chair=10, 
-          model_name='gpt-4o', 
-          output_dir='outputs', 
-          max_num_words=16384, 
-          visual_dir='outputs/visual', 
-          device='cuda', 
-          data_dir='./data', # Directory to all paper PDF
-          acceptance_rate=0.32, 
-          task='paper_review')
-
-os.environ['OPENAI_API_VERSION'] = args.api_version
-
-# In[13]:
-
-
-malicious_Rx1_setting = {
-    "AC": [
-        "BASELINE"
-    ],
-
-    "reviewer": [
-        "malicious",
-        "BASELINE",
-        "BASELINE"
-    ],
-
-    "author": [
-        "BASELINE"
-    ],
-    "global_settings":{
-        "provides_numeric_rating": ['reviewer', 'ac'],
-        "persons_aware_of_authors_identities": []
-    }
-}
-
-all_settings = {"malicious_Rx1": malicious_Rx1_setting}
-args.experiment_name = "malicious_Rx1_setting"
-
-
-# 
-# `malicious_Rx1` means 1 reviewer is a malicious reviewer, and the other reviewers are default (i.e. `BASELINE`) reviewers.
-# 
-# 
-
-# ## Reviews
-# 
-# Define the review pipeline
-
-# In[10]:
-
-
-from agentreview.environments import PaperReview
-
-def review_one_paper(paper_id, setting):
-    paper_decision = paper_id2decision[paper_id]
-
-    experiment_setting = get_experiment_settings(paper_id=paper_id,
-                                                 paper_decision=paper_decision,
-                                                 setting=setting)
-    print(f"Paper ID: {paper_id} (Decision in {args.conference}: {paper_decision})")
-
-    players = initialize_players(experiment_setting=experiment_setting, args=args)
-
-    player_names = [player.name for player in players]
-
-    env = PaperReview(player_names=player_names, paper_decision=paper_decision, paper_id=paper_id,
-                          args=args, experiment_setting=experiment_setting)
-
-    arena = PaperReviewArena(players=players, environment=env, args=args)
-    arena.launch_cli(interactive=False)
-
-
-# In[11]:
-
-
-import os
-import sys
-
-sys.path.append(os.path.abspath(os.path.join(os.getcwd(), "agentreview")))
-
-from agentreview.paper_review_settings import get_experiment_settings
-from agentreview.paper_review_arena import PaperReviewArena
-from agentreview.utility.experiment_utils import initialize_players
-from agentreview.utility.utils import project_setup, get_paper_decision_mapping
-
-
-# In[14]:
-
-
-sampled_paper_ids = [39]
-
-paper_id2decision, paper_decision2ids = get_paper_decision_mapping(args.data_dir, args.conference)
-
-for paper_id in sampled_paper_ids:
-    review_one_paper(paper_id, malicious_Rx1_setting)
-
-
-
-def run_paper_decision():
-    args.task = "paper_decision"
-
-    # Make sure the same set of papers always go through the same AC no matter which setting we choose
-    NUM_PAPERS = len(const.year2paper_ids[args.conference])
-    order = np.random.choice(range(NUM_PAPERS), size=NUM_PAPERS, replace=False)
-
-
-    # Paper IDs we actually used in experiments
-    experimental_paper_ids = []
-
-    # For papers that have not been decided yet, load their metareviews
-    metareviews = []
-    print("Shuffling paper IDs")
-    sampled_paper_ids = np.array(const.year2paper_ids[args.conference])[order]
-
-    # Exclude papers that already have AC decisions
-    existing_ac_decisions = load_llm_ac_decisions(output_dir=args.output_dir,
-                                                             conference=args.conference,
-                                                             model_name=args.model_name,
-                                                             ac_scoring_method=args.ac_scoring_method,
-                                                             experiment_name=args.experiment_name,
-                                                             num_papers_per_area_chair=args.num_papers_per_area_chair)
-
-    sampled_paper_ids = [paper_id for paper_id in sampled_paper_ids if paper_id not in existing_ac_decisions]
-
-
-
-
-# In[ ]:
-
-
-
-

run.sh

@@ -0,0 +1,12 @@
+# If you use OpenAI API, you need to set OPENAI_API_KEY.
+export OPENAI_API_KEY=...
+
+# If you use AzureOpenAI API, you need to set the following
+export AZURE_ENDPOINT=...  # Format: https://<your-endpoint>.openai.azure.com/
+export AZURE_DEPLOYMENT=...  # Your Azure OpenAI deployment here
+export AZURE_OPENAI_KEY=... # Your Azure OpenAI key here
+
+python run_paper_review_cli.py --conference ICLR2024 \
+    --openai_client_type azure_openai \
+    --data_dir data \
+    --experiment_name malicious_Rx1

run_paper_decision_cli.py

@@ -39,6 +39,8 @@ def main(args):
     """
     args.task = "paper_decision"
 
+    print(const.AGENTREVIEW_LOGO)
+
     # Sample Paper IDs from each category
     paper_id2decision, paper_decision2ids = get_paper_decision_mapping(args.data_dir, args.conference)
 
@@ -98,31 +100,6 @@ def main(args):
 
         players = initialize_players(experiment_setting=experiment_setting, args=args)
 
-        # players = []
-        #
-        # for role, players_li in experiment_setting["players"].items():
-        #
-        #     for i, player_config in enumerate(players_li):
-        #
-        #         # This phase should only contain the Area Chair
-        #         if role == "AC":
-        #
-        #             player_config = get_ac_config(env_type="paper_decision",
-        #                                           scoring_method=args.ac_scoring_method,
-        #                                           num_papers_per_area_chair=args.num_papers_per_area_chair,
-        #                                           global_settings=experiment_setting['global_settings'],
-        #                                           acceptance_rate=args.acceptance_rate
-        #                                                           ** player_config)
-        #
-        #             # player_config = AgentConfig(**player_config)
-        #             player_config['model'] = args.model_name
-        #             player = AreaChair(**player_config)
-        #
-        #         else:
-        #             raise NotImplementedError(f"Unknown role: {role}")
-        #
-        #         players.append(player)
-
         player_names = [player.name for player in players]
 
         if batch_index >= num_batches - 1:  # Last batch. Include all remaining papers
@@ -139,7 +116,6 @@ def main(args):
         arena = PaperReviewArena(players=players, environment=env, args=args, global_prompt=const.GLOBAL_PROMPT)
         arena.launch_cli(interactive=False)
 
-
 if __name__ == "__main__":
     project_setup()
     main(parse_args())

run_paper_review_cli.py

@@ -45,6 +45,8 @@ def main(args: Namespace):
 
     args.task = "paper_review"
 
+    print(const.AGENTREVIEW_LOGO)
+
     paper_id2decision, paper_decision2ids = get_paper_decision_mapping(args.data_dir, args.conference)
 
     # Sample paper IDs for the simulation from existing data.
@@ -67,65 +69,6 @@ def main(args: Namespace):
 
         player_names = [player.name for player in players]
 
-        # for role, players_list in experiment_setting["players"].items():
-        #
-        #     for i, player_config in enumerate(players_list):
-        #         if role == "Paper Extractor":
-        #
-        #             player_config = get_paper_extractor_config(global_settings=experiment_setting['global_settings'], )
-        #
-        #             player = PaperExtractorPlayer(data_dir=args.data_dir, paper_id=paper_id,
-        #                                           paper_decision=paper_decision,
-        #                                           args=args,
-        #                                           conference=args.conference, **player_config)
-        #
-        #             player_names.append(player.name)
-        #
-        #
-        #         elif role == "AC":
-        #
-        #             player_config = get_ac_config(env_type="paper_review",
-        #                                           scoring_method=args.ac_scoring_method,
-        #                                           num_papers_per_area_chair=args.num_papers_per_area_chair,
-        #                                           global_settings=experiment_setting['global_settings'],
-        #                                           acceptance_rate=args.acceptance_rate,
-        #                                           **player_config)
-        #
-        #             player_config['model'] = args.model_name
-        #
-        #             player = AreaChair(data_dir=args.data_dir,
-        #                                conference=args.conference,
-        #                                args=args,
-        #                                **player_config)
-        #
-        #             player_names.append(player.name)
-        #
-        #
-        #         elif role == "Author":
-        #
-        #             # Author requires no behavior customization.
-        #             # So we directly use the Player class
-        #             player_config = get_author_config()
-        #             player = Player(data_dir=args.data_dir,
-        #                             conference=args.conference,
-        #                             args=args,
-        #                             **player_config)
-        #
-        #             player_names.append(player.name)
-        #
-        #         elif role == "Reviewer":
-        #             player_config = get_reviewer_player_config(reviewer_index=i + 1,
-        #                                                        global_settings=experiment_setting['global_settings'],
-        #                                                        **player_config)
-        #             player_config['model'] = args.model_name
-        #             player = Reviewer(data_dir=args.data_dir, conference=args.conference, **player_config)
-        #             player_names.append(player.name)
-        #
-        #         else:
-        #             raise NotImplementedError(f"Unknown role: {role}")
-        #
-        #         players.append(player)
-
         env = PaperReview(player_names=player_names, paper_decision=paper_decision, paper_id=paper_id,
                           args=args, experiment_setting=experiment_setting)