Simulation of Research Activities with LLM x Tree Search

• By having a Large Language Model (LLM) read multiple existing papers, it is possible to generate fictional follow-up research. I thought that by repeating this process and expanding the directed acyclic graph of papers, researchers could simulate the process of reading papers and generating new research.

• I constructed this simulation using an algorithm similar to MCTS / UCB Tree Search. The output is evaluated by another LLM acting as a reviewer, using the evaluation scores to expand descendants in a positive direction while pruning bad branches.

• The top node (yellow) represents an existing paper, while the other nodes are generated by the LLM.

  • 

• The results are somewhat cherry-picked, so there is not complete confidence in the conclusions. However, the increase in evaluations for each node indicates that the utilization of good nodes is working well.

  • 

• It would be interesting to see how changing the definition of “good papers” given to the LLM affects the growth patterns.

• By altering the parameter c of UCB, changes in growth can also be observed.

  • When placing emphasis on exploration (c=0.3):
    • 
  • When placing emphasis on exploitation (c=0.1):
    • 

Some potential directions:

• It could be interesting to incorporate human feedback (e.g., gaze attention) to guide the graph’s growth in relevant directions.
• The same methodology could be applied to other brainstorming tasks, such as generating a game idea or a story plot.
• Creating multiple agents with different “interests” and allowing them to collaboratively expand the graph could be an intriguing approach.

Repository: https://github.com/blu3mo/simulacra-of-academia