Replication Package for "Who's Who? LLM-assisted Software Traceability with Architecture Entity Recognition"

by Dominik Fuchß, Haoyu Liu, Sophie Corallo, Tobias Hey, Jan Keim, Johannes von Geisau, and Anne Koziolek

Requirements

The requirements are defined in INSTALL.md.

Structure

• Replication-Package-ICSA25: contains the original replication package of the ICSA 2025 paper with all information about the experiments, data and results.
• Replication-Package-TAAS25: contains the new version of our tooling and the data for the new experiments.
• TAAS25_Raw_Outputs: contains the output of our test runs for TAAS 2025

Important Notes regarding TAAS 2025

• The new version of the tooling is very similar to the one used in ICSA 2025, but it has been updated to work with the latest version of the LLMs and includes some bug fixes
• You will need an OPENAI_API_KEY to run the experiments
• You need to setup the environment exactly as for the ICSA 2025 replication package
• We provide a docker image that encapsulates the complete as already done for the ICSA 2025 replication package: ghcr.io/ardoco/taas25:latest
• The human-readable cache for the LLMs is stored in Replication-Package-TAAS25/.cache-llm
• To execute the experiments, you need to execute the following tests (we suggest using IntelliJ to execute the tests) but in Replication-Package-TAAS25 you can run mvn -q test -Dsurefire.failIfNoSpecifiedTests=false -Dtest=<<TEST_NAME>> ...
  1. ArtemisIT : to run the Artemis + TransArC evaluation
  2. TransarcIT : to run the TransArC evaluation
  3. ExArchIT : to run the ExArch + TransArC evaluation
  4. ArdocodeIT : to run the Ardocode evaluation
  5. ArtemisInExArchIT : to run the Artemis + ExArch + TransArC evaluation
  6. ArtemisInTransarcIT : to run the Artemis + TransArC evaluation