In the task of code translation, neural network-based models frequently generate semantically incorrect code that deviates from the original logic of the source code. This problem persists even with advanced large models. While a recent approach suggests using test cases to identify these semantic errors, its efectiveness is highly dependent on the quality of the test cases, making it unsuitable for code snippets that lack test cases in real-world scenarios. To automatically locate semantic errors in code translation without valid test cases, we propose the Knowledge-guided Semantic Analysis Framework (KSAF). KSAF decomposes the source and translated code synchronously and performs static analysis to detect semantic errors. This is achieved by leveraging fine-grained knowledge in conjunction with an AI chain-driven Large Language Model (LLM). In a previously studied benchmark of Python programs, our framework based on the GPT-3.5-turbo model achieved a correctness rate of 47.8% through a static evaluation method. This result represents a 37.2% improvement over the baseline using the same base model and a 13.4% improvement in correctness compared to the baseline using the GPT-4-turbo-based model.
CEUR ceur-ws.org
Code translation involves converting a program written in one programming language into
another, ensuring that the original functionality remains intact. Neural network models have
achieved significant success in this task, but recent studies [
Based on this, Wang et al. [
To automatically locate semantic errors in code translation in the absence of valid test cases, we propose a framework KSAF, which decomposes the source code and translated code synchronously and statically analyses the code to locate semantic errors with fine-grained knowledge combined with AI chain-driven LLM. Experiments show that our approach can achieve better results. KSAF is the first method to locate semantic errors in code translation without test cases. It only requires API documentation, does not need model training, and is adaptable to low-resource languages.
Translated Code javascript code=" function f_gold(s, k) { s = s.replace('-', ' ).toUpperCase(); let res = []; let cnt = (s.length % k) || k; let t = 0; for (let i = 0; i < s.length; i++) { res.push(s[i]); t += 1; if (t == cnt) { t = 0; cnt = k; if (i != s.length - 1) { } }ret}ur}nrreess..pjouisnh('(')-;');
Source Code def f_gold(s: str, k: int) -> str: s = s.replace('-', ' ).upper() res = [] cnt = (len(s) % k) or k t = 0 for i, c in enumerate(s): res.append(c) t += 1 if t == cnt: t = 0 cnt = k if i != len(s) - 1: res.append('-’) return ' .join(res) (A) Neural Source Map
Generator (B) Code AST Decomposition
Knowledge Base (C) Checking (C) Comparing
(C) Locating
Suspicious Lines s = s.replace('-', ' ).toUpperCase(); translated code, and a fixed prompt are input into the LLM. This module produces a mapping between atomic fragments in the source code and their corresponding parts in the translated code, providing an ordered list of these atomic fragments.
In the Code AST Decomposition module, as shown in Figure 1 B, the abstract syntax tree(AST) of the source code is traversed to extract ”subtrees” from eight types of nodes as sub-code [7]. Using the mapping list from Module A, the corresponding translated code for each sub-code is obtained. Each sub-code pair is then passed to the next module.
After obtaining the sub-code and its corresponding translated code, KSAF uses LLM for static analysis to identify semantic inconsistencies between the source and translated code. We designed a knowledge-driven LLM AI Chain workflow, as shown in Figure 1 C, which includes three steps: Checking, Comparing, and Locating, all using the same LLM. In the Checking step, KSAF inputs the source code, translated code, and a fixed prompt into the LLM to extract the fully qualified names (FQN) of operators and APIs, then passes the results to the Comparing step. In the Comparing step, the FQNs are linked with an ofline-built API knowledge base to obtain the corresponding API function descriptions. These descriptions and the results from the Checking step form a prompt fed into the LLM to precisely summarize the diferences in operators and APIs between the source and translated code. In the Locating step, the Comparing step results, source code, and translated code are input into the LLM as a prompt to identify suspicious code lines that might cause semantic inconsistencies between the source and translated code.
In this module, our objective is to compare the efectiveness of KSAF with other methods. To ensure fairness in the experiments, we selected methods that, like KSAF, do not require test cases for static code analysis. Specifically, we chose the widely recognized prompt-based methods that aim to fully leverage the potential of foundational models: LLMs with Few-Shot Learning [8]: A few examples are provided as demonstration examples in the prompt to guide the LLM in achieving better performance on the task. LLMs with Chain of Thought (CoT) [9]: By appending ”Let’s think step by step” at the end of the prompt, the LLM is prompted to explain the reasoning or steps before providing the final answer.
We used the dataset (excluding test cases) and metrics [
As shown in Table 1, our method outperforms all baseline approaches. Additionally, the
method proposed by Wang et al. is unable to handle code without test cases, resulting in zero
values for all metrics. And following the experimental setup of previous work [
This paper propose a method based on code AST decomposition and fine-grained knowledge combined with an AI chain-driven LLM to locate semantic inconsistencies between source and translated code. This method efectively handles code without test cases. We plan to extend our approach to multi-language datasets and conduct comprehensive experiments to further validate KSAF’s efectiveness in the future.
This work was supported by the Project of Science and Technology Research and Development Plan of China Railway Corporation (N2023J044). [5] Mozilla, Javascript reference, https://developer.mozilla.org/en-US/docs/Web/JavaScript/
Reference, 2024. Last accessed: March 11, 2024. [6] beautiful-soup 4, Beautiful soup 4, 2024. https://beautiful-soup-4.readthedocs.io/en/latest/. [7] T. Hu, Z. Xu, Y. Fang, Y. Wu, B. Yuan, D. Zou, H. Jin, Fine-grained code clone detection with block-based splitting of abstract syntax tree, in: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 89–100. [8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [9] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners, Advances in neural information processing systems 35 (2022) 22199–22213.