Code translation involves converting a program written in one programming language into another, ensuring that the original functionality remains intact. Neural network models have achieved significant success in this task, but recent studies [1,2] have found that these models often introduce subtle errors. These subtle errors can be grouped into syntactic and semantic errors. Syntax errors violate the syntax rules of destination languages, which a grammar checker can often identify. In contrast, semantic errors are more subtle and may result in translated code that either fails to execute without violating the target language's syntax or produces outputs that are inconsistent with the original code [3]. For example, as shown in the replace function in Figure 1, s.replace('-', ' ') in Python replaces all occurrences of '-' with ' ', while in JavaScript, it only replaces the first occurrence by default.

Based on this, Wang et al. [3] rely on test cases that can expose semantic errors to analyze code and locate these errors dynamically. However, their method is highly dependent on the quality of the test cases, requiring them to reveal semantic errors effectively, and it cannot handle code snippets lacking valid test cases. Additionally, In the code translation domain, relying on test cases to execute code is not only costly but also poses potential security risks [4].

To automatically locate semantic errors in code translation in the absence of valid test cases, we propose a framework KSAF, which decomposes the source code and translated code synchronously and statically analyses the code to locate semantic errors with fine-grained knowledge combined with AI chain-driven LLM. Experiments show that our approach can achieve better results. KSAF is the first method to locate semantic errors in code translation without test cases. It only requires API documentation, does not need model training, and is adaptable to low-resource languages. Figure 1 illustrates the general framework of our approach. We first build an API knowledge base by crawling the official JavaScript documentation [5]. Then, we design a framework based on the knowledge-driven AI chain and code decomposition to locate errors in code translation statically.

In this work, we collect API documents from the online resource [5] through the web crawler tool [6], where each API document is a crawled web page containing rich information such as the name of the API, syntax, parameters, samples, and function descriptions. We only keep half-structured API statements and functional descriptions, where an API statement describes the fully qualified name(FQN) of an API, which serves as a retrieval index for the knowledge base. The functional description contains the functional logic and behavior of that API.

In the Neural Source Map Generator module, as shown in Figure 1 A, the source code, translated code, and a fixed prompt are input into the LLM. This module produces a mapping between atomic fragments in the source code and their corresponding parts in the translated code, providing an ordered list of these atomic fragments.

In the Code AST Decomposition module, as shown in Figure 1 B, the abstract syntax tree(AST) of the source code is traversed to extract "subtrees" from eight types of nodes as sub-code [7]. Using the mapping list from Module A, the corresponding translated code for each sub-code is obtained. Each sub-code pair is then passed to the next module.

After obtaining the sub-code and its corresponding translated code, KSAF uses LLM for static analysis to identify semantic inconsistencies between the source and translated code. We designed a knowledge-driven LLM AI Chain workflow, as shown in Figure 1 C, which includes three steps: Checking, Comparing, and Locating, all using the same LLM. In the Checking step, KSAF inputs the source code, translated code, and a fixed prompt into the LLM to extract the fully qualified names (FQN) of operators and APIs, then passes the results to the Comparing step. In the Comparing step, the FQNs are linked with an offline-built API knowledge base to obtain the corresponding API function descriptions. These descriptions and the results from the Checking step form a prompt fed into the LLM to precisely summarize the differences in operators and APIs between the source and translated code. In the Locating step, the Comparing step results, source code, and translated code are input into the LLM as a prompt to identify suspicious code lines that might cause semantic inconsistencies between the source and translated code.

In this module, our objective is to compare the effectiveness of KSAF with other methods. To ensure fairness in the experiments, we selected methods that, like KSAF, do not require test cases for static code analysis. Specifically, we chose the widely recognized prompt-based methods that aim to fully leverage the potential of foundational models: LLMs with Few-Shot Learning [8]: A few examples are provided as demonstration examples in the prompt to guide the LLM in achieving better performance on the task. LLMs with Chain of Thought (CoT) [9]: By appending "Let's think step by step" at the end of the prompt, the LLM is prompted to explain the reasoning or steps before providing the final answer.

We used the dataset (excluding test cases) and metrics [3] of Wang et al. to evaluate our method and baseline. Where 𝒮 𝑠𝑒𝑚 , 𝒮 ℎ𝑖𝑑 , and 𝒮 𝑑𝑖𝑓 denote the ratios of successfully identified errors to the total number of semantic errors, hidden errors, and errors leading to results that differ from the source code output, respectively. Semantic errors are when the code is syntactically correct but logically flawed, causing the program to behave in a way that is not expected. Hidden errors are a special kind of semantic error, which usually can't be immediately localized to a specific fix, even when running test cases. Errors leading to results that differ from the source code output are also a type of semantic error, which does not cause a runtime error but causes the output of the translated code in unit tests to be inconsistent with the source code [3].

As shown in Table 1, our method outperforms all baseline approaches. Additionally, the method proposed by Wang et al. is unable to handle code without test cases, resulting in zero values for all metrics. And following the experimental setup of previous work [3], we found that our framework KSAF detected an average of 3.0 suspicious lines, which represents 16.5% of the total lines of code. This indicates that users typically need to review only 1 to 3 lines to understand and fix semantic errors.

This paper propose a method based on code AST decomposition and fine-grained knowledge combined with an AI chain-driven LLM to locate semantic inconsistencies between source and translated code. This method effectively handles code without test cases. We plan to extend our approach to multi-language datasets and conduct comprehensive experiments to further validate KSAF's effectiveness in the future.