This paper describes the usage of CRC checking in an application named FMFT (Fast Manager of File Transfer) that is based on Xinan Liu's Reliable File Transfer Protocol. The system consists of a server and a client C++ program utilizing UDP connection. The aim is to transfer big files quickly by using this program on busy network connections. The -based minimum viable product relying on Xinan Liu's solution was rewritten in C++ and enhanced by adding control over chunk size. To make this program available for as many platforms as possible, WebAssembly programming language was used. This article introduces the performance results of the utilization of CRC checking of sent and received packages. Finally, it will be discussed that thanks to an advanced scheduling, our application can utilize the network more eficiently.
Transferring a huge amount of data between workstations and supercomputer nodes
for processing introduces new problems: not only the upload process is slow, but
the result can be huge also to download. Even gigabit connections can slow down
to a few megabit range in a busy network in real-life use cases having even packet
loss or damage [
Transmission Control Protocol (TCP) based solutions have a lot of problems with trafic control. Usually, the transfer rate fluctuates a lot, even having just a small extra data transfer on the network from other nodes. A User Datagram Protocol (UDP) based software handles big data transfers a way better. Albeit, they have a challenge of the feedback of the unsuccessfully transferred packages. By creating a UDP based high-speed file transfer system, the demand to quickly move huge data between supercomputer nodes and workstations could be satisfied.
The UFTP and UFTPD software pair mostly accomplishes our needed features
[
Xinan Liu’s Reliable File Transfer Protocol was chosen for the starting point of
the development work, because beside its small code base it achieved the first place
in CS2105 (Introduction to Computer Networks) Speed Contest AY15/16 Sem1.
Moreover, we implemented the Stream Control Transmission Protocol (SCTP [
The structure of the paper consists of four main chapters. The development environment of the Fast Manager of File Transfer application is described in chapter two. In chapter three CRC checking is presented. Performance hints related to scheduling of packages are given in chapter four. The achievements, possible continuation, future research and development aims are enlisted in chapter five.
We chose Qt 5.15.1 stable version for the platform-independent development. The
work was carried out on a Debian 10.5 workstation. Mobile equipment as Android
and iOS has Qt support. Furthermore, it has a lot of desktop operating system
targets as Windows, Linux and macOS. In recent years one of the most interesting
new platform is WebAssembly [
The chosen programming language is the highly portable ++11, on the
contrary of QML. The server was created for command line usage. The client was
designed to have a native GUI on the actual operating system (see Figure 1). The
WebAssembly client is lacking SCTP support right now. After we had tested our
SCTP file transfer system for desktop operating systems, we found that the
performance of the TCP based FTP transfer and our SCTP and UDP based file transfer
implementations need more comprehensive investigations [
Tests were carried out on a gigabit network. Firstly, we worked with the
TCPbased FTP protocol. Then the initial approach in our software was to have a UDP
data channel with a TCP control channel similar to SABUL [
The testing was carried over with a test file of 100 MB, we transferred data between a desktop workstation and the test server which is in the Gyires supercomputer’s server room. Figure 2 shows values of the successive transfers starting from 200 bytes to 4000 bytes of chunk sizes. It can be seen that without computing CRC-16 for data packets, the transfer rate maintains over 800 Mbps from chunk size of 1100 bytes up to 4000 bytes.
1,000 ] s 800 p b M [ e t a r 600 r e f s n a r T 400 200
200 1,000 2,000 3,000 4,000
By computing CRC-16 for data packets, the transfer rate cross the 800 Mbps boundary with 824 Mbps just at chunk size of 2400 bytes. The curve of results of transfer rates with computing CRC-16 values stays almost entirely under the curve of raw transfer of packages. This was consistent over various sized test data with the diference being around 5% with higher than 2400 bytes chunk sizes. For small chunk sizes as 900 bytes the diference can be as high as 37%. During our test runs we have not encountered any package to fail the CRC-16 check. The reason for this was maybe the fact that the packages had to travel a relatively few hops. For longer data transfer paths having more hops it is potential to start having failing packages at CRC-16 checking.
The test server was a Cisco UCS C240 M5 having gigabit Ethernet connection, connected to the Gyires supercomputer. The applied virtual machine runs Ubuntu 18.04.5 LTS server version under VMWare’s vSphere 6.7 having 20 GB memory and 8 cores of an Intel® Xeon® Gold 6130 CPU @ 2.10 GHz. The test machine has an Intel® Core™ i7 CPU 920 @ 2.67 GHz with 18 GB memory having gigabit Ethernet connection, connected to the academic Internet network running Debian 10.5 desktop version.
Qt has a function for calculating a checksum value since version 5.9: quint16
qChecksum(const char *data, uint len), which is applied for the sent and
received data packages in our server-client pair of programs. This function is using
the CRC-16-CCITT algorithm having a 16-bit cache conserving (16 entry table)
implementation [
The first solution was to send packages continuously from the client to the server. It
meant that a lot of packages got lost, while the CPU and the network stack was at
maximum stress. The best packet transmission success rate for tests was 92.6% in
the based minimum viable product. The reimplemented ++ pair of software
decreased the packet transmission success rate to 83.3% what is explained by the
twofold increase in eficiency to stress both the CPU and the network stack. By
introducing a transfer rate cap at software side the packet transmission success rate
was increased over 90.0% again. The scheduling was based on one second length
bursts of packages. It means that after sending out enough number of packages of a
preset chunk size for the predetermined transfer rate, there was a pause in sending
until the next one second time slot. It gives network bufers an unnecessarily heavy
load at regular intervals. The next iteration was to fine tune the time interval to
one millisecond. The backside of this solution was that the precomputed number
of packages for one slot was rather small for higher chunk sizes. This means that
the predetermined transfer rate may be achieved by only a relatively big margin,
while the success rate improvement was not that great. Although, the CPU and
network stress became smaller. The final solution was to use a nanosecond timer
for scheduling. At each timer event we check one second history of sent packages.
If it shows that the actual transfer rate is under the desired target, the required
number of packages of the given chunk size is sent out to achieve the predetermined
transfer rate. Under normal circumstances it means only one extra packet to be
sent. The predetermined transfer rate is approximated accurately at a very low
margin this way. While it has even a little more stress on the CPU side, it further
minimizes the stress on the network stack and bufers. Consequently, it leads to
fewer lost and resent packages [
Our first implementation was enhanced to a high-speed connectionless file transfer system by using UDP control channel relying on Xinan Liu’s work. File transfers were increased over 800 Mbps on busy connections. Using WebAssembly, the client even runs in browsers. There is a chance for malicious nodes on the network for tampering with packets using a man in the middle attack. Besides this the integrity of the packets can be damaged by accidental bit alternations anywhere in the network chain.
CRC checking is a good and eficient way for identifying packet modifications. A CRC-16-CCITT algorithm based solution was implemented to mitigate this kind of damage. The transfer ratewith packets using a man-in-the-middle attack hit is around 5% for usable chunk sizes, while the performance hit on CPU is negligible. We have not encountered packets failing the CRC check. However, on longer network paths it will be a more useful tool for reducing packet retransmissions.
The entire data file should have a CRC-32 or CRC-64 test for ensuring the integrity of the process. We have started to work on encryption also. It may be improved by utilizing DTLS for securing the whole connection. The server shall be made fully multithreaded to have a thread for each transfer session to service more clients eficiently. Broken transfers will be enabled to resume later. The client will be modified to be a dual-pane file transfer and manager software.