The data collected during various kinds of scienti c research may be represented both by well known binary le formats and by custom formats specially developed for some unique device. While thorough understanding of the le format may be required for the former case of the well known format, for the latter case of custom formats it is of critical importance. For the custom formats usually only few people know how they are organized, and this expertise can easily be lost. We have developed the language FlexT (Flexible Types) for speci cation of binary data formats. The language is declarative and designed to be well understood for human readers. Its main elements are the data type declarations, which look very much like the usual type declarations of the imperative programming languages, but are more exible. The primary purpose of the language FlexT development is to be able to view the binary data and to check that the data conform to the speci cation, and that the speci cation conforms to the data samples available. As a result of the tests we can be sure, that the speci cation is correct. The FlexT speci cations doesn`t contain any surplus information besides from that about the le format. They are compact and human readable. We have also developed the algorithm for data reading code generation from the speci cation. In the report we'll consider some FlexT language details and the experience of its application to speci cation of some scienti c data formats.
In the eld of natural sciences the outcomes of some research are usually represented by the articles, which summarize the main results of the analysis of the collected data. But the results of the analysis depend also on the methods chosen, their parameters, and some other subjective factors, so the other researchers are interested in the access to the data (both raw original and processed) to be able to check the results, try their own ways of the data analysis, or compare the data to their own ones. The concept of scienti c data life-cycle should meet this demand.
So, it becomes not enough to just obtain the data, process them and write some articles using the results of the processing. It is also required to share the data with other researchers. These researchers may be not only our contemporaries but also our descendants, living in a few decades from now. Our descendants, we hope, will have more advanced devices, computers and software. But they still will not be able to measure the values of physical quantities for previous periods (say, several years ago). So the data, that we have stored for them, may become very valuable. Let us consider the possible ways of the data representation, compare their features, and describe some approaches, that should simplify data sharing. 2
When deciding on the data representation it is required to select between several groups of options. We may consider the groups as a coordinates in the space of possible le formats. Let us consider the dimensions and their possible values.
The rst classi cation is by the level of customization: { well-known le formats with ready to use data processing libraries and software (JPEG,TIFF,DBF,CSV,SHP,...); { data exchange frameworks with the data format setup/data processing/code generation libraries, software; { "universal" formats with the data format setup/data processing/code generation libraries, software; { custom le formats.
So it is required to select between the well-known le formats, which may already have the ready to use libraries for their processing; the data exchange frameworks and "universal" formats, which require some additional information about the particular data structures, but provide a lot of ready to use services for their handling and the custom le formats, which will require to write all the software from scratch. Since the existing le formats may impose some unacceptable limitations on the data structures or have too large memory overhead, the choice here is far from being obvious. The "universal" le formats will be addressed in more detail in the next sections.
The second classi cation is by the le format usage: there exist internal and exchange le formats. The internal le formats are designed to maximize the e ciency and ease of processing by a speci c software. They may contain some auxiliary data structures (e.g. indices) and be very complex. The internal le formats are usually binary, unless the les are very small. The exchange le formats should be rather simple and easily understandable for other programmers, but they may be not very well-suited for everyday work (e.g. data editing).
The next option to consider when selecting the data representation is whether to use a text-based or a binary data representation. The binary data formats are much more space- and time-e cient, than the text-based ones. The main disadvantage of the binary data is that they look opaque to the users and it is hard to control their contents with a "naked eye". That's why programmers nevertheless often prefer to use text formats, and among them the XML-based ones are of great popularity in spite of the fact that it becomes impossible for a human being to comprehend the extremely large text les. The language FlexT can make binary data transparent. 3
The "Universal" formats allow you to store a wide (but still limited) range of information. They are always accompanied by some software libraries and utilities for handling the les of the format.
Some examples:
{ XML (eXtensible Markup Language) is a text based format family, they may
use XML schemata, various utilities and editors, libraries/code generation
software to simplify data processing. Various le format projects claim to be
the binary version of the XML format, but no proposal has been accepted
as a binary XML standard.
{ Hierarchical Data Format (HDF) [
The method involves an interface description language that describes the structure of some data and a program that generates source code from that description for writing or reading a stream of bytes that represents the structured data. In contrast to HDF the resulting les are not self-contained and require the structure speci cation (or, better, the code generated from it) to read the les, corresponding to this speci cation.
"Universal" data formats are often used to represent scienti c data. Not to mention XML, which is used everywhere, HDF les, for example, are used for representation of remote sensing data, while the Protocol Bu ers are the main way to represent the weights of deep neural networks. The main reason for this is the availability of tools and libraries that simplify the development and processing of "Universal" formats at the cost of following their prescripts. Our approach makes it possible to do nearly the same with an almost arbitrary binary le format. 4
The main goal of the language FlexT is to provide the instrument, that can help us to explore and understand the contents of the binary les, for which there exists a speci cation of their format. It can also help to check whether the format speci cation is correct using the samples of data in this format. FlexT speci cations can be used to check, view, process, and document both well-known and custom le formats.
Our experience shows that the vast majority of format speci cations written in natural language contain errors and ambiguities, which can be detected and xed by trying to apply various versions of the speci cation to the sample data in order to nd the correct variant of understanding of the format description.
The information about a le format may also be obtained from the source code of a program that reads or writes the data. But the data access code contains a lot of unessential details about a speci c way to read or write data and it is usually intermixed with the other code, that somehow processes the data. So, the resulting FlexT speci cation, which leaves out all the surplus details, will be much more concise and of much higher quality than the source code.
In contrast to many other projects intended for speci cation of binary le formats, the FlexT speci cations are human-readable, and it is possible to consider them as a concise notation for representation of the information about a binary le format.
We also have a successful experience of reverse engineering of some le formats using just the samples of data without any description or source code.
The format speci cations are also required to write a correct program, that should work with the les of the format. Because the FlexT language data types look similar to that of imperative languages, it is possible to immediately use some parts of speci cation to declare the data types, constants, and so on, which are required to write the data processing code. Anyway the process of writing the code manually is still time-consuming and error-prone and we have implemented the code generator, which can automatically produce the data reading code in imperative languages from the FlexT speci cations.
By now we have implemented the code generation for the most widely used FlexT data types, but some complex types like that used in speci cations of machine instructions' encoding are not supported yet. 5
The major part of the information about a le format is represented in the FlexT speci cation by the data type declarations. In contrast to the data types of imperative programming languages, the FlexT data types can contain data elements, the size of which is determined by the speci c data represented in the format. Thus we can say that the types exibly adjust to the data. It explains the name of the language FlexT (from Flexible Types). After de ning the data types, it is required to specify the placement in memory of some data elements which have some of these types. Such top-level data elements by analogy with imperative programming languages we will call variables (although they represent immutable data).
The language syntax was chosen to be well-understandable by human reader.
When designing some of the languages discussed in the review, such as the EAST
language, this task was explicitly formulated in the requirements for
development. The language FlexT was discussed in more details in [
By static data types we will understand the data types analogous to the traditional types of procedural programming languages. Their distinctive feature is that the size of a data element of this type and the internal placement of its constituent parts are determined at the time of compilation and do not depend on speci c data. In procedural programming languages that are actually used at present, the composite data types can contain only the static elements.
The size of a data element of dynamic type and the internal placement of its sub-elements may depend on speci c data, which it represents. Hereinafter the word "dynamic" will be used in this sense. String constants are an example of dynamic types in traditional procedural languages. Thus, to determine the size that occupies the value of an ASCIIZ string, it is required to look through the entire its contents to nd the null terminating character.
The dynamic data types can't be used as the types of variables, at least if this type is mutable, because assignment of a new value to a sub-element of a variable may cause a change in its size, and no compiler can e ectively support this kind of operations. The modern string data types, which look like dynamic for the programmer, use dynamic memory to store actual data, and the variables of the types themselves are of xed size, i.e they are static in terms of our classi cation. The good example of the complex dynamic data structures, that we can encounter in the modern imperative languages, is the run-time type information (RTTI). The RTTI is located in the constant sections of the executable les and its representation supports compact encoding of identi ers and some tables of variable lengths.
But when it comes to storing the information in binary les, the dynamic data types become a natural choice. That's why they are so important for the speci cations of binary le formats. If we ignore the need to adhere to the requirements to the types of variables of imperative languages, then we can naturally support descriptions of fairly complex dependencies between data elements in the language. We use this approach in the design of the FlexT language. 5.2
Data types can have a number of properties, the set of which depends on the kind of the type. For example, the size and the number of elements are the properties of arrays, and the selected case number is the property of variants. Each data type has the property Size. The values of the properties can be speci ed in the statements of type declaration, and also by expressions that compute the value of this property using the values and properties of the nested data elements, and using the values of the parameters of the type. The parameters in the type declaration represent the information that needs to be speci ed additionally when the type is used (called). 5.3
The main data types of the language FlexT are shown in the Table 1. The footnote a marks the data types, which are supported by the current version of the data reader code generation algorithm. 5.4
The language FlexT has the statements, which are designed to de ne various kinds of data types (array, record, variant, etc.), of their own syntax. At the same time, there are many tasks that need to be addressed for all the types, regardless of their kinds. For this purpose the language has the blocks of additional information, that can be placed using the symbol ':' after any type de nition. The most important of the blocks are: { the block of assignments intended to specify the values of the data type properties and the parameters of its sub-elements; { the block assert, that we use to specify the condition of correctness of the data type value; { the block displ, that speci es the way, the data type value should be displayed; { the blocks let, that can de ne new computed properties of the data type.
Raw dataa
menta
declarationa Machine codes of TOpPDP ?( @. Op instruc- >= TWOpCode . br ) and ...; tions aSupported by the data reading code generator
Let us consider some examples of data formats described in FlexT, which demonstrate some capabilities of the language. 6.1
The STL
les
The STL (from STereo-Lithography) le format [
The Listing 1.1 shows the speci cation of the STL format in FlexT. The STL format has binary Fig. 1: A 3D model shown and text versions. The binary version has no signa- from an STL le ture or magic value, that could be used to check whether the le is correct, and to distinguish the binary STL les from the text STL les. Instead, the binary STL les start with an arbitrary 80-char header. Fortunately, the text version should start with the keyword solid, so we can split the rst 5 chars from the header and check, that they don't contain the text "solid" using the assert statement.
The FlexT library has no prede ned oating point data types. So, the
single precision (4-byte IEEE) oating point data type TSingle speci cation is
Listing 1.1: The STL format
speci cation in FlexT
data
0 array [
TSTLPoint Normal
array [
Word Attr ends data 5 array [75] of char Hdr1 80 ulong Count assert 84+ Count * TSTLFace : Size =
FileSize ; data 84 array [ Count ] of TSTLFace Faces Fig. 2: An STL le parse results taken from the le Float.rfi. The 3-D vectors are represented by the array TSTLPoint. The mesh face (always triangle) is represented by the TSTLFace record, that contains the face normal, three vertices and two bytes of attributes.
The le header is followed by the total face Count and the array of Faces. The second assert checks, that the le size corresponds to the face count.
The Fig. 2 shows the initial fragment of the parse results for the STL le from the Fig. 1 using this speci cation. Here we can easily understand the le data, so using the FlexT speci cation the binary le becomes absolutely transparent. 6.2
The HiSCORE custom
le format is designed to represent the readings of the
TAIGA-HiSCORE instrument [
Listing 1.2: The complete HiSCORE format speci cation in FlexT type bit TBit5 num +(5) : displ =( int (@)) TBit6 num +(6) : displ =( int (@)) TDNS num +(7) : displ =( int (@ *10) ) TBit10 num +(10) : displ =( int (@)) TTime struc
TDNS dns TBit10 mks TBit10 mls TBit6 s TBit6 m TBit5 h num +(20) rest ends : displ =( int (@.h) ,':', int (@.m) ,'''', int (@.s) ,'.', int (@. mls ) ,' ',int (@. mks ) ,' ',int (@. dns )) type TVal num +(2) : displ =( int (@)) TTrackInfo struc word offset // track offset TVal N // length N array [@.N] of TVal Data // N bytes - track data ends : displ =( '[ ' , ADDR (& @) ,']',@) TPkgData ( Sz ) struc array [9] of TTrackInfo Tracks ulong Stop // 4 bytes - FF FF FF FF - package end raw [] rest // Just in case ends :[ @: Size =@: Sz ]: assert [@. Stop =0 xFFFFFFFF ] TPkgHdr struc // Package header (24 bytes ): word idf // data type id = 3008 word NumBytes // package size ( without the 24 bytes of the header ) ulong NumEvent // event counter number ulong StopTrigger // position of stop trigger in DRS counts TTime EventT // event time word IP // IP adress word NumSt // Number of station
TPkgData (@. NumBytes ) Data ends : assert [@. idf =3008] data 0 array of TPkgHdr :[ @: Size = FileSize ] Hdr
The section type bit contains the declarations of the bit-oriented types, which are used to describe the time representation in the HiSCORE les by the TTime bit record.
From the top-level view the HiSCORE le is a sequence of packages, represented by the TPkgHdr record. So, the size of the array of packages is set to the FileSize value. Each package contains a header of constant size and a data block of the size speci ed in the NumBytes header eld. The main content of the package data block is the 9 tracks (the TTrackInfo data type), each track contains the value count N and a sequence of N values.
So, the eld NumBytes is in fact redundant, because this size could be computed from the values of the TTrackInfo.N elds. To check, that these values match the TPkgData record has an extra eld rest: this eld should normally be of zero size, and the zero-size record elds are not shown by default. But if something goes wrong, we will see a hex dump here.
The Fig. 3 shows an initial fragment of the results of parsing a HiSCORE le.
read hisc.c - 278 lines, Hiscore.r - 47 lines 6.3
One of the possible sources of information about a le format is the source code, which can process (read or write) les of this format. The advantages of the source code over the descriptions in natural language are its proved correctness (the code can indeed process the data) and the lack of ambiguity. So, it may seem that understanding a le format by analyzing a source code for its processing will always be easy and preferable to reading the speci cations in natural language. The only expected problem is that it may be hard to lter out the data reading/writing code from the much bigger source intended for some particular data processing purposes.
In our experience of FlexT usage we have an indicative example, which demonstrates, that sometimes it may be very hard to understand the le format using the source code.
The le format MM5 is used for representation of the weather forecast data,
computed by the same-name Earth climate model [
Listing 1.3: Excerpts from the le readv3.f for reading the MM5 data version 3 program readv3 ! This utility program is written in free - format Fortran 90. ...
integer , dimension (50 ,20) :: bhi real , dimension (20 ,20) :: bhr character ( len =80) , dimension (50 ,20) :: bhic character ( len =80) , dimension (20 ,20) :: bhrc character ( len =120) :: flnm integer :: iunit = 10 ...
print *, ' flnm = ', trim ( flnm ) open ( iunit , file = flnm , form = ' unformatted ', status = ' old ', action = ' read ') ...
read ( iunit , iostat = ierr ) flag do while ( ierr == 0) if ( flag == 0) then
read ( iunit , iostat = ier ) bhi , bhr , bhic , bhrc
call printout_big_header ( bhi , bhr , bhic , bhrc ) elseif ( flag == 1) then
After writing the FlexT speci cation of the MM5 v3 le format and parsing some les it became obvious that the les contain some additional bytes, which prevent it from reading in C. The additional 4-byte values match the sizes of the data written or read by each of the read/write Fortran operators.
The further investigation revealed, that Fortran write operators for binary les consider the list of their arguments as a block, and add information about the size of the block to the le. The four bytes of the size are written before each such block and also after it. The corresponding Fortran read operators expect to nd the block sizes in the le and correctly skip them when reading. They can also check the correctness of the le by comparing the argument list sizes to the block sizes from the le.
As a result, even the Fortan programmers may be unaware about the Fortran
binary data blocks representation details. Of course, they should know, that it
is required to read the data grouped the same way, as they were grouped when
written, but no more. For example, consider the following tutorial about the
Fortran le IO [
Note that the case in question dealt with the Fortran language, which is very ancient, but still alive. And now imagine what may happen if some cryptic esoteric language would be used instead.
The Listing 1.4 contains the fragment of the MM5 le format speci cation in FlexT, which corresponds to the last read operator in the Listing 1.3. Listing 1.4: Excerpt from the MM5 data version 3 format speci cation in FlexT TBHi array [50] of array [20] of i4 Tbhr array [20] of array [20] of TReal TComment array [80] of Char , <0 x20 ; TBHiC array [50] of array [20] of TComment TbhrC array [20] of array [20] of TComment TBigHeader struc u4 BHSize // Size of Data - added automatically by Fortran write TBHi BHi Tbhr bhr TBHiC BHiC TbhrC bhrC u4 BHSize_ // Size of Data - added automatically by Fortran write ends : assert [@. BHSize =@: size -8 ,@. BHSize_ =@: size -8]
The record TBigHeader corresponds to the single Fortran read/write block. It explicitly contains the elds BHSize and BHSize with the values of the block size added by the Fortran IO operators. This speci cation discloses the peculiarities of the Fortran system library and makes them evident to the programmers in any programming language. And it is easy to understand this human-readable format speci cation even without any prior knowledge about the language FlexT. So, the text of speci cation itself can help the future digital archaeologists to parse the MM5 les. 7 7.1
The format speci cations are required to write a correct program, that should
work with the les of the format. Because the FlexT language data types look
similar to that of imperative languages, it is possible to immediately use some
parts of speci cation to declare the data types, constants, and so on, which are
required to write the data processing code. Anyway the process of writing the
code manually is still time-consuming and error-prone. So, we have implemented
the code generator, which can automatically produce the data reading code in
imperative languages from the FlexT speci cations. By its expressive power the
FlexT language outperforms the other projects developing the binary format
speci cations, so the task of code generation for the FlexT speci cations is rather
nontrivial, because it is required to reconstruct all the features in the generated
code. By now we have implemented the code generation for the most widely used
FlexT data types, but some complex types are not supported yet. The main
principles and algorithms we use for code generation from FlexT speci cations
were considered in [
The Listing 1.5 contains an excerpt from the ESRI Shape le format speci cation { a data type TArcData de nition. The Listing 1.6 shows the code of the method for getting the eld T from the accessor (i.e. the auxiliary class generated for reading the complex data type) of the data type of the eld TArcData.Points. The code uses the values from the array Parts to determine the number of elements in the array T, and the if operator was generated for the expression of the TXPointTbl parameter.
Listing 1.5: FlexT speci cation of polygon/polyline data in Shape le format TArcData struc
TBBox BBox long NumParts long NumPoints array [@. NumParts ] of long Parts array [@. NumParts ] of struc
TXPointTbl (( @@@ . Parts [@ :#+1] exc @@@ . NumPoints ) -@@@ . Parts [@ :#]) T ends Points ends 7.2
The rst thing any programmer will want to do after generating a data reader is to test whether it works well. To perform the test it is required to write an application, which will use the data reader somehow. The most obvious and illustrative task here is to print using the data reader. Having manually created several test programs of this kind, we found that this process is rather tedious and should be automated. So, we have developed the algorithm, which automatically generates the test code. The test program generated together with the data reader makes it possible to immediately check the reader. Of no less importance is the fact that the source code of the program demonstrates the main patterns of data access using the reader. The Listing 1.7 shows some fragments of the test application code automatically generated for the ESRI Shape le format.
Listing 1.6: Generated Pascal code, which provides accessor for the eld T function TTArcData_Sub1Accessor .T: TTXPointTblAccessor ; var i0 : Integer ; ndx0 : Integer ; begin if not Assigned ( FT ) then begin ndx0 := Index +1; if ( ndx0 >=0) and ( ndx0 < TTArcDataAccessor ( TTArcData_Sub2Accessor ( Parent ).
Parent ). Parts . Count ) then i0 := TTArcDataAccessor ( TTArcData_Sub2Accessor ( Parent ). Parent ). Parts .
Fetch ( ndx0 ) else i0 := TTArcDataAccessor ( TTArcData_Sub2Accessor ( Parent ). Parent ).
NumPoints ; FT := TTXPointTblAccessor . Create ( Self ,0 ,0 ,i0
TTArcDataAccessor ( TTArcData_Sub2Accessor ( Parent ). Parent ). Parts . Fetch (
Index )); Listing 1.7: Fragments of the test application code in C++, immediate write style std :: unique_ptr < TSHPReader > must_free_Reader ( new TSHPReader ( FN )); Reader = must_free_Reader . get () ; if (! AssertTShpHeader ( Reader -> Hdr () , Reader ))
exit (2) ; cout <<" Hdr :" << endl ; cout << sIndent <<" Magic : " <<Reader -> Hdr () -> Magic . Value () << endl ; ... cout << sIndent <<" FileLength : " << Reader -> Hdr () -> FileLength . Value () << endl ; cout << sIndent <<" Ver : " <<Reader -> Hdr () ->Ver << endl ; ... cout <<" Tbl :" << endl ; for (i =0; i < Reader -> Tbl () -> Count () ; i ++) {
V = Reader -> Tbl () -> Fetch (i); cout << sIndent <<"[" <<i <<" ]: " << endl ; cout << sIndent <<" RecNo : " <<V -> RecNo () << endl ; cout << sIndent <<" Len : " <<V -> Len () << endl ; if (!V -> Data () -> GetAssert () )
exit (2) ; cout << sIndent <<" Data :" << endl ; cout << sIndent <<" ST : " << TShapeTypeToStr (V -> Data () ->ST () ) << endl ; cout << sIndent <<" SD :" << endl ; switch ( ( TShapeRecData_Sub0_Case )V -> Data () ->SD () -> hCase () ) { case hcPoint : cout << sIndent <<" Point :" << endl ; cout << sIndent <<"X: " <<V -> Data () ->SD () -> cPoint () ->X << endl ; cout << sIndent <<"Y: " <<V -> Data () ->SD () -> cPoint () ->Y << endl ; break ; ... case hcMultiPointZ : cout << sIndent <<" MultiPointZ :" << endl ; ... cout << sIndent <<" Points :" << endl ; for ( i13 =0; i13 <V -> Data () ->SD () -> cMultiPointZ () ->A () -> Points () -> Count () ; i13 ++) { V13 = V -> Data () ->SD () -> cMultiPointZ () ->A () -> Points () -> Fetch ( i13 ); cout << sIndent <<"[" <<i13 <<" ]: " << endl ; cout << sIndent <<"X: " <<V13 ->X << endl ; cout << sIndent <<"Y: " <<V13 ->Y << endl ; }
We have considered the possible options, which should be examined when selecting a le format for scienti c data representation.
The formal speci cations of binary le formats, especially for the custom ones, are very important, because the natural language speci cations are ambiguous, and it may be hard to fetch the data format information from the source code (the MM5 le format case demonstrates it very well).
The language FlexT is designed to write compact, human-readable and powerful speci cations, which can be used to check the correctness of data and resolve the ambiguities in the understanding of the other kinds of information about le formats.
It is also possible to generate from the FlexT speci cation the data reading code and the code of the application, that can immediately test the generated reader by printing the whole content of a binary le according to the speci cation using the reader. The current level of capabilities of the code generator is well characterized by the fact that it has successfully produced a full-featured code for reading data for the well-known in the GIS community Shape le format. The FlexT speci cation of the Shape format takes approximately 180 lines of code. The code generator have produced 1570 lines of the reader code, and 375 lines of the test program. The algorithm described in the present paper has also been used for the generation of the data readers for the scienti c le formats of the experiments in the TAIGA observatory.