DNA sequencing is the process of determining the order of nucleotides in DNA. The rapid speed of sequencing attained with modern DNA sequencing technology has been instrumental in the sequencing of complete DNA sequences, including the human genome. Nevertheless it is a sensitive data which needs safe but ecfiient storage methods. The goal in this research was to analyze diferent models and algorithms to determine which is the most applicable for storage, and query considering the need of user permissions, and encryption.
Due to cheaper methods it is possible for everyone to keep their own DNA on their own device. However these are sensitive data which needs to be secured. Genomes are stored mostly in text files. The size of these files could be even 3GB or more depending on the analyzed species. Secured data management is not solved in these files. By using database managers, the levels of permissions can be easily managed, because many database manager systems have built-in security and encryption possibility.
Previous researches are also available about using database manager systems for
storing genotypic data, describing and comparing multiple techniques [
In this paper the authors construct a DNA relational database with a simple
data model in which one DNA molecule stores one piece of data, [
3.1. DNA Deoxyribonucleic acid (DNA) is a molecule composed of two chains that coil around each other to form a double helix carrying genetic instructions. Nucleic acids are one of the four major types of macromolecules that are essential for all known forms of life. Each nucleotide is composed of one of four nitrogen-containing nucleobases (cytosine [C], guanine [G], adenine [A] or thymine [T]). DNA sequencing is the process of determining the nucleic acid sequence. In this paper we worked with these sequences.
Firstly we created a database in local host, followed by a database table. We split the sequences into 1680-character-long pieces and uploaded many diferent DNA sequences into the table.
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
seqPosition INT NOT NULL,
sequence TEXT NOT NULL
);
Only with this structure, a simple pattern searching SQL query took about 5-10
minutes in MySQL [
where seq like "%GACCGCGGCGCCGAGCGGCAGCCGCGCCGGCCCGGAC%" and id=101; 1 row(s) returned 550.953 sec / 0.000 sec
We did the same in PostgreSQL.
id SERIAL PRIMARY KEY NOT NULL,
seqPosition INT NOT NULL,
sequence TEXT NOT NULL
);
However in PostgreSQL [
A B-tree index can be used for column comparisons in expressions that use the
=, >, >=, <, <=, or BETWEEN operators. The index also can be used for
LIKE comparisons if the argument to LIKE is a constant string that does not start
with a wildcard character. MySQL uses the Turbo Boyer-Moore [
SELECT * FROM dna where pos=111; 10 row(s) returned 0.110 sec / 0.015 sec
In bioinformatics, k-mers are subsequences of length k contained within a biological sequence. Primarily used within the context of computational genomics and sequence analysis. Usually, the term k-mer refers to all of a sequence’s subsequences of length k, such that the sequence AGAT would have four monomers (A, G, A, and T), three 2-mers (AG, GA, AT), two 3-mers (AGA and GAT) and one 4-mer (AGAT). The maximum number of variations of the possible substrings are 4 .
We developed a program in python to determine the number of k-mers. With reading the file line by line it counts the occurring nucleotid variations.
We examined human, hominoidea, plant and bacteria dna sequence too. The next diagram shows the diference in percentage between each of the genom 3-mers self-compared value of occurrence.
We had an idea to use multiprocessing to increase the program eficiency. The basic idea was that if we split the file into smaller pieces, (for example four equal sized files) and if we add each piece into a thread, it should reduce the runtime. In python, multiprocessing is a package that supports spawning processes using an API similar to the threading module. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.
The multiprocessing module also introduces APIs which do not have analogs in the threading module. A prime example of this is the Pool object which ofers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism).
We get the following results without and with multiprocessing: The runtime with a 3gb sized dna sequence is more than 24 minutes (1440.026 sec) without parallelism.
The runtime with the same, but divided file is less than 10 minutes (585.83 sec) with parallelism. This means a 59% performance improvement.
For a better representation we tested our program on a 820MB sized DNA sequence. At first we did not use multiprocessing, then split the file into 3/6/11 equal pieces and used multiprocessing for our calculation. The following figure represents the taken time by each run.
The Lempel–Ziv–Markov chain algorithm (LZMA) is an algorithm used to perform
lossless data compression. This algorithm uses a dictionary compression scheme
somewhat similar to the LZ77 algorithm published by Abraham Lempel and Jacob
Ziv in 1977. It is used by 7zip nowadays [
In discussing security, it is necessary to consider fully protecting the entire server host (not just the server) against all types of applicable attacks: eavesdropping, altering, playback, and denial of service. Both MySQL and PostgreSQL have the option to create and add privileges with diferent roles to the database. Which provides the opportunity to determine which user could access the database and which could edit it. Every fresh MySql and PostgreSql system has an already defined role, which is always the superuser. 6.1. MySQL The privilege system is the most basic security feature MySQL has to ofer. It allows defining who can access database. Each user may be assigned a number of privileges. They define what each user can do once they have been authorized to connect. The privileges can be either global, granting specific right without restrictions, or limited to a specific object such as database or table.
MySQL implements an SSL library to provide connection encryption. All endpoints – server and clients – require their own set of private keys and certificates to be able to use encryption.
It oefrs a number of cryptographic [
Same as in MySQL, PostgreSQL also use user roles to manage access privileges. Roles can act as users, groups or both. Roles can own database objects and can manage the permissions on these objects. Both of the discussed database management systems the owner of a created object is the user who created it.
PostgreSQL has native support for SSL connections to encrypt client/server communications for increased security.
We would like to mention that it also has the possibility to encrypt data, both one way (e.g passwords) or two way(e.g credit cards). For one way encryption, the crypt function packaged in pgcrypto provides an added level of security above the md5 way. We have several ways to accomplish the two way encryption, these implements the encryption part of the OpenPGP(RFC 4880) standard. However we did not try these methods out.
We created an environment where we can manage the user privileges and store the data securely and encrypted. The database manager gives us many options to easily add new users and create user hierarchy, from the principle of least privilege to a user with admin authority. We also managed to implement the first basic version of our processing tool that gives us the opportunity to compare diferent species genomes. Even started to test the multi-threaded version of the program for faster process of the genomes.
In the future we want to try new data storage options and compare our results with other database managers, e.g NoSQL. As our reviewers insisted it would be interesting to examine another compression algorithms, and compare them to our results. Moreover, the processing optimization and performance improvement is in our main scope. Implementing a user friendly interface with options to import genomic data, which also capable of creating diferent graphs and reports based on the results is also one of our main goals.