Once upon a time programming was done just writing commands of a programming language in a proper order, but currently software is created using libraries, API-s (Application Programming Interface), frameworks, Dockers, Kubernetes etc. Libraries load other libraries, API-s call other API-s and as a result seemingly short and simple programs may have amazing depth of code and complexity, what causes for programmers many problems, especially for students. For interpreted code this depth could be (approximately) measured with the ratio of the visual code vs code in libraries. It is shown that the number of LOC (Lines Of Code) in invisible code - code in libraries, modules, API-s etc. is even in small practical programs thousands-millions times greater than the number of lines in the visible code. Innovations (cloud computing, multicore CPU-s etc.) cause introduction of new libraries and modules which enable use of new possibilities in existing software ecosystem, but also introduce bigger and bigger amounts of invisible code. During the pandemic grow student's use of WWW tutorials, but abundance of code in programming examples/tutorials on WWW is sometimes unnecessary, caused by obsolete or unneeded packages and libraries and sometimes also by desire to earn on adverting (un-needed, but popular) code packages for high-paying customers. In the following are analyzed some Python 3 and JavaScript examples.
With exhaustion of Earth's natural resources humanity is constantly increasing scientific research, which nowadays is based on world-wide programming and communication ecosystem. Once upon a time programming was done just writing commands of a programming language in a proper order. Modern software is created using libraries, API-s (Application Programming Interface), selecting commands from menus in frameworks, packed with Dockers, Kubernetes etc. Libraries load other libraries, API-s call other API-s and as a result programs have amazing depth of code. For notcompiled (interpreted code) that depth could be (approximately) measured with the ratio of the visual code vs code in libraries - the total number of lines used in visible code is thousands-millions times greater than the number of lines in the visible code. This abundance of code is often unnecessary, caused by obsolete or unneeded packages and libraries, but sometimes also by desire to earn on adverting (un-needed, but popular) code packages for high-paying customers. This creates many difficulties for students, who often cannot understand the programming task but try to solve it using 'top-down' programming – first import all libraries and packages, what have been used in some previous project and then just start googling trying to get from Internet sources as many snippets (which may be useful) as possible. Such 'top-down' programmers often do never become a 'real' programmers, but continue creating new packages and libraries, which 'just work' and further contaminate the software ecosystem. 1.1.
With exhaustion of Earth's natural resources humanity is constantly increasing scientific
research. Spending on sciences has increased in last four years worldwide by 19% and the
number of scientists grew by 13.7% reaching currently already to 8.8 million [
Once upon a time programming was done just writing commands of a programming language in a proper order, but these times are long gone. While e.g. the language Python 3.7 has only 35 keywords and 59 symbols, the real power of the language is in its libraries and modules, e.g. in the computer used to write this text are installed 361 Python 3.7 modules. Currently programming is done with modules, libraries, API-s, Dockers, Kubernetes etc. Every module presents some functionality what is (usually) far more complex than a single command of a programming language. Modules and API-s call other modules, libraries, API-s, what makes modern programs complex hierarchical systems, where the visible part (code what is inspect able in the program) is a minuscule part of the invisible in final code, hidden in libraries, API-s etc. Modern software is like icebergs in Antarctic – the main part is invisible.
In the following are presented some (simple) examples showing this 'iceberg-style' programming and an analyse of the reasons for programs 'blow-up', since the depth/complexity is causing many problems for students, but also for the whole modern world-wide software ecosystem. To illustrate the issue here is introduced program's visibility index: ratio of visible code vs code loaded in libraries/modules. 2.1.
The following graph depicts development of the SQAMIA conference over 2012..2019 in terms of number of pages in the conference proceedings; the yellow line is a second-order (square) approximation of the process.
Growth of the SQAMIA conference –by number of pages in Proceedings (the yellow line – second-order approximation) The above graph was produced by 17 lines (including commentaries – they are important to make a program easier to understand for humans) of Python 3 code : 1. 2. 3. 4. 5. 6. 7. 8.
9. 10. 11. 12. 13. 14. 15. 16. 17.
import numpy as np
import matplotlib.pyplot as plt
# x axis values
x_sqamia = range(2012,2020)
# corresponding y axis values
y_sqamia = [18,108,114,80,109,192,154,176] # Sqamia
plt.plot(x_sqamia, y_sqamia)
# naming the x axis
plt.xlabel('year')
# naming the y axis
plt.ylabel('pages in Proceedings ')
# giving to graph a title
plt.title("Growth of the Sqamia conference")
# calculating coefficients of the approximation line
a2, a1, a0 = np.polyfit(x_sqamia, y_sqamia, 2)
plt.plot(x_sqamia, a2*x_sqamia*x_sqamia + a1*x_sqamia + a0)
plt.show()
This text is understandable also for non-programmers. But programmers looking the above code
will wonder – no loops ??? The 'classical' program, i.e. using only commands from a programming
language without any libraries would have to use several loops – using the x_sqamia iterator,
projecting values to axes, solving the system of equations for creating square approximation in
order to calculate correlation parameters a2,a1,a0. Totally is hidden the AI (Artificial
Intelligence) – calculating the size and visually pleasing placement of axes and their scales.
All the functionality of the above program is hidden in the two explicitly called libraries numpy,
matplotlib and many others, called from these two. The pyinstaller [
The 'blow-up' and then 'missing' together make programming (at least for students) a nondeterministic exercise, thus difficult/impossible to learn) is in (this case mainly) caused by modern exceedingly diverse executing environments – Windows (32/64 bit), MacOS, GNU/Linux, AIX, Solaris, FreeBSD and OpenBSD, IBM-style mainframes, diverse lot of mobile devices, which all have some differences, thus for them are called different modules. Many of them are specific to current environment, e.g. ntpath exists only on Windows, but the posixpath exists only on Posix/Unix/Mac systems and is missing in Windows environment (but the compiler is still trying to load it). This creates (in every environment) many modules which compiler marks as 'missing'. There are several types of missing modules: optional - imported within a try-except-statement delayed - imported from within a function conditional - imported within an if-statement top-level - imported at the top-level; they are most important and may cause the program misfunction The same module may fall into several categories, e.g. for this program the missing module org.python was called by module copy (optional), xml.sax (delayed, conditional) and by module setuptools.sandbox (conditional).
In Windows 64bit environment was the PyInstaller able to access only 41 packages (ca 25%). The PyInstaller (in the same environment) even searched for more and did not find 238 modules, but it also warned "This does not necessarily mean this module is required for running you program" – missing modules are required in some other operating environment, e.g. the list of modules contained 125 encodings: encodings.ascii • encodings.base64_codec • encodings.big5 • encodings.big5hkscs • encodings.bz2_codec • encodings.charmap • encodings.cp037 • encodings.cp1006 • encodings.cp1026 … (missing were codecs for Egypt, Cuba, UAR etc, ).
All together were among missing modules 150 optional, 106 delayed, 94 conditional and 313
toplevel. Large number of missing top-level modules does not mean, that the code can not be
executed – a missing top-level module were called from a module, which itself is optional.
But the highest level of program's execution is the OS (Operating System, often called also as
Operating Environments) [
0.0000979431 It is rather difficult (e.g. for students) to understand fully a program when they can inspect only 0.0000979431-th size part of it – e.g. what should be done with the program for the above graph in order to include two missing years 2020, 2021 to it ? It turns out, that a quite new concept – data filtering – is needed, and from inspecting the above program (as an educational simple example for producing data graphs) this is not easy to produce.
Unnecessary lines of code may be also inserted deliberately – either by ignorance of authors or by desire to advertise some popular library.
A typical program for beginners (using any programming language/system) is to produce a
program to create a visible display "Hello World". In html5 for this are needed only some lines
of html5 code (in basic mode).
But the programming powerhouse Facebook (currently Meta) needs thousands of such files (with
largely increased functionality for tracking users). Thus they created a Javascript+Html5 library
react.js advertising it with "React makes it painless to create interactive User Interfaces." [
<script src="https://unpkg.com/react@18/umd/react.development.js" crossorigin></script>
<script src="https://unpkg.com/react-dom@18/umd/react-dom.development.js" crossorigin></script>
<script src="https://unpkg.com/@babel/standalone/babel.min.js"></script> </head> <body> <div id="mydiv"></div> <script type="text/babel"> function Hello() {
return <h1>Hello World!</h1>; }
ReactDOM.render(<Hello />, document.getElementById('mydiv'))
</script>
</body>
</html>
This script has some strange features:
(lines 11:.13) - why for displaying a static string "Hello World!" (without any variables) is
created a function in an environment ReactDOM – the html5 language (without any
Javascript) can perfectly show a static string in a pre-defined div?
(line 14) - why is for displaying/rendering the string used a proprietary function
"ReactDim.render" – in an html-document rendering anything on screen is always done
by browser, a script can only use the available CSS functions (CSS is not used)?
(lines 4..6) - why for creating this functionality are loaded three Javascript libraries with
altogether 73571+3358+26275 = 103204 lines (after un-minifying) of JavaScript code,
which are not needed at all?
(lines 4..5) - the crossorigin scripting has been for years considered extremely unsafe
[
visibility
0.000164722 Using plain html5 would return an easily understandable script, producing exactly the same page: <!DOCTYPE html> <html> <head>
<title> Hello !</title> </head> <body>
<div id="mydiv">Hello World!</div> </body> </html>
visibility 1.0
The main burden – 73571 lines of code (after un-minifying) comes from the Babel library for
interpreting Ecmascript ES6 (the official name for Javascript). All major browsers can handle
ES6 already for last five years [
At least for a student this is not "painless", but (very) painful.
Our computing environments are constantly changing and their complexity is rapidly growing.
Every day are uploaded/modified thousands of new libraries/API-s which are also used in new
programs, e.g. in one week - August 3 – August 10, 2022 in the Github repository for node.js
"1,479 files have changed and there have been 34,550 additions and 6,640 deletions." [
The Tensorflow example from Google [
The search for Python 3.7 modules was able to find 55 of 158 modules with 24465 lines of code. The Tensorflow example program had 52 lines of code, thus in this case
visibility 0.002125485
But here the main growth in complexity comes from the program.
Definition of the Tensorflow model contains only 6 lines of code: def build_model(vocab_size, embedding_dim, rnn_units, batch_size): model = tf.keras.Sequential([tf.keras.layers.Embedding(vocab_size, embedding_dim,batch_input_shape=[batch_size, None]), tf.keras.layers.GRU(rnn_units,return_sequences=True,stateful=True,recurrent_init ializer='glorot_uniform'),tf.keras.layers.Dense(vocab_size)] return model But this definition creates a model with over 4 million parameters as shown in output for Python command model.summary():
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (128, None, 256) 43776 _________________________________________________________________ gru (GRU) (128, None, 1024) 3938304 _________________________________________________________________ dense (Dense) (128, None, 171) 175275 ================================================================= Total params: 4,157,355 Trainable params: 4,157,355
Non-trainable params: 0
Handling more than 4 million parameters is (very) slow in 'plain' Python, since Python uses only
a single CPU core and cannot parallelize computation. In most of modern computers are already
multicore CPU-s and their speed can be utilized with another library - bodo, which is [
Many (most) of the classical programming textbooks considered programming as a process of creating a linear artefact - putting programming language statements one-after-another in a proper order.
But already creators of the compilers for the first high-level language – Fortran, Cobol – understood that programs and their functionality (also creating the programs itself) are far from linearity and programs are always presented as trees – e.g. as the AST (Abstract Syntax Tree). Programming, human thought has never been linear, although many of created by ourselves tools (standards, texts, constitutions etc.) are trying to force to us this style of thinking. We are not linear. Programming has constantly becoming more and more non-linear hierarchical. Programmers still have to create linear text, but programs are including libraries, IDE-s, modules etc. which all make programs functionality (inner working) strongly multidimensional. Software development is not a smooth or simple process, many researchers are worried with its growing complexity, see e.g. [18]. Emily Freeman, head of DevOps at Amazon Web Services Inc characterized it "It’s a study in entropy, and it is not getting any more simple”[19]. Unfortunately we are still not aware how to teach to our non-linearly thinking students the proper non-linear thinking methods/styles, but they should (at least) be aware of tremendous dimensionality of our programs (and our thoughts).