=Paper=
{{Paper
|id=Vol-1755/181-189
|storemode=property
|title=Towards a Fully Cooperative Multi-Agent Reinforcement Learning based Media Access Control Protocol for Underwater Acoustic Wireless Sensor Networks
|pdfUrl=https://ceur-ws.org/Vol-1755/181-189.pdf
|volume=Vol-1755
|authors=Aliyu Ahmed,Jonathan G. Kolo,Mikail O. Olaniyi,James Agajo
|dblpUrl=https://dblp.org/rec/conf/cori/AhmedKOA16
}}
==Towards a Fully Cooperative Multi-Agent Reinforcement Learning based Media Access Control Protocol for Underwater Acoustic Wireless Sensor Networks==
https://ceur-ws.org/Vol-1755/181-189.pdf
Towards a Fully Cooperative Multi-Agent Reinforcement
Learning based Media Access Control Protocol for
Underwater Acoustic Wireless Sensor Networks
Olaniyi O. Mikail
Ahmed Aliyu Computer Engineering Federal University of
Computer Engineering Federal University of Technology Minna, Nigeria
Technology Minna, Nigeria mikail.olaniyi@futminna.edu.ng
aliyu.ahmed@futminna.edu.ng
Agajo James
Kolo G. Jonathan Computer Engineering
Computer Engineering Federal University of Federal University of Technology Minna, Nigeria
Technology Minna, Nigeria james.agajo@futminna.edu.ng.com
jgkolo@futminna.edu.ng
ABSTRACT Keywords
Reinforcement Learning, MAC protocol, ALOHA, QoS, Self-
Underwater Acoustic Sensor Networks (UWASNs) has gain a organization, UWASN, Multi-Agent
widespread recognition recently due to some technological
break- through, and thus, beginning a new era of research in the
industry with potential for vast applications that are important 1. INTRODUCTION
to our livelihood. Despite all these potentials, deploying a Increased researches in WSNs has made a plethora of real life
reliable UWASNs based systems still remain very far from applications possible, particularly in underwater scenario.
perfect and there are only limited experimental trials at the UWASNs is recently becoming an important area of research
moment. This is due to challenges of reliabilty, QoS and energy with promising potential for various applications ranging from
efficiency, which is due to inherent characteristics of underwater oil and gas extraction (seismic imaging), pipeline
underwater acoustic channel. These pose signif- icant and infrastructure monitoring, marine life monitoring and
challenges for the design of network protocols, especially, the control, monitoring of underwater Carbon(IV)Oxide (CO2)
Media Access Control (MAC) protocol for UWASNs. Various storage facility, border control, Fish farming, freshwater
MAC protocols have been developed for UWASNs and some reservoirs management, Autonomous Underwater Vehicles
few adopted from Wireless Sensor Networks (WSNs). (AUVs), Naval Network centric warfare- mine reconnaissance
However, most of these protocols do not provide acceptable etc. to tsunami and seaquake early warning systems [1–4].
QoS in terms of delay, throughput, fairness and energy Despite all these promising applications, Underwater Sensor
efficiency. This paper presents a review of some of the Networks remain quite limited as compared to the terrestrial
prominent MAC protocols for UWASNs and adaptable WSNs Sensor Networks technologies. Thus, this makes underwater
based MAC protocols for UWASNs and propose a Fully operations limited to remotely controlled submersibles which
Cooperative Multi-Agent Reinforcement Learning based MAC are large, very costly and are almost temporarily deployed [1]
protocol for UWASNs. The proposed scheme will apply Multi- as com- pared to sensor network nodes which are relatively
Agent based Reinforcement Learning (RL) to ALOHA MAC cheaper and can be permanently deployed on the sea flow for
scheme to create a dynamic contention-free-like slotted MAC to real time communications.
aid nodes cooperation and interactions within themselves and Radio based communication for terrestrial sensor networks is
the underwater environment to significantly achieve “self- not suitable for underwater usage because of extremely limited
organization” and “self-adaptability” to changes in the propagation delay as current mote radios transmit between 50 to
environment which would provide means for coping with long 100cm and within 30-300 Hz of frequency underwater. The
and variable propagation delay, low data rates and energy implication is that extraordinary transmission power and very
efficiency and in turn can significantly improve the QoS of large antennas are required for deployment [1, 5]. Therefore,
UWASN systems by having better convergence time and establishing communication in UWSN effectively largely
Energy efficiency. depends on acoustic communications. However, Underwater
CCS Concepts Acoustic communications bring about new challenges due to
• Networks ➝ Network components ➝ Wireless access unique characteristics of under- water acoustic communication
points, base stations and infrastructure channels such as: High propagation delay caused by low speed
of acoustic signals (speed of sound is approximately 1500 m/s)
which is by 5 orders of magnitude slower than radio waves
(3x108m/s) for terrestrial Wireless Sensor Net- works (WSN)
[1, 3], low data rate (between 5-20Kb/s) due to limited channel
CoRI’16, Sept 7–9, 2016, Ibadan, Nigeria. bandwidth, high error rates, highly dynamic environment and
high energy consumption (typical consumption between 50 to
100 W) [3, 6–8].
UWASNs is made up of a large number of sensors deployed
underwater with capability to communicate via acoustic links.
181
Its then worthy to mention that special consideration need to be communication problems such as MAC layer protocols [10, 12]
taken with respect to channel modelling, medium access, and similar design strategies could be employed for developing
routing and other sensitive issues when designing UWASNs MAC protocol for UWASNs.
[9]. For a successful UWASNs design and deployment, Media The rest of the paper is organized as follows. Findings on the
Access Control (MAC) protocol is very important. It is a Layer review of some prominent MAC protocols are presented in
2 (Data Link) protocol which define how channels are accessed section 2, while, section 3 presents design challenges of MAC
for efficient and successful communication. Various MAC protocols for UWASNs. Section 4 presents the overview of the
protocols have been proposed for the terrestrial WSNs to proposed MAC protocol and section 5 concludes the paper.
provide significant improvement on energy efficiency and
throughput performance [10], however, these schemes cannot
be directly adopted for UWASNs due to the afore- mentioned 2. REVIEW OF SOME PROMINENT MAC
unique characteristics of underwater environment [8]. MAC
schemes can be classified into contention based and contention PROTOCOLS
free schemes. This section reviews some of the important MAC protocols that
Contention based schemes mostly employ carrier sense (CS) have been recently proposed for WSNs and UWASNs to
techniques such as Carrier Sense Multiple Access (CSMA) address the pertinent problems of energy efficiency, throughput
proto- cols and Random access techniques such as ALOHA and de- lay. Contention-based and contention-free schemes
protocols. In CSMA schemes, exchange of control packets have been considered for UWASNs. For contention-free
causes long packet delay due to long preamble in real acoustic schemes, it is already established that Frequency Division
modems which in- creases packet collisions and control packets Multiple Access (FDMA) is not suitable as a result of limited
have long preamble and load which degrade network bandwidth of acoustic channel [3]. Time Division Multiple
performance, increased energy consumption and hidden Access (TDMA), another form of contention-free schemes, has
terminal problem [10], thus not suitable for applications such as also been studied but its efficiency is limited by strict
UWASNs requiring low delay. Also, pure ALOHA relies on synchronization and large guard time [13]. More- over, Code
packet retransmission for reliable data delivery. This may be Division Multiple Access (CDMA) (another contention- free
suitable for terrestrial WSNs because of its simplicity, but may scheme) is known for high autocorrelation and low cross-
not be appropriate for UWASNs, because, packet re- correlation properties to minimize interference among users
transmission can quickly saturate the network due to limited which make its design for UWASNs very complex [6].
channel capacity [3]. Contention-free access schemes (TDMA, On the contrary, ALOHA [14–17] and CSMA [8, 18–21]
FDMA, CDMA, etc.) use slot-scheduling techniques for media (contention-based schemes) have recently received significant
access. This could have been the right candidate for UWASNs, consideration for UWASNs owing to their simplicity and good
because of low collision rates, but they are rather too complex throughput [3]. A contention-based scheme that depends on
for literally speak- ing, primitive underwater sensor hand- shake called propagation delay tolerant collision
technologies. Another problems of contention-free schemes are avoidance proto- col (PCAP) was proposed by [22], it allows
high system overhead, high propa- gation delay, strict time the sender to transmit another data packet or perform handshake
synchronization and not flexible to changes in the number of for the next queued data packet while waiting for the clear to
nodes. Therefore, for a successful UWASNs de- ployment to send (CTS) packet, thereby, favourably utilizing long
solve unique challenges earlier discussed, possible solutions propagation delay. But it requires strict clock synchronization
are: To design a new sleep and wake-up schemes from scratch which makes it complex for UWASNs. Moreover, [23]
for UWASNs, reduce control packet exchange or to com- bine proposed distance aware collision avoidance proto- col
contention-based and schedule-based schemes. (DACAP), a handshake based protocol which creates a waiting
Thus, there is need for a much simple MAC protocols scheme window of time (based on the distance between the sender and
that will be “self-organized” and “self-adaptive” after been de- the receiver node) for the sender after receiving CTS to allow
ployed and can nonetheless provide energy efficient for in- tending receiver to receive warning to avoid collision.
communica- tion, good throughput and acceptable delay. This The control packets and long preamble can cause long packet
research proposes development of an intelligent ALOHA based delay and in turn reduce the network performance.
MAC protocol for UWASNs. The research will explore the use In addition, Tone signals have also been employed in
of machine learning techniques, specifically, Multi-Agent based contention- based approaches as evidenced in [24] called T-
Reinforcement Learn- ing (RL) to assist with nodes cooperation Lohi protocol. In this approach, short tones are transmitted by
and interactions with the environment to achieve “self- nodes to alert neigh- boring nodes of intending transmissions to
organization and adaptation”. This would provide means for detect the channel con- tenders before sending data. The time
coping with long and variable propagation delay, low data rates instances of arrival of such tones at various nodes varies for
and energy efficiency. The ability of nodes to learn from their different nodes due to different propagation delays. Thus, nodes
interactions with the wireless environment pro- vides scope to only transmit data whenever tone signals are not received,
significantly enhance their ability to self-organize and adapt to otherwise, a calculated back-off interval is activated and back-
changes in the environment. off performed. The downside is that a special Wake-up tones
Reinforcement Learning (RL) is an approach or technique of receiver hardware is required by T-Lohi nodes to be able to
Machine Learning (ML) that makes use of agent(s) to learn detect tones using low energy consumption.
effective strategies through trial-and-error interactions with the A Handshake based Ordered Scheduling MAC (HOSM) has
dynamic environment, take future actions which are determined also been proposed by [25] for underwater acoustic LANs. In
by scalar reward based on prior experience to transit from initial this technique, Channel reservation phase is firstly created by
state to a new one with the ultimate goal of maximizing the the intending nodes to transmit data packets and a calculated
cumulative re- ward along the course of interaction [11]. It has ordered lists are used by the nodes for data transmission. The
found established usage in Artificial Intelligence researches key idea of this technique is to utilize the information of
such as robotics, controls and automation, and recently in propagation delay to adjust the time instant of control packets
182
transmission to reduce collisions of control packets to achieve consumption. Slotted-FAMA, due to frequent exchanges of
high throughput, low de- lay spatial fairness. However, energy RTS/CTS, reduces the channel utilization, results in poor
is not given fair attention and could be adaptable to a traffic performance such as throughput, end-to-end delay in UWASN
with different priorities. Furthermore, [26] proposed an hybrid characterized by long propagation delay, low bandwidth and
scheme called hybrid reservation-based MAC (HRMAC) high bit error rate. R-MAC schedules the transmissions of
protocol where the nodes use declaration to re- serve channel control pack- ets and data packets to avoid data packet
and collision of control packets is reduced by spec- trum collision. This creates serious overhead issues and further
spreading technology. The good news is that many nodes with dampens its performance in UWASNs.
intending data packet transmission can reserve the channel As a result, [31] proposed Multi-session FAMA (M-FAMA) for
simul- taneously but transmit their data in a given order. This UWASNs to solve the problem of bandwidth limitation
significantly improves the channel efficiency as compared to associated with Slotted-FAMA. It takes the advantage of the
typical MAC pro- tocols for UWASNs. But the scheme cannot propagation de- lay information of the neighboring nodes and
be extended to general multi-hop underwater acoustic networks. expected transmis- sion schedules to initiate multiple sessions
In addition, very recently, [3] proposed an adaptive retransmis- simultaneously. In this scheme, the inherent problem of fairness
sion scheme for contention-based MAC Protocols for across multiple contend- ing sources is solved by introducing an
Underwater Acoustic Sensor Networks. This try to address the algorithm that balances the bandwidth. Compared with its
problem of low performance (low Packet Delivery Ratio (PDR) predecessor, Slotted-FAMA, it has the advantages of
and high End-to- End (E2E) delay) associated with contention- temporal/spatial reuse and collision avoidance to some degree.
based MAC proto- cols for UWASNs by using adaptive However, due to large number of control packets in order to
retransmission scheme (ARS) to dynamically selects an optimal initiate multiple sessions, M-FAMA performs low in terms of
value of the maximum number of retransmissions, such that the energy efficiency as compared to most channel reservation pro-
successful delivery probability of a packet is maximized for a tocols. Also, in bursty-traffics, RTS/CTS handshake degrades
given network load. ARS ALOHA and ARS CSMA its performance in terms of throughput. Also, throughput is
significantly improve network performance in terms of PDR affected significantly when deployed on highly mobile nodes as
and E2E delay, however, it could not extend ARS to support a result of increase in failure of channel reservation due to
different performance requirements in UWASNs such that each changing network topology and it is not developed with self-
node can adapt its transmissions to satisfy a specific organization and adap- tation capabilities. Moreover, the
performance requirement from applications or users. performance in terms of delay is poor, because, the RTS/CTS
ALOHA-Q, an intelligent based protocol is proposed by [10] handshake processes keep the propa- gation delay at high
for terrestrial WSNs. This applies reinforcement learning to values and this problem is not solved by the multiple session
frame based ALOHA to select slots intelligently with mechanism.
capabilities to mi- grate from random access to perfect In contrast, the technique in [27] introduces probability
scheduling using Q-learning technique. It utilizes a simple function- based code assignment algorithm to reduce code
learning process and has much lower complexity and collision; mean- while, without RTS/CTS handshake, state-
overheads. This greatly improve QoS of WSN in terms of based channel access mechanism maximizes the channel
energy efficiency, delay and throughput as compared with utilization. It also supports con- current transmissions between
slotted ALOHA, S-MAC and Z-MAC. However, all of the nodes by adopting CDMA communication
overestimating frame size can generate unused slots and technology, which improves the network perfor- mance of end-
underestimating frame size can introduce packet collisions to-end delay, energy-consumption, network through- put and
which both may affect channel performance. In the same vain, delivery ratio. However, CDMA is not practical because it is
there is concern about the ability of the network to adapt to difficult to assign pseudo-random codes among large numbers
different densities of node deployment without requiring a fixed of sensor nodes, thus, it is not scalable as evidenced in the
and pre-estimated frame size configura- tion. Although, this throughput becoming poor with increase in network load. Also,
technique has promising performance, it was designed for it did not con- sider self-organization and self-adaptability
terrestrial WSNs. This can be adapted for UWASNs by careful issues, which are very important when designing MAC for
modifications to suit the challenges of limited channel ca- UWASNs.
pacity, long propagation delay and energy efficiency for DTMAC was proposed by [32] for UWASN and was based on
underwater acoustic communications. Most importantly, frame
size estimation could further be tuned to reduce packet distributed coupon collection algorithm that allocate a certain
collisions to acceptable val- ues for UWASNs. num- ber of times an intending transmitting nodes will repeat a
[27] proposed a hierarchical and distributed code assignment trans- mission which is a function of transmission probability
algorithm based on divisive probability function which can that only requires the number of neighboring nodes and not the
avoid conflict between spread codes with high probability, and
provide a state-based MAC protocol for UWASNs. The exact net- work topology so as to improve the network
technique tries to eliminate the RTS/CTS handshake prominent throughput in burst short-packet traffic deployments and
in POCA-CDMA (Path oriented Code Assignment) MAC overcome the challenges of long propagation delay and swarm
protocol [28], Slotted- FAMA [29] and R-MAC [30]. POCA-
CDMA MAC adopts the mobility. The technique tried to map throughput-optimal value
CDMA technology to make the sink receive packets from with the successful transmission probability as a turning
multi- ple neighbors at the same time. It achieves higher parameter to avoid the use of channel reser- vation and
throughput, but suffers from the hidden terminal effects and low handshake mechanisms in order to curb the problem of
energy efficiency as a result of higher control packets overhead.
propagation delay. It also tried to solve the problem of space
That may be toler- able in terrestrial WSNs, but becomes
serious underperformance issue in UWSNs as a result of low unfairness by eliminating transmission distance factor. All these
bandwidth, long propagation de- lay and high energy considerably improve the performance of DTMAC in
183
UWASNs. However, DTAMC is designed with the goal of slot selection for optimal data transmission which will bring
short data packets transmission, thus it makes an assumption of about su- perior channel performance as against low throughput
a single-hop target network. In addition, DTMAC protocol may associated with ALOHA based MAC protocols.
be suitable for high bandwidth demand deployments, but pay
less attention on success- ful packet transmission probability 3. DESIGN CHALLENGES OF MAC
which means that the through- put in this sense is affected.
PROTOCOLS FOR UWASNs
Another weakness of this protocol is that it was only designed
for short data packets. Also, scalability is an associated problem Media Access Control (MAC) protocol is a Layer 2 (Data Link)
as performances in terms of throughput, delay and energy protocol that defines how channels are accessed for efficient
efficiency degrade with increase in node density, this is and successful data packets communications. It is the backbone
because, the technique only takes into consideration the de- lay of data packet transmission between sensor nodes in UWASNs
factor. and QoS and Energy efficiency of UWASNs largely depends
on it [37]. Thus, it is important to carry out comprehensive
Intelligent protocols have also been recently employed in wire-
study on the design of MAC protocols for underwater in order
less communications including Wireless Sensor Networks as
to have an effective communication between the sensor nodes,
ev- idenced in [10, 33, 34], also cooperative communications
acceptable QoS and reasonable energy efficiency.
have shown to have significant effect solving the problem of
multiple fading effects in wireless networks, and thereby
3.1 Why not Radio Frequency Communication?
improve QoS of the network in terms of adaptivity, reliability,
Radio waves are strongly attenuated in underwater water
data throughput and net- work lifetime. [33] investigated the
enviro- ments and as a result has limited propagation ranges,
use of cooperative communi- cations based on Multi-Agent
e.g. in sea water, just up to 10 meters [38]. The implication is
Reinforcement Learning (MRL-CC) algorithm on multi-hop
that to cover large distances, large antennae and high
mesh cooperative communication mecha- nism in order to
transmitter powers are required. This is costly and non-
achieve QoS provisioning in terrestrial WSNs. Is- sues of
practical. Long-wave radio, however, can be used for short
Spatial and time diversity gains in wireless networks using
distances 1-8kbps at 122kHz carrier for ranges up to 6-10
cooperative communications have also been investigated
meters [38]. Propagation rate is also low due to low-bandwidth
recently in [35, 36]. This strategy complements the M-FAMA
modems which are currently available conditions that the range
approach by intelligently creating multiple transmission
becomes appreciable, approximately 100 meters [38], and only
sessions. Owing to the broadcast nature of the wireless (RF)
with high bandwidth modems can several Mbps of data rate be
medium and spatial distri- bution of sensor nodes, cooperative
achieved. However, developing high band- width modems for
communications can be used to improve the network
underwater communications is still a open re- search area, and
performance of WSNs. This can also be further extended
there is need for transceiver alignment, thus only short-range
through a careful design to take advantage of un- derwater
connections of order 1-2m at 57.6kbps data rate are possible [4,
acoustic channel to develop a cooperative MAC protocol for
38]. This is not practical for numerous underwater ap-
Underwater Acoustic Sensor Networks.
plications. Thus, it is very clear that for effective
Performance comparison of some of the prominent MAC proto-
communication in underwater channel, acoustic communication
cols for Underwater Acoustic Sensor Networks are summarised
is the ultimate alter- native.
in Table 1 as shown:
From the comparative analysis summerised in Table 1, it can be 3.3 Characteristics of Acoustic channels
concluded that the current researches on MAC protocol de- sign Acoustic Communication is the only communication
for UWASNs strive to achieve optimal channel performance at technology that supports all required transmission ranges in
the cost of architectural complexity. Consequently, control underwater, it is cheaper and practical as compared to radio and
over- heads are increased and as a result, performs poorly in optical communication in underwater environment. However,
energy effi- ciency. There is need for consideration of MAC acoustic channels have some unique features that pose
protocols design for UWASNs that should make a trade-off challenges to effective communication in underwater
between energy effi- ciency and channel performance with environments. Some of these features are as follows:
respect to application area.
Very long and variable propagation delay as a result
This paper proposes a technique that apply Reinforcement of low speed of sound which is approximately 1500
Learning on framed ALOHA MAC protocol. This hope to take m/s, about 5 orders of magnitude slower than radio
the advantage of low architectural complexity and overheads waves (3x108 m/s) and the speed of sound varies
associated with ALOHA MAC protocol to provide intelligent
184
considerably with respect to temperature, pressure and ambient and man made noises. Noise in underwater
salinity, thus, it is depth dependent (1450-1540 m/s.) environment can be given as:
[37, 38]. Noise = Turb + Ship + Surf + Therm + Others
Bandwidth is severely limited as a result of (1) Where Turb is Turbulence, Ship is
attenuation and interactions with bottom and surface Shipping, Therm is Thermal and Others refer to man-
of the water body. Thus, the available bandwidth made, biological, ice, rain, seismic, etc
becomes transmission distance dependent. Data rate is noises.
also low as a result, about 100Kbps [37].
Path/propagation loss as a result of attenuation as a
Extensive multi-path arrivals/propagations cause result of de- crease of the sound intensity through the
Intersymbol interference (ISI) delay in hundreds of path from the sender to the receiver caused by
symbols, which can severely degrade acoustic absorption due to conversion of acoustic energy into
communication signal and also leads to high bit error heat, and it increases with distance and frequency.
rate.
Thus, The Transmission Loss (TL), is given as:
Channel Dynamism with respect to time and high −3
TL = SS + α × 10 (2)
Dopller spread (especially for horizontal
communication). It is also clear that one water body is
different from the other and different from itself at Where, SS is the Spherical spreading factor given as ss = 20 log r, r
different times, this makes channel tracking to be- is the range in meters and α is the attenuating factor put forward
come difficult. empirically by Thorp formula.
Dopller-shift ratio is of several orders higher than that 3.2 Why not Optical Communication?
of the RF channels which makes symbol Light is strongly scattered and absorbed underwater, drastically
synchronization difficult. limiting communication range [38]. It is only in very clear water
Noise which is caused by majorly two factors, the
Table 1. Performance Comparison of UWASNs MAC protocols
Author, Year MAC Technique Energy Efficiency Throughput Delay Channel utilization
Marcal et al, 2006 Slotted-FAMA MAC Handshake based Low Low High Low
Peng et al, 2007 R-MAC Schedule based Low High Moderate Moderate
Guangu et al, 2011 POCA-CDMA MAC Handshake based Low Moderate Low Moderate
Han et al, 2013 M-FAMA Multiple Acess Low Moderate High Moderate
Du et al, 2015 state-based CDMA MAC Virtual Handshake Moderate Moderate Low Moderate
Li et al, 2016 DTMAC Coupon Collection Low Moderate Moderate High
underwater environment and in turn create a dynamic
3.4 Motivation for self-organize and self- network topology.
adaptive MAC protocols for UWASNs
In addition, node failure is more prone to UWASNs
There are many factors that necessitate the development of a because of energy-depletion or failure in hardware as a
self- organize and adaptive MACs for UWASNs, some of result of corrosion or fouling.
those are described here:
It is very difficult to achieve accurate time
The communication module in UWASNs called acoustic synchronization of the nodes because of the variable and
modems consume more energy as compared with the long propagation delay which can limit approaches that
conventional motes for terrestrial WSNs. However, nodes depends entirely on duty cy- cling.
are powered by batteries which will be extremely difficult
Also, situations of Hidden node and exposed node in
to recharge or replace and solar power cannot be exploited
underwater channel become prominent with contention-
in underwater environment.
based collision avoidance MAC protocols.
Another challenge is that, due to vast nature of water body
Due to low propagation speed in underwater channel,
such as sea and ocean, deployments are mostly sparsely
hand- shaking experience high delay, and this can
based and this can cause passive movement of nodes due
negatively affect the performances of MAC protocols that
to water current or other disturbances which are prone to
185
depend on RTS/CTS handshake process. their tasks thus, makes the system robust and good for “self-
adaptability” and “self-organization”. Because of this full
Since, UWASNs are known for power challenge, MAC
cooperation among the agents, the system will also allow easy
proto- cols for UWASNs should be able to avoid power
insertion of new agents into the system without bringing the
wastage in collision.
entire system down, this provide for high degree of scalability.
It is also important to know that centralized networking is
It is understood from the literatures that valid and standard
not suitable for UWASNs, because, it will create a single
models for UWASNs do not exist, in order to realize the
point of failure. That is why a scheme that can self-
proposed tech- nique, an underwater pipeline infrastructure
organize and be self- adaptive is required to fully improve
monitoring scenario is considered. The model of this scenario
the reliabilty of UWASN systems.
will be firstly developed based on the requirements for this
Studies on MAC protocols have also shown that most of study. This model shall be used as an application base on
the MAC protocols designed for (radio based) WSNs are which the Fully Cooperative Multi-Agent Reinforcement
not opti- mized for very long propagation delay, low data Learning based Q-ALOHA MAC (FCQ-ALOHA MAC) is
rates and en- ergy efficiency in underwater acoustic deployed. The development of the FCQ-ALOHA MAC will be
channel. some of the In- telligent MAC schemes are also achieved by designing fully cooperative mechanism struc- ture,
marred by issues of imbalance fairness, difficulty in frame Q-Learning algorithm initialization and Markov Decision Pro-
size estimation and degraded delay performances. cess (MDP) model.
Current Reinforcement Learning based MAC protocols
are based upon single-agent learning, which is 4.1 Architecture of the proposed MAC
independent learn- ing without cooperation and intelligent
protocol
interactions among the nodes and the channel. This not
practical for UWASNs since cooperation and adaptability The block diagram of the proposed FCQ-ALOHA MAC
with the dynamic channel enviro- ment is paramount. protocol for sender node within cooperative nodes is shown in
Figure 1.
Owing to the aforementioned challenges, UWASNs always
exhibit dynamic network topology. Alongside other Sensor nodes are modelled as frames and each frame divided
challenges such as long and variable propagation delay, low into optimal number of slots. As fixed frame size estimation
bandwidth, high bit error rate, etc., all bring about serious has been identified from the literatures to be a difficult task
challenges for deigning MAC protocol for UWASN. and frame size over or under estimation could lead to poor
However, adaptive MAC protocols can have significant performances in- terms of QoS and energy efficiency, we
positive impact on hash channels with low link quality such as therefore, will employ dy- namic frame size estimation. This
underwater acoustic channel. will exclude the task of pre- allocation of frame sizes. At initial
instance, Q-values for all the Cooperative Nodes (CNs) in the
network are set to zero, (Q1,1, = Q1,2.... = Qi,n = 0), to create
4. OVERVIEW OF THE PROPOSED MAC
a complete random access transmission scenario, TxALOHA.
PROTOCOL Where Qi,nis the Q-value associ- ated with the nth slot of the
The technique will involve the use of a model-free Reinforce- ith frame and TxALOHA is the pure ALOHA transmission
ment Learning (Q-Learning) algorithm to explore and exploit scenario. Optimal slot within the optimal frame will then be
Fully Cooperative Multi-Agent based learning experience on a selected for data packet transmission. As ex- pected, this initial
frame based ALOHA MAC scheme. This will aid nodes transmission scenario will have maximum data collision,
cooperation and interactions with the underwater environment however, the experience (reward calculated from Re- ward
to achieve “self- organization” and “self-adaptability” which in Function) from this transmission will be fed back to the Q-
turn can signicantly improve the QoS of UWASN systems by value tables which will then inform a better transmission
having better conver- gence time and high Energy efficiency. policy for next transmission. The algorithm is expected to
With Multi-agent based Reinforcement Learning, faster converge within the shortest possible period of time depending
learning and convergence can be achieved due to experience on network size, node mobility and node density. After
sharing among the agents. when one or more agents fail, which convergence, the CNs would have learn the Joint Policy, π, and
is synonymous to node(s) failure (inher- ent in Underwater on their own can take proper actions for future data
Acoustic Sensor Networks), others agents take over some of transmission.
186
The flow chart depicting the proposed implementation
strategy
of data transmission based on FCQ-ALOHA MAC protocol
is shown in Figure 3.
4.2 Expected Outcomes
The study is expected to develop an intelligent ALOHA based
Media Access Control protocol for Underwater Acoustic
Sensor Net- works for underwater pipeline monitoring. This
will provide acceptable QoS performance in terms of
throughput, delay, and en efficiency and in turn make
UWASN systems more reliable, efficient and effective for
Figure 1. Block diagram of FCQ-ALOHA MAC for sender various applications. A model of the underwater pipeline
node infrastructure monitoring system will also be developed, and
this will be used as an application base for evaluating the
developed MAC protocol.
The cooperative network model is shown in Figure 2.
5. CONCLUSION
The challenges associated with underwater acoustic
communications have made the deployment of UWASNs
unpopular for potential applications in underwater
operations. Medium Access Con- trol protocol is largely
responsible for successful UWASNs development which is
marred by challenges such as long and variable propagation
delay, limited channel capacity, low data rates and en- ergy
efficiency. The state of the art MAC protocols for UWASNs
have room for improvement in terms of QoS and energy
efficiency. In view of the above, a Fully Cooperative Multi-
Agent Q- learning based MAC protocol that would be “self-
organized” and “self-adaptive” with improved performances
in delay, throughput and energy efficiency is proposed here
to provide solutions for the aforementioned challenges of
UWASNs systems. It is expected that meaningful impact
with respect to data transmission at MAC layer would be
made in applications of UWASNs systems.
Figure 2. Conceptual diagram of Cooperative network
model
187
hydroelectric reservoirs. In WUWNET â12, Los Angeles,
California USA., 2012.
[3] N. Thi-Tham and S. Yoon. Ars: An adaptive
retransmission scheme for contention-based mac protocols
in underwater acoustic sensor networks. International
Journal of Distributed Sensor Networks, 15, 2015.
[4] Mohsin Murad, Adil A Sheikh, Muhammad Asif
Manzoor, Emad Felemban, and Saad Qaisar. A survey on
current underwater acous- tic sensor network applications.
International Journal of Computer Theory and Engineering,
7(1):51, 2015.
[5] Christina Peach and A. Yarali. An overview of
underwater sensor networks. In ICWMC 2013: The Ninth
International Conference on Wireless and Mobile
Communications, 2013.
[6] I. F. Akyildiz, D. Pompili, and T. Melodia.
Underwater acoustic sensor networks: research challenges.
Ad Hoc Networks, 3(2):257– 279., 2005.
[7] J. Heidemann, M. Stojanovic, and M. Zorzi. Underwater
sensor net- works: applications, advances and challenges.
Philosophical Transac- tions of the Royal Society,
370(1958):158–175, 2012.
[8] L. Jin and D. Huang. A slotted csma based reinforcement
learning ap- proach for extending the lifetime of underwater
acoustic wireless sen- sor networks. Computer
Communications, 36(9):1094â1099., 2013.
[9] Fatih Senel, Kemal Akkaya, Melike Erol-Kantarci, and
T. Yilmaz. Self-deployment of mobile underwater acoustic
sensor networks for maximized coverage and guaranteed
connectivity. Journal of Ad Hoc Networks, 34:170â183.,
2014.
[10] Chu Yi, Selahattin Kosunalp, Paul D. Mitchell, David
Grace, and T. Clarke. Application of reinforcement
Learning to medium access control for wireless sensor
networks. Elsevier J. of Engineering Applications of
Artificial Intelligence, 46:23–32., 2015.
[11] R. S. Sutton and A. G. Barto. (1998). Reinforcement
learning: An introduction. MIT press Cambridge, 1 edition.
[12] H. D. Li, D. Grace, and P. D. Mitchell. Cognitive radio
multiple access control for unlicensed and open spectrum
with reduced spectrum sensing requirements. In
International Symposium on Wireless Communication
Systems (ISWCS), 2010.
[13] J. G. Proakis, E.M. Sozer, J. A. Rice, and M.
Stojanovic Shallow water acoustic networks. IEEE
CommunicationsMagazine, 39(11):114â119., 2001.
[14] J. Ahn, A. Syed, B. Krishnamachari, and J. Heidemann.
Design and analysis of a propagation delay tolerant aloha
protocol for underwater networks. Journal of Ad Hoc
Networks, 9(5):752–766., 2011.
[15] L. Vieira, J. Kong, U. Lee, and M. Gerla. Analysis of
aloha protocols for underwater acoustic sensor networks. In
1st International Work- shop on Underwater Networks
(WUWNet â06), Los Angeles, Calif, USA., September 2006.
Figure 3. Flow chart of data transmission strategy
[16] N. Chirdchoo, W.-S. Soh, and K. C. Chua. Aloha-
based mac proto- cols with collision avoidance for
6. REFERENCES underwater acoustic networks. In 26th IEEE International
Conference on Computer Communications (INFOCOM
[1] J. Heidemann, Y. Wei, J. Wills, A. Syed, and L. Yuan. â07), Anchorage, Alaska, USA„ 2007.
Research challenges and applications for underwater sensor
networking. In IEEE Wireless Communications and [17] P. Mandal, S. De, and S. S. Chakraborty. A receiver
Networking Conference, 2006. WCNC 2006., 2006. synchronized slotted aloha for underwater wireless
networks with imprecise propagation delay information.
[2] F. M. V. Luiz, P. David, and S. S. Viana. Hydronode: An Journal of Ad Hoc Networks, 11(4):1443â1455., 2013.
underwater sensor node prototype for monitoring
188
[18] D. Wang, X. Hu, F. Xu, H. Chen, and Y. Wu. [28] Guangyu Fan, Huifang Chen, Lei Xie, and Kwang
Performance analysis of p-csma for underwater acoustic Wang. Poca-cdma mac: An improved cdma-based mac
sensor networks. In OCEANS, Yeosu, Republic of Korea., protocol for underwater acoustic wireless sensor networks. In
May 2012. Proceedings of IEEE International Conference on Wireless
Communications, Networking and Mobile Computing
[19] D. Fang, Y. Li, and H. Huang L. Yin. A csma/ca- (WiCOM), pages 1–4, 2011.
based mac proto- col for underwater acoustic networks. In
6th International Conference on Wireless Communications, [29] Marcžal Molins and Milica Stojanovic. Slotted fama:
Networking and Mobile Computing (WiCOM â10), Chengdu, a mac protocol for underwater acoustic networks. In
China., September 2010. Proceedings of IEEE OCEANSâ06, pages 16–22,
Singapore, 2006.
[20] M. Molins and M. Stojanovic. Slotted fama: a mac
protocol for underwater acoustic networks paper presented [30] Peng Xie and Junhong Cui. R-mac: An energy-efficient
at the. In OCEANS 2006, Asia Pacific, Singapore., 2007. mac protocol for underwater sensor networks. In
Proceedings of International Conference on Wireless
[21] S. M. Smith, J. C. Park, and A. Neel. A peer-to-peer Algorithms, Systems and Applications, pages 187–196, 20
communication protocol for underwater acoustic
communication paper presented at the. In MTS/IEEE [31] S. Han, Y. Noh, U. Lee, and M. Gerla. M-fama: A
Conference (OCEANS â97), IEEE, Halifax, Canada, October multi-session mac protocol for reliable underwater acoustic
1997. streams. In in Proc. INFOCOM, pages 665–673, 2013.
[22] X. Guo, M. R. Frater, and M. J.Ryan. (2007, may [32] Chao Li, Yongju Xu, Chaonong Xu, Zhulin An, Diao
2007). a propagation-delay tolerant collision avoidance Boyu, and Li Xiaowei. Dtmac: A delay tolerant mac protocol
protocol for underwater acoustic sensor networks paper for underwater wireless sensor networks. In IEEE Sensors J.,
presented at the. In Asia Pacific (OCEANS â06), Singapore, 16(11):4137â4146, 2016.
2007.
[33] X. Liang, M. Chen, Y. Xiao, and V. C. M. Leung. Mrl-
[23] B. Peleato and M. Stojanovic. Distance aware cc: A novel cooperative communication protocol for qos
collision avoidance protocol for ad-hoc underwater acoustic provisioning in wireless sensor networks. International
sensor networks. IEEE Communications Letters, Journal of Sensor Networks, 8(2):98– 108, 2010.
11(12):1025–1027, 2007.
[34] Ning Li, José-Fernán Martínez, Juan Manuel Meneses
[24] A. A. Syed, W. Ye, and J. Heidemann. T-lohi: a new Chaus, and Martina Eckert. A survey on underwater acoustic
class of mac protocols for underwater acoustic sensor sensor network routing protocols. Sensors, 16(3):414, 2016.
networks. In 27th IEEE Communications Society
Conference on Computer Communications (INFOCOM [35] A. Nosratinia, T. E. Hunter, and A. Hedayat.
â08), Phoenix, Ariz, USA, April 2008. Cooperative commu- nication in wireless networks. In IEEE
Communications Magazine, volume 42, pages 74–80.
[25] Z. Liao, D. Li, and J. Chen. (2015). a handshake October 2004.
based ordered scheduling mac protocol for underwater
acoustic local area networks. International Journal of [36] Y. W. Hong, W. J. Haung, F. H. Chiu, and C. C. Kuo.
Distributed Sensor Networks, 2015(15), 2015. Cooperative communication in resource constraint wireless
networks. In IEEE Signal Processing Magazine, volume 24,
[26] G. Fan, H. Chen, L. Xie, and K. Wang. A hybrid pages 47–57. May 2007.
reservation-based mac protocol for underwater acoustic
sensor networks. Journal of Ad Hoc Networks, 11(3):1178– [37] Keyu Chen, M. M. En Cheng, Fei Yuan, and W. Su. A
1192., 2013. survey on mac protocols for underwater wireless sensor
networks. IEEE COMMUNICATIONS SURVEYS &
[27] Du Xiujuan, Peng Chunyan, Liu Xiuxiu, and Liu Yuchi. TUTORIALS, 2014.
Hierarchical code assignment algorithm and state-based
cdma protocol for uwsn. China Commun. (China [38] Jesús Llor and Manuel P Malumbres. Modelling
Communications), 12(3):50–61, 2015. underwater wireless sensor networks. Intech open, pages 1–
20, 2010.
189