Towards a Fully Cooperative Multi-Agent Reinforcement Learning based Media Access Control Protocol for Underwater Acoustic Wireless Sensor Networks

Abstract

Underwater Acoustic Sensor Networks (UWASNs) has gain a widespread recognition recently due to some technological break- through, and thus, beginning a new era of research in the industry with potential for vast applications that are important to our livelihood. Despite all these potentials, deploying a reliable UWASNs based systems still remain very far from perfect and there are only limited experimental trials at the moment. This is due to challenges of reliabilty, QoS and energy efficiency, which is due to inherent characteristics of underwater acoustic channel. These pose signif- icant challenges for the design of network protocols, especially, the Media Access Control (MAC) protocol for UWASNs. Various MAC protocols have been developed for UWASNs and some few adopted from Wireless Sensor Networks (WSNs). However, most of these protocols do not provide acceptable QoS in terms of delay, throughput, fairness and energy efficiency. This paper presents a review of some of the prominent MAC protocols for UWASNs and adaptable WSNs based MAC protocols for UWASNs and propose a Fully Cooperative Multi-Agent Reinforcement Learning based MAC protocol for UWASNs. The proposed scheme will apply MultiAgent based Reinforcement Learning (RL) to ALOHA MAC scheme to create a dynamic contention-free-like slotted MAC to aid nodes cooperation and interactions within themselves and the underwater environment to significantly achieve “selforganization” and “self-adaptability” to changes in the environment which would provide means for coping with long and variable propagation delay, low data rates and energy efficiency and in turn can significantly improve the QoS of UWASN systems by having better convergence time and Energy efficiency.

Keywords

• eol>Reinforcement Learning
• MAC protocol
• ALOHA
• QoS
• Selforganization
• UWASN
• Multi-Agent

CCS Concepts

• Networks ➝ Network components ➝ points, base stations and infrastructure Wireless access 1.

INTRODUCTION

Increased researches in WSNs has made a plethora of real life applications possible, particularly in underwater scenario. UWASNs is recently becoming an important area of research with promising potential for various applications ranging from underwater oil and gas extraction (seismic imaging), pipeline and infrastructure monitoring, marine life monitoring and control, monitoring of underwater Carbon(IV)Oxide (CO2) storage facility, border control, Fish farming, freshwater reservoirs management, Autonomous Underwater Vehicles (AUVs), Naval Network centric warfare- mine reconnaissance etc. to tsunami and seaquake early warning systems [ 1–4 ]. Despite all these promising applications, Underwater Sensor Networks remain quite limited as compared to the terrestrial Sensor Networks technologies. Thus, this makes underwater operations limited to remotely controlled submersibles which are large, very costly and are almost temporarily deployed [ 1 ] as com- pared to sensor network nodes which are relatively cheaper and can be permanently deployed on the sea flow for real time communications.

Radio based communication for terrestrial sensor networks is not suitable for underwater usage because of extremely limited propagation delay as current mote radios transmit between 50 to 100cm and within 30-300 Hz of frequency underwater. The implication is that extraordinary transmission power and very large antennas are required for deployment [ 1, 5 ]. Therefore, establishing communication in UWSN effectively largely depends on acoustic communications. However, Underwater Acoustic communications bring about new challenges due to unique characteristics of under- water acoustic communication channels such as: High propagation delay caused by low speed of acoustic signals (speed of sound is approximately 1500 m/s) which is by 5 orders of magnitude slower than radio waves (3x108m/s) for terrestrial Wireless Sensor Net- works (WSN) [ 1, 3 ], low data rate (between 5-20Kb/s) due to limited channel bandwidth, high error rates, highly dynamic environment and high energy consumption (typical consumption between 50 to 100 W) [ 3, 6–8 ].

UWASNs is made up of a large number of sensors deployed underwater with capability to communicate via acoustic links. Its then worthy to mention that special consideration need to be taken with respect to channel modelling, medium access, routing and other sensitive issues when designing UWASNs [ 9 ]. For a successful UWASNs design and deployment, Media Access Control (MAC) protocol is very important. It is a Layer 2 (Data Link) protocol which define how channels are accessed for efficient and successful communication. Various MAC protocols have been proposed for the terrestrial WSNs to provide significant improvement on energy efficiency and throughput performance [ 10 ], however, these schemes cannot be directly adopted for UWASNs due to the afore- mentioned unique characteristics of underwater environment [ 8 ]. MAC schemes can be classified into contention based and contention free schemes.

Contention based schemes mostly employ carrier sense (CS) techniques such as Carrier Sense Multiple Access (CSMA) proto- cols and Random access techniques such as ALOHA protocols. In CSMA schemes, exchange of control packets causes long packet delay due to long preamble in real acoustic modems which in- creases packet collisions and control packets have long preamble and load which degrade network performance, increased energy consumption and hidden terminal problem [ 10 ], thus not suitable for applications such as UWASNs requiring low delay. Also, pure ALOHA relies on packet retransmission for reliable data delivery. This may be suitable for terrestrial WSNs because of its simplicity, but may not be appropriate for UWASNs, because, packet retransmission can quickly saturate the network due to limited channel capacity [ 3 ]. Contention-free access schemes (TDMA, FDMA, CDMA, etc.) use slot-scheduling techniques for media access. This could have been the right candidate for UWASNs, because of low collision rates, but they are rather too complex for literally speak- ing, primitive underwater sensor technologies. Another problems of contention-free schemes are high system overhead, high propa- gation delay, strict time synchronization and not flexible to changes in the number of nodes. Therefore, for a successful UWASNs de- ployment to solve unique challenges earlier discussed, possible solutions are: To design a new sleep and wake-up schemes from scratch for UWASNs, reduce control packet exchange or to com- bine contention-based and schedule-based schemes.

Thus, there is need for a much simple MAC protocols scheme that will be “self-organized” and “self-adaptive” after been deployed and can nonetheless provide energy efficient communica- tion, good throughput and acceptable delay. This research proposes development of an intelligent ALOHA based MAC protocol for UWASNs. The research will explore the use of machine learning techniques, specifically, Multi-Agent based Reinforcement Learn- ing (RL) to assist with nodes cooperation and interactions with the environment to achieve “selforganization and adaptation”. This would provide means for coping with long and variable propagation delay, low data rates and energy efficiency. The ability of nodes to learn from their interactions with the wireless environment pro- vides scope to significantly enhance their ability to self-organize and adapt to changes in the environment.

Reinforcement Learning (RL) is an approach or technique of Machine Learning (ML) that makes use of agent(s) to learn effective strategies through trial-and-error interactions with the dynamic environment, take future actions which are determined by scalar reward based on prior experience to transit from initial state to a new one with the ultimate goal of maximizing the cumulative re- ward along the course of interaction [ 11 ]. It has found established usage in Artificial Intelligence researches such as robotics, controls and automation, and recently in communication problems such as MAC layer protocols [ 10, 12 ] and similar design strategies could be employed for developing MAC protocol for UWASNs.

The rest of the paper is organized as follows. Findings on the review of some prominent MAC protocols are presented in section 2, while, section 3 presents design challenges of MAC protocols for UWASNs. Section 4 presents the overview of the proposed MAC protocol and section 5 concludes the paper.

2. REVIEW OF SOME PROMINENT MAC

PROTOCOLS

This section reviews some of the important MAC protocols that have been recently proposed for WSNs and UWASNs to address the pertinent problems of energy efficiency, throughput and de- lay. Contention-based and contention-free schemes have been considered for UWASNs. For contention-free schemes, it is already established that Frequency Division Multiple Access (FDMA) is not suitable as a result of limited bandwidth of acoustic channel [ 3 ]. Time Division Multiple Access (TDMA), another form of contention-free schemes, has also been studied but its efficiency is limited by strict synchronization and large guard time [ 13 ]. More- over, Code Division Multiple Access (CDMA) (another contention- free scheme) is known for high autocorrelation and low crosscorrelation properties to minimize interference among users which make its design for UWASNs very complex [ 6 ]. On the contrary, ALOHA [ 14–17 ] and CSMA [ 8, 18–21 ] (contention-based schemes) have recently received significant consideration for UWASNs owing to their simplicity and good throughput [ 3 ]. A contention-based scheme that depends on hand- shake called propagation delay tolerant collision avoidance proto- col (PCAP) was proposed by [ 22 ], it allows the sender to transmit another data packet or perform handshake for the next queued data packet while waiting for the clear to send (CTS) packet, thereby, favourably utilizing long propagation delay. But it requires strict clock synchronization which makes it complex for UWASNs. Moreover, [ 23 ] proposed distance aware collision avoidance proto- col (DACAP), a handshake based protocol which creates a waiting window of time (based on the distance between the sender and the receiver node) for the sender after receiving CTS to allow for in- tending receiver to receive warning to avoid collision. The control packets and long preamble can cause long packet delay and in turn reduce the network performance.

In addition, Tone signals have also been employed in contention- based approaches as evidenced in [ 24 ] called TLohi protocol. In this approach, short tones are transmitted by nodes to alert neigh- boring nodes of intending transmissions to detect the channel con- tenders before sending data. The time instances of arrival of such tones at various nodes varies for different nodes due to different propagation delays. Thus, nodes only transmit data whenever tone signals are not received, otherwise, a calculated back-off interval is activated and backoff performed. The downside is that a special Wake-up tones receiver hardware is required by T-Lohi nodes to be able to detect tones using low energy consumption.

A Handshake based Ordered Scheduling MAC (HOSM) has also been proposed by [ 25 ] for underwater acoustic LANs. In this technique, Channel reservation phase is firstly created by the intending nodes to transmit data packets and a calculated ordered lists are used by the nodes for data transmission. The key idea of this technique is to utilize the information of propagation delay to adjust the time instant of control packets transmission to reduce collisions of control packets to achieve high throughput, low de- lay spatial fairness. However, energy is not given fair attention and could be adaptable to a traffic with different priorities. Furthermore, [ 26 ] proposed an hybrid scheme called hybrid reservation-based MAC (HRMAC) protocol where the nodes use declaration to re- serve channel and collision of control packets is reduced by spec- trum spreading technology. The good news is that many nodes with intending data packet transmission can reserve the channel simul- taneously but transmit their data in a given order. This significantly improves the channel efficiency as compared to typical MAC pro- tocols for UWASNs. But the scheme cannot be extended to general multi-hop underwater acoustic networks. In addition, very recently, [ 3 ] proposed an adaptive retransmission scheme for contention-based MAC Protocols for Underwater Acoustic Sensor Networks. This try to address the problem of low performance (low Packet Delivery Ratio (PDR) and high End-to- End (E2E) delay) associated with contentionbased MAC proto- cols for UWASNs by using adaptive retransmission scheme (ARS) to dynamically selects an optimal value of the maximum number of retransmissions, such that the successful delivery probability of a packet is maximized for a given network load. ARS ALOHA and ARS CSMA significantly improve network performance in terms of PDR and E2E delay, however, it could not extend ARS to support different performance requirements in UWASNs such that each node can adapt its transmissions to satisfy a specific performance requirement from applications or users. ALOHA-Q, an intelligent based protocol is proposed by [ 10 ] for terrestrial WSNs. This applies reinforcement learning to frame based ALOHA to select slots intelligently with capabilities to mi- grate from random access to perfect scheduling using Q-learning technique. It utilizes a simple learning process and has much lower complexity and overheads. This greatly improve QoS of WSN in terms of energy efficiency, delay and throughput as compared with slotted ALOHA, S-MAC and Z-MAC. However, overestimating frame size can generate unused slots and underestimating frame size can introduce packet collisions which both may affect channel performance. In the same vain, there is concern about the ability of the network to adapt to different densities of node deployment without requiring a fixed and pre-estimated frame size configura- tion. Although, this technique has promising performance, it was designed for terrestrial WSNs. This can be adapted for UWASNs by careful modifications to suit the challenges of limited channel capacity, long propagation delay and energy efficiency for underwater acoustic communications. Most importantly, frame size estimation could further be tuned to reduce packet collisions to acceptable val- ues for UWASNs. [ 27 ] proposed a hierarchical and distributed code assignment algorithm based on divisive probability function which can avoid conflict between spread codes with high probability, and provide a state-based MAC protocol for UWASNs. The technique tries to eliminate the RTS/CTS handshake prominent in POCA-CDMA (Path oriented Code Assignment) MAC protocol [ 28 ], Slotted- FAMA [ 29 ] and R-MAC [ 30 ]. POCACDMA MAC adopts the CDMA technology to make the sink receive packets from multi- ple neighbors at the same time. It achieves higher throughput, but suffers from the hidden terminal effects and low energy efficiency as a result of higher control packets overhead. That may be toler- able in terrestrial WSNs, but becomes serious underperformance issue in UWSNs as a result of low bandwidth, long propagation de- lay and high energy consumption. Slotted-FAMA, due to frequent exchanges of RTS/CTS, reduces the channel utilization, results in poor performance such as throughput, end-to-end delay in UWASN characterized by long propagation delay, low bandwidth and high bit error rate. R-MAC schedules the transmissions of control pack- ets and data packets to avoid data packet collision. This creates serious overhead issues and further dampens its performance in UWASNs.

As a result, [ 31 ] proposed Multi-session FAMA (M-FAMA) for UWASNs to solve the problem of bandwidth limitation associated with Slotted-FAMA. It takes the advantage of the propagation de- lay information of the neighboring nodes and expected transmis- sion schedules to initiate multiple sessions simultaneously. In this scheme, the inherent problem of fairness across multiple contend- ing sources is solved by introducing an algorithm that balances the bandwidth. Compared with its predecessor, Slotted-FAMA, it has the advantages of temporal/spatial reuse and collision avoidance to some degree. However, due to large number of control packets in order to initiate multiple sessions, M-FAMA performs low in terms of energy efficiency as compared to most channel reservation protocols. Also, in bursty-traffics, RTS/CTS handshake degrades its performance in terms of throughput. Also, throughput is affected significantly when deployed on highly mobile nodes as a result of increase in failure of channel reservation due to changing network topology and it is not developed with selforganization and adap- tation capabilities. Moreover, the performance in terms of delay is poor, because, the RTS/CTS handshake processes keep the propa- gation delay at high values and this problem is not solved by the multiple session mechanism.

In contrast, the technique in [ 27 ] introduces probability function- based code assignment algorithm to reduce code collision; mean- while, without RTS/CTS handshake, statebased channel access mechanism maximizes the channel utilization. It also supports con- current transmissions between all of the nodes by adopting CDMA communication technology, which improves the network perfor- mance of endto-end delay, energy-consumption, network through- put and delivery ratio. However, CDMA is not practical because it is difficult to assign pseudo-random codes among large numbers of sensor nodes, thus, it is not scalable as evidenced in the throughput becoming poor with increase in network load. Also, it did not con- sider self-organization and self-adaptability issues, which are very important when designing MAC for UWASNs.

DTMAC was proposed by [ 32 ] for UWASN and was based on distributed coupon collection algorithm that allocate a certain num- ber of times an intending transmitting nodes will repeat a trans- mission which is a function of transmission probability that only requires the number of neighboring nodes and not the exact net- work topology so as to improve the network throughput in burst short-packet traffic deployments and overcome the challenges of long propagation delay and swarm mobility. The technique tried to map throughput-optimal value with the successful transmission probability as a turning parameter to avoid the use of channel reser- vation and handshake mechanisms in order to curb the problem of propagation delay. It also tried to solve the problem of space unfairness by eliminating transmission distance factor. All these considerably improve the performance of

DTMAC in UWASNs. However, DTAMC is designed with the goal of short data packets transmission, thus it makes an assumption of a single-hop target network. In addition, DTMAC protocol may be suitable for high bandwidth demand deployments, but pay less attention on success- ful packet transmission probability which means that the through- put in this sense is affected. Another weakness of this protocol is that it was only designed for short data packets. Also, scalability is an associated problem as performances in terms of throughput, delay and energy efficiency degrade with increase in node density, this is because, the technique only takes into consideration the de- lay factor.

Intelligent protocols have also been recently employed in wireless communications including Wireless Sensor Networks as ev- idenced in [ 10, 33, 34 ], also cooperative communications have shown to have significant effect solving the problem of multiple fading effects in wireless networks, and thereby improve QoS of the network in terms of adaptivity, reliability, data throughput and net- work lifetime. [ 33 ] investigated the use of cooperative communi- cations based on Multi-Agent Reinforcement Learning (MRL-CC) algorithm on multi-hop mesh cooperative communication mecha- nism in order to achieve QoS provisioning in terrestrial WSNs. Is- sues of Spatial and time diversity gains in wireless networks using cooperative communications have also been investigated recently in [ 35, 36 ]. This strategy complements the M-FAMA approach by intelligently creating multiple transmission sessions. Owing to the broadcast nature of the wireless (RF) medium and spatial distri- bution of sensor nodes, cooperative communications can be used to improve the network performance of WSNs. This can also be further extended through a careful design to take advantage of un- derwater acoustic channel to develop a cooperative MAC protocol for Underwater Acoustic Sensor Networks.

Performance comparison of some of the prominent MAC protocols for Underwater Acoustic Sensor Networks are summarised in Table 1 as shown: From the comparative analysis summerised in Table 1, it can be concluded that the current researches on MAC protocol de- sign for UWASNs strive to achieve optimal channel performance at the cost of architectural complexity. Consequently, control over- heads are increased and as a result, performs poorly in energy effi- ciency. There is need for consideration of MAC protocols design for UWASNs that should make a trade-off between energy effi- ciency and channel performance with respect to application area.

This paper proposes a technique that apply Reinforcement Learning on framed ALOHA MAC protocol. This hope to take the advantage of low architectural complexity and overheads associated with ALOHA MAC protocol to provide intelligent slot selection for optimal data transmission which will bring about su- perior channel performance as against low throughput associated with ALOHA based MAC protocols.

3. DESIGN CHALLENGES OF MAC

PROTOCOLS FOR UWASNs

Media Access Control (MAC) protocol is a Layer 2 (Data Link) protocol that defines how channels are accessed for efficient and successful data packets communications. It is the backbone of data packet transmission between sensor nodes in UWASNs and QoS and Energy efficiency of UWASNs largely depends on it [ 37 ]. Thus, it is important to carry out comprehensive study on the design of MAC protocols for underwater in order to have an effective communication between the sensor nodes, acceptable QoS and reasonable energy efficiency. 3.1

Why not Radio Frequency Communication?

Radio waves are strongly attenuated in underwater water enviro- ments and as a result has limited propagation ranges, e.g. in sea water, just up to 10 meters [ 38 ]. The implication is that to cover large distances, large antennae and high transmitter powers are required. This is costly and nonpractical. Long-wave radio, however, can be used for short distances 1-8kbps at 122kHz carrier for ranges up to 6-10 meters [ 38 ]. Propagation rate is also low due to low-bandwidth modems which are currently available conditions that the range becomes appreciable, approximately 100 meters [ 38 ], and only with high bandwidth modems can several Mbps of data rate be achieved. However, developing high band- width modems for underwater communications is still a open re- search area, and there is need for transceiver alignment, thus only short-range connections of order 1-2m at 57.6kbps data rate are possible [ 4, 38 ]. This is not practical for numerous underwater applications. Thus, it is very clear that for effective communication in underwater channel, acoustic communication is the ultimate alter- native. 3.3

Characteristics of Acoustic channels

Acoustic Communication is the only communication technology that supports all required transmission ranges in underwater, it is cheaper and practical as compared to radio and optical communication in underwater environment. However, acoustic channels have some unique features that pose challenges to effective communication in underwater environments. Some of these features are as follows: 

Very long and variable propagation delay as a result of low speed of sound which is approximately 1500 m/s, about 5 orders of magnitude slower than radio waves (3x108 m/s) and the speed of sound varies      considerably with respect to temperature, pressure and salinity, thus, it is depth dependent (1450-1540 m/s.) [ 37, 38 ].

Bandwidth is severely limited as a result of attenuation and interactions with bottom and surface of the water body. Thus, the available bandwidth becomes transmission distance dependent. Data rate is also low as a result, about 100Kbps [ 37 ].

Extensive multi-path arrivals/propagations cause Intersymbol interference (ISI) delay in hundreds of symbols, which can severely degrade acoustic communication signal and also leads to high bit error rate.

Channel Dynamism with respect to time and high Dopller spread (especially for horizontal communication). It is also clear that one water body is different from the other and different from itself at different times, this makes channel tracking to become difficult.

Dopller-shift ratio is of several orders higher than that of the RF channels which makes symbol synchronization difficult.

Noise which is caused by majorly two factors, the

3.4 Motivation for self-organize and selfadaptive MAC protocols for UWASNs

There are many factors that necessitate the development of a self- organize and adaptive MACs for UWASNs, some of those are described here:  The communication module in UWASNs called acoustic modems consume more energy as compared with the conventional motes for terrestrial WSNs. However, nodes are powered by batteries which will be extremely difficult to recharge or replace and solar power cannot be exploited in underwater environment.  Another challenge is that, due to vast nature of water body such as sea and ocean, deployments are mostly sparsely based and this can cause passive movement of nodes due to water current or other disturbances which are prone to  ambient and man made noises. Noise in underwater environment can be given as: Noise = Turb + Ship + Surf + Therm + Others (1) Where Turb is Turbulence, Ship is Shipping, Therm is Thermal and Others refer to manmade, biological, ice, rain, seismic, etc noises.

Path/propagation loss as a result of attenuation as a result of de- crease of the sound intensity through the path from the sender to the receiver caused by absorption due to conversion of acoustic energy into heat, and it increases with distance and frequency.

Thus, The Transmission Loss (TL), is given as: TL = SS + α × 10−3

(2) Where, SS is the Spherical spreading factor given as ss = 20 log r, r is the range in meters and α is the attenuating factor put forward empirically by Thorp formula. 3.2

Why not Optical Communication?

Light is strongly scattered and absorbed underwater, drastically limiting communication range [ 38 ]. It is only in very clear water Energy Efficiency

Low Low Low

Low Moderate

Low

Throughput

Low

High Moderate Moderate Moderate Moderate

Delay High Moderate Low High Low Moderate

Channel utilization

Low Moderate Moderate Moderate Moderate

High underwater environment and in turn create a dynamic network topology.  In addition, node failure is more prone to UWASNs because of energy-depletion or failure in hardware as a result of corrosion or fouling.  It is very difficult to achieve accurate time synchronization of the nodes because of the variable and long propagation delay which can limit approaches that depends entirely on duty cy- cling.  Also, situations of Hidden node and exposed node in underwater channel become prominent with contentionbased collision avoidance MAC protocols.  Due to low propagation speed in underwater channel, hand- shaking experience high delay, and this can negatively affect the performances of MAC protocols that depend on RTS/CTS handshake process.  Since, UWASNs are known for power challenge, MAC proto- cols for UWASNs should be able to avoid power wastage in collision.  It is also important to know that centralized networking is not suitable for UWASNs, because, it will create a single point of failure. That is why a scheme that can selforganize and be self- adaptive is required to fully improve the reliabilty of UWASN systems.  Studies on MAC protocols have also shown that most of the MAC protocols designed for (radio based) WSNs are not opti- mized for very long propagation delay, low data rates and en- ergy efficiency in underwater acoustic channel. some of the In- telligent MAC schemes are also marred by issues of imbalance fairness, difficulty in frame size estimation and degraded delay performances.  Current Reinforcement Learning based MAC protocols are based upon single-agent learning, which is independent learn- ing without cooperation and intelligent interactions among the nodes and the channel. This not practical for UWASNs since cooperation and adaptability with the dynamic channel enviro- ment is paramount. Owing to the aforementioned challenges, UWASNs always exhibit dynamic network topology. Alongside other challenges such as long and variable propagation delay, low bandwidth, high bit error rate, etc., all bring about serious challenges for deigning MAC protocol for UWASN. However, adaptive MAC protocols can have significant positive impact on hash channels with low link quality such as underwater acoustic channel.

4. OVERVIEW OF THE PROPOSED MAC

PROTOCOL

The technique will involve the use of a model-free Reinforcement Learning (Q-Learning) algorithm to explore and exploit Fully Cooperative Multi-Agent based learning experience on a frame based ALOHA MAC scheme. This will aid nodes cooperation and interactions with the underwater environment to achieve “self- organization” and “self-adaptability” which in turn can signicantly improve the QoS of UWASN systems by having better conver- gence time and high Energy efficiency. With Multi-agent based Reinforcement Learning, faster learning and convergence can be achieved due to experience sharing among the agents. when one or more agents fail, which is synonymous to node(s) failure (inher- ent in Underwater Acoustic Sensor Networks), others agents take over some of their tasks thus, makes the system robust and good for “selfadaptability” and “self-organization”. Because of this full cooperation among the agents, the system will also allow easy insertion of new agents into the system without bringing the entire system down, this provide for high degree of scalability. It is understood from the literatures that valid and standard models for UWASNs do not exist, in order to realize the proposed tech- nique, an underwater pipeline infrastructure monitoring scenario is considered. The model of this scenario will be firstly developed based on the requirements for this study. This model shall be used as an application base on which the Fully Cooperative Multi-Agent Reinforcement Learning based Q-ALOHA MAC (FCQ-ALOHA MAC) is deployed. The development of the FCQ-ALOHA MAC will be achieved by designing fully cooperative mechanism struc- ture, Q-Learning algorithm initialization and Markov Decision Process (MDP) model.

4.1 Architecture

protocol of the proposed

MAC The block diagram of the proposed FCQ-ALOHA MAC protocol for sender node within cooperative nodes is shown in Figure 1.

Sensor nodes are modelled as frames and each frame divided into optimal number of slots. As fixed frame size estimation has been identified from the literatures to be a difficult task and frame size over or under estimation could lead to poor performances in- terms of QoS and energy efficiency, we therefore, will employ dy- namic frame size estimation. This will exclude the task of pre- allocation of frame sizes. At initial instance, Q-values for all the Cooperative Nodes (CNs) in the network are set to zero, (Q1,1, = Q1,2.... = Qi,n = 0), to create a complete random access transmission scenario, TxALOHA. Where Qi,nis the Q-value associ- ated with the nth slot of the ith frame and TxALOHA is the pure ALOHA transmission scenario. Optimal slot within the optimal frame will then be selected for data packet transmission. As ex- pected, this initial transmission scenario will have maximum data collision, however, the experience (reward calculated from Re- ward Function) from this transmission will be fed back to the Qvalue tables which will then inform a better transmission policy for next transmission. The algorithm is expected to converge within the shortest possible period of time depending on network size, node mobility and node density. After convergence, the CNs would have learn the Joint Policy, π, and on their own can take proper actions for future data transmission.

The flow chart depicting the proposed implementation strategy of data transmission based on FCQ-ALOHA MAC protocol is shown in Figure 3. 4.2

Expected Outcomes

The study is expected to develop an intelligent ALOHA based Media Access Control protocol for Underwater Acoustic Sensor Net- works for underwater pipeline monitoring. This will provide acceptable QoS performance in terms of throughput, delay, and en efficiency and in turn make UWASN systems more reliable, efficient and effective for various applications. A model of the underwater pipeline infrastructure monitoring system will also be developed, and this will be used as an application base for evaluating the developed MAC protocol. 5. CONCLUSION The challenges associated with underwater acoustic communications have made the deployment of UWASNs unpopular for potential applications in underwater operations. Medium Access Con- trol protocol is largely responsible for successful UWASNs development which is marred by challenges such as long and variable propagation delay, limited channel capacity, low data rates and en- ergy efficiency. The state of the art MAC protocols for UWASNs have room for improvement in terms of QoS and energy efficiency. In view of the above, a Fully Cooperative MultiAgent Q- learning based MAC protocol that would be “selforganized” and “self-adaptive” with improved performances in delay, throughput and energy efficiency is proposed here to provide solutions for the aforementioned challenges of UWASNs systems. It is expected that meaningful impact with respect to data transmission at MAC layer would be made in applications of UWASNs systems.

References

[1] J. Heidemann , Y. Wei , J. Wills , A. Syed , and L. Yuan . Research challenges and applications for underwater sensor networking . In IEEE Wireless Communications and Networking Conference , 2006 . WCNC 2006 ., 2006 .

[2] F. M. V. Luiz , P. David , and S. S. Viana . Hydronode: An underwater sensor node prototype for monitoring hydroelectric reservoirs . In WUWNET â12 , Los Angeles, California USA., 2012 .

[3] N. Thi-Tham and S. Yoon . Ars: An adaptive retransmission scheme for contention-based mac protocols in underwater acoustic sensor networks . International Journal of Distributed Sensor Networks , 15 , 2015 .

[4] Mohsin Murad , Adil A Sheikh, Muhammad Asif Manzoor, Emad Felemban, and Saad Qaisar . A survey on current underwater acous- tic sensor network applications . International Journal of Computer Theory and Engineering , 7 ( 1 ): 51 , 2015 .

[5] Christina Peach and A. Yarali . An overview of underwater sensor networks . In ICWMC 2013: The Ninth International Conference on Wireless and Mobile Communications , 2013 .

[6] I. F. Akyildiz , D. Pompili , and T. Melodia . Underwater acoustic sensor networks: research challenges . Ad Hoc Networks , 3 ( 2 ): 257 - 279 ., 2005 .

[7] J. Heidemann , M. Stojanovic , and M. Zorzi . Underwater sensor net- works: applications, advances and challenges . Philosophical Transac- tions of the Royal Society , 370 ( 1958 ): 158 - 175 , 2012 .

[8] L. Jin and D. Huang . A slotted csma based reinforcement learning ap- proach for extending the lifetime of underwater acoustic wireless sen- sor networks . Computer Communications , 36 ( 9 ): 1094â1099 ., 2013 .

[9] Fatih Senel , Kemal Akkaya, Melike Erol-Kantarci, and T. Yilmaz . Self-deployment of mobile underwater acoustic sensor networks for maximized coverage and guaranteed connectivity . Journal of Ad Hoc Networks , 34 : 170â183 ., 2014 .

[10] Chu Yi , Selahattin Kosunalp, Paul D. Mitchell , David Grace, and T. Clarke . Application of reinforcement Learning to medium access control for wireless sensor networks . Elsevier J. of Engineering Applications of Artificial Intelligence , 46 : 23 - 32 ., 2015 .

[11] R. S. Sutton and A. G. Barto. ( 1998 ). Reinforcement learning: An introduction . MIT press Cambridge, 1 edition .

[12] H. D. Li , D. Grace , and P. D. Mitchell . Cognitive radio multiple access control for unlicensed and open spectrum with reduced spectrum sensing requirements . In International Symposium on Wireless Communication Systems (ISWCS) , 2010 .

[13] J. G. Proakis , E.M. Sozer , J. A. Rice , and M. Stojanovic Shallow water acoustic networks . IEEE CommunicationsMagazine , 39 ( 11 ): 114â119 ., 2001 .

[14] J. Ahn , A. Syed , B. Krishnamachari , and J. Heidemann . Design and analysis of a propagation delay tolerant aloha protocol for underwater networks . Journal of Ad Hoc Networks , 9 ( 5 ): 752 - 766 ., 2011 .

[15] L. Vieira , J. Kong , U. Lee , and M. Gerla . Analysis of aloha protocols for underwater acoustic sensor networks . In 1st International Work- shop on Underwater Networks (WUWNet â06) , Los Angeles, Calif, USA., September 2006 .

[16] N. Chirdchoo , W.-S. Soh, and K. C. Chua . Alohabased mac proto- cols with collision avoidance for underwater acoustic networks . In 26th IEEE International Conference on Computer Communications (INFOCOM â07) , Anchorage, Alaska, USA„ 2007 .

[17] P. Mandal , S. De, and S. S. Chakraborty . A receiver synchronized slotted aloha for underwater wireless networks with imprecise propagation delay information . Journal of Ad Hoc Networks , 11 ( 4 ): 1443â1455 ., 2013 .

[18] D. Wang , X. Hu , F. Xu , H. Chen , and Y. Wu . Performance analysis of p-csma for underwater acoustic sensor networks . In OCEANS , Yeosu, Republic of Korea., May 2012 .

[19] D. Fang , Y. Li , and H. Huang L. Yin . A csma/cabased mac proto- col for underwater acoustic networks . In 6th International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM â10) , Chengdu, China., September 2010 .

[20] M. Molins and M. Stojanovic . Slotted fama: a mac protocol for underwater acoustic networks paper presented at the . In OCEANS 2006 , Asia Pacific , Singapore., 2007 .

[21] S. M. Smith , J. C. Park , and A. Neel . A peer-to-peer communication protocol for underwater acoustic communication paper presented at the . In MTS/IEEE Conference (OCEANS â97) , IEEE, Halifax, Canada, October 1997 .

[22] X. Guo , M. R. Frater , and M. J.Ryan. ( 2007 , may 2007 ). a propagation-delay tolerant collision avoidance protocol for underwater acoustic sensor networks paper presented at the . In Asia Pacific (OCEANS â06) , Singapore, 2007 .

[23] B. Peleato and M. Stojanovic . Distance aware collision avoidance protocol for ad-hoc underwater acoustic sensor networks . IEEE Communications Letters , 11 ( 12 ): 1025 - 1027 , 2007 .

[24] A. A. Syed , W. Ye , and J. Heidemann . T-lohi: a new class of mac protocols for underwater acoustic sensor networks . In 27th IEEE Communications Society Conference on Computer Communications (INFOCOM â08) , Phoenix, Ariz, USA, April 2008 .

[25] Z. Liao , D. Li , and J. Chen. ( 2015 ). a handshake based ordered scheduling mac protocol for underwater acoustic local area networks . International Journal of Distributed Sensor Networks , 2015 ( 15 ), 2015 .

[26] G. Fan , H. Chen , L. Xie , and K. Wang . A hybrid reservation-based mac protocol for underwater acoustic sensor networks . Journal of Ad Hoc Networks , 11 ( 3 ): 1178 - 1192 ., 2013 .

[27] Du Xiujuan , Peng Chunyan, Liu Xiuxiu, and Liu Yuchi. Hierarchical code assignment algorithm and state-based cdma protocol for uwsn . China Commun. (China Communications) , 12 ( 3 ): 50 - 61 , 2015 .

[28] Guangyu Fan , Huifang Chen, Lei Xie, and Kwang Wang . Poca-cdma mac: An improved cdma-based mac protocol for underwater acoustic wireless sensor networks . In Proceedings of IEEE International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM) , pages 1 - 4 , 2011 .

[29] Marcžal Molins and Milica Stojanovic . Slotted fama: a mac protocol for underwater acoustic networks . In Proceedings of IEEE OCEANSâ06 , pages 16 - 22 , Singapore, 2006 .

[30] Peng Xie and Junhong Cui . R-mac: An energy-efficient mac protocol for underwater sensor networks . In Proceedings of International Conference on Wireless Algorithms, Systems and Applications , pages 187 - 196 , 20

[31] S. Han , Y . Noh, U. Lee , and M. Gerla . M-fama: A multi-session mac protocol for reliable underwater acoustic streams . In in Proc. INFOCOM , pages 665 - 673 , 2013 .

[32] Chao Li , Yongju Xu, Chaonong Xu, Zhulin An , Diao Boyu , and Li Xiaowei . Dtmac: A delay tolerant mac protocol for underwater wireless sensor networks . In IEEE Sensors J. , 16 ( 11 ): 4137â4146 , 2016 .

[33] X. Liang , M. Chen , Y. Xiao , and V. C. M. Leung . Mrlcc: A novel cooperative communication protocol for qos provisioning in wireless sensor networks . International Journal of Sensor Networks , 8 ( 2 ): 98 - 108 , 2010 .

[34] Ning Li , José-Fernán Martínez , Juan Manuel Meneses Chaus, and Martina Eckert . A survey on underwater acoustic sensor network routing protocols . Sensors , 16 ( 3 ): 414 , 2016 .

[35] A. Nosratinia , T. E. Hunter , and A. Hedayat . Cooperative commu- nication in wireless networks . In IEEE Communications Magazine , volume 42 , pages 74 - 80 . October 2004 .

[36] Y. W. Hong , W. J. Haung , F. H. Chiu , and C. C. Kuo . Cooperative communication in resource constraint wireless networks . In IEEE Signal Processing Magazine , volume 24 , pages 47 - 57 . May 2007 .

[37] Keyu Chen , M. M. En Cheng, Fei Yuan, and W. Su . A survey on mac protocols for underwater wireless sensor networks . IEEE COMMUNICATIONS SURVEYS & TUTORIALS , 2014 .

[38] Jesús Llor and Manuel P Malumbres. Modelling underwater wireless sensor networks . Intech open , pages 1 - 20 , 2010 .