=Paper=
{{Paper
|id=Vol-1576/181
|storemode=property
|title=Numerical Hydrodynamics Simulation of Astrophysical Flows at Intel Xeon Phi supercomputers
|pdfUrl=https://ceur-ws.org/Vol-1576/181.pdf
|volume=Vol-1576
|authors=Igor Kulikov,Igor Chernykh,Eduard Vorobyov,Aleksey Snytnikov,Dmitriy Weins,Aleksandr Moskovsky,Aleksey Shmelev,Viktor Protasov,Aleksandr Serenko,Vladislav Nenashev,Vitaliy Vshivkov,Aleksey Rodionov,Boris Glinsky,Aleksandr Tutukov
}}
==Numerical Hydrodynamics Simulation of Astrophysical Flows at Intel Xeon Phi supercomputers==
<pdf width="1500px">https://ceur-ws.org/Vol-1576/181.pdf</pdf>
<pre>
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                                agora.guru.ru/pavt




       ×èñëåííîå ãèäðîäèíàìè÷åñêîå ìîäåëèðîâàíèå

àñòðîôèçè÷åñêèõ òå÷åíèé íà ãèáðèäíûõ ñóïåðÝÂÌ,
                                                                                              ∗
          îñíàùåííûõ óñêîðèòåëÿìè Intel Xeon Phi

                 1,2                  1,2                        3,4                    1,2          1
È.Ì. Êóëèêîâ          , È.Ã. ×åðíûõ     , Ý.È. Âîðîáüåâ              , À.Â. Ñíûòíèêîâ     , Ä.Â. Âèíñ ,
                       5                    5                        7              7                7
 À.À. Ìîñêîâñêèé , À.Á. Øìåëåâ , Â.À. Ïðîòàñîâ , À.À. Ñåðåíêî , Â.Å. Íåíàøåâ ,
                       1               1
           Â.À. Âøèâêîâ , À.Ñ. Ðîäèîíîâ , Á.Ì. Ãëèíñêèé
                                                       1,2, À.Â. Òóòóêîâ6

 Èíñòèòóò âû÷èñëèòåëüíîé ìàòåìàòèêè è ìàòåìàòè÷åñêîé ãåîôèçèêè Ñèáèðñêîãî
                       1                                                            2
  îòäåëåíèÿ ÐÀÍ            , Íîâîñèáèðñêèé ãîñóäàðñòâåííûé óíèâåðñèòåò , Department of
                                          3
        Astrophysics, University of Vienna , ÍÈÈ Ôèçèêè, Þæíûé Ôåäåðàëüíûé
                  4                             5
   Óíèâåðñèòåò , ÐÑÊ Òåõíîëîãèè , Èíñòèòóò àñòðîíîìèè ÐÀÍ , Íîâîñèáèðñêèé
                                                                                6
                                                                                7
                            ãîñóäàðñòâåííûé òåõíè÷åñêèé óíèâåðñèòåò



           ðàáîòå ïðåäñòàâëåíû èññëåäîâàíèÿ êîäà AstroPhi äëÿ ÷èñëåííîãî ìîäåëèðî-
          âàíèÿ àñòðîôèçè÷åñêèõ òå÷åíèé íà ãèáðèäíûõ ñóïåðÝÂÌ, îñíàùåííûõ óñêîðè-
          òåëÿìè Intel Xeon Phi. Îïèñàí ñî-äèçàéí âû÷èñëèòåëüíîé ìîäåëè äëÿ îïèñàíèÿ
          àñòðîôèçè÷åñêèõ îáúåêòîâ. Äåòàëüíî îïèñàíû îñîáåííîñòè ïàðàëëåëüíîé ðåà-
          ëèçàöèè è èññëåäîâàíèÿ ïðîèçâîäèòåëüíîñòè êîäà AstroPhi. Ïðåäñòàâëåíû ðå-
          çóëüòàòû ìîäåëèðîâàíèÿ âçàèìîäåéñòâèÿ ìåæãàëàêòè÷åñêîãî âåòðà è äèñêîâîé
          ãàëàêòèêè. Äëÿ êîäà AstroPhi áûëî äîñòèãíóòî 134-êðàòíîå óñêîðåíèå â ðàìêàõ
          îäíîãî óñêîðèòåëÿ Intel Xeon Phi, 75-ïðîöåíòíàÿ ìàñøòàáèðóåìîñòü ïðè èñïîëü-
          çîâàíèè 224 óñêîðèòåëåé Intel Xeon Phi. Íà ðàñ÷åòíîé ñåòêå 7168 × 1024 × 1024
          áûëî äîñòèãíóòî 47 ïðîöåíòîâ îò ïèêîâîé ñêàëÿðíîé ïðîèçâîäèòåëüíîñòè óñêî-
          ðèòåëÿ Intel Xeon Phi ïðè èñïîëüçîâàíèè 53760 íèòåé.
          Êëþ÷åâûå ñëîâà: Âûñîêîïðîèçâîäèòåëüíûå âû÷èñëåíèÿ, âû÷èñëèòåëüíàÿ àñòðî-
          ôèçèêà, óñêîðèòåëè Intel Xeon Phi.

1. Ââåäåíèå

      Ìàòåìàòè÷åñêîå ìîäåëèðîâàíèå èãðàåò êëþ÷åâóþ ðîëü â ñîâðåìåííîé àñòðîôèçèêå.
Îíî ÿâëÿåòñÿ óíèâåðñàëüíûì èíñòðóìåíòîì äëÿ èññëåäîâàíèÿ íåëèíåéíûõ ýâîëþöèîííûõ
ïðîöåññîâ âî Âñåëåííîé. Îäíèìè èç âàæíåéøèõ çàäà÷, ðåøàåìûõ âû÷èñëèòåëüíîé àñòðî-
ôèçèêîé, ÿâëÿþòñÿ çàäà÷è ñòîëêíîâåíèÿ [1] è ýâîëþöèè ãàëàêòèê [2], ïðîöåññû êîëëàïñà
çâåçä [3], õèìîêèíåòè÷åñêèå ïðîöåññû â ãàëàêòèêàõ [4]. Ïðè êîíñòðóèðîâàíèè ìàòåìàòè-
÷åñêîé ìîäåëè ñëåäóåò ó÷èòûâàòü äîñòèæåíèÿ ñîâðåìåííîé àñòðîíîìèè. Òàê àêòóàëüíûì
ÿâëÿåòñÿ ó÷åò ìàãíèòíîãî ïîëÿ â ãàëàêòèêàõ, òàê êàê åãî íàëè÷èå îáíàðóæåíî â ðóêàâàõ
ãàëàêòèêè M51 [5] è âëèÿåò íà ïðîöåññ çâåçäîîáðàçîâàíèÿ. Òàêèì îáðàçîì, èçó÷åíèå àñò-
ðîôèçè÷åñêèõ ïðîöåññîâ óñëîæíÿåòñÿ íåîáõîäèìîñòüþ ó÷åòà áîëüøîãî ÷èñëà ïîäñåòî÷íûõ
ôèçè÷åñêèõ ïðîöåññîâ. Êðîìå òîãî, ñîñòàâ àñòðîôèçè÷åñêèõ îáúåêòîâ ñîñòîèò èç íåñêîëü-
êèõ èíãðèäèåíòîâ, äëÿ îïèñàíèÿ êîòîðûõ èñïîëüçóþòñÿ ðàçëè÷íûå ìàòåìàòè÷åñêèå ìîäå-
ëè. Äàííîå îáñòîÿòåëüñòâî óñëîæíÿåò ðàçðàáîòêó ýôôåêòèâíûõ êîäîâ äëÿ èññëåäîâàíèÿ
àñòðîôèçè÷åñêèõ ïðîáëåì íà ñóïåðêîìïüþòåðàõ.
      Äëÿ ìîäåëèðîâàíèÿ ñëîæíûõ àñòðîôèçè÷åñêèõ ïðîöåññîâ â âûñîêîì ðàçðåøåíèè íåîá-
õîäèìî èñïîëüçîâàòü íàèáîëåå ìîùíûå ñóïåðêîìïüþòåðû. Äâà èç Top-3 (÷åòûðå èç Top-10)
ñóïåðêîìïüþòåðà â íîÿáðüñêîé âåðñèè 2015 ãîäà ñïèñêà Top-500 îñíàùåíû ãðàôè÷åñêèìè
óñêîðèòåëÿìè è óñêîðèòåëÿìè Intel Xeon Phi. Îæèäàåòñÿ, ÷òî ïåðâûé ñóïåðêîìïüþòåð ýê-

  ∗
    Ðàáîòà ïîääåðæàíà ãðàíòîì Ðîññèéñêîãî ôîíäà ôóíäàìåíòàëüíûõ èññëåäîâàíèé 15-31-20150 ìîë-à-âåä,
15-01-00508 è 16-07-00434, ãðàíòîì Ïðåçèäåíòà ÐÔ MK  6648.2015.9. Ðàáîòà âûïîëíåíà ïðè ÷àñòè÷íîé
ïîääåðæêå ïðîåêòíîé ÷àñòè ãîñçàäàíèÿ  3.961.2014/Ê Ìèíèñòåðñòâà Îáðàçîâàíèÿ è Íàóêè Ðîññèéñêîé
Ôåäåðàöèè (Ý.È. Âîðîáüåâ).



                                                      205
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



çàôëîïñíîé ïðîèçâîäèòåëüíîñòè áóäåò ïîñòðîåí íà îñíîâå ãèáðèäíîãî ïîäõîäà. Ðàçðàáîòêà
êîäîâ äëÿ ãèáðèäíûõ ñóïåðêîìïüþòåðîâ íå ñóãóáî òåõíè÷åñêàÿ çàäà÷à, à îòäåëüíàÿ ñëîæ-
íàÿ íàó÷íàÿ çàäà÷à, òðåáóþùàÿ ñî-äèçàéíà àëãîðèòìîâ íà âñåõ ñòàäèÿõ ðåøåíèÿ çàäà÷è 
îò ôèçè÷åñêîé ïîñòàíîâêè äî èíñòðóìåíòîâ ðàçðàáîòêè.
   Íåñìîòðÿ íà áîëüøîå ÷èñëî ðàçðàáîòàííûõ êîäîâ äëÿ ðåøåíèÿ àñòðîôèçè÷åñêèõ çà-
äà÷ [6] îñòàåòñÿ áîëüøîå ÷èñëî íåðåøåííûõ ïðîáëåì â îáëàñòè ìàòåìàòè÷åñêèõ ìîäåëåé,
÷èñëåííûõ ìåòîäîâ è ïðîãðàììíûõ ðåàëèçàöèé äëÿ èçó÷åíèÿ àñòðîôèçè÷åñêèõ òå÷åíèé. Àâ-
òîðñêèì êîëëåêòèâîì óæå íà ïðîòÿæåíèè íåñêîëüêèõ ëåò ðàçâèâàåòñÿ ãèáðèäíûé ýéëåðîâî-
ëàãðàíæåâûé ïîäõîä äëÿ ðåøåíèÿ àñòðîôèçè÷åñêèõ çàäà÷.  íàñòîÿùåé ñòàòüå áóäåò ïðè-
âåäåíî êðàòêîå îïèñàíèå è ïîäðîáíîå èññëåäîâàíèå îðèãèíàëüíîãî êîäà AstroPhi [7] äëÿ
ìîäåëèðîâàíèÿ äèíàìèêè àñòðîôèçè÷åñêèõ îáúåêòîâ.


2. Êîíöåïöèÿ ñî-äèçàéíà âû÷èñëèòåëüíîé ñõåìû

   Ãëàâíûé ôîêóñ íàøèõ èññëåäîâàíèé íàïðàâëåí íà ìîäåëèðîâàíèå äèíàìèêè ãàëàêòèê.
Ïîýòîìó ÷èñëåííàÿ ìîäåëü àñòðîôèçè÷åñêèõ òå÷åíèé îðèåíòèðîâàíà â îñíîâíîì íà îïèñà-
íèå êîìïîíåíò ãàëàêòèê è ïîäñåòî÷íûõ ïðîöåññîâ.  ðàáîòå [8] áûëè èññëåäîâàíû âîïðîñû
ñî-äèçàéíà ÷èñëåííûõ ìîäåëåé àñòðîôèçèêè è ôèçèêè ïëàçìû. Ðàñññìîòðèì îñíîâíûå ýòà-
ïû ñî-äèçàéíà ÷èñëåííûõ ìîäåëåé äëÿ ðåøåíèÿ àñòðîôèçè÷åñêèõ ïðîáëåì.


  1. Ýòàï ôîðìóëèðîâêè ôèçè÷åñêîé çàäà÷è. Ãëàâíûìè èíãðèäèåíòàìè ãàëàêòèê ÿâëÿåòñÿ
     ãàçîâàÿ êîìïîíåíòà, êîòîðàÿ îïèñûâàåò ìåæçâåçäíûé ãàç è ðàâíîìåðíî ðàñïðåäåëåí-
     íóþ ïûëü, è áåññòîëêíîâèòåëüíàÿ êîìïîíåíòû, êîòîðàÿ èñïîëüçóåòñÿ äëÿ îïèñàíèÿ
     çâåçäíîé êîìïîíåíòû è òåìíîé ìàòåðèè. Îñíîâíûìè ïîäñåòî÷íûìè ôèçè÷åñêèìè ïðî-
     öåññàìè ÿâëÿþòñÿ ïðîöåññû çâåçäîîáðàçîâàíèÿ, ýôôåêò îò âçðûâà ñâåðõíîâûõ, ôóíê-
     öèè îõëàæäåíèÿ è íàãðåâàíèÿ, à òàêæå õèìè÷åñêèå ðåàêöèè [9].


  2. Ýòàï ìàòåìàòè÷åñêîé ôîðìàëèçàöèè. Äëÿ îïèñàíèÿ ãàçîâîé êîìïîíåíòû èñïîëüçó-
     þòñÿ óðàâíåíèÿ ãðàâèòàöèîííîé ãàçîâîé äèíàìèêè, êîòîðûå ðàñøèðÿþòñÿ íà óðàâíå-
     íèÿ îäíîñêîðîñòíîé ìíîãîêìîïíåíòíîé ãðàâèòàöèîííîé ãàçîâîé äèíàìèêè ñ ýôôåê-
     òèâíûì ïîêàçàòåëåì àäèàáàòû â ñëó÷àå ó÷åòà õèìè÷åñêèõ ðåàêöèé. Äëÿ îïèñàíèÿ
     áåññòîëêíîâèòåëüíîé êîìïîíåíòû èñïîëüçóþòñÿ óðàâíåíèÿ äëÿ ïåðâûõ ìîìåíòîâ áåñ-
     ñòîëêíîâèòåëüíîãî óðàâíåíèÿ Áîëüöìàíà. Òàêîé ïîäõîä áûë èññëåäîâàí è óñïåøíî
     èñïîëüçîâàí äëÿ ðåøåíèÿ çàäà÷ ýâîëþöèè [2, 9] è ñòîëêíîâåíèÿ ãàëàêòèê [6, 10]. Òà-
     êîé ñïîñîá îïèñàíèÿ áåññòîëêíîâèòåëüíîé êîìïîíåíòû ïîçâîëÿåò ïîçâîëÿåò ñôîðìó-
     ëèðîâàòü òåðìîäèíàìè÷åñêè ñîãëàñîâàííóþ ìîäåëü çâåçäîîáðàçîâàíèÿ è ýôôåêòà îò
     âçðûâà ñâåðõíîâûõ.


  3. Ýòàï ïîñòðîåíèÿ ÷èñëåííîãî ìåòîäà ðåøåíèÿ. Îñîáåííîñòüþ ìàòåìàòè÷åñêîé ôîð-
     ìàëèçàöèè ÿâëÿåòñÿ îïèñàíèå ãàçîâîé è áåññòîëêíîâèòåëüíîé êîìïîíåíò ãàëàêòèê ñ
     ïîìîùüþ ñèñòåìû ãèïåðáîëè÷åñêèõ óðàâíåíèé. Òàêèì îáðàçîì, ìû ìîæåì ñôîðìóëè-
     ðîâàòü åäèíûé ÷èñëåííûé ìåòîä ðåøåíèÿ ãèïåðáîëè÷åñêèõ óðàâíåíèé.  ñëåäóþùåì
     ðàçäåëå ÷èñëåííûé ìåòîä áóäåò îïèñàí áîëåå ïîäðîáíî. Èñïîëüçîâàíèå åäèíîãî ÷èñ-
     ëåííîãî ìåòîäà ïîçâîëÿåò çàïèñàòü åäèíûé ïàðàëëåëüíûé àëãîðèòì. Â îñíîâå òàêîãî
     àëãîðèòìà ëåæèò ëîêàëüíîñòü âû÷èñëåíèé, ÷òî äîñòàòî÷íî ýôôåêòèâíî ïðîåöèðóåòñÿ
     íà ñîâðåìåííûå àðõèòåêòóðû ñóïåðêîìïüþòåðîâ.


  4. Ýòàï âûáîðà ñòðóêòóð äàííûõ. Èñïîëüçóåìûå ñòðóêòóðû äàííûõ â ñëó÷àå ðåøåíèÿ ãè-
     ïåðáîëè÷åñêèõ óðàâíåíèé ïîëíîñòüþ ñîãëàñóþòñÿ ñ âûáîðîì ðàñ÷åòíûõ ñåòîê.  îðè-
     ãèíàëüíîì ïîäõîäå èñïîëüçóþòñÿ ðåãóëÿðíûå ñåòêè, ÷òî ïîçâîëÿåò ñôîðìóëèðîâàòü
     ïðîñòîé ïîäõîä ê îðãàíèçàöèè ïàðàëëåëüíûõ âû÷èñëåíèé [11]. Îñíîâíûì èç òðåíäîâ
     ñîâðåìåííîãî ÷èñëåííîãî ðåøåíèÿ ãèïåðáîëè÷åñêèõ óðàâíåíèé ÿâëÿåòñÿ òåõíîëîãèÿ
     ïîäâèæíûõ ñåòîê.  ñëó÷àå èñïîëüçîâàíèÿ ðåãóëÿðíûõ ñòðóêòóð äàííûõ ìîæåò áûòü



                                               206
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



     îïèñàíà òåõíîëîãèÿ êîìôîðíûõ ïîäâèæíûõ ñåòîê, êîòîðàÿ ïîçâîëÿåò ýôôåêòèâíî ìî-
     äåëèðîâàòü áîëüøîå ÷èñëî çàäà÷ ìåõàíèêè ñïëîøíîé ñðåäû. Ïðè ýòîì ñîõðàíÿþòñÿ
     ïàðàëëåëüíûå àëãîðèòìû, èñïîëüçóåìûå äëÿ âû÷èñëåíèé íà ðåãóëÿðíûõ ñåòêàõ.  íà-
     ñòîÿùåå âðåìÿ, òàêàÿ òåõíîëîãèÿ ïîäâèæíûõ ñåòîê íå ðåàëèçîâàíà â êîäå AstroPhi,
     íî â ïåðñïåêòèâå òàêàÿ ðåàëèçàöèÿ ïëàíèðóåòñÿ.


  5. Ýòàï ó÷åòà àðõèòåêòóðû ñóïåðêîìïüþòåðà.  íàøèõ èññëåäîâàíèÿõ èñïîëüçóþòñÿ ãè-
     áðèäíûå ñóïåðêîìïüþòåðû ñ óñêîðèòåëÿìè Intel Xeon Phi. Ëîãè÷åñêàÿ àðõèòåêòóðû
     òàêîãî ñóïåðêîìïüþòåðà ïðåäñòàâëÿåòñÿ â âèäå ëèíåéêè óñêîðèòåëåé, âçàèìîäåéñòâó-
     þùèõ íàïðÿìóþ (â ñëó÷àå èñïîëüçîâàíèÿ native ðåæèìà) èëè ÷åðåç CPU (â ñëó÷àå
     èñïîëüçîâàíèÿ ooad ðåæèìà).  ðàìêàõ îäíîãî óñêîðèòåëÿ âû÷èñëåíèÿ ðàçáèâàþò-
     ñÿ íà áîëüøîå (íåñêîëüêî ñîòåí) íèòåé. Îðãàíèçàöèÿ âû÷èñëåíèé â îðèãèíàëüíîì
     ìåòîäå ïîçâîëÿåò èñêëþ÷èòü âçàèìîäåéñòâèå ìåæäó íèòÿìè â ðàìêàõ îäíîãî óñêîðè-
     òåëÿ íà îñíîâíûõ ýòàïàõ ìåòîäà, ëèáî ñâîäèòü òàêèå âû÷èñëåíèÿ ê ìèíèìóìó. Òàêîå
     âçàèìîäåéñòâèå âîçíèêàåò â ñëó÷àå âû÷èñëåíèÿ øàãà ïî âðåìåíè èç óñëîâèÿ Êóðàíòà.


  6. Ýòàï èñïîëüçîâàíèÿ ñðåäñòâ ðàçðàáîòêè. Îðãàíèçàöèÿ âû÷èñëåíèé îðèãèíàëüíîãî ÷èñ-
     ëåííîãî ìåòîäà è àðõèòåêòóðû èñïîëüçóåìûõ ñóïåðêîìïüþòåðîâ ïîçâîëÿþò íàì îãðà-
     íè÷èòüñÿ áèáëèîòåêîé MPI äëÿ îðãàíèçàöèè ìåæïðîöåññíûõ âçàèìîäåéñòâèé è òåõ-
     íîëîãèåé OpenMP äëÿ îðãàíèçàöèè ìíîãîïîòî÷íûõ âû÷èñëåíèé.


 ñëåäóþùåì ðàçäåëå áóäåò ïîäðîáíåå îïèñàíà ðåàëèçàöèÿ êàæäîãî ýòàïà.


3. Êîä AstroPhi

   Äëÿ îïèñàíèÿ ãàçîâîé êîìïîíåíòû áóäåì èñïîëüçîâàòü ñèñòåìó óðàâíåíèé îäíîñêîðîñò-
íîé ìíîãîêîìïîíåíòíîé ãðàâèòàöèîííîé ãàçîâîé äèíàìèêè, çàïèñàííóþ â ýéëåðîâûõ êîîð-
äèíàòàõ:
                                    ∂ρ
                                        + ∇ · (ρ⃗u) = S − D,
                                    ∂t
                              ∂ρi                         ρi  ρi
                                  + ∇ · (ρi ⃗u) = −sij + S − D ,
                              ∂t                          ρ   ρ
                          ∂ρ⃗u
                               + ∇ · (ρ⃗u⃗u) = −∇p − ρ∇(Φ) + ⃗v S − ⃗uD,
                           ∂t
               ∂ρE                                                     S   D
                   + ∇ · (ρE⃗u) = −∇ · (p⃗v ) − (ρ∇(Φ), ⃗u) − Λ + Γ + ε − ε ,
                ∂t                                                     ρ   ρ
                    ∂ρε                                            S   D
                        + ∇ · (ρϵ⃗u) = −(γ − 1)ρϵ∇ · ⃗u − Λ + Γ + ε − ε ,
                     ∂t                                            ρ   ρ
                                               1
                                         ρE = ρ⃗u2 + ρε,
                                               2
                                          p = (γ − 1)ρε,
Äëÿ îïèñàíèÿ áåññòîëêíîâèòåëüíîé êîìïîíåíòû áóäåì èñïîëüçîâàòü ñèñòåìó óðàâíåíèé äëÿ
ïåðâûõ ìîìåíòîâ áåññòîëêíîâèòåëüíîãî óðàâíåíèÿ Áîëüöìàíà, çàïèñàííóþ òàêæå â ýéëå-
ðîâûõ êîîðäèíàòàõ:
                                     ∂n
                                        + ∇ · (n⃗v ) = D − S,
                                     ∂t
                       ∂n⃗v
                            + ∇ · (n⃗v⃗v ) = −∇Π − n∇(Φ) + ⃗uD − ⃗v S,
                        ∂t
                   ∂ρW                                               D   S
                       + ∇ · (ρW⃗v ) = −∇ · (Π⃗v ) − (n∇(Φ), ⃗v ) + ε − ε ,
                    ∂t                                               ρ   ρ


                                               207
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



                          ∂Πξξ                                D    S
                               + ∇ · (Πξξ ⃗v ) = −2Π∇ · ⃗u + ε − ε ,
                           ∂t                                 3ρ   3ρ
                                      1          Πxx + Πyy + Πzz
                               ρW = ρ⃗v 2 +                      ,
                                      2                 2
Óðàâíåíèå Ïóàññîíà äëÿ îáåèõ êîìïîíåíò çàïèñûâàåòñÿ â âèäå:

                                        ∆Φ = 4πG(ρ + n),
ãäå p  äàâëåíèå ãàçà, ρi  ïëîòíîñòü i êîìïîíåíòû ñìåñè ãàçà, sij  ñêîðîñòü ïðîõîæ-
                                    ∑
äåíèÿ õèìè÷åñêèõ ðåàêöèé, ρ =         i ρi  ïëîòíîñòü ñìåñè ãàçà, n  ïëîòíîñòü áåññòîëê-
íîâèòåëüíîé êîìïîíåíòû, ⃗    u  ñêîðîñòü ãàçîâîé êîìïîíåíòû, ⃗v  ñêîðîñòü áåññòîëêíîâè-
òåëüíîé êîìïîíåíòû, ρE  ïëîòíîñòü ïîëíîé ìåõàíè÷åñêîé ýíåðãèè ãàçà, ρW  ïëîòíîñòü
ïîëíîé ìåõàíè÷åñêîé ýíåðãèè áåññòîëêíîâèòåëüíîé êîìïîíåíòû, Φ  ãðàâèòàöèîííûé ïî-
òåíöèàë, ε  ïëîòíîñòü âíóòðåííåé ýíåðãèè ãàçà, γ  ýôôåêòèâíûé ïîêàçàòåëü àäèàáàòû,
Πξξ = (Πxx , Πyy , Πzz )  äèàãîíàëüíûé òåíçîð äèñïåðñèè ñêîðîñòåé áåññòîëêíîâèòåëüíîé
êîìïîíåíòû, S  ñêîðîñòü îáðàçîâàíèÿ ñâåðõíîâûõ çâåçä, D  ñêîðîñòü çâåçäîîáðàçîâàíèÿ,
Λ  ôóíêöèÿ îõëàæäåíèÿ, Γ  ôóíêöèÿ íàãðåâàíèÿ îò âçðûâà ñâåðõíîâûõ çâåçä. Ìû íå
áóäåì ââîäèòü ïîäðîáíîñòè îïèñàíèÿ êàæäîãî òåðìà äëÿ îïèñàíèÿ ïîäñåòî÷íîé ôèçèêè,
òàê ïîäðîáíîñòè èõ ïðèìåíåíèÿ ìîãóò áûòü íàéäåíû â ðàáîòàõ [4, 9, 12].


3.1. Îïèñàíèå ÷èñëåííîãî ìåòîäà


      Äëÿ ÷èñëåííîãî ðåøåíèÿ óðàâíåíèé ãðàâèòàöèîííîé ãàçîâîé äèíàìèêè áûë èñïîëüçî-
âàí îðèãèíàëüíûé ÷èñëåííûé ìåòîä, îñíîâàííûé íà êîìáèíàöèè ìåòîäà Ãîäóíîâà, ìåòîäà
ðàçäåëåíèÿ îïåðàòîðîâ è êóñî÷íî-ïàðàáîëè÷åñêîãî ìåòîäà íà ëîêàëüíîì øàáëîíå äëÿ îáåñ-
ïå÷åíèÿ âûñîêîãî ïîðÿäêà òî÷íîñòè [13, 14]. Ñèñòåìà óðàâíåíèé ðåøàåòñÿ â äâà ýòàïà: ýé-
ëåðîâ, íà êîòîðîì ðåøàþòñÿ óðàâíåíèÿ áåç àäâåêòèâíûõ ÷ëåíîâ, è ëàãðàíæåâ, íà êîòîðîì
ïðîèñõîäèò àäâåêòèâíûé ïåðåíîñ ãèäðîäèíàìè÷åñêèõ âåëè÷èí. Íà ýéëåðîâîì ýòàïå ãèäðî-
äèíàìè÷åñêèå óðàâíåíèÿ äëÿ îáåèõ êîìïîíåíò çàïèñûâàþòñÿ â íåêîíñåðâèòèâíîé ôîðìå è
èñêëþ÷àþòñÿ àäâåêòèâíûå ÷ëåíû.  ðåçóëüòàòå òàêàÿ ñèñòåìà íà èíòåðôåéñå äâóõ ÿ÷ååê
èìååò àíàëèòè÷åñêîå ðåøåíèå, êîòîðîå èñïîëüçóåòñÿ äëÿ çàïèñè ïîòîêîâ ÷åðåç èíòåðôåéñ
äâóõ ÿ÷ååê [15]. Äëÿ ïîâûøåíèÿ ïîðÿäêà òî÷íîñòè èñïîëüçóåòñÿ êóñî÷íî-ïàðàáîëè÷åñêèé
ìåòîä íà ëîêàëüíîì øàáëîíå (PPML), êîòîðûé ñîñòîèò â ïîñòðîåíèè ëîêàëüíûõ ïàðàáîë
âíóòðè ÿ÷ååê äëÿ êàæäîé ãèäðîäèíàìè÷åñêîé âåëè÷èíû. Ãëàâíîå îòëè÷èå PPML îò êëàñ-
ñè÷åñêîãî PPM ìåòîäà ñîñòîèò â èñïîëüçîâàíèè ëîêàëüíîãî øàáëîíà äëÿ âû÷èñëåíèé. Ýòî
ïîçâîëÿåò íà ýòàïå ïàðàëëåëüíîé ðåàëèçàöèè, â îñíîâå êîòîðîé ãåîìåòðè÷åñêàÿ äåêîìïîçè-
öèÿ ðàñ÷åòíîé îáëàñòè, èñïîëüçîâàòü òîëüêî îäèí ñëîé ïåðåêðûòèÿ ïîäîáëàñòåé, ÷òî óïðî-
ùàåò ðåàëèçàöèþ ãðàíè÷íûõ óñëîâèé è óìåíüøàåò êîëè÷åñòâî ïåðåñûëîê, ñëåäîâàòåëüíî
ñïîñîáñòâóåò ðîñòó ýôôåêòèâíîñòè ïàðàëëåëüíîé ðåàëèçàöèè. Íà ëàãðàíæåâîì ýòàïå èñ-
ïîëüçóåòñÿ àíàëîãè÷íûé ÷èñëåííûé ïîäõîä. Íà äàííûé ìîìåíò ðåøåíèå óðàâíåíèÿ Ïóàñ-
ñîíà îñíîâàíî íà Fast Fourier Transform ìåòîäå. Ýòî ñâÿçàíî ñ òåì, ÷òî ðåøåíèå óðàâíåíèÿ
Ïóàññîíà çàíèìàåò íåñêîëüêî ïðîöåíòîâ îò âðåìåíè ñ÷åòà, íî â äàëüíåéøåì ìû ïëàíè-
ðóåì ïåðåéòè ê èòåðàöèîííûì ìåòîäàì ðåøåíèÿ òàêèì êàê SOR è CGM. Ïîñëå ðåøåíèÿ
óðàâíåíèÿ Ïóàññîíà è ãèäðîäèíàìè÷åñêèõ óðàâíåíèé ïðîèñõîäèò êîððåêòèðîâêà ðåøåíèÿ
overdenition ñèñòåìû óðàâíåíèé, äëÿ ýòîãî èñïîëüçóåòñÿ îðèãèíàëüíàÿ ïðîöåäóðà äëÿ ñî-
õðàíåíèÿ ïîëíîé ýíåðãèè ñèñòåìû è ãàðàíòèè íåóáûâàíèÿ ýíòðîïèè [16, 17]. Â ðåçóëüòà-
òå ðàçðàáîòàííûé ÷èñëåííûé ìåòîä ðåøåíèÿ îáëàäàåò ñëåäóþùèìè ñâîéñòâàìè: âûñîêèé
ïîðÿäîê òî÷íîñòè íà ãëàäêèõ ðåøåíèÿõ è ìàëàÿ äèññèïàöèÿ â ñëó÷àå ðàçðûâíûõ ðåøå-
íèé; îòñóòñòâèå íåîáõîäèìîñòè ââåäåíèÿ ÷ëåíà èñêóññòâåííîé âÿçêîñòè èëè îãðàíè÷èòå-
ëåé; èíâàðèàíòíîñòü ïîëó÷àåìîãî ÷èñëåííîãî ðåøåíèÿ îòíîñèòåëüíî ïîâîðîòà è îòñóòñòâèå
êàðáóíêóë-ýôôåêòîâ; ãàðàíòèðîâàííîå íåóáûâàíèå ýíòðîïèè; âîçìîæíîñòü ðàñøèðåíèÿ íà
áîëåå ñëîæíûå ãèäðîäèíàìè÷åñêèå ìîäåëè; ïðîñòîòà ïðîãðàììíîé ðåàëèçàöèè; ïîòåíöèàëü-
íî áåñêîíå÷íàÿ ìàñøòàáèðóåìîñòü. Ïîñëåäíèé ïóíêò íàì íàèáîëåå âàæåí è îñíîâàí íà òîì



                                               208
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



ôàêòå, ÷òî âñå âû÷èñëåíèÿ â ÿ÷åéêàõ ïðîèñõîäÿò íåçàâèñèìî, ðåãóëÿðíî è íà ëîêàëüíîì
øàáëîíå. ×èñëåííûé ìåòîä áûë ïðîòåñòèðîâàí íà ñëåäóþùèõ çàäà÷àõ:

  1. îäíîìåðíûå òåñòû Ãîäóíîâà î ðàñïàäå ðàçðûâà,

  2. îäíîìåðíûé òåñò Àêñåíîâà ñ íåïðåðûâíûì ïåðèîäè÷åñêèì ðåøåíèåì,

  3. çàäà÷à Ñåäîâà î òî÷å÷íîì âçðûâå,

  4. äâóìåðíàÿ íåóñòîé÷èâîñòü Ðåëåÿ-Òåéëîðà,

  5. äâóìåðíàÿ íåóñòîé÷èâîñòü Êåëüâèíà-Ãåëüìãîëüöà,

  6. çàäà÷à êîëëàïñà Ýâðàðäà.

Ïîäðîáíîå îïèñàíèå ÷èñëåííîãî ìåòîäà è åãî âåðèôèêàöèÿ ïðèâåäåíà â ðàáîòå [18]. Òàêæå
ðàçðàáîòàíî ðàñøèðåíèå ÷èñëåííîãî ìåòîäà íà ðåøåíèå ÌÃÄ óðàâíåíèé [19].


3.2. Äåêîìïîçèöèÿ ðàñ÷åòíîé îáëàñòè


   Ñî-äèçàéí [8] ôèçèêî-ìàòåìàòè÷åñêîé ìîäåëè, ÷èñëåííîãî ìåòîäà è ñòðóêòóð äàííûõ
ïîçâîëÿåò èñïîëüçîâàòü ãåîìåòðè÷åñêóþ äåêîìïîçèöèþ ðàñ÷åòíîé îáëàñòè ñ îäíèì ñëîåì
ïåðåêðûòèÿ ïîäîáëàñòåé. Òàêóþ âîçìîæíîñòü ìû èìååì çà ñ÷åò ïîñòðîåíèÿ ïàðàáîë íà
ïðåäûäóùåì øàãå, ÷òî òðåáóåò òîëüêî ëîêàëüíîãî âçàèìîäåéñòâèÿ ìåæäó ÿ÷åéêàìè. Íà
ðèñóíêå (1) ïðèâåäåíû ïðîöåíòíûå ñîîòíîøåíèÿ ìåæäó ýòàïàìè Äëÿ ðåøåíèÿ óðàâíåíèÿ




                       The Lagrangian Stage

                                     80%




                                                               Poisson Solver




                                                                   6%
                                                              4%
                                                   10%
                                                              Other
                                  The Eulerian Stage




                Ðèñ. 1. Ïðîöåíòíîå ñîîòíîøåíèå ìåæäó ýòàïàìè â êîäå AstroPhi
Ïóàññîíà, â îñíîâå êîòîðîãî áûñòðîå ïðåîáðàçîâàíèå Ôóðüå äëÿ ñóïåðÝÂÌ ñ ðàñïðåäå-
ëåííîé ïàìÿòüþ áûëà èñïîëüçîâàíà áèáëèîòåêà FFTW [20]. Â îñíîâå ýòîé áèáëèîòåêè ëå-
æèò ïðîöåäóðà ALLTOALL, êîòîðàÿ òðàíñïîíèðóåò òðåõìåðíûé ìàññèâ, ïåðåðàñïðåäåëÿÿ
çíà÷èòåëüíûå îáúåìû ïàìÿòè ìåæäó âñåìè ïðîöåññàìè. Áåçóñëîâíî, ýòî äîðîãàÿ ñåòåâàÿ
îïåðàöèÿ, êîòîðàÿ òðåáóåò îòêàçà îò âñåãî àëãîðèòìà â ñëó÷àå èñïîëüçîâàíèè ñêîëü ëèáî
çíà÷èòåëüíîãî êîëè÷åñòâà âû÷èñëèòåëåé. Îäíàêî, ýòà ïðîöåäóðà â ñëó÷àå èñïîëüçîâàíèÿ
ñåòåâîé èíôðàñòðóêòóðû InniBand íå çàíèìàåò êðèòè÷åñêîå âðåìÿ è, ïî âñåé âèäèìîñòè,
îïòèìèçèðîâàíà íà íèçêîì ñåòåâîì óðîâíå [21].
   Îñíîâíûìè ýòàïàìè âû÷èñëèòåëüíîé ñõåìû ÿâëÿþòñÿ ýéëåðîâ è ëàãðàíæåâ ýòàïû. Ìû
ñîñðåäîòî÷èìñÿ èìåííî íà ýòèõ ýòàïàõ, êàê íà íàèáîëåå çàòðàòíûõ. Òàêæå âíå íàøåãî ðàñ-
ñìîòðåíèÿ â ïëàíå óñêîðåíèÿ îñòàíóòñÿ ïðîöåäóðû, â êîòîðûõ äåëåíèå èìïóëüñà íà ôóíê-
öèþ ïëîòíîñòè è ïåðåçàïèñü ìàññèâîâ.  ýòèõ ïðîöåäóðàõ ôàêòè÷åñêè ïðîèñõîäèò êîïèðî-
âàíèå ïàìÿòè èç îäíîé îáëàñòè â äðóãóþ, â äàëüíåéøåì ìû òàêæå ðàññìîòðèì ýòè îïåðàöèè
îòäåëüíî ñ òî÷êè çðåíèÿ îáîáùåííîé ôóíêöèè MEMCPY.



                                               209
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



   Îòäåëüíî îñòàíîâèìñÿ íà ïðîöåäóðå âû÷èñëåíèÿ øàãà ïî âðåìåíè, èñõîäÿ èç óñëîâèÿ
Êóðàíòà.  ñëó÷àå èñïîëüçîâàíèÿ ãðàôè÷åñêèõ óñêîðèòåëåé äàííàÿ ïðîöåäóðà áûëà ðåà-
ëèçîâàíà òîëüêî íà CPU [6] (òàêæå áûëî ñäåëàíî è â êîäå GAMER [22]). Ïðè÷èíà ýòîãî 
îòñóòñòâèå ýôôåêòèâíîé ðåàëèçàöèè ðåäóöèðóþùåé îïåðàöèåé min â òåõíîëîãèè CUDA. Â
òî âðåìÿ êàê â OpenMP òàêàÿ îïåðàöèÿ ýôôåêòèâíî ðåàëèçîâàíà. Ñòîèìîñòü ýòîé ïðîöå-
äóðû ñîñòàâëÿåò ïîðÿäêà îäíîãî ïðîöåíòà îò îáùåãî âðåìåíè âû÷èñëåíèé è ïðàêòè÷åñêè
íå âëèÿåò íà ýôôåêòèâíîñòü ïàðàëëåëüíîé ðåàëèçàöèè. Îäíàêî, ïðè óâåëè÷åíèè êîëè÷å-
ñòâà ãðàôè÷åñêèõ ÿäåð äî íåñêîëüêèõ òûñÿ÷ è ñòîêðàòíîãî óñêîðåíèÿ â ðàìêàõ îäíîãî
ãðàôè÷åñêîãî ïðîöåññîðà ñóììàðíî âñåõ îñòàëüíûõ ïðîöåäóð, ìîæåò âîçíèêíóòü êóðüåç-
íàÿ ñèòóàöèÿ, êîãäà ïðîöåäóðà âû÷èñëåíèÿ øàãà ïî âðåìåíè áóäåò âûïîëíÿòüñÿ äîëüøå
âñåõ îñòàëüíûõ. Ïðè òîì, ÷òî àâòîðàìè óæå áûëî äîñòèãíóòî 55-êðàòíîå óñêîðåíèå â ðàì-
êàõ îäíîãî GPU [6] è êîëè÷åñòâî ãðàôè÷åñêèõ ÿäåð â îäíîì óñêîðèòåëå óâåëè÷èâàåòñÿ, òî
òàêàÿ ñèòóàöèÿ ìîæåò áûòü äîñòèãíóòà â áëèæàéøèå ïàðó ëåò. Ñòîèò îòìåòèòü, ÷òî òàêàÿ
ïðîáëåìà â ïðèíöèïå íåâîçìîæíà íà óñêîðèòåëÿõ Intel Xeon Phi.
   Èñïîëüçîâàíèå ðàâíîìåðíîé ñåòêè â äåêàðòîâûõ êîîðäèíàòàõ äëÿ ðåøåíèÿ óðàâíåíèé
ãèäðîäèíàìèêè ïîçâîëÿåò èñïîëüçîâàòü ïðîèçâîëüíóþ äåêàðòîâó òîïîëîãèþ äëÿ äåêîìïî-
çèöèè ðàñ÷åòíîé îáëàñòè. Òàêàÿ îðãàíèçàöèÿ âû÷èñëåíèé èìååò ïîòåíöèàëüíî áåñêîíå÷íóþ
ìàñøòàáèðóåìîñòü. Â êîäå AstroPhi èñïîëüçóåòñÿ ìíîãîóðîâíåâàÿ îäíîìåðíàÿ äåêîìïîçè-
öèÿ ðàñ÷åòíîé îáëàñòè. Ïî îäíîé êîîðäèíàòå âíåøíåå îäíîìåðíîå ðàçðåçàíèå ïðîèñõîäèò
ñðåäñòâàìè òåõíîëîãèè MPI, âíóòðè êàæäîé ïîäîáëàñòè ðàçðåçàíèå ïðîèñõîäèò ñðåäñòâàìè
OpenMP, àäàïòèðîâàííîãî äëÿ MIC-àðõèòåêòóð (ðèñ. 2).




                  Ðèñ. 2. Ñõåìà ãåîìåòðè÷åñêîé äåêîìïîçèöèè â êîäå AstroPhi
   Òàêîé ïîäõîä èñïîëüçîâàëñÿ òàêæå è â ïåðâîé âåðñèè ïðîãðàììíîãî êîäà AstroPhi [7]
c ó÷åòîì èñïîëüçîâàíèÿ ooad ðåæèìà. Òàêàÿ äåêîìïîçèöèÿ ñâÿçàíà ñ òîïîëîãèåé è àðõè-
òåêòóðîé ãèáðèäíîãî ñóïåðÝÂÌ RSC PetaStream, êîòîðûé áûë èñïîëüçîâàí äëÿ âû÷èñëè-
òåëüíûõ ýêñïåðèìåíòîâ.




                                               210
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



3.3. Øàáëîíû ïðîãðàììèðîâàíèÿ äëÿ Intel Xeon Phi


   Äëÿ èñïîëüçîâàíèÿ Intel Xeon Phi èñïîëüçîâàí ðåãóëÿðíûé øàáëîí âû÷èñëåíèé, êîòî-
ðûé ñëåäóåò èç ñõåìû äåêîìïîçèöèè ðàñ÷åòíîé îáëàñòè è ñîñòîèò â ðàñïðåäåëåíèè ðàáîò
ïî íèòÿì (ñì. ðèñ. 3). Â ëèñòèíãå ïðèâåäåíà çàãîòîâêà äëÿ èñïîëüçîâàíèÿ ooad ðåæèìà


...
// Offload / Native mode
#define NATIVE /* OFFLOAD */
// Number of MIC - threads
#define MIC_NUM_THREADS 240
...
#ifdef OFFLOAD
#pragma offload_attribute ( push , target ( mic ))
#endif
double foo ( double *a , double x , int index )
{
  return a[ index ] * x ;
}
#ifdef OFFLOAD
#pragma offload_attribute ( pop )
#endif
...
#ifdef OFFLOAD
#pragma offload target ( mic ) in (a: length (N )) \
           out (c: length (N ))
#endif
{
  # pragma omp parallel for default ( none ) shared (a ,x ,c) \
             num_threads ( MIC_NUM_THREADS )
  for (i =0; i <N;i ++)
  c[i] = foo (a ,x ,i );
}
...

                       Ðèñ. 3. Ïàòòåðí äëÿ ïðîöåäóðû íà Intel Xeon Phi
èñïîëüçîâàíèÿ óñêîðèòåëÿ Intel Xeon Phi, àíàëîãè÷íûé ïîäõîäó èñïîëüçóåìîìó â ðàáîòå [7].


3.4. Øàáëîíû ñåòåâûõ âçàèìîäåéñòâèé


   Ìåæïðîöåññíîå âçàèìîäåéñòâèå ñðåäñòâàìè MPI îñóùåñòâëÿåòñÿ ñ ïîìîùüþ øàáëîíà
ïåðåäà÷è ïî äâóíàïðàâëåííîìó ñïèñêó (ñì. ðèñ. 4) êðàéíèõ ýëåìåíòîâ îäíîìåðíîãî ìàññèâà
ðàçìåðîì    N ýëåìåíòîâ. Óêàçàííûé øàáëîí ÿâëÿåòñÿ î÷åíü ïðîñòûì, îäíàêî èìåííî íà
íåì ïîñòðîåíû áîëåå ñëîæíûå ìåæïðîöåññíûå âçàèìîäåéñòâèÿ îáìåíà ñðåçàìè òðåõìåðíûõ
ìàññèâîâ.


4. Èññëåäîâàíèå ïðîèçâîäèòåëüíîñòè

   Äëÿ ýêñïåðèìåíòîâ áûëè èñïîëüçîâàíû äâà ãèáðèäíûõ ñóïåðêîìïüþòåðà íà îñíîâå àð-
õèòåêòóðû RSC PetaStream: ÌÂÑ-10Ï ÌÑÖ ÐÀÍ (64 óñêîðèòåëÿ Intel Xeon Phi 7120 D) è
Ïîëèòåõíèê RSC PetaStream ÑÏáÏÓ (256 óñêîðèòåëÿ Intel Xeon Phi 5120 D). Äàëåå ïðèâå-
äåì èññëåäîâàíèÿ ïðîèçâîäèòåëüíîñòè ðàçëè÷íûõ ïîäñèñòåì êîäà: èññëåäîâàíèå óñêîðåíèÿ,
ìàñøòàáèðóåìîñòè, èìèòàöèîííîãî ìîäåëèðîâàíèÿ ìàñøòàáèðóåìîñòè, ïðîïóñêíîé ñïîñîá-
íîñòè ïàìÿòè è ñêîðîñòü ñåòåâûõ êîììóíèêàöèé. Â íàøèõ èññëåäîâàíèÿõ âîïðîñû ìàñøòà-
áèðóåìîñòè è óñêîðåíèÿ áûëè èññëåäîâàíû íà îáåèõ àðõèòåêòóðàõ, âîïðîñû ñâÿçàííûå ñ
îðãàíèçàöèåé âû÷èñëåíèé áûëè ïðîâåäåíû íà ñóïåðêîìïüþòåðå ÌÂÑ-10Ï ÌÑÖ ÐÀÍ.




                                               211
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



#define COMM MPI_COMM_WORLD
#define STATUS MPI_STATUS_IGNORE
#define TR 1 // " to right " communications
#define TL 2 // " to left " communications
...
i f ( rank == 0)
{ buffer [0] = a[N -2];
  MPI_Send ( buffer ,1 , MPI_DOUBLE , rank +1 , TR , COMM );
  MPI_Recv ( buffer ,1 , MPI_DOUBLE , rank +1 , TL , COMM , STATUS );
  a[N -1] = buffer [0]; }
if( rank == size -1)
{ MPI_Recv ( buffer ,1 , MPI_DOUBLE , rank -1 , TR , COMM , STATUS );
  a [0] = buffer [0];
  buffer [0] = a [1];
  MPI_Send ( buffer ,1 , MPI_DOUBLE , rank -1 , TL , COMM ); }
if( rank !=0 && rank != size -1)
{ MPI_Recv ( buffer ,1 , MPI_DOUBLE , rank -1 , TR , COMM , STATUS );
  a [0] = buffer [0];
  buffer [0] = a[N -2];
  MPI_Send ( buffer ,1 , MPI_DOUBLE , rank +1 , TR , COMM );
  MPI_Recv ( buffer ,1 , MPI_DOUBLE , rank +1 , TL , COMM , STATUS );
  a[N -1] = buffer [0];
  buffer [0] = a [1];
  MPI_Send ( buffer ,1 , MPI_DOUBLE , rank -1 , TL , COMM ); }
...

                            Ðèñ. 4. Ïàòòåðí äëÿ MPI êîììóíèêàöèé

4.1. Èññëåäîâàíèå óñêîðåíèÿ


     Â ñèëó ðàçíîãî îáúåìà ïàìÿòè íà óñêîðèòåëÿõ Intel Xeon Phi 7120 D èññëåäîâàíèå óñêî-
                                  3
ðåíèÿ ïðîâîäèëîñü íà ñåòêå 512 , íà óñêîðèòåëÿõ Intel Xeon Phi 5120 D áûëà èñïîëüçîâàíà
             2
ñåòêà 512×256 . Ýòî ìàêñèìàëüíûå ðàçìåðû ñåòîê, êîòîðûå ìîãóò ïîìåñòèòüñÿ â îäèí óñêî-
ðèòåëü. Äëÿ èçìåðåíèÿ óñêîðåíèÿ çàìåðÿëîñü âðåìÿ êàæäîãî ýòàïà ÷èñëåííîãî ìåòîäà, â
ñåêóíäàõ, à çàòåì âû÷èñëÿëàñü èõ ñóììà ïðè ðàçëè÷íîì ÷èñëå èñïîëüçóåìûõ ëîãè÷åñêèõ
ÿäåð (Threads). Óñêîðåíèå P (SpeedUp) âû÷èñëÿëîñü ïî ôîðìóëå (1):


                                                 T otal1
                                           P =                                                   (1)
                                                 T otalK
ãäå T otal1  âðåìÿ âû÷èñëåíèé íà îäíîì ëîãè÷åñêîì ÿäðå, T otalK  âðåìÿ âû÷èñëåíèé ïðè
èñïîëüçîâàíèè K ëîãè÷åñêèõ ÿäåð. Ðåçóëüòàòû èññëåäîâàíèé óñêîðåíèÿ äëÿ ñóïåðêîìïüþ-
òåðà ÌÂÑ-10Ï ÌÑÖ ÐÀÍ (JSCC) è Ïîëèòåõíèê RSC PetaStream ÑÏáÏÓ (SPb) ïðèâåäåíû
íà ðèñóíêå (5). Òàêèì îáðàçîì, áûëî ïîëó÷åíî 134-êðàòíîå óñêîðåíèå (ìàñøòàáèðóåìîñòü
â ñèëüíîì ñìûñëå) â ðàìêàõ îäíîãî óñêîðèòåëÿ Intel Xeon Phi 7120 D è 84-êðàòíîå óñêîðå-
íèå â ðàìêàõ îäíîãî óñêîðèòåëÿ Intel Xeon Phi 5120 D. Òàêèå çíà÷åíèÿ óñêîðåíèÿ ïî âñåé
âèäèìîñòè íàïðÿìóþ ñâÿçàíû ñ ïðîèçâîäèòåëüíîñòüþ êàæäîãî óñêîðèòåëÿ. Òàê èññëåäî-
âàíèÿ óñêîðåíèÿ (è äàëüíåéøåé ìàñøòàáèðóåìîñòè) íà ñóïåðêîìïüþòåðå ÌÂÑ-10Ï ÌÑÖ
ÐÀÍ áûëî ñäåëàíî â àïðåëå 2015 ãîäà, à èññëåäîâàíèÿ íà ñóïåðêîìïüþòåðå Ïîëèòåõíèê
RSC PetaStream ÑÏáÏÓ áûëî ñäåëàíî â íîÿáðå 2015 ãîäà.


4.2. Èññëåäîâàíèå ìàñøòàáèðóåìîñòè


     Ïðîâîäèëîñü èññëåäîâàíèå ìàñøòàáèðóåìîñòè êîäà AstroPhi íà óñêîðèòåëÿõ Intel Xeon
Phi 7120 D íà ñåòêå 512p × 512 × 512, íà óñêîðèòåëÿõ Intel Xeon Phi 5120 D áûëà èñïîëü-
çîâàíà ñåòêà 512p × 256 × 256 â îáåèõ ñëó÷àÿõ èñïîëüçîâàëèñü ÷åòûðå ëîãè÷åñêèõ ÿäðà íà
êàæäûé óñêîðèòåëü, ãäå p  ÷èñëî èñïîëüçóåìûõ óñêîðèòåëåé. Òàêèì îáðàçîì, íà êàæäûé
óñêîðèòåëü ïðèõîäèòñÿ îäèíàêîâûé ðàçìåð ïîäîáëàñòè ïðè ëþáîì ÷èñëå èññëåäóåìûõ óñêî-



                                               212
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                                    agora.guru.ru/pavt




                                      140

                                                    JSCC
                                      120
                                                    SPb
                                      100




                        Speed-Up
                                       80



                                       60



                                       40



                                       20



                                        0


                                            1   2   4     8      16     32   64   128   256


                                                              Threads




                             Ðèñ. 5. Èññëåäîâàíèå óñêîðåíèÿ êîäà AstroPhi

ðèòåëåé. Äëÿ èññëåäîâàíèÿ ìàñøòàáèðóåìîñòè çàìåðÿëîñü âðåìÿ êàæäîãî ýòàïà ÷èñëåííîãî
ìåòîäà, â ñåêóíäàõ, à çàòåì âû÷èñëÿëàñü èõ ñóììà (Total) ïðè ðàçëè÷íîì ÷èñëå èñïîëüçó-
åìûõ óñêîðèòåëåé Intel Xeon Phi (MIC). Ìàñøòàáèðóåìîñòü T (Scalability) âû÷èñëÿëîñü ïî
ôîðìóëå
                                                              T otal1
                                                        T =                                      (2)
                                                              T otalp
ãäå T otal1  âðåìÿ âû÷èñëåíèé íà îäíîì óñêîðèòåëå ïðè èñïîëüçîâàíèè îäíîãî óñêîðèòå-
ëÿ, T otalp  âðåìÿ âû÷èñëåíèé íà îäíîì óñêîðèòåëåé ïðè èñïîëüçîâàíèè p óñêîðèòåëåé.
Ðåçóëüòàòû èññëåäîâàíèé ìàñøòàáèðóåìîñòè ïðèâåäåíû íà ðèñóíêå (6). Òàêèì îáðàçîì,




                                      1,1




                                      1,0
                        Scalability




                                      0,9




                                      0,8




                                                    JSCC
                                      0,7
                                                    SPb


                                      0,6
                                            1   2   4     8      16     32   64   128   256


                                                              MICs




                     Ðèñ. 6. Èññëåäîâàíèå ìàñøòàáèðóåìîñòè êîäà AstroPhi
áûëà ïîëó÷åíà 92-ïðîöåíòíàÿ ýôôåêòèâíîñòü (ìàñøòàáèðóåìîñòü â ñëàáîì ñìûñëå) íà 64
óñêîðèòåëÿõ Intel Xeon Phi 7120 D è 75-ïðîöåíòíàÿ ýôôåêòèâíîñòü íà 224 óñêîðèòåëÿõ Intel
Xeon Phi 5120 D. Çàìåòèì, ÷òî ýôôåêòèâíîñòü áûñòðåå ïðîñàæèâàåòñÿ íà ñóïåðêîìïüþòåðå
ÑÏáÏÓ, ÷òî âåðîÿòíî ñâÿçàíî ñî ñëîæíîñòüþ ñåòåâîé èíôðàñòðóêòóðû è äîïîëíèòåëüíûìè



                                                           213
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



ñåòåâûìè ðàñõîäàìè íà îðãàíèçàöèþ îáìåíîâ.


4.3. Ìîäåëèðîâàíèå ìàñøòàáèðóåìîñòè


   Îñîáåííîñòüþ îðèãèíàëüíîãî ïîäõîäà ÿâëÿåòñÿ âîçìîæíîñòü ïðîñòîé ãåîìåòðè÷åñêîé
äåêîìïîçèöèè ðàñ÷åòíîé îáëàñòè è ïîñëåäóþùèì îáìåíîì ãðàíè÷íûõ çíà÷åíèé ìåæäó
òîëüêî ñîñåäíèìè âû÷èñëèòåëüíûìè óçëàìè. Èìèòàöèîííàÿ ìîäåëü îðãàíèçàöèè âû÷èñ-
ëåíèé ñòðîèòñÿ èç ñëåäóþùèõ ïðåäïîëîæåíèé:


  1. äëÿ íàõîæäåíèÿ îáùåãî âðåìåíè âûïîëíåíèÿ âû÷èñëåíèé íà êàæäîì ýòàïå áóäåì
     ïðåäïîëàãàòü, ÷òî íàì èçâåñòíî ñðåäíåå âðåìÿ âû÷èñëåíèé íà îäíó ÿ÷åéêó, òàêèì
     îáðàçîì, ïðåäïîëàãàÿ îäíîðîäíîñòü âû÷èñëåíèé ïî âñåé ðàñ÷åòíîé îáëàñòè;


  2. â êà÷åñòâå âû÷èñëèòåëüíîãî óçëà âûáèðàåòñÿ óñêîðèòåëü Intel Xeon Phi ïîëíîñòüþ,
     òåì ñàìûì íå ìîäåëèðóåòñÿ ìàñøòàáèðóåìîñòü âû÷èñëåíèé âíóòðè îäíîãî óñòðîéñòâà;


  3. âðåìÿ âûïîëíåíèÿ êîììóíèêàöèé áóäåì ñ÷èòàòü ëèíåéíîé ôóíêöèåé îò ÷èñëà ïåðå-
     äàâàåìûõ ýëåìåíòîâ ñ ó÷åòîì ëàòåíòíîñòè;


  4. êîëè÷åñòâî ïåðåäàâàåìûõ ýëåìåíòîâ, à, ñëåäîâàòåëüíî, è âðåìÿ ïåðåäà÷è, ïîñëå êàæ-
     äîãî èç ýòàïîâ ÷èñëåííîãî ìåòîäà îäèíàêîâî;


  5. èñïîëüçóåòñÿ ñåòåâàÿ èíôðàñòðóêòóðà ÑÑÊÖ ÈÂÌèÌà ÑÎ ÐÀÍ;


  6. èñïîëüçóåòñÿ îäíîìåðíàÿ äåêîìïîçèöèÿ ðàñ÷åòíîé îáëàñòè;


  7. ðàññìàòðèâàåòñÿ òîëüêî ýêñòåíñèâíîñòü âû÷èñëèòåëüíîé ñèñòåìû ïðè ñîõðàíåíèè ïðî-
     èçâîäèòåëüíîñòè îòäåëüíîãî óñòðîéñòâà.


Ïîñòðîåííàÿ íà òàêèõ äîïóùåíèÿõ èìèòàöèîííàÿ ìîäåëü êîäà AstroPhi áûëà ñìîäåëèðîâà-
íà ñ ïîìîùüþ êîìïëåêñà AGNES [23] íà ðàçëè÷íîì ÷èñëå ìîäåëüíûõ óñêîðèòåëåé. Âû÷èñ-
ëèòåëüíûå ýêñïåðèìåíû ïîêàçàëè, ÷òî ïðîãðàììíûé êîìïëåêñ AstroPhi ìîæåò áûòü ñ 70-
ïðîöåíòíîé ýôôåêòèâíîñòüþ ìàñøòàáèðóåì äî îäíîãî ìèëëèîíà âû÷èñëèòåëüíûõ óñòðîéñòâ.
Òàêîå ÷èñëî óñêîðèòåëåé ñîîòâåòñòâóåò ýêçàôëîïñíîìó óðîâíþ âû÷èñëåíèé.



5. Âû÷èñëèòåëüíûå ýêñïåðèìåíòû

    êà÷åñòâå âû÷èñëèòåëüíîãî ýêñïåðèìåíòà âûáðàíà çàäà÷à âûñîêî-ñêîðîñòíîãî ñòîëê-
íîâåíèÿ äèñêîâîé ãàëàêòèêè ñ ìåæãàëàêòè÷åñêèì âåòðîì â ãèäðîäèíàìè÷åñêîé ìîäåëè. Â
ðåçóëüòàòå òàêîãî âçàèìîäåéñòâèÿ îáðàçóåòñÿ ìåõàíèçì íàáåãàþùåãî ïîòîêà è ïðîèñõîäèò
îáòåêàíèå ñ îáðàçîâàíèåì íåóñòîé÷èâîñòüþ çà ãàëàêòèêîé. Îáðàçîâàíèå ïîäîáíûõ íåóñòîé-
÷èâîñòåé è õâîñòîâ âàæíû äëÿ èçó÷åíèÿ ìåõàíèçìà îáðàçîâàíèÿ ïåêóëÿðíûõ ãàëàêòèê è
ïðîöåññà çâåçäîîáðàçîâàíèÿ [24, 25]. Äèñêîâàÿ ãàëàêòèêà çàäàåòñÿ ðàâíîâåñíîé êîíôèãó-
ðàöèåé ñôåðè÷åñêîãî ãàëî ñ NFW-ïðîôèëåì ïëîòíîñòè è ðàâíîâåñíûì ýêñïîíåöèàëüíûì
ïðîôèëåì ïëîòíîñòè ñ äèôôåðåíöèàëüíûì âðàùåíèåì. Îáùàÿ ìàññà ãàëàêòèêè ñîñòàâëÿåò
M = 1013 M⊙ . Ñêîðîñòü íàáåãàþùåãî ïîòîêà ñîñòàâëÿåò v = 600 km/sec. Ïîñòàíîâêà çàäà÷è
èçîáðàæåíà íà ðèñóíêå (7). Ðåçóëüòàòû ìîäåëèðîâàíèÿ ïðåäñòàâëåíû íà ðèñóíêå (8), êî-
òîðûå ñîãëàñóþòñÿ ñ ðåçóëüòàòàìè àíàëîãè÷íîãî ìîäåëèðîâàíèÿ [25] è íàáëþäåíèÿìè [26].
Ðàñ÷åòû áûëè ïðîâåäåíû íà ïîñëåäîâàòåëüíîñòè ñåòîê îò 896×128×128 äî 7168×1024×1024.
Íà ïîñëåäíåé ñåòêå áûëî äîñòèãíóòî 47 ïðîöåíòîâ îò ïèêîâîé ñêàëÿðíîé ïðîèçâîäèòåëü-
íîñòè óñêîðèòåëÿ Intel Xeon Phi ïðè èñïîëüçîâàíèè 53760 íèòåé. Íà ðèñóíêå (8) âèäíî
îáðàçîâàíèå õâîñòà çà ôðîíòîì ãàëàêòèêè, êîòîðûé îáðàçóåòñÿ â ñëåäñòâèè íàáåãàíèÿ ãàçà
íà ãàëàêòèêó.




                                               214
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                                   agora.guru.ru/pavt




                    Ðèñ. 7. Ïîñòàíîâêà çàäà÷è íàáåãàíèÿ ãàçà íà ãàëàêòèêó

                                   3
                                                                                0



                                   2
                                                                                20


                                   1

                                                                                40
                        [10 kpc]




                                   0


                                                                                60
                                   -1



                                                                                80
                                   -2




                                   -3                                           100

                                        -3   -2   -1      0         1   2   3


                                                       [10 kpc]




              Ðèñ. 8. Ðåçóëüòàòû ìîäåëèðîâàíèÿ. Ñòîëáöåâàÿ ïëîòíîñòü â M pc           ⊙
                                                                                          −2




6. Çàêëþ÷åíèå

    ðàáîòå áûëè ïðåäñòàâëåíû èññëåäîâàíèÿ êîäà AstroPhi äëÿ ÷èñëåííîãî ìîäåëèðîâà-
íèÿ àñòðîôèçè÷åñêèõ òå÷åíèé íà ãèáðèäíûõ ñóïåðÝÂÌ, îñíàùåííûõ óñêîðèòåëÿìè Intel
Xeon Phi. Ïîäðîáíî îïèñàí ñî-äèçàéí âû÷èñëèòåëüíîé ìîäåëè äëÿ îïèñàíèÿ àñòðîôèçè-
÷åñêèõ îáúåêòîâ. Äåòàëüíî îïèñàíû îñîáåííîñòè ïàðàëëåëüíîé ðåàëèçàöèè è èññëåäîâà-
íèÿ ïðîèçâîäèòåëüíîñòè êîäà AstroPhi. Äëÿ êîäà AstroPhi áûëî äîñòèãíóòî 134-êðàòíîå
óñêîðåíèå â ðàìêàõ îäíîãî óñêîðèòåëÿ Intel Xeon Phi, 75-ïðîöåíòíàÿ ìàñøòàáèðóåìîñòü
ïðè èñïîëüçîâàíèè 224 óñêîðèòåëåé Intel Xeon Phi. Ïðåäñòàâëåíû ðåçóëüòàòû ìîäåëèðîâà-
íèÿ âçàèìîäåéñòâèÿ ìåæãàëàêòè÷åñêîãî âåòðà è äèñêîâîé ãàëàêòèêè. Íà ðàñ÷åòíîé ñåòêå
7168×1024×1024 áûëî äîñòèãíóòî 47 ïðîöåíòîâ îò ïèêîâîé ñêàëÿðíîé ïðîèçâîäèòåëüíîñòè
óñêîðèòåëÿ Intel Xeon Phi ïðè èñïîëüçîâàíèè 53760 íèòåé.
   Àâòîðû âûðàæàþò áëàãîäàðíîñòü êîëëåãàì èç îðãàíèçàöèé Intel (Íèêîëàþ Ìåñòåðó è
Äìèòðèþ Ïåòóíèíó) è RSC Group (Âëàäèìèðó Ìèðîíîâó, Ïàâëó Ëàâðåíêî è Áîðèñó Ãàãà-
ðèíîâó), çà ïðåäîñòàâëåíèå äîñòóïà ê êëàñòåðàì RSC PetaStream è ïîäðîáíûõ êîíñóëüòà-
öèé ïî èõ èñïîëüçîâàíèþ. Ý.È. Âîðîáüåâ áëàãîäàðèò Ìèíèñòåðñòâî Îáðàçîâàíèÿ è Íàóêè
Ðîññèéñêîé Ôåäåðàöèè, ïðîåêò  3.961.2014/K, çà ôèíàíñîâóþ ïîääåðæêó.




                                                              215
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



Ëèòåðàòóðà

 1. Tutukov A., Lazareva G., Kulikov I. Gas Dynamics of a Central Collision of Two Galaxies:
    Merger, Disruption, Passage, and the Formation of a New Galaxy // Astronomy Reports.
    2011. Vol. 55, No. 9. P. 770783.


 2. Mitchell N., Vorobyov E., Hensler G. Collisionless Stellar Hydrodynamics as an Ecient
    Alternative to N-body Methods // Monthly Notices of the Royal Astronomical Society.
    2013. V. 428, No. 3. P. 26742687.


 3. Ardeljan N.V., Bisnovatyi-Kogan G.S., Kosmachevskii G.S., Moiseenko S.G. An implicit
    Lagrangian code for the treatment of nonstationary problems in rotating astrophysical
    bodies // Astronomy and Astrophysics Supplement Series. 1996. Vol. 115. P. 573594.


 4. Khoperskov S.A, Vasiliev E.O., Sobolev A.M., Khoperskov A.V. The simulation of
    molecular clouds formation in the Milky Way // Monthly Notices of the Royal
    Astronomical Society. 2013. Vol. 428. P. 23112320.


 5. Fletcher A., Beck R., Shukurov A., Berkhuijsen E., Horellou C. Magnetic elds and spiral
    arms in the galaxy M51 // Monthly Notices of the Royal Astronomical Society. 2014.
    Vol. 412. P. 23962416.


 6. Kulikov I. GPUPEGAS: A New GPU-accelerated Hydrodynamic Code for Numerical
    Simulations of Interacting Galaxies // The Astrophysical Journal Supplement Series. 2014.
    Vol. 214, Id. 12.


 7. Kulikov I.M., Chernykh I.G., Snytnikov A.V., Glinskiy B.M., Tutukov A.V. AstroPhi: A
    code for complex simulation of dynamics of astrophysical objects using hybrid
    supercomputers // Computer Physics Communications. 2015. Vol. 186. P. 7180.


 8. Glinskiy B., Kulikov I., Snytnikov A., Romanenko A., Chernykh I., Vshivkov V. Co-design
    of Parallel Numerical Methods for Plasma Physics and Astrophysics // Supercomputing
    frontiers and innovations. 2015. Vol. 1, No. 3. P. 8898.


 9. Vorobyov E., Recchi S., Hensler G. Stellar hydrodynamical modeling of dwarf galaxies:
    simulation methodology, tests, and rst results // Astronomy & Astrophysics. 2015.
    Vol. 579, Id. A9.


10. Kulikov I., Chernykh I. Glinskiy B., Weins D., Shmelev A. Astrophysics simulation on RSC
    massively parallel architecture // Proceedings  2015 IEEE/ACM 15th International
    Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015. 2015. P. 11311134.


11. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I. Supercomputer Simulation of an
    Astrophysical Object Collapse by the Fluids-in-Cell Method // Lecture Notes of Computer
    Science. 2009. Vol. 5698. P. 414422.


12. Kulikov I., Chernykh I., Katysheva E., Protasov A., Serenko A. The numerical simulation
    of interacting galaxies by means of hybrid supercomputers // Bulletin NCC: Numerical
    analysis. 2015. Vol. 17. P. 1733.


13. Popov M., Ustyugov S. Piecewise parabolic method on local stencil for gasdynamic
    simulations // Computational Mathematics and Mathematical Physics. 2007. Vol. 47,
    No. 12. P. 19701989.


14. Popov M., Ustyugov S. Piecewise parabolic method on a local stencil for ideal
    magnetohydrodynamics // Computational Mathematics and Mathematical Physics. 2008.
    Vol. 48, No. 3. P. 477499.



                                               216
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



15. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I., Tutukov A. Hydrodynamical code for
   numerical simulation of the gas components of colliding galaxies // The Astrophysical
   Journal Supplement Series. 2011. Vol. 194, Id. 47.


16. Godunov S., Kulikov I. Computation of Discontinuous Solutions of Fluid Dynamics
   Equations with Entropy Nondecrease Guarantee // Computational Mathematics and
   Mathematical Physics. 2014. Vol. 54. P. 10121024.


17. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I., Tutukov A. Computational methods
   for ill-posed problems of gravitational gasodynamics // Journal of Inverse and Ill-posed
   Problems. Vol. 19, No. 1. P. 151166.


18. Kulikov I., Vorobyov E. Using the PPML approach for constructing a low-dissipation,
   operator-splitting scheme for numerical simulations of hydrodynamic ows // The Journal
   of Computational Physics. (submitted, 2016)


19. Kulikov I., Chernykh I., Snytnikov A., Protasov V., Tutukov A., Glinsky B. Numerical
   Modelling of Astrophysical Flow on Hybrid Architecture Supercomputers // In Parallel
   Programming: Practical Aspects, Models and Current Limitations (ed. M. Tarkov). 2015.
   P. 71116.


20. Frigo M., Johnson S. The Design and Implementation of FFTW3 // Proceedings of the
   IEEE. 2005. Vol. 93, No. 2. P. 216231.


21. Kalinkin A., Laevsky Y., Gololobov S. 2D Fast Poisson Solver for High-Performance
   Computing // Lecture Notes in Computer Science. 2009. Vol. 5698. P. 112120.


22. Schive H., Tsai Y., Chiueh T. GAMER: a GPU-accelerated Adaptive-Mesh-Renement
   Code for Astrophysics // The Astrophysical Journal. 2010. Vol. 186. P. 457484.


23. Podkorytov D., Rodionov A., Sokolova O., Yurgenson A. Using Agent-Oriented Simulation
   System AGNES for Evaluation of Sensor Networks // Lecture Notes of Computer Science.
   2010. Vol. 6235. P. 247250.


24. Jae Y.L., Smith R., Candlish G., Poggianti B.M., Sheen Y.-K., Verheijen M.A.W.
   BUDHIES II: A phase-space view of HI gas stripping and star-formation quenching in
   cluster galaxies // Monthly Notices of the Royal Astronomical Society. 2015. Vol. 448.
   P. 17151728.


25. Vollmer B., Cayatte V., Balkowski C., Duschl W.J. Ram pressure stripping and galaxy
   orbits: The case of the Virgo cluster // The Astrophysical Journal. 2001. Vol. 561.
   P. 708726.


26. Cayatte V., Kotanyi C., Balkowski C., van Gorkom J.H. A very large array survey of
   neutral hydrogen in Virgo Cluster spirals. 3: Surface density proles of the gas // The
   Astronomical Journal. 1994. Vol. 107, No. 3. P. 10031017.




                                               217
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                             agora.guru.ru/pavt




Numerical Hydrodynamics Simulation of Astrophysical

             Flows at Intel Xeon Phi supercomputers

  I.M. Kulikov
                 1,2, I.G. Chernykh1,2, E.I. Vorobyov3,4, A.V. Snytnikov1,2, D.V. Weins1,
                  5                  5                     7               7
A.A. Moskovsky , A.B. Shmelev , V.A. Protasov , A.A. Serenko , V.E. Nenashev , V.A.
                                                                                                  7
                          1                      1                1,2                 6
                 Vshivkov , A.S. Rodionov , B.M. Glinsky            , A.V. Tutukov


   Institute of Computational Mathematics and Mathematical Geophysics SB RAS,

 Novosibirsk, Russian Federation
                                         1, Novosibirsk State University, Novosibirsk, Russian
             2                                                             3
 Federation , Department of Astrophysics, University of Vienna , Research Institute of
                                                                                       4
Physics, Southern Federal University, Rostov-on-Don, Russian Federation , RSC Group,
                                         5
     Moscow, Russian Federation , Institute of Astronomy RAS, Moscow, Russian
           6
 Federation , Novosibirsk State Technical University, Novosibirsk, Russian Federation
                                                                                     7


        In this paper we present the AstroPhi code for numerical simulation of astrophysical
        ows at Intel Xeon Phi supercomputers. The co-design of a computational astrophysics
        model is described. The parallel implementation and scalability tests of the AstroPhi
        code are presented. The results of simulation of interaction between intergalactic wind
        and a disk galaxy are provided. For AstroPhi code a 134x speed-up with one Intel Xeon
        Phi accelerator and 75% weak scaling eciency on 224x Intel Xeon Phi accelerators
        was obtained. We obtained the peak performance on a 7168 × 1024 × 1024 mesh size
        using 53760 RSC PetaStream threads.
        Keywords:   High performance computing, Numerical astrophysics, Intel Xeon Phi
        accelerators.

References

1. Tutukov A., Lazareva G., Kulikov I. Gas Dynamics of a Central Collision of Two Galaxies:
   Merger, Disruption, Passage, and the Formation of a New Galaxy // Astronomy Reports.
   2011. Vol. 55, No. 9. P. 770783.


2. Mitchell N., Vorobyov E., Hensler G. Collisionless Stellar Hydrodynamics as an Ecient
   Alternative to N-body Methods // Monthly Notices of the Royal Astronomical Society.
   2013. V. 428, No. 3. P. 26742687.


3. Ardeljan N.V., Bisnovatyi-Kogan G.S., Kosmachevskii G.S., Moiseenko S.G. An implicit
   Lagrangian code for the treatment of nonstationary problems in rotating astrophysical
   bodies // Astronomy and Astrophysics Supplement Series. 1996. Vol. 115. P. 573594.


4. Khoperskov S.A, Vasiliev E.O., Sobolev A.M., Khoperskov A.V. The simulation of
   molecular clouds formation in the Milky Way // Monthly Notices of the Royal
   Astronomical Society. 2013. Vol. 428. P. 23112320.


5. Fletcher A., Beck R., Shukurov A., Berkhuijsen E., Horellou C. Magnetic elds and spiral
   arms in the galaxy M51 // Monthly Notices of the Royal Astronomical Society. 2014.
   Vol. 412. P. 23962416.


6. Kulikov I. GPUPEGAS: A New GPU-accelerated Hydrodynamic Code for Numerical
   Simulations of Interacting Galaxies // The Astrophysical Journal Supplement Series. 2014.
   Vol. 214, Id. 12.


7. Kulikov I.M., Chernykh I.G., Snytnikov A.V., Glinskiy B.M., Tutukov A.V. AstroPhi: A
   code for complex simulation of dynamics of astrophysical objects using hybrid
   supercomputers // Computer Physics Communications. 2015. Vol. 186. P. 7180.



                                                     218
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



 8. Glinskiy B., Kulikov I., Snytnikov A., Romanenko A., Chernykh I., Vshivkov V. Co-design
    of Parallel Numerical Methods for Plasma Physics and Astrophysics // Supercomputing
    frontiers and innovations. 2015. Vol. 1, No. 3. P. 8898.


 9. Vorobyov E., Recchi S., Hensler G. Stellar hydrodynamical modeling of dwarf galaxies:
    simulation methodology, tests, and rst results // Astronomy & Astrophysics. 2015.
    Vol. 579, Id. A9.


10. Kulikov I., Chernykh I. Glinskiy B., Weins D., Shmelev A. Astrophysics simulation on RSC
    massively parallel architecture // Proceedings  2015 IEEE/ACM 15th International
    Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2015. 2015. P. 11311134.


11. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I. Supercomputer Simulation of an
    Astrophysical Object Collapse by the Fluids-in-Cell Method // Lecture Notes of Computer
    Science. 2009. Vol. 5698. P. 414422.


12. Kulikov I., Chernykh I., Katysheva E., Protasov A., Serenko A. The numerical simulation
    of interacting galaxies by means of hybrid supercomputers // Bulletin NCC: Numerical
    analysis. 2015. Vol. 17. P. 1733.


13. Popov M., Ustyugov S. Piecewise parabolic method on local stencil for gasdynamic
    simulations // Computational Mathematics and Mathematical Physics. 2007. Vol. 47,
    No. 12. P. 19701989.


14. Popov M., Ustyugov S. Piecewise parabolic method on a local stencil for ideal
    magnetohydrodynamics // Computational Mathematics and Mathematical Physics. 2008.
    Vol. 48, No. 3. P. 477499.


15. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I., Tutukov A. Hydrodynamical code for
    numerical simulation of the gas components of colliding galaxies // The Astrophysical
    Journal Supplement Series. 2011. Vol. 194, Id. 47.


16. Godunov S., Kulikov I. Computation of Discontinuous Solutions of Fluid Dynamics
    Equations with Entropy Nondecrease Guarantee // Computational Mathematics and
    Mathematical Physics. 2014. Vol. 54. P. 10121024.


17. Vshivkov V., Lazareva G., Snytnikov A., Kulikov I., Tutukov A. Computational methods
    for ill-posed problems of gravitational gasodynamics // Journal of Inverse and Ill-posed
    Problems. Vol. 19, No. 1. P. 151166.


18. Kulikov I., Vorobyov E. Using the PPML approach for constructing a low-dissipation,
    operator-splitting scheme for numerical simulations of hydrodynamic ows // The Journal
    of Computational Physics. (submitted, 2016)


19. Kulikov I., Chernykh I., Snytnikov A., Protasov V., Tutukov A., Glinsky B. Numerical
    Modelling of Astrophysical Flow on Hybrid Architecture Supercomputers // In Parallel
    Programming: Practical Aspects, Models and Current Limitations (ed. M. Tarkov). 2015.
    P. 71116.


20. Frigo M., Johnson S. The Design and Implementation of FFTW3 // Proceedings of the
    IEEE. 2005. Vol. 93, No. 2. P. 216231.


21. Kalinkin A., Laevsky Y., Gololobov S. 2D Fast Poisson Solver for High-Performance
    Computing // Lecture Notes in Computer Science. 2009. Vol. 5698. P. 112120.


22. Schive H., Tsai Y., Chiueh T. GAMER: a GPU-accelerated Adaptive-Mesh-Renement
    Code for Astrophysics // The Astrophysical Journal. 2010. Vol. 186. P. 457484.



                                               219
Параллельные вычислительные технологии (ПаВТ’2016) || Parallel computational technologies (PCT’2016)

                                         agora.guru.ru/pavt



23. Podkorytov D., Rodionov A., Sokolova O., Yurgenson A. Using Agent-Oriented Simulation
   System AGNES for Evaluation of Sensor Networks // Lecture Notes of Computer Science.
   2010. Vol. 6235. P. 247250.


24. Jae Y.L., Smith R., Candlish G., Poggianti B.M., Sheen Y.-K., Verheijen M.A.W.
   BUDHIES II: A phase-space view of HI gas stripping and star-formation quenching in
   cluster galaxies // Monthly Notices of the Royal Astronomical Society. 2015. Vol. 448.
   P. 17151728.


25. Vollmer B., Cayatte V., Balkowski C., Duschl W.J. Ram pressure stripping and galaxy
   orbits: The case of the Virgo cluster // The Astrophysical Journal. 2001. Vol. 561.
   P. 708726.


26. Cayatte V., Kotanyi C., Balkowski C., van Gorkom J.H. A very large array survey of
   neutral hydrogen in Virgo Cluster spirals. 3: Surface density proles of the gas // The
   Astronomical Journal. 1994. Vol. 107, No. 3. P. 10031017.




                                               220

</pre>