ESPResSo
Extensible Simulation Package for Research on Soft Matter Systems
Loading...
Searching...
No Matches
ghosts.hpp
Go to the documentation of this file.
1/*
2 * Copyright (C) 2010-2022 The ESPResSo project
3 * Copyright (C) 2002,2003,2004,2005,2006,2007,2008,2009,2010
4 * Max-Planck-Institute for Polymer Research, Theory Group
5 *
6 * This file is part of ESPResSo.
7 *
8 * ESPResSo is free software: you can redistribute it and/or modify
9 * it under the terms of the GNU General Public License as published by
10 * the Free Software Foundation, either version 3 of the License, or
11 * (at your option) any later version.
12 *
13 * ESPResSo is distributed in the hope that it will be useful,
14 * but WITHOUT ANY WARRANTY; without even the implied warranty of
15 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 * GNU General Public License for more details.
17 *
18 * You should have received a copy of the GNU General Public License
19 * along with this program. If not, see <http://www.gnu.org/licenses/>.
20 */
21
22#pragma once
23
24/** \file
25 * Ghost particles and particle exchange.
26 *
27 * In this file you find everything concerning the exchange of
28 * particle data (particles, ghosts, positions and forces) for short
29 * range interactions between the spacial domains of neighbouring
30 * nodes.
31 *
32 * <h2> How does this work </h2>
33 * The ghost communication transfers data from cells on one node to cells on
34 * another node during the integration process. Note that data can only be
35 * transferred from one cell to one other, and the contents of the other cell
36 * will be overwritten.
37 *
38 * The particle data that has to be transferred, and especially from where to
39 * where, heavily depends on the cell system. In ESPResSo, this is abstracted
40 * in form of ghost communicators (\ref GhostCommunicator) and ghost
41 * communications (\ref GhostCommunication). The ghost communicators represent
42 * the four communications above and consist of the data to transfer (which is
43 * determined by their type) and a list of ghost communications. The data
44 * types are described by the particle data classes:
45 * - @ref GHOSTTRANS_PROPRTS transfers the @ref ParticleProperties
46 * - @ref GHOSTTRANS_POSITION transfers the @ref ParticlePosition
47 * - @ref GHOSTTRANS_MOMENTUM transfers the @ref ParticleMomentum
48 * - @ref GHOSTTRANS_FORCE transfers the @ref ParticleForce
49 * - @ref GHOSTTRANS_RATTLE transfers the @ref ParticleRattle
50 * - @ref GHOSTTRANS_PARTNUM transfers the cell sizes
51 *
52 * Each ghost communication describes a single communication of the local with
53 * another node (or all other nodes). The data transferred can be any number
54 * of cells, there are five communication types:
55 * - @ref GHOST_SEND sends data to one other node
56 * - @ref GHOST_RECV recvs data from one other node. In the case of
57 * forces, they are added up, everything else is overwritten
58 * - @ref GHOST_BCST sends data to all nodes
59 * - @ref GHOST_RDCE recvs data from all nodes. In the case of
60 * forces, they are added up, everything else is overwritten
61 * - @ref GHOST_LOCL transfer data from a local cell to another local cell.
62 * In this case, the first half of the cells are the sending cells, the
63 * other half are the corresponding receivers.
64 *
65 * Note that for the first four communications you have to make sure that
66 * when one node sends to another, that the sender has @ref GHOST_SEND
67 * and the receiver @ref GHOST_RECV. In the case of @ref GHOST_BCST resp.
68 * @ref GHOST_RDCE, all nodes have to have the same communication type
69 * and the same master sender/receiver (just like the MPI commands).
70 *
71 * A special topic are @ref GHOST_PREFETCH and @ref GHOST_PSTSTORE. For
72 * example, if all nodes broadcast to the other, the naive implementation will
73 * be that @c n_nodes times a @ref GHOST_BCST is done with different master
74 * nodes. But this means that each time <tt>n_nodes - 1</tt> nodes wait for
75 * the master to construct its send buffer. Therefore there is the prefetch
76 * flag which can be set on a pair of recv/send operations. If the ghost
77 * communication reaches a recv operation with prefetch, the next send
78 * operation (which must have the prefetch set!!) is searched and the send
79 * buffer already created. When sending, this precreated send buffer is used.
80 * In the scenario above, all nodes create the send buffers simultaneously in
81 * the first communication step, thereby reducing the latency a little bit.
82 * The pststore is similar and postpones the write back of received data
83 * until a send operation (with a precreated send buffer) is finished.
84 *
85 * The ghost communicators are created by the cell systems.
86 */
87
88#include "BoxGeometry.hpp"
89#include "ParticleList.hpp"
90
91#include <utils/Vector.hpp>
92
93#include <boost/mpi/communicator.hpp>
94
95#include <cstddef>
96#include <utility>
97#include <vector>
98
99/** \name Transfer types, for \ref GhostCommunicator::type */
100/************************************************************/
101/**@{*/
102
103/// send to a single node
104#define GHOST_SEND 0
105/// recv from a single node
106#define GHOST_RECV 1
107/// broadcast, the node entry gives the sender
108#define GHOST_BCST 2
109/// reduce, the node entry gives the receiver
110#define GHOST_RDCE 3
111/// transfer data from cell to cell on this node
112#define GHOST_LOCL 4
113
114/// mask to the job area of the transfer type
115#define GHOST_JOBMASK 15
116/// additional flag for prefetching
117#define GHOST_PREFETCH 16
118/// additional flag for poststoring
119#define GHOST_PSTSTORE 32
120/**@}*/
121
122/** Transfer data classes, for \ref ghost_communicator */
123enum : unsigned {
125 /// transfer \ref ParticleProperties
127 /// transfer \ref ParticlePosition
129 /// transfer \ref ParticleMomentum
131 /// transfer \ref ParticleForce
133#ifdef BOND_CONSTRAINT
134 /// transfer \ref ParticleRattle
136#endif
137 /// resize the receiver particle arrays to the size of the senders
139 GHOSTTRANS_BONDS = 128u
141
143 /** Communication type. */
144 int type;
145 /** Node to communicate with (to use with all MPI operations). */
146 int node;
147
148 /** Pointer array to particle lists to communicate. */
149 std::vector<ParticleList *> part_lists = {};
150
151 /** Position shift for ghost particles. The shift is done on the sender side.
152 */
154};
155
156/** Properties for a ghost communication. */
158 GhostCommunicator() = default;
159 GhostCommunicator(boost::mpi::communicator comm, std::size_t size)
160 : mpi_comm(std::move(comm)), communications(size) {}
161
162 /** Attached mpi communicator */
163 boost::mpi::communicator mpi_comm;
164
165 /** List of ghost communications. */
166 std::vector<GhostCommunication> communications;
167};
168
169/**
170 * @brief Do a ghost communication with the specified data parts.
171 */
173 BoxGeometry const &box_geo, unsigned int data_parts);
Vector implementation and trait types for boost qvm interoperability.
@ GHOSTTRANS_MOMENTUM
transfer ParticleMomentum
Definition ghosts.hpp:130
@ GHOSTTRANS_RATTLE
transfer ParticleRattle
Definition ghosts.hpp:135
@ GHOSTTRANS_PARTNUM
resize the receiver particle arrays to the size of the senders
Definition ghosts.hpp:138
@ GHOSTTRANS_POSITION
transfer ParticlePosition
Definition ghosts.hpp:128
@ GHOSTTRANS_PROPRTS
transfer ParticleProperties
Definition ghosts.hpp:126
@ GHOSTTRANS_FORCE
transfer ParticleForce
Definition ghosts.hpp:132
@ GHOSTTRANS_NONE
Definition ghosts.hpp:124
@ GHOSTTRANS_BONDS
Definition ghosts.hpp:139
void ghost_communicator(GhostCommunicator const &gcr, BoxGeometry const &box_geo, unsigned int data_parts)
Do a ghost communication with the specified data parts.
Definition ghosts.cpp:442
Utils::Vector3d shift
Position shift for ghost particles.
Definition ghosts.hpp:153
int node
Node to communicate with (to use with all MPI operations).
Definition ghosts.hpp:146
std::vector< ParticleList * > part_lists
Pointer array to particle lists to communicate.
Definition ghosts.hpp:149
int type
Communication type.
Definition ghosts.hpp:144
Properties for a ghost communication.
Definition ghosts.hpp:157
GhostCommunicator(boost::mpi::communicator comm, std::size_t size)
Definition ghosts.hpp:159
GhostCommunicator()=default
boost::mpi::communicator mpi_comm
Attached mpi communicator.
Definition ghosts.hpp:163
std::vector< GhostCommunication > communications
List of ghost communications.
Definition ghosts.hpp:166