Chap_Terms.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chapter: Terms and Conventions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{PMIx Terms and Conventions}
\label{chap:terms}

The \ac{PMIx} Standard has adopted the widespread use of key-value \textit{attributes} to add flexibility to the functionality expressed in the existing \acp{API}. Accordingly, the community has chosen to require that the definition of each standard \ac{API} include the passing of an array of attributes. These provide a means of customizing the behavior of the \ac{API} as future needs emerge without having to alter or create new variants of it. In addition, attributes provide a mechanism by which researchers can easily explore new approaches to a given operation without having to modify the \ac{API} itself.

The \ac{PMIx} community has further adopted a policy that modification of existing released \acp{API} will only be permitted under extreme circumstances. In its effort to avoid introduction of any such backward incompatibility, the community has avoided the definitions of large numbers of \acp{API} that each focus on a narrow scope of functionality, and instead relied on the definition of fewer generic \acp{API} that include arrays of directives for ``tuning'' the function's behavior. Thus, modifications to the PMIx standard increasingly consist of the definition of new attributes along with a description of the \acp{API} to which they relate and the expected behavior when used with those \acp{API}.

One area where this can become more complicated relates to the attributes that provide directives to the client process and/or control the behavior of a \ac{PMIx} standard \ac{API}. For example, the \refattr{PMIX_TIMEOUT} attribute can be used to specify the time (in seconds) before the requested operation should time out.
The intent of this attribute is to allow the client to avoid hanging in a request that takes longer than the client wishes to wait, or may never return (e.g., a \refapi{PMIx_Fence} that a blocked participant never enters).

If an application truly relies on the \refattr{PMIX_TIMEOUT} attribute in a call to \refapi{PMIx_Fence}, it should set the \textit{required} flag in the \refstruct{pmix_info_t} for that attribute. This informs the library and its \ac{SMS} host that it must return an immediate error if this attribute is not supported. By not setting the flag, the library and \ac{SMS} host are allowed to treat the attribute as optional, silently ignoring it if support is not available.

\adviceuserstart
It is critical that users and application developers consider whether or not a given attribute is required (marking it accordingly) and always check the return status on all \ac{PMIx} function calls to ensure support was present and that the request was accepted. Note that for non-blocking \acp{API}, a return of \refconst{PMIX_SUCCESS} only indicates that the request had no obvious errors and is being processed. The eventual callback will return the status of the requested operation itself.
\adviceuserend

While a \ac{PMIx} library implementer, or an \ac{SMS} component server, may choose to support a particular \ac{PMIx} \ac{API}, they are not required to support every attribute that might apply to it. This would pose a significant barrier to entry for an implementer as there can be a broad range of applicable attributes to a given \ac{API}, at least some of which may rarely be used in a specific market area. The \ac{PMIx} community is attempting to help differentiate the attributes by indicating in the standard those that are generally used (and therefore, of higher importance to support) versus those that a ``complete implementation'' would support.

In addition, the document refers to the following entities and process stages when describing use-cases or operations involving \ac{PMIx}:

\begin{itemize}
\item \declareterm{session}\emph{session} refers to an allocated set of resources assigned to a particular user by the system \ac{WLM}. Historically, \ac{HPC} sessions have consisted of a static allocation of resources - i.e., a block of resources are assigned to a user in response to a specific request and managed as a unified collection. However, this is changing in response to the growing use of dynamic programming models that require on-the-fly allocation and release of system resources. Accordingly, the term \emph{session} in this document refers to the current block of assigned resources and is a potentially dynamic entity.
\item \declareterm{slot}\declareterm{slots}\emph{slot} refers to an allocated entry for a process. \acp{WLM} frequently allocate entire nodes to a \emph{session}, but can also be configured to define the maximum number of processes that can simultaneously be executed on each node. This often corresponds to the number of hardware \acp{PU} (typically cores, but can also be defined as hardware threads) on the node. However, the correlation between hardware \acp{PU} and slot allocations strictly depends upon system configuration.
\item \declareterm{job}\emph{job} refers to a set of one or more \emph{applications} executed as a single invocation by the user within a session. For example, ``\textit{mpiexec -n 1 app1 : -n 2 app2}'' is considered a single \ac{MPMD} job containing two applications.
\item \declareterm{namespace}\emph{namespace} refers to a character string value assigned by the \ac{RM} to a \textit{job}. All \textit{applications} executed as part of that \textit{job} share the same \emph{namespace}. The \emph{namespace} assigned to each \emph{job} must be unique within the scope of the governing \ac{RM}.
\item \declareterm{application}\emph{application} refers to a single executable (binary, script, etc.) member of a \emph{job}. Applications consist of one or more \emph{processes}, either operating independently or in parallel at any given time during their execution.
\item \declareterm{rank}\emph{rank} refers to the numerical location (starting from zero) of a process within the defined scope. Thus, {global rank} is the rank of a process within its \emph{job}, while \emph{application rank} is the rank of that process within its \emph{application}.
\item \declareterm{workflow}\declareterm{workflows}\emph{workflow} refers to an orchestrated execution plan frequently spanning multiple \emph{jobs} carried out under the control of a \emph{workflow manager} process. An example workflow might first execute a computational job to generate the flow of liquid through a complex cavity, followed by a visualization job that takes the output of the first job as its input to produce an image output.
\item \declareterm{scheduler}\emph{scheduler} refers to the component of the \ac{SMS} responsible for scheduling of resource allocations. This is also generally referred to as the \emph{system workflow manager} - for the purposes of this document, the \emph{WLM} acronym will be used interchangeably to refer to the scheduler.
\item \declareterm{resource manager}\emph{resource manager} is used in a generic sense to represent the subsystem that will host the \ac{PMIx} server library. This could be a vendor's \ac{RM}, a programming library's \ac{RTE}, or some other agent.
\item \declareterm{host environment}\emph{host environment} is used interchangeably with \emph{resource manager} to refer to the process hosting the \ac{PMIx} server library.
\item \declareterm{network plane}\declareterm{network planes}\emph{network plane} refers to a collection of \acp{NIC} and switches in a common logical or physical configuration. Network planes are often implemented in \ac{HPC} clusters as separate overlay or physical networks controlled by a dedicated fabric manager.
\end{itemize}


This document borrows freely from other standards (most notably from the \ac{MPI} and OpenMP standards) in its use of notation and conventions in an attempt to reduce confusion. The following sections provide an overview of the conventions used throughout the \ac{PMIx} Standard document.

%%%%%%%%%%%
\section{Notational Conventions}

Some sections of this document describe programming language specific examples or \acp{API}.
Text that applies only to programs for which the base language is C is shown as follows:

\cspecificstart
C specific text...
\begin{codepar}
int foo = 42;
\end{codepar}
\cspecificend

Some text is for information only, and is not part of the normative specification.
These take several forms, described in their examples below:

\notestart
\noteheader
General text...
\noteend

\rationalestart
Throughout this document, the rationale for the design choices made in the interface specification is set off in this section.
Some readers may wish to skip these sections, while readers interested in interface design may want to read them carefully.
\rationaleend

\adviceuserstart
Throughout this document, material aimed at users and that illustrates usage is set off in this section.
Some readers may wish to skip these sections, while readers interested in programming with the \ac{PMIx} \ac{API} may want to read them carefully.
\adviceuserend

\adviceimplstart
Throughout this document, material that is primarily commentary to \ac{PMIx} library implementers is set off in this section.
Some readers may wish to skip these sections, while readers interested in \ac{PMIx} implementations may want to read them carefully.
\adviceimplend

\advicermstart
Throughout this document, material that is primarily commentary aimed at host environments (e.g., \acp{RM} and \acp{RTE}) providing support for the \ac{PMIx} server library is set off in this section.
Some readers may wish to skip these sections, while readers interested in integrating \ac{PMIx} servers into their environment may want to read them carefully.
\advicermend

%%%%%%%%%%%
\section{Semantics}

The following terms will be taken to mean:

\begin{itemize}
\item \emph{shall}, \emph{must} and \emph{will} indicate that the specified behavior is \emph{required} of all conforming implementations
\item \emph{should} and \emph{may} indicate behaviors that a complete implementation would include, but are not required of all conforming implementations
\end{itemize}

%%%%%%%%%%%
\section{Naming Conventions}

The \ac{PMIx} standard has adopted the following conventions:

\begin{itemize}
\item \ac{PMIx} constants and attributes are prefixed with \textbf{\code{PMIX}}.
\item Structures and type definitions are prefixed with \code{pmix}.
\item Underscores are used to separate words in a function or variable name.
\item Lowercase letters are used in \ac{PMIx} client \acp{API} except for the \ac{PMIx} prefix (noted below) and the first letter of the word following it.
For example, \refapi{PMIx_Get_version}.
\item \ac{PMIx} server and tool \acp{API} are all lower case letters following the prefix - e.g., \refapi{PMIx_server_register_nspace}.
\item The \code{PMIx_} prefix is used to denote functions.
\item The \code{pmix_} prefix is used to denote function pointer and type definitions.
\end{itemize}

Users should not use the \textbf{\code{PMIX}}, \textbf{\code{PMIx}}, or \textbf{\code{pmix}} prefixes in their applications or libraries so as to avoid symbol conflicts with current and later versions of the \ac{PMIx} standard and implementations such as the \ac{PRI}.

%%%%%%%%%%%
\section{Procedure Conventions}

While the current \acf{PRI} is solely based on the C programming language, it is not the intent of the \ac{PMIx} Standard to preclude the use of other languages.
Accordingly, the procedure specifications in the \ac{PMIx} Standard are written in a language-independent syntax with the arguments marked as IN, OUT, or INOUT.
The meanings of these are:
\begin{itemize}
\item IN:
The call may use the input value but does not update the argument from the perspective of the caller at any time during the calls execution,
\item OUT:
The call may update the argument but does not use its input value
\item INOUT:
The call may both use and update the argument.
\end{itemize}

Many \ac{PMIx} interfaces, particularly nonblocking interfaces, use a \code{void*}cbdata object passed to the function that is then passed to the associated callback. In a client-side \ac{API}, the cbdata is a client-provided context (opaque object) that the client can pass to the nonblocking call (e.g., \refapi{PMIx_Get_nb}). When the nonblocking call (e.g., \refapi{pmix_value_cbfunc_t}) completes, the cbdata is passed back to the client without modification by the \ac{PMIx} library, thus allowing the client to associate a context with that callback. This is useful if there are many outstanding nonblocking calls.

A similar model is used for the server module functions (see \ref{server:module_fns}). In this case, the \ac{PMIx} library is making an upcall into its host via the \ac{PMIx} server module function and passing a specific cbfunc and cbdata. The \ac{PMIx} library expects the host to call the cbfunc with the necessary arguments and pass back the original cbdata upon completing the operation. This gives the server-side \ac{PMIx} library the ability to associate a context with the call back (since multiple operations may be outstanding). The host has no visibility into the contents of the cbdata object, nor is permitted to alter it in any way.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Standard vs Reference Implementation}

The \textit{PMIx Standard} is implementation independent. The \textit{PMIx Reference Implementation} (PRI) is one implementation of the Standard and the \ac{PMIx} community strives to ensure that it fully implements the Standard. Given its role as the community's testbed and its widespread use, this document cites the attributes supported by the \ac{PRI} for each \ac{API} where relevant by marking them in {\color{red}red}. This is not meant to imply nor confer any special role to the \ac{PRI} with respect to the Standard itself, but instead to provide a convenience to users of the Standard and \ac{PRI}.

Similarly, the \textit{PMIx Reference RunTime Environment} (PRRTE) is provided by the community to enable users operating in non-\ac{PMIx} environments to develop and execute \ac{PMIx}-enabled applications and tools. Attributes supported by the \ac{PRRTE} are marked in {\color{green!60!black}green}.