Authors Barry F. Smith Fande Kong Hong Zhang Jacob Faibussowitsch Jed Brown Junchao Zhang Lois Curfman McInnes Mark Adams Matthew Knepley Oana Marin Patrick Sanan Richard Tran Mills Satish Balay Scott Kruger Todd Munson Victor Eijkhout
License CC-BY-4.0
THEME ARTICLE: COLLEGEVILLE WORKSHOP 2021:
SCIENTIFIC SOFTWARE TEAMS
The PETSc Community as Infrastructure
Mark Adams, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
Satish Balay , Oana Marin, Lois Curfman McInnes , Richard Tran Mills , Todd Munson , Hong Zhang , and
Junchao Zhang, Argonne National Laboratory, Lemont, IL, 60439, USA
Jed Brown , University of Colorado at Boulder, Boulder, CO, 80309, USA
Victor Eijkhout, The University of Texas at Austin, Austin, TX, 78712, USA
Jacob Faibussowitsch, University of Illinois at Urbana-Champaign, Urbana, IL, 61801, USA
Matthew Knepley , University of New York at Buffalo, Buffalo, NY, 14260, USA
Fande Kong, Idaho National Laboratory, Idaho Falls, ID, 83415, USA
Scott Kruger, Tech-X Corporation, Boulder, CO, 80303, USA
€ rich, Switzerland
Patrick Sanan, ETH Zurich, 8092 Zu
Barry F. Smith , Flatiron Institute, New York, NY, 10010, USA
Hong Zhang, Illinois Institute of Technology, Chicago, IL, 60616, USA
The communities that develop and support open-source scientific software packages
are crucial to the utility and success of such packages. Moreover, they form an
important part of the human infrastructure that enables scientific progress. This article
discusses aspects of the Portable Extensible Toolkit for Scientific Computation
community, its organization, and technical approaches that enable community
members to help each other efficiently and effectively.
T
o meet the technological challenges of the 21st software paradigm.1,3,5,9 Many numerically oriented
century, the simultaneous revolutions in data sci- open-source projects, including SciPy, Julia, and
ence and computing architectures need to be mir- Stan, thrive because of their communities; those
rored by a revolution in scientific simulation that provides without a community die out, have only a fringe
flexible, scalable, multiphysics multiscale capabilities in usership, or are maintained (as orphan software) by
both traditional and new areas. This simulation technol- other communities.
ogy rests on a foundation of numerical algorithms and In addition, an explicit focus on software ecosys-
software for high-performance computing. This founda- tems—collections of interdependent products
tion raises in importance to the level of classical hard whose development teams have incentives to col-
infrastructures but it requires human investment and laborate to provide aggregate value—is addressing
new ways of organizing the effort for software and algo- growing HPC complexity.6 Notable efforts include
rithm development, support, and maintenance. the xSDK, where community policiesb are helping
Much simulation technology today is developed with coordination among numerical packages, and
and supported with a communitya open-source E4S,c a broader effort addressing functionality
across the HPC software stack. These endeavors
are funded by the U.S. Department of Energy (DOE)
a
The source is developed in a public environment with con- within the Exascale Computing Project (ECP).d
tributors from a variety of institutions. Moreover, communities are tackling challenges in
how research software is developed and sustained10
as well as software stewardship.8
This work is licensed under a Creative Commons Attribution
4.0 License. For more information, see https://creativecom-
mons.org/licenses/by/4.0/
Digital Object Identifier 10.1109/MCSE.2022.3169974 b
htt_ps://xsdk.info/policies
Date of publication 11 May 2022; date of current version 31 c
htt_ps://E4S.io
d
August 2022. htt_ps://exascaleproject.org
6 Computing in Science & Engineering Published by the IEEE Computer Society May/June 2022
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
This article presents an explanatory case study of institutions and application fields. Also, community inter-
the Portable, Extensible Toolkit for Scientific Compu- actions promote algorithmic development, enable state-
tation (PETSc),2 considering community as infrastruc- of-the-art advances, and benefit the scientific
ture. PETSc began in the early 1990s at Argonne community.
National Laboratory as a project for research on paral- We organize this article as follows. In the next sec-
lel numerical algorithms. Since then, developers, users, tion, we discuss the purposes of the PETSc community
and functionality have grown substantially, driven by and the various roles that members play. After that, we
continually expanding community needs to exploit introduce several key organizational principles and
advances in HPC architectures for next-generation communication patterns. We then introduce the para-
science fully. The authors have over 160 years of com- digm of debugging by e-mail, which encapsulates the
bined experience with PETSc; their training ranges philosophy and software technologies we use to help
from mathematics to computer science to science each other (regardless of location) use, debug, and
and engineering. One author also supports PETSc at a improve PETSc. This section shows how technical
supercomputing center; another maintains the testing choices in design and software details can be made
and merge request infrastructure. Several authors are specifically to enhance the community experience.
liaisons with other software communities. PETSc com-
prises software infrastructure (code and tools) plus COMMUNITY
human infrastructure: the community of people who
This section outlines the myriad purposes of the PETSc
develop, support, maintain, use, and fund PETSc, their
software and its roles within the PETSc community.
interactions, and their culture. The human infrastruc-
First, of course, the PETSc community (similar to other
ture—people and their interactions as a community,
package communities) is embedded in the DOE, HPC,
within the broader DOE, HPC, and computational sci-
and broader computational science communities.
ence communities—is foundational and enables the
creation of sustainable software infrastructure.
PETSc was not originally purposefully designed to sup- Purposes of PETSc
port long-term community software infrastructure. PETSc serves many purposes as a software library that
Rather, work on the software inspired the creation of a connects research in applied mathematics to usage
set of practices to enable a small development team with within applications in science and engineering. These
large ambitions and a long time horizon to develop and include:
support software capable of solving problems of interest
to the developers and their collaborators. However, › a research platform targeting innovative algo-
these practices, reviewed in the following, have wider rithmic development;
benefits, and certain community properties could serve › a well-supported HPC library;
as a template for long-term software infrastructure: › a repository of template applications via a
wealth of example codes;
› enabling swift, in-depth engagement, especially › a compendium of algorithms, with an algorith-
for new users; mic management system that provides con-
› encouraging and offering opportunities for anyone crete, scalable implementations of a wide range
to contribute to the software and documentation; of methods described in the applied mathemat-
› providing a virtual institution for collaboration; ics literature;
› developing extensible interfaces that enable peo- › an application development framework;
ple interested in mathematics and algorithms to › a pedagogical tool for training numerical ana-
experiment and deploy research; lysts on HPC platforms4;
› supporting developer autonomy to pursue topics › a source of best choice numerical methods in its
aligned with individual research needs; role as an interface between academic algorith-
› enabling strong ties to academia, industry, and mic development and the needs of users in sci-
laboratories worldwide; ence and engineering;
› committing to continually advancing library › an extensible interface to complementary HPC
capabilities as needed by next-generation sci- software, such as SuperLU and hypre.
ence and HPC architectures.
Roles of PETSc Community Members
Spread throughout the world, the PETSc community Virtually all active PETSc community members are
allows the real-time transfer of knowledge across PETSc users; a smaller subset of these, who often
May/June 2022 Computing in Science & Engineering 7
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
began as PETSc users, are also PETSc developers. All face the challenge of writing scalable imple-
PETSc users provide important contributions, including mentations, which, even in PETSc, can be
bug reports, bug fixes, improved documentation, and time consuming with a steep learning curve.
suggestions for new features. Individuals often move However, such people find the benefits of
between different roles in the PETSc community. There using PETSc, including the broad impact of
is a “long tail” who contributes less frequently than the their work on HPC applications, outweigh
most active developers, yet collectively contributes a using packages with a less steep learning
great deal. Over one hundred people are contributors curve, such as MATLAB.
to the PETSc Git repository; hundreds more communi- › Scientific toolkit developers build systems that
cate through e-mail and GitLab issues each year, and tackle a subset of PETSc functionality but at a
thousands use PETSc directly or via other toolkits. higher level of abstraction, with more specific sup-
The community structure is crucial in providing a port for their target class of problems. Such tool-
pathway to increasing involvement for interested kits, including Firedrake, MOOSE, and Deal.II,
users. One way is to characterize PETSc community leverage PETSc capabilities and introduce addi-
members along institutional lines. tional infrastructure.
› Application developers focus on creating a code
› Academic-oriented users are students, faculty, that addresses one specific simulation. They are
and staff focusing on research and development, often discipline scientists or engineers, who ben-
employed by universities and research laborato- efit from performance enhancements provided
ries. Students may use PETSc to do homework or through PETSc composability, where upgrades
develop a paper or thesis code. Students often in algorithms and data structures can occur
contribute code back to PETSc, and, as they seamlessly from a user perspective, yet provide
graduate, bring PETSc to their new institutions. significant performance increases.
Some students have become PETSc developers.
› Industrial users employ PETSc in their company’s
research or commercial products. PETSc’s 2-clause
BSD license eases its commercial use. These users
ORGANIZATION AND
may request support that is unlikely to be funded
COMMUNICATION
by research grants, such as support for Microsoft We now summarize the organizational and communi-
Windows; fortunately, there are avenues for PETSc cation patterns in PETSc due to its various purposes
community members to help with these requests. and member roles. Besides communicating within
Industrial users require discretion and confidential- funded projects, institutional settings, and events,
ity. They cannot always share their use cases, so we users engage in the PETSc community through online
must provide general solutions without details of support mailings, GitLab issues, and Slack channels.
the specifics. Developing the trust needed for indus- Annual PETSc user meetings include tutorials on
trial users is a gradual process whose importance leveraging library functionality for research while also
must be recognized by both sides. highlighting users’ science achievements made possi-
ble by advances in PETSc features.
Another way to categorize
PETSc community members is
by the goals of their work, as
shown in Figure 1.
› Algorithm developers focus
on devising and analyzing
algorithms and hence may
be less concerned about
generality and usability. They
use PETSc because it pro-
vides infrastructure for HPC
architectures, allowing them
FIGURE 1. PETSc is a scientific junction, where the boundary between types of users and
to avoid unnecessary cod-
ing. Algorithm developers developers is fluid; roles shift and change.
8 Computing in Science & Engineering May/June 2022
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
Engagement When Problems Occur party libraries, such as MUMPS, SuperLU, and hypre
Support is a crucial aspect of a healthy software com- with little effort. Often “technical language” barriers
munity. Members of the PETSc community usually exist between communities. For example, an expert in
respond to requests within hours, if not minutes. This contact mechanics may describe a solver convergence
engagement helps new users feel welcomed and val- issue as “we found a PETSc error when contacts occur
ued, make rapid progress, and gain confidence. with a frictionless model.” A solvers expert might find it
Through users’ feedback, developers learn what works, hard to resolve such an issue. Fortunately, other PETSc
what does not, and where improvements are needed. community members may have domain expertise and
PETSc community members discover new research can serve as liaisons between communities. Such indi-
topics from feature requests and discussions. viduals speak the languages of both communities,
However, providing excellent support is a substan- understanding both PETSc’s capabilities and the needs
tial effort, particularly when users encounter difficult of their communities. Thus, they can explore, explain,
bugs or performance issues at scale. User-developer and introduce PETSc features to their communities.
communication on a particular topic can span weeks Such liaisons help expand PETSc’s reach across disci-
or even months. PETSc developers need to be patient plines and reduce the centralized maintenance burden
and consistently engaged with users. One may ques- by addressing many questions directly in their commu-
tion whether this practice is sustainable, but it has nities while contributing patches, feature requirements,
worked reasonably well. In the last section, we discuss and even serving as testers of software releases.
some technical approaches to providing support. An important PETSc subcommunity is systems engi-
neers who manage institutions’ computational infra-
Trust Within the Community structure. They often are the first to encounter problems
To maintain the vitality of the library, new algorithmic that need the attention of PETSc developers. Their exper-
developments must be rapidly integrated, bugs promptly tise can help rapidly debug problems and develop fixes.
fixed, and awkward constructions removed. These activi- Package maintainers, for example, for APT, are also a
ties require the PETSc community to establish a high valuable resource, as they track PETSc on particular con-
level of trust, communicating that the library will be well figurations; they often have excellent suggestions for
supported even in the face of rapid evolution, and that improvements to PETSc’s configuration and installation.
code will continue to run with help from the community.
Members of the PETSc community have a wide range of
professions, backgrounds, and levels of involvement, Responding to Change
with individuals often participating in several ways over Communities must respond with innovative and creative
the years. Engagement is key to disseminating tacit solutions to changing circumstances. For numerical soft-
knowledge and developing users’ skills and social sup- ware, this includes the continual emergence of new sci-
port so that people can transition to become developers ence drivers and techniques, currently data science and
and mentors. The PETSc community has developed a artificial intelligence, as well as new hardware architec-
broad base of people with expertise and kindness to tures. For example, a large shift in HPC is underway with
reduce and report bugs, mentor newcomers, and con- incorporating graphical processing units (GPUs) into sci-
tribute in other ways. To help improve the atmosphere, entific computing. Major organizations, such as DOE
which can be difficult for newcomers, the PETSc com- have responded with, for example, the ECP, where com-
munity has adopted a code of conduct.e munity open-source projects, including PETSc, are
aggressively developing innovations in data structures
Community to Community and algorithms for new architectures,7,11 The PETSc com-
Application communities often treat PETSc as a software munity empowers developers to be creative by providing
ecosystem instead of a stand-alone package. As a result, the autonomy to be innovative while still maintaining
they rely on PETSc to manage necessary low-level tools, guidelines for developmentf and town squares to orga-
such as MPI, BLAS/LAPACK, and vendor packages used nize overall development plans. This approach, along
on accelerators. Application communities also appreci- with PETSc’s wide variety of contributors, enables a level
ate unified solver interfaces, particularly linear precondi- of agility that might not otherwise occur. This approach
tioners, which enable application codes to access third- also promotes project-specific planning (for example, as
needed for work proposed and funded in particular
e
htt_ps://gitlab.com/petsc/petsc/-/blob/main/
f
CODE_OF_CONDUCT.md htt_ps://petsc.org/release/developers
May/June 2022 Computing in Science & Engineering 9
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
grants) and coordination among development communi-
ties overall.
Enabling Research Collaborations
PETSc’s community helps members identify funding
opportunities, access expertise, and transition between
roles in the project. One of the greatest difficulties in
maintaining a coherent software project over decades is
providing career paths for contributors. PETSc gives aca-
demic contributors a solid foundation for advancement
via awards (e.g., SC Gordon Bell prizes and SIAM prizes),
the highly cited users’ manual, professional recognition,
and productive collaborations born from PETSc develop-
ment, maintenance, and support. In addition, PETSc pro-
vides resource sharing from collaborative grants and FIGURE 2. PETSc has a holistic remote debuggable design,
collaboration opportunities that extend beyond the enabling a feature dubbed debugging by email, so PETSc
development group. Sometimes, PETSc affiliation may community members can pinpoint causes of problems
be more important than departmental affiliation, espe- through conversations in PETSc mailing lists and GitLab
cially since modern academic departments are often
issues.
atomized, with little internal collaboration. The commu-
nity provides strong academic connections for industrial
and laboratory members, tangible outputs recognized by
solution is beyond the expertise of PETSc developers.
future employers, and active participation in the wider
Because of its complexity—users call PETSc through
computational science community.
its Python, Fortran, or C bindings; in Linux, macOS, or
Windows operating systems; on machines from lap-
tops to the world’s most powerful supercomputers; on
Engagement With Funding Institutions x86, Arm, Power, or other CPU architectures, possibly
Even the smallest community open-source projects can- accelerated by GPUs from different vendors—PETSc
not exist without some funding and institutional support. community helpers cannot reproduce all problems
Usually, it is a combination of grants from governmental met by individuals. Thus, PETSc has a holistic remote
or nongovernmental agencies, in-house funding within debuggable design, enabling a feature dubbed debug-
particular institutions, and less formal systems that allow ging by e-mail, so PETSc community members can pin-
employees to contribute to open-source packages dur- point causes of problems through conversations in
ing a portion of their regular employment. Members of PETSc mailing lists and GitLab issues.
the PETSc community are actively engaged with pro- This approach enables the community to do more
gram development, including communicating with pro- with less. In this section, we introduce the debuggabil-
gram managers at the U.S. Department of Energy, the ity design of PETSc, which is designed and imple-
National Science Foundation, and with institutional man- mented to improve engagement and support. PETSc’s
agement to ensure that support is provided and main- debuggability spans from configuration to code execu-
tained. This form of interaction is crucial to the long- tion, with the salient features highlighted in Figure 2.
term viability of all open-source software communities; Our commitment to software support leads us to
PETSc users and community members have played maintain our own simple, integrated tools for many
important roles in various local, national, and interna- tasks that conventional software wisdom would dic-
tional conversations, including recent DOE activities tate should be performed exclusively by full-featured
related to software sustainability.8,10 external tools. The combination of internally devel-
oped and external tools used in PETSc is unique to its
SOFTWARE DESIGN TO SUPPORT history and “high-end” HPC focus and is not necessar-
THE COMMUNITY ily the best approach for other open-source packages.
Helping people when they encounter problems is
essential for the PETSc community. Members of the Configuration Debugging
community need to diagnose quickly where problems HPC software systems have complex execution envi-
occur and refer users to corresponding support if a ronments, including great variance in hardware and
10 Computing in Science & Engineering May/June 2022
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
LISTING 1. Sample showing error checking in PETSc.
software. Software developers must expend consider- early rather than later to avoid obscure error mes-
able effort to configure and build their code for different sages. The default error handler prints the stack trace
situations. When failures occur, the software developers leading to the error, including function names, file
need to have information available in a usable format to names, and line numbers. The stack trace is built
diagnose and fix the problems. Configuration failures are inside the two macros PetscFunctionBegin and PetscFunc-
the most common support issues the PETSc community tionReturn(0); see lines 3 and 16. PETSc also provides
faces. Thus, having a debuggable configuration system utilities to check the integrity of function parameters;
with comprehensive logging is critical. Rather than using see lines 4–10. All application programming interfaces
a standard configuration system, such as GNU Autotools (APIs) shown here are public; users are encouraged to
or CMake, PETSc, has a bespoke configuration system apply the same strategy in their code.
with extensive checking, written in Python, which logs PETSc also has APIs to assert properties of the
everything during configuration in a single file configure. code so that useful error messages are generated
log. When a check fails, it generates clear error messages promptly if the code behaves unexpectedly. For exam-
and a Python stack trace. PETSc users attach configure. ple, once a matrix is preallocated or assembled, one
log and another file make.log generated by make when they can set a property of the matrix to indicate that in sub-
meet configuration errors. By examining the two files, sequent insertions one will insert only to existing non-
PETSc developers can quickly determine why the config- zero locations.
uration system made specific choices and what went
wrong. In addition, since the configuration system is
Memory Debugging
bespoke, the PETSc community can easily add new
Memory corruption problems are common; therefore,
checking, testing, and logging. CMake is notoriously diffi-
memory allocations are done through a PETSc-spe-
cult to debug by e-mail because it logs information in var-
cific API that records information and sets sentinels
ious directories and does not log much of its process.
around the allocations in debug mode. With com-
mand-line options, PETSc will initialize the allocated
Runtime Error Debugging memory with not a number; using the uninitialized
The PETSc library strives to provide descriptive error memory in floating-point operations will generate
messages that explain why and where errors have an appropriate error message. Also, PETSc can check
occurred, making it easy for PETSc developers to diag- the integrity of the entire heap of PETSc-allocated
nose by e-mail what went wrong and assist users with memory at every allocation. PETSc codes also can out-
fixes. PETSc has extensive code to assist in this put information about memory that has never been
regard. See Listing 1, which shows code that adds two freed during the PETSc finalization stage, to detect
vectors with y += alpha*x. Every PETSc function returns memory leaks. As lightweight Valgrind-like features,
a PetscErrorCode, indicating whether the function is suc- the output can be shared with PETSc developers to
cessfully executed, and if not, what error occurred. In help understand a code’s misbehavior. But, of course,
PETSc source code, every function call is error we also recommend using more sophisticated tools,
checked, as in lines 12–14. We make errors manifest including Valgrind and debuggers.
May/June 2022 Computing in Science & Engineering 11
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
LISTING 2. Sample output of -log_view.
Performance Debugging processes might wait for messages from a lagging
Another challenging support task, which the PETSc partner), giving confusing results.
community also handles routinely, is debugging per- Having the profiling tools integrated with the
formance problems, particularly for high levels of par- numerical algorithms in use, outputting by default to
allelism. PETSc provides APIs to allow developers and stdout, is crucial because it allows all users to provide
users to set stages in their code and log the perfor- quickly information on their usage, independent of
mance of events of interest. For example, lines 12 and what computational systems they may use or which
14 in Listing 1 are for the VEC_AXPY event, rendering a additional analysis or logging tools they have available.
lightweight, integrated logging system that allows This allows PETSc developers to quickly and directly
users to quickly gather timings. Listing 2 shows a snip- view timings on the user’s system and facilitate perfor-
pet of the stdout output. From the top, we know that mance debugging of scalable solvers at “production”
the computation has two stages, labeled as setup and scale, by e-mail, where direct reproduction of a user’s
solve. PETSc summarizes the computation and com- issue is infeasible.
munication statistics of the two stages. Below that, it
lists detailed statistics of events (functions) within Algorithm Debugging
each stage (only the first stage is shown), including PETSc includes an extensive suite of parallel precondi-
the number of times an event has been called, time tioners, linear solvers, nonlinear solvers, and time inte-
and floating-point operations (flops) an event has grators. Composable and nested solvers are among
spent, MPI messages and reductions an event has the most powerful PETSc features since they facilitate
incurred, and other statistics. Interested readers are numerical experimentation on a novel, complex prob-
referred to the PETSc/TAO Users Manual2 or hints in lems, but keeping track of them can be difficult. PETSc
the log view message itself. Because MPI processes developers must see the detailed solver configura-
have different statistics in parallel, PETSc shows max- tions to spot potential problems. Hence, PETSc pro-
ima overall processes and ratios of maxima to minima. vides APIs to display all solver options being used.
This information is useful because whenever we Listing 3 shows a snippet of a longer output from a
encounter a large ratio in time or flops in output, we nonlinear solver. With indention reflecting levels of the
know a load imbalance in the corresponding event composite solvers, we can see the nested solvers
might exist. Sometimes, imbalance in one event can used and key parameters employed at various levels
distort the timing of other events (for instance, of the solvers.
12 Computing in Science & Engineering May/June 2022
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
LISTING 3. Sample output of -snes_view.
PETSc provides flexible monitors to be used with output is printed in well-formatted plain text for
solver views, which print the residual or function norm copy-and-paste or file attachments to e-mails and
at each iteration of an iterative solver so that users GitLab issues.
can check the convergence of the solver and compare
different algorithms. Listing 4 shows the output of
nonlinear and linear solver monitors. CONCLUSION
In summary, PETSc offers a wide set of comple- The increased prominence of data science and the
mentary options to aid debugging by e-mail with transition to computing architecture heterogeneity
the following common themes: users can enable require more, not less, high-quality numerical simula-
debugging regardless of their computing environ- tion and analysis software. This software is often cre-
ment; errors appear as early as possible; and the ated in community open-source environments; the
LISTING 4. Sample output of -snes_monitor -ksp_monitor.
May/June 2022 Computing in Science & Engineering 13
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
communities are crucial to the utility of such software. 5. N. Eghbal, “Roads and Bridges: The unseen labor
We have outlined some aspects of the open-source behind our digital infrastructure,” 2016. [Online].
PETSc community and its collaboration strategies. Available: https://www.fordfoundation.org/work/
Most of what was discussed apply to other numerical learning/research-reports/roads-and-bridges-the-
software communities. We concluded by focusing on unseen-labor-behind-our-digital-infrastructure/
mechanisms we use to allow community members to 6. L. C. McInnes, M. A. Erik, W. Heroux, A. Draeger,
efficiently help one another at a distance using S. S. Coghlan, and K. Antypas, “How community
straightforward communication channels. The science software ecosystems can unlock the potential of
and engineering of scientific software communities are exascale computing,” Nat. Comput. Sci., vol. 1,
only just beginning, and this topic is starting to receive pp. 92–94, 2021, doi: 10.1038/s43588-021-00033-y.
more consideration at institutional levels. By sharing 7. R. T. Mills et al., “Toward performance-portable
some of the PETSc community approaches, we hope to PETSc for GPU-based exascale systems,” Parallel
contribute to the wider scientific computing commu- Comput., vol. 108, 2021, Art. no. 102831,
nity as it seeks to improve the software programming doi: 10.1016/j.parco.2021.102831.
process. 8. “Request for information on the stewardship of
software for scientific and high-performance
computing,” U.S. Dept. Energy, Office Adv. Sci. Comput.
Res., Oct. 2021. [Online]. Available: https://www.govinfo.
ACKNOWLEDGMENTS
gov/app/details/FR-2021-10-29/2021-23582
The authors thank all PETSc users and developers for
9. M. J. Turk, “How to scale a code in the human
their many software, organizational, and conceptual
dimension,” in Proc. Conf. Extreme Sci. Eng. Discov.
contributions to the community. This material was
Environ.: Gateway Discov., 2013, pp. 1–7, doi: 10.1145/
based upon work funded in part by the U.S. Depart-
2484762.2484782.
ment of Energy, Office of Science, under contract
10. “Workshop on the science of scientific-software
DE-AC02-06CH11357. This work was supported in part
development and use,” U.S. Dept. Energy, Office Adv.
by the Exascale Computing Project under Grant
Sci. Comput. Res., Dec. 2021. [Online]. Available:
17-SC-20-SC, in part by a collaborative effort of the
https://www.orau.gov/SSSDU2021
U.S. Department of Energy Office of Science and the
11. J. Zhang et al., “The PetscSF scalable
National Nuclear Security Administration, and in part
communication layer,” IEEE Trans. Parallel Distrib.
by the U.S. Department of Energy, Office of Science,
Syst., vol. 33, no. 4, pp. 842–853, Apr. 2022,
Office of Advanced Scientific Computing Research
doi: 10.1109/TPDS.2021.3084070.
under Award Number DE-SC0016140. The work of
Matt Knepley and Jed Brown were supported in part MARK ADAMS is a staff scientist in the Scalable Solvers
by U.S. DOE Contract DE-AC02-0000011838. Group at Lawrence Berkeley National Laboratory, Berkeley,
CA, 94720, USA. Adams received his Ph.D. degree in civil engi-
neering from the University of California at Berkeley. Contact
REFERENCES him at mfadams@lbl.gov.
1. G. Avelino, E. Constantinou, M. T. Valente, and
A. Serebrenik, “On the abandonment and survival of SATISH BALAY is a software engineer at Argonne National
open source projects: An empirical investigation,” in Laboratory, Lemont, IL, 60439, USA. Balay received his
Proc. ACM/IEEE Int. Symp. Empirical Softw. Eng. Meas., M.S. degree in computer science from Old Dominion Univer-
2019, pp. 1–12. sity. Contact him at balay@mcs.anl.gov.
2. S. Balay et al., “PETSc/TAO users manual,” Argonne
Nat. Lab., Lemont, IL, USA, Tech. Rep. ANL-21/39, OANA MARIN is a numerical analyst at Argonne National Labo-
Revision 3.16, 2021. [Online]. Available: https://petsc. ratory, Lemont, IL, 60439, USA. Marin received her Ph.D. degree
org/ applied mathematics from the Royal Institute of Technology.
3. W. Bangerth and T. Heister, “What makes
Contact her at oanam@anl.gov.
computational open source software libraries
successful?,” Comput. Sci. Discov., vol. 6, no. 1, 2013,
LOIS CURFMAN MCINNES is a senior computational scien-
Art. no. 015010, doi: 10.1088/1749-4699/6/1/015010.
4. E. Bueler, PETSc for Partial Differential Equations: tist at Argonne National Laboratory, Lemont, IL, 60439, USA.
Numerical Solutions in C and Python. Philadelphia, PA, McInnes received her Ph.D. degree in applied mathematics
USA: SIAM, 2020, doi: 10.1137/1.9781611976311. from the University of Virginia. Contact her at curfman@anl.gov.
14 Computing in Science & Engineering May/June 2022
COLLEGEVILLE WORKSHOP 2021: SCIENTIFIC SOFTWARE TEAMS
RICHARD TRAN MILLS is a computational scientist at 61801, USA. Faibussowitsch received his B.Sc. degree in engi-
Argonne National Laboratory, Lemont, IL, 60439, USA. Mills neering mechanics from the University of Illinois at Urbana-
received his Ph.D. degree in computer science from the Col- Champagne. Contact him at faibuss2@illinois.edu.
lege of William and Mary. He is a member of IEEE. Contact
him at rtmills@anl.gov. MATTHEW KNEPLEY is an associate professor at the Univer-
sity of New York at Buffalo, Buffalo, NY, 14260, USA. Kenpley
TODD MUNSON is a senior computational scientist at received his Ph.D. degree in computer science from Purdue
Argonne National Laboratory, Lemont, IL, 60439, USA. He is a University. Contact him at knepley@gmail.com.
member of IEEE. Munson received his Ph.D. degree in com-
puter science from the University of Wisconsin at Madison. FANDE KONG is a computational scientist and software
Contact him at tmunson@anl.gov. developer at Idaho National Laboratory, Idaho Falls, ID,
83415, USA. Kong received his Ph.D. degree in computer sci-
HONG ZHANG is an assistant computational mathematician ence from the University of Colorado Boulder. Contact him at
at Argonne National Laboratory, Lemont, IL, 60439, USA. fande.kong@inl.gov.
Zhang received his Ph.D. degree in computer science from
Virginia Tech. Contact him at hongzhang@anl.gov. SCOTT KRUGER is a scientist/VP with Tech-X Corporation,
Boulder, CO, 80303, USA. Kruger received his Ph.D. degree in
JUNCHAO ZHANG is a software developer at Argonne nuclear engineering and engineering physics from the Univer-
National Laboratory, Lemont, IL, 60439, USA. Zhang received sity of Wisconsin-Madison. Contact him at kruger@txcorp.com.
his Ph.D. degree in computer science from ICT, Chinese Acad-
emy of Science. Contact him at jczhang@anl.gov. PATRICK SANAN is a postdoctoral researcher at ETH Zurich,
€ rich, Switzerland and an assistant computational
8092, Zu
JED BROWN is an assistant professor at the University of mathematician at Argonne National Laboratory, Lemont, IL,
Colorado at Boulder, Boulder, CO, 80309, USA. Brown 60439, USA. Sanan received his Ph.D. degree in applied and
received his Dr.Sc. degree in civil and environmental engi- computational mathematics from the California Institute of
neering from ETH, Zurich, Switzerland. Contact him at Technology. Contact him at psanan@anl.gov.
jed@jedbrown.org.
BARRY F. SMITH is a senior research scientist at the
VICTOR EIJKHOUT is a research scientist at the Texas Flatiron Institute, New York, NY, 10010, USA. Smith received
Advanced Computing Center of The University of Texas at Austin, his Ph.D. degree in mathematics from New York University.
Austin, TX, 78712, USA. Eijkhout received his Ph.D. degree in math- Contact him at bsmith@flatironinstitute.org.
ematics from Radboud University, The Netherlands. Contact him
at eijkhout@tacc.utexas.edu. HONG ZHANG is a research professor of computer science at
the Illinois Institute of Technology, Chicago, IL, 60616, USA.
JACOB FAIBUSSOWITSCH is a graduate research student at Zhang received his Ph.D. degree in applied mathematics from
the University of Illinois at Urbana-Champaign, Urbana, IL, Michigan State University. Contact her at hzhang@mcs.anl.gov.
May/June 2022 Computing in Science & Engineering 15