Cogito, Ergo Sumana for 2020 February

# 06 Feb 2020, 10:27AM: My First Exascale Computing Project Annual Meeting: Some interesting things about attending the Exascale Computing Project Annual Meeting for the first time, and stuff I have learned here so far!

[Edited 1:10pm CT to add: By the way, here is a contextual note for people who don't usually read my blog. I'm Sumana Harihareswara, a project manager and open source consultant who hadn't heard of ECP before November, and who primarily works in Python and outside of government stuff. I haven't done any kind of systematic survey of all ECP participants/attendees so these are my impressions based on people I've talked with and talks I've attended.]

Here is the overview of the Exascale Computing Project, which started a few years ago. Giant high-performance computing hardware, software, applications, training, and so on, working a lot at the United States's National Laboratories (like Lawrence Berkeley, Oak Ridge, Argonne, Los Alamos, and so on). Thus there is a lot that is public (for instance, see this report on improving scientific productivity, or this capability assessment), but then there are talks I'm not allowed to attend because I haven't signed the relevant nondisclosure agreement.
They contribute a bunch to LLVM and to Spack, a package manager. There are like 6-7 full-time funded people working on Spack [Edited 1:11 CT for correction: no, this is more like 6-7 people who work full-time and who spend at least a chunk of their time on Spack], and dozens of people attended the Spack state of the project/feedback roundtable session. Researchers and developers within ECP are working on a bunch of open source projects (example), some extremely specific to high-performance computing math things, but some more generally useful tools, and many folks in the project would like to get broader publicity and adoption for the latter. There are some opportunities here for cross-pollination, funding, user testing, and de-duplication between work being done by DoE and work being done in the larger open source industry.
The ECP is sponsored by the US Department of Energy. And, you know, that means fossil fuels too. There's an Industry Council and ExxonMobil is on it. The National Labs do a bunch of work for DoE and other US government departments -- and ~~for the private clients who can afford it~~ [Edited 1:23pm to correct this; those orgs aren't paying the labs to do work, they're getting to use the facilities just like anyone else could (example)], which is often the fossil fuels companies who want to run simulations having to do with oil and gas. When I've talked to folks here about how that feels weird, I get a variety of responses. Some people point out that there is a National Renewable Energy Laboratory among the ECP Participating Labs, or that the combustion work in the labs helps energy companies figure out how to use gas more efficiently so we burn less fuel, and so on. One person basically said: They're an important industry and it's part of our job to help them; it's the Department of Energy and that means all energy. Another person basically said: As soon as feasible, I want us to not do that work anymore.
[Edited 1:12pm CT to note: of course these are my personal observations and not a "here is an official position" thing.] I don't think anyone here denies that climate change is happening. I think they're supposed to make an attempt to not use that phrase in official published materials and they're not supposed to talk about it when they go to DC, though. In one talk a speaker mentioned that one of the categories he was listing was "Earth and Space Science -- what we used to call climate." I said, "Sorry, I'm new. What do we call it now?" and got the answer: "Earth Systems."
Weapons! Yeah the DoE includes the Office of Science (SC) and the National Nuclear Security Administration (NNSA). And the National Labs do some work for the military, the Department of Homeland Security, National Security Agency, and so on. Also there's some back-and-forth where sometimes people, for instance, start at the Department of Defense and then start working within DoE. ~~Approximately everyone at this meeting is fine with the fact that some of their work (or maybe a lot of their work) has to do with weapons.~~ [Edited 1:16pm and 4:50pm CT to say: so I've been told that this is mega inaccurate and that a buuuuuuunch of people's work here has NOTHING to do with weapons, is just pure open science, that there are several labs where nearly no one directly works on weapons stuff, or that there are several labs where no one does. Also I've been pointed to the DoE budget where only a fraction of the yearly spend goes to NNSA labs, and those labs also do a bunch of open science research. I need to look into this more to understand the nuances. Also, it was pointed out to me that, if I'm saying "this work is not directly weapons work but it is foundational to weapons work," then, one could also justly say that my work in Python also supports weapons research. Yup, it sure does! I am definitely complicit in things I am uncomfortable with! It's complicated.] Again, some people, when I bring this up, point out how much of the work has nothing to do with weapons, or talk about the work of stockpile stewardship as being primarily about safekeeping of and knowledge transfer about nuclear warheads where there is no likely near-term path to the US completely getting rid of them, or talk about defense in a world where nukes are out there and not about to go away. And at least one person said, basically, I have no problem with the weapons stuff and it's cool.
The vast majority of people here have doctorates, usually in one of the mathematical, computational, or physical sciences. I haven't seen a single name badge that has "Dr." on it; I think it would take up room and seem egotistical. Also, I am very rarely the only woman in the room, and some of the leadership are women, but I'm often the only person in the room who doesn't know C (or Fortran; the software ECP is writing for or adapting to the new machines is basically 2/3 C and C++, 1/3 Fortran). So my particular configuration of insecurities this week is different than it often is at tech conferences.
I am, here, extremely unusual in that I do not work for Department of Energy, one of the National Labs, a university, or a big company that is in the Industry Council. People squint at my badge, which says "Sumana Harihareswara, Changeset Consulting," and ask "where are you from?" And I say "New York City" and they say, "Oh, Brookhaven?" and then I explain that I'm a Better Scientific Software Fellowship Honorable Mention and that I'm working on materials to help people maintain open source software better. On the second day of the conference, I took a pen and added "BSSw" to the badge to help jump-start this process.
People here will refer to "a code" to mean an application or a particular simulation, where I might say "a tool". A person might refer to "running industrial codes" or "legacy codes that have been used for decades".
One of the kinds of sessions I'm not allowed in is the detailed PathForward stuff; DoE is contracting with chipmakers to do research and development and get big cutting-edge supercomputers for the ECP.
Following a rigorous review process, six responses were selected for award and contract negotiations began. All six selected responses successfully led to contracts that were awarded and announced in June 2017. The six awardees were Advanced Micro Devices (AMD), Cray Inc. (Cray), Hewlett Packard Enterprise (HPE), International Business Machines (IBM), Intel Corp. (Intel), and NVIDIA Corp. (NVIDIA).

HPE has bought Cray so that reduces the competition among these vendors -- and the redundancy in case one of them delivers late, goes bankrupt, or what have you.
Some people who are not US citizens work at the National Labs, including the more weapons-centric ones. [Edited 1:13 CT to note: I said "many" originally, but this is not to say that non-US-citizens are a majority! There are thousands of people working at the National Labs; "many" does not mean "most," just, like, there are some. I don't have exact numbers here and am changing "many" to "some".] They are open to hiring people from other countries. Also, National Labs employees are kiiiiiiiinda US government employees and kinda not in a way that I don't understand well enough to explain. But there are national security projects within the US government that would appreciate if more US citizens got into science and engineering research -- hence, for example, the National Science Foundation Graduate Research Fellowship Program (GRFP) which
helps ensure the vitality of the human resource base of science and engineering in the United States and reinforces its diversity. The program recognizes and supports outstanding graduate students in NSF-supported science, technology, engineering, and mathematics disciplines who are pursuing research-based master's and doctoral degrees at accredited United States institutions.
Fellows share in the prestige and opportunities that become available when they are selected. Fellows benefit from a three-year annual stipend of $34,000 along with a $12,000 cost of education allowance for tuition and fees (paid to the institution), opportunities for international research and professional development, and the freedom to conduct their own research at any accredited U.S. institution of graduate education they choose.

And they don't require a GRE score, by the way. Also you can sign up to help review applications!
The researchers at the National Labs, like a lot of scholars within academia, care about getting papers published, and sometimes that gets in the way of good maintainership for their open source projects. For instance, if you are worried that sharing your feature roadmap for your open source tool will let someone else get the jump on you and get a paper submitted sooner, you might hold that information kinda secret, which makes it more likely users will duplicate that work in their own forks.
The different National Labs have different cultures and "the further they are from a city, the weirder they get".

Thanks to BSSw for bringing me here! [Edited 4:57pm CT to add: I went on so long about these pseudo-anthopological observations that I need to start a new entry about cool tools I found out about here! Hope that will be next.]

Filed under: Open Source and Free Culture Transparency in Government Code