[Project Proposal] The Chapel Programming Language #11
Comments
|
In filing this application, we understand that the application process may still be getting revised based on initial experiences with the founding projects, and we will update ours to track the evolution of that process. |
|
For anyone subscribed to this issue, I've updated our application with the names of our TAC sponsors, and now consider this application complete (modulo additional changes that may occur to the questions going forward). I've edited my previous comment to reflect this. |
Project Proposal
Name of Project
Chapel / The Chapel Programming Language
Project Description
Chapel is an open-source programming language designed for productive parallel computing. Its features support the specification of the parallelism and locality within a computation using clean, powerful abstractions. Chapel's design and implementation are portable and scalable, targeting multicore CPUs and GPUs of laptops, desktops, commodity clusters, clouds, or supercomputers.
Chapel's chief value is that it supports code that is similarly readable/writeable as Python while resulting in performance that can match or beat conventional HPC programming technologies, such as C/C++/Fortran, MPI, OpenMP, CUDA, or ROCm. Chapel benefits from a modern language design and productive features for HPC systems, including distributed arrays, a global namespace, and features for controlling explicit task creation and placement.
Chapel was pioneered at Cray Inc. in the early 2000s as part of the DARPA High Productivity Computing Systems program (HPCS). Cray's HPCS program was named 'Cascade', and Chapel's name comes from the phrase Cascade High Productivity Language. The earliest release of Chapel was version 0.4, made available to those requesting it in December 2006. The first public release was in November 2008.
After HPCS wrapped up in 2012, the project pivoted from creating a research prototype to building a production-grade implementation capable of helping users solve their programming problems. Starting in 2019, users began writing the first large-scale applications in Chapel. Since then, Chapel applications have been used in production for computations as diverse as:
In practice, code from these applications has scaled to over 8,000 compute nodes, 1,000,000 CPU cores, and 1,000 GPUs.
Chapel development has continued at Hewlett Packard Enterprise with its acquisition of Cray Inc. In March 2024, the team at HPE, in collaboration with community contributors, released Chapel version 2.0, stabilizing the core language and standard libraries to prevent breaking user codes going forward. In terms of semantic versioning, Chapel 2.0 can be considered our 1.0 release. This month, Chapel released version 2.2, which reflects 20+ years of code contributions from over 200 contributors representing over 100 affiliations worldwide.
Statement on Alignment with High Performance Software Foundation's Mission
Chapel is a unique project within the HPC software community due to its focus on making the parallelism within HPC systems—as well as desktop CPUs and GPUs—accessible to computational scientists. Often these users are HPC experts who have recognized that Chapel improves their time-to-solution and lets them leverage their systems more quickly than when using standard techniques. Others are programmers who are not necessarily well-versed in parallel computing or HPC technologies. We have spoken with many Chapel programmers who know little-to-nothing about SPMD programming models, message passing, or other traditional HPC concerns; yet through Chapel, they are able to express their algorithms, leverage parallel hardware, and incrementally tune their programs to reap the benefits of parallelism at any scale. By bringing the programming of HPC systems within reach of more people, the Chapel project can help HPSF with its goals of accessibility and workforce growth.
A historical challenge to Chapel's adoption has been the concern that it is only supported by a single vendor, so could go away at any time if Cray/HPE loses interest. Meanwhile, other vendors or HPC centers may have been disincentivized from contributing to it due to Cray/HPE's historical control over the technology. Housing Chapel under HPSF would provide a neutral setting for greater collaboration within the community while also providing users with more assurance about the neutrality and vendor-independence of Chapel's future.
In addition, HPSF provides an opportunity for the Chapel project to address other potential shortcomings, such as: a community governance model for the language and its implementation, granting more users and institutions commit privileges, and security screening and maintenance. In return, we hope that Chapel will bring to HPSF an easy entry point for more users to get started with HPC, a platform for promoting understanding of HPC and its use in practice, and a language in which new HPC applications and frameworks can be developed.
Project Website
Project Website
Note that we are currently in the process of dramatically revamping our website in order to support a modernized look and feel, better browsing on smartphones, and simpler maintenance. The goal is to launch that website in the coming month or so (prior to SC24), and an ever-evolving preview of it can be viewed here.
Code of Conduct
Code of Conduct
Governance Practices
We do not have a formal, established governance process at present, and this is an area that we would like to address and improve in, as an HPSF project.
Practically speaking, governance today is led by the development team at HPE with input and feedback from the user community. We have an established code review policy in which all non-trivial pull requests must be reviewed and approved by at least one core developer other than the PR author. A core developer must also test the PR using our suite of 18,000+ tests. Decision-making is handled through GitHub issues, discussion forums, and direct communication (email, pair-programming, meetings, etc.). More details on our decision-making process are provided in question 14 below.
This model has been reasonably satisfactory so far, but everyone recognizes that for the language to grow and gain increased user confidence, it needs to move to a model in which there are more stakeholders and a more formal process for decision-making to protect users' investments in their code. Moving Chapel under HPSF seems like a natural and healthy stepping stone toward that vision.
Two Sponsors from the High Performance Software Foundation's Technical Advisory Committee
John Mellor-Crummey and Sameer Shende have agreed to serve as our TAC sponsors.
What is the project's solution for source control?
GitHub
What is the project's solution for issue tracking?
GitHub issues
Please list all external dependencies and their license
Chapel itself is licensed as Apache 2.0. It relies on the following third-party packages, which are summarized in greater detail in our top-level LICENSE file:
For convenience, these packages are bundled within the Chapel source tree, and in releases, within our third-party directory, though system-installed or packager-managed versions can typically be used in place of the bundled versions for most major components.
Chapel also relies on a number of Python packages that are dynamically installed within a virtual environment:
Please describe your release methodology and mechanics
Chapel is released quarterly (March, June, September, December) in a variety of formats:
Most of the release artifacts are built using Jenkins with some manual steps around things like documentation or opening pull requests on external repositories (Spack, Homebrew, etc.). Recent releases have been announced on the Chapel blog, highlighting major features and changes. Semi-annually, we also publish release notes in slide format, providing more detail on changes within releases, covering two at a time.
Please describe Software Quality efforts (CI, security, auditing)
To keep our releases high-quality, we perform:
quick pre-merge CI/CD checks using GitHub actions, performing a combination of correctness checks (build, test) and linter-style checks
lengthier post-merge smoke tests to avoid catastrophes in nightly testing (build more configurations and test a larger number of cases)
nightly correctness and performance testing across a number of axes:
In all, these nightly tests involve running somewhere between 100–200 configurations.
We do not currently have automated security screenings set up, and this is an area where we would like to improve under HPSF, making use of the organization's knowledge and best practices. Code reviews by core developers are our main means of avoiding security issues today (realizing that is imperfect).
Please list the project's leadership team
At the time of this writing, Chapel leadership consists of:
Please list the project members with access to commit to the mainline of the project
The following HPE employees are the project members that currently have merge privileges to our repository:
The full list of contributors/committers to the Chapel project is maintained in our CONTRIBUTORS.md file.
Please describe the project's decision-making process
Most decisions are made on a consensus basis, either internally by the development team at HPE, or by soliciting opinions and feedback from users, depending on the type of decision and affected audience.
For non-trivial internal decisions, or initial explorations into public decisions, we typically form an ad hoc subteam of at least three people to explore the topic and propose a solution, where anyone on the team is encouraged to participate if they are interested. Others are expected to go along with the subteam's decision, barring some sort of unforeseen show-stopper or major objection. Once the subteam has reached a conclusion, it is summarized back to the team, giving others a chance to object if they have an insight or strong reason that the proposal is untenable. In practice, this has happened only very rarely.
Decisions that relate to the design of the language or libraries are captured and discussed on public GitHub issues as a means of soliciting input from the community. Many of these issues are opened or requested by users directly. Decisions that are particularly impactful to user codes are advertised directly to users to get their feedback by pointing them to the GitHub issues or contacting them directly. For some of the most impactful changes, we have run polls and worked hard to involve as many users as possible in order to avoid surprises and attempt to develop a consensus community view.
A key example of this was a change that was spearheaded by users several years ago as part of the push for Chapel 2.0 in which it was proposed that Chapel's built-in types like strings, tuples, and bytes change from 1-based indexing to 0-based indexing (note that a user's arrays can use any lower bound in Chapel). This was a contentious topic within the team and broader user community, leading to much discussion, polling, and weighing of the tradeoffs before deciding to make the switch. Fortunately, with Chapel 2.0 complete, many of these breaking changes are behind us, and those that remain are considered unstable features and flagged as such.
What is the maturity level of your project?
Using the HPSF terminology, we consider ourselves to be at the 'Established' stage.
Specifically, we believe we easily meet the Sandbox and Established requirements, yet we are also clearly not yet a Core project in terms of governance, number of organizations with merge privileges, or security protocols.
Meanwhile, the characterization of Established projects fits us well: We are interested in creating a lifecycle plan and leveraging the expertise within HPSF and the Linux Foundation in doing so; we have a significant, diverse, and devoted user base, but are interested in growing in all these areas; and we are interested in working toward becoming a Core project.
Please list the project's official communication channels
Please list the project's social media accounts
Please describe any existing financial sponsorships
The Chapel team at HPE is primarily funded through contracts focused on improving Chapel for all open-source users. HPE's hardware resources are used for development and testing infrastructure.
Please describe the project's infrastructure needs or requests
We have no specific or pressing needs at this time. However, under HPSF, it could be attractive to look into moving some subset of our testing and release process into a (more) public setting that would permit non-HPE contributors to access the systems and logs. Currently, this is all done within HPE's firewall, making it inaccessible to external developers without having an HPE developer relay information to them or try things on their behalf.
Ongoing costs of the project include web hosting (currently at Dreamhost) and DNS registration. Other potential "plus-ups" that would be attractive, but that we have lived without so far, might include paid runners for CI/CD or higher performance servers for trying Chapel within GitHub codespaces for people who want to experiment with it in a more parallel and performant setting. We have also discussed the creation of a mechanism for giving the public the chance to try Chapel on public cloud systems like AWS.
The text was updated successfully, but these errors were encountered: