The US Government, Open Source Software and Analyzing 786 Pages of Responses - Highlights
Serkan Holat
Posted on December 15, 2023
A semi-professional content analysis on the submitted responses to the US Government's Request for Information on Open Source Software
Part II: Highlights
Welcome to the second part of the series, covering the sections that stood out while reviewing the responses to the US Government's Request for Information (RFI) on Open Source Software.
You will find a compilation of highlights from 25 organizations, including tech companies, foundations, universities, and security firms. Mainly in a single-paragraph format and alphabetically ordered, they are collected under seven topics, each with a link to the original response and without additional commentary.
As a reminder, my priority centered on how the organizations define open source software, its challenges, and the proposed solutions, particularly from a funding perspective. Therefore, you will see fewer items related to technical solutions, even though the RFI's primary focus is on software security.
Enjoy reading the collection, which I hope should provide a distinct insight into the overall state of the ecosystem. As always, please feel free to share your thoughts and questions!
Topics
- Current State
- Facts and Figures
- Most Important Area
- Public Funding
- Ideas and Insights
- Let's Be Frank
- Rules and Regulations
Current State
Modern day funding of core open source projects is not likely to generate revenue, because the current development model is not sustainable for many small open source projects. As a result, the current open source model does not incentivize or reward secure development practices.
In sum, open source software security is a public policy problem in the same way that “health” or “housing” or “foreign affairs” is a public policy problem. It’s just newer. And that newness is exciting, but that same newness should also be a reason for some caution and modesty.
One of the greatest challenges to sustaining open source software lies in its economic model. Access to the benefits of software which has been developed and licensed as open source is freely available to all actors in the ecosystem. Many companies deriving the benefits of OSS for a range of well-understood reasons contribute back to the projects which serve as the basis of their commercialized offerings, while many more do not. This later behavior is referred to as “free riding” and is not one of the industry’s better practices.
To deal with [the lack of investment and participation in open-source software] challenges effectively, governments must adopt a long-term view that considers the security, sustainability, and resilience of these vital systems. Instead of prioritizing short-term interventions, governments should invest in holistic solutions and strategic partnerships that lower systemic risk, improve efficiency, and increase competitiveness.
However, many companies that rely on OSS do not contribute back financially to the OSS projects and communities upon which their service offerings rely. The very licenses that govern open source require freedom of use, which means that the commercial entities that use OSS are free to do so without contributing to the security and health of the software. This is sometimes referred to as the “free-rider” or “tragedy of the commons” problem. This freedom to use and reuse has resulted in a boon to innovation, but not in economic equity and sustainability. Inconsistent support for open source projects by the commercial entities that use them has been a difficult problem to solve.
Open@RIT - Rochester Institute of Technology
In an era where digital advancements are critical for addressing global challenges, including climate change, public health crises, and technological innovations, there is a growing recognition that the siloed nature of digital infrastructure development is unsustainable. To harness the full potential of digital knowledge, it is imperative to foster an environment that promotes openness, collaboration, and the removal of barriers to access.
As a public good, there is a market failure when it comes to dedicating resources to open-source communities. There are few incentives for many organizations to participate, and yet those organizations all benefit when another organization does commit resources and personnel to the cause – a variation of the "tragedy of the commons."
A related issue that represents a real security and stability risk in Open-Source is the reliance of Open-Source Software Foundations upon the goodwill of private sector companies to donate their resources in-kind to cover these infrastructure costs. As it currently stands, the world's basic infrastructure and hosting costs for Open-Source languages (not only Rust, but several of the top 10 most used languages in the world) are supported by only a handful of vendors. Any one of these vendors exiting the market or pivoting their strategy would deal a severe blow to the security and stability of the ecosystem, and would have the potential to effectively bankrupt the Foundation affected.
The bottom line for the impact of volunteerism on government and industry is that deep, fundamental security work requires constant, consistent work and uncompensated volunteers don’t typically get work done on a constant, consistent basis.
Facts and Figures
While that scenario of individual developers may seem like an outlier, it is not an insignificant portion of the ecosystem. The 2022 Census II of Free and Open Source Software found that 94% of projects had fewer than ten developers accounting for more than 90% of the lines of code added. In 49 of the top 50 non-npm projects reviewed, nearly a quarter of them had only one developer responsible for more than 80% of the lines of code added.
The FreeBSD Foundation has also identified three security roles that it needs to fill in order to strengthen the project’s security posture to meet current and future needs. These three people will add $500,000 in currently unfunded annual salary expenses. Significantly securing critical OSS software will require a significant increase in investment.
The first version of PyPI was opened for public use over twenty years ago in 2002, and it continues to be the main repository for open-source Python packages. PyPI has grown to host over 490,000 distinct projects, with over 5 million releases, which are downloaded collectively over a billion times per day. Maintainers upload roughly between 12,000-15,000 new releases per day to PyPI. PyPI website typically receives between 3.7 and 4 million unique visitors (humans) per month. PyPI receives approximately 3 billion web requests per day for projects hosted on its infrastructure (approx. 90 billion per month).
A large volume of projects on PyPI ... are maintained by volunteers, not by corporations. Many projects are even maintained by a sole maintainer, often without the resources or even the knowledge to update their package publishing processes to more secure methods. Out of 490,000 projects on PyPI, 91% of projects have a single account with the maintainer role.
The scaling and adoption of memory-safe languages such as Rust has related challenges in the associated scaling of the infrastructure that supports the language. The Rust Foundation is responsible for hosting Crates.io, the package repository for Rust, which alone costs upwards of $40k per month as of August 2023, and this cost rises monthly as the development and adoption of Rust increases. This is one of several infrastructure costs that the Rust Foundation is obligated to meet in order for individuals and companies to use the language to build.
Over the last decade, Sonatype has seen first-hand an escalation of attacks that directly target developer and development infrastructure; Solarwinds is just one example of this type of attack. As part of our efforts to secure software repositories, Sonatype has also invested in human and AI/ML approaches to address these new attacks via malicious packages. Having identified over 250,000 malicious packages, the threat to OSS via public repositories and package managers is real.
Similarly, somewhere around half of open source maintainers do that critical maintenance work alone. In our survey, about 44% of maintainers said they were solo maintainers, and other studies report similar numbers (eg, 57% in the [referenced] study of the most popular projects, and perhaps 93% in the [referenced] study of Python).
Most Important Area
Carnegie Mellon University - Software Engineering Institute
In summary, the Software Engineering Institute (SEI) considers the behavioral and economic incentives area to be the most important as the incentives will need to influence the behavior of the OSS ecosystem thus achieving the goals of the Open-Source Software Security Initiative's (OS3I) proposed initiatives. Without such incentives, the OSS ecosystem will likely continue its current path and trends likely without the changes the OS3I is attempting to achieve. Though this area is the most important for improvement, it may also be the most challenging. The OS3I and government must devise incentives that will work for the diverse and global open-source software ecosystem. That understanding is not as much a technical one as it is a socioeconomic one. Without that understanding, no amount of technical advancement will be effective. With that understanding, OS3I can engage with the open-source software ecosystem to understand the gaps that exists between OS3I's desired state and the interests of the other stakeholders within the ecosystem as to what foundational needs, technical as well as socioeconomic, are required to close that gap.
[Our] response to the ONCD Request for Information will focus specifically on the RFI topic area of “incentives for securing the open source ecosystem.” We believe that this is, in many ways, the most important area called out in the RFI. Open source is not magic! While other areas called out in the RFI are important, if incentives and motivations of open source maintainers are not well-understood by policy-makers, those other improvements will not happen, or will happen only slowly. That puts the question of incentives and motivation on the critical path for almost all other improvements to the security of the open source ecosystem.
Public Funding
Many, if not all, of the recommendations, actions, or potential regulations that may emerge as a result of this RFI will require significant work from maintainers and producers of OSS projects. In some cases the outputs of this process may lead to ongoing costs for things like additional infrastructure and recurring audits. Many OSS projects are run on an entirely voluntary basis by people who are not paid for their efforts, providing them with a significant challenge to fund this kind of additional work. Additionally, any risk of inequitable liability requirements will diminish their available funds to support OSS security best practices and requirements. Just as many other items discussed in this response require funding, finding a way to fund the work needed from small, independent projects is an important element of this overall effort that must be prioritized.
Organized funds to improve open source security are already having an impact. Linux Foundation’s Alpha-Omega Project, Germany’s Sovereign Tech Fund, and, while not its specific focus, the United States’ Open Technology Fund have all contributed to improving the open source supply chain. Innovative efforts like the Open Source Technology Improvement Fund (OSTIF) is finding a way to bridge the communications gap between security professionals and open source project maintainers. More is needed to secure the critical foundations of our shared digital infrastructure, and the USG is in a good position to help advance these existing efforts, and potentially establish new ones.
Atlantic Council - Cyber Statecraft Initiative
Several federal entities such as NASA and the NSF already provide forms of funding for OSS projects, communities, or research. OS3I can leverage its position as an interagency coordinator to collate lessons from these entities learned in their OSS funding experiences into a best-practices framework for government funding.
For timely responses to urgent needs in key OSS projects—for example, those heavily used in government or critical infrastructure—government should have a means of quickly allocating funding or other security resources such as developers, infrastructure, or security audits to those projects when required.
In addition to targeted and timely interventions, government should seek a vehicle for longer-term, sustainable investments in the health of the OSS ecosystem. Government is uniquely well-positioned to take a long-term view on the public goods created by OSS and invest in initiatives to shore up the health of the community to promote these public goods and to forestall costly crises down the line.
Explore the feasibility of creating the equivalent of the Sovereign Tech Fund in the US, with a focus on contributing to sustaining and improving security of the open source software which the US federal government depends upon which might otherwise go untended. Choose a federal agency to create an appropriate agreement to underwrite the effort and give the program a three year pilot period to demonstrate impact.
The federal government should establish an open-source software infrastructure fund like the German Sovereign Tech Fund model. One implementation option would be to expand the Open Technology Fund’s recently established Free and Open Source Software Sustainability Fund with a simple mandate to support important open source digital infrastructure.
Open source software is a critical element of US digital infrastructure that needs financial support through new and existing public funding sources. The federal government should partner with existing OSS foundations such as the Rust Foundation, Python Software Foundation, Eclipse Foundation, Linux Foundation, Open Source Security Foundation (OpenSSF), and many others with expertise in this space. The federal government can leverage their collective knowledge and relationships to most effectively direct funding and support to key projects.
The federal government should consider expanding the availability of public funding programs for open source technologies, similar to grant programs such as the OpenSSF’s Alpha-Omega Program, US AGM’s Open Tech Fund, Mozilla’s Open Source Support Awards, and the German government’s Sovereign Tech Fund. The National Science Foundation’s Pathways to Enable Open Source Ecosystems (POSE) program is a promising experiment in direct public funding of open source innovation, and we applaud its mission to foster wholly new open source foundations and ecosystems. POSE would be most effective if paired with a funding vehicle in the form of an “Open Source Tech Fund,” providing financial support for US and international organizations that maintain key open source projects. This support will help ensure the security and sustainability of critical, widely used and free public services like software repositories.
Identify critical OSS projects/foundations and support them to improve their security, through funds or other contributions. One example is the Alpha-Omega project, a technical initiative of the OpenSSF. Alpha-Omega provides a path for industry to come together to catalyze critical OSS projects and foundations as they improve their OSS security in systemic and durable ways. It does this by funding security staffing, ecosystem-wide improvements, project audits, and security tooling. We would also like to see the OS3I supporting a government funding parallel to Alpha-Omega, similar to Germany's Sovereign Tech Fund or the differently focused Open Technology Fund.
Ideas and Insights
Atlantic Council - Cyber Statecraft Initiative
Government OSPO: The OS3I should serve as a prototype for and coordinator of US government open source program offices (OSPOs). First, OS3I should collate lessons learned from OSPOs in industry, academia, and US and foreign governments into guidance for US government agencies developing their own. It should also coordinate existing and future agency-level OSPOs or similarly tasked offices and serve as a functional single point of contact for OSS issues communicated to the US government in coordination with the ONCD, rerouting outreach to the correct offices and backstopping where none yet exists.
Tracking dependencies: The rapid pace of digital innovation and the informal relationships between OSS dependencies and their downstream beneficiaries has created a digital ecosystem prone to stacking risk on a relatively small number of critical OSS projects. It’s also created challenges for entities—including government—seeking visibility into those points of concentration. Tracking dependencies is key to managing risk, and OSS dependencies run deep and quiet—US government cannot secure what it does not know it relies upon, hampering its ability to preempt or respond to crisis.
Cybersecurity Coalition Comments
All software: While utilizing memory safe languages can contribute towards more resilient software, it cannot be viewed in isolation. Practicing safe cyber hygiene and abiding by industry best practices are also important factors contributing to the overall security of open source software security. It is important to remember that all software has the potential to contain vulnerabilities, and therefore we should avoid concentrating on specific technical issues and instead focus on the overarching security of all critical software.
Tech is evolving: We would also caution against mandating certain controls or specific memory safe languages. Technology is always evolving, and what may be considered best-in-class today could change tomorrow. Therefore, mandating specific controls in a regulation that could stand for decades is short sided. Instead, requiring ‘adherence to security best practices’ or referring to standards or frameworks that are more regularly updated is preferred.
Federally-funded bug bounty programs: The current and most popular bug bounty programs are “Hack the Pentagon” and “Hack Department of Homeland Security (DHS)”. One drawback to these efforts are that they are time-bound events, which some cybersecurity experts believe may not deliver consistent security improvements. A dramatically improved bug bounty program would persist over time and would involve increased public transparency (e.g., live streaming with Social Media presence, drawing broad viewership), industry collaboration, increasing the incentives, and expanding the scope of targets.
SBOM registry: Creating a centralized, accessible SBOM registry that not only stores these materials but also performs automated analysis could be a game-changer. The registry would offer to store an SBOM for any official release of any software product. This could include simple tasks such as automatic conversions (when possible) between CycloneDX and SPDX, automatic dependency checks, license compatibility evaluations, and even security vulnerability assessments. It could also compare SBOMs generated by different providers (for example different Linux distributions shipping the same software with slightly different options). Such a service would provide immediate, actionable insights for developers, making SBOM generation a value-added activity, because of the access of those additional functions. Results of the analysis should be publicly available.
Institute for Security and Technology (IST)
Centralized database of known vulnerabilities: We encourage the U.S. government to focus first on developing a centralized database of known vulnerabilities, along with tools and approaches to identify vulnerabilities at scale. It is critical to first understand the scale and scope of the open-source ecosystem and its associated security issues before identifying opportunities to increase its security.
On memory safety: Memory safety is not a panacea. Memory safety helps with a bug class not highly present in web application languages, frameworks, and APIs - buffer overflows. OWASP notes that memory-safe languages do not address the following bug classes, including insecure, insufficient, or missing:
• Architecture
• Authentication and Session Management
• Authorization
• Input validation and output encoding
• Cryptographic Flaws
• Error Handling
• Data Protection and Privacy
• Secure communications
• Malicious code checks
• Business Logic Flaws
• File and Resource Handling
• API Security
• Configuration
Updating procurement rules: Through a multi-agency effort, the federal government should update its procurement rules to prefer technology suppliers that demonstrate financial and/or engineering support for the OSS in the suppliers own solution stack. This preference should apply to both cloud and on-premises solutions provided to the US Government.
Centrally curated toolset: Right now, projects need to adopt each standard or tool implementing the standard individually and must use customized workflows or configurations, which dissuades many projects from adopting. Toward these standards being adopted by critical open-source Python projects, we propose creating a centrally curated toolset for building Python packages with optimal security practices enabled by default. A central toolset would require a relatively small investment to leverage existing security technology to create widespread adoption in one of the largest open source ecosystems in the world.
Surveying the ecosystem: Because every migration of a project into memory safety requires time and resources, it is critical to prioritize projects that are most important to migrate and to avoid placing undue burden on projects which are less or not safety-critical. The nature of open source consumption doesn’t lend itself to knowing how and where a project is being used, so many project maintainers don’t know whether security must be prioritized for their own projects, increasing the complexity of this task. Effectively prioritizing candidate projects for migration would require surveying the ecosystem with usage information (number of downloads, dependency graph information, and input from consumers like the federal government) and whether their primary function would benefit from using a memory safe language (such as packages implementing cryptography or processing uncontrolled inputs).
On package repos: Engagement with these key centers of distribution [‘warehouses’ (also known as package repositories)], which fill a vital function and are heavily depended on by the ecosystem, to improve their security posture could reap significant systemic benefits for software developers, vendors, service providers and users. Many of the packages and much of the community code found in warehouses are unsigned and have little information on their provenance. As a result, it is challenging to validate that the software received is authentic and intended for use by the maintainer, whether it is legitimate, or whether it is safe.
On memory safety: The fact is that there is a growing use of memory safe languages in development communities. This is a major step forward in good programming practices. However, it is of no value when hastily implemented by developers lacking the proper training and experience to develop quality software. An experienced and security-focused C++ developer is more likely to produce code of greater efficiency and execution security than an inexperienced Rust developer.
The Open Source Technology Improvement Fund
Identifying the under-maintained projects: Projects first need to be identified and categorized by their lifecycle status, language, and relationship to critical infrastructure. There are millions of projects across billions of installs, so we need to prioritize appropriately. A critical exercise that will take approximately $10 million and between one to two years to complete is the identifying of projects in our infrastructure that are under maintained. The administration can start with supporting the Software Bill of Materials and any initiatives that identify software that is no longer maintained.
Let's Be Frank
Un-asked question: ... the U.S. federal government should first define, assess and improve its own open source software security and, next, that of critical infrastructure, before seeking to contribute broadly to the safety and security of free and open source software writ large.
Why focus on the federal government’s own open source software supply chain? Because it’s mostly a black box and likely full of the known dangers and risks that companies and open source software developers have been coping with. Does any entity within the federal government have a machine-readable list of all the software that entity depends upon and the open source software components within it? If the answer is no, and all indications are that the answer is no, then assessing the open source software security of the government isn’t even an unsolved problem; it’s more like an un-asked question.
No intention for change: Unfortunately, we have observed and heard directly from organizations that do not believe the government intends to change the status quo. Their belief is supported, in their opinion, by a lack of specific action by the federal government to hold software organizations directly responsible and accountable. Given this, they hold much of the current rhetoric and public discourse as nothing more than pageantry with no intention for change.
Sonatype does not hold this belief but recommends that the government should take clear, public steps in demonstrating the actions and capabilities of the federal government on these issues.
The Open Source Technology Improvement Fund
Very few good feelings: You need to build trust with the community which has been slowly chipped away over decades of government secrecy, spying, oversight, and perceived miserliness. In order to accomplish even the short-term projects suggested by this RFI, it will be necessary for the government agencies working on this to set aside their egos in the name of enacting change. Frankly, no one in the open source industry is going to willingly sign up for projects that are directly and singularly run by government agencies. There are very few good feelings towards government management or involvement in open source, especially in security. What you need to do is demonstrate that this government is transparent in its funding and support of security improvements for projects with no ulterior motives.
The biggest challenge: Possibly the biggest challenge in defining and acting upon open source projects’ security is that big tech companies will not easily accept having to spend their profits on security efforts that do not just benefit themselves. Proprietary firms do not feel that using open source code is a commitment to the community of users and buyers of their software and hardware. Not only do they not practice any accountability back to open source, they benefit massively from the contributions and code of others for free.
Rules and Regulations
While the general intentions of Europe’s Cyber Resilience Act (CRA) and Product Liability Directive (PLD) are welcomed in general–and their much-needed bolstering of security in particular–we are concerned by the proposed specifics. Proposed regulation imposes obligations too early in the supply chain, risking stifling innovation and jeopardizing the ability of open source organizations, such as the Apache Software Foundation, to make a positive contribution by coordinating the downstream industry with regard to security.
Funds for the support of OSS will require international disbursement with minimum bureaucracy in order for those funds to be allocated appropriately and to generate the desired impact. Any financial contribution or investment frameworks developed to funnel funds towards this work must ensure that they are free from substantial bureaucracy or restrictions.
International collaboration can multiply the impact of federal government activities across all open-source software security priority areas. As with other global challenges and public goods, national governments share common interests in security, even among competitors. Each of the areas identified by the RFI should be on the agendas of most national governments. US diplomacy should encourage other national governments to increase and coordinate investments in open-source software security, utilizing the motion itself to build trust, as well as increasing global resilience by decreasing cyber risk.
MITRE's Center for Data-Driven Policy
The U.S. government should work with and lobby the European Union to modify the current draft language of the CRA to ensure these developers are protected from liability. If we fail to do so, and the CRA is passed with the current language, then the negative effects on the U.S. and global FOSS ecosystem may be substantial.
RTX Technology Research Center
The United States should collaborate with the European Union on the Cyber Resiliency Act to seek standardization of terms such as “Critical Class I/II” and “Critical Infrastructure.” This will enable greater collaboration across international supply chains.
We must collaborate with other governments and security agencies to ensure future regulations and policies prevent fragmentation within the OSS ecosystem. OSS is a public good; the contributors and custodians of this public good that underpins all software must be protected and supported through thought policy and regulation.
Posted on December 15, 2023
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
December 15, 2023