What we could do with funding

[tkralphs]

This is a follow-up to #18, which I'm throwing out as a new discussion. An important question that always comes up when we discuss funding is what we could really do with the money if we had it, so I wanted to give some additional background and then a partial answer to this. These are just some thoughts---the list below is certainly not exhaustive. These thoughts are my own, not those of the Foundation.

For background, a big focus of the Foundation's Technical Leadership Council (TLC) over its history has not really been on developing and maintaining individual projects, but rather on providing services (for projects that opt in) that (1) help users navigate COIN-OR and use the software we distribute; (2) make it as easy as possible for developers to write portable, well-tested software that installs easily in a variety of environments; and (3) allow all the supported projects to function as a coherent and interoperable whole. One tangible result of all this effort has been the emergence of the COIN-OR Optimization Suite, which would not exist as a coherent whole without the efforts of the Foundation.

There is a lot of infrastructure needed to make this possible that probably wouldn't be developed at all if there weren't a centralized model for maintaining it, mostly due to the limited bandwidth of individual developers. This infrastructure can be shared, to the benefit of all projects, but the effort involved in providing it, as well as maintaining those legacy projects that continue to provide important functionality, is large. It remains one of the primary value-added things the Foundation has done and can continue to do going forward. Although less effort needs to be focused now on certain kinds of infrastructure (we can leverage GIthub for many things we used to do ourselves), there are still many services that the Foundation is in a unique position to provide.

Although there was a perceived emphasis on the need for funding primarily to support legacy projects in our original blog post, I fully agree with the idea of focusing on the active, up-and-coming projects. We have to figure out what kinds of support those projects actually need and that's effectively what I wanted to talk about here. To be clear, the modular way in which the current suite of projects interoperates allows new projects to be integrated and old projects to be removed, while maintaining a coherent whole. One way of moving forward would be to focus on how to continue to evolve the whole suite of tools, whatever that consists of and whatever that entails.

So to give a little more detail to the kinds of infrastructure and tools I'm talking about, here is a (probably still incomplete) list some of the things that the Foundation has done and could continue to do with enough funding. Many of these are focused on supporting developers, but at least some are also about supporting users, which is also an important goal that we could put more focus on if we had funding.

1. Help navigating the landscape. Over time, we've put a lot of effort into making our Web site a friendly entry point for users, but there is a huge potential to make it into a much more vibrant portal where users can get help to find the right projects for their needs; understand the landscape and how everything hangs together; ask general questions before they've figured out what project to engage with; find tutorials and documentation, etc.
2. Support for sustainable software development. We've spent a lot of time thinking about what sorts of software engineering practices and operating procedures are needed to help individual project managers in using industry best practices to keep their project sustainable, as well as how to enable the development of a suite of software as large as the COIN-OR suite in tandem by dozens of different people in a decentralized way. The TLC has developed documentation for project managers here, but this obviously need to be continually updated and expanded, which it hasn't been.
3. Standards for interoperability. There are many challenges to be addressed in keeping projects interoperable. Standards for interoperability have allowed individual solvers and libraries to be developed in a modular fashion, which has been the key to developing the COIN-OR Optimization suite into what it is with effectively very little active management. Implementing such standards requires the development of a whole host of additional infrastructure and tools, such as those detailed in the next couple of items.
4. Source control, versioning, and release management. To make it possible for projects to link more easily to their dependencies and to make it easier for many aspects of routine maintenance to be done in a centralized way across all projects, we spent a lot of time thinking about common workflows, including standardized ways of doing source control, versioning, testing, dependency and release management, etc. This required the development of documentation, helper scripts, and other infrastructure. Currently, much of this is documented in the Developer's Guide, which could be further enhanced and requires on-going maintenance.
5. Build tooling. Perhaps the biggest centralized development effort over time has gone into the common build system that allows all projects to be built in a standardized way, with mechanisms for projects to detect and link to dependencies while ensuring version compatibility. The build system we currently use is based on the GNU autotools and has been a big focus of development efforts of the TLC over time. Mainly, this effort has gone into the common core of m4 functions that are custom-developed to allow COIN-OR projects to be auto-configured based on the build-time environment to work both with each other and with other third-party projects. This common core is centrally maintained for use by all projects, which saves effort, but bug fixes still need to be propagated out to all 20+ projects and possibly to multiple versions of each of these projects. This means maintaining additional auxiliary scripts that automate this process. It also means more documentation that should be maintained.
6. Dependency management. One of the keys to keeping the whole stack interoperating is effective dependency management. Back when projects were managed with the subversion version control system (And CVS before that), dependencies were specified using the subversion "externals" mechanism, but the move to git over the last 5 years spurred the development of the new coinbrew script. This script parses the Dependencies files that live in each branch of each project's repository to figure out what additional projects must be fetched and built and to incrementally build and install dependencies in the right order. There is an on-going effort to replace the Dependencies files with YAML files (config.yml) which contain more fine-grained project dependence information and additional metadata that could be used for other purposes. Of course, we could also consider moving to a package manager such as conan.
7. Packaging and distributing. Over time, there's been more and more effort at packaging the various projects that comprise the suite for distribution in various places: Linux distros, homebrew for OS X, conan, conda, vcpkg, Julia, etc. This is all done more efficiently with centralized effort.    
8. ThirdParty and Data projects. Many projects have dependencies on third-party open source projects, which are themselves not packaged. Over time, we've developed a mechanism for downloading source for these projects and building them using our own build system for consistency (see https://github.com/coin-or/coin-or-tools).  
9. Testing and continuous integration. A lot of effort has gone into various mechanisms for automatic build and test. We struggled for years with our own self-hosted Jenkins server before moving to a combination of Travis and Appveyor and now to Github Actions. What makes it possible to do this across the entire suite is the standardization of tooling, the common build system, the dependency management mechanism, and coinbrew. This is again all stuff that individual project managers may struggle to do on their own.  
10. Maintaining portability. Many of the above items contribute to helping developers with something that many find both vexing and time-consuming---maintaining the portability of their code. Many developers work most of the time on a single platform (often Linux) and may not have the knowledge, motivation, and/or the access to infrastructure needed to ensure their codes are building properly and can be easily used across a wide range of platforms. This is something that the combination of build tools, standard procedures for test and CI, and other SOPs can help developers to achieve and that greatly benefits the entire community of users.
11. Docker images. I've been maintaining a set of Docker images that would make it easy to spin up an installation of the full suite of tools and integrate it, .e.g., into a Web app.
12. Binary distributions and installers. We've tried over many years to develop a system of automatically building and distributing binaries with mixed success. On Linux, the easiest path to installing binaries is typically the package manager for your Linux distro, but in any case, building from source is usually easy on Linux. On OS X, homebrew makes it relatively straightforward. Windows is always the challenge. While it has become immensely easier to build from source on Windows with options like WSL and MSys easily available, it may still be desirable to have some kind of GUI installer, such as the one I was developing using the seemingly defunct InstallJammer. On the other hand, making it possible to install on Windows using Conda could be a good solution as well. Recent efforts have also been aimed at utilizing the ability to archive build artifacts from Github Actions as a possible way of providing up-to-date binaries in an automated way (see, for example, https://github.com/coin-or/CoinUtils/actions/workflows/release.yml).
13. Documentation. Lack of documentation has dogged many COIN-OR projects from the beginning and there's a been a lot of effort at improving this. In recent years, common documentation for both users and developers of all projects in the Optimization Suite has been developed and launched on Github pages. Naturally, documentation is something that most project developers find it hard to devote time to, with their limited bandwidth, and this is something that the Foundation can certainly help on.
14. Interfaces and APIs. Another role the Foundation can play is in developing APIs, file formats, and other standards that allow COIN-OR projects (and other solvers) to talk to modeling languages, Python extensions, programming languages, etc. in standardized ways, primarily allow users to build solver-agnostic models natively in various modelling and programming languages. The Open Solver Interface is an example of an effort focused on developing a solver-agnostic API that has enabled many users to interface easily to a variety of linear solvers in C++. This was wrapped and extended to Python. Both FlopC++ and Rehearse were efforts to develop an algebraic modeling language as a C++ class library and CyLP is a similar effort to interface directly to Cbc in Python. The Optimization Services project provides a much more general framework for building solver-agnostic models in memory that we have envisioned for many years as the basis for development of a more general Open Solver Interface v2.0 with the ability to handle non-linear constraints, stochastic optimization problems, perhaps even multilevel. Much of the glue code for doing this already exists in the Optimization Services project, but development has foundered.
15. Data interchange formats. Another innovative (at the time) aspect of the Optimization Services project was its definition of various XML-based formats for data interchange. These had the potential to revolutionize the development of optimization-based applications in the cloud, but XML has since fallen out of favor. An exciting recent development is COIN-OR's association with a project called MOSDEX to develop JSON-based data interchange formats. This is another example of the kind of thing an organization like COIN-OR could sponsor to benefit a wide variety of projects.

All of the above efforts are things that can benefit the entire broader community of developers and users of open-source solvers generally and even developers and users or proprietary optimization software. With the kind of funding we're talking about, the Foundation would surely be able to continue to move things forward in a number of possible dimensions, such as the ones outlined above. What things to pursue in the end is something that would need to be discussed and decided in consultation with the community and those providing the funding, but I hope this makes the answer to the question of what we could do with the funding a little bit more concrete. I'd be happy to hear any comments. 

[sebheger]

@tkralphs I really appreciate most of your ideas. They are showing absolutely in the right direction from my business-driven perspective. In my opinion, the foundation should think about professionalizing some of these services. Regular "funds" could secure that newly developed things could be also sustainably maintained on a long-term basis.