Apache CMS Retrospective
Joe SchaeferThe Apache CMS — invented in October 2010 by select members of The Apache Infrastructure Team (Paul Querna (VP), Daniel Shahaf, Ph.D. (SVN dev), and myself), formally deprecated in June 2015, and finally decommissioned in January 2022 — was always ahead of its time. At its peak, over 100 Apache Top-Level Projects and over 4K users relied on it, but none more than Apache OpenOffice. Never was its prescient performance technology more clearly evidenced than in its content dependency management functionality in the early years of the Oracle donation of OpenOffice to Apache in June 2011.
To be clear: when others speak of dependency management, they are predominately concerned with software dependencies, not content dependencies. It all boils down to well-regulated content “includes” in the templating+build system, which is not at all the same thing as software deps.
This feature was absolutely critical in supporting Apache’s massive https://OpenOffice.org (OOo) website. The RDBMS CMS Sun originally provided to OOo would fall over and die even if you just wanted to fix a typo. In contrast, The Apache CMS ran on a FreeBSD jail on baldr.apache.org: a moderately provisioned, Dell 1950 box that ran on 8 CPU and 24 GB of RAM with a pair of 96GB mirrored hard drives across several jails, and flew through the workflow with relative ease.
Without some gluing CMS like service that can get a contributor into an editing session for the page they want to repair IN A SINGLE CLICK, the cognitive energy is far too great to fix a typo on a webpage today:
- go fish that page out of a github repo,
- fork the repo,
- edit the page,
- commit the change,
- push it,
- create a PR,
- wait until a committer approves and merges the PR,
- wait 10-15 minutes for the staging build to finish while it grinds through all 40K+ of buildable assets (~4GB total size),
- wait for a committer to find and review the published changed content somewhere on the staging site,
- wait for that committer to promote the entire site to production,
- wait another 5-10 minutes for the publication build to finish,
- wait for gitpubsub to push the new content out to Apache’s edge webservers.
With the Apache CMS (webgui), sharing a committable “patch/diff” over email was a one-click operation for any person on Earth, as well as a one-click operation to commit+build+publish for a committer to apply on the project. The whole thing revolved around the sharing of capability URLs in the context of a live Markdown editor with dual-pane instantaneous HTML previews. They allowed an Apache committer on the project to “clone” the baldr-jail-hosted zfs filesystem of a contributor’s (uncommitted) checkout; and subsequently inspect, alter, and commit that cloned checkout by the Apache committer themselves as the committer and not the contributor. Once that commit happens, the CMS not only built it in seconds (since it’s only building the changed files and their handful of dependendent files), but it also provided links to the build and to the live rendering of the content on the staging site for review prior to promotion to production.
The whole One-Click Amazon Patent was critical to customer satisfaction for them. Same thing here; but The Apache CMS was all alone in this space.
The Apache CMS (webgui) was that essential coordination switchboard between all that volunteer energy that has sadly left the org behind.
There are several excuses Apache’s Infrastructure Leadership made as to why it was dropped:
a bus-factor of 1 (me),
phasing out FreeBSD (OpenZFS runs on Ubuntu),
buggy (unreliable zfs clones from a tiny FreeBSD jail),
ugly (thanks, Rich!),
git is better (thanks Greg!)
But the real motive was spite. Between the time it was deprecated in March 2015 and the time it was finally decommissioned in January 2022, it had been running on autopilot in a FreeBSD jail on baldr.apache.org for nearly 7 years. The only maintenance required was (quarterly?) host reboots due to item 4 above, and annual SSL certificate renewals. That’s IT.
When push came to shove in late 2021, I offered Dave Fisher to host the OpenOffice website on Orion at a steep discount. At first, Dave petitioned the board and they approved the expense. Dave offered The ASF to waive the finding fee I agreed to pay him and told me to put that money towards the hosting costs.
What happened next was truly remarkable: the Apache Infrastructure Team, immediately and persistently over a span of weeks, put Dave in the unenviable position of declaring his now exclusive loyalties, according to them: to me and by extension the project’s volunteer community, or to The ASF.
Dave was the key innovator and collaborator behind the scalability successes of The Apache CMS’s incremental build technology. He didn’t invent the solutions; but he worked productively with me on how I added the scalability features he needed to ensure high performance builds from The Apache CMS as applied to the OpenOffice website- which at the time was delivering north of 25M requests a day! Together we discovered a profound application of SSI in his efforts, which I believe are still carried through in the JBake templating system in use today.
Unfortunately, if you look at the content editing going on with OpenOffice’s website on GitHub lately, you can clearly see a massive dropoff in activity when The Apache Infrastructure Team forced Dave to move them off the Apache CMS, and as a result channeled all contribution activity through GitHub alone.
I don’t blame Dave for the ugly choice he was forced to make, nor for the predetermined outcome, but we are obviously no longer on speaking terms since that day.
What is shocking about the whole affair was the absolute belligerence of The Apache Infrastructure Team about the whole effort. I personally held the hand of every project we onboarded to the Apache CMS; to see all of the goodwill that created burned to the ground by autocratic decree was simply unfathomable to me as a long time member of the organization.
All they ever did was give the stranded Apache projects callous ultimatums, and never a single hour of effort dedicated to personally offboarding them to anything else. It was no different with OOo; they simply abused, hazed, and coerced Dave to do their bidding. And so he did.
I resigned in 2018. Couldn’t carry on witnessing it. What I did instead in 2020 was build Orion from the ashes. But even that avenue was foreclosed for Apache projects by the Apache Infrastructure Team.
All for spite.
Sad.