[Discussion] Non-root interferes with helm chart

[MirandaStreeter]

Since #141, OpenVoxProject/openvox-helm-chart is facing the following issue:
OpenVoxProject/openvox-helm-chart#44

/container-entrypoint.d/20-use-templates-initially.sh: line 12: cd: /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet: No such file or directory

I'm reporting this here because the causes (and most fixes) also land here.

Long story short, /opt/puppetlabs/server/data/puppetserver is puppetserver's master-var-dir (separate from vardir (that tripped me up)). It also holds the non-root homedir. It is also a directory people commonly mount a volume over. By default the helm chart does this via the "serverdata" PVC. Prior to #141 this was fine because the container was rootful and the home data lived elsewhere. Now, we get the above error.

If you work around this in any way puppetserver still dies with another, different error:

Caused by: org.jruby.exceptions.LoadError: (LoadError) cannot load such file -- puppet

This is because in #141 the Ubuntu container changed to match the Alpine behavior where we use puppetserver gem install openvox instead of installing puppet via the .deb file. Meaning, we're installing the gems during build time rather than runtime, and it's saving to that same master-var-dir, so it's similarly getting overwritten by the PVC.

On top of all that, anyone's workflow that was relying on the previous homedir location (say, for example, you mounted files into .ssh/ or .gitconfig/) has broken. The same goes for anyone mounting gems to the old location instead of using puppetserver gem install. And even then the latter gets wiped on container reboot.

I'm opening this to discussion because there are multiple approaches, and none of them are perfect.

• Move the homedir and gemdir (maybe puppetserver/apps instead of puppetserver/data?)
  • The homedir is already set via ENV HOME to work around random UIDs, and the gem-path and gem-home are already set via puppetserver.conf, so this would be easy enough.
  • If someone is relying on the old non-root paths they'll have to update it to the new non-root path.
• Specify --confdir/--config for all the puppet/puppetserver commands in entrypoint scripts
  • Less moving parts and easier to understand, imo
  • Anything calling puppet ad-hoc inside the container might not return correct results (don't know why someone would do this)
  • We'd still have to move the gemdir
• Set data persistence on the serverdata PVC to default false
  • Don't have to move the gemdir
  • Another step in the long march towards immutability (of which I have strong feelings about)
  • May break certain workflows. For example, if you're extracting the reports manually instead of sending it to PDB.
• Create an environment variable upstream for the confdir et al
  • I'm seeing prior art for run_dir
  • Blast radius is far larger if anything goes wrong
  • We'd still have to move the gemdir
  • Probably quite useful even outside of this use case
  • Likely duplicating work in the XDG basedir PR

Let me know what people think.

[rwaffen]

#153 is handling some of the points from here already, doesn't it? maybe we should rebase and use this?

for any other topic i'm not good enough. and dont want to paste just wall-of-text from some llm 😅

EDIT:
but if you think this could be somewhat benficial i can post what my codex thinks about this.

[dotconfig404]

Good analysis, thanks.

I am quite interested in hearing about the "march towards immutability" - that is basically the idea that one should simply not have a need to mount that dir in the first place?

Back when I wrote the PR to make this image OpenShift compatible, I also considered pinning --confdir namely to /etc/puppetlabs/puppet, but decided against it, as I was actually running into the issue you mentioned, as I was testing module-related things in the server using puppet module install. Besides, non-root users have the confdir in ~/.puppetlabs/etc/puppet by convention and the $HOME for puppet is (at least in Debian and Red Hat systems) in /opt/puppetlabs/server/data/puppetserver, so /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc would be the correct place for the confdir.

So regarding the confdir location / homedir I would rather suggest an entrypoint script to recreate the structure at runtime and fail with a clear message when $HOME isn't writable.

Regarding the gem-home though, I would say relocation out of the master-var-dir makes sense, with subsequent addition to puppetserver.conf in the gem-path.

Generally though, the best fix imo is upstream. This is something I also considered back then. IIRC the way to determine the confdir and other paths was pretty finicky, in both the ruby openvox/puppet gem and the clojure server code. I am happy to dig deeper into this and get back to speed, been some time since I investigated this. But yea, not sure how deep the rabbit hole goes here right now, a lot of historical baggage, e.g. the cadir moving and being inconsistent between server and gem was one thing that tripped me back then, besides all the obvious puppet branding which I wish we could move away from towards openvox.

[MirandaStreeter]

Ahhh I can't believe I missed #153, that is far more elegant. I kept trying to move the static content out of the vardir (and kept coming up with different approaches because none of them were very satisfying), but that PR just moves the vardir down a level. Much nicer. I'm closing this one in favor of that.

[dotconfig404]

I'd still be interested to hear about your "immutability march". Using the new directory means a migration, as the new var dir lives inside the currently mounted path. Would need some coordination with a PR in the helm repo as well to change the mount path to the new var dir. Certainly doable, but not convinced about a quick decision here.

Also, /opt/puppetlabs/server/data/puppetserver is still something that will easily get mounted anyways in an attempt to mount the vardir, even if sufficiently documented in the image doc. Moving the gem dir instead of the var dir and using a self-healing entrypoint script that fails with a clear error message on permission issues appears to me to be more frictionless.

[MirandaStreeter]

I'm reopening this for further discussion.

I'd still be interested to hear about your "immutability march".

I've had my fair share of customer interactions regarding large fleets of compiler servers. A running theme was wishing for puppetserver orchestration that could be scaled up and torn down as needed. And part of making that viable is the cliche "cattle, not pets" adage. The more we encourage treating container images like mini-VMs to be exec'd into and babied, the more we have to deal with statefulness and brittleness.

If I could wave a magic wand I'd make it possible to set the JAR as the entrypoint itself, load it up with environment variables, separate the puppet code into a sidecar, and rip out the CA components into its own container. From my perspective wrangling a dozen mountpoints for config is antithetical to the advantages containers bring.

Using the new directory means a migration, as the new var dir lives inside the currently mounted path. Would need some coordination with a PR in the helm repo as well to change the mount path to the new var dir. Certainly doable, but not convinced about a quick decision here.

Also, /opt/puppetlabs/server/data/puppetserver is still something that will easily get mounted anyways in an attempt to mount the vardir, even if sufficiently documented in the image doc. Moving the gem dir instead of the var dir and using a self-healing entrypoint script that fails with a clear error message on permission issues appears to me to be more frictionless.

Yes, so, it is mentioned in #153 that it's a breaking change. But from what I can tell it's just holding a restartcounter and the facts (which the agent rewrites on its next run anyway). Maybe if you set reports = store you'll have something worth preserving, but doing that within a container rather than sending it out to the DB is treating containers like mini-VMs which, again, I feel strongly we should discourage at every opportunity.

But even then the "Follow-up" in #153 alludes to the solution. I believe the helm chart can be updated for both the data relocation and image update simultaneously. When you update the mountPath it shouldn't orphan the data. The storage exists in the volume independently of the pod. :)

In either case I'd much prefer to avoid any kind of self-healing scripts. The more moving parts we create the more structural weak points we have to live with.

[anthonysomerset]

Set data persistence on the serverdata PVC to default false

• Don't have to move the gemdir
• Another step in the long march towards immutability (of which I have strong feelings about)
• May break certain workflows. For example, if you're extracting the reports manually instead of sending it to PDB.

Does anybody even do any of these workflows today that would be impacted by this change, because the way i read it this is probably the most elegant and yes anything that reduces the need for persistant volumes and/or increases immutability ftw in my humblest of opinions

[dotconfig404]

Agreed, and to take it further, I would really like to know if the "JAR-container" vision is something more feasible than it is assumed - what do we really need to mount, the SSL dir and the puppet code/data?

Sth like auth.conf is a bit tricky perhaps to set via ENV variables, but surely there's a solution there as well. Right now the image is pretty messy in any case with all the scripts and mountpoints, and I agree that another entrypoint script (aka the self-healing script I mentioned) would add to that.

[anthonysomerset]

Isn't configmap the direct analog for complex configs that can't easily be broken into ENV variables anyway? The challenge there becomes the judgment call between complex enough to use configmap vs ENV vars

[MirandaStreeter]

Oh, yes, configmaps are also perfectly cromulent. My story was more about taking steps to avoid the tech sin of creating a PVC just to mount a *.conf file.

I don't see us being able to whittle down the entrypoint scripts and gunning for the JRE itself until we start separating out components like the CA subsystem, but that's outside the scope of this issue and a discussion for 10.x and beyond.

Does anybody even do any of these workflows today that would be impacted by this change, because the way i read it this is probably the most elegant and yes anything that reduces the need for persistant volumes and/or increases immutability ftw in my humblest of opinions

Quite possibly? I honestly don't know. That's part of why I opened this discussion, to see what assumptions we can break. I'm perfectly happy with submitting that in place of the gem/home path shuffling.