DIP17: describe native build extensions

[philpax]

As required for XIVR and other plugins that have additional build components.

Unless any objections arise, this will be ratified by 2022-11-18T00:00:00Z.

https://discord.com/channels/581875019861328007/1040413030959284244

[philpax]

todo, integrate the following:

• describe current available docker images and where they're defined, as well as what they are at the time of writing
• describe process/location for updating/creating new docker images. at present, we maintain our own community repository of images so that the relevant staff can review them and make sure they don't do anything naughty.
• clarify what needs is (a way to download dependencies for your build process), what we'll accept, when they're downloaded
• rename the script file to something like goatcorp-build-script.sh or similar (open to suggestions). clarify when it's run. make clear it's only meant for direct dependencies to your plugin, and shouldn't be used to build other things. clarify no network access. clarify build environment. clarify location at project_path.
• for now, images are unversioned, but we'll revisit this when something inevitably breaks

future extensions:

• publish a public key somewhere that you can encrypt secrets with, and the build script can decrypt. future extension as getting this right will require some work to ensure the private key doesn't leak

[Blooym]

For the script, I think something as easy as plogonbuild.sh would work fine, since that's the name of the system building it anyway.

Aa for image versioning, if we publish via actions to GitHub Container Repository, we can tag them with very very little extra effort with a GitHub action on git tag creation.

[philpax]

Apologies for the delay on this. New deadline for changes is 2022-11-27T00:00:00Z.

[KazWolfe]

Approving these changes, but I do have some inline notes that some may find useful.

All that being said, I still feel as though this system is a bit overcomplicated/overengineered for our use case. While I don't think it's worth tearing down what we currently have (as it is, well, already built for the one plugin that would reasonably use it), I suspect this would be a good candidate to revisit in the future. For example, may it suffice to simply build a developer-provided Dockerfile and then mandate that the final output be placed in a specific directory?

Lastly, I think we should make absolutely clear that the extensions provided here are not designed for developers to mutate or significantly change the build process to suit their wishes (e.g. disabling debug symbols), and instead only exists to assist staging of non-C# code. We can get into an entire debate of whether it's better or not (given that we're not on Windows images, probably not) to just make the developer deal with MSBuild as well, but this solution seems fine for now.

[KazWolfe]

Ideally, we would allow users to specify build scripts as part of the manifest to allow for more flexibility in the process, potentially allowing caching of steps, and similar. This would also give developers more flexibility in deciding where to place files (e.g. those that wish to have .plogon/build.sh) as well as generally limiting the amount of hardcoded things for us to deal with.

This, however, is a very low priority need and would be more of a wishlist item than anything. The defined build script may already call other scripts as necessary, rendering a viable workaround.

[philpax]

I'm not going to say "no", but I do think that's potentially overcomplicating things. To some extent, if you need multiple script files, there might be other issues that need to be resolved?

Re caching of steps: that's a decent point. For now, leaning towards "call make in your build script" and revisiting this at a later stage once we have a better idea of how it's being used.

[KazWolfe]

The main reason for "overcomplication" and allowing multiple scripts is to give developers flexibility and freedom in structuring their repositories in a way that best suits their purposes, as well as allow certain components to be more isolated (useful for testing). Granted, this can be done by having the script we call also call subscripts, but then we get back into the entire "hardcoded paths thing." This doesn't have to be solved now, however.

[KazWolfe]

Ultimately, closed-source plugins need to go away. At present, the two (known) closed-source plugins have security concerns related to API keys, which is reasonable but also problematic.

I like the idea of our images having a known GPG key loaded into its keychain (stored as a secret in our runner configs), and then allowing users to specify whatever decryption process as part of their pre-build steps. This will, of course, not solve the issue of the final build artifact leaking whatever was created, but that's the nature of (trusted) client-side code. Plugins that require secure connections to backend servers are, as always, advised to do account or session management themselves.

[philpax]

Agree on all of this. Feel free to propose a follow-up extension in another PR - I don't know enough about how to securely handle secrets in a scenario like this

[KazWolfe]

So long as the image is unversioned, we need to be careful to not introduce any breaking changes. Until such time as we have a system to version build images, may it make sense to guarantee that updates to the base image coincide with Dalamud API bumps?

[philpax]

@goaaats Do you have any problems with that? My concern is that we might need to update the image at a time that doesn't coincide with an API bump, and we're left in the unfortunate situation of either holding back a plugin or forcing an API bump on everyone else.

My feeling is that if we run into issues with breaking changes, we'll just revert the relevant change and either a) make a new image or b) add some kind of versioning scheme (which could be as simple as creating extended-v2 or something while we get the rest of our ducks in line)

[KazWolfe]

Please clarify this to better suit the below additions - would we rather assets be built in plogonbuild.sh or still pre-baked?

[philpax]

For now, pre-baked, depending on the type of asset. Rule of thumb, I guess: if it can be built with the tools available in the image, it's probably fine, but if it can't, then it should be prebaked. If uncertain, ask goat or another build system tech.

[philpax]

For example, may it suffice to simply build a developer-provided Dockerfile and then mandate that the final output be placed in a specific directory?

The concern with that is the Dockerfile could do arbitrary things (like wasting bandwidth / storage space / building Chromium and holding up the box for everyone else). Suppose we could catch that in review. Not sure if a custom Dockerfile is simpler than a script running in a known Docker environment? Guess it's up to your personal preferences. Will think about it some more.

Lastly, I think we should make absolutely clear that the extensions provided here are not designed for developers to mutate or significantly change the build process to suit their wishes (e.g. disabling debug symbols), and instead only exists to assist staging of non-C# code. We can get into an entire debate of whether it's better or not (given that we're not on Windows images, probably not) to just make the developer deal with MSBuild as well, but this solution seems fine for now.

Sure. I'll update that in the next draft.

---

Given the feedback, I'll update the draft in a day or two and we can ratify by the end of the month. Depends on how busy I am.

[karashiiro]

A custom Dockerfile will always be easier than a prebaked sandbox with a custom build script, because you can use an arbitrary base image, which defeats the purpose of the DIP. If we allowed arbitrary Dockerfiles and then restricted the base image to a few allowed images, then we'd be right back where we are now (though that would benefit from built-in caching so there might be merit to that).

[karashiiro]

Also, versioning our base images should be seriously considered at some point for reproducibility. DIP17 gives us confidence that builds are managed by a trustworthy authority, but it doesn't necessarily make our builds reproducible, since the build artifacts may change if the build image changes. Having some record of a build being done on a particular image version would fix that.

[KazWolfe]

Not sure if a custom Dockerfile is simpler than a script running in a known Docker environment?

A script is simpler given:

• The developer actually knows the known Docker environment.
• The developer knows where everything they need is located in said environment.
• The known environment contains everything the developer needs (e.g. loading in OpenCV might be... fun)

A Dockerfile would allow a developer to more easily test their build processes locally, build an environment that's suited for their purposes, and otherwise handle things in a more graceful manner. I do think the point about local testing is particularly salient though - right now if I wanted to make and test an extended build I'd need to pull down the extended container, browse around, figure out where files are dumped, and run the container and build steps locally until they worked - or, worse, reconfigure my local bare-metal environment to behave like the extended image.

[KazWolfe]

The review team specifically should also encourage the use of clearly-defined URLs that point to a specific version (e.g. don't use https://get.myapp.io/download/sdk).

Files or URLs that seem to have a high tendency to break should be rejected by the review team. This does pose a bit of a problem though, as certain providers don't necessarily like "playing nice" and providing versioned files - we'd likely need to recommend users upload these SDK versions to their own file store (or something else?).

[philpax]

Word of Goat says we shan't be supporting custom Docker images, so we'll stick with predefined images for now