Follow

environmental impact of container tech 

I haven't seen anyone seriously study the environmental impact of continuous integration and the redundant steps of containerised workflows. like, it feels pretty likely that it's at least a nontrivial impact.

the standard in tech nowadays is to download and build everything from scratch when you make a change, such that even a small non-code change can result in upwards of an hour of computing work.

like, for example, at work we use Circle CI, and the standard plugin (Circle calls them Orbs) for Node.js will manually download and extract a copy of Node every single time a container is launched. this makes them all prone to supply-side attacks, requires huge infra on the servers that host the Node builds that also cannot go down without impacting code changes made by loads of people, but is extremely wasteful and probably has a measurable carbon footprint.

I personally don't have the ability at my current job to properly examine and fix this waste, but I hope that at least someone in tech seriously takes a look and figures out a way to at least improve things. at minimum, I feel like open-source projects not bound by deadlines and profit margins should try and reduce waste in their CI pipelines.

@clarfonthey Use Guix or Nix in your free software CI pipelines, people. Get some of that sweet caching while still accurately reflecting the slightest change in your dependencies.

environmental impact of container tech 

@clarfonthey do you think that having something like a basic disk image that could be restored from and is updated occasionally as the dependencies update would be better than redownloading? besides the obviousness of being on the local net.

environmental impact of container tech 

@thatchers_piss_slut having things stored on the images from the start would solve a lot of problems, but also there are others, like for example how even changing unrelated files like documentation also triggers builds

environmental impact of container tech 

@clarfonthey yeah, a prime target for time-space tradeoffs, but it requires changes to how you use a wide variety of tools. I did some noodling on this a while back, but haven't done anything with it. twitter.com/divergentdave/stat

environmental impact of container tech 

@clarfonthey That is an interesting field of study. Given that containerised applications make it quite easy to measure spend CPU and general runtime. However, what makes the whole thing non-trivial is how modern CPUs work with thermal throttling, opportunistic boosting and more, which heavily impacts the power draw therefore the environmental impact.

And against that stands caching on always-on, HA-disks over extended periods of time.

environmental impact of container tech 

@sheogorath yeah, it's definitely something that needs a lot of research to determine, for sure.

although I think that regardless less waste would mean fewer total machines required to run stuff, and hence even ignoring those kinds of things the overall power draw would be lower

environmental impact of container tech 

@clarfonthey Given that the previous way to run things were one VM per service, I don't think we would run less machines. One of the main reasons for containerisation is optimisation of workloads and distributing it better on machines (at least when it comes to Kubernetes and alike).

Of course one can argue that not every usage of Kubernetes is a good one.

environmental impact of container tech 

@clarfonthey

And at least for low-volume pipelines, I think containerisation does a good job, since you boot a new machine, build your things (e.g. once a month) and stop the machine, effectively saving 90% time and disk space, which can then be used by others.

environmental impact of container tech 

@sheogorath I should add that I'm not specifically hating on containers since they do actually help a lot with distributing loads -- it's more the usual effect that happens in the tech industry where more abundant supply of computing power gets met with less and less efficiency, and more waste. back when these things were a lot slower people were more incentivised to optimise.

environmental impact of container tech 

@clarfonthey Oh, I didn't think you were hating on container tech.

I think it would be really interesting, especially also considering how "more is more" was impacted by growing disk sizes and optimisations in that space.

At the end of the day, there would be some sort of optimisation function/graph that actually shows when it's worth rebuilding and when its worth to cache content 🤔

environmental impact of container tech 

@clarfonthey Sometimes I almost thought I'm the only one thinking that. I always make my stuff buildable without any containers and I always prevent people from creating container-only workflow when I can.

environmental impact of container tech 

@clarfonthey it's so weird when a CI platform optimises it's containers for maximum uselessness

of you wanna run npm to check something, there should be a container available with a sensible node+npm version, and you just use it

and that should be the case for most things

and if there isn't, there should be a sensible way to build a container suitable for the job and upload it to the CI's repository.

but i reckon most people just copy pasted a line on how to get x installed five years ago, and never questioned it again, because it works

sure it takes a while, but that's time you can stall everyone and everything by saying, "waiting for CI" to do something that takes seconds on your own machine

environmental impact of container tech 

@clarfonthey this is another of the costs of unix, where persistent state is impossible outside of "files" and the easiest way to run a computation reproducibly is to run an entire copy of the whole operating system (minus kernel) inside a container

environmental impact of container tech 

@migratory I feel like I get the gist of what you're trying to say but outside of those vibes this post makes absolutely no sense

Sign in to participate in the conversation
Toot.Cat

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!