New Mastodon chart feedback

[timetinytim]

We are in the process of releasing a brand new mastodon chart to replace the old chart for various reasons (See this issue for more details).

As such, we are creating this space as a place to leave feedback on the new chart: features we could add, adjustments to how it currently flows, etc.

[ShadowJonathan]

Will there be any place for auto-scaling and the likes, or is the chart meant for statically scaled deployments?

[b2cc]

Some feedback from my experience with the old chart:

• GatewayAPI support (ingress nginx is dead by end of Q1 '26)
• Allow definition of existing storage (don't assume some CSI readily available, allow definition of existing PV/PVC)
• Images should be able to run root-less and only use ports >1024. Modern clusters often come with SCCs that remove many permissions/capabilities from the runtime environment for security reasons.
• For starters: don't depend on external charts, concentrate on deploying Mastodon.
  • Allow definition of external endpoints for service dependencies
  • For PostgreSQL, Redis and Elastic/Opensearch point users to existing operators or docker-maintained images
• Configmaps for file overrides (to allow easier maintenance of image customizations, e.g custom themes etc.)

[timetinytim]

@ShadowJonathan Autoscaling these deployments can be tricky, since whether a web or sidekiq pod is overloaded is not just a matter of CPU/memory usage. However this is something we would definitely like to have, and something we've been experimenting with internally, so I'd love to include this in the future.

[timetinytim]

@b2cc
In order of each of your points

• We literally had a discussion about this last week, and will definitely be included soon 🙂
• I can see the value in that. @renchap, is there a good reason we shouldn't support in-cluster storage at all?
• 100% agree
• This chart was created with the whole idea of decoupling from dependencies, and the README for this chart has some suggestions for people to use. Though I'd definitely love to hear about others that are not on there.
• Oh that's a good idea, and honestly surprising we didn't have that already.

As time allows I'll try to implement these as soon as I can, and make corresponding issues for the work being done~

[jorijn]

I think it would be cool to have KEDA objects that look at the Sidekiq detailed metrics like queue latency to scale up workers. :)

[b2cc]

@timetinytim : Thanks for the feedback, I just missed the "v0" replacement chart with additional information already arrived. 🙏

In regards to the autoscaling: I wrote a detailed article about autoscaling via the KEDA operator here:

• https://wiki.fediverse.foundation/books/fediverse-foundation-announcements/page/autoscaling-mastodons-sidekiq-queues-with-the-keda-operator

It looks at the Redis queue size an starts to bring up additional pods at a certain threshold. We have this running since years now and it works really nicely. I understand that this goes against the "no external dependencies" principle, but maybe there's still value in that, or you (and others) can check it out to figure out a better way?

fyi: @jorijn @ShadowJonathan

[ShadowJonathan]

FWIW with web node scalability; I use a puma exporter that exports the amount of backlogged connections per container, and bring up more containers if these start approaching a certain threshold

One thing of note here, though, is that these requests are pre-queued by a traefik instance which puts api requests before fediverse requests, meaning users never notice large "thundering herd" spikes.

It may be worth it to create a similar thing with this chart, meaning it's automatically resistant against large influxes of requests. There's an nginx-map based implementation here, though I know this is possible with haproxy ACLs alone (even though I haven't yet looked at converting it yet); https://tech.lgbt/@ShadowJonathan/112150648546817861

[crmason2]

• Allow definition of existing storage (don't assume some CSI readily available, allow definition of existing PV/PVC)

To add to this also allow the setting of storageClassName. I may not want to use my default. It would also be a nice to have to do this by item (Redis, Postgresql, Elasticsearch, etc.) but even having it once would be a huge improvement.

[KrisSimon]

I would like to see a helm chart and a default vaues.yaml that rollout by default a working system. All non-set passwords have to be generated, all components should talk to each other, ... just a running start from where we can start adjusting the deployment.

Currently there are to many settings needed without a proper documentation.

[timetinytim]

@KrisSimon It might be a little tricky to make it 100% deployable without any modification, because there are things like specifying the instance name/URL, which need to be done before it connects to the rest of the fediverse (otherwise rolling it back involves more extra steps).

Although I do see your point about things like generating passwords or secrets from the start. I'll try to see how much I can clean up & automate~

[timetinytim]

So there have been a lot of good suggestions so far in this thread. I'm going to start creating issues branching off from a lot of these requests, both to separate out the discussion of these things in particular, and to track the implementation of each one.

Thank you all again for your suggestions, and please keep them coming! I want to make this chart something that people actually want to use 😄

[b2cc]

@timetinytim : I just discovered something that might be helpful: being able to add DNS aliases into the pods.

Use case:
We're currently hosting several instances behind a firewall and reverse proxy in a private subnet. This can cause issues like instances not being able to connect to each other since standard NAT doesn't route traffic back between instances on a single NAT network interface, or problems with automatic certificate renewal.

Config goes into the pod spec. See e.g.:
https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/#adding-additional-entries-with-hostaliases