This is a commissioned study.

  1. Why self host Hubs Community Edition
  2. How to do it
  3. Why synchronisation for cooperation is how Hubs will survive, thrive, or disappear

If you are short on time and "just" want to get it done :

Si vous parlez francais, regardez la video francophone 🇫🇷 !

Why self host Hubs Community Edition

This is a technical tutorial on how to setup Mozilla Hubs Community Edition. The goal is to be able to host your own social virtual world based on Hubs CE for short. That said it is important to ask why in the first place. If you only care about how, please feel free to skip ahead.

So you might not care for virtual worlds. You might think they are pointless or just a fad. You might even think that, worst than that, they are dangerous. The truth is, you might be right! In fact I personally believe some virtual worlds are detrimental in a lot of different ways. They might be risky in terms of privacy, addiction, etc. So, again, why bother?

Well, I believe if something is not OK then maybe it is possible to... fix it! In order to be able to fix it though one needs the tool and knowledge to do so. Sure, it is possible to build virtual worlds from the ground up. I even tinkered a bit with that to give workshops with kids. It's a wonderful way to learn but honestly it is quite demanding. What is more pragmatic is to adapt existing solutions to your needs and your requirements. This, in my humble opinion, is the genuine value of Hubs Community Edition (HubsCE). What HubsCE allows you to do is self-host your virtual world.

What self-hosting a virtual world mean in practice? Well it means instead of using the official hubs.mozilla.com I can now use hubs.ovh hosted on my own server with my own domain! That might look like a detail but in terms of governancy, in terms of my ability to make HubsCE my own, it is fundamental. Note that as it has recently been announced with Hubs sunset it is even more important than before, few points about that specifically at the end.

So yes, this tutorial explain how to setup HubsCE on your own infrastructure. I will take few shortcuts in order to make this practical today but hopefully it will help sufficiently to consider hosting it on any hardware you might need, not just in the example provided.

More on motivation see :

How to do it

Note that video has a detailed table of content allowing you to jump directly to the right section.

The end goal, a HubsCE instance hosted on an OVH server with my own domain :

In the image above you can see a properly working HubsCE instance running with few participants moving along with their own avatar. Do keep in mind, if you are still discovering Hubs, that the visuals are entirely yours to define. Here in this example I left the most basic room and avatars but you can find other ones, even make your own.

Now in order to get there, you will need :

  1. a cloud service provider, e.g an OVH account
  2. a domain that you own and can modify, e.g hubs.ovh
  3. an emailing provider, e.g a (free) elasticemail account
  4. a terminal, e.g konsole
  5. kubectl installed

You do not need to be a Kubernetes expert nor know OVH to follow this tutorial. You are though expected to be familiar with the command line and have some concept about backend deployment.

In practice the bare minimum is : a browser, a credit card and a terminal. You can buy your own domain with OVH in minutes, you can register with elasticemail also very quickly. If you do not have kubectl installed check https://kubernetes.io/docs/tasks/tools/#kubectl


Shemas of the steps required

From now on I assume you have all the requirements listed above. If you do the rest of the should take you an hour at most if you are familiar with Kubernetes. If you are not, you might hit a snag or two and it might take you half a day but probably a lot less, depending of course on the problem. Once you are familiar with the process, getting a brand new HubsCE instance running should take you less than 30min.

So, finally, the core of the how-to, here are the summarized steps :

  1. login to OVH
  2. create a new Kubernetes cluster
  3. configure your HubsCE setup files locally
  4. deploy that setup to your cluster
  5. update your domain with your cluster assigned IP
  6. update your cluster with certificates
  7. login to your HubsCE instance

Voila! All in all there are 7 key steps to get your own HubsCE instance running.

This might sound very abstract if you are not familiar with Kubernetes but thanks to kubectl and HubsCE scripts it is rather straightforward. Each step also has ways to get feedback to insure that we are in a good state.

Now for the actual steps, assuming you are in a new environment :

  1. login to OVH https://www.ovh.com/manager/#/public-cloud/
  2. create a new project, e.g "HubsCE" with a payment method
    1. with my settings for testing, the monthly bill was less than 10e/month
  3. create cluster (see screenshots below)
    1. my recommended settings are
      1. closest geographical location to you, any sublocation
      2. follow their recommended version of Kubernetes
      3. private network: none
      4. node type : discovery, e.g D2-8
      5. node pool size : default (3, autoscaling disabled)
      6. billing : hourly, no anti-affinity
      7. name : hubsce (can be something else, just write it down)
  4. editing render_hcce.sh file
    1. cf https://github.com/mozilla/hubs-cloud/blob/feature/ce/community-edition/render_hcce.sh
    2. generate yaml file hcce.yaml
  5. download kubeconfig file from OVH pannel
    1. move it to ~/.kube/config or export it
  6. apply it and wait (3min)
    1. kubectl apply -f hcce.yaml
  7. get public IP
    1. kubectl -n hubsce get svc lb
  8. open OVH DNS file and put the public IP there
    1. (empty),*,assets,cors,stream IN A <public-IP>
    2. ping you server and after a minute it should return the new IP

You can connect now! Certificates are missing so see below.

PS: if you are impatient you can use watch kubectl get pods -n hubsce to get the result every 2s (default)

PS: if something goes wrong check the related pod using e.g kubectl logs -f certbotbot-http -n hubsce

Generate and add certificates

  1. do a normal setup with unmodified hcce.yaml
  2. test if it is accessible despite lack of certificate
  3. edit cbb.sh then run it
    1. requires a bit of interaction to validate the 4 domains but takes 5min top
  4. comment the HAProxy certificates line in hcce.yaml
    1. specifically comment - --default-ssl-certificate=hubsce/cert-hcce
  5. apply the modified hcce.yaml
  6. wait for the hap pod to get terminated and a new one to spawn
  7. delete ALL pods if somehow it doesn't work right away
    1. kubectl -n hubsce delete pods --all
  8. voila!

PS: remember than in the current Kubernetes setup the DB is not saved when nodes get deleted. Hopefully a solution using volumes will be clarified soon.

OVH configuration screenshots


Naming your cloud project and accepting conditions

Payment method, know that you can be billed hourly or monthly later

Creating your Kubernetes cluster, starting by picking the geographical region

Selecting the type of nodes for your pool, discovery nodes are cheaper

Created cluster, note how the kubeconfig file can be downloaded now

Details of an active node pool

Multiple pools live, note that you only need one

kubectl example screenshots


Using kubectl to get the load balancer IP

Listing running pods from the pool of a cluster

Getting a custom client

  1. update hcce.yaml with utopiah/ce:stable-latest
    1. cf https://hubs.mozilla.com/labs/community-edition-case-study-quick-start-on-gcp-w-aws-services/
    2. waiting on actions, e.g https://github.com/Utopiah/hubs/actions/runs/7596376153/job/20690008773
    3. build step taking more than 6min
      1. first time at least
    4. failed? check path
    5. using /ce as default repository did work

The process is relying on Github Action https://github.com/Utopiah/hubs/actions/workflows/ce-build.yml

One must use their own Docker Hub repository, e.g https://hub.docker.com/repository/docker/utopiah/ce/general


Docker Hub example of token used to push an image on a custom repository

Github Action to generate a build that will be pushed as a Docker image

Github secret to be able to securely push an image

Why synchronisation for cooperation is how Hubs will survive, thrive, or disappear

Following the recent annoucement of Hubs sunset by Mozilla managing to setup your own HubsCE instance has become even more important.

Ok hopefully by now either you managed to have HubsCE on your own server running, or you have a better understanding on how to do it, or you feel confident you could find the right person to do it for you.

That's honestly wonderful. It means you have already, or soon will have, a working version of your portion of the metaverse! You can invite friends, acquaintances, colleagues or even complete stranger to YOUR space. It's wonderful because you get to define the rules. You get to define how people will behave, and to be pragmatic, what to do when some misbehave. It's also a lot of responsability but it gives you the chance to make a community grow, florish, learn from each other.

Now, I would argue individually it is a good place to be in. Yet, I do not think it is sufficient. Indeed as time passes you and other participants to your virtual social world will find imperfections, get ideas for improvements, etc. It means your version of HubsCE today should evolve to a better, more adapted to your need version. It is the case for you, but also for me, and for pretty much everybody else self-hosting HubsCE!

So, if we ALL need a better version of HubsCE, how do we do it? Well I believe we must coordinate. We must agree on where, e.g :

  • a Website describing HubsCE is and how to install it
  • a repository tracking changes with a license and issues
  • a forum and chatrooms to discuss
  • guidelines describing what is appropriate behavior

Right now you might think "Well, we already do have all this, the official Mozilla ones!" and unfortunately that's not the case anymore. All those are things of the past we must as a community distance ourselves from. Mozilla made the decision that they will not support Hubs, so we must do it instead.

We must now insure that HubsCE can be not just independant from Mozilla but even sustainable. That means finding resources for everything listed above. Having forks everywhere is a typical consequence from a project changing governance and that is fine. What is more challenging is everybody relying on Hubs as it is, self-hosted or not, and have few weeks, maybe even days, to migrate to a host that will be able to handle their next online session. This is the most urgent need but once this is covered, which fork will they pick? Who will maintained the "best" fork? How can we as a community not only help identify such a fork but also support the efforts of those maintaining it?

I believe have a clear governance for the future of HubsCE is the most important aspect. Some of the most eager support of Hubs today are discussing about funding. I do believe it is also fundamental but I think it is secondary. First we must agree on what HubsCE managed by the community, without Mozilla, should be. Only after agreeing on this, does pulling resources makes sense.

This might look superfluous. We might all think "But I know what Hubs is, I just *more* of it, simply better!" to which I encourage you all to spend 5min with a stranger and ask what they think Hubs is. Chances are every single answer will be different. Some people might believe it is first and foremost an XR platform, others a social platforms, other a responsive social platform that also works in XR, others a keystone to the metaverse, etc. So, unless this is crystal clear, finding what the next thing to focus on should be, what to spend resources on, is practically impossible.

I do have ideas on how to do that, transitioning to another structure, bylaws, funding, etc but to be honest it does not really matter. What matters more is that, as has already started on the official Hubs Discord, the people who are the most touched by the transition to both voice their needs but also coordinate, decide collectively what the future of Hubs should be.

Finally, my call to action is simple : get involved!

PS: some possibly useful resources https://fosdem.org/2024/schedule/event/fosdem-2024-2023-ensuring-longevity-strategies-for-sustainable-floss-projects-/

Demonstration in public with Q&A

Francophone

Digging further

Note that you can use https://github.com/mozilla/hubs-compose/ in order to tinker with your own custom client without deploying.

OVH API

instead of clicking on the interface one can use the OVH API using their node client https://ovh.github.io/node-ovh/

consider cloud-init

  • would simplify quite a bit
    • install all related tools i.e kubelet containerd kubeadm
    • run kubeadm join
      • passing the right information, e.g token

email not received

  • "553 Envelope FROM 'noreply@hubs.ovh' email address not allowed"
    • using elasticmail must add each domain as verified
    • updating DNS zone takes minutes then sending email works
Q&A
Beyond OVH

investigate https://community.hetzner.com/tutorials/install-kubernetes-cluster

I did manage to start pods but couldn't get load balancer external IP

consider kubeadm

  • still requires to install kubectl and kubelet on each node

consider k3d on Pi rather than kind

OVH after few weeks of usage gave me the option to use much larger instances

  1. Instances : 20
  2. vCPU : 40
  3. RAM (Go) : 240

installing Immers as a way to fedeate Hubs

Note that this video significantly predates HubsCE. Consequently some work of adaption might be required.


Note

My notes on Tools gather what I know or want to know. Consequently they are not and will never be complete references. For this, official manuals and online communities provide much better answers.