Self Hosting Artificial Intelligence {Content}

AI self hosted, privacy gains (potential Beyond Monopolies episode)

Motivation

Artificial Intelligence (AI) is often presented like a complex field, the state of the art being impossible to understand, models too large to train, incredible work in progress moving forward that could change anything, yet a black box inscrutable for anyone except the selected few.

This is truly damaging to the field as it is a fascinating topic and even though indeed nobody can understand it all, we can all benefit from tinkering with it, learning from it and possibly even using it.

It's also often showcased as impractical for a "normal" person with their "normal" computer. Consequently everything must be done in the "cloud", far away from our scrutiny.

This is also damaging as it is simply false. Some very large models are indeed too large to run on a single computers but most, including what was considered the state of the art just a couple of years ago. In fact the trend to scale might be problematic for the entire field.

Regardless of all those limitation the goal here is to showcase that even though not everything can be done on your desktop, a lot can. Composing from that and learning how it works can help to reconsider a potential feeling of helplessness.

Not only can you self-host AI models, use them, adapt them, but there is a whole community and set of tools to help you do so. This movement itself is very encouraging. AI does not have to be a block box. Your digital life does not have to be owned by someone else, even for the state of the art.

Requirements

Software

Familiarity with self-hosting, e.g Linux command line, see Shell, and containerization, e.g Docker. Ideally familiarity with Python the most popular work in AI is currently often done in that Programming language.

Hardware

A desktop with a proper graphic card is recommended. Some solutions do not need it all while others need the last generation of GPU. It is also possible to rend such a configuration in the cloud if necessary, while insuring that the cloud provided has terms of services and overall practices aligned with your needs.

Ideal

Linux desktop with latest generation NVIDIA GPU, Docker installed and running with NVIDIA support.

Used locally

• NLP (NLTK)
• face tracking (OpenCV)
  • PostureMaintenance
• OCR (Tesseract)
• ..? (TensorFlowJS)
• VR and AI workshops with Yannel
• face tracking (Human)
• live deep fake (avatarify)
• green screen (OBS plugin)
• swarm robotics simulation (Argos)
• STT (vosk, deepspeech, long before that PockerSphinx)
• image to 3D avatar (PifHUD)
• video understanding (PySlowFast)
• translation (mozilla, OpenNMT via LibreTranslate)
• edge computer vision (OAK-D)
• segmentation (DIS)
• tf-idf (gensim)
• in browser NLP (nlp_compromise)
• text-to-image (min(DALL·E), Stable Diffusion)
• summarization (bart-large-cnn)
• alpaca.cpp (chat based on LLaMA)
  • also on a standalone VR headset via Termux
• bloomz.cpp (completion relying on BLOOM)
• SantaCoder (code generation built from The Stack)
• CoquiTTS (to test HuggingFace Spaces deploy to Docker)
  • reddit post with limitations
• turbocopilot (code generation relying on llama.cpp)
  • integrated back in Vim, based on previous santacoder in Vim)
• whisper.cpp (STT to generate caption, e.g https://twitter.com/utopiah/status/1649332186233950208 )
• GPT4All-J via Nomic AI chat UI Installer
• shape-E
  • thanks to https://ngwaifoong92.medium.com/introduction-to-shap-e-text-to-3d-a4fb5304642b
  • converting to obj then using `obj2gltf` to get the model for WebXR
    • could rely on trimesh to convert to glTF, see https://trimsh.org/trimesh.exchange.gltf.html after https://trimsh.org/trimesh.exchange.load.html
  • could provide a Web API using Flask for in VR generation
    • generating 4 latent objects in .ply takes 1min on a 2080ti, 1 takes 30s, 10 takes 2min
• Immich

Used non locally but still open

• SantaCoder in SpaSca (XR) https://twitter.com/utopiah/status/1621760472461594624
• article summarization using bart-large-cnn https://twitter.com/utopiah/status/1569917194598862849

example as tweets

• long video, via its transcript, to short text summary
  • https://twitter.com/utopiah/status/1563869342181298176
• motivation
  • https://twitter.com/utopiah/status/1562496197662687232
• image to shape (PiFHUD)
  • https://twitter.com/utopiah/status/1272452811058548739
• STT (local) and translation (remote, Deepl)
  • https://twitter.com/utopiah/status/1529081850534166528
  • https://twitter.com/utopiah/status/1534452229784195074
• text-to-image for XR
  • https://twitter.com/utopiah/status/1562356260615356416

Remarks

• Suppose text input. including dataAsURI
• model card to CO2 equivalent
  • https://twitter.com/utopiah/status/1562398263495593984

Typical process

1. discover news about AI
2. live demo availability, if so try with own data
3. open-source repository
4. models availability
5. image availability, as Docker or replicate (cog)
   1. otherwise make Dockerfile
      1. very often relying on Anaconda3
6. try locally
7. if out of memory (e.g OOM error with Torch) look for slimmed down version
8. if no Web interface available, use Python or NodeJS to make a basic interface
9. build demo using the Web API, integrating with other tools, e.g WebXR

Tooling

• generic
  • Docker
  • notebooks
    • Jupyter
    • Observable
      • https://observablehq.com/d/b92fecbba00dbe31
• dedicated
  • cog
  • Gradio
    • see https://observablehq.com/@utopiah/hf-gradio-discovery
  • HuggingFace Transformers with its pipeline
  • NLTK with its .download() to directly get models and datasets
  • torch
  • tensorflow
  • jax

Constrained model pathfinder

• starting locally then server then cloud
  • cascading : [url1, url2, url3]
• usage base, eg try to use generic terms from model cards and if not fallback to model names
  • as done in https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.pipeline
• Telegram bot as mobile interface
• cloud-init Docker https://stackoverflow.com/questions/24418815/how-do-i-install-docker-using-cloud-init/62540068 that could point to a specific Dockerfile URL or to an existing image on official Hub image or nvcr.io, etc
• can be prioritizing
  • locality for privacy
  • remote for CO2 equivalence
  • a composite based on user weighting

Training followed

• university training
  • AI** at UTC
    • https://webapplis.utc.fr/uvs/
• MOOCs
  • AIClass
  • MLClass
  • Coursera
    • Sequences, Time Series and Prediction
    • Natural Language Processing in TensorFlow
    • Convolutional Neural Networks in TensorFlow
    • Introduction to TensorFlow for Artificial Intelligence, Machine Learning, and Deep Learning
    • Computational Molecular Evolution
  • Dataquest.io
    • Data Cleaning Course
    • Data Science Projects Course
    • Data Visualization Course

To try

• on toolformers
  • https://github.com/cognitiveailab/neurosymbolic
  • https://github.com/minosvasilias/toolformer-zero
• nanoGPT https://github.com/karpathy/nanoGPT
• minGPT https://github.com/karpathy/minGPT/
• on mobile, e.g Docker on the PinePhone Pro
  • camera is not well supported
  • ARM architecture
  • e.g OCR on camera output
  • could start with transformers with a specific model to limit dependencies
• https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/
• https://www.oreilly.com/library/view/machine-learning-design/9781098115777/
• ts_server https://bellard.org/ts_server/

See also

• MyServer
• MyCloudTransition
• Presentation format SelfHostingArtificialIntelligence