Exploring Llamafile: Mozilla's Attempt in the World of Open Source AI

Now Mozilla tries to make running LLMs easier with Llamafile project. Here's my experience with it.

4 comments

The issue isnt if its easy to use the issue is a matter of compute. A majority of people on the internet only have mobile access. We need a way to let the masses utilise distributed compute in a secure and private way.
- You can run many 7B models on phones with 8GB RAM
  
  I find the 7b models dont have the capabilities id like and i cant image the tokens per second would be very good.