Use the vitals package with ellmer to evaluate and compare the accuracy of LLMs, including writing evals to test local models.

This desktop app for hosting and running LLMs locally is rough in a few spots, but still useful right out of the box.

PolyCouncil is an open-source application that orchestrates multiple local Large Language Models (LLMs) running in LM Studio. Instead of querying a single model, PolyCouncil runs multiple models in ...