@galaxyinferno.codes: well I never really only do one thing, but of my main tasks has been to figure out how to test our RAG system: we are uploading and embedding multiple documents about our company to the AI chatbot database and the goal is that users can ask stuff like “what’s department A responsible for?” Or “what’s the role of Peter Smith?” and get the right answer 💡 But as we change settings and add more info, it gets more and more complex to test if everything still works as intended, you don’t want to ask the chatbot all of these examples by hand every week ❓😅 So I looked at a few (open-source!!) frameworks and for now we’re using “promptfoo”, which supports quite a few different metrics and is fairly easy to setup since you just need to generate some yaml-configuration files and use it as a command line interface 👍🏻 There is honestly so much I could talk about in the area of testing - especially around small-scale applications but I also feel like such a noob because we’re just getting started in this area 🫣🤷🏻♀️
