Since creating the first version of the Ada Lovelace chatbot in Dialogflow, one of the main problems has been generating accurate answers.
This week, I’ve been testing the different approaches Dialogflow’s beta knowledgebase offers.
For the first version of the knowledge base I added what is universally seen as one of the most reliable internet sources: Wikipedia. In this case, I downloaded Ada Lovelace’s Wikipedia page and added as a knowledge base. Dialogflow really struggled with extracting the correct answers as they often made no sense, as you can see in the image above.
The next approach was adding texts written about Ada Lovelace, as some research on the internet told me this type of text would be better than Wikipedia. I added these two articles written about her to the knowledge base:
The results of adding these more descriptive texts were slightly better. The answers contained more relevant information, however it was often hidden in a whole paragraph of text. Better, but nowhere near a working conversational agent.
Q & A
The second way of adding knowledge is by adding a CSV file with question and answer pairs. Based on the website of one of the museums I researched, I quickly created 10 Q&A pairs and added these to Dialogflow. Unsurprisingly, this approach worked best! The chatbot could easily answer questions related to the ones in the Q&A document. Succes!
After these tests, I tried a hybrid version of the two. I added both the dense texts and the Q&A file to the knowledgebase. This has a very strange effect, as mentioned in Dialogflow’s documentation:
“Confidence scores are not yet calibrated between FAQs and knowledge base articles. We suggest using only one of them at a time, as the best result may not always score the highest.”
Dialogflow had troubles picking an answer, usually resulting in a fallback answer like “Sorry, could you say that again?’. It also made it very very slow.
So to conclude, adding Q & A data has the best results. Now I’ll need to research if this is possible without manually writing every single question and answer pair…