I wouldn't have suggested those models. Just use a semantically fine-tuned BERT.
> GPT-3 Embeddings by @OpenAI was announced this week. I was excited and tested them on 20 datasets. Sadly they are worse than open models that are 1000 x smaller
I am a fairly technical guy (check out my submissions) and I read your links and have no idea how to use these to make responses the way I can with OpenAI.
It says I can input a Source Sentence and compare it to other sentences.
For example, how do I get it to reply to a question as if I am George from Seinfeld?
Embeddings are not for that. Embeddings take text and encode it into a high dimensional vector space. Similar texts will be closer together in the vector space.
The idea I was proposing was to use embeddings as a way to store and retrieve relevant "memories" so the AI could maintain coherence across time. I.e. whenever the user sends a message, we pull up the N most relevant memories (where relevance == closeness in the vector space) and include those in the prompt, so GPT3 can use the information when it forms its response.
I just implemented exactly this. In the corpus I put a few hundred papers I am interested in. Now I can ask a question, the search engine will find a few snippets and put them in the GPT-3 prompt.
HN prevents users from responding to responses to their own comments without some delay to prevent flame wars -- just wait a few minutes next time, or click on the link to the comment directly and you'll be able to reply.
Yes you would still need GPT3 in this system. Right now, the incredibly simple system just wires gives GPT3 a window of the last 100 messages and has it output the next message to send.
The following is an excerpt SMS conversation between two friends:
Transcript:
<splice in the last 100 messages here>
Then you can have GPT3 output what it believes the most likely next message is, and you send it. But this system means it loses context if a message is outside the window. So you can augment this system by creating an embedding of the last few messages of the conversation, and creating a prompt like:
The following is an excerpt SMS conversation between two friends, and relevant past memories that are related to the current conversation:
Relevant past memories:
<splice in the N past messages with the most similar embedding to the most recent messages>
Transcript:
<splice in the last 100 messages>
So this gets you a kind of short term memory (the last 100 messages) and a long term memory (the embeddings).
