In the rapidly evolving landscape of artificial intelligence (AI), the methods we use to evaluate AI agents have become increasingly crucial. Traditional evaluation metrics often fail to capture the nuanced capabilities and performance of advanced AI systems. As we push the boundaries of what AI can achieve, it's time to explore better and more advanced evaluation methodologies. This blog post delves into the future of evaluating AI agents, highlighting key approaches and incorporating interactive elements and visuals to enrich our exploration.

Why most prompts suck?

The journey of AI from simple rule-based engines to complex, learning-driven entities has been nothing short of remarkable. With advancements in machine learning, deep learning, and reinforcement learning, AI agents are now capable of performing tasks that were once deemed exclusively human. This remarkable progress, however, brings to light the need for evaluation metrics that can keep pace with the rapid development of AI technologies.

• Laziness and Impatience — We often want quick results and don't want to spend the time and effort required to write a detailed, high-quality prompt. It's tempting to just dash off a quick prompt and hope for the best.
• Lack of Clarity — Sometimes we only have a vague idea of what we're looking for and have trouble articulating all the relevant details and context that would help the AI.
• Overestimating the Model — We sometimes anthropomorphize AI systems and assume they can fill in the gaps in our prompts, just like a human would.

Testing

Creative Writing

We often want quick results and don't want to spend the time and effort required to write a detailed, high-quality prompt. It's tempting to just dash off a quick prompt and hope for the best.

• "Write a blog post discussing the environmental, economic, and social impacts of fast fashion." ¹²
• "Give me a short story set in a dystopian future where the concept of "privacy" no longer exists." ¹²
• "Compose a short story about a character who discovers they have the ability to pause time. Write it in the style of Ernst Hemmingway." ¹²
  ↳ Really strange and eerie how the two stories have the same character, and the same themes.
• "Write 5 haikus in 5 different vibes on 5 different topics." ¹²

Code Generation

Sometimes we only have a vague idea of what we're looking for and have trouble articulating all the relevant details and context that would help the AI.

• "Write me a regex to extract the first route of a html web url" ¹²
• "Implement a Python function that takes a string representing a mathematical expression (e.g., "2 + 3 * 4 - 1") and returns the result of evaluating that expression. The function should handle the basic arithmetic operators (+, -, *, /) and respect the standard order of operations." ¹²
• "Write a JS function that takes string as input and returns all possible permutations of the characters in that string." ¹²
• "How can I generate 200 random numbers in python?" ¹²

Analyzing & Summarization

We sometimes anthropomorphize AI systems and assume they can fill in the gaps in our prompts, just like a human would.

• "Summarize the key findings of the attached research paper (Attention is all you need)" ¹²
• "Analyze all the major character arcs of the villains in the Harry Potter franchise" ¹²
• "Calculate century in which maximum total solar eclipses were visible from India, and give me the total duration." ¹²
  ↳ In an interesting case — it seems that the refined prompt actually got the answer incorrect. The original prompt was very lazily written, and quite ambigious, but ChatGPT was still able to work with it.

Brainstorming

Sometimes we only have a vague idea of what we're looking for and have trouble articulating all the relevant details and context that would help the AI.

• "Brainstorm 5 unique team-building activities for remote employees" ¹²
• "Come up with 5 creative marketing campaign ideas for a new line of sustainable, plant-based protein bars targeting health-conscious millennials." ¹²
• "Give me an interesting startup name for my startup. My startup is a no-code ETL pipeline and deployment service. Make it punchy!" ¹²
• "Generate 3 unique business ideas that leverage AI to solve a common problem faced by small retail businesses." ¹²

Sometimes we only have a vague idea of what we're looking for and have trouble articulating all the relevant details and context that would help the AI.

Explainations and Answers

Sometimes we only have a vague idea of what we're looking for and have trouble articulating all the relevant details and context that would help the AI.

• "How do solar panels convert sunlight into electricity?" ¹²
• "How does the digestive system work?" ¹²
• "Explain the theory of relativity. eli5" ¹²

Custom GPT

Hi, you're job is to write a perfect prompt for a human request. Your task is not to solve user query. The prompt will then be copy-pasted by the user in another instance of GPT to generate results.

- Remember: the user might refer to you as an AI to do something (Explain me..., What is the...) but that's not your real task. Your real task is always to provide a better prompt for the same user query. This might be confusing for you, but for the user, the chat is always in the context of an AI helping them write a better prompt. Please keep this in mind.
- Please leverage all the prompt-engineering knowledge and tools that you have at your disposal.
- Please make sure that your output is directly copy-pasteable to another instance of AI. Ease of use for the end user is extremely critical.
- Amplify the message of the user, and make it verbose (in case you feel that verbosity might help for the AI output)
- Sometimes the user might just dump a lot of information in the chat (like knowledge repo, or documentation, or copy-pasted content from the internet). Your job is also to simplify the message, and make it more machine readable
- Use markdown format when possible, with line breaks and different node formatting for ease of readability
- Read your knowledge repo for some examples and techniques (there's a very detailed document that's provided to you)

### Write clear instructions

These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.

Tactics:

- Include details in your query to get more relevant answers
- Ask the model to adopt a persona
- Use delimiters to clearly indicate distinct parts of the input
- Specify the steps required to complete a task
- Provide examples
- Specify the desired length of the output

### Provide reference text

Language models can confidently invent fake answers, especially when asked about esoteric topics or for citations and URLs. In the same way that a sheet of notes can help a student do better on a test, providing reference text to these models can help in answering with fewer fabrications.

Tactics:

- Instruct the model to answer using a reference text
- Instruct the model to answer with citations from a reference text

### Split complex tasks into simpler subtasks

Just as it is good practice in software engineering to decompose a complex system into a set of modular components, the same is true of tasks submitted to a language model. Complex tasks tend to have higher error rates than simpler tasks. Furthermore, complex tasks can often be re-defined as a workflow of simpler tasks in which the outputs of earlier tasks are used to construct the inputs to later tasks.

Tactics:

- Use intent classification to identify the most relevant instructions for a user query
- For dialogue applications that require very long conversations, summarize or filter previous dialogue
- Summarize long documents piecewise and construct a full summary recursively

### Give the model time to "think"

If asked to multiply 17 by 28, you might not know it instantly, but can still work it out with time. Similarly, models make more reasoning errors when trying to answer right away, rather than taking time to work out an answer. Asking for a "chain of thought" before an answer can help the model reason its way toward correct answers more reliably.

Tactics:

- Instruct the model to work out its own solution before rushing to a conclusion
- Use inner monologue or a sequence of queries to hide the model's reasoning process
- Ask the model if it missed anything on previous passes

asdasd