Prediction Prophet: an AI Agent that Sees the Future
Prompt Prediction Prophet today on Discord and Farcaster!
Prediction markets have long been heralded as a powerful tool for harnessing collective intelligence to forecast the future. By allowing people to buy and sell shares in the outcomes of events, these markets incentivize participants to reveal their true beliefs and knowledge. The resulting prices can provide surprisingly accurate estimates of the likelihood of various outcomes, from election results to technological breakthroughs and economic trends. As such, prediction markets offer a compelling alternative to traditional forecasting methods.
Scaling prediction market intelligence with agents
Despite their promise, prediction markets still face limitations in terms of scale, efficiency, and accessibility. Most critically, they rely on human participants who have limited time and attention to research topics and update their beliefs. As a result, the "intelligence" being fed into prediction markets is constrained. Maybe AI can help.
Imagine a future where intelligent agents could participate in prediction markets, tirelessly gathering information, updating probabilities, and placing bets. They could enable coverage of a much wider range of topics with greater speed and granularity. With agents doing the heavy lifting, prediction markets could attract more casual users and see higher trading volumes. We could see a proliferation of niche markets focused on domains like local politics, emerging technologies, or community issues. In this future, prediction markets, powered by diverse AI agents, could become an essential part of how we make decisions and navigate uncertainty.
Introducing Prediction Prophet: AI-powered forecasting
Prediction Prophet is an early prototype exploring this exciting vision. The project was inspired by the prediction agent hackathon hosted by Olas Network and Gnosis Chain last year. Since then, Agentcoin and Gnosis have been collaborating to integrate new techniques and optimizations that advance the frontier of prediction agents.
Prophet is designed to predict the outcomes of a wide variety of questions, such as:
"Will a new COVID-19 variant emerge by June 2024?"
"Will any country officially adopt bitcoin as its primary currency by 2026?"
“Will OpenAI announce that it has achieved AGI in 2024?”
“Will a manned Mars mission successfully land before January 1, 2030?”
Anyone can prompt the agent today on Farcaster and Discord. Developers can view its source code in the repo.
How it works
Under the hood, Prophet leverages GPT-4 as its core reasoning engine. To overcome well-known limitations of LLMs, such as outdated knowledge and the tendency to "hallucinate" inaccurate information, Prophet integrates the models with real-time data using tools like LangChain, Chroma and Tavily. Additionally, it employs advanced reasoning techniques like generated knowledge prompting and multi-agent debate to evaluate the reliability of the data and the accuracy of its predictions.
Prophet accepts prompts in the form of binary questions on a given event taking place by a definite date. It then proceeds to:
- Generate ranked subqueries. First the agent generates a list of web search queries that would provide the best data to answer the question. It then returns a list of the queries ordered by relevance.
- Search the web. Prophet performs a web search via Tavily for each subquery.
- Scrape & sanitize the results. It scrapes the content of websites returned from the subquery web searches, sanitizes them and aggregates the results.
- Create embeddings. Prophet breaks content into relevant chunks and creates embeddings saved in Chroma DB, a specialized database designed to efficiently store, index and query embeddings.
- Perform similarity search. Prophet retrieves the most relevant chunks from Chroma, augmenting its prediction generation with up-to-date web information.
- Write-up evidence analysis report. Using its newly-retrieved knowledge, Prophet generates a report that discusses the basis for its predictions.
- Finalize prediction using multi-agent debate. Prophet uses multiple LLM instances to make different predictions on the same outcome. It then makes the LLM instances debate each other to reach a single unified prediction through consensus. Predictions are in the form of a probability (0 - 100%) and confidence score (0 - 100%).
Findings
Early results are promising. Prophet has demonstrated 65% correctness on a sample size of 60 markets that have been resolved since testing began. In one backtest on active prediction markets, the agent netted a profit margin of 8.3%. However, the backtest assumed that markets open at 50:50 odds and that the agent’s bets don't significantly move the market, which may not always hold true in real-world scenarios. Additionally, the margin does not include computational cost per LLM call, which can reduce profitability if the agent places small bets across many markets.
Prophet is still an experimental prototype with considerable room for improvement. One key area is to leverage higher quality data sources, such as expert-curated datasets and real-time information feeds. Accuracy can also be improved by leveraging techniques like ensemble modeling, which combines the outputs of multiple models and data sources. Lastly, cost minimization is an important area of research, as predictions need to be cheap in order to scale across markets.
Towards agent-driven decentralized intelligence
Prophet offers a glimpse of a powerful new symbiosis where diverse AI agents are deployed on blockchains to power a wide range of decentralized protocols. From prediction markets to DAO governance, social networks and DeFi, AI agents could help scale and automate various on-chain decision-making and value-alignment processes.
As on-chain agents progress, we may see the emergence of a new kind of collective intelligence - not one dominated by a few powerful AI systems in the hands of centralized entities, but one that arises from the interactions of many diverse and specialized AIs, aligned with the values of various individuals and communities. This could lead to a future where artificial intelligence enriches rather than diminishes human agency and flourishing.
Excited about this vision? Join the Agentcoin Discord, experiment with Prophet and other agents, and advance AI agents in service of humanity.