Compare trading vs prediction accuracy #544
Conversation
WalkthroughThis pull request introduces several modifications across multiple files within the Changes
Possibly related PRs
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (6)
scripts/compare_trading_vs_prediction_accuracy.py (4)
1-6: Consider adding type hints and remove extra blank line.Consider importing
typingrelated utilities for better type safety. Also, there's an unnecessary extra blank line.import typer from web3 import Web3 +from typing import List, Optional from prediction_market_agent_tooling.markets.omen.omen import OmenSubgraphHandler -
7-9: Add input validation and documentation.The function would benefit from input validation and documentation explaining its purpose and parameters.
def main(public_key: str) -> None: + """Compare trading and prediction accuracy for a given Ethereum address. + + Args: + public_key: Ethereum address in hex format + + Raises: + ValueError: If the public_key is not a valid Ethereum address + """ + if not Web3.is_address(public_key): + raise ValueError(f"Invalid Ethereum address: {public_key}") + public_key_checksummed = Web3.to_checksum_address(public_key)
34-40: Enhance results reporting and add data persistence.The current output format could be improved with more detailed statistics and structured output options (e.g., JSON, CSV).
- print("N bets:", len(all_bets)) - print("Bet accuracy:", len(correct_bets) / len(all_bets) if all_bets else None) - print( - "Prediction accuracy:", - len(correct_results) / len(all_bets_with_results) if all_bets else None, - ) - print("N bets without results:", len(all_bets_without_results)) + import json + from datetime import datetime + + results = { + "timestamp": datetime.utcnow().isoformat(), + "address": public_key_checksummed, + "statistics": { + "total_bets": len(all_bets), + "bet_accuracy": len(correct_bets) / len(all_bets) if all_bets else None, + "prediction_accuracy": len(correct_results) / len(all_bets_with_results) if all_bets_with_results else None, + "bets_without_results": len(all_bets_without_results), + "total_bets_with_results": len(all_bets_with_results), + } + } + + # Print human-readable format + print(json.dumps(results, indent=2)) + + # Optionally save to file + output_file = f"accuracy_report_{public_key_checksummed}_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}.json" + with open(output_file, 'w') as f: + json.dump(results, f, indent=2)
43-44: Enhance CLI with additional options and documentation.The CLI could be improved with better documentation and additional options for output format and verbosity.
+def version_callback(value: bool): + if value: + print(f"Prediction Market Analysis Tool v1.0.0") + raise typer.Exit() + if __name__ == "__main__": - typer.run(main) + app = typer.Typer(help="Analyze prediction market trading and accuracy.") + + @app.command() + def analyze( + public_key: str = typer.Argument(..., help="Ethereum address to analyze"), + output_format: str = typer.Option("text", "--format", "-f", help="Output format: text/json/csv"), + verbose: bool = typer.Option(False, "--verbose", "-v", help="Enable verbose logging"), + version: bool = typer.Option(False, "--version", callback=version_callback, help="Show version"), + ): + """Analyze trading and prediction accuracy for a given address.""" + if verbose: + logging.basicConfig(level=logging.INFO) + main(public_key) + + app()prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py (1)
912-924: Implementation looks good with room for optimization.The method correctly retrieves and filters agent results for a specific bet. The error handling is appropriate, raising a clear error message for multiple results and returning None when no results are found.
Consider adding a query parameter for transaction hash in the subgraph schema to optimize the filtering process, as the current implementation fetches all market results before filtering in memory.
prediction_market_agent_tooling/markets/omen/omen.py (1)
452-452: LGTM! Consider adding a comment for clarity.Good refactoring to replace the magic number with
BPS_CONSTANT. This improves maintainability and consistency.Consider adding a brief comment explaining the BPS (basis points) conversion:
+ # Convert probability to basis points (1 BPS = 0.01%) estimated_probability_bps=int(traded_market.answer.p_yes * BPS_CONSTANT),
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (5)
prediction_market_agent_tooling/markets/omen/data_models.py(3 hunks)prediction_market_agent_tooling/markets/omen/omen.py(2 hunks)prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py(1 hunks)prediction_market_agent_tooling/tools/utils.py(1 hunks)scripts/compare_trading_vs_prediction_accuracy.py(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- prediction_market_agent_tooling/tools/utils.py
🧰 Additional context used
📓 Learnings (1)
prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py (2)
Learnt from: evangriffiths
PR: gnosis/prediction-market-agent-tooling#300
File: prediction_market_agent_tooling/markets/omen/omen.py:344-352
Timestamp: 2024-10-08T17:30:32.487Z
Learning: The `get_resolved_bets_made_since` method in the `OmenAgentMarket` class should include an `end_time` argument to make it more flexible.
Learnt from: evangriffiths
PR: gnosis/prediction-market-agent-tooling#300
File: prediction_market_agent_tooling/markets/omen/omen.py:344-352
Timestamp: 2024-07-08T07:05:58.507Z
Learning: The `get_resolved_bets_made_since` method in the `OmenAgentMarket` class should include an `end_time` argument to make it more flexible.
🔇 Additional comments (4)
prediction_market_agent_tooling/markets/omen/data_models.py (3)
27-27: LGTM: BPS_CONSTANT import added for probability calculations.
The import is correctly placed and will be used for probability calculations in the ContractPrediction class.
789-796: LGTM: New properties added to ContractPrediction.
The new properties enhance the ContractPrediction class with:
estimated_probability: Correctly converts BPS to probability using BPS_CONSTANTboolean_outcome: Provides a clear boolean interpretation of the probability
The implementation is clean and follows best practices.
506-506: Verify the impact of changing transactionHash type.
The type change from HexAddress to HexBytes is logically correct as transaction hashes are typically represented as bytes. However, we should verify that all consumers of this field handle the new type correctly.
✅ Verification successful
Let me generate additional scripts to verify the type handling in the codebase.
Based on the analysis of the codebase, I can now provide the final response:
The type change from HexAddress to HexBytes is safe and properly handled.
The codebase shows consistent handling of transactionHash as HexBytes:
- All usages of
transactionHashin transaction receipts are properly handled by calling.hex()when needed for string representation - The
HexBytestype is properly imported and used throughout the codebase - No direct dependencies rely on the old
HexAddresstype fortransactionHash - The type change aligns with Web3's transaction hash representation
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for usages of OmenBet.transactionHash to verify type compatibility
rg -l "OmenBet.*transactionHash|transactionHash.*OmenBet" | xargs rg -A 5 "transactionHash"
Length of output: 5318
Script:
#!/bin/bash
# Search for HexBytes imports and usage
rg "HexBytes" -A 2
# Search for type conversion or hex-related operations on transactionHash
rg "transactionHash.*hex|hex.*transactionHash"
# Search for any error handling or type checking related to transaction hashes
ast-grep --pattern 'try {
$$$
transactionHash
$$$
}'
Length of output: 38060
prediction_market_agent_tooling/markets/omen/omen.py (1)
81-81: LGTM!
The import of BPS_CONSTANT is correctly placed and follows the project's import organization.
| all_bets_with_results = [] | ||
| all_bets_without_results = [] | ||
| correct_results = [] | ||
| for bet in all_bets: | ||
| result = OmenSubgraphHandler().get_agent_results_for_bet(bet) | ||
| if result is None: | ||
| all_bets_without_results.append(bet) | ||
| continue | ||
| all_bets_with_results.append(bet) | ||
| if result.boolean_outcome == bet.fpmm.boolean_outcome: | ||
| correct_results.append(result) | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Optimize performance with batch processing.
The sequential processing of results could be slow for large datasets. Consider implementing batch processing and adding progress tracking.
+ from tqdm import tqdm
+ from concurrent.futures import ThreadPoolExecutor, as_completed
+
all_bets_with_results = []
all_bets_without_results = []
correct_results = []
- for bet in all_bets:
- result = OmenSubgraphHandler().get_agent_results_for_bet(bet)
- if result is None:
- all_bets_without_results.append(bet)
- continue
- all_bets_with_results.append(bet)
- if result.boolean_outcome == bet.fpmm.boolean_outcome:
- correct_results.append(result)
+
+ handler = OmenSubgraphHandler()
+
+ def process_bet(bet):
+ return bet, handler.get_agent_results_for_bet(bet)
+
+ with ThreadPoolExecutor(max_workers=10) as executor:
+ futures = [executor.submit(process_bet, bet) for bet in all_bets]
+ for future in tqdm(as_completed(futures), total=len(all_bets), desc="Processing bets"):
+ bet, result = future.result()
+ if result is None:
+ all_bets_without_results.append(bet)
+ continue
+ all_bets_with_results.append(bet)
+ if result.boolean_outcome == bet.fpmm.boolean_outcome:
+ correct_results.append(result)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| all_bets_with_results = [] | |
| all_bets_without_results = [] | |
| correct_results = [] | |
| for bet in all_bets: | |
| result = OmenSubgraphHandler().get_agent_results_for_bet(bet) | |
| if result is None: | |
| all_bets_without_results.append(bet) | |
| continue | |
| all_bets_with_results.append(bet) | |
| if result.boolean_outcome == bet.fpmm.boolean_outcome: | |
| correct_results.append(result) | |
| from tqdm import tqdm | |
| from concurrent.futures import ThreadPoolExecutor, as_completed | |
| all_bets_with_results = [] | |
| all_bets_without_results = [] | |
| correct_results = [] | |
| handler = OmenSubgraphHandler() | |
| def process_bet(bet): | |
| return bet, handler.get_agent_results_for_bet(bet) | |
| with ThreadPoolExecutor(max_workers=10) as executor: | |
| futures = [executor.submit(process_bet, bet) for bet in all_bets] | |
| for future in tqdm(as_completed(futures), total=len(all_bets), desc="Processing bets"): | |
| bet, result = future.result() | |
| if result is None: | |
| all_bets_without_results.append(bet) | |
| continue | |
| all_bets_with_results.append(bet) | |
| if result.boolean_outcome == bet.fpmm.boolean_outcome: | |
| correct_results.append(result) |
| all_bets = [ | ||
| bet | ||
| for bet in OmenSubgraphHandler().get_bets( | ||
| better_address=public_key_checksummed, | ||
| filter_by_answer_finalized_not_null=True, | ||
| ) | ||
| if bet.fpmm.is_resolved_with_valid_answer | ||
| ] | ||
| correct_bets = [ | ||
| bet for bet in all_bets if bet.boolean_outcome == bet.fpmm.boolean_outcome | ||
| ] |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add error handling and logging.
The bet retrieval logic needs error handling for API calls and would benefit from progress logging, especially for large datasets.
+ import logging
+ logger = logging.getLogger(__name__)
+
+ try:
+ logger.info(f"Retrieving bets for address {public_key_checksummed}")
all_bets = [
bet
for bet in OmenSubgraphHandler().get_bets(
better_address=public_key_checksummed,
filter_by_answer_finalized_not_null=True,
)
if bet.fpmm.is_resolved_with_valid_answer
]
+ logger.info(f"Retrieved {len(all_bets)} bets")
+ except Exception as e:
+ logger.error(f"Failed to retrieve bets: {e}")
+ raiseCommittable suggestion skipped: line range outside the PR's diff.
No description provided.