Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

examples: Adding Sugarscape IG - polars with numba vectorized loop #91

Merged

Conversation

adamamer20
Copy link
Collaborator

@adamamer20 adamamer20 commented Sep 1, 2024

This PR introduces a Numba implementation for the loop in SugarscapePolars to optimize agent move calculations. Key points:

  1. Implementation: Added a Numba-accelerated function to determine the best moves for agents, replacing the more complex DataFrame-based logic.

  2. Validation: Implemented equality checks across different Polars implementations to ensure consistent results.

  3. Performance Comparison:

    • Numba-CPU and Numba-Parallel are the fastest.
    • DataFrame looping is now the slowest due to added complexity for correct agent priority handling.
    • Numba GPU implementation is slower than non-vectorized CPU, likely due to data transfer overhead and the nature of the parallelism in this problem.
  4. Numba Considerations:

    • Requires explicit type and size specifications for vectors.
    • Input vector must match output size.
    • Limited to 1D vector operations.
  5. Performance Bottlenecks:

    • Major time consumption in join operations, particularly in neighborhood calculations and cell updates.
    • Potential for optimization with lazy evaluation and an improved backend.
  6. Future Optimizations:

    • Explore lazy evaluation approaches for join operations.
    • Non-vectorized loop directly on Polars DataFrame (without converting to numpy vectors) might offer potential speed improvements.

@adamamer20 adamamer20 linked an issue Sep 1, 2024 that may be closed by this pull request
@adamamer20 adamamer20 marked this pull request as draft September 1, 2024 14:38
Copy link

codecov bot commented Sep 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

…o 67-sugarscape-instantaneous-growback-polars-with-numba
…hborhood. For both numba and completely vectorized it's easier to reason this way then update the current sugar and "best moves" ranking when agents move
…ght make the same move and haven't found the optimal move yet). This avoids race conditions.
…ssary since we prepare the neighborhood looking at potential/max sugar anyway)
@adamamer20 adamamer20 marked this pull request as ready for review October 2, 2024 07:13
@adamamer20 adamamer20 changed the title Adding Sugarscape IG - polars with numba vectorized loop examples: Adding Sugarscape IG - polars with numba vectorized loop Oct 2, 2024
@adamamer20 adamamer20 self-assigned this Oct 2, 2024
@adamamer20 adamamer20 added enhancement Improvements to existing features or performance. examples additions or modifications to sample models labels Oct 2, 2024
@adamamer20
Copy link
Collaborator Author

This is ready.


# Filter impossible moves
# Filter only possible moves (agent is in his cell, blocking agent has moved before him or there is no blocking agent)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the previous one, although more concise, not considered favorable?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean the previous implementation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pl.col("blocking_agent_order").is_null() is to allow agents to move if there is no blocking agent.

I changed the title for clarity

@rht
Copy link
Contributor

rht commented Oct 3, 2024

Approved because my comments are questions instead of blocking requests.

@adamamer20 adamamer20 merged commit 1d7b3ed into main Oct 28, 2024
5 checks passed
@adamamer20 adamamer20 deleted the 67-sugarscape-instantaneous-growback-polars-with-numba branch October 28, 2024 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvements to existing features or performance. examples additions or modifications to sample models
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SugarScape Instantaneous Growback (Polars-With-Numba)
2 participants