Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a tabular output for run-all-attacks #39

Open
suhacker1 opened this issue Oct 1, 2020 · 2 comments
Open

Create a tabular output for run-all-attacks #39

suhacker1 opened this issue Oct 1, 2020 · 2 comments
Labels
blocked good first issue Good for newcomers metrics visualization Part of the three phase metrics visualization project user-facing Features that will directly impact users

Comments

@suhacker1
Copy link
Collaborator

suhacker1 commented Oct 1, 2020

Is your feature request related to a problem? Please describe.
When running multiple attacks, it should be easy to compare the effectiveness of each one.

Describe the solution you'd like.
Returning a table of each attack and respective metrics is a start. This could eventually be turned into graphs and other visuals.
A pandas data frame is a potential format.

Detail any additional context.
TensorFlow Privacy is an excellent example.

@suhacker1 suhacker1 added user-facing Features that will directly impact users metrics visualization Part of the three phase metrics visualization project labels Oct 1, 2020
@suhacker1 suhacker1 added the good first issue Good for newcomers label Oct 3, 2020
@JosephTLucas
Copy link

I'm thinking about working on this. Still introducing myself to the codebase. When you say run-all-attacks, is that run_all_extraction()?

It seems like I'd build this table out of the various ModelExtractionAttack.label_agreement attributes after for s in synths. Does that sound right?

@suhacker1
Copy link
Collaborator Author

Thank you so much for your interest in PrivacyRaven!

That sounds correct! Right now, unfortunately, this issue is blocked on #79 (the new library updates means that the HopSkipJump-based extraction attack no longer works). However, if you're still interested in contributing to PrivacyRaven, here's the project plan and more good first issues.

Feel free to contact me through the Empire Hacking Slack or at [email protected] if you'd like a tour of the codebase or you want to talk about PrivacyRaven more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked good first issue Good for newcomers metrics visualization Part of the three phase metrics visualization project user-facing Features that will directly impact users
Projects
None yet
Development

No branches or pull requests

2 participants