Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose bin/distmap hardcoded number_of_processors? #81

Open
magicDGS opened this issue Feb 27, 2018 · 1 comment
Open

Expose bin/distmap hardcoded number_of_processors? #81

magicDGS opened this issue Feb 27, 2018 · 1 comment

Comments

@magicDGS
Copy link
Collaborator

In distmap, there is a variable called $number_of_processors, which is explained as "This number need to define. How many processors Hadoop can use on each nodes". A quick search finds that it is used for some mapper scripts to set how many processors the node would use (e.g., number of threads param in bwa).

I have two suggestions for this to work in an user-friendly way:

  • Expose as an "advance argument" and default to some constant
  • Grab this information from somewhere in the hadoop configuration

Other suggestions, @robmaz?

@robmaz
Copy link
Owner

robmaz commented Feb 28, 2018

I guess this number can reasonably depend on both the mapper and the hadoop configuration. I do not think it should be exposed to the user. Currently, it is only set for each mapper type, not taking into account possible hadoop constraints. But probably this does not matter - if the job requests more resources than it can get, it will just get less than requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants