Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEBUG mode fails at assert(logger->f); #122

Open
cjlegg opened this issue Feb 18, 2021 · 3 comments
Open

DEBUG mode fails at assert(logger->f); #122

cjlegg opened this issue Feb 18, 2021 · 3 comments

Comments

@cjlegg
Copy link
Collaborator

cjlegg commented Feb 18, 2021

DEBUG mode fails at assert(logger->f); which is line 493 of mpi_alltoall.c. Geoff has noticed the same for alltoallv. Code works overall without debug mode.

@cjlegg
Copy link
Collaborator Author

cjlegg commented Feb 18, 2021

Here are my notes of tracing through the call graph for where logger->f is referred to. This ends with a suggestion for a fix, which I shall try out.

logger->f is assigned to a new file at l536 in _log_data in logger.c (not ever ecluded by conditional compilation)

logger is passed to that call

_log_data is called (twice) by log_data, but inside #if ENABLE_RAW_DATA

logger is passed to that call too

log_data is called by log_profiling_data (no conditional compilation therein)

logger is passed to that call too.

log_profiling_data is called by _commit_data (no conditional comilation round it there)

calls to commit_data have no condtional compilation round them.

in mpi_alltoallv, for example) logger is intialsied at l789 in _mpi_init with logger_init()

logger_init() does not initialise the file

Therefore suggestion - wrap assert(logger->f) in #if ENABLE_RAW_DATA

@cjlegg
Copy link
Collaborator Author

cjlegg commented Feb 18, 2021

wrapping assert(logger->f) in #if ENABLE_RAW_DATA did not work

note of investigation:
first had to fix some marcro calls and wrong variables in the call discussed below to make it compile - can do a pull request for those

first reporduced the error: set DEBUG =1 (in both alltoallv and alltoall configs)- thus:

ran run script ...-counts-variants,
got seg faults at _mpi_alltoall() l1328 using liballtoall_counts.so, and liballtoall_counts_unequal.so and liballtoall_counts_compact.so adn liballtoall_counts_unequal_compact.so
which prints to logger->f

so trying to mask out that call with ENABLE_RAW_DATA

recompiled and ran test again, as job 96232 - seg fault again at mpi_alltoall.c:1328

added an asstiopn to check that it is indeed logger->f that is seg faulting
aslo put some brackets around the test in #if, in case needed - ran as job 96233
alltoall: mpi_alltoall.c:1328: _mpi_alltoall: Assertion `logger->f' failed.

So why has logger-f not yet been initialised here?

@cjlegg
Copy link
Collaborator Author

cjlegg commented Feb 18, 2021

from the comment above showing about the call graph down to where logger-f> is initiallised, that takes place in _commit_data, which does not happen until line 1392 (still in _mpi_alltoall()) or even later, when all the the mpi client program has called MPI_Finalize

temporary solution is to comment out these trace prints to logger->f, then a decision has to be made to make that permanent or to initialise earlier, if that is not detrimental in some way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant