prepfold "amount complete" above 100%

[wcfiore]

I'm performing a fairly standard prepfold command-
prepfold -nsub 128 -mask guppi_57955_P86Y1272_0003_rfifind.mask -p 0.00246068325 -dm 33.08 -searchpdd -o joecandsearchpdd *.fits
After it's done folding it searches over DMs, periods, pdots, and pdotdots as normal, but when "amount complete" reached 100% it didn't stop. It's currently at 145% and climbing...

I'll note I performed this exact same prepfold command sans the -searchpdd flag and it worked just fine.

[scottransom]

Whoa! Really? That's very bizarre... Note that I definitely do *not* recommend that you search over 4 parameters. That could take forever! In fact, it might be a bug because I didn't expect that. It might need the loop to go to 10000% or something. What you should do if you get a good candidate is find the DM where the search signal was maximized and do a search over p, pd, and (if needed) pdd there. Once that is optimized, fold the raw data using -nopsearch and -nodsearch and only allow it to search over DM. Scott

…

On 5/24/19 2:43 PM, William Fiore wrote: I'm performing a fairly standard prepfold command- |prepfold -nsub 128 -mask guppi_57955_P86Y1272_0003_rfifind.mask -p 0.00246068325 -dm 33.08 -searchpdd -o joecandsearchpdd *.fits| After it's done folding it searches over DMs, periods, pdots, and pdotdots as normal, but when "amount complete" reached 100% it didn't stop. It's currently at 145% and climbing... I'll note I performed this exact same prepfold command sans the -searchpdd flag and it worked just fine. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#99?email_source=notifications&email_token=AABAOITE3KXGLGOOPPG5OXLPXAZMHA5CNFSM4HPR3X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVYTVOQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABAOISADSNACBEX23WG6WDPXAZMHANCNFSM4HPR3X4A>.Web Bug from https://github.com/notifications/beacon/AABAOIQBSRHVXA22TQ3G53LPXAZMHA5CNFSM4HPR3X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVYTVOQ.gif [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#99?email_source=notifications\u0026email_token=AABAOITE3KXGLGOOPPG5OXLPXAZMHA5CNFSM4HPR3X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVYTVOQ", "url": "#99?email_source=notifications\u0026email_token=AABAOITE3KXGLGOOPPG5OXLPXAZMHA5CNFSM4HPR3X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVYTVOQ", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

-- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: [email protected] Charlottesville, VA 22903 USA GPG Fingerprint: A40A 94F2 3F48 4136 3AC4 9598 92D5 25CB 22A6 7B65

[wcfiore]

Alright, thanks! Agreed, it's quite strange but it does make sense that searching over too many parameters would cause issues.

[scottransom]

I'll still check into it. Thanks for making an issue!

[parulj3795]

Hi, I ran the following command after finding the DM and period from the cands.txt file -
prepfold -n 512 -p 0.005757491 -dm 3.0 J4037.fil
The amount complete went up to 890% and didn't stop.

[scottransom]

Hmmm. This is really bizarre. I can't reproduce this with data that I have. I'm trying it with the same exact types and numbers of arguments as well as with an input .fil file. It is really bizarre to me.

[parulj3795]

J4037.fil is a typo, it is J0437.fil. The similar command works for other data set (for different pulsar). But even in this case, I am able to get a profile if I use -nodmsearch.

prepfold -n 512 -p 0.005757491 -nodmsearch -dm 3.0 J4037.fil

[keegansmith21]

Hi, I'm trying to execute the command:

prepfold -psr J0255-5304 -o 1224859816_1024_bins -n 1024 -dmstep 1 -npart 120 -npfact 1 -ndmfact 1 -ncpus 8 -dm 17.961 -p 447.723924479956 *.fits -pstep 1 -pdstep 2 -noxwin -runavg -noclip -nsub 256

And i'm having the same issue. Prepfold is trying to search over 2049 dm, p, pd parameters but the search % keeps rising. Strangely, this works fine when i fold over 100 bins instead.

[scottransom]

@keegansmith21 I looking into this right now again and am worried that I won't be able to trigger the issue. But we'll see.

I do have a couple of comments about the command line, though:

1. If you use the command line option "-fine", it sets -ndmfact 1 -dmstep 1 -npfact 1 -pstep 1 -pdstep 2 all at once.
2. -noclip is rarely what you want for radio data unless you are looking at a pulsar with very small amounts of DM sweep across the band. It is quite good at getting rid of impulsive RFI and not messing with your pulsar signal
3. -ncpus greater than 1 doesn't really do anything very useful at present. Working on the OpenMP capabilities of PRESTO is definitely on my ToDo list
4. If you use -psr, you will get the ATNF values for folding a pulsar, but you are also using -p and specifcying it at 447 seconds! If you wanted to specify the pulsar period, don't give -psr and a name, just use -p in seconds (you should be using 0.4477...)
5. -runavg is kind of dangerous for slow pulsars as you can end up removing some of the signal of your pulsar (since it is a form of high-pass filter). However, sometimes it is useful if the power levels in an observation change a lot

Hope these tips help. And I'll let you know if I can figure out what is causing the search to go above 100%

[scottransom]

@keegansmith21 and @parulj3795 I think I just fixed this issue with the latest commit (d365d2d). Basically, if you are searching over too many trails in DM, P, Pdot, and/or Pdotdot space, the integer holding the number of trials overflowed! And when that got turned into a float, there also wasn't enough precision! The new fixes actually tell you how many trials you will really search and will give you a Warning (or a really stern warning) if your number of trials is >1e8 (>1e9).

Please test this and let me know if it helps! Also, using lots and lots of bins in the profiles makes the number of trials a lot bigger. So if you do that, definitely try to use at least one of -nosearch, -nopsearch, -nopdsearch, or -nodmsearch.

[keegansmith21]

Thanks Scott, I'll try out the fix and let you know if it's still an issue.