Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when plotting AMP vs. FREQ over 8 SPWs #94

Open
Jordatious opened this issue Mar 10, 2021 · 6 comments
Open

Crash when plotting AMP vs. FREQ over 8 SPWs #94

Jordatious opened this issue Mar 10, 2021 · 6 comments

Comments

@Jordatious
Copy link
Contributor

Jordatious commented Mar 10, 2021 •
edited
Loading

I'm getting the error below when calling shadeMS in the following way:

shadems -j 32 --xaxis FREQ --yaxis MODEL_DATA:amp 1587795059_sdp_l0.J03323-28075.ms

I managed to get it working when plotting amplitude as a function of time, but not as a function of frequency, as with this call.

Here's some info about the data in case it's useful. There are discontinuities between SPWs, either side of RFI bands. And then obviously the number of channels isn't consistent between SPWs. I'm not sure if that's problematic or not.

190G	1587795059_sdp_l0.J03323-28075.ms
nant: 59
nbaseline: 1711
Columns present: ['UVW', 'FLAG', 'FLAG_CATEGORY', 'WEIGHT', 'SIGMA', 'ANTENNA1', 'ANTENNA2', 'ARRAY_ID', 'DATA_DESC_ID', 'EXPOSURE', 'FEED1', 'FEED2', 'FIELD_ID', 'FLAG_ROW', 'INTERVAL', 'OBSERVATION_ID', 'PROCESSOR_ID', 'SCAN_NUMBER', 'STATE_ID', 'TIME', 'TIME_CENTROID', 'DATA', 'WEIGHT_SPECTRUM', 'SIGMA_SPECTRUM', 'MODEL_DATA', 'CORRECTED_DATA', 'DIR1_DATA']
nspw: 8
Freqs: 880-1419 MHz
Channel width: 836 kHz
Freqs (spw 0): 880.4119873046875-932.2401123046875 MHz (nchan: 63)
Freqs (spw 1): 960.4268798828125-1008.9112548828125 MHz (nchan: 59)
Freqs (spw 2): 1010.4263916015625-1058.9107666015625 MHz (nchan: 59)
Freqs (spw 3): 1060.4259033203125-1108.9102783203125 MHz (nchan: 59)
Freqs (spw 4): 1110.4254150390625-1162.2535400390625 MHz (nchan: 63)
Freqs (spw 5): 1301.4110107421875-1331.5047607421875 MHz (nchan: 37)
Freqs (spw 6): 1332.4190673828125-1379.2315673828125 MHz (nchan: 57)
Freqs (spw 7): 1380.4071044921875-1418.8602294921875 MHz (nchan: 47)
nchans: 444
fields: ['J03323-28075']
npol: 2
Tobs: 9.940239161427598 hours
Dump time: 7.9966169712
nscans: 22

Log:

2021-03-10 11:58:58 - shadems - INFO - using colourmap colorcet.bkr
2021-03-10 11:58:59 - shadems - INFO - using colourmap cmasher.pride
2021-03-10 11:58:59 - shadems - INFO - using colourmap colorcet.glasbey_dark
2021-03-10 11:58:59 - shadems - INFO - /idia/users/jcollier/software/shadems-venv/bin/shadems -j 32 --xaxis FREQ --yaxis MODEL_DATA:amp 1587795059_sdp_l0.J03323-28075.ms
2021-03-10 11:58:59 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:58:59 - shadems - INFO - : MS 1587795059_sdp_l0.J03323-28075.ms contains 44937704 rows
2021-03-10 11:58:59 - shadems - INFO - :   (8, 63) spectral windows and channels
2021-03-10 11:58:59 - shadems - INFO - :   1 fields: J03323-28075
2021-03-10 11:59:05 - shadems - INFO - :   22 scans: 3 5 7 9 11 13 15 17 19 21 25 27 29 31 33 35 37 39 41 43 47 49
2021-03-10 11:59:17 - shadems - INFO - :   59/59 antennas: 0:m000 1:m001 2:m002 3:m003 4:m004 5:m005 6:m006 7:m007 8:m008 9:m009 10:m010 11:m011 12:m012 13:m013 14:m014 15:m015 16:m016 17:m017 18:m018 19:m019 20:m020 21:m021 22:m022 23:m024 24:m026 25:m027 26:m028 27:m029 28:m030 29:m031 30:m032 31:m033 32:m034 33:m035 34:m038 35:m039 36:m040 37:m041 38:m042 39:m044 40:m045 41:m046 42:m047 43:m048 44:m049 45:m050 46:m051 47:m052 48:m053 49:m054 50:m055 51:m056 52:m057 53:m058 54:m059 55:m060 56:m061 57:m062 58:m063
2021-03-10 11:59:23 - shadems - INFO - :   1711/1770 baselines present
2021-03-10 11:59:24 - shadems - INFO - :   corrs/Stokes XX YY I Q
2021-03-10 11:59:24 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:59:24 - shadems - INFO - : Data selected for plotting:
2021-03-10 11:59:24 - shadems - INFO - Antenna(s)       : all
2021-03-10 11:59:24 - shadems - INFO - Baseline(s)      : all except autocorrelations
2021-03-10 11:59:24 - shadems - INFO - Field(s)         : all
2021-03-10 11:59:24 - shadems - INFO - SPW(s)           : all
2021-03-10 11:59:24 - shadems - INFO - Scan(s)          : all
2021-03-10 11:59:24 - shadems - INFO - Channels         : all
2021-03-10 11:59:24 - shadems - INFO - Corr/Stokes      : XX YY
2021-03-10 11:59:24 - shadems - INFO - ------------------------------------------------------
2021-03-10 11:59:24 - shadems - INFO - loading minmax cache from 1587795059_sdp_l0.J03323-28075-minmax-cache.json
2021-03-10 11:59:24 - shadems - INFO - axis: FREQ, range (None, None), discretization None
2021-03-10 11:59:24 - shadems - INFO - axis: amp(MODEL_DATA), corr None, range (None, None), discretization None
2021-03-10 11:59:24 - shadems - INFO -                  : you have asked for 1 plots employing 2 unique datums
2021-03-10 11:59:43 - shadems - INFO - : Indexing MS and building dataframes (44937704 rows, chunk size is 5000)
Traceback (most recent call last):
  File "/idia/users/jcollier/software/shadems-venv/bin/shadems", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/idia/users/jcollier/software/shadeMS/bin/shadems", line 8, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/idia/users/jcollier/software/shadeMS/shade_ms/main.py", line 426, in main
    row_chunk_size=options.row_chunk_size)
  File "/idia/users/jcollier/software/shadeMS/shade_ms/data_plots.py", line 191, in get_plot_data
    value = axis.get_value(group, corr, extras, flag=flag, flag_row=flag_row, chanslice=chanslice)
  File "/idia/users/jcollier/software/shadeMS/shade_ms/data_mappers.py", line 337, in get_value
    self.minmax = min(self.minmax[0], min1) if self.minmax[0] is not None else min1, \
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/array/core.py", line 1622, in __bool__
    return bool(self.compute())
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/base.py", line 283, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/base.py", line 565, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/threaded.py", line 84, in get
    **kwargs
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 487, in get_async
    raise_exception(exc, tb)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 317, in reraise
    raise exc
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/optimization.py", line 963, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/daskms/reads.py", line 176, in getter_wrapper
    return future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/daskms/reads.py", line 38, in ndarray_getcol
    getcolnp(column, result[rr:rr + rl], startrow=rs, nrow=rl)
  File "/idia/users/jcollier/software/shadems-venv/lib/python3.6/site-packages/casacore/tables/table.py", line 1052, in getcolnp
    return self._getcolvh(columnname, startrow, nrow, rowincr, nparray)
RuntimeError: Table DataManager error: Internal error: StManIndArray::get/put shapes not conforming
srun: error: compute-056: task 0: Exited with exit code 1
srun: Terminating job step 11005.0
@Jordatious
Copy link
Contributor Author

Btw, when installing shadeMS via pip (following #95), I get the same error.

@Jordatious
Copy link
Contributor Author

Ok perhaps I should have checked #85... It's a different error that is output though.

@o-smirnov
Copy link
Collaborator

Same underlying cause as #85 I'll bet. Good excuse to bite the bullet and sort out SPW support properly. Could you please split out one scan of your MS for me @Jordatious, for testing?

@Jordatious
Copy link
Contributor Author

Hey @o-smirnov,

Sure! I put the first scan of the target (scan 3) here:

/scratch3/users/oms/1587795059_sdp_l0.J03323-28075-scan3.ms/

Interestingly, I get the same error when I try to plot this manually... I have no idea how plotms managed to get it working...

In [4]: tb.open('1587795059_sdp_l0.J03323-28075.ms')
Out[4]: True

In [5]: dat=tb.query('ANTENNA1==1 AND ANTENNA2==2',columns='MODEL_DATA,TIME')

In [6]: dat.getcol('MODEL_DATA')[0]
2021-03-11 14:25:35	SEVERE	getcol::MODEL_DATA	Exception Reported: Table DataManager error: getArrayColumnCells shape mismatch in column MODEL_DATA

@o-smirnov
Copy link
Collaborator

That's not too surprising. getcol() wants to read the column as a (row,chan,corr) array, and since your selection has a "ragged" channel axis, it simply can't.

I suppose plotms iterates over spectral windows to avoid this, so I should just do the same in shadeMS...

@Jordatious
Copy link
Contributor Author

Ahhh ok that makes sense. You may already know, but this works for me, which you can see corresponds to the channel count from here:

In [9]: dat = tb.query("ANTENNA1==0 AND ANTENNA2==1 AND DATA_DESC_ID==0", columns='MODEL_DATA,TIME')
In [10]: data=dat.getcol('MODEL_DATA')
In [11]: data.shape
Out[11]: (2, 63, 3283)
In [12]: dat2 = tb.query("ANTENNA1==0 AND ANTENNA2==1 AND DATA_DESC_ID==1", columns='MODEL_DATA,TIME')
In [13]: data2=dat2.getcol('MODEL_DATA')
data
In [14]: data2.shape
Out[14]: (2, 59, 3283)

@Jordatious
Copy link
Contributor Author

Yep so this is working for me when I iterate over the above, and it's quite fast actually. The plotting bit is what takes so long...

Just a quick correction to the above, which I've edited - this was the run where I have all 8 SPWs, without overlap, only avoiding RFI bands, so there's no missing bands. It's the one for which I'm running CubiCal, which was partially successful, and which I'll report back about later. Part of determining why it wasn't 100% successful is plotting the MODEL_DATA and DIR1_DATA columns.

This was referenced Mar 16, 2021
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@o-smirnov @Jordatious