Created tests for USGS Spectral Library Version 7 convert function #146

bdegley4789 · 2018-04-26T09:08:09Z

This doesn't need to get accepted right now. I still need to get all the tests to pass, I had said I would do a pr either today or yesterday so you could review what I have been working on before our meeting on Friday at 8am.
It looks like this gets some errors with test_classify_image. I'm guessing this is because I changed the envi .hdr and .sli files from the spectral v06 to v07 files

update branch

Fix broken location of GDAL CURRENT to fixed 2.2.4 in Dockerfile

lewismc

@bdegley4789 please look at my comments. Thank you for writing unit tests. Lets talk tomorrow.

lewismc · 2018-04-26T21:35:01Z

examples/example_mineral.py

@@ -61,7 +61,7 @@


 input_filename = 'avng.jpl.nasa.gov/AVNG_2015_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr'
-library_filename='../pycoal/tests/s06av95a_envi.hdr'
+library_filename='../pycoal/tests/s07_AV95_envi.hdr'


You need to make sure that there are no regressions what-so-ever. We are NOT dropping support for USGS v6. We are adding functionality to use v7... this means that all v6 code should not be touched.
The solution here is to either

introduce a brand new example file for v7 mineral classification, or

introduce a new CLI flag which determines the spectral library at runtime based upon the program input.

lewismc · 2018-04-26T21:35:13Z

examples/example_mineral.py

@@ -120,7 +120,7 @@ def main(argv=None):
        # Setup argument parser
        parser = ArgumentParser(description=program_license, formatter_class=RawDescriptionHelpFormatter)
        parser.add_argument("-i", "--image", dest="image", default=input_filename, help="Input file to be processed [default: ang20150420t182050_corr_v1e_img.hdr]")
-        parser.add_argument("-s", "--slib", dest="slib", default=library_filename, help="Spectral Library filename [default: s06av95a_envi.hdr]")
+        parser.add_argument("-s", "--slib", dest="slib", default=library_filename, help="Spectral Library filename [default: s07_AV95_envi.hdr]")


Same here...

lewismc · 2018-04-26T21:35:56Z

examples/example_spectral07.py

+
+Dependencies
+`USGS Spectral Library Version 7 <https://speclab.cr.usgs.gov/spectral-lib.html>`_
+must be downloaded to the examples directory


must be downloaded to the examples directory

should be changed to

must be downloaded and unzipped to the examples directory

lewismc · 2018-04-26T21:36:11Z

examples/example_spectral07.py

+
+@author:     COAL Developers
+
+@copyright:  2018 COAL Developers. All rights reserved.


change to 2017-2018

lewismc · 2018-04-26T21:36:55Z

examples/example_spectral07.py

+@contact:    [email protected]
+'''
+
+#!/usr/bin/python


This always goes at the top of the file.

lewismc · 2018-04-26T21:40:31Z

pycoal/mineral.py

+        input_file.write('Measurement:  Unknown\n')
+        input_file.write('First Column:  X\n')
+        input_file.write('Second Column: Y\n')
+        input_file.write('X Units:  Wavelength (micrometers)\n')


Writing all of these values still seems way too tightly tied with some very specific file. It is not correct and needs to be much more flexible.

lewismc · 2018-04-26T21:41:29Z

pycoal/mineral.py

@@ -362,3 +368,184 @@ def convert(cls, data_dir="", db_file="", hdr_file=""):
        library = aster_database.create_envi_spectral_library(spectrum_ids, band_info)

        library.save(hdr_file)
+
+class SpectralToAsterConversion:


Change this Class name to something more descriptive.

lewismc · 2018-04-26T21:42:36Z

pycoal/mining.py

-proxy_class_names = [u'Schwertmannite BZ93-1 s06av95a=b',
-                   u'Renyolds_TnlSldgWet SM93-15w s06av95a=a',
-                   u'Renyolds_Tnl_Sludge SM93-15 s06av95a=a']
+proxy_class_names = [u'Schwertmannite BZ93-1         BECKb AREF',


Again, this cannot change... we need to maintain backwards compatibility and add functionality not take away.

lewismc · 2018-04-26T21:44:44Z

pycoal/tests/test.py

@@ -26,7 +26,7 @@ def _remove_files(listOfFileNames):
            pass

 # file names of USGS Digital Spectral Library 06 in ENVI format
-libraryFilenames = ["s06av95a_envi.hdr", "s06av95a_envi.sli"]
+libraryFilenames = ["s07_AV95_envi.hdr", "s07_AV95_envi.sli"]


... and again. You need to add new entries not take away from what we already have.

lewismc · 2018-04-26T21:45:51Z

pycoal/tests/test.py

@@ -26,7 +26,7 @@ def _remove_files(listOfFileNames):
            pass

 # file names of USGS Digital Spectral Library 06 in ENVI format
-libraryFilenames = ["s06av95a_envi.hdr", "s06av95a_envi.sli"]
+libraryFilenames = ["s07_AV95_envi.hdr", "s07_AV95_envi.sli"]

 # set up test module before running tests
 def setup_module(module):


You are not using any of the test files you've introduced. You either need to use them or remove them.

lewismc · 2018-04-26T21:51:56Z

@bdegley4789 I would also like to remind you to always test your code before you submit a pull request. It is VERY simple and takes no longer than about 2 seconds to type and around a minute or so to execute.
https://github.com/capstone-coal/pycoal#tests
Thanks

lewismc · 2018-05-02T02:54:47Z

@bdegley4789 any updates here?

bdegley4789 · 2018-05-02T14:56:56Z

@lewismc I can’t really figure why my tests aren’t passing. I re-downloaded pycoal from the master. I moved the files I changed over to the new pycoal I downloaded and now everything passes. I must have changed something in a directory I’m unaware of. I’d like to figure out what is going on first, if I can't figure it out, I'll just push this new version to my fork and update this pr tomorrow or after I finish work today.

This is what I get when I move all the files I changed over to a new pycoal directory.

Using /Users/bryceegley/anaconda3/lib/python3.6/site-packages
Finished processing dependencies for pycoal==0.5.2
Bryces-MacBook-Air:pycoal bryceegley$ nosetests
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
...........
----------------------------------------------------------------------
Ran 11 tests in 9.731s

OK
Bryces-MacBook-Air:pycoal bryceegley$

On my current fork I still get these 2 failures and 1 error

ERROR: mineral_test.test_classify_image_subset
Traceback (most recent call last):
  File "/Users/bryceegley/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/bryceegley/Desktop/temp/pycoal/pycoal/tests/mineral_test.py", line 135, in test_classify_image_subset
    mc = mineral.MineralClassification(libraryFilenames[0], class_names=mining.proxy_class_names)
  File "/Users/bryceegley/Desktop/temp/pycoal/pycoal/mineral.py", line 60, in __init__
    self.library = self.subset_spectral_library(self.library, class_names)
  File "/Users/bryceegley/Desktop/temp/pycoal/pycoal/mineral.py", line 309, in subset_spectral_library
    old_index = spectral_library.names.index(class_name)
ValueError: 'Schwertmannite BZ93-1 s06av95a=b' is not in list


FAIL: mineral_test.test_classify_image
Traceback (most recent call last):
  File "/Users/bryceegley/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/bryceegley/Desktop/temp/pycoal/pycoal/tests/mineral_test.py", line 69, in test_classify_image
    assert expected.metadata.get(u'class names') == actual.metadata.get(u'class names')
AssertionError: 


FAIL: mineral_test.test_classify_image_in_memory
Traceback (most recent call last):
  File "/Users/bryceegley/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/bryceegley/Desktop/temp/pycoal/pycoal/tests/mineral_test.py", line 94, in test_classify_image_in_memory
    assert numpy.array_equal(expected.asarray(), actual.asarray())
AssertionError:

bdegley4789 · 2018-05-02T15:04:43Z

I also finished running the mineral classification using the Spectral Version 7 envi .hdr and .sli on the default example and staged it here
https://drive.google.com/drive/folders/1LTJv5H6_YxMDz5fff074MUj59QO9mwvH

Based on the three mineral classifications I’ve done with Spectral v7, everything seems to perform normally using the Spectral Version 7 envi .hdr and .sli files

lewismc · 2018-05-02T15:21:55Z

Bryce, look at the stack trace...

ValueError: 'Schwertmannite BZ93-1 s06av95a=b' is not in list

lewismc · 2018-05-02T15:23:47Z

You’ve also neglected to address any of my comments... come on.

bdegley4789 · 2018-05-02T20:03:23Z

I'm working on implementing the comments you made, I was just giving an update on where I was with that.

I changed mining.py back to the names of Spectral Version 6 Library so I thought 'Schwertmannite BZ93-1 s06av95a=b' would be in the list.
I did that to keep support for Spectral v6 library like we discussed.

I was thinking about doing multiple examples one for Spectral v7 and one for Spectral v6. But mining.py requires the spectra names so I think the only option
of doing this is through the other method you mentioned using the CLI

lewismc · 2018-05-02T20:36:44Z

I was thinking about doing multiple examples one for Spectral v7 and one for Spectral v6. But mining.py requires the spectra names so I think the only option
of doing this is through the other method you mentioned using the CLI

Why don't you just do the following

# classes identified as proxies for coal mining classification using USGSv6
proxy_class_names_usgsv6 = [u'Schwertmannite BZ93-1 s06av95a=b',
                   u'Renyolds_TnlSldgWet SM93-15w s06av95a=a',
u'Renyolds_Tnl_Sludge SM93-15 s06av95a=a']

# classes identified as proxies for coal mining classification using USGSv7
proxy_class_names_usgsv7 = [u'Schwertmannite BZ93-1 BECKb AREF',
                   u'Renyolds_TnlSldgWet SM93-15w BECKa AREF',
u'Renyolds_Tnl_Sludge SM93-15 BECKa AREF']

Then use the proxy_class_names_usgsv6 and proxy_class_names_usgsv7 depending on whether USGSv6 or USGSv7 classification is required?

lewismc · 2018-05-02T20:37:54Z

Take a look at the comments... you've not addressed them. If this is not done by Thursday morning I will just do it myself.

bdegley4789 · 2018-05-03T10:41:29Z

@lewismc Ok, I’m just using two different examples scripts for mineral classification to run USGS Spectral v7 and v6 libraries. Mining.py now has a new argument in the CLI which can be passed as a number either 6 or 7. Or you can just change the number in example_mining.py

I tested running this with 6 and 7 and it works. But it must be the same version number as you did in the mineral classification to get the classified file. If the classified image was created with USGS Spectral Version 6, you can’t do a mining classification with version 7 since the spectra names are different.

Bryces-MacBook-Air:examples bryceegley$ python example_mining.py
Saving /Users/bryceegley/Desktop/pycoal/examples/ang20150420t182050_corr_v1e_img_class_mining.img
Bryces-MacBook-Air:examples bryceegley$

I fixed errors in nose tests, now everything passes and I made all the requested changes you had

Bryces-MacBook-Air:pycoal bryceegley$ nosetests
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
............
----------------------------------------------------------------------
Ran 12 tests in 9.768s

OK
Bryces-MacBook-Air:pycoal bryceegley$

bdegley4789 · 2018-05-03T10:47:14Z

I don't know why travis-ci keeps saying FileNotFoundError: [Errno 2] No such file or directory: 'pycoal/tests'

lewismc · 2018-05-03T15:45:07Z

@bdegley4789 what are all of the files

ASCIIdata/ASCIIdata_splib07a/splib07a_Bandpass_(FWHM)_ASDNG_High-Res_NextGen.txt

If you re not using them remove them.

lewismc · 2018-05-03T15:53:21Z

I don't know why travis-ci keeps saying FileNotFoundError: [Errno 2] No such file or directory: 'pycoal/tests'

Look at the error traces in the TravisCI logs

======================================================================
ERROR: test suite for <module 'environment_test' from '/home/travis/build/capstone-coal/pycoal/pycoal/tests/environment_test.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/nose/suite.py", line 210, in run
    self.setUp()
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/nose/suite.py", line 293, in setUp
    self.setupContext(ancestor)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/nose/suite.py", line 316, in setupContext
    try_run(context, names)
  File "/home/travis/virtualenv/python3.5.5/lib/python3.5/site-packages/nose/util.py", line 469, in try_run
    return func(obj)
  File "/home/travis/build/capstone-coal/pycoal/pycoal/tests/test.py", line 47, in setup_module
    ftp.retrbinary('RETR %s' % f, lib_f.write)
  File "/opt/python/3.5.5/lib/python3.5/ftplib.py", line 443, in retrbinary
    with self.transfercmd(cmd, rest) as conn:
  File "/opt/python/3.5.5/lib/python3.5/ftplib.py", line 400, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/opt/python/3.5.5/lib/python3.5/ftplib.py", line 362, in ntransfercmd
    source_address=self.source_address)
  File "/opt/python/3.5.5/lib/python3.5/socket.py", line 712, in create_connection
    raise err
  File "/opt/python/3.5.5/lib/python3.5/socket.py", line 703, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

There is a timeout when executing the FTP file retrieval in File "/home/travis/build/capstone-coal/pycoal/pycoal/tests/test.py", line 47, in setup_module

lewismc · 2018-05-03T15:53:51Z

How large are these file? If they are not large then we should just cache them locally within the tests directory as a resource.

bdegley4789 · 2018-05-03T17:09:15Z

Those files in pycoal/pycoal/tests/usgs_splib07/ASCIIdata/ are for testing. I did the same thing as the ASTER tests. Take a small subset of the files in the spectral library and use them to run my conversion function on to verify that it generates a .sli and .hdr file. I could probably remove some of them but to test my conversion functions we should have some files in there. It is 221 KB for the files I included for testing. The ASTER test files are about half that 106KB so I could remove some if that is too much space.

lewismc · 2018-05-03T17:12:49Z

Can you please squash your commits. This PR is very messy.

Those files in pycoal/pycoal/tests/usgs_splib07/ASCIIdata/ are for testing.

Unless they are being used in unit tests (which per your PR they are not) then they should be removed from the PR.

remove s06_envi files from gitignore

bdegley4789 · 2018-05-08T09:41:46Z

@lewismc I fixed the errors with the ftp statements we discussed, I pushed the convolved envi .hdr and .sli files and I fixed another error I found when running nosetests. It looks like all the checks now pass. I think this pull request is good to be accepted and fulfills the work needed on issue #105 . In the next two days I'll add documentation for this issue to the website. Let me know what you think

lewismc · 2018-05-08T19:49:54Z

Excellent work on this issue @bdegley4789 it's been a long time coming and a lot of back and forth these last few months... but it's now ready to commit to master branch. Excellent work, thank you for the persistence.

bdegley4789 and others added 26 commits March 8, 2018 08:53

splib07

d9f451c

Merge pull request #1 from capstone-coal/master

a9bb2d4

update branch

spectral 7 converter start

840ff4b

Splib 07 test one file

b5505a6

Finished Spectral Version 7 convert

7b10aa9

Finish client changes to spectral convert function

0052322

Condensed to one script

0db1578

Fix oswalk and other changes

ddb47a3

Remove duplicate spectra

3f662ad

remove aster files

b8d6826

Remove duplicate spectra

a715062

Merge branch 'master' of https://github.com/bdegley4789/pycoal

6ef85ba

Change to unknown

2fede26

Fix spelling

48a7063

Move convert function to mineral.py

efd214c

Fix band num

ff7154b

Merge pull request #2 from capstone-coal/master

c56ec34

Fix broken location of GDAL CURRENT to fixed 2.2.4 in Dockerfile

Fix reload error travis ci

06e3df4

update reload to py3

8e8b46a

remove unused import

73fb86c

fix mineral import

27474dc

remove sys path import

ae4f4e1

removed bad import statements

c7e4865

Merge branch 'master' of https://github.com/bdegley4789/pycoal

72b8b1a

Change spectra names to v7

c6d4feb

Worked on testing

2e83470

lewismc requested changes Apr 26, 2018

View reviewed changes

Allow cli to work with spectral v7 and v6

4d9db97

Fix ftp url

1e0ad1b

bdegley4789 added 4 commits May 8, 2018 02:14

Remove ftp calls, fix overwrite error in nosetests

c7988a1

Remove Spectral 7 library

11592a3

Merge pull request #3 from capstone-coal/master

c22132d

remove s06_envi files from gitignore

Repush with gitignore updated

7aa86d6

lewismc approved these changes May 8, 2018

View reviewed changes

lewismc merged commit 2c59bac into capstone-coal:master May 8, 2018

lewismc mentioned this pull request May 8, 2018

Upgrade to use USGS Spectral Library Version 7 #105

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Created tests for USGS Spectral Library Version 7 convert function #146

Created tests for USGS Spectral Library Version 7 convert function #146

bdegley4789 commented Apr 26, 2018

lewismc left a comment

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc Apr 26, 2018

lewismc commented Apr 26, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 2, 2018

bdegley4789 commented May 2, 2018 •

edited

Loading

lewismc commented May 2, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 2, 2018

lewismc commented May 2, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 3, 2018

bdegley4789 commented May 3, 2018

lewismc commented May 3, 2018

lewismc commented May 3, 2018

lewismc commented May 3, 2018

bdegley4789 commented May 3, 2018

lewismc commented May 3, 2018

bdegley4789 commented May 8, 2018 •

edited

Loading

lewismc commented May 8, 2018


		@author: COAL Developers

		@copyright: 2018 COAL Developers. All rights reserved.

Created tests for USGS Spectral Library Version 7 convert function #146

Created tests for USGS Spectral Library Version 7 convert function #146

Conversation

bdegley4789 commented Apr 26, 2018

lewismc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewismc commented Apr 26, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 2, 2018

bdegley4789 commented May 2, 2018 • edited Loading

lewismc commented May 2, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 2, 2018

lewismc commented May 2, 2018

lewismc commented May 2, 2018

bdegley4789 commented May 3, 2018

bdegley4789 commented May 3, 2018

lewismc commented May 3, 2018

lewismc commented May 3, 2018

lewismc commented May 3, 2018

bdegley4789 commented May 3, 2018

lewismc commented May 3, 2018

bdegley4789 commented May 8, 2018 • edited Loading

lewismc commented May 8, 2018

bdegley4789 commented May 2, 2018 •

edited

Loading

bdegley4789 commented May 8, 2018 •

edited

Loading