Gotta do something with those stale downloads...

[jotegui]

https://console.cloud.google.com/appengine/taskqueues/compose?project=vertnet-portal

There are 18 downloads "in progress", all of them from many days ago (up to 197). We should checkout the code to see what is causing them and fix it. It is taking a lot of backend instance time (and that means $$)

[tucotuco]

Below is a list of the queries that were hung up so that we can try to find
a pattern or try to reproduce them. I removed the tasked from the queue.

q=M%C3%A9xico+class%3AActinopterygii+country%3A%22M%C3%A9xico%22&requesttime=2016-05-06T16%3A31%3A31.001020&name=Peces.M%C3%A9xico&fileindex=0&filepattern=Peces.M%C3%A9xico-356f3b60858b41bbaef04418e2eb3b40&reccount=1000&email=gicruz%40ecosur.edu.mx

q=preparations%3A+wet+type%3Aspecimen+class%3AMammalia&requesttime=2016-02-24T22%3A14%3A07.161640&name=Thayn%C3%A1_Brito&fileindex=0&filepattern=Thayn%C3%A1_Brito-83ed7d47fb294dd68662e780e1f70706&reccount=1000&email=thay_na96%40hotmail.com

q=Brehm+type%3Aspecimen&requesttime=2015-11-05T08%3A07%3A59.809910&name=Brehm_%C3%9Cbersee&fileindex=0&filepattern=Brehm_%C3%9Cbersee-971a80f30ea2417caea1635d2d69ec35&reccount=1000&email=t.toepfer%40zfmk.de

q=specificepithet%3Ahudsonius+genus%3Azapus+locality%3AYakutat+Bay+stateprovince%3A%22Alaska%22+institutioncode%3AUSNM&requesttime=2016-02-13T23%3A06%3A59.391770&name=Zahu_al_type&fileindex=0&filepattern=Zahu_al_type-cc543119a54d42e186978ceae51266ff&reccount=1&email=

q=Selasphorus+type%3Aspecimen+mappable%3A1&requesttime=2015-12-08T19%3A50%3A30.052400&name=MyResults&fileindex=0&filepattern=MyResults-65b1e51ec7cf4f4991c0272dd735a932&reccount=1000&email=cjbattey%40uw.edu

q=Cardellina+pusilla&requesttime=2015-12-08T17%3A11%3A43.545530&name=MyResults&fileindex=0&filepattern=MyResults-55de92961bf241239cf8a2db739c3999&reccount=1000&email=ryleec999%40gmail.com

q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A25.829150&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-f15e6813b8f14fafb802546e4489f782&reccount=31821&email=mmims%40usgs.gov

q=specificepithet%3Aprinceps+genus%3Aochotona&requesttime=2015-12-08T15%3A46%3A45.684890&name=test-bigsearch1&fileindex=0&filepattern=test-bigsearch1-e0d031980f6849d68c5e378c549370dd&reccount=1000&email=myrmecocystus%40gmail.com

q=specificepithet%3Acollaris+genus%3Aochotona+year%3A1960&requesttime=2015-12-08T15%3A46%3A45.886060&name=test-bigsearch2&fileindex=0&filepattern=test-bigsearch2-2073dee868a247f592dbb8bca913f48f&reccount=1&email=myrmecocystus%40gmail.com

q=Ecuador+amphibia&requesttime=2016-02-04T19%3A38%3A52.527740&name=Mario_H._Yanez-Mu%C3%B1oz&fileindex=0&filepattern=Mario_H._Yanez-Mu%C3%B1oz-e91ea0f24b63462bba4568d0e96d733f&reccount=1000&email=mayamu%40hotmail.com

q=Ecuador+reptilia&requesttime=2016-02-04T19%3A39%3A41.078270&name=Mario_H._Yanez-Mu%C3%B1oz&fileindex=0&filepattern=Mario_H._Yanez-Mu%C3%B1oz-096bd7c775d14ddc87a98869796e812d&reccount=1000&email=mayamu%40hotmail.com

q=family%3APhrynosomatidae+stateprovince%3A%22Nuevo+Le%C3%B3n%22+country%3A%22M%C3%A9xico%22&requesttime=2016-01-26T02%3A28%3A49.131840&name=Phrynosomatidae_Nuevo_Le%C3%B3n&fileindex=0&filepattern=Phrynosomatidae_Nuevo_Le%C3%B3n-47f10562f3fb4b49a1735293c49959b9&reccount=1000&email=manuelocampo_13%40hotmail.com

q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A25.294150&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-3cddf114a4f94ab48b0d4f1bee18abe5&reccount=31821&email=mmims%40usgs.gov

q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A24.878130&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-e7576d2bb2fb45b4ad0fdef5f70dfa2f&reccount=31821&email=mmims%40usgs.gov

q=class%3AAmphibia+stateprovince%3AOregon&requesttime=2016-01-21T22%3A38%3A24.878130&name=ORAmphibians&fileindex=31&filepattern=ORAmphibians-e7576d2bb2fb45b4ad0fdef5f70dfa2f&reccount=31821&email=mmims%40usgs.gov

q=Troglodytes&requesttime=2015-12-08T19%3A52%3A48.103150&name=Troglodytes&fileindex=41&filepattern=Troglodytes-788c6905ca7e47c7bf3d8943efdee04d&reccount=41572&email=barke042%40umn.edu

q=distance%28location%2Cgeopoint%28-28.071980301779845%2C-61.5234375%29%29%3C400129&requesttime=2016-03-30T14%3A09%3A55.222070&name=MyResults&fileindex=6&filepattern=MyResults-8bf52987989443cb8128aac8bad46957&reccount=6646&email=agustinomen%40hotmail.com

q=mammals+and+nayarit+%28mammals%29&requesttime=2016-04-25T18%3A27%3A31.636690&name=Mam%C3%ADferos_de_Nayarit_VERTNET&fileindex=0&filepattern=Mam%C3%ADferos_de_Nayarit_VERTNET-38a517599e3141dabed2ad5d6dee0da9&reccount=1000&email=zacatuchemx%40hotmail.com

On Fri, May 20, 2016 at 7:49 AM, Javier Otegui [email protected]
wrote:

https://console.cloud.google.com/appengine/taskqueues/compose?project=vertnet-portal

There are 18 downloads "in progress", all of them from many days ago (up
to 197). We should checkout the code to see what is causing them and fix
it. It is taking a lot of backend instance time (and that means $$)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#597

[tucotuco]

This remains a recurring theme. The only solution so far is a brute force one. Open the downloadwrite task queue and delete everything in it. This will affect downloads in progress. After doing so, the files in the vn-dltest bucket in Google Cloud storage have to be removed. Use the GCS_cleaner.py script to do this. https://github.com/VertNet/post-harvest-processor/tree/master/lib