Additional data for IssueList and SeriesList

[mizaki]

Hi,

I’ve been working on allowing the use of other comic DB information sources for Comic Tagger. I have a working version of one for Metron but I have a slight issue with the lack of data for certain API calls. This makes for a rather bare GUI.

I was wondering if there is any chance of returning some additional fields for some of the API calls? The two options I have as of now are to leave a lot of information blank or, make multiple calls. Obviously, the multiples call within the 30 a minute limit would mean multiple minutes delays for a large result. Not a great user experience.

Here are the calls and the extra information I would like for the GUI:

Series search:
https://metron.cloud/api/series/?name={title} (I presume this is the correct way to search for a series name?)

Current fields for SeriesList:

id
series
modified

Additional fields requested:

year_began
publisher
desc
issue_count
image*

*There is no image for series. I notice the website seems to use the image of the first issue. It would be nice to include the first issue image url with the series search.

Issue list for series:

https://metron.cloud/api/issue/?series_id={id}

Current fields for IssueList:

id
issue
cover_date
modified

Additional fields requested:

desc
image
number [added in an edit because I missed it]
name*

*issue returns the series name and number which is usable but having the “name” field from the issue would be nicer.

How it looks currently:

Thanks.

[bpepple]

Hi,

I'm not opposed to adding more information, but I do have a couple of concerns:

• Potential API breaking changes: The API has been stable the last 4 years, and I'm hesitant to make any breaking changes since there a lot of tools and users that use the current API, and would need plenty of time to migrate/inform existing users.
• Implementation: If there are breaking API changes, it would make sense to add versioning to the API for future changes, which would require a decent amount of developer time.
• Community: I'd want to get some input from current users/developers, before making any decisions.
• Lack of time: Currently, the amount of free time I've got for a new project is pretty limited, so someone from the community would probably need to step up and implement this.
• ComicTagger Question: Does ComicTagger still do image hashing for issue matching? If so, I'd have some questions about the average data consumption per user. Depending of what that is, that could impact our server costs since our CDN's charge for usage over 1 TiB a month.

Regarding your field suggestions, most of them make sense, but I don't think adding the desc field is a good idea due to it's potential size (it's a Django TextField without a max length).

Anyway, I'll point this discussion to some of the folks in our Matrix rooms later today, and get their thoughts..

[bpepple]

Finally got around to opening a feature request for cover hashing. #170

[bpepple]

Did some preliminary testing with ImageHash, and using Perceptual hashing seems to be the best choice based on a greater tolerance of minor modifications. What are other people's thoughts on this?

[lordwelch]

I've seen tiny color differences change the hash when using both the Difference Hash and the Perceptual hash. The Average Hash works on much bigger differences but will produce the same hash

[lordwelch]

In particularly I've been trying to re-create the hashes from the Python ImageHash library in Go which ends up being difficult because webp as an example stores color as YCbCr and PIL converts it to RGB which doesn't use the same calculation as Go which means that some of the colors have an imperceptible shift but throws off the Difference Hash and the Perceptual Hash by a few points

[bpepple]

Hmm, I did some testing on around 50 issues (using Metron Images & Comic Archive covers) using Average & Perceptual Hashes, and Perceptual hashing had much better matching results.

That being said, it might be worth using a larger dataset (maybe 1,000 or so) to see if that is accurate conclusion. If I can free up some cycles next week, I'll write a better testing suite to check this and publish the results.

[bpepple]

Additional fields requested:

desc image number [added in an edit because I missed it] name*

*issue returns the series name and number which is usable but having the “name” field from the issue would be nicer.

@mizaki What information are you looking for with the name field (issue, series, series + vol or year)?

[mizaki]

It's a "nice to have" and in no way required for the identification list. As it was first designed around CV's data, I imagine it was what was available and to make it look less bare. As you can see, I've used what was available in the IssueList (issue) to fill it out.

The flow through the GUI is:

• Search for series
• Select series
• List issues for series (image above)
• Select issue (it will auto-select based on issue number)
• Okay to tag (full information for issue is retrieved)

[bpepple]

Ok, did some testing on the changes you're asking for. If I add everything you asked for to the IssueList endpoint it see's around a 279% increase in response size. If only series, number, and image fields are added it's see about 131% increase in response size. Right now I think it make sense to not add the Issue desc & name fields since they aren't necessary for the task of identifying an issue.

Since fields are just being added (for now, I' not removing the semi-redundant issue field) there shouldn't be any API breakage, so I'll most likely push these changes out later today, and publish a new version of Mokkari.

[bpepple]

I'm also going to add the year_began and issue_count fields to the SeriesList endpoint. I'm going to hold off on adding the image field for now, since I'll need to do some optimizing of the SQL query on that, since we'll be adding a JOIN on the Issue model.

[mizaki]

That's great, thanks. I have a couple of queries for Mokkari but I'll put them in its repo.

[mizaki]

I also realise it seems I didn't want name but title (Collection Title) that should make more sense. Still not a issue if it's missing but thought I'd clear it up.

[mizaki]

Sorry to be that guy... digging deeper into creating the talker, I didn't notice that Metron has variant cover images (reading is hard sometimes). Having the URL of the variant images in the IssueList would allow checking and displaying of those (obviously). In the GUI you can flip between them. During an auto-tag I can work around it with an additional call for the full issue (which should be quite limited as it's filter by issue number but if it becomes a problem it can always be removed). It's in no way a game stopper of course, another nice to have.

I also noticed that the series in IssueList doesn't have the series id. I don't need it but I thought it was a little odd. Maybe it is odd to need it? :)

[bpepple]

Sorry to be that guy... digging deeper into creating the talker, I didn't notice that Metron has variant cover images (reading is hard sometimes). Having the URL of the variant images in the IssueList would allow checking and displaying of those (obviously). In the GUI you can flip between them. During an auto-tag I can work around it with an additional call for the full issue (which should be quite limited as it's filter by issue number but if it becomes a problem it can always be removed). It's in no way a game stopper of course, another nice to have.

No worries. I don't think it make a lot sense to add it to the IssueList endpoint. I'm think a better workaround would be to expose a GET method for the Variant Covers table (which only exposes the POST/PUT/PATCH methods currently).

I also noticed that the series in IssueList doesn't have the series id. I don't need it but I thought it was a little odd. Maybe it is odd to need it? :)

Can't remember the exact reason it's not exposed off the top-of-my-head, but I haven't ran across any use-cases that have needed it.

[mizaki]

A theoretical on the series id not being in IssueList, I could see this possibility if you did something like: https://metron.cloud/api/issue/?number=510&series_name=Uncanny%20x-men It would take a search of series to get the id to then get the full series info. In that instance, it would remove an additional API call.

[mizaki]

If anyone wants to give this a try, an alpha version is here.

[bpepple]

If I can get some free time I'll give it a try.

[bpepple]

In particularly I've been trying to re-create the hashes from the Python ImageHash library in Go which ends up being difficult because webp as an example stores color as YCbCr and PIL converts it to RGB which doesn't use the same calculation as Go which means that some of the colors have an imperceptible shift but throws off the Difference Hash and the Perceptual Hash by a few points

I suspect the cover hash won't be terribly useful for projects that don't use Python, for exactly the reasons you cite above. Luckily, for tasks like tagging comics, Python is a good choice since a low-level language doesn't provide much benefit (if any).

[mizaki]

I would like to raise the issue title/name in IssueList again.

As mentioned before the IssueList uses display_name and creates that from the series title (year) #issue number. When the full issue information is retrieved there is title/collection_title and the array of name. As I understand it, title is solely used for TPB etc.?

From a user standpoint the issue list will contain only the display_name generated title which does no reflect what will be populated once the issue is tagged. CT does fetch the full information when an issue is clicked in the issue window which (will, issue outstanding) update that title. However, there is also a free text filter for the title which obviously is not much use unless the title or name is available for all issues. (And we're not going to do a full issue retrieve for each issue in a series.)

I expect your previous answer stands but I thought I'd bring it up again just in case :)

[bpepple]

Yeah, for now my opinion hasn't changed in regard to adding the Collection Title & Story Titles fields to the issue_list endpoint, since it doesn't really help much in identifying the series.