Daily Papers API

[hlky]

Fixes #2553

This PR introduces list_papers using the Daily Papers API, search_papers using the papers/search endpoint and get_paper using papers/{paper_id} endpoint.

We add DailyPaper dataclass, containing Paper and associated metadata.

Paper dataclass containing metadata about the paper itself.

PaperAuthor dataclass containing metadata about the paper's author.

PaperAuthor's user and DailyPaper's submitted_by use existing User dataclass, although these contain fewer fields than User itself so could have their own dataclasses.

We add list_papers to HfApi which accepts date as str, YYYY-MM-DD is the expected format, this could also accept datetime as a parameter. The endpoint itself also accepts a full datetime in format %Y-%m-%dT%H:%M:%S.%fZ. Invalid dates will return HTTP 400.

We add PaperSearchInfo dataclass, containing minimal metadata, returned by search_papers.

We add search_papers to HfApi which accepts query as str, this can be a text query or arXiv paper ID.

We add get_paper to HfApi which accepts either paper_id as str or a PaperSearchInfo object with paper_search. Due to slight differences between the data returned from papers/{paper_id} and Daily Papers endpoint we add a static method from_get_paper to DailyPaper. Some fields are unavailable from papers/{paper_id}, namely thumbnail and numComments, when providing a PaperSearchInfo we copy thumbnail into the DailyPaper object.

We add tests test_papers_by_date, test_search_papers, test_get_paper_by_id, test_get_paper_by_paper_search_info under DailyPaperApiTest.

[hlky]

Here's an example DailyPaper:

DailyPaper(
    paper=Paper(
        paper_id="2409.11340",
        authors=[
            PaperAuthor(
                author_id="66ea3b25353c1b9b84254825",
                user=User(
                    username="Shitao",
                    fullname="Xiao",
                    avatar_url="/avatars/c0675d05a52192ee14e9ab1633353956.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Shitao Xiao",
                status="claimed_verified",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 7, 1, 29, 215000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b84254826",
                user=None,
                name="Yueze Wang",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b84254827",
                user=User(
                    username="JUNJIE99",
                    fullname="JUNJIE ZHOU",
                    avatar_url="/avatars/42f09356a1282896573ccb44830cd327.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Junjie Zhou",
                status="claimed_verified",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 7, 1, 31, 41000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b84254828",
                user=User(
                    username="avery00",
                    fullname="huaying Yuan",
                    avatar_url="/avatars/2537cee66afecc2d999e05b01c78d319.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Huaying Yuan",
                status="admin_assigned",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 7, 12, 24, 40000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b84254829",
                user=None,
                name="Xingrun Xing",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b8425482a",
                user=User(
                    username="Ruiran",
                    fullname="Ruiran Yan",
                    avatar_url="/avatars/26aef5944759c2e4366a71eb8c7fc50a.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Ruiran Yan",
                status="admin_assigned",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 7, 12, 36, 909000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b8425482b",
                user=User(
                    username="stingw",
                    fullname="Shu-Ting Wang",
                    avatar_url="/avatars/3486af06cc2c1562e09b04bb03360912.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Shuting Wang",
                status="admin_assigned",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 7, 12, 43, 24000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b8425482c",
                user=None,
                name="Tiejun Huang",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ea3b25353c1b9b8425482d",
                user=User(
                    username="zl101",
                    fullname="zhengliu",
                    avatar_url="/avatars/ef13dc7ce243819bc0da9b04e778b432.svg",
                    details=None,
                    is_following=None,
                    is_pro=False,
                    num_models=None,
                    num_datasets=None,
                    num_spaces=None,
                    num_discussions=None,
                    num_papers=None,
                    num_upvotes=None,
                    num_likes=None,
                    num_following=None,
                    num_followers=None,
                    orgs=[],
                ),
                name="Zheng Liu",
                status="extracted_pending",
                status_changed_at=datetime.datetime(
                    2024, 9, 18, 2, 30, 1, 852000, tzinfo=datetime.timezone.utc
                ),
                hidden=False,
            ),
        ],
        published_at=datetime.datetime(
            2024, 9, 17, 16, 42, 46, tzinfo=datetime.timezone.utc
        ),
        title="OmniGen: Unified Image Generation",
        summary="In this work, we introduce OmniGen, a new diffusion model for unified image\ngeneration. Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen\nno longer requires additional modules such as ControlNet or IP-Adapter to\nprocess diverse control conditions. OmniGenis characterized by the following\nfeatures: 1) Unification: OmniGen not only demonstrates text-to-image\ngeneration capabilities but also inherently supports other downstream tasks,\nsuch as image editing, subject-driven generation, and visual-conditional\ngeneration. Additionally, OmniGen can handle classical computer vision tasks by\ntransforming them into image generation tasks, such as edge detection and human\npose recognition. 2) Simplicity: The architecture of OmniGen is highly\nsimplified, eliminating the need for additional text encoders. Moreover, it is\nmore user-friendly compared to existing diffusion models, enabling complex\ntasks to be accomplished through instructions without the need for extra\npreprocessing steps (e.g., human pose estimation), thereby significantly\nsimplifying the workflow of image generation. 3) Knowledge Transfer: Through\nlearning in a unified format, OmniGen effectively transfers knowledge across\ndifferent tasks, manages unseen tasks and domains, and exhibits novel\ncapabilities. We also explore the model's reasoning capabilities and potential\napplications of chain-of-thought mechanism. This work represents the first\nattempt at a general-purpose image generation model, and there remain several\nunresolved issues. We will open-source the related resources at\nhttps://github.com/VectorSpaceLab/OmniGen to foster advancements in this field.",
        upvotes=38,
        discussion_id="66ea3b29353c1b9b842549ac",
    ),
    published_at=datetime.datetime(
        2024, 9, 18, 1, 0, 6, 728000, tzinfo=datetime.timezone.utc
    ),
    title="OmniGen: Unified Image Generation",
    thumbnail="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2409.11340.png",
    comments=3,
    submitted_by=User(
        username="",
        fullname="AK",
        avatar_url="https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg",
        details=None,
        is_following=None,
        is_pro=False,
        num_models=None,
        num_datasets=None,
        num_spaces=None,
        num_discussions=None,
        num_papers=None,
        num_upvotes=None,
        num_likes=None,
        num_following=None,
        num_followers=None,
        orgs=[],
    ),
)

Although not returned by the API we could add a link to arXiv page and PDF link, https://arxiv.org/abs/{paper_id} and https://arxiv.org/pdf/{paper_id} respectively.

Also the API doesn't appear to allow retrieval of the paper's discussion.

[hlky]

Example PaperSearchInfos from search_paper:

[
    PaperSearchInfo(
        paper_id="2409.07146",
        title="Gated Slot Attention for Efficient Linear-Time Sequence Modeling",
        thumbnail="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2409.07146.png",
        source="hf",
    ),
    PaperSearchInfo(
        paper_id="2409.03752",
        title="Attention Heads of Large Language Models: A Survey",
        thumbnail="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2409.03752.png",
        source="hf",
    ),
    ...
]

Example DailyPaper from get_paper with paper_id:

DailyPaper(
    paper=Paper(
        paper_id="2409.11074",
        authors=[
            PaperAuthor(
                author_id="66ead57361228b02f8144cdf",
                user=None,
                name="Adrian Cosma",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ead57361228b02f8144ce0",
                user=None,
                name="Ana-Maria Bucur",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ead57361228b02f8144ce1",
                user=None,
                name="Emilian Radoi",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
        ],
        published_at=datetime.datetime(
            2024, 9, 17, 11, 3, 46, tzinfo=datetime.timezone.utc
        ),
        title="RoMath: A Mathematical Reasoning Benchmark in Romanian",
        summary="Mathematics has long been conveyed through natural language, primarily for\nhuman understanding. With the rise of mechanized mathematics and proof\nassistants, there is a growing need to understand informal mathematical text,\nyet most existing benchmarks focus solely on English, overlooking other\nlanguages. This paper introduces RoMath, a Romanian mathematical reasoning\nbenchmark suite comprising three datasets: RoMath-Baccalaureate,\nRoMath-Competitions and RoMath-Synthetic, which cover a range of mathematical\ndomains and difficulty levels, aiming to improve non-English language models\nand promote multilingual AI development. By focusing on Romanian, a\nlow-resource language with unique linguistic features, RoMath addresses the\nlimitations of Anglo-centric models and emphasizes the need for dedicated\nresources beyond simple automatic translation. We benchmark several open-weight\nlanguage models, highlighting the importance of creating resources for\nunderrepresented languages. We make the code and dataset available.",
        upvotes=1,
        discussion_id="66ead57461228b02f8144d31",
    ),
    published_at=datetime.datetime(
        2024, 9, 19, 17, 17, 31, 279000, tzinfo=datetime.timezone.utc
    ),
    title="RoMath: A Mathematical Reasoning Benchmark in Romanian",
    thumbnail="",
    comments=0,
    submitted_by=User(
        username="IAMJB",
        fullname="JB D.",
        avatar_url="/avatars/1208629f14f010dbc2cd94f3c30f9baf.svg",
        details=None,
        is_following=None,
        is_pro=False,
        num_models=None,
        num_datasets=None,
        num_spaces=None,
        num_discussions=None,
        num_papers=None,
        num_upvotes=None,
        num_likes=None,
        num_following=None,
        num_followers=None,
        orgs=[],
    ),
)

Example DailyPaper from get_paper with paper_search:

DailyPaper(
    paper=Paper(
        paper_id="2409.11074",
        authors=[
            PaperAuthor(
                author_id="66ead57361228b02f8144cdf",
                user=None,
                name="Adrian Cosma",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ead57361228b02f8144ce0",
                user=None,
                name="Ana-Maria Bucur",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
            PaperAuthor(
                author_id="66ead57361228b02f8144ce1",
                user=None,
                name="Emilian Radoi",
                status="",
                status_changed_at=None,
                hidden=False,
            ),
        ],
        published_at=datetime.datetime(
            2024, 9, 17, 11, 3, 46, tzinfo=datetime.timezone.utc
        ),
        title="RoMath: A Mathematical Reasoning Benchmark in Romanian",
        summary="Mathematics has long been conveyed through natural language, primarily for\nhuman understanding. With the rise of mechanized mathematics and proof\nassistants, there is a growing need to understand informal mathematical text,\nyet most existing benchmarks focus solely on English, overlooking other\nlanguages. This paper introduces RoMath, a Romanian mathematical reasoning\nbenchmark suite comprising three datasets: RoMath-Baccalaureate,\nRoMath-Competitions and RoMath-Synthetic, which cover a range of mathematical\ndomains and difficulty levels, aiming to improve non-English language models\nand promote multilingual AI development. By focusing on Romanian, a\nlow-resource language with unique linguistic features, RoMath addresses the\nlimitations of Anglo-centric models and emphasizes the need for dedicated\nresources beyond simple automatic translation. We benchmark several open-weight\nlanguage models, highlighting the importance of creating resources for\nunderrepresented languages. We make the code and dataset available.",
        upvotes=1,
        discussion_id="66ead57461228b02f8144d31",
    ),
    published_at=datetime.datetime(
        2024, 9, 19, 17, 17, 31, 279000, tzinfo=datetime.timezone.utc
    ),
    title="RoMath: A Mathematical Reasoning Benchmark in Romanian",
    thumbnail="https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2409.11074.png",
    comments=0,
    submitted_by=User(
        username="IAMJB",
        fullname="JB D.",
        avatar_url="/avatars/1208629f14f010dbc2cd94f3c30f9baf.svg",
        details=None,
        is_following=None,
        is_pro=False,
        num_models=None,
        num_datasets=None,
        num_spaces=None,
        num_discussions=None,
        num_papers=None,
        num_upvotes=None,
        num_likes=None,
        num_following=None,
        num_followers=None,
        orgs=[],
    ),
)