Named Entity Detection (Human Names etc.)

[TheTechromancer]

Extract companies, humans, and locations from HTTP_RESPONSE.

Would require:

• Extracting visible text from HTML
• A helper for detecting human names, etc.

[TheTechromancer]

This is possible with Named Entity Recognition:

# pip install spacy
# python -m spacy download en_core_web_sm

import spacy

# Load the pre-trained model
nlp = spacy.load("en_core_web_sm")

# Your text
text = "Alice and Bob are meeting at Central Park in New York with Dr. Smith."

# Process the text
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text} : {ent.label_}")

    # Alice : PERSON
    # Bob : PERSON
    # Central Park : LOC
    # New York : GPE
    # Smith : PERSON

However, this produces a large number of false positives.

[TheTechromancer]

With @domwhewell-sage's unstructured module, this has become more feasible since we can use it to extract visible text from HTML. This will cut down significantly on the number of false positives.

from unstructured.partition.html import partition_html

url = "https://www.cnn.com/2023/01/30/sport/empire-state-building-green-philadelphia-eagles-spt-intl/index.html"
elements = partition_html(url=url)
print("\n\n".join([str(el) for el in elements]))