Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce components experimental #1339

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

shahules786
Copy link
Member

@shahules786 shahules786 commented Sep 21, 2024

Experimental feature: Component in metrics

# load desired NLI model as component

from ragas.experimental.metrics.component import TextClassificationNLIComponent

classifier = pipeline(
            "text-classification",
            model='vectara/hallucination_evaluation_model',
            tokenizer=AutoTokenizer.from_pretrained('google/flan-t5-base'),
            trust_remote_code=True
        )
classifier = classifier(input_pairs, top_k=1)
prompt = "<pad> Determine if the hypothesis is true given the premise?\n\nPremise: {text1}\n\nHypothesis: {text2}"
vectara_hhem = TextClassificationNLIComponent(hf_pipeline=classifier,prompt=prompt,batch_size=8,label="consistent")

# customise faithfulness NLI component
from ragas.dataset_schema import SingleTurnSample
from ragas.experimental.metrics._faithfulness import FaithfulnessExperimental

sample = SingleTurnSample(
    user_input = "Where is Eiffer Towerr",
    response="The Eiffel Tower is located in Paris.",
    retrieved_contexts=["The Eiffel Tower, an iconic symbol of Paris and one of the most recognizable structures in the world, was completed in 1889 as the entrance arch for the 1889 World's Fair. Designed by the engineer Gustave Eiffel, this wrought-iron lattice tower stands 324 meters tall and was initially met with skepticism but has since become a global cultural icon of France. It attracts millions of visitors each year, offering stunning panoramic views of the city from its observation decks."]
)

scorer = FaithfulnessExperimental()
scorer.llm = openai_model
scorer.nli_component = component
# scorer.nli_component.llm = openai_model
await scorer.single_turn_ascore(sample)

Also would suggest to move current version of experimental faithfulness before merging this.

@shahules786 shahules786 marked this pull request as ready for review September 21, 2024 10:48
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Sep 21, 2024
@shahules786 shahules786 changed the title feat: introduce components feat: introduce components experimental Sep 21, 2024
]
for i in range(0, len(prompt_input_list), self.batch_size):
prompt_input_list_batch = prompt_input_list[i : i + self.batch_size]
response = self.hf_pipeline(prompt_input_list_batch, **self.model_kwargs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might be a problem with async calls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants