Skip to main content

Dataset Creation Helpers

Helper functions for programmatically creating and loading common dataset types.

create_qa_dataset​

def create_qa_dataset(
    questions: List[str],
    answers: List[str],
    name: str = "qa_dataset",
    **kwargs
) -> Dataset:
    ...

Create question-answer dataset.

from benchwise import create_qa_dataset

dataset = create_qa_dataset(
    questions=["What is AI?", "What is ML?"],
    answers=["Artificial Intelligence", "Machine Learning"],
    name="ai_qa"
)

create_summarization_dataset​

def create_summarization_dataset(
    documents: List[str],
    summaries: List[str],
    name: str = "summarization_dataset",
    **kwargs
) -> Dataset:
    ...

Create summarization dataset.

from benchwise import create_summarization_dataset

dataset = create_summarization_dataset(
    documents=["Long article..."],
    summaries=["Summary..."],
    name="news_summ"
)

create_classification_dataset​

def create_classification_dataset(
    texts: List[str],
    labels: List[str],
    name: str = "classification_dataset",
    **kwargs
) -> Dataset:
    ...

Create classification dataset.

from benchwise import create_classification_dataset

dataset = create_classification_dataset(
    texts=["Great product!", "Terrible experience"],
    labels=["positive", "negative"],
    name="sentiment"
)

Loading Pre-built Datasets​

Functions to load pre-built benchmark datasets for common tasks.

load_mmlu_sample​

def load_mmlu_sample() -> Dataset:
    ...
Loads a sample of the Massive Multitask Language Understanding (MMLU) dataset.

load_hellaswag_sample​

def load_hellaswag_sample() -> Dataset:
    ...
Loads a sample of the HellaSwag dataset, a common sense reasoning benchmark.

load_gsm8k_sample​

def load_gsm8k_sample() -> Dataset:
    ...
Loads a sample of the GSM8K dataset, a grade school math word problems benchmark.

See Also​