Skip to content

[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation #332

Open

Description

@achad4

The current underlying implementation of BERT score supports a limited set of transformer models, and FMEval further truncates this list to microsoft/deberta-xlarge-mnli and roberta-large-mnli.

Torchmetrics provides a more generic BERT score implementation. The specific request here is to not limit what transformer models users can configure. The behavior should be as follows:

  • If microsoft/deberta-xlarge-mnli and roberta-large-mnli is specified, use the bert-score implementation to avoid regressions for existing customers.
  • Otherwise, use the torchmetrics implementation of BERT score.

Use cases for broader set of models underlying BERT score:

  1. Monolingual BERT models have been shown to outperform multi-lingual BERT models on certain tasks. https://aclanthology.org/2021.acl-long.243.pdf
  2. Customers may fine tune their own transformers which can be downloaded into the container running AWSFMeval and passed into the torchmetrics BERT score implementation.

Activity

  1. Riccardo231 commented on Dec 28, 2024

    @Riccardo231

    Hello, I'd be glad to work on this. Can i get the issue assigned? Thanks

  2. Zhenshan-Jin commented on Jan 8, 2025

    @Zhenshan-Jin

    @ricor07 Thank you for helping out. Let us know when the PR is ready and we are happy to review. ^

    see https://github.com/aws/fmeval/blob/main/CONTRIBUTING.md#contributing-via-pull-requests for detailed PR guidance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions