You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation #332
Torchmetrics provides a more generic BERT score implementation. The specific request here is to not limit what transformer models users can configure. The behavior should be as follows:
If microsoft/deberta-xlarge-mnli and roberta-large-mnli is specified, use the bert-score implementation to avoid regressions for existing customers.
Otherwise, use the torchmetrics implementation of BERT score.
Use cases for broader set of models underlying BERT score:
Customers may fine tune their own transformers which can be downloaded into the container running AWSFMeval and passed into the torchmetrics BERT score implementation.
Activity
Riccardo231 commentedon Dec 28, 2024
Hello, I'd be glad to work on this. Can i get the issue assigned? Thanks
Zhenshan-Jin commentedon Jan 8, 2025
@ricor07 Thank you for helping out. Let us know when the PR is ready and we are happy to review. ^
see https://github.com/aws/fmeval/blob/main/CONTRIBUTING.md#contributing-via-pull-requests for detailed PR guidance