Zobrazeno 1 - 10
of 48
pro vyhledávání: '"Opitz, Juri"'
Large Language Models (LLMs) have become capable of generating highly fluent text in certain languages, without modules specially designed to capture grammar or semantic coherence. What does this mean for the future of linguistic expertise in NLP? We
Externí odkaz:
http://arxiv.org/abs/2405.05966
Autor:
Opitz, Juri
Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many w
Externí odkaz:
http://arxiv.org/abs/2404.16958
Autor:
Opitz, Juri
The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and benchmarking of models that predict faithfulness of generated text. But w
Externí odkaz:
http://arxiv.org/abs/2404.03344
Autor:
Nawrath, Marcel, Nowak, Agnieszka, Ratz, Tristan, Walenta, Danilo C., Opitz, Juri, Ribeiro, Leonardo F. R., Sedoc, João, Deutsch, Daniel, Mille, Simon, Liu, Yixin, Zhang, Lining, Gehrmann, Sebastian, Mahamood, Saad, Clinciu, Miruna, Chandu, Khyathi, Hou, Yufang
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate
Externí odkaz:
http://arxiv.org/abs/2404.01701
With an increasing number of parameters and pre-training data, generative large language models (LLMs) have shown remarkable capabilities to solve tasks with minimal or no task-related examples. Notably, LLMs have been successfully employed as evalua
Externí odkaz:
http://arxiv.org/abs/2310.19792
Autor:
Opitz, Juri
The effectiveness of compression in text classification ('gzip') has recently garnered lots of attention. In this note we show that `bag-of-words' approaches can achieve similar or better results, and are more efficient.
Comment: improved writin
Comment: improved writin
Externí odkaz:
http://arxiv.org/abs/2307.15002
The task of natural language inference (NLI) asks whether a given premise (expressed in NL) entails a given NL hypothesis. NLI benchmarks contain human ratings of entailment, but the meaning relationships driving these ratings are not formalized. Can
Externí odkaz:
http://arxiv.org/abs/2306.00936
Conditional language models still generate unfaithful output that is not supported by their input. These unfaithful generations jeopardize trust in real-world applications such as summarization or human-machine interaction, motivating a need for auto
Externí odkaz:
http://arxiv.org/abs/2305.16819
Arguments often do not make explicit how a conclusion follows from its premises. To compensate for this lack, we enrich arguments with structured background knowledge to support knowledge-intense argumentation tasks. We present a new unsupervised met
Externí odkaz:
http://arxiv.org/abs/2305.08495
Autor:
Opitz, Juri
The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g
Externí odkaz:
http://arxiv.org/abs/2305.06993