Initial commit

README.md

@@ -1,11 +1,13 @@
 ---
 title: Test of Time Accuracy
 datasets:
--  
+- baharef/ToT
+- aauss/ToT_separate_instructions 
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+- temporal reasoning
+description: "Accuracy metric for the Test of Time benchmark by Bahar et al. (2025)."
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
@@ -14,37 +16,107 @@ pinned: false
 
 # Metric Card for Test of Time Accuracy
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+
+This metric is designed for the **Test of Time (ToT)** benchmark (Bahar et al., 2025). It measures the accuracy of model predictions against reference answers. The metric expects model outputs to be formatted as a JSON object (e.g., `{"answer": "...", "explanation": "..."}`).
+
+It performs the following steps:
+
+1. Extracts the first valid JSON object from the model's prediction string.
+2. Processes the JSON based on the specified `subset`:
+   - **semantic**: Extracts the value of the "answer" field.
+   - **arithmetic**: Removes the "explanation" field and compares the remaining dictionary (containing the answer) to the reference.
+3. Compares the processed prediction with the reference to calculate accuracy, which is a dictionary for the artihmetic subset and a string for the semantic subset.
 
 ## How to Use
-*Give general statement of how to use the metric*
 
-*Provide simplest possible example for using the metric*
+You can load the metric using the `evaluate` library:
+
+```python
+import evaluate
+metric = evaluate.load("aauss/test_of_time_accuracy")
+
+predictions = [
+    '{"explanation": "Some explanation...", "unordered_list": ["London"]}',
+    ' "Response without opening curly brackets...", "answer": "2005-04-07"}',
+]
+
+references = [
+        '{"unordered_list": ["London"]}',
+        "{'answer': '2005-04-07'}",
+    ]
+
+print(
+    metric.compute(
+        predictions=predictions,
+        references=references,
+        subset="arithmetic",
+    )
+)
+>>> 0.5
+
+print(
+    metric.compute(
+        predictions=predictions,
+        references=references,
+        subset="arithmetic",
+        return_average=False
+    )
+)
+>>> [True, False]
+
+predictions = [
+    '{"explanation": "Some explanation leading to a wrong answer...", "answer": 1}',
+    '{"explanation": "Some explanation ...", "answer": "1985"}'
+]
+
+references = ["0", "1985"]
+
+print(
+    metric.compute(
+        predictions=predictions,
+        references=references,
+        subset="semantic",
+    )
+)
+>>> 0.5
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+
+- **predictions** (`list` of `str`): List of predictions to score. Each prediction should be a string that contains a JSON object (e.g., generated by an LLM).
+- **references** (`list` of `str`): List of reference answers.
+- **subset** (`str`): The subset of the benchmark being evaluated. Must be one of:
+  - `"arithmetic"`: Used for arithmetic tasks where the answer might need structure preservation (ignores explanation).
+  - `"semantic"`: Used for semantic tasks where only the "answer" value is compared.
+- **return_average** (`bool`, optional): If `True`, returns the average accuracy. If `False`, returns a list of boolean scores (correct/incorrect) for each sample. Defaults to `True`.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+The metric returns a dictionary with the following keys:
+
+- **accuracy** (`float` or `list` of `bool`): The accuracy score (0.0 to 1.0) if `return_average=True`, or a list of booleans indicating correctness per sample if `return_average=False`.
 
 *State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
 
-### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+Checkout the original [paper](https://openreview.net/pdf?id=44CoQe6VCq) for some reference performances.
+
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+
+- The metric relies on `json.JSONDecoder` to find the first JSON object in the prediction string. If the model output is malformed or does not contain a valid JSON, extraction may fail (returning `None`), leading to an incorrect prediction.
+- It strictly expects the extracted JSON to follow the format as described in the task and optionally `explanation` for the logic to work as intended.
 
 ## Citation
-*Cite the source where this metric was introduced.*
 
-## Further References
-*Add any useful further references.*
+Evaluation was not described in more detail in the paper but we can expect that model answers were parsed to allow for a more robust evaluation.
+
+```bibtex
+@InProceedings{huggingface:module,
+title = {Test of Time Accuracy},
+authors={Auss Abbood},
+year={2025}
+}
+```

requirements.txt

@@ -1 +1,2 @@
-git+https://github.com/huggingface/evaluate@main
+git+https://github.com/huggingface/evaluate@main
+pytest

test_of_time_accuracy.py

@@ -11,30 +11,31 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
+"""Accuracy metric for the Test of Time benchmark by Bahar et al. (2025)."""
 
-import evaluate
-import datasets
+import ast
+import json
+from typing import Literal
 
+import datasets
+import evaluate
 
-# TODO: Add BibTeX citation
 _CITATION = """\
 @InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
+title = {Test of Time Accuracy},
+authors={Auss Abbood},
+year={2025}
 }
 """
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+The Test of Time (ToT) benchmarks expects models format their answers as a JSON with an explanation field and an answer field that follows a predefined format. The metrics extracts JSONs objects from the model's output, retains only the first JSON, drops the explanation field and compares it with the reference answer.
 """
 
 
 # TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
+Compares the extracted answer from the model's output with the reference answer.
 Args:
     predictions: list of predictions to score. Each predictions
         should be a string with tokens separated by spaces.
@@ -53,43 +54,91 @@ Examples:
     {'accuracy': 1.0}
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
-
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class TestofTimeAccuracy(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+    """Accuracy metric for the Test of Time benchmark by Bahar et al. (2025)."""
+
+    __test__ = False
 
     def _info(self):
         # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
             module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
             # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Value("string"),
+                    "references": datasets.Value("string"),
+                    "subset": datasets.Value("string"),
+                    "return_average": datasets.Value("bool"),
+                }
+            ),
             # Homepage of the module for documentation
-            homepage="http://module.homepage",
+            # homepage="http://module.homepage",
             # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            # codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
+            # reference_urls=["http://path.to.reference.url/new_module"],
         )
 
-    def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
+    @staticmethod
+    def _extract_first_json_object(s: str) -> dict | None:
+        decoder = json.JSONDecoder()
+        idx, end = 0, len(s)
+        while idx < end:
+            try:
+                obj, next_idx = decoder.raw_decode(s, idx)
+                idx = next_idx
+                if isinstance(obj, dict):
+                    return obj
+            except ValueError:
+                idx += 1
+        return None
+
+    @staticmethod
+    def _pop_explanation(d):
+        if isinstance(d, dict):
+            d.pop("explanation", None)
+        return d
+
+    @staticmethod
+    def _get_answer(d):
+        if isinstance(d, dict):
+            return d.get("answer", None)
+        return d
+
+    @staticmethod
+    def _parse_label(s):
+        """Parses a string that could be a JSON object or a Python dict."""
+        try:
+            return json.loads(s)
+        except json.JSONDecodeError:
+            try:
+                # Safe: only parses literals, does not execute code
+                return ast.literal_eval(s)
+            except (ValueError, SyntaxError):
+                return None
 
-    def _compute(self, predictions, references):
+    def _compute(
+        self,
+        predictions,
+        references,
+        subset: Literal["arithmetic", "semantic"],
+        return_average: bool = True,
+    ):
         """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
-        return {
-            "accuracy": accuracy,
-        }
+        predictions = [self._extract_first_json_object(p) for p in predictions]
+        if subset == "semantic":
+            predictions = [self._get_answer(p) for p in predictions]
+        elif subset == "arithmetic":
+            predictions = [self._pop_explanation(p) for p in predictions]
+            references = [self._parse_label(r) for r in references]
+        else:
+            raise ValueError(f"Invalid subset: {subset}")
+        accuracy = [i == j for i, j in zip(predictions, references)]
+        if return_average:
+            return {"accuracy": sum(accuracy) / len(accuracy)}
+        return {"accuracy": accuracy}

tests.py

@@ -1,17 +1,81 @@
-test_cases = [
-    {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
-    },
-    {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
-    },
-    {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]
+import pytest
+
+from test_of_time_accuracy import TestofTimeAccuracy
+
+arithmetic_test_cases = {
+    "predictions": [
+        'JSON = {"explanation": "The war began in 360 BC. Since BC years count backwards, adding 8 years to 360 BC means subtracting 8 from 360, resulting in 352 BC.", "answer": "352 BC"}',
+        '```json\n{\n  "explanation": "The dates provided are March 2012, September 2011, June 2017, September 2019, and June 2015. These correspond to visits to Miami, Sydney, Tokyo, London, and Nairobi respectively. The latest date among these is September 2019, which is associated with London. Therefore, London is the last city visited.",\n  "unordered_list": ["London"]\n}\n```',
+        ' "To find the date of the second most important game, we need to subtract 7 days from the date of the most important game. We can do this by counting back 7 days from April 14, 2005. April 14 - 7 days = April 7, 2005", "answer": "2005-04-07"}',
+    ],
+    "references": [
+        '{"answer": "352 BC"}',
+        '{"unordered_list": ["London"]}',
+        "{'answer': '2005-04-07'}",
+    ],
+    "result": {"accuracy": 2 / 3},
+    "per_item_accuracy": [True, True, False],
+}
+
+semantic_test_cases = {
+    "predictions": [
+        ' "First, we need to find the third occurrence of E33 being the R53 of E22. We can see that it happened from 1959 to 1962, then from 1967 to 1968, and then from 1982 to 1984. The third occurrence happened from 1982 to 1984. We can then compute the duration by subtracting the start time from the end time.", "answer": 2}',
+        ' "To find the duration, we need to find the start and end time when E97 was the R71 of E67. From the given facts, we can see that E97 was the R71 of E67 from 1961 to 1961, and also from 1964 to 1964. We need to find the first occurrence, which is from 1961 to 1961.", "answer": 1}',
+        '{"explanation": "To find when E92 stopped being the R88 of E11, we need to look at the temporal facts where E92 was the R88 of E11 and find the end time. We see that E92 was the R88 of E11 from 1982 to 1985, and there is no other fact that indicates E92 stopped being the R88 of E11 before 1985. However, we also see that E92 was the R17 of E42 from 1986 to 1992, and E92 was the R88 of E42 from 1977 to 1979, but this is irrelevant to the question. Therefore, E92 stopped being the R88 of E11 in 1985.", "answer": "1985"}',
+    ],
+    "references": ["2", "0", "1985"],
+    "result": {"accuracy": 1 / 3},
+    "per_item_accuracy": [False, False, True],
+}
+
+
+def test_arithmetic_accuracy():
+    metric = TestofTimeAccuracy()
+    results = metric.compute(
+        predictions=arithmetic_test_cases["predictions"],
+        references=arithmetic_test_cases["references"],
+        subset="arithmetic",
+    )
+    assert results == arithmetic_test_cases["result"]
+
+
+def test_semantic_accuracy():
+    metric = TestofTimeAccuracy()
+    results = metric.compute(
+        predictions=semantic_test_cases["predictions"],
+        references=semantic_test_cases["references"],
+        subset="semantic",
+    )
+    assert results == semantic_test_cases["result"]
+
+
+def test_per_item_arithmetic_accuracy():
+    metric = TestofTimeAccuracy()
+    results = metric.compute(
+        predictions=arithmetic_test_cases["predictions"],
+        references=arithmetic_test_cases["references"],
+        subset="arithmetic",
+        return_average=False,
+    )
+    assert results["accuracy"] == arithmetic_test_cases["per_item_accuracy"]
+
+
+def test_per_item_semantic_accuracy():
+    metric = TestofTimeAccuracy()
+    results = metric.compute(
+        predictions=semantic_test_cases["predictions"],
+        references=semantic_test_cases["references"],
+        subset="semantic",
+        return_average=False,
+    )
+    assert results["accuracy"] == semantic_test_cases["per_item_accuracy"]
+
+
+def test_invalid_subset():
+    metric = TestofTimeAccuracy()
+    with pytest.raises(ValueError):
+        metric.compute(
+            predictions=arithmetic_test_cases["predictions"],
+            references=arithmetic_test_cases["references"],
+            subset="invalid",
+        )