t5-small-nl-to-regex

google-t5/t5-small fully fine-tuned to translate an English description of a text pattern into a regular expression, on the three DeepRegex corpora (KB13, NL-RX-Synth, NL-RX-Turk).

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("thealper2/t5-small-nl-to-regex")
model = AutoModelForSeq2SeqLM.from_pretrained("thealper2/t5-small-nl-to-regex")

PREFIX = "translate natural language to regex: "

# T5's SentencePiece vocabulary cannot represent \\ ^ ` { } ~, so the
# model emits them as '#b #c #g #l #r #n'. Decode them back:
CODEC = {"#b": "\\", "#c": "^", "#g": "`", "#l": "{", "#r": "}", "#n": "~"}

def decode_regex(text):
    out, i = [], 0
    while i < len(text):
        pair = text[i:i + 2]
        if pair in CODEC:
            out.append(CODEC[pair]); i += 2
        elif pair == "#h":
            out.append("#"); i += 2
        else:
            out.append(text[i]); i += 1
    return "".join(out)

def generate_regex(text):
    ids = tokenizer(PREFIX + text, return_tensors="pt")
    out = model.generate(**ids, max_length=128, num_beams=4, do_sample=False)
    return decode_regex(tokenizer.decode(out[0], skip_special_tokens=True))

generate_regex("lines that contain the word 'dog'")

Output dialect

Outputs follow the DeepRegex regex representation, which is not Python re syntax. It adds two operators:

Operator	Meaning	Example
~X	complement — matches every string X does not	~(.*e.*)
X&Y	intersection — matches strings matched by both	([A-Z])&([AEIOUaeiou])

Matching is whole-line (full-match) semantics. re.compile accepts these strings but reads ~ and & as literal characters, so outputs must be translated before use with Python re.

String literals follow the datasets' own preprocessing: words in quotes are normalised to the placeholders dog, truck, ring, lake in order of appearance. A description mentioning two quoted words yields a regex over dog and truck, which the caller substitutes back.

Training data

20824 examples from the three DeepRegex corpora:

Corpus	Examples	Unique regexes	Description style
KB13	824	732	Kushman & Barzilay (2013), human-written
NL-RX-Synth	10000	9648	synthetic descriptions generated from the regex
NL-RX-Turk	10000	9648	Mechanical Turk paraphrases of the NL-RX-Synth regexes

Split 80/10/10 with seed 42 (train 16659 / validation 2082 / test 2083), stratified by corpus. NL-RX-Synth and NL-RX-Turk have byte-identical targ.txt files, so they are split on a shared permutation of the line index; otherwise every NL-RX-Turk test regex would also be an NL-RX-Synth training target.

127 of 2083 test regexes (6.1%) also occur as a training target; these are genuine repeats in the corpora and were not removed.

Training procedure

Full fine-tuning of all parameters — no LoRA, adapters or quantisation.

Setting	Value
Base model	google-t5/t5-small
Parameters	60,506,624
Task prefix	translate natural language to regex:
Learning rate	0.0003
Weight decay	0.01
Warmup ratio	0.05
LR schedule	linear
Epochs (max)	15.0
Effective batch size	64
Max source / target length	128 / 128
Precision	bf16
Early stopping patience	3 epochs on validation exact match
Best checkpoint	checkpoint-3915

Evaluation

Held-out test split (2083 examples), beam search with num_beams=4, do_sample=False.

Metrics

• Exact match — string equality after stripping surrounding whitespace only.
• Regex validity — the output parses as a well-formed expression of the dialect.
• DFA equivalence — exact language equivalence decided with automata. Not decidable when either side contains a \b assertion, so it is reported with its coverage.
• Probe agreement — approximate: the prediction and the reference agree on every generated probe string. Finite-sample agreement, not a proof of equivalence.
• Functional accuracy — exact matches plus predictions judged equivalent by DFA equivalence, falling back to probe agreement where DFA equivalence is undecidable. The headline number; it mixes an exact and an approximate criterion and is reported as such.

Overall

Metric	Value
Exact match	66.78%
Regex validity	99.76%
Functional accuracy	78.54%
DFA equivalence (exact, 77.39% of examples decidable)	80.40%
Probe agreement (approximate)	79.98%
Character similarity	0.9016
Baseline: most frequent training regex	0.00% exact match

Per corpus

Corpus	n	Exact match	Functional accuracy	Validity
KB13	83	42.17%	68.67%	100.00%
NL-RX-Synth	1000	85.90%	89.70%	99.90%
NL-RX-Turk	1000	49.70%	68.20%	99.60%

Limitations

• Output is in the DeepRegex dialect, not Python re; ~ and & need translation before use.
• Quoted strings are placeholders (dog, truck, ring, lake), inherited from the datasets' preprocessing. The model does not copy real literals.
• The descriptions are short, single-sentence and template-like; behaviour on free-form or multi-sentence requests is untested.
• Probe agreement is finite-sample and can call two regexes equivalent when a longer counter-example exists.
• DFA equivalence cannot decide expressions containing \b, roughly 20% of the corpus.
• 60M parameters and a domain-specific vocabulary — not a general-purpose regex assistant.
• Generated regexes are not validated for catastrophic backtracking; do not run them on untrusted input without review.

Citation

Datasets from Locascio et al., Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge (EMNLP 2016), and Kushman & Barzilay, Using Semantic Unification to Generate Regular Expressions from Natural Language (NAACL 2013).