The reference: phonetics and tajweed rulings

The model hears. What a recitation should be, sound by sound and rule by rule, is Quran Lab's reference for the phonetics and tajweed rulings of the Quran, in the riwaya of Hafs 'an 'Asim via tariq al-Shatibiyyah. Every ayah is written out as phones, and every phone carries the rulings that made it what it is. It is the ground truth the model's labels and its expected units (model.reference) come from. It is public in two places: the engine, quran-g2p, and the dataset made with it, quran-tajweed-phonetics.

From a rule to the book and the page

Every phone of the dataset lists in rules the ids of the rulings that produced it, such as R185_MUTTASIL. Each id resolves in the dataset's rulings.jsonl to the ruling's name in Arabic and in English, its phase, and its classical citation as the file gives it: the book, with the volume and page or the bayt. So when your app's own judge finds, from what the model heard, that a learner did not recite a place as the reference prescribes, the app can show the learner which ruling applies there and where the scholars state it. The worked example below shows the whole chain on real data.

The citation says where the classical works state the ruling that the engine implements. It does not say that the learner erred. That judgement is your app's, made on the model's output (Build your own feedback); neither the SDK nor Quran Lab ever tells a learner that they broke a rule.

What the published files cover, counted for this page:

• The phones carry 45 distinct ids in rules. 44 of them resolve in rulings.jsonl. The 45th, EMIT, is not a ruling: it records that the phone was read from the written text. Skip it.
• rulings.jsonl has 62 rulings, each equal field by field to the engine's register. The register has 63 today: the one the file lacks, R162_NAKHLUQKUM_NAQIS (the second wajh at 77:20), was added to the engine after the dataset was last made, and no phone of the dataset carries it. The engine's README and the dataset's card still say 62; the register file says 63, and 63 is the count today.
• 53 of the 62 citations give a place in the book: a volume and page, a page, or a bayt number. 9 give the work without a place: R134_LAM_JALALA_ALIF, R180_TABEEI, R189_AARED and R212_LAM_JALALA quote the line of the poem they cite, and R112_STRIP_INITIAL_SHADDA, R121_MADD_EWAD, R130_WASL_ELISION, R221_IMALA and R222_TASHEEL name the work only.
• Each ruling has reviewed rows in register_reviewed.jsonl: an ayah that witnesses it, the ruling's wording for that ayah, and the reviewer's decision (صحيح or فيه وجهان). Those rows carry the review's own ids (the engine's R185_MUTTASIL is reviewed as R182_MUTTASIL), so the way from an engine id to its rows is the engine's register: its review column names the golden rows (tests/goldens/) that put the ruling before the reviewers, and each golden row names its ayah. All 63 rulings reach at least one reviewed row that way.

The engine and the dataset

quran-g2p is the engine: text in, tajweed-attributed phones out, in Python with no dependencies. What a developer uses in it:

• src/quran_g2p/: the engine. phonemize gives the phones of an ayah, and tokenlayer_tj3 turns phones into the model's tokens.
• docs/RULINGS-REGISTER.md: the register, generated from the engine's code. Each ruling with its id, its Arabic and English name, its sources and its review rows.
• artifacts/tokenizer_tj3/: the model's vocabulary, tokens.txt and its manifest.
• artifacts/spans/: the rulings as character offsets over all 6,236 ayat, for a project that cannot run Python.

quran-tajweed-phonetics is the dataset: 6,236 ayat and 522,475 phones, made with the engine.

word_index counts the phonetic words of the ayah: one per written word, except in the 27 ayat whose opening letters are spelled out (R011_MUQATTAAT), where each letter's name is a word of its own.

The licence, in the card's own words: "Quran-Lab No-Profit License 1.2 (see LICENSE): free to use, adapt and redistribute, share-alike. The work itself is never for sale, so you may not charge for it, for access to it, or for a feature it powers, and the same holds for derivatives including models trained with it."

How it was verified

From the dataset's card and the engine's README. Every ruling carries its classical citation, and each citation was checked against the printed books. The rulings register the reviewers read has 208 entries. An ijazah-holding hafiz, Shaikh Sami Almadani, ruled on all 208 from his own talaqqi: 191 confirmed, 17 confirmed while documenting a second transmitted wajh, and none judged wrong. The complete register then underwent formal scholarly arbitration (تحكيم علمي) by two serving professors of Qira'at at Umm al-Qura University in Makkah, each engaged separately and blind to every other reviewer: the first confirmed the register, and none of the second's notes overturned a phonetic ruling. Over the whole corpus, the two pinned mushaf editions (Tanzil and KFGQPC) give identical phones for all 6,236 ayat; the phones agree with the mushaf's own dabt at about 8,900 tanween sites; they were checked in both directions against the quran-tajweed rule-span dataset (60,057 annotations) and word by word against the colour-coded Madinah mushaf of the King Fahd Complex (77,429 words); the card lists the differences.

The model and the reference

The 38 tokens are the engine's. model.tokens of TajweedFormer V4 is the engine's tj3 vocabulary: the tokens.txt of the 4.0.0 model is byte for byte artifacts/tokenizer_tj3/tokens.txt (SHA-256 66fe6eb81b15506270f9e3c6efb0724cda757cd54326f26ca4bcd6be4f3b2aa4). One token per sound: a long vowel is one token (the vowel and its madd letter), and a geminate consonant is one token. Madd counts, gemination, ghunna length, rawm and sakt are not tokens, and neither are tafkheem, qalqalah and the class of a ghunna: the model gives them as heads (timing_log_probs, gem, attr_tafkheem, attr_qalqalah, attr_ghunna; Build your own feedback). That is where shadda, ghunna and idgham are: heads in the model, and geminated, ghunna and the rule ids in the reference.

model.reference is made from the dataset. Quran Lab's release pipeline renders each row of quran_phonetics.jsonl with the engine's tokenlayer_tj3.phones_to_tj3_units: the units of the ayah read from its start to its end with a stop, with the word boundary | between words, and the word of each unit. Checked for this page on all 6,236 ayat: the table that ships with 4.0.0 equals what quran-g2p at its current public commit (7eeba12) produces, and the dataset's phones equal the engine's.

The join. phones_to_tj3_units([[p] for p in row["phones"]]) returns the ayah's units and, for each phone, the index of the unit it became. Those units are exactly the ones model.reference gives for the ayah, so unit i of the reference is made of the phones whose index is i: a long vowel is its vowel and its madd letter, and every other unit is one phone. The table the SDK ships holds unit ids and words only. The phones, their marks and their rules come from the dataset through this join. For the letters, rule_index.jsonl gives each phone that carries a rule the characters of text it was read from (span), and the engine gives every phone the same (src_span).

A worked example

Clone the engine and put three files of the dataset beside it: quran_phonetics.jsonl (about 150 MB), rulings.jsonl and register_reviewed.jsonl.

git clone https://github.com/Quran-Lab/quran-g2p

Save this as reference_example.py in the same folder:

# reference_example.py: the reference for one ayah, joined to the model's expected units.
#   python reference_example.py 1 1             each unit with its phones, their rules, and the rules' sources
#   python reference_example.py R185_MUTTASIL   one ruling: its row of rulings.jsonl and its reviewed rows
# Beside this file: a clone of quran-g2p, and rulings.jsonl, quran_phonetics.jsonl and register_reviewed.jsonl
# from the dataset. The ruling mode also needs pyyaml (pip install pyyaml).
import glob, json, sys
sys.path.insert(0, "quran-g2p/src")
from quran_g2p import tokenlayer_tj3 as tj3

def rows(name):
    with open(name, encoding="utf-8") as f:
        yield from map(json.loads, f)

rulings = {r["rule_id"]: r for r in rows("rulings.jsonl")}

def ayah(surah, number):
    row = next(r for r in rows("quran_phonetics.jsonl") if (r["surah"], r["ayah"]) == (surah, number))
    phones = row["phones"]
    units, first = tj3.phones_to_tj3_units([[p] for p in phones])  # first[k]: the unit phone k became
    print(row["text"])
    print("ids", [tj3.INDEX[u] for u in units])                      # what model.reference gives for the ayah
    named = []
    for i, u in enumerate(units):
        mine = [p for k, p in enumerate(phones) if first[k] == i]
        if not mine:                                                  # "|", the word boundary
            continue
        rules = list(dict.fromkeys(r for p in mine for r in p.get("rules", []) if r != "EMIT"))
        marks = ["geminated" for p in mine if p["geminated"]]
        marks += [f"{k} {p[k]}" for p in mine for k in ("ghunna", "qalqalah") if k in p]
        marks += [f"length {p['length']['allowed']}" for p in mine if "length" in p]
        print(f"{i:3} {tj3.INDEX[u]:2} {u:3} word {mine[0]['word_index']}", "+".join(p["base"] for p in mine), *marks, *rules)
        named += [r for r in rules if r not in named]
    for r in named:
        print(r, "|", rulings[r]["name_ar"], "|", rulings[r]["name_en"], "|", rulings[r]["citation"])

def ruling(rule_id):
    import yaml
    from quran_g2p.rules.registry import COVERAGE                     # the golden rows put before the reviewers
    print(json.dumps(rulings[rule_id], ensure_ascii=False))
    golden = {g["id"]: g for f in glob.glob("quran-g2p/tests/goldens/*.yaml")
              for g in yaml.safe_load(open(f, encoding="utf-8")) or []}
    sites = {(golden[g]["surah"], golden[g]["ayah"], golden[g]["rule"]) for g in COVERAGE[rule_id]}
    for r in rows("register_reviewed.jsonl"):
        if (r["surah"], r["ayah"], r["rule_id"]) in sites:
            print("row", r["row"], f"{r['surah']}:{r['ayah']}", r["rule_id"], "|", r["ruling_ar"], "|", r["review_verdict"])

if sys.argv[1].startswith("R"):
    ruling(sys.argv[1])
else:
    ayah(int(sys.argv[1]), int(sys.argv[2]))

python reference_example.py 1 1, Al-Fatihah 1:1. A line reads: the unit's index in model.reference for the ayah, its id in model.tokens, the token, its word, the phones it was made from with their marks, and their rule ids. The indexes the list skips (5, 10, 18) are the word boundaries. The last lines name each ruling from rulings.jsonl, with its citation exactly as the file has it.

بِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ
ids [3, 32, 13, 25, 32, 1, 24, 34, 27, 32, 1, 11, 31, 7, 25, 34, 26, 32, 1, 11, 31, 7, 35, 25]
  0  3 ب   word 0 beh
  1 32 i   word 0 kasra
  2 13 س   word 0 seen
  3 25 م   word 0 meem
  4 32 i   word 0 kasra
  6 24 ل   word 1 lam geminated R133_R160_IDGHAM_KAMIL
  7 34 aa  word 1 fatha+alef_madd length [2] R134_LAM_JALALA_ALIF R180_TABEEI
  8 27 ه   word 1 heh
  9 32 i   word 1 kasra
 11 11 ر   word 2 reh geminated R133_R160_IDGHAM_KAMIL R211_REH
 12 31 a   word 2 fatha
 13  7 ح   word 2 hah
 14 25 م   word 2 meem
 15 34 aa  word 2 fatha+alef_madd length [2] R180_TABEEI
 16 26 ن   word 2 noon
 17 32 i   word 2 kasra
 19 11 ر   word 3 reh geminated R133_R160_IDGHAM_KAMIL R211_REH
 20 31 a   word 3 fatha
 21  7 ح   word 3 hah
 22 35 ii  word 3 kasra+yeh_madd length [2, 4, 6] R189_AARED
 23 25 م   word 3 meem
R133_R160_IDGHAM_KAMIL | الإدغام الكامل (اللام الشمسية والمثلان المتصلان) | complete idgham: lam shamsiyya and adjacent mithlayn | تحفة الأطفال «للام أل حالان قبل الأحرف»؛ النشر 2:18-19
R134_LAM_JALALA_ALIF | ألف لفظ الجلالة المحذوفة رسمًا | the dagger alif of the jalala | الجزرية «وفخم اللام من اسم الله عن فتح أو ضم»
R180_TABEEI | المد الطبيعي | madd tabee'i (2) | تحفة الأطفال «والمد أصلي وفرعي له»
R211_REH | أحكام الراء تفخيمًا وترقيقًا | the raa decision table | الجزرية باب الراءات «ورقق الراء إذا ما كسرت»؛ هداية القاري 1:130
R189_AARED | المد العارض للسكون | madd 'aared lil-sukun (2/4/6) | تحفة الأطفال «ومثل ذا إن عرض السكون»

python reference_example.py 2 5, an ayah with a muttasil madd (units 3 and 35) and an idgham with ghunna (unit 19). Where the output is cut, a line shows ....

أُو۟لَٰٓئِكَ عَلَىٰ هُدًى مِّن رَّبِّهِمْ وَأُو۟لَٰٓئِكَ هُمُ ٱلْمُفْلِحُونَ
ids [2, 33, 24, 34, 2, 32, 23, 31, 1, 19, 31, 24, 34, 1, 27, 33, 9, 31, 1, 25, 32, 1, 11, 31, 3, 32, 27, 32, 25, 1, 28, 31, 2, 33, 24, 34, 2, 32, 23, 31, 1, 27, 33, 25, 33, 1, 24, 25, 33, 21, 24, 32, 7, 36, 26]
  0  2 ء   word 0 hamza
  1 33 u   word 0 damma
  2 24 ل   word 0 lam
  3 34 aa  word 0 fatha+alef_madd length [4, 5] R185_MUTTASIL
  4  2 ء   word 0 hamza
  5 32 i   word 0 kasra
  6 23 ك   word 0 kaf
  7 31 a   word 0 fatha
...
 14 27 ه   word 2 heh
 15 33 u   word 2 damma
 16  9 د   word 2 dal
 17 31 a   word 2 fatha
 19 25 م   word 3 meem geminated ghunna idgham length [2] R141_IDGHAM_GHUNNA
 20 32 i   word 3 kasra
 22 11 ر   word 4 reh geminated R133_R160_IDGHAM_KAMIL R211_REH
 23 31 a   word 4 fatha
 24  3 ب   word 4 beh geminated
 25 32 i   word 4 kasra
 26 27 ه   word 4 heh
 27 32 i   word 4 kasra
 28 25 م   word 4 meem
...
 53 36 uu  word 7 damma+waw_madd length [2, 4, 6] R189_AARED
 54 26 ن   word 7 noon
R185_MUTTASIL | المد الواجب المتصل | madd muttasil (4-5) | الشاطبية باب المد؛ النشر 1:315 «فوجب أن لا يعتقد أن قصر المتصل جائز عند أحد من القراء»
R180_TABEEI | المد الطبيعي | madd tabee'i (2) | تحفة الأطفال «والمد أصلي وفرعي له»
R141_IDGHAM_GHUNNA | الإدغام بغنة الكامل (ن، م) | complete idgham with ghunna | تحفة الأطفال «في يرملون»؛ النشر 2:22
R133_R160_IDGHAM_KAMIL | الإدغام الكامل (اللام الشمسية والمثلان المتصلان) | complete idgham: lam shamsiyya and adjacent mithlayn | تحفة الأطفال «للام أل حالان قبل الأحرف»؛ النشر 2:18-19
R211_REH | أحكام الراء تفخيمًا وترقيقًا | the raa decision table | الجزرية باب الراءات «ورقق الراء إذا ما كسرت»؛ هداية القاري 1:130
R189_AARED | المد العارض للسكون | madd 'aared lil-sukun (2/4/6) | تحفة الأطفال «ومثل ذا إن عرض السكون»

The ب of رَّبِّهِمْ (unit 24) is geminated with no rule: a shadda of the word's own spelling is not a ruling. One place where the rule id on a phone is not the one the register reviews: where a noon sakinah ends a word and ر or ل begins the next, as in مِّن رَّبِّهِمْ (unit 22), the geminated letter carries R133_R160_IDGHAM_KAMIL, while the reviewed row for this ayah is under R142_IDGHAM_BILA_GHUNNA, which the phones carry only after a tanween. There are 338 such places. The phones and the units there are as prescribed; only the rule id is the general one.

python reference_example.py 112 1, a qalqalah kubra at the stop (unit 18):

قُلْ هُوَ ٱللَّهُ أَحَدٌ
ids [22, 33, 24, 1, 27, 33, 28, 31, 1, 24, 34, 27, 33, 1, 2, 31, 7, 31, 9]
  0 22 ق   word 0 qaf R210_ISTILA
  1 33 u   word 0 damma
  2 24 ل   word 0 lam
  4 27 ه   word 1 heh
  5 33 u   word 1 damma
  6 28 و   word 1 waw
  7 31 a   word 1 fatha
  9 24 ل   word 2 lam geminated R133_R160_IDGHAM_KAMIL R212_LAM_JALALA
 10 34 aa  word 2 fatha+alef_madd length [2] R134_LAM_JALALA_ALIF R180_TABEEI
 11 27 ه   word 2 heh
 12 33 u   word 2 damma
 14  2 ء   word 3 hamza
 15 31 a   word 3 fatha
 16  7 ح   word 3 hah
 17 31 a   word 3 fatha
 18  9 د   word 3 dal qalqalah kubra R201_QALQALAH_KUBRA
R210_ISTILA | تفخيم حروف الاستعلاء ومراتبه | isti'la tafkheem and its maraatib | المقدمة الجزرية (باب صفات الحروف)؛ النويري 1:237-238
R133_R160_IDGHAM_KAMIL | الإدغام الكامل (اللام الشمسية والمثلان المتصلان) | complete idgham: lam shamsiyya and adjacent mithlayn | تحفة الأطفال «للام أل حالان قبل الأحرف»؛ النشر 2:18-19
R212_LAM_JALALA | تفخيم لام الجلالة وترقيقها | lam al-jalala tafkheem/tarqeeq | الجزرية «وفخم اللام من اسم الله عن فتح أو ضم»
R134_LAM_JALALA_ALIF | ألف لفظ الجلالة المحذوفة رسمًا | the dagger alif of the jalala | الجزرية «وفخم اللام من اسم الله عن فتح أو ضم»
R180_TABEEI | المد الطبيعي | madd tabee'i (2) | تحفة الأطفال «والمد أصلي وفرعي له»
R201_QALQALAH_KUBRA | القلقلة الكبرى | qalqalah kubra (waqf) | الجزرية «وبينن مقلقلًا إن سكنا وإن يكن في الوقف كان أبينا»؛ هداية القاري 1:84-87

From a place to the book: the ruling at each of those places, with its row of rulings.jsonl printed whole and its reviewed rows. python reference_example.py R185_MUTTASIL (the madd at unit 3 of 2:5):

{"rule_id": "R185_MUTTASIL", "phase": "P10", "name_ar": "المد الواجب المتصل", "name_en": "madd muttasil (4-5)", "spec": "SPEC-185", "citation": "الشاطبية باب المد؛ النشر 1:315 «فوجب أن لا يعتقد أن قصر المتصل جائز عند أحد من القراء»"}
row 44 4:43 R182_MUTTASIL | مد واجب متصل (جَآءَ) | صحيح
row 45 2:19 R182_MUTTASIL | مد متصل (ٱلسَّمَآءِ) | صحيح

python reference_example.py R141_IDGHAM_GHUNNA (unit 19 of 2:5):

{"rule_id": "R141_IDGHAM_GHUNNA", "phase": "P6", "name_ar": "الإدغام بغنة الكامل (ن، م)", "name_en": "complete idgham with ghunna", "spec": "SPEC-141", "citation": "تحفة الأطفال «في يرملون»؛ النشر 2:22"}
row 115 2:61 R141_IDGHAM_GHUNNA | إدغام كامل بغنة في النون (لَن نَّصْبِرَ) | صحيح
row 116 86:6 R141_IDGHAM_GHUNNA | إدغام كامل بغنة في الميم (مِن مَّآءٍ دَافِقٍ) | صحيح
row 127 2:26 R141_IDGHAM_GHUNNA | إدغام تنوين (مَثَلًا) في ميم (مَّا) | صحيح
row 176 12:32 R141_IDGHAM_GHUNNA | نون التوكيد الخفيفة تدغم في الميم وصلًا (وَلَيَكُونًا مِّنَ ٱلصَّٰغِرِينَ)؛ والوقف عليها بالألف | صحيح

python reference_example.py R201_QALQALAH_KUBRA (unit 18 of 112:1; the long row is cut):

{"rule_id": "R201_QALQALAH_KUBRA", "phase": "P11", "name_ar": "القلقلة الكبرى", "name_en": "qalqalah kubra (waqf)", "spec": "SPEC-200", "citation": "الجزرية «وبينن مقلقلًا إن سكنا وإن يكن في الوقف كان أبينا»؛ هداية القاري 1:84-87"}
row 18 113:1 R201_QALQALAH_KUBRA | قلقلة كبرى وقفًا (ٱلْفَلَقِ) | صحيح
row 20 112:1 R201_QALQALAH_KUBRA | قلقلة كبرى وقفًا (أَحَدٌ ...
row 21 85:20 R201_QALQALAH_KUBRA | قلقلة كبرى وقفًا للطاء (مُّحِيطُۢ) | صحيح
row 22 22:5 R201_QALQALAH_KUBRA | قلقلة كبرى وقفًا للجيم (بَهِيجٍ) | صحيح

Where each rule is in the data

The names and citations of every id are in rulings.jsonl; the whole register, with the rulings that no phone carries, is docs/RULINGS-REGISTER.md.

What it is not

• Lengths are prescriptions, not observations. In the card's words: "Where the tradition gives the reciter a choice (munfasil, 'aared, leen), the value actually realized in a recording is a measurement this text layer cannot supply. Do not treat canonical values as ground truth at free-choice positions." A learner who recites a munfasil at 4 or at 5 is within allowed.
• One reading of each ayah. The rows are phonemized as read from the start of the ayah to its end, with a stop there, and so is model.reference. A stop inside the ayah, rawm and ishmam, and two ayat read without a stop between them can all be generated with the engine; the dataset does not hold them.
• One riwaya. Hafs 'an 'Asim via tariq al-Shatibiyyah, and no other.
• Not a judgement of anyone. In the card's words: "This dataset states the prescription of the riwaya; certifying reciters remains the sanad's work."

Building feedback with it

A judge compares what the model heard with what the ayah prescribes. The SDK gives the first (the heads, the timing, the calibration: Build your own feedback) and model.reference the expected units; this reference gives, for each unit, its phones, its rules and their sources. Use the reference for what is expected and the model's output for what was heard, and phrase whatever your app shows as your app's reading of the model, never as a ruling. Never write Quran text, phones or rule assignments from memory: take them from the reference.