The seven tables, the sources they come from, the coverage each enrichment column actually has, and how to query the result.
Seven tables, exported as pipe-delimited UTF-8 CSV and Apache Parquet with identical columns:
position_index.fragrance_allergens rows — the EU Annex III fragrance-allergen entries, one row per legal name and INCIDB match, review rows and unmatched names included.regulatory_status rows — EU Annex II–VI status per ingredient and list entry, conditions verbatim. A row exists only where a list says something.INCIDB Korea, sold separately, adds three tables in a second archive from the same edition: kr_mfds_entries, kr_mfds_substances and kr_mfds_links, built from Annexes 1 and 2 of South Korea's MFDS Notice 2026-19, with the Korean text as printed and INCIDB links by exact CAS. An ingredient without a link has no status in this data.
Three sources, each with a documented join. Every source file's URL, download timestamp and SHA-256 is recorded in the build_report.json shipped with the data.
categories_tags, raw ingredient declarations) under the Open Database License (ODbL).NULL in every CosIng-derived column, never a placeholder. The Annex II–VI exports also feed regulatory_status, matched by the CosIng glossary name or the CosIng "Identified INGREDIENTS" name only (no CAS route). Absence of a row is not a status, and nothing in it is legal advice.rating_source, and are NULL everywhere else.South Korea MFDS (INCIDB Korea only) — Annexes 1 and 2 of MFDS Notice 2026-19 (Regulation on Safety Standards etc. of Cosmetics), from law.go.kr. Each entry is linked to INCIDB only where one of its printed CAS numbers equals the single CAS INCIDB holds for an ingredient. Links on botanical entries, links a hand check found wrong and links whose CAS number several INCIDB names share ship as review links; they, and links not yet hand-checked, stay outside the published counts; a shared CAS still links normally for a name hand-checked as that same substance. Many Annex 1 entries apply only as limited by a condition or exception written in the entry. The notice URL and the annex PDF hash are on every entry row (source_url, pdf_sha256), not in build_report.json. Not legal advice.
Reported two ways, because the two differ by an order of magnitude and quoting only one would mislead:
functions: 12.0% of distinct names; 84.3% of label occurrences.cas_number: 9.0% of distinct names; 78.8% of label occurrences.chemical_description: 9.5% of distinct names, 4,322 distinct values.A label corpus contains far more distinct strings — botanical variants, multilingual spellings, marketing tokens — than any regulatory inventory lists, which is why the first number in each pair is low. The slots on a real label are filled overwhelmingly by ingredients CosIng does cover, which is why the second is high. Neither number alone describes the data.
Raw label tokens are resolved to canonical INCI names by an explicit ordered procedure — entity decode, character strip, whitespace collapse, upper-case, then exact match, cleaned match, a small cited typo map, percentage stripping, parenthesis stripping, a synonym map, slash-variant splitting, and concatenation splitting only where both halves match the inventory. Tokens that none of them resolves are re-split by explicit rules (brackets, (and), bullets, colons, . separators, line breaks), never fuzzily: every recovered name must match the inventory or the cited maps exactly, and unmatched text between recovered names is kept as unresolved_residual. A token that nothing resolves is kept verbatim and recorded as unresolved, never guessed. The method used is stored per row in ingredient_name_map.method, with part_index and split_rule recording each cut, so every decision is auditable and reversible.
import pyarrow.parquet as pq
ingredients = pq.read_table('ingredients.parquet').to_pandas()
links = pq.read_table('product_ingredients.parquet').to_pandas()
# Coverage the way it actually matters: weighted by label occurrence.
merged = links.merge(ingredients, on='ingredient_id', how='left')
print(merged['functions'].notna().mean())
# EU Annex III fragrance allergens
allergens = ingredients[ingredients['is_common_allergen'] == 1]import duckdb
duckdb.query("""
SELECT f.function, COUNT(*) AS label_slots
FROM 'product_ingredients.parquet' pi
JOIN 'ingredients.parquet' i USING (ingredient_id)
CROSS JOIN UNNEST(string_split(i.functions, ';')) AS f(function)
WHERE i.functions IS NOT NULL
GROUP BY 1 ORDER BY 2 DESC LIMIT 10;
""").show()
# Audit the canonicalisation itself
duckdb.query("""
SELECT method, COUNT(*) AS n
FROM 'ingredient_name_map.parquet'
GROUP BY 1 ORDER BY 2 DESC;
""").show()
# INCIDB Korea: Annex 2 preservatives linked to INCIDB ingredients
duckdb.query("""
SELECT l.inci_name, e.name_ko, e.limit_ko, l.scope_note
FROM 'kr_mfds_links.parquet' l
JOIN 'kr_mfds_entries.parquet' e USING (entry_id)
WHERE e.section = 'preservative' AND l.review = 0
ORDER BY l.inci_name;
""").show()