Tag Database Architecture

The tag database provides O(1) lookup for 32,000+ metadata tag definitions auto-generated from ExifTool source.

Overview

Metric	Value
Total Tags	16,684 (tag definitions; see Tag Coverage for the measured extraction score)
Modules Parsed	140+
Lookup Time	O(1)
Memory	~5-10MB (lazy loaded)

Workspace Architecture

The tag database is implemented as a separate workspace crate (oxidex-tags-*) to solve debug build memory issues.

Structure

oxidex/
├── oxidex-tags-core/     # Core types (TagDescriptor, TagId, etc.)
├── oxidex-tags-camera/   # Camera MakerNotes tags
├── oxidex-tags-media/    # Audio/video format tags
├── oxidex-tags-image/    # Image format tags (EXIF, PNG, etc.)
├── oxidex-tags-document/ # Document format tags (PDF, etc.)
├── oxidex-tags-specialty/# Specialized format tags (DICOM, etc.)
└── src/                  # Main crate (uses oxidex-tags-*)

Profile Configuration

# In root Cargo.toml
[profile.dev.package.oxidex-tags-core]
opt-level = 2        # Always optimize tag crates
codegen-units = 16   # Parallel compilation

[profile.dev.package.oxidex-tags-camera]
opt-level = 2
codegen-units = 16

# ... similar for other tag crates

Why Separate Crates?

• Debug builds: 100GB+ RAM → 11GB (91% reduction)
• Main crate stays in debug mode (fast iteration)
• Tag crates always optimized (prevents OOM)
• Industry-standard pattern (used by rustc, diesel, syn)

Tag Generation Pipeline

Tags are regenerated by explicitly running the sync-tags binary against a locally-installed exiftool — never as a side effect of cargo build:

cargo run --release --bin sync_tags

1. Run     → `exiftool -f -listx` dumps ExifTool's own resolved tag database as XML
2. Parse   → src/tag_sync/mod.rs::parse_listx reads id/name/writable/type/description
3. Route   → each tag's table name maps to one of 6 oxidex-tags-* domain crates
4. Generate → writes oxidex-tags-{domain}/src/{domain}_tags.yaml directly
5. Record  → `.exiftool-version` is updated with the exiftool release used

Because ExifTool has already resolved table-level WRITABLE inheritance by the time it emits -listx output, this captures per-tag type data that the old Perl-regex parser missed for the vast majority of tags.

Generated Code Structure

Each domain crate pre-compiles its YAML tag definitions to binary format at build time. The build.rs script eliminates the cold-start YAML parsing penalty by:

1. Reading the YAML source file (e.g. oxidex-tags-camera/src/camera_tags.yaml)
2. Deserializing with serde_yaml::from_str into TagDatabase structures
3. Serializing to efficient binary format with bincode::serde
4. Writing the binary blob to OUT_DIR for embedding via include_bytes!

This converts one-time deserialization work from runtime to compile time, avoiding repeated YAML parsing on every program start while keeping the source readable and maintainable.

Supported Formats

By Tag Count

Module	Tags	Description
DICOM	3,149	Medical imaging
NikonCustom	3,512	Nikon custom settings
Nikon	2,398	Nikon MakerNotes
Sony	1,148	Sony MakerNotes
QuickTime	1,069	Video metadata
Canon	930	Canon MakerNotes
Casio	930	Casio MakerNotes
Pentax	876	Pentax MakerNotes
EXIF	718	Core EXIF specification

By Category

Standard Formats:

• EXIF, GPS, XMP, IPTC, JFIF, TIFF

MakerNotes (30+ vendors):

• Canon, Nikon, Sony, Olympus, Panasonic, Pentax, FujiFilm
• Samsung, Minolta, Kodak, Casio, Ricoh, etc.

Video:

• QuickTime, MP4, Matroska, Flash, ASF, MPEG, H264

Audio:

• ID3, FLAC, Ogg, Vorbis, AAC, APE

Specialized:

• DICOM (medical), FITS (astronomy), MXF, PDF, PostScript

RAW:

• DNG, CR2, NEF, ARW, CanonRaw, SigmaRaw, MinoltaRaw

Graphics:

• PNG, GIF, BMP, PSD, JPEG, JPEG2000, OpenEXR, ICO

Documents:

• HTML, XML, SVG, VCard, LNK

Usage

Lookup by Tag Name

use oxidex::tag_db::get_tag_descriptor;

if let Some(tag) = get_tag_descriptor("EXIF:Make") {
    println!("Tag: {} (ID: {:?})", tag.tag_name, tag.tag_id);
}

Get All Tags for Format

use oxidex_tags_camera::canon::get_tags;

for (name, descriptor) in get_tags().iter() {
    println!("{}: {:?}", name, descriptor.value_type);
}

Rebuilding

To regenerate the tag database from a locally-installed exiftool:

cargo run --release --bin sync_tags

This overwrites oxidex-tags-*/src/*_tags.yaml and .exiftool-version directly — review the resulting git diff before committing.

Performance

• Lookup: O(1) via HashMap
• Memory: ~5-10MB (heap-allocated lazily)
• Build Time: ~4 minutes (cached after first build)
• Compilation: Uses Lazy initialization to avoid static limits

Build Requirements

Memory

Build Mode	Memory
Release	~5GB
Debug (with workspace)	~11GB
Debug (without workspace)	100GB+ (OOM)

Recommendation: Always use release builds for final testing.

Commands

# Development build
cargo build

# Release build (recommended for testing)
cargo build --release

# Run tests (release recommended)
cargo test --release --workspace

XML Parser Features

The XML tag parser (parse_listx in src/tag_sync/mod.rs) handles:

• XML element parsing: <table> (table group) and <tag> (individual tag definitions)
• Tag attributes: id, name, writable (boolean), type (optional)
• Both element forms: self-closing tags (<tag/>) and full tags (<tag>...</tag>)
• Nested descriptions: <desc lang='en'> text extraction (English locale only)
• XML entity unescaping: &, ', etc. in description text
• Writable inheritance resolution: ExifTool pre-resolves table-level inheritance in -listx output

Known Limitations

• Some ExifTool composite tags are excluded (calculated values)
• Shortcut tags are excluded (aliases to other tags)
• Some tags have platform-specific or format-specific variations