Using Regular Expressions to Parse PDF Test Data into JSON (with Data Validation)

Working with test reports stored in PDF files can be frustrating — especially when you need to extract structured data for automated processing. In this guide, we’ll walk through how to use Python’s regular expressions (re module) to:

• Parse test results from PDF files using PyPDF2

• Extract patterns using readable and maintainable named capture groups

• Validate structured output using jsonschema

• Understand common regex syntax, flags, and best practices

---

The Challenge: Test Reports in PDF Format

Imagine you’re given a batch of PDFs containing test results in this format:

Test1234    2025-05-10    PASS
Test1235    2025-05-11    FAIL

The goal is to extract these results into structured JSON. Regular expressions let us define patterns to find these entries reliably.

---

Step 1: Extracting Text with PyPDF2

Use PyPDF2 to read PDF text:

import PyPDF2

def extract_text_from_pdf(file_path):
    with open(file_path, 'rb') as file:
        reader = PyPDF2.PdfReader(file)
        return "\n".join([page.extract_text() for page in reader.pages])

---

Step 2: Using Regex with Named Groups for Readability

Use named capture groups ((?P<name>...)) to improve readability:

import re

pattern = re.compile(
    r'(?P<test_id>Test\d+)\s+(?P<date>\d{4}-\d{2}-\d{2})\s+(?P<result>PASS|FAIL)',
    re.MULTILINE | re.IGNORECASE
)

matches = pattern.finditer(text)
results = [match.groupdict() for match in matches]

---

Step 3: Validate Extracted Data with jsonschema

from jsonschema import validate

schema = {
    "type": "object",
    "properties": {
        "test_id": {"type": "string"},
        "date": {"type": "string", "format": "date"},
        "result": {"type": "string", "enum": ["PASS", "FAIL"]}
    },
    "required": ["test_id", "date", "result"]
}

for result in results:
    validate(instance=result, schema=schema)

---

Regex Syntax Reference

Basic Elements

Pattern	Description	Example Match
.	Any character except newline	a.c → abc, a-c
\d	Digit (0–9)	\d+ → 123
\D	Non-digit	\D+ → abc
\w	Word character (a-zA-Z0-9_)	\w+ → hello123
\W	Non-word character	\W → #, .
\s	Whitespace (space, tab, newline)	\s → ' ', \n
\S	Non-whitespace	\S → a, 1, _
[]	Character set	[aeiou] → a, e
[^]	Negated set	[^0-9] → a, B
^	Start of line or string	^Hello matches Hello
$	End of line or string	world$ matches world
\b	Word boundary	\bword\b matches word only

---

Quantifiers

Pattern	Description	Example Match
*	0 or more repetitions	lo*l → ll, lool
+	1 or more repetitions	lo+l → lol, lool
?	0 or 1 repetition	colou?r → color, colour
{m}	Exactly m repetitions	a{3} → aaa
{m,n}	Between m and n repetitions	a{2,4} → aa, aaa

---

🧩 Grouping and Capturing

Pattern	Description	Example
(...)	Capture group	(abc)+ → captures abc
(?:...)	Non-capturing group	(?:abc)+
(?P<name>...)	Named capturing group	(?P<year>\d{4})
(?P=name)	Backreference to named group

---

⚙️ Common Regex Functions in Python

Function	Purpose
re.findall()	Return all matches
re.search()	Return first match
re.match()	Match from beginning of string
re.sub()	Substitute matched string
re.split()	Split string on pattern
re.compile()	Compile regex with optional flags
re.escape()	Escape regex special characters

---

🚩 Regex Flags

Flag	Description
re.IGNORECASE	Case-insensitive matching
re.MULTILINE	^ and $ match line boundaries
re.DOTALL	. matches newline too
re.VERBOSE	Allows whitespace and comments in pattern

---

💡 Tips for Readable Regex

• Use re.compile() for clarity and reuse.

• Label parts of your regex with (?P<name>...) to make results self-descriptive.

• Use re.escape() when searching for literal strings that may include regex characters:

term = "Test(123)"
escaped = re.escape(term)
pattern = re.compile(escaped)

---

✅ Conclusion

By combining PDF parsing, regex with named groups, and JSON Schema validation, you can automate test data extraction and verification cleanly and reliably.

Regex can be intimidating at first, but with practice and smart organization — especially using named groups and clear structure — it becomes a powerful ally in any data extraction task.