Logo OnlineCBT
PYTHON Tutorial
PYTHON · LESSON 43

Regular Expressions

Search, validate and transform patterned text responsibly.

Level: Beginner to Advanced
Duration: ~60 Mins Deep-Dive
Updated: 26 Jul 2026

Lesson 43: Regular Expressions

1. Introduction to Advanced Pattern Matching & Text Parsing

Welcome to Module 6 of our Python Mastery Series: Text Processing, Pattern Matching and Advanced Data Parsing! Up to this point, our text manipulation relies primarily on built-in string methods such as .find(), .split(), .startswith(), and .replace().

While standard string methods work well for exact character matches, enterprise software engineering frequently deals with unstructured text payloads containing complex patterns—such as parsing log files for IP addresses, validating user email formats, extracting transaction IDs from raw API webhooks, or redacting sensitive credit card numbers. Attempting to parse complex rules using standard string methods results in fragile, deeply-nested string splitting code.

To solve this challenge, computer science provides Regular Expressions (RegEx). A Regular Expression is a specialized, compact domain-specific language used to describe search patterns within strings.

In this lesson, we will master Python's standard `re` module, explore metacharacters, character shorthand classes (`\d`, `\w`, `\s`), anchors, quantifiers, capturing groups, named groups (`(?P...)`), pattern substitution (`re.sub()`), and integrate regex patterns seamlessly with Structural Pattern Matching (`match-case`).


2. RegEx Metacharacters & Character Shorthands Cheat Sheet

1. Core Metacharacters & Anchors

  • .
  • Wildcard Dot
  • Matches **any single character** except newline (\n).
  • ^
  • Start Anchor
  • Matches the **start of a string** or line.
  • $
  • End Anchor
  • Matches the **end of a string** or line.
  • [abc]
  • Character Set
  • Matches any **single character** inside the square brackets.
  • [^abc]
  • Negated Set
  • Matches any **single character NOT** inside the square brackets.
  • a | b
  • Alternation (OR)
  • Matches pattern a OR pattern b.
Symbol Pattern Name Matching Behavior Description

2. Character Shorthand Classes

  • \d
  • [0-9]
  • Matches any single **Decimal Digit**.
  • \D
  • [^0-9]
  • Matches any single **Non-Digit character**.
  • \w
  • [a-zA-Z0-9_]
  • Matches any single **Word Character** (alphanumeric + underscore).
  • \W
  • [^a-zA-Z0-9_]
  • Matches any single **Non-Word character** (punctuation, spaces).
  • \s
  • [ \t\n\r\f\v]
  • Matches any single **Whitespace character** (space, tab, newline).
  • \S
  • [^ \t\n\r\f\v]
  • Matches any single **Non-Whitespace character**.
Shorthand Full Class Equivalent Matching Behavior Description

3. Quantifiers (Repetition Controls)

  • *
  • 0 or more times
  • Greedy match (matches as much as possible).
  • +
  • 1 or more times
  • Greedy match (requires at least one occurrence).
  • ?
  • 0 or 1 time
  • Makes the preceding token optional.
  • {n}
  • Exactly n times
  • Matches exact integer repetition count.
  • {n,m}
  • Between n and m times
  • Matches bounded repetition range.
  • *? / +?
  • Non-Greedy (Lazy)
  • Appended to quantifiers to match as **few characters as possible**.
Quantifier Repetition Count Matching Strategy Behavior
The Raw String Prefix Requirement (`r"..."`): ALWAYS prefix Python regular expression strings with an r (e.g., r"\d{3}-\d{4}")! In Python raw strings, backslashes are treated as literal characters rather than Python string escape sequences, preventing backslash escape syntax conflicts.

3. The Standard `re` Module Functions

Python's built-in re module provides five fundamental search and extraction functions:

  • re.search(pattern, string): Scans the string and returns a **Match object** for the FIRST match found anywhere in text. Returns None if no match exists.
  • re.match(pattern, string): Checks if the pattern matches strictly starting from the **beginning of the string**.
  • re.findall(pattern, string): Returns a **List of strings** containing ALL non-overlapping matches across the text.
  • re.finditer(pattern, string): Returns an **Iterator yielding Match objects** lazily, ideal for low-memory processing of large log streams.
  • re.sub(pattern, replacement, string): Replaces matched patterns with replacement strings (essential for data sanitization!).
REGEX_CORE_FUNCTIONS_DEMO.PY
import re

raw_log_payload = """
2026-07-26 19:45:00 [INFO] User ID #USR_1001 logged in from IP 192.168.1.45
2026-07-26 19:45:02 [ERROR] Transaction TX_9901 failed for User ID #USR_1002 from IP 10.0.0.12
"""

# 1. re.search(): Find first IP Address match
ip_pattern = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"
search_result = re.search(ip_pattern, raw_log_payload)

print("=== 1. re.search() First Match Inspection ===")
if search_result:
    print("Matched IP String :", search_result.group())
    print("Match Index Span  :", search_result.span())

# 2. re.findall(): Extract ALL User IDs
user_pattern = r"USR_\d{4}"
all_users = re.findall(user_pattern, raw_log_payload)

print("\n=== 2. re.findall() Full Extraction ===")
print("Extracted User IDs List:", all_users)

# 3. re.sub(): Redacting IP Addresses for Data Privacy
sanitized_log = re.sub(ip_pattern, "[REDACTED_IP]", raw_log_payload)

print("\n=== 3. re.sub() Data Masking ===")
print(sanitized_log.strip())
OUTPUT
=== 1. re.search() First Match Inspection ===
Matched IP String : 192.168.1.45
Match Index Span  : (62, 74)

=== 2. re.findall() Full Extraction ===
Extracted User IDs List: ['USR_1001', 'USR_1002']

=== 3. re.sub() Data Masking ===
2026-07-26 19:45:00 [INFO] User ID #USR_1001 logged in from IP [REDACTED_IP]
2026-07-26 19:45:02 [ERROR] Transaction TX_9901 failed for User ID #USR_1002 from IP [REDACTED_IP]

4. Advanced Grouping: Capturing & Named Groups

Regular Expression Parentheses (...) define Capturing Groups, allowing you to isolate and extract specific sub-components from a complex text match.

1. Named Capturing Groups (`(?Ppattern)`)

Instead of relying on index numbers (e.g., match.group(1), match.group(2)), professional RegEx uses **Named Groups** to assign semantic keys directly to extracted components!

NAMED_GROUPS_PARSER_DEMO.PY
import re

# RegEx defining Named Groups for an Email Address: user, domain, tld
email_regex = re.compile(
    r"^(?P[a-zA-Z0-9._%+-]+)@(?P[a-zA-Z0-9.-]+)\.(?P[a-zA-Z]{2,})$"
)

sample_email = "alice.developer@apex-tech.com"
match_obj = email_regex.match(sample_email)

print("=== Named Group Extraction Inspection ===")
if match_obj:
    # 1. Extract named groups as a Python dictionary via .groupdict()
    extracted_dict = match_obj.groupdict()
    print("Extracted Dictionary:", extracted_dict)
    
    # 2. Access individual named fields using .group("key")
    print(f"Username : {match_obj.group('username')}")
    print(f"Domain   : {match_obj.group('domain')}")
    print(f"TLD      : {match_obj.group('tld')}")
OUTPUT
=== Named Group Extraction Inspection ===
Extracted Dictionary: {'username': 'alice.developer', 'domain': 'apex-tech', 'tld': 'com'}
Username : alice.developer
Domain   : apex-tech
TLD      : com

5. Performance Optimization: Pre-Compiling RegEx (`re.compile`)

When processing thousands of text lines inside loops, calling raw functions like re.search(pattern, text) repeatedly forces Python to re-parse and compile the regular expression pattern string into internal bytecode on every single iteration.

The Compilation Rule (`re.compile`): ALWAYS pre-compile regular expression patterns using pattern = re.compile(r"...") outside iteration loops! Pre-compiling converts the RegEx string into an optimized Pattern object once, accelerating execution speed across subsequent loop cycles.
REGEX_COMPILE_PERFORMANCE.PY
import re

# Pre-compiling pattern object with re.IGNORECASE flag
TRANSACTION_PATTERN = re.compile(r"TXN_(?P\d{6})", re.IGNORECASE)

stream_lines = [
    "PAYMENT CONFIRMED: TXN_880192 AMOUNT: $450.00",
    "REFUND PROCESSED: txn_102938 AMOUNT: $50.00",
    "INVALID RECORD NO PATTERN HERE"
]

print("=== Pre-Compiled Stream Iterator Extraction ===")
for idx, line in enumerate(stream_lines, start=1):
    match = TRANSACTION_PATTERN.search(line)
    if match:
        print(f"Line #{idx}: Found Transaction ID -> {match.group('tx_id')}")
    else:
        print(f"Line #{idx}: No transaction pattern detected.")
OUTPUT
=== Pre-Compiled Stream Iterator Extraction ===
Line #1: Found Transaction ID -> 880192
Line #2: Found Transaction ID -> 102938
Line #3: No transaction pattern detected.

6. Integrating RegEx Extraction with `match-case` Pattern Matching

Python 3.10+ `match-case` Structural Pattern Matching can inspect RegEx match objects directly when combined with conditional pattern guards or dynamic string tuple dispatchers.

MATCH_CASE_REGEX_DISPATCHER.PY
import re

# Compiled RegEx Patterns
EMAIL_REGEX = re.compile(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")
PHONE_REGEX = re.compile(r"^\+?\d{1,3}[-.\s]?\d{10}$")
CARD_REGEX = re.compile(r"^\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}$")

def classify_contact_input(raw_input_str: str) -> str:
    """
    Classifies raw string inputs using RegEx pattern match guards inside match-case.
    Demonstrates combining RegEx validation with structural pattern dispatching.
    """
    clean_str = raw_input_str.strip()

    match clean_str:
        case s if EMAIL_REGEX.match(s):
            return f"CLASSIFIED [EMAIL]: '{s}' validated successfully."

        case s if PHONE_REGEX.match(s):
            return f"CLASSIFIED [PHONE]: '{s}' validated as international phone format."

        case s if CARD_REGEX.match(s):
            masked_card = f"****-****-****-{s[-4:]}"
            return f"CLASSIFIED [CREDIT_CARD]: Card {masked_card} captured."

        case _:
            return f"CLASSIFIED [UNKNOWN_FORMAT]: String '{clean_str}' failed format validation!"

# Testing Pattern Dispatcher
print(classify_contact_input("admin.support@company.org"))
print(classify_contact_input("+91-9876543210"))
print(classify_contact_input("4532-1102-8890-4321"))
print(classify_contact_input("INVALID_DATA_PAYLOAD"))
OUTPUT
CLASSIFIED [EMAIL]: 'admin.support@company.org' validated successfully.
CLASSIFIED [PHONE]: '+91-9876543210' validated as international phone format.
CLASSIFIED [CREDIT_CARD]: Card ****-****-****-4321 captured.
CLASSIFIED [UNKNOWN_FORMAT]: String 'INVALID_DATA_PAYLOAD' failed format validation!

7. Frequently Asked Interview Questions with Answers

Q1: What is the difference between `re.search()` and `re.match()` in Python?
Answer: re.match() checks for a pattern match strictly starting at the **very beginning of the string**. re.search() scans through the **entire string** and returns a match object for the first occurrence found anywhere in the text.
Q2: Why should regular expressions always be defined as Raw Strings (e.g., `r"\d+\s+\w+"`)?
Answer: Standard Python strings interpret backslashes as string escape characters (e.g., \n for newline, \t for tab). Prefixing with r creates a raw string literal where backslashes pass through to the RegEx engine directly without Python string escaping interference.
Q3: How do Greedy vs Non-Greedy (Lazy) quantifiers behave in RegEx?
Answer: Standard quantifiers (*, +) are **Greedy**: they consume as much matching text as possible. Appending a question mark (*?, +?) makes them **Non-Greedy (Lazy)**, causing the engine to stop matching at the earliest possible character that satisfies the pattern.
Q4: What are Named Capturing Groups, and how do you define them?
Answer: Named Capturing Groups allow assigning a string key identifier to a parentheses group using the syntax (?Ppattern). Extracted groups can then be accessed as a dictionary via match.groupdict() or individually using match.group("group_name").
Q5: Why is pre-compiling RegEx with `re.compile()` recommended inside performance-critical loops?
Answer: Pre-compiling converts the string pattern into compiled RegEx bytecode once. Reusing the compiled pattern object inside execution loops avoids repeating the pattern parsing and compilation overhead on every iteration.

8. Homework & Practical Assignments

Task 1: Log File IP and Timestamp Extractor

Create a script named log_regex_task.py inside your lesson_43 folder:

  • Import re and define a log string containing multiple entries with timestamps, severity levels, and IP addresses.
  • Pre-compile a RegEx pattern with Named Groups: (?P\d{4}-\d{2}-\d{2}), (?PINFO|WARN|ERROR), (?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).
  • Iterate through lines using re.finditer() and display extracted dictionary records via .groupdict().

Task 2: Pattern-Matched Credit Card Masker and Validator

Create a script named card_masker_task.py:

  • Define a function sanitize_credit_card(card_str: str) -> str.
  • Use re.sub() to replace all spaces and hyphens with empty strings.
  • Write a match-case dispatcher to validate card digits length using re.match(r"^\d{16}$", clean_card) and return a masked string format: "****-****-****-1234".
  • Test function with valid formatted cards, unformatted raw digits, and invalid inputs.

Task 3: Master Review Capstone Project — Production Enterprise RegEx Parsing & Telemetry OS (`regex_parser_os.py`)

Create a script named regex_parser_os.py inside your lesson_43 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 43**.

Project Architectural Requirements Specification:

  1. Regular Expressions & Pattern Extraction (Lesson 43):
    • Pre-compile domain RegEx patterns with Named Groups for parsing emails, phone numbers, transaction IDs, and IP addresses.
    • Perform text sanitization and data masking using re.sub().
    • Use re.finditer() for low-memory lazy extraction streams.
  2. Type Hints, Generics & Static Safety (Lesson 42):
    • Apply explicit modern type annotations (X | Y, list[str], dict[str, Any]) throughout all functions and classes.
    • Define domain TypeAlias definitions and Generic Response Envelopes.
  3. Metaprogramming, Closures & Decorators (Lessons 40-41):
    • Build function factories generating customized RegEx validation closures.
    • Build a 3-level configurable decorator audit_pattern_execution(level="INFO") with @functools.wraps.
  4. Iterators, Generators & Functional Processing (Lesson 39):
    • Implement generator functions streaming log file lines lazily with $O(1)$ memory footprint.
  5. Full OOP Architecture (Lessons 32-38):
    • Define abstract base class BasePatternParser(ABC) and concrete Data Classes ParsedLogRecord and SecurityAlert.
    • Build composite manager RegExParserOS composing storage drivers and custom context managers.
  6. Observability & Fault Tolerance (Lessons 29-31):
    • Set up multi-handler logging to console and regex_os.log file.
    • Incorporate developer assertions (assert) verifying pattern match bounds.
    • Define custom exception hierarchy (RegexParsingError, PatternMismatchError).
  7. Context Managers & Persistence Layer (Lessons 27-30):
    • Use custom class-based context manager ParserVaultLock to manage persistent database lock files.
    • Persist JSON records to parsed_db.json and export CSV audit reports to regex_audit.csv using pathlib.Path.
  8. Nested Collections & Comprehensions (Lessons 21-26):
    • Maintain in-memory registry dictionaries mapping IDs to Data Class Instances.
    • Use List and Dictionary Comprehensions to clean and transform records.
  9. Advanced Function Parameters & Scope (Lessons 15-17):
    • Structure logic into pure modular functions with type hints, docstrings, early return Guard Clauses, and *args / **kwargs logging.
  10. Indefinite & Definite Loops (Lessons 13-14):
    • Run the interactive CLI interface inside a continuous while True menu loop.
    • Iterate through output reports using for loops with enumerate() and .items().
  11. Pattern Matching CLI Dispatcher (Lesson 12):
    • Process user CLI command tokens inside a match-case block with RegEx pattern guards:
      • case ["PARSE", "EMAIL", raw_email] → Parse email using named groups RegEx and persist.
      • case ["MASK", "CARD", raw_card] → Redact card number using re.sub() and output.
      • case ["STREAM", filename] → Stream file lines lazily and apply RegEx pattern extraction.
      • case ["EXPORT", "CSV"] → Export parsed database state to CSV.
      • case ["EXIT" | "QUIT"] → Terminate session safely using a sentinel flag.
      • case _ → Output command error message.
  12. Conditionals, Formatting & Foundations (Lessons 1-11):
    • Sanitize all inputs using .strip() and .upper().
    • Format tabular reports using f-strings with field width alignment specifiers (:<15, :>10) and clear visual borders.

9. File & Workspace Directory Structure

Standard Course Workspace Layout:

Ensure all exercise files are stored inside their corresponding lesson directories:

python_mastery_course/
│
├── lesson_01/ ... lesson_42/
│
└── lesson_43/
    ├── regex_core_functions_demo.py
    ├── named_groups_parser_demo.py
    ├── regex_compile_performance.py
    ├── match_case_regex_dispatcher.py
    ├── log_regex_task.py            <-- (Task 1)
    ├── card_masker_task.py          <-- (Task 2)
    └── regex_parser_os.py           <-- (Task 3: Master Review Capstone)
        

10. What We Will Learn Next

Next Up: Lesson 44 — Dates, Times and Time Zones

Now that you master Regular Expressions, text parsing, and data extraction, we will explore managing temporal data cleanly across global enterprise systems!

In the next lesson, we will cover:

  • Working with Python's standard datetime module (date, time, datetime, timedelta).
  • Parsing and Formatting dates using strptime() and strftime() specifiers.
  • Understanding Naive vs Aware datetime objects.
  • Managing Time Zones using Python 3.9+ zoneinfo module.
  • Combining datetime calculations with match-case pattern dispatchers.

📝 Live Lesson Practice

HTML/CSS JavaScript Python C++ C PHP
⌨️ Practice Inputs (लाइव इनपुट भरें) (खाली होने पर RED, भरने पर GREEN underline)
💻 Code Editor (Monaco VS Code Engine)
👀 Live Preview
Address Contact

+91 7877547686

E-mail

onlinecbtportal@gmail.com

Helpline Number

+91 7877547686


Click To Download
Get it on Google Play

ऐप डाउनलोड करने
के लिए Google Play पर
उपलब्ध है

WhatsApp Chat