1. Introduction to Industry Standard Structured Data Formats
In Lesson 27, we mastered reading and writing unstructured raw text files using Python's open() function and pathlib.Path. While unstructured text files work well for simple logging, modern enterprise systems require Structured Serialization Formats to exchange complex data between databases, spreadsheets, backend web servers, and third-party REST APIs.
The two most widely adopted data exchange standards across software engineering are:
- CSV (Comma-Separated Values): A tabular, plain-text format where records represent rows and commas delimit column attributes. Essential for spreadsheet exports (Excel, Google Sheets) and data science pipelines (Pandas).
- JSON (JavaScript Object Notation): A lightweight, human-readable, language-independent data format based on key-value mappings and ordered sequences. Standard for web APIs, microservice payloads, and configuration files.
In this lesson, we will explore Python's standard library modules: `csv` and `json`. We will learn how to read, write, parse, serialize, and deserialize complex data structures seamlessly, while orchestrating structured payloads using Structural Pattern Matching (`match-case`).
2. Working with Tabular CSV Data (`csv` Module)
Parsing CSV files manually using string .split(",") is dangerous because real-world CSV fields often contain escaped commas inside quoted strings (e.g., "New York, NY", 10001). Python's built-in csv module handles quoted fields, custom delimiters, and newlines transparently.
1. Reading CSV Files (`csv.reader` vs `csv.DictReader`)
csv.reader(file): Reads each row as a List of Strings indexed by position (e.g.,row[0],row[1]).csv.DictReader(file): Reads each row as a Dictionary where keys are extracted automatically from the CSV header row (e.g.,row["name"]).
newline="" inside the open() function! This prevents the csv module from adding extra blank lines or mangling carriage returns across Windows and Unix platforms.
import csv
from pathlib import Path
csv_path = Path.cwd() / "sample_employees.csv"
# 1. Writing sample CSV file with headers
data_rows = [
["emp_id", "name", "department", "salary"],
["101", "Alice Smith", "Engineering", "95000.00"],
["102", "Bob Jones", "Marketing", "72000.00"],
["103", "Charlie Brown", "Engineering", "105000.00"]
]
with open(csv_path, mode="w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerows(data_rows)
print("=== Reading CSV using csv.DictReader (Pythonic Standard) ===")
with open(csv_path, mode="r", encoding="utf-8", newline="") as f:
reader = csv.DictReader(f)
for row in reader:
# Values are automatically mapped to header column names as dictionary keys
emp_name = row["name"]
dept = row["department"]
salary = float(row["salary"])
print(f"ID #{row['emp_id']} | Name: {emp_name:<15} | Dept: {dept:<12} | Salary: ${salary:,.2f}")
=== Reading CSV using csv.DictReader (Pythonic Standard) === ID #101 | Name: Alice Smith | Dept: Engineering | Salary: $95,000.00 ID #102 | Name: Bob Jones | Dept: Marketing | Salary: $72,000.00 ID #103 | Name: Charlie Brown | Dept: Engineering | Salary: $105,000.00
2. Writing CSV Files (`csv.writer` vs `csv.DictWriter`)
Similarly, csv.DictWriter allows exporting lists of Python dictionaries directly to disk by defining a fieldnames header schema.
import csv
from pathlib import Path
out_path = Path.cwd() / "exported_inventory.csv"
products = [
{"product_id": "P01", "title": "Wireless Mouse", "price": 25.50, "stock": 120},
{"product_id": "P02", "title": "Mechanical Keyboard", "price": 89.99, "stock": 45},
{"product_id": "P03", "title": "4K Monitor", "price": 320.00, "stock": 15}
]
headers = ["product_id", "title", "price", "stock"]
with open(out_path, mode="w", encoding="utf-8", newline="") as f:
writer = csv.DictWriter(f, fieldnames=headers)
# 1. Write the header row
writer.writeheader()
# 2. Write dictionary records batch
writer.writerows(products)
print(f"Successfully exported {len(products)} product records to: {out_path.name}")
Successfully exported 3 product records to: exported_inventory.csv
3. Working with Hierarchical JSON Data (`json` Module)
JSON (JavaScript Object Notation) represents complex multi-level data hierarchies natively. Python maps JSON types directly to native Python data structures:
object({"key": "val"})dict- Key-Value Mapping
array([1, 2, 3])list- Ordered Sequence
string("hello")str- Text Sequence
number (real)/number (int)float/int- Numerical Scalar
boolean(true/false)bool(True/False)- Boolean Literal
nullNone- Null Pointer Singleton
| JSON Data Type | Python Data Type Equivalent | Python Type Category |
|---|---|---|
4. The Four Core JSON Functions Matrix
The standard json module provides four fundamental functions. Understanding when to use string-based functions vs file-stream functions is essential:
- Serialization
(Python Object → JSON Format) json.dumps(obj)
("dump string")json.dump(obj, file_stream)
("dump file")- Deserialization
(JSON Format → Python Object) json.loads(json_str)
("load string")json.load(file_stream)
("load file")
| Target Operation Type | In-Memory String Function | Disk File Stream Function |
|---|---|---|
import json
from pathlib import Path
json_file_path = Path.cwd() / "system_config.json"
# Complex Python nested dictionary structure
app_config = {
"app_name": "Apex Enterprise Engine",
"version": "2.4.0",
"server": {
"host": "127.0.0.1",
"port": 8080,
"ssl_enabled": True
},
"active_modules": ["auth", "billing", "analytics"],
"max_connections": None
}
# 1. Serializing to formatted JSON String using json.dumps()
formatted_json_str = json.dumps(app_config, indent=4)
print("=== Formatted JSON String Output ===")
print(formatted_json_str)
# 2. Writing JSON directly to Disk File using json.dump()
with open(json_file_path, mode="w", encoding="utf-8") as f:
json.dump(app_config, f, indent=4)
print(f"\nConfig successfully saved to: {json_file_path.name}")
=== Formatted JSON String Output ===
{
"app_name": "Apex Enterprise Engine",
"version": "2.4.0",
"server": {
"host": "127.0.0.1",
"port": 8080,
"ssl_enabled": true
},
"active_modules": [
"auth",
"billing",
"analytics"
],
"max_connections": null
}
Config successfully saved to: system_config.json
import json
from pathlib import Path
json_file_path = Path.cwd() / "system_config.json"
# Reading and deserializing JSON file from Disk using json.load()
with open(json_file_path, mode="r", encoding="utf-8") as f:
loaded_config = json.load(f)
print("=== Deserialized Python Dictionary Object ===")
print(f"Loaded Object Type : {type(loaded_config)}")
print(f"App Title : {loaded_config['app_name']}")
print(f"Host Port Target : {loaded_config['server']['host']}:{loaded_config['server']['port']}")
print(f"First Active Module: {loaded_config['active_modules'][0]}")
=== Deserialized Python Dictionary Object === Loaded Object Type :App Title : Apex Enterprise Engine Host Port Target : 127.0.0.1:8080 First Active Module: auth
5. Integrating CSV & JSON Processing with `match-case` Pattern Matching
Building resilient data transformation pipelines involves combining csv and json modules with match-case structural pattern matching to route dynamic file payloads based on schema structure.
import csv
import json
from pathlib import Path
def process_data_payload(command_payload: list):
"""
Routes and transforms CSV/JSON data using Structural Pattern Matching.
Demonstrates handling structured file formats dynamically.
"""
match command_payload:
case ["EXPORT_JSON", filename, dict_data] if isinstance(dict_data, dict):
target_path = Path.cwd() / filename
with open(target_path, mode="w", encoding="utf-8") as f:
json.dump(dict_data, f, indent=2)
return f"SUCCESS: Exported JSON dictionary to '{filename}'."
case ["EXPORT_CSV", filename, [head_row, *data_rows]] if len(data_rows) > 0:
target_path = Path.cwd() / filename
with open(target_path, mode="w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(head_row)
writer.writerows(data_rows)
return f"SUCCESS: Exported CSV table with {len(data_rows)} rows to '{filename}'."
case ["PARSE_JSON_KEY", filename, target_key]:
target_path = Path.cwd() / filename
if not target_path.exists():
return f"ERROR: File '{filename}' does not exist!"
with open(target_path, mode="r", encoding="utf-8") as f:
payload = json.load(f)
extracted = payload.get(target_key, "KEY_NOT_FOUND")
return f"EXTRACTED [{target_key}]: {extracted}"
case _:
return "ERROR: Unrecognized structured payload command schema."
# Testing the dispatcher
sample_user = {"id": 881, "username": "alice_dev", "status": "ACTIVE"}
table_data = [["ID", "Role"], [101, "Admin"], [102, "Editor"]]
print(process_data_payload(["EXPORT_JSON", "user_881.json", sample_user]))
print(process_data_payload(["EXPORT_CSV", "roles.csv", table_data]))
print(process_data_payload(["PARSE_JSON_KEY", "user_881.json", "username"]))
SUCCESS: Exported JSON dictionary to 'user_881.json'. SUCCESS: Exported CSV table with 2 rows to 'roles.csv'. EXTRACTED [username]: alice_dev
6. Frequently Asked Interview Questions with Answers
json.dumps(obj) serializes a Python object into a formatted **JSON text string** in RAM. json.dump(obj, file_stream) serializes a Python object directly into an open **disk file stream**.
csv module handles newline translation internally according to dialect standards. Specifying newline="" prevents the underlying Operating System's file stream layer from inserting extra carriage returns (\r\n) or generating blank lines between rows on Windows platforms.
csv.reader returns rows as index-based lists (e.g., row[2]), making code fragile if CSV column positions change. csv.DictReader maps column headers to dictionary keys (e.g., row["email"]), making code self-documenting and resilient to column re-ordering.
dict, JSON arrays map to Python list, strings to str, numbers to int/float, booleans (true/false) to True/False, and JSON null maps to Python's None singleton.
TypeError: Object of type X is not JSON serializable. To serialize custom non-standard types, developers must provide a custom encoder function via the default argument in json.dumps() or convert types to serializable primitives (strings/lists) beforehand.
7. Homework & Practical Assignments
Task 1: CSV DictWriter Data Exporter
Create a script named csv_exporter_task.py inside your lesson_28 folder:
- Import
csvandPathfrompathlib. - Define a list of student dictionary records:
students = [ {"student_id": "S101", "name": "Alice", "gpa": 3.9, "grade": "A"}, {"student_id": "S102", "name": "Bob", "gpa": 3.2, "grade": "B"}, {"student_id": "S103", "name": "Charlie", "gpa": 3.7, "grade": "A"} ] - Use
csv.DictWriterwithnewline=""andencoding="utf-8"to write header and records tostudents_report.csv. - Re-open the CSV using
csv.DictReader, filter students withgpa >= 3.5using a List Comprehension, and print their names.
Task 2: JSON Configuration Reader & Formatter
Create a script named json_config_task.py:
- Define a nested server configuration dictionary containing app name, port, database host, and allowed IP sets.
- Convert the allowed IP set to a list to ensure JSON serializability.
- Save the dictionary to
server_settings.jsonusingjson.dump()withindent=4. - Read the file back using
json.load(), inspect nested properties using early return Guard Clauses, and print formatted configuration details.
Task 3: Master Review Capstone Project — Enterprise File Database Engine (`file_database_os.py`)
Create a script named file_database_os.py inside your lesson_28 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 28**.
Project Architectural Requirements Specification:
- Environment, Modules & Entry Guard (Lessons 18-20):
- Import standard library modules (
sys,datetime,csv,json) andPathfrompathlib. - Wrap main application execution inside an
if __name__ == "__main__":guard block.
- Import standard library modules (
- CSV & JSON Persistence Layer (Lessons 27-28):
- Maintain a storage directory
db_vault/usingpathlib.Path. - Store catalog databases on disk as JSON files (
catalog.json) usingjson.dump()andjson.load(). - Export transactional audit history tables as CSV files (
audit_export.csv) usingcsv.DictWriter.
- Maintain a storage directory
- Nested Collections & Data Modelling (Lesson 26):
- Maintain an in-memory active cache of product catalogs as multi-level nested dictionaries.
- Perform multi-key sorting on dataset records using
operator.itemgetteror custom lambda key tuples.
- Comprehensions & Core Data Structures (Lessons 21-25):
- Use List and Dictionary Comprehensions to clean, filter, and map record sets dynamically before file exports.
- Use
setalgebra (Union, Intersection, Difference) to evaluate access permission overlaps.
- Advanced Function Parameters & Scope (Lessons 15-17):
- Structure code into pure modular functions with type hints, docstrings, early return Guard Clauses, and
*args/**kwargslogging.
- Structure code into pure modular functions with type hints, docstrings, early return Guard Clauses, and
- Indefinite & Definite Loops (Lessons 13-14):
- Run the main application inside a continuous
while Trueinteractive CLI menu loop. - Iterate through record streams using
forloops withenumerate()and.items().
- Run the main application inside a continuous
- Pattern Matching CLI Dispatcher (Lesson 12):
- Process user CLI command tokens inside a
match-caseblock:case ["ADD", pid, name, price_str, stock_str]→ Parse inputs, update nested catalog dict, and persist tocatalog.json.case ["EXPORT", "CSV", filename]→ Export catalog dictionary list to CSV usingcsv.DictWriter.case ["LOAD", "JSON", filename]→ Read JSON database from disk and hydrate memory cache.case ["EXIT" | "QUIT"]→ Terminate session safely using a sentinel flag.case _→ Output command error message.
- Process user CLI command tokens inside a
- Conditionals, Formatting & Foundations (Lessons 1-11):
- Sanitize all string inputs using
.strip()and.upper(). - Format tabular reports using f-strings with field width alignment specifiers (
:<15,:>10) and currency formatting (:,.2f).
- Sanitize all string inputs using
8. File & Workspace Directory Structure
Standard Course Workspace Layout:
Ensure all exercise files are stored inside their corresponding lesson directories:
python_mastery_course/
│
├── lesson_01/ ... lesson_27/
│
└── lesson_28/
├── csv_reader_demo.py
├── csv_dictwriter_demo.py
├── json_serialization_demo.py
├── json_deserialization_demo.py
├── match_case_structured_dispatcher.py
├── csv_exporter_task.py <-- (Task 1)
├── json_config_task.py <-- (Task 2)
└── file_database_os.py <-- (Task 3: Master Review Capstone)
OnlineCBT