Robust JSON Serialization in Python with Custom Types
Learn to build a custom JSON encoder in Python that handles datetime, set, Path, and Enum types. Type markers keep serialized data recoverable.
In this article, you will learn how to build a robust JSON serialization pipeline in Python that handles custom data types, graceful error recovery, and schema migrations.
Topics we will cover include:
- How to define a custom JSON encoder to handle Python types like
datetime,set,Path, andEnum. - How to write a custom decoder function that reconstructs those original Python objects from a serialized JSON string, including graceful handling of malformed data.
- How to manage schema changes over time using a factory function that applies defaults and migrations during deserialization.

Introduction and Motivation
Anyone fairly experienced in constructing machine learning pipelines might have noticed at some stage that only a small percentage of their time is dedicated to writing the model architecture. What do they spend the rest of that time on? Data plumbing: the process of cleaning, preprocessing, and transforming data to make it look perfectly digestible for downstream models.
This article tackles a hidden payoff often discovered along the machine learning engineering journey: the constant need to move structured datasets between files, processes, microservices, and APIs. Throughout these processes, you need a solution to serialize and deserialize data in a reliable and robust fashion.
We will explore some practical strategies for serializing custom types in a way that enables parsing them back securely, as well as managing data schema changes and dealing with malformed data — all without necessarily crashing the whole pipeline!
Before going deep into the three-step solution, let’s look at the root problem: JSON limitations when encountering certain data types like datetime, set, Path, or Enum. Suppose this is the configuration for a machine learning experiment:
from datetime import datetime
from enum import Enum
from pathlib import Path
import json
class ModelType(Enum):
RANDOM_FOREST = "rf"
NEURAL_NET = "nn"
experiment_config = {
"experiment_id": 1042,
"model_type": ModelType.NEURAL_NET,
"features": {"age", "income", "location"}, # A set!
"data_path": Path("/data/v2/train.csv"),
"timestamp": datetime.now(),
"nested_meta": {
"retries": 3
}
}
The moment you try to serialize this experiment configuration using json.dumps(experiment_config), you will immediately come across an error like this:
TypeError: Object of type ModelType is not JSON serializable
These kinds of errors are what we are about to fix by defining a robust pipeline for encoding and decoding JSON objects with custom types.
1. Defining a Custom Encoder
First, we will define a custom encoder class that inherits from json.JSONEncoder. This is crucial to be able to “customize” the existing json.dumps() with the extra logic we will implement in our class to handle custom data types that JSON cannot handle by default.
Besides subclassing json.JSONEncoder, we also need to override the default() method. This is where, one by one, we will define the custom approach to handle each specific data type we might need to work with, using conditionals in the form if isinstance(obj, DATA_TYPE):, replacing DATA_TYPE with a specific data type or class name, as shown below:
class RobustEncoder(json.JSONEncoder):
def default(self, obj):
# Handling Datetimes
if isinstance(obj, datetime):
return {"__type__": "datetime", "value": obj.isoformat()}
# Handling Sets (convert to list for JSON)
if isinstance(obj, set):
return {"__type__": "set", "value": list(obj)}
# Handling Paths (convert to string)
if isinstance(obj, Path):
return {"__type__": "path", "value": str(obj)}
# Handling Enums (extract the underlying value)
if isinstance(obj, Enum):
return obj.value
# Letting the base class handle everything else (or raise TypeError)
return super().default(obj)
# Serializing our config object
json_string = json.dumps(experiment_config, cls=RobustEncoder, indent=2)
print(json_string)
One more important detail: the extra trick for robust serialization in the above code is attaching the __type__ marker to objects that might lose their original typing when they are turned into JSON primitives. This occurs, for instance, when converting Python sets into lists.
Notice how we use cls=RobustEncoder in this new call to json.dumps() to use our custom encoder.
Result:
{
"experiment_id": 1042,
"model_type": "nn",
"features": {
"__type__": "set",
"value": [
"age",
"income",
"location"
]
},
"data_path": {
"__type__": "path",
"value": "/data/v2/train.csv"
},
"timestamp": {
"__type__": "datetime",
"value": "2025-01-15T10:30:00.123456"
},
"nested_meta": {
"retries": 3
}
}
The configuration now serializes cleanly, and the __type__ markers on the set, path, and datetime values preserve the information needed to rebuild those objects later. With this encoder in place, the next steps are a custom decoder that reconstructs the original Python types and a schema-migration layer that keeps older serialized data compatible as your configurations evolve.