Enhancing JSON Serialization In Python With Typing
--
Maintaining valid types when deserializing JSON with Python is hard. This article discusses an approach using typing. You can find the package on GitHub.
The Problem
Here we have a python dict with a mix of data types.
from datetime import datetime
from decimal import Decimal
dct = {
'some_text': 'Hello, World!',
'some_date': datetime.fromisoformat('2020-01-29T12:56:13'),
'some_int': 42,
'some_float': 3.14,
'some_decimal': Decimal("2.414"),
'some_list': [
{
'other_text': 'Hello, World!',
'other_date': datetime.fromisoformat('2020-01-29T12:56:13'),
'other_decimal': Decimal("2.414")
},
{
'other_text': 'Hello, World!',
'other_date': datetime.fromisoformat('2020-01-29T12:56:13'),
'other_decimal': Decimal('2.414')
}
]
}Because the dict contains datetime and Decimal this won’t serialize without some extra work:
import json
try:
text = json.dumps(dct)
except Exception as error:
print(error)
Object of type datetime is not JSON serializableTo support the serialization of these types we need to provide an encoder.
class MyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return obj.isoformat()
if isinstance(obj, Decimal):
return float(obj)This is quite nice. We simply check to see if the object is of a particular type and return a JSON compatible object. For the date this is a string (although we’d prefer it to be in proper ISO 8601 format), and the Decimal becomes a float. Now our serialization works:
text = json.dumps(dct, cls=MyEncoder, indent=4)
print(text){
"some_text": "Hello, World!",
"some_date": "2020-01-29T12:56:13",
"some_int": 42,
"some_float": 3.14,
"some_decimal": 2.414,
"some_list": [
{
"other_text": "Hello, World!",
"other_date": "2020-01-29T12:56:13",
"other_decimal": 2.414
},
{
"other_text": "Hello, World!",
"other_date": "2020-01-29T12:56:13",
"other_decimal": 2.414
}
]
}
Now lets deserialize our text.
obj1 = json.loads(text)
print(obj1)
{
'some_text': 'Hello, World!',
'some_date': '2020-01-29T12:56:13',
'some_int': 42,
'some_float': 3.14,
'some_decimal': 2.414,
'some_list': [
{
'other_text': 'Hello, World!',
'other_date': '2020-01-29T12:56:13',
'other_decimal': 2.414
},
{
'other_text': 'Hello, World!',
'other_date': '2020-01-29T12:56:13',
'other_decimal': 2.414
}
]
}
print(dct == obj1)
FalseObviously we haven’t provided any information about the dates and decimals, so we just get back the standard JSON types. To do this we need to provide an object hook function. The object hook takes in any dictionary the JSON parser finds and changes it in any way it sees fit.
def object_hook(dct):
for key, value in dct.items():
try:
dct[key] = datetime.fromisoformat(value)
except:
pass
return dctDetecting the desired type is obviously a problem. Here I “detected” the type by trying to convert every entry. I could used a regular expression to check first, but I still have to check everything.
However my main problem is that there is no way of telling between a float and a decimal. Rob sad :(
The deserialization now looks like this.
obj2 = json.loads(text, object_hook=object_hook)
print(obj2)
{
'some_text': 'Hello, World!',
'some_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'some_int': 42,
'some_float': 3.14,
'some_decimal': 2.414,
'some_list': [
{
'other_text': 'Hello, World!',
'other_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'other_decimal': 2.414
},
{
'other_text': 'Hello, World!',
'other_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'other_decimal': 2.414
}
]
}
print(dct == obj2)
FalseWe could put in some business logic to use the key names, but this would require a custom decoder for each data set. We could put some type information in the JSON, but we may not be in control of its generation, and anyway that looks ugly.
Type Annotations
With type annotations we should be able to solve this problem. The data structure we are streaming can be “typed” by using a TypedDict. in Python 3.7 this can be found in the typing_extensions package, while in 3.8 it is standard.
from typing import List
try:
from typing import TypedDict
except:
from typing_extensions import TypedDict
class InnerDict(TypedDict):
other_text: str
other_date: datetime
other_decimal: Decimal
class OuterDict(TypedDict):
some_text: str
some_date: datetime
some_int: int
some_float: float
some_decimal: Decimal
some_list: List[InnerDict]Using the jetblack-serializationpackage we can use these type annotations. It relies heavily on the typing_inspect package.
Lets see what happens when we serialize our data.
from jetblack_serialization.config import SerializerConfig
from jetblack_serialization.json import deserialize, serialize
from stringcase import camelcase, snakecase
config = SerializerConfig(camelcase, snakecase, pretty_print=True)
text = serialize(dct, OuterDict, config)
print(text)
{
"someText": "Hello, World!",
"someDate": "2020-01-29T12:56:13.00Z",
"someInt": 42,
"someFloat": 3.14,
"someDecimal": 2.414,
"someList": [
{
"otherText": "Hello, World!",
"otherDate": "2020-01-29T12:56:13.00Z",
"otherDecimal": 2.414
},
{
"otherText": "Hello, World!",
"otherDate": "2020-01-29T12:56:13.00Z",
"otherDecimal": 2.414
}
]
}First it’s camel-cased the keys! The SerializerConfig takes a key serializer and deserializer (using the stringcase package). We can see the date got formatted nicely (the config option also allows for value serializers).
Now lets see how the JSON is deserialized.
obj = deserialize(text, OuterDict, config)
print(obj)
{
'some_text': 'Hello, World!',
'some_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'some_int': 42,
'some_float': 3.14,
'some_decimal': Decimal('2.414'),
'some_list': [
{
'other_text': 'Hello, World!',
'other_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'other_decimal': Decimal('2.414')
},
{
'other_text': 'Hello, World!',
'other_date': datetime.datetime(2020, 1, 29, 12, 56, 13),
'other_decimal': Decimal('2.414')
}
]
}The keys are now snake-case and the types have been correctly identified, even the decimals. A quick check shows a successful round trip.
print(dct == obj)
TrueRob happy :)
And just for fun, we can serialize into XML.
from typing_extensions import Annotated
from jetblack_serialization.xml import serialize as serialize_xml, XMLEntity
from stringcase import pascalcase
xml_config = SerializerConfig(pascalcase, snakecase, pretty_print=True)
text = serialize_xml(dct, Annotated[OuterDict, XMLEntity('Dict')], xml_config)
print(text)
<Dict>
<SomeText>Hello, World!</SomeText>
<SomeDate>2020-01-29T12:56:13.00Z</SomeDate>
<SomeInt>42</SomeInt>
<SomeFloat>3.14</SomeFloat>
<SomeDecimal>2.414</SomeDecimal>
<SomeList>
<OtherText>Hello, World!</OtherText>
<OtherDate>2020-01-29T12:56:13.00Z</OtherDate>
<OtherDecimal>2.414</OtherDecimal>
</SomeList>
<SomeList>
<OtherText>Hello, World!</OtherText>
<OtherDate>2020-01-29T12:56:13.00Z</OtherDate>
<OtherDecimal>2.414</OtherDecimal>
</SomeList>
</Dict>As XML requires a tag to wrap the object we needed to put in some extra information which was achieved with the Annotated tag available through the typing extensions package and described in PEP 593.
I’ve been using this with a REST framework which uses typing to automatically bind variables to reduce the effort to produce the API. You can find that code here.
