Avro format in python

To Nha Notes | June 27, 2023, 8:34 a.m.

Installation

$ python3 -m pip install avro

Defining a schema

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

schema = avro.schema.parse(open("user.avsc", "rb").read())

writer = DataFileWriter(open("users.avro", "wb"), DatumWriter(), schema)
writer.append({"name": "Alyssa", "favorite_number": 256})
writer.append({"name": "Ben", "favorite_number": 7, "favorite_color": "red"})
writer.close()

reader = DataFileReader(open("users.avro", "rb"), DatumReader())
for user in reader:
    print user
reader.close()

References

https://avro.apache.org/docs/1.11.1/getting-started-python/#notice-for-python-3-users

https://mahaboob.medium.com/read-and-write-to-avro-format-file-with-schema-in-python-bad275e33d63

https://www.perfectlyrandom.org/2019/11/29/handling-avro-files-in-python/

https://github.com/bashamsc/Avro_Read_Write_Python/blob/master/avro_read_write_file_using_schema.py