Serialbox Python

This section is concerned with the python3 interface of Serialbox. A set of examples can be found in examples/python/.

Building

The Python interface of Serialbox is built if the CMake variable SERIALBOX_ENABLE_PYTHON is ON (which is the default behaviour) and installed in install/python/serialbox. Serialbox requires atleast Python3.4 and relies on numpy.

To get started, add the path of the Serialbox module to the Python environment variable PYTHONPATH

$ export PYTHONPATH=<path-to-serialbox-installation>/python:$PYTHONPATH

To check if everything works:

$ export PYTHONPATH=<path-to-serialbox-installation>/python:$PYTHONPATH
$ python3 -c "from serialbox import __version__; print(__version__)"

which should print the current version string of the library.

Quick start - Writing

This section will show you how to write numpy.arrays using the Python interface of Serialbox. The written data set can be read by any language supported by Serialbox. To get started, import the serialbox module:

>>> import serialbox as ser

To serialize data, you have to create a Serializer object. The Serializer is initialized with a mode (OpenModeKind), a directory and a specific prefix, with which all the written files in the data set start. All files will be placed in directory and the directory will be created if necessary. For writing there are two modes: Write and Append: Write will erase all files of a previous run of a serializer the with same directory and prefix; Append will import all existing information and allow you to add more data.

To open a Serializer for writing in the current directory and prefix all files with field:

>>> serializer = ser.Serializer(ser.OpenModeKind.Write, ".", "field")

To serialize a field, you have to register the field within the Serializer (Serializer.register_field) and specify a specifc Savepoint at which the field will be written. Savepoints are used within the Serializer to discriminate fields at different points in time. Savepoints in the Serializer are unique and primarily identified by their name and further distinguished by their metainfo. Savepoints can thus have the same name as long as their metainfo differs. As a short example, we will serialize two numpy.arrays foo and bar at two different Savepoints (MySavepoint with time=1 and time=2).

We create the fields:

>>> import numpy as np
>>> foo = np.random.rand(10, 10)
>>> bar = np.random.rand(25, 25, 25)

Register the fields (FieldMetainfo) within the Serializer:

>>> foo_info = ser.FieldMetainfo(ser.TypeID.Float64, foo.shape)
>>> bar_info = ser.FieldMetainfo(ser.TypeID.Float64, bar.shape)
>>> serializer.register_field("foo", foo_info)
>>> serializer.register_field("bar", bar_info)
>>> serializer.fieldnames()
['foo', 'bar']

Write foo and bar at MySavepoint with time=1

>>> MySavepoint_t1 = ser.Savepoint("MySavepoint", {"time": 1})
>>> serializer.register_savepoint(MySavepoint_t1)
>>> serializer.write("foo", MySavepoint_t1, foo)
>>> serializer.write("bar", MySavepoint_t1, bar)

Perform a silly timestep:

>>> foo *= 2
>>> bar *= 2

and write the updated fields at MySavepoint with time=2

>>> MySavepoint_t2 = ser.Savepoint("MySavepoint", {"time": 2})
>>> serializer.register_savepoint(MySavepoint_t2)
>>> serializer.write("foo", MySavepoint_t2, foo)
>>> serializer.write("bar", MySavepoint_t2, bar)

The current directory will now contain the following files

.
├── ArchiveMetaData-field.json
├── MetaData-field.json
├── field_bar.dat
└── field_foo.dat

With field_foo.dat and field_bar.dat containing the actual data while the JSON files hold the meta-information.

Note: The Python interface is actually more powerful and you can omit all the calls to register_field and register_savepoint. The example above can thus be written as:

>>> import numpy as np
>>> foo = np.random.rand(10, 10)
>>> bar = np.random.rand(25, 25, 25)
>>> serializer.write("foo", ser.Savepoint("MySavepoint", {"time": 1}), foo) # implicitly register savepoint and field
>>> serializer.write("bar", ser.Savepoint("MySavepoint", {"time": 1}), bar)
>>> foo *= 2
>>> bar *= 2
>>> serializer.write("foo", ser.Savepoint("MySavepoint", {"time": 2}), foo)
>>> serializer.write("bar", ser.Savepoint("MySavepoint", {"time": 2}), bar)

Quick start - Reading

This section will show you how to read data from an existing Serialbox data set, written by any language supported by Serialbox. In this example, we will use the data set written in the Quick start - Writing section. To get started, import the serialbox module:

>>> import serialbox as ser

To access the data, you have to create a Serializer object. The Serializer is initialized with a mode (OpenModeKind), a directory and a specific prefix, with which all the files in the data set start. The directory has to contain the data in question and the prefix has to match the one of the data set (i.e the same as used in the writing).

To open a Serializer in the current directory for reading of fields matching field_*.dat:

>>> serializer = ser.Serializer(ser.OpenModeKind.Read, ".", "field")

In case of an error, a SerialboxError is thrown. To access a particular field, you have to specify a specific Savepoint of this field. Savepoints are used within the Serializer to discriminate fields at different points in time. Savepoints in the Serializer are unique and primarily identified by their name and further distinguished by their metainfo. To get a full list of all registered savepoint (in the order they were registered):

>>> serializer.savepoint_list()
[Savepoint MySavepoint {"time":1}, Savepoint MySavepoint {"time":2}]

For more advanced queries, see Serializer.savepoint. To get a list of all registered fields:

>>> serializer.fieldnames()
['foo', 'bar']

To read foo at Savepoint with time=1, you can use one of the read methods of the Serializer (Serializer.read, Serializer.read_slice, Serializer.read_async)

>>> savepoint = serializer.savepoint["MySavepoint"].time[1]
>>> foo = serializer.read("foo", savepoint)

foo is a newly allocated numpy.array containing the deserialized data.

Quick start - Visualization

To help visualizing the data serialbox2 contains a built-in Visualizer based on matplotlib . The visualizer is able to visualize any 3D numpy field and expects two arguments: a reference to the numpy field and the name of the plot. For exmaple, to visualize the field pp:

>>> from serialbox.visualizer import Visualizer
>>> Visualizer(pp, 'pp')

This will open a window displaying the data:

Serializer

class serialbox.Serializer(mode, directory, prefix, archive='Binary')

Serializer implementation of the Python Interface.

The serializer is attached to a specific directory and to a specific prefix, with which all files read and written will start. There are three modes to open a serializer: Read, Write and Append (see OpenModeKind). Read will give a read-only access to the serialized data; Write will erase all files of a previous run of a serializer with same directory and prefix; Append will import all existing information and allow the user to add more data.

It is possible to store multiple serializer in the same directory as long as they differ in their prefix.

Example:

>>> field = np.array([1,2,3])
>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> savepoint = Savepoint("savepoint")
>>> ser.write("field", savepoint, field)
>>> [f for f in os.listdir(".") if os.path.isfile(os.path.join(".", f))] # List files in current directory
['field_field.dat', 'ArchiveMetaData-field.json', 'MetaData-field.json']

__init__(mode, directory, prefix, archive='Binary')

Initialize the Serializer.

Parameters:

• mode (OpenModeKind) – Mode of the Serializer
• directory (str) – The directory where the files will be stored/read (will be created if necessary)
• prefix (str) – The prefix of the files
• archive (str) – String used to construct the Archive (see Archive.registered_archives)

Raises:

serialbox.SerialboxError – if initialization failed

mode

Return mode of the Serializer.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.mode
<OpenModeKind.Write: 1>

Returns:Mode of the Serializer Return type:OpenModeKind

prefix

Return the prefix of all filenames.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.prefix
'field'

Returns:prefix of all filenames Return type:str

directory

Return the directory of the Serializer.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.directory
'.'

Returns:directory of the Serializer Return type:str

update_meta_data()

Write meta-data to disk.

static enable()

Enable serialization.

Serialization is enabled by default, but it can be disabled either by setting the environment variable STELLA_SERIALIZATION_DISABLE to a positive value or by calling the function Serializer.disable. With this function you enable the serialization independently of the current environment.

The serialization can be only globally enabled or disabled. There is no way to enable or disable only a specific serializer.

static disable()

Disable serialization.

Serialization is enabled by default, but it can be disabled either by setting the environment variable STELLA_SERIALIZATION_DISABLE to a positive value or by calling the function Serializer.disable. With this function you disable the serialization independently of the current environment.

The serialization can be only globally enabled or disabled. There is no way to enable or disable only a specific serializer.

static status()

Get the status of serialization

The status is represented as an integer which can take the following values:

• 0: the variable is not yet initialized i.e the serialization is enabled if the environment variable STELLA_SERIALIZATION_DISABLE or SERIALBOX_SERIALIZATION_DISABLE is not set to a positive value. The first Serializer which is initialized has to set this value either to +1 or to -1 according to the environment.
• +1: the serialization is enabled, independently of the environment
• -1: the serialization is disabled, independently of the environment

See Serializer.enable and Serializer.disable.

Returns:serialization status Return type:int

global_metainfo

Get a reference to the global meta-information of the serializer.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.global_metainfo
<MetainfoMap {}>
>>> ser.global_metainfo.insert("key", 5)
>>> ser.global_metainfo
<MetainfoMap {"key": 5}>

Returns:Refrence to the meta-information map Return type:MetainfoMap

register_savepoint(savepoint)

Register savepoint within the Serializer.

Savepoints can have the same name but their meta-data has to be different, thus Savepoints have to be unique.

Parameters:savepoint (Savepoint) – Savepoint to add Raises:serialbox.SerialboxError – if savepoint already exists within the Serializer

has_savepoint(savepoint)

Check if savepoint exists within the Serializer.

Parameters:savepoint (Savepoint) – Savepoint to search for Returns:True if Savepoint exists, False otherwise Return type:bool

get_savepoint(name)

Get a list of Savepoint(s) identified by name in the order they were registered

The Savepoints in the list are copy-constructed from the Savepoints in the Serializer and inserted in the order they were registered.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.register_savepoint(Savepoint("sp", {"key": 5}))
>>> ser.register_savepoint(Savepoint("sp", {"key": 6}))
>>> ser.get_savepoint("sp")
[<Savepoint sp {"key": 5}>, <Savepoint sp {"key": 6}>]

Parameters:name (str) – Name of the savepoint(s) Returns:List of registered Savepoint(s) with savepoint.name() == name Return type:list [Savepoint]

fields_at_savepoint(savepoint)

Get a list of the registered fieldnames at savepoint.

Parameters:savepoint (Savepoint) – Savepoint to use Returns:list of fieldsnames Return type:list [str] Raises:serialbox.SerialboxError – if savepoint does not exists

savepoint_list()

Get a list of registered savepoints within the Serializer.

The Savepoints in the list are copy-constructed from the Savepoints in the Serializer and inserted in the order they were registered.

Returns:List of registered savepoints Return type:list [Savepoint]

savepoint

Access the savepoint collection of this Serializer

Savepoints are primarily identified by their name and further distinguished by their meta_info but they are unique within the Serializer.

Most of the time you will be interested in a particular Savepoint. This requires you to specify the meta-information key=value pairs. There are two ways of doing this. But first, recall that there is NO ordering in the meta-information, hence it does not matter in which sequence you specify them! Consider the following example in which we register two savepoints, both named identically but with slightly different meta-information:

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser.register_savepoint(Savepoint("my-savepoint", {"key1": 5, "key2": "str"}))
>>> ser.register_savepoint(Savepoint("my-savepoint", {"key1": 6, "key2": "str"}))

To access the first savepoint (i.e the one with key=5) we can use the element access:

>>> ser.savepoint["my-savepoint"]["key1"][5]["key2"]["str"]
<SavepointCollection [my-savepoint {"key2": str, "key1": 5}]>

Note that a SavepointCollection can be converted to a Savepoint (See SavepointCollection.as_savepoint())

Again, the order in which you access the meta-information does not matter, meaning the follwoing is completely identical:

>>> ser.savepoint["my-savepoint"]["key2"]["str"]["key1"][5]
<SavepointCollection [my-savepoint {"key2": str, "key1": 5}]>

The second way uses the member access of python which can be more convenient. Note that this way has some downsides if you don’t use savepoint names or keys which can be mapped to valid Python identifiers. For example instead of using a ‘.’ you can use ‘_’, same goes for ‘-‘. Therefore, we can also write:

>>> ser.savepoint.my_savepoint.key1[5].key2["str"]
<SavepointCollection [my-savepoint {"key2": str, "key1": 5}]>

The following transformations are applied to construct a valid Python identifier.

Character	Replacement
-	_
.	_
[0-9]	_[0-9]

It is also possible to mix the two approaches and thus bypass any problems with the identifiers.

>>> ser.savepoint["my-savepoint"].key1[5].key2["str"]
<SavepointCollection [my-savepoint {"key2": str, "key1": 5}]>

Returns:Collection of all savepoints Return type:SavepointCollection

register_field(name, fieldmetainfo)

Add field given by name with field meta-information fieldmetainfo to the Serializer.

The write methods can do the registration implicitly.

Parameters:

• name (str) – Name of the newly registered field
• fieldmetainfo (FieldMetainfo) – Field meta-information of the newly registered field

Raises:

serialbox.SerialboxError – if field with given name already exists within the Serializer

has_field(field)

Check if field is registered within the Serializer.

Parameters:field (str) – Name of the field to check for Returns:True if field exists, False otherwise Return type:bool

get_field_metainfo(field)

Get the FieldMetainfo of field.

Parameters:field (str) – Name of the field Returns:Copy of the field meta-information of field Return type:FieldMetainfo Raises:serialbox.SerialboxError – if field does not exist within the Serializer

fieldnames()

Get a list of registered fieldnames within the Serializer.

Returns:list of fieldnames Return type:list [str]

write(name, savepoint, field, register_field=True)

Serialize field identified by name at savepoint to disk

The savepoint will be registered at field name if not yet present. If register_field is True, the field will be registered if necessary.

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> field = np.random.rand(3,3)
>>> ser.write("myfield", Savepoint("mysavepoint"), field)
>>> ser.has_field("myfield")
True
>>> ser.fields_at_savepoint(Savepoint("mysavepoint"))
['myfield']

Parameters:

• name (str) – Name of the field
• savepoint (Savepoint) – Savepoint at which the field will be serialized
• field (numpy.array) – Field to serialize
• register_field (bool) – Register the field if not present

Raises:

serialbox.SerialboxError – if serialization failed

read(name, savepoint, field=None)

Deserialize field identified by name at savepoint from disk

If field is a numpy.array it will be filled with data from disk. If field is None, a new numpy.array will be allocated with the registered dimensions and type.

>>> ser = Serializer(OpenModeKind.Read, ".", "field", "Binary")
>>> ser.fieldnames()
['myfield']
>>> ser.get_field_metainfo("myfield")
<FieldMetainfo type = double, dims = [3, 3], metainfo = {}>
>>> field = ser.read("myfield", Savepoint("mysavepoint"))
>>> field
array([[ 0.56079736,  0.21627747,  0.87964583],
       [ 0.94684836,  0.12496717,  0.47460455],
       [ 0.11462436,  0.86608157,  0.57855988]])

Parameters:

• name (str) – Name of the field
• savepoint (Savepoint) – Savepoint at which the field will be deserialized
• field (numpy.array) – Field to fill or None

Returns:

Newly allocated and deserialized field

Return type:

numpy.array

Raises:

serialbox.SerialboxError – if deserialization failed

read_async(name, savepoint, field=None)

Asynchronously deserialize field name at savepoint from disk.

If field is a numpy.array it will be filled with data from disk. If field is None, a new numpy.array will be allocated with the registered dimensions and type.

This method runs the Serializer.read function asynchronously (potentially in a separate thread which may be part of a thread pool), meaning this function immediately returns. To synchronize all threads, use Serializer.wait_for_all.

If the archive is not thread-safe or if the library was not configured with SERIALBOX_ASYNC_API the method falls back to synchronous execution.

>>> ser = Serializer(OpenModeKind.Read, ".", "field", "Binary")
>>> ser.fieldnames()
['field_1', 'field_2', 'field_3']
>>> field_1 = ser.read_async("field1", Savepoint("sp"))
>>> field_2 = ser.read_async("field2", Savepoint("sp"))
>>> field_3 = ser.read_async("field3", Savepoint("sp"))
>>> ser.wait_for_all()

Parameters:

• name (str) – Name of the field
• savepoint (Savepoint) – Savepoint at which the field will be deserialized
• field (numpy.array) – Field to fill or None

Returns:

Newly allocated and deserialized field

Return type:

numpy.array

Raises:

SerialboxError – Deserialization failed

wait_for_all()

Wait for all pending asynchronous read operations and reset the internal queue.

read_slice(name, savepoint, slice_obj, field=None)

Deserialize sliced field identified by name and slice at savepoint from disk

The method will allocate a numpy.array with the registered dimensions and type and fill it at specified positions (given by slice_obj) with data from disk. If field is a numpy.array it will be filled with data from disk. If field is None, a new numpy.array will be allocated with the registered dimensions and type.

Assume we are given a three-dimensional field but we are only interested in a certain layer of the data (k = 50), we can use the slice object (ser.Slice) to encode this information and instruct the serializer to only load the desired data. Note that we still need to allocate memory for the whole field.

>>> ser = Serializer(OpenModeKind.Read, ".", "field", "Binary")
>>> ser.fieldnames()
['field']
>>> ser.get_field_metainfo("field")
<FieldMetainfo type = double, dims = [1024, 1024, 80], metainfo = {}>
>>> field = np.zeros(1024, 1024, 80)
>>> ser.read_slice('field', Savepoint("sp"), ser.Slice[:, :, 50], field)

You can of course load the full data and slice it afterwards with numpy which yields the same result, though is most likely slower.

>>> ser.read('field', Savepoint("sp"), field)[:, :, 50]

Parameters:

• savepoint (Savepoint) – Savepoint at which the field will be deserialized
• slice_obj (Slice) – Slice of the data to load
• field (numpy.array) – Field to fill or None

Returns:

Newly allocated and deserialized field

Return type:

numpy.array

Raises:

serialbox.SerialboxError – if deserialization failed

static to_file(name, field, filename, archive=None)

Serialize field identified by name directly to file.

This method allows stateless serializations i.e serialize fields without the need to register fields or savpoints.

If a file with filename already exists, it’s contents will be discarded. If archive is None, the method will try to deduce the archive using the extensions of filename (See Archive.archive_from_extension()).

>>> field = np.random.rand(3,3)
>>> Serializer.to_file("field", field, "field.dat")

Parameters:

• name (str) – Name of the field (may not be needed for certain archives)
• field (numpy.array) – Field to serialize
• name – Name of the file

Raises:

serialbox.SerialboxError – if archive could not be deduced or deserialization failed

from_file(name, field, filename, archive=None)

Deserialize field identified by name directly from file.

This method allows stateless deserializations i.e serialize fields without specifying the savepoints or fields.

If archive is None, the method will try to deduce the archive using the extensions of filename (See Archive.archive_from_extension()).

>>> field = np.zeros((3,3))
>>> field = Serializer.from_file("field", field, "field.dat")
>>> field
array([[ 0.56079736,  0.21627747,  0.87964583],
       [ 0.94684836,  0.12496717,  0.47460455],
       [ 0.11462436,  0.86608157,  0.57855988]])

Warning

This method performs no consistency checks concerning the dimensions and type of the data. You have to know what you are doing!

Parameters:

• name (str) – Name of the field (may not be needed for certain archives)
• field (numpy.array) – Field to serialize
• name – Name of the file

Raises:

serialbox.SerialboxError – if archive could not be deduced or deserialization failed

__str__()

Convert Serializer to string

>>> ser = Serializer(OpenModeKind.Write, ".", "field", "Binary")
>>> ser
<Serializer mode = Write
directory = "."
prefix = "field"
archive = "Binary"
metainfo = {}
savepoints = []
fieldmetainfo = []>

Returns:Multi-line string representation of the Serializer Return type:str

MetaInfoMap

class serialbox.MetainfoMap(metainfo=None, impl=None)

Meta-information implementation of the Python Interface.

Objects of this class contain a map of meta-information in form of key = value or key = {value1, … valueN} pair. The keys are strings and unique, while the values can be integers, booleans, floating point numbers (either single or double precision) or strings.

The elements are internally stored as a hash-map and thus the order of insertion is irrelevant. The MetainfoMaps can be constrcuted from python dictionary dict.

>>> m = MetainfoMap({'key1': 1, 'key2': 'str'})
>>> m
<MetainfoMap {"key2": str, "key": 5}>
>>>

__init__(metainfo=None, impl=None)

Initialize the Metainfo map.

If metainfo is None, an empty map is created. Elements can be added later with MetainfoMap.insert.

Parameters:

• metainfo (dict) – Key-value pair dictionary used for initialization
• impl – Directly set the implementation pointer [internal use]

clone()

Clone the Metainfo map by performing a deepcopy.

>>> m = MetainfoMap({'key1': 1, 'key2': 'str'})
>>> m_clone = m.clone()
>>> m.clear()
>>> m
<MetainfoMap {}>
>>> m_clone
<MetainfoMap {"key2": str, "key": 5}>

Returns:Clone of the map Return type:MetainfoMap

insert(key, value, typeid=None)

Insert a new element in the form key = value or key = {value1, … valueN} pair.

The element is inserted only if its key is not equivalent to the key of any other element already in the map (i.e keys must be unique). If the optional parameter typeid is omitted, the function will try it’s best effort to deduce the typeid, otherwise the value will be converted to match the type of typeid (see TypeID).

>>> m = MetainfoMap()
>>> m.insert('key', 5)
>>> m
<MetainfoMap {"key": 5}>
>>> m.insert('Array', [1, 2, 3, 4])
>>> m
<MetainfoMap {"Array": [1, 2, 3, 4], "key": 5}>
>>> m.insert('Float', 5.0, TypeID.Float32)
>>> m
<MetainfoMap {"Array": [1, 2, 3, 4], "Float": 5.000000, "key": 5}>
>>> type(m['Float'])
<class 'float'>

Parameters:

• key (str) – Key of the new element
• value – Object to be copied to the value of the new element
• typeid (serialbox.TypeID) – Type-id to use

Raises:

• SerialboxError – if element with key already exists or typeid could not be deduced
• TypeError – if typeid is not a serialbox.TypeID or int

size()

Get number of elements in the map.

Returns:Number of elements in the map Return type:int

empty()

Check if mao is empty.

Returns:True if map is empty, False otherwise Return type:bool

has_key(key)

Check if and element with key key exists.

Parameters:key (str) – Key of the element Returns:True if element with key exists, False otherwise Return type:bool

clear()

Clear the map.

All the elements in the MetainfoMap are dropped: their destructors are called, and they are removed from the container, leaving it with a size of 0.

to_dict()

Convert MetainfoMap to a python dictionary dict.

The MetainfoMap is copied into the dictionary.

>>> d = {'key': 5, 'string': 'str'}
>>> m = MetainfoMap(d)
>>> map_as_dict = m.to_dict()
>>> map_as_dict
{'key': 5, 'string': 'str'}
>>> d == map_as_dict
True

Returns:copy of the Metainfo map as a dictionary Return type:dict

__eq__(other)

Test for equality.

MetainfoMaps are equal if all their elements are equal.

Returns:True if self == other, False otherwise Return type:bool

__ne__(other)

Test for inequality.

MetainfoMaps are equal if all their elements are equal.

Returns:True if self != other, False otherwise Return type:bool

__getitem__(key, typeid=None)

Get value of element given by key. The correct type will be inferred.

>>> m = MetainfoMap()
>>> m.insert('key', 5)
>>> m['key']
5
>>>

Parameters:

• key (str) – Key of the element
• typeid (int) – Type-id [internal use]

Returns:

Copy of the value of the element

Raises:

serialbox.SerialboxError – if Element with key does not exist

__iter__()

Iterate the MetainfoMap.

>>> m = MetainfoMap({'key1': 5, 'key2': 6})
>>> for elements in m:
    ...     print(elements)
    ('key1', 5)
    ('key2', 6)
>>>

Returns:Iterator of MetainfoMap Return type:MetainfoMapIterator

FieldMetaInfo

class serialbox.FieldMetainfo(type, dims, metainfo=None, impl=None)

FieldMetainfo implementation of the Python Interface.

FieldMetainfos store the meta-information of fields. Each FieldMetainfo stores the type (TypeID) and dimension of the corresponding field and, optionally, arbitrary meta-information in the form of a MetainfoMap.

>>> f = FieldMetainfo(TypeID.Float64, [256, 256, 80])
>>> f
<FieldMetainfo type = double, dims = [256, 256, 80], metainfo = {}>
>>> f.metainfo.insert('key', 5)
>>> f
<FieldMetainfo type = double, dims = [256, 256, 80], metainfo = {"key": 5}>
>>>

__init__(type, dims, metainfo=None, impl=None)

Initialize the FieldMetainfo.

Parameters:

• type (TypeID, int) – Type of the field.
• dims (list [int]) – List of dimensions.
• metainfo (dict, MetainfoMap) – Key-value pair dictionary used to set the meta-information
• impl – Directly set the implementation pointer [internal use]

clone()

Clone the FieldMetainfo map by performing a deepcopy.

>>> f = FieldMetainfo(TypeID.Float64, [256, 256, 80])
>>> f_clone = f.clone()
>>> del f
>>> f_clone
<FieldMetainfo type = double, dims = [256, 256, 80], metainfo = {}>
>>>

Returns:Clone of the FieldMetainfo Return type:FieldMetainfo

type

Type of the associated field

Returns:Type of the field Return type:TypeID

dims

Dimensions of the associated field.

Returns:Dimensions of the field Return type:list [int]

metainfo

Meta-information of the associated field.

Returns:Refrence to the meta-info map Return type:MetainfoMap

__eq__(other)

Test for equality.

FieldMetainfos compare equal if their type, dimensions and meta-infos compare equal.

Returns:True if self == other, False otherwise Return type:bool

__ne__(other)

Test for inequality.

FieldMetainfos compare equal if their type, dimensions and meta-infos compare equal.

Returns:True if self != other, False otherwise Return type:bool

Savepoint

class serialbox.Savepoint(name, metainfo=None, impl=None)

Savepoints are used within the Serializer to discriminate fields at different points in time. Savepoints in the Serializer are unique and primarily identified by their name

>>> savepoint = Savepoint('savepoint')
>>> savepoint.name
'savepoint'
>>>

and further distinguished by their metainfo

>>> savepoint = Savepoint('savepoint', {'key': 5})
>>> savepoint.metainfo
<MetainfoMap {"key": 5}>
>>>

__init__(name, metainfo=None, impl=None)

Initialize the Savepoint.

This method prepares the Savepoint for usage and gives a name, which is the only required information for the savepoint to be usable. Meta-information can be added after the initialization has been performed.

Parameters:

• name (str) – Name of the savepoint
• metainfo (dict) – {Key:value} pair dictionary used for initializing the meta-information of the Savepont
• impl (SavepointImpl) – Directly set the implementation pointer [internal use]

Raises:

serialbox.SerialboxError – if Savepoint could not be initialized

name

Name of the Savepoint.

>>> s = Savepoint('savepoint')
>>> s.name
'savepoint'
>>>

Return str:Name of the savepoint Return type:str

metainfo

Refrence to the meta-information of the Savepoint.

>>> s = Savepoint('savepoint', {'key': 5})
>>> s.metainfo['key']
5
>>> type(s.metainfo)
<class 'serialbox.metainfomap.MetainfoMap'>
>>> s.metainfo.insert('key2', 'str')
>>> s
<MetainfoMap {"key": 5, "key2": str}>
>>>

Returns:Refrence to the meta-information map Return type:MetainfoMap

clone()

Clone the Savepoint by performing a deepcopy.

>>> s = Savepoint('savepoint', {'key': 5})
>>> s_clone = s.clone()
>>> s.metainfo.clear()
>>> s_clone
<Savepoint sp {"key": 5}>
>>>

Returns:Clone of the savepoint Return type:Savepoint

__eq__(other)

Test for equality.

Savepoints compare equal if their names and metainfos compare equal.

>>> s1 = Savepoint('savepoint', {'key': 'str'})
>>> s2 = Savepoint('savepoint', {'key': 5})
>>> s1 == s2
False
>>>

Returns:True if self == other, False otherwise Return type:bool

__ne__(other)

Test for inequality.

Savepoints compare equal if their names and metainfos compare equal.

>>> s1 = Savepoint('savepoint', {'key': 'str'})
>>> s2 = Savepoint('savepoint', {'key': 5})
>>> s1 != s2
True
>>>

Returns:True if self != other, False otherwise Return type:bool

class serialbox.SavepointCollection

Collection of savepoints. A collection can be obtained by using the savepoint attribute of the Serializer.

>>> ser = Serializer(OpenModeKind.Write, '.', 'field')
>>> ser.register_savepoint(Savepoint('s1'))
>>> ser.register_savepoint(Savepoint('s2'))
>>> isinstance(ser.savepoint, SavepointCollection)
True
>>> ser.savpoint.savepoints()
[<Savepoint s1 {}>, <Savepoint s2 {}>]
>>> ser.savepoint.as_savepoint()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "savepoint.py", line 227, in as_savepoint
    raise SerialboxError(errstr)
serialbox.error.SerialboxError: Savepoint is ambiguous. Candidates are:
  s1 {}
  s2 {}
>>>

savepoints()

Get the list of savepoints in this collection. The savepoints are ordered in the way they were inserted.

Returns:List of savepoints in the collection. Return type:list [Savepoint]

as_savepoint()

Return the unique savepoint in the list or raise an SerialboxError if the list has more than 1 element.

Returns:Unique savepoint in this collection. Return type:Savepoint Raises:serialbox.SerialboxError – if list has more than one Savepoint

Archive

class serialbox.Archive

Provide information about the registered archives

static registered_archives()

Get a list of strings of the registered archives.

Returns:Registered archives Return type:list [str]

static archive_from_extension(filename)

Deduce the name of the archive according to the extension of the filename.

Only the registered archives are taken into account!

Extensions	Archives
.dat, .bin	Binary
.nc	NetCDF

Parameters:filename (str) – Name of the file Returns:Name of the registered archive matching the file extension Return type:str Raises:serialbox.SerialboxError – if extensions is invalid or no registered archive supports it.

Logging

class serialbox.Logging

Logging infastrucutre

By default, logging is disabled. If SERIALBOX_DISABLE_LOGGING was used during compilation of the library, the functions do nothing (see Config).

static enable()

Enable logging.

static disable()

Disable logging.

static is_enabled()

Check if logging is enabled.

Returns:True if logging is enabled, False otherwise Return type:bool

Config

class serialbox.Config

Configurations used when compiling the serialboxC library. The configuration is stored as a python dictionary dict.

>>> config = Config()
>>> d = config.get_dict()
>>> d
{'SERIALBOX_HAS_NETCDF': '1',
 'SERIALBOX_CXX_FLAGS': '-std=c++11  -march=native -fPIC -O2 -g -DNDEBUG',
 'BOOST_VERSION': '1_61',
 'SERIALBOX_ASYNC_API': '1',
 'SERIALBOX_CXX_COMPILER': '/usr/bin/clang',
 'SERIALBOX_HAS_OPENSSL': '1'}
>>>

get_dict()

Dictionary of compile options

Returns:Compile options Return type:dict

OpenModeKind

class serialbox.OpenModeKind

Policy for opening files in the Serializer and Archive

Mode	Value
Read	0
Write	1
Append	2

TypeID

class serialbox.TypeID

Type-id of types recognized by serialbox.

This enum corresponds to the enum definitions of the C/C++ library.

TypeID	Python Type	C++ Type
TypeID.Boolean	bool	bool
TypeID.Int32	int, np.int32	int
TypeID.Int64	int, np.int64	std::int64_t
TypeID.Float32	float, np.float32	float
TypeID.Float64	float, np.float64	double
TypeID.String	str	std::string

Slice

Specification of the slice indices which are used for partial loading of serialized data. To avoid instantiation, use the global object serialbox.Slice.

>>> Slice[:, :, 5]
(slice(None, None, None), slice(None, None, None), 5)

See here for furhter information on indexing.

SerialboxError

class serialbox.SerialboxError(message)

Raised when an operation results in an error.

message

Explanation of the error.

Returns:Explanation string Return type:str

Visualizer

class serialbox.visualizer.Visualizer(field, fieldname, halospec=None)

Visualization a field in a matplotlib window

__init__(field, fieldname, halospec=None)

Visualization a field in a matplotlib window

Each k level is represented as a simple slice in the window.

Parameters:

• field (numpy.array) – Field to display
• fieldname (str) – Name of the field to display
• halospec (array[array]) – 2x2 array with the definition of the halo size (e.g [[3, 3], [3, 3]])