IO

class finalfusion.io.Chunk

Basic building blocks of finalfusion files.

write(file: Union[str, bytes, int, os.PathLike])

Write the Chunk as a standalone finalfusion file.

Parameters

file (Union[str, bytes, int, PathLike]) – Output path

Raises

TypeError – If the Chunk is a Header.

abstract static chunk_identifier() → finalfusion.io.ChunkIdentifier

Get the ChunkIdentifier for this Chunk.

Returns

chunk_identifier

Return type

ChunkIdentifier

abstract static read_chunk(file: BinaryIO) → finalfusion.io.Chunk

Read the Chunk and return it.

The file must be positioned before the contents of the Chunk but after its header.

Parameters

file (BinaryIO) – a finalfusion file containing the given Chunk

Returns

chunk – The chunk read from the file.

Return type

Chunk

abstract write_chunk(file: BinaryIO)

Write the Chunk to a file.

Parameters

file (BinaryIO) – Output file for the Chunk

class finalfusion.io.ChunkIdentifier(value)

Bases: enum.IntEnum

Known finalfusion Chunk types.

Header = 0

SimpleVocab = 1

NdArray = 2

BucketSubwordVocab = 3

QuantizedArray = 4

Metadata = 5

NdNorms = 6

FastTextSubwordVocab = 7

ExplicitSubwordVocab = 8

class finalfusion.io.FinalfusionFormatError

Bases: Exception

Exception to specify that the format of a finalfusion file was incorrect.

class finalfusion.io.TypeId(value)

Bases: enum.IntEnum

Known finalfusion data types.

u8 = 1

f32 = 10