HuwCampbell/avro/1.0.1

Avro.Codec

This modules defines how to build Avro Codecs.

Codec's represent how to encode and decode data for your domain types, as well as the schema with which they will be written.

For example, below we define a type alias and how it would be represented as an Avro record

type alias Person =
    { name : String, age : Maybe Int }


personCodec : Codec Person
personCodec =
    success Person
        |> requiring "name" string .name
        |> optional "age" int .age
        |> record { baseName = "person", nameSpace = [] }

Furthermore, one of several variants in an Avro union can represent an Elm custom type, by using the union family of functions

personOrPetCodec : Codec (Result Person Pet)
personOrPetCodec =
    union personCodec petCodec

Records, unions, named types, and basic types can be composed to easily represent complex models.

Core Type

type alias Codec a =
{ schema : Schema
, decoder : Value -> Maybe a
, writer : a -> Value
}

This type defines the schema, encoder, and decoder for an elm type.

imap : (b -> a) -> (a -> b) -> Codec a -> Codec b

Map a Codec to a new type.

A Codec can transform data from avro, as well as to avro. Therefore to change its type, you need to provide function to transform back and forth between the original type and the new type.

The categorical term for this is that Codec is an invariant functor, you can map between isomorphic types.

The main utility of this is transforming basic types to custom types.

For example, here's the definition of the maybe function, which uses imap with union.

maybe : Codec b -> Codec (Maybe b)
maybe just =
    let
        note =
            Result.fromMaybe ()
    in
    imap note Result.toMaybe <|
        union null just
emap : (b -> a) -> (a -> Maybe b) -> Codec a -> Codec b

Partially map a Codec to a new type.

Like imap this function requires a from and to mapping to the new type, but this also allows one to fail to parse an unexpected value.

For example, the Logical UUID Type is backed by an Avro string. As not all strings are valid UUIDs, this function would be used to print and parse the string. Using the elm-uuid library:

import Prng.Uuid as Uuid exposing (Uuid)


uuid : Codec Uuid
uuid =
    string
        |> emap Uuid.toString Uuid.fromString
        |> withLogicalType "uuid"

This function can be overused as if the mapping fails the user will receive a parse error from the binary decoder, and lose any specific information as to the nature of the error.

withDocumentation : String -> Codec a -> Codec a

Add documentation to a Codec.

If the Schema does not support documentation (i.e, it's not a Record or Enum) this function has no effect.

withAliases : List TypeName -> Codec a -> Codec a

Add aliases to a Codec.

If the Schema does not support aliases (i.e, it's not a named type) this function has no effect.

withLogicalType : String -> Codec a -> Codec a

Add a logical type annotation to a Codec.

Basic Builders

int : Codec Int

A Codec for an int type

bool : Codec Bool

A Codec for a boolean type

long : Codec Int

A Codec for a long type.

Nota bene: This parses to an elm Int type, which has a maximum precision of 53 bits. Numbers larger than this will fail to parse, yielding a decoding error.

int64 : Codec Int64

A Codec for a long type.

Nota bene: Elm integers cannot represent the full 64 bit range on standard platforms. This Codec employs two Ints using their 32 bit range to encode a single 64 bit signed integer.

float32 : Codec Float

A Codec for a float type

Warning: This uses an elm Float type to encode from, which is a 64 bit type underneath. Using double may be a better choice unless an expected reader can only accept a 32 bit float.

float64 : Codec Float

A Codec for a double type

null : Codec ()

A Codec for an avro null type

string : Codec String

A Codec for a string type

array : Codec a -> Codec (List a)

A Codec for an array type

dict : Codec a -> Codec (Dict String a)

A Codec for a map type.

enum : TypeName -> List String -> Maybe String -> Codec Int

Construct a Avro enumeration encoded with a 0 base index.

Arguments are:

  • The name of the enum;
  • A list of symbols which the enum represents; and
  • An optional default, used when the reader encounters a symbol from the writer that isn’t defined in the reader’s original set of symbols.

This can be used with emap to map to a custom type.

namedType : Codec a -> Codec a

Use a Codec as a Named Type.

The Schema for this Codec will only be its name. This can be useful to separate definitions logically and share them.

When using this function, one will need to put the original schema in an environment before resolving reader and writer schemas.

Working with Record Types

Records are easily constructed using a pipeline style to compose field Codecs.

success Person
    |> requiring "name" string .name
    |> optional "age" int .age
    |> record { baseName = "person", nameSpace = [] }

Core types

type alias StructCodec a =

Definition of a Struct Codec

Usually one will work with this type alias directly, as the builder is most useful when contructing the final value.

type StructBuilder b a

Builder for a Struct Codec.

The first type parameter b is the type we're mapping from when writing an Avro record, the second type parameter a is the type we're reading avro data into. When a builder is completely constructed, the types a and b will be for the same type, and can be turned into a Codec a using the record function.

The technical term for a type like this is that StructBuilder is an applicative profunctor; it wraps a parser and a builder for lists of Avro fields (along with their Schemas).

The usual pattern is to use the requiring, or optional functions to chain together a sequence of codecs for each field in turn.

Construction

record : TypeName -> StructCodec a -> Codec a

Build a Codec for an Avro record from a StructCodec.

This function requires a "completed" StructCodec, which writes and reads the same type.

A struct builder which parses and writes no fields and always succeeds.

String
-> Codec a
-> (c -> a)
-> StructBuilder c (a -> b)

Compose a required field's Codecs to build a record.

This function is designed to be used in a pipeline, and therefore consumes fields in order when written in this style. For example:

success Person
    |> requiring "name" string .name
    |> requiring "age" int .age

Will expect a string column then an int column in the record; and apply the constructor Person to the arguments in that order.

The explicit arguments one should write are:

  • The field name which will be written into the Avro Schema;
  • The Codec for the individual field; and
  • How to extract the fields from the type in order to write it.

The final argument in the type signature is the pipelined builder for the record under construction.

String
-> Codec a
-> (c -> Maybe a)
-> StructBuilder c (Maybe a -> b)

Compose an optional field's Codec to build a record.

This will create a union in the Schema with null as the first field and set a default value of null.

String
-> Codec a
-> a
-> (c -> a)
-> StructBuilder c (a -> b)

Use a field in a struct codec which falls back if it doesn't exist.

In the avro specification, the default value for Union values must be a value from the first sub-schema of the union. If the default value violates this contstraint it will not be emitted when serializing the Schema to JSON.

String
-> List String
-> Maybe String
-> Maybe SortOrder
-> Codec a
-> Maybe a
-> (c -> a)
-> StructBuilder c (a -> b)

Use a field in a struct codec, supplying all possible parts.

If you need to add documentation, ordering, or aliases, this function allows full customisation of the struct field.

The arguments are:

  • name
  • aliases
  • documentation
  • ordering
  • point codec
  • default value
  • extractor

Working with Union Types

Union types are similar to User defined types, and can be used to encode them from Elm.

These functions will ensure that unions don't directly contain other unions (which would violate the Avro specification), but one should also ensure that type names aren't replicated in the resulting union.

The best way to achieve this is to use these functions with record types of distinct names where possible.

maybe : Codec b -> Codec (Maybe b)

A codec for a potentially missing value.

If using this in a record, it may be best to use the optional function instead, as that will apply this function as well as setting a default value.

union : Codec a -> Codec b -> Codec (Result a b)

Construct a Codec for an Avro union.

As Avro unions can not be nested (i.e., they can not directly contain other unions) this function takes care to flatten unions passed in to it.

Often it is useful to use the imap function to turn this into a Custom Type.

union3 : Codec a -> Codec b -> Codec c -> Codec (Result a (Result b c))

Construct a union from 3 codecs.

Codec a
-> Codec b
-> Codec c
-> Codec d
-> Codec (Result a (Result b (Result c d)))

Construct a union from 4 codecs.

Codec a
-> Codec b
-> Codec c
-> Codec d
-> Codec e
-> Codec (Result a (Result b (Result c (Result d e))))

Construct a union from 5 codecs.

Working with Recursive Types

recursiveRecord : TypeName -> (Codec a -> StructCodec a) -> Codec a

Build a record type which may be recursive, by providing a Codec which embeds the record as a NamedType in the Schema.

type LinkedList
    = LinkedList Int (Maybe LinkedList)


linkedCodec : Codec LinkedList
linkedCodec =
    let
        struct rec =
            success LinkedList
                |> requiring "item" int (\(LinkedList a _) -> a)
                |> optional "rest" rec (\(LinkedList _ a) -> a)
    in
    struct
        |> recursiveRecord { baseName = "LinkedList", nameSpace = [] }

Fancy Records

These functions are lower level than those above, but provide an alternative method for building records.

String
-> List String
-> Maybe String
-> Maybe SortOrder
-> Codec a
-> Maybe a
-> StructCodec a

Construct a struct parser from a Codec.

Usually one can use the requiring, or optional faimly of functions to chain together a sequence of point parsers.

The earlier example is equivalent to:

example =
    success Person
        |> using (structField name [] Nothing Nothing string Nothing |> lmap .name)
        |> using (structField age [] Nothing Nothing int Nothing |> lmap .age)


example2 =
    map2 Person
        (structField "name" [] Nothing Nothing string Nothing |> lmap .name)
        (structField "age" [] Nothing Nothing int Nothing |> lmap .age)
using : StructBuilder c a -> StructBuilder c (a -> b) -> StructBuilder c b

Apply a function across 2 codecs.

This can also be used with success if a part of a constructor is not to be included in the Avro definition at all.

dimap : (x -> y) -> (a -> b) -> StructBuilder y a -> StructBuilder x b

Profunctor mapping of struct codec.

lmap : (x -> y) -> StructBuilder y a -> StructBuilder x a

Contravariant mapping of struct codec.

map : (a -> b) -> StructBuilder x a -> StructBuilder x b

Mapping of struct codec.

(a -> b -> c)

Apply a function through struct codecs.

This can be used as an alternative to using

(a -> b -> c -> d)

Apply a function through struct codecs.

(a -> b -> c -> d -> e)

Apply a function through struct codecs.

To map across more fields, one can build additional layers with using.

map5 f a b c d e =
    map4 f a b c d
        |> using e