This modules defines how to build Avro Codecs.
Codec's represent how to encode and decode data for your domain types, as well as the schema with which they will be written.
For example, below we define a type alias and how it would be represented as an Avro record
type alias Person =
{ name : String, age : Maybe Int }
personCodec : Codec Person
personCodec =
success Person
|> requiring "name" string .name
|> optional "age" int .age
|> record { baseName = "person", nameSpace = [] }Furthermore, one of several variants in an Avro union can represent
an Elm custom type, by using the union family
of functions
personOrPetCodec : Codec (Result Person Pet)
personOrPetCodec =
union personCodec petCodecRecords, unions, named types, and basic types can be composed to easily represent complex models.
Map a Codec to a new type.
A Codec can transform data from avro, as well as to avro. Therefore to change its type, you need to provide function to transform back and forth between the original type and the new type.
The categorical term for this is that Codec is an invariant functor, you can map between isomorphic types.
The main utility of this is transforming basic types to custom types.
For example, here's the definition of the maybe function, which
uses imap with union.
maybe : Codec b -> Codec (Maybe b)
maybe just =
let
note =
Result.fromMaybe ()
in
imap note Result.toMaybe <|
union null justPartially map a Codec to a new type.
Like imap this function requires a from and to
mapping to the new type, but this also allows one to fail to parse an
unexpected value.
For example, the Logical UUID Type is backed by an Avro string.
As not all strings are valid UUIDs, this function would be used to
print and parse the string. Using the
elm-uuid
library:
import Prng.Uuid as Uuid exposing (Uuid)
uuid : Codec Uuid
uuid =
string
|> emap Uuid.toString Uuid.fromString
|> withLogicalType "uuid"This function can be overused as if the mapping fails the user will receive a parse error from the binary decoder, and lose any specific information as to the nature of the error.
Add documentation to a Codec.
If the Schema does not support documentation (i.e, it's not a Record or Enum) this function has no effect.
Add aliases to a Codec.
If the Schema does not support aliases (i.e, it's not a named type) this function has no effect.
Add a logical type annotation to a Codec.
A Codec for an int type
A Codec for a boolean type
A Codec for a long type.
Nota bene: This parses to an elm Int type, which has a maximum
precision of 53 bits. Numbers larger than this will fail to parse,
yielding a decoding error.
A Codec for a long type.
Nota bene: Elm integers cannot represent the full 64 bit range on standard platforms. This Codec employs two Ints using their 32 bit range to encode a single 64 bit signed integer.
A Codec for a float type
Warning: This uses an elm Float type to encode from, which is
a 64 bit type underneath. Using double may be a better choice unless
an expected reader can only accept a 32 bit float.
A Codec for a double type
A Codec for an avro null type
A Codec for a string type
A Codec for an array type
A Codec for a map type.
Construct a Avro enumeration encoded with a 0 base index.
Arguments are:
This can be used with emap to map to a custom type.
Use a Codec as a Named Type.
The Schema for this Codec will only be its name. This can be useful to separate definitions logically and share them.
When using this function, one will need to put the original schema in an environment before resolving reader and writer schemas.
Records are easily constructed using a pipeline style to compose field Codecs.
success Person
|> requiring "name" string .name
|> optional "age" int .age
|> record { baseName = "person", nameSpace = [] }Definition of a Struct Codec
Usually one will work with this type alias directly, as the builder is most useful when contructing the final value.
Builder for a Struct Codec.
The first type parameter b is the type we're mapping from when writing
an Avro record, the second type parameter a is the type we're reading
avro data into. When a builder is completely constructed, the types a
and b will be for the same type, and can be turned into a Codec a
using the record function.
The technical term for a type like this is that StructBuilder is an
applicative profunctor; it wraps a parser and a builder for lists of
Avro fields (along with their Schemas).
The usual pattern is to use the requiring,
or optional functions to chain together a
sequence of codecs for each field in turn.
Build a Codec for an Avro record from a StructCodec.
This function requires a "completed" StructCodec, which writes and reads the same type.
A struct builder which parses and writes no fields and always succeeds.
Compose a required field's Codecs to build a record.
This function is designed to be used in a pipeline, and therefore consumes fields in order when written in this style. For example:
success Person
|> requiring "name" string .name
|> requiring "age" int .ageWill expect a string column then an int column in the record; and apply the constructor Person to the arguments in that order.
The explicit arguments one should write are:
The final argument in the type signature is the pipelined builder for the record under construction.
Compose an optional field's Codec to build a record.
This will create a union in the Schema with null as the first field and set a default value of null.
Use a field in a struct codec which falls back if it doesn't exist.
In the avro specification, the default value for Union values must be a value from the first sub-schema of the union. If the default value violates this contstraint it will not be emitted when serializing the Schema to JSON.
Use a field in a struct codec, supplying all possible parts.
If you need to add documentation, ordering, or aliases, this function allows full customisation of the struct field.
The arguments are:
Union types are similar to User defined types, and can be used to encode them from Elm.
These functions will ensure that unions don't directly contain other unions (which would violate the Avro specification), but one should also ensure that type names aren't replicated in the resulting union.
The best way to achieve this is to use these functions with record types of distinct names where possible.
A codec for a potentially missing value.
If using this in a record, it may be best to use the
optional function instead, as that will
apply this function as well as setting a default value.
Construct a Codec for an Avro union.
As Avro unions can not be nested (i.e., they can not directly contain other unions) this function takes care to flatten unions passed in to it.
Often it is useful to use the imap function to
turn this into a Custom Type.
Construct a union from 3 codecs.
Construct a union from 4 codecs.
Construct a union from 5 codecs.
Build a record type which may be recursive, by providing a Codec
which embeds the record as a NamedType in the Schema.
type LinkedList
= LinkedList Int (Maybe LinkedList)
linkedCodec : Codec LinkedList
linkedCodec =
let
struct rec =
success LinkedList
|> requiring "item" int (\(LinkedList a _) -> a)
|> optional "rest" rec (\(LinkedList _ a) -> a)
in
struct
|> recursiveRecord { baseName = "LinkedList", nameSpace = [] }These functions are lower level than those above, but provide an alternative method for building records.
Construct a struct parser from a Codec.
Usually one can use the requiring, or optional
faimly of functions to chain together a sequence of point parsers.
The earlier example is equivalent to:
example =
success Person
|> using (structField name [] Nothing Nothing string Nothing |> lmap .name)
|> using (structField age [] Nothing Nothing int Nothing |> lmap .age)
example2 =
map2 Person
(structField "name" [] Nothing Nothing string Nothing |> lmap .name)
(structField "age" [] Nothing Nothing int Nothing |> lmap .age)Apply a function across 2 codecs.
This can also be used with success if a part of a constructor is not to
be included in the Avro definition at all.
Profunctor mapping of struct codec.
Contravariant mapping of struct codec.
Mapping of struct codec.
Apply a function through struct codecs.
This can be used as an alternative to using
Apply a function through struct codecs.
Apply a function through struct codecs.
To map across more fields, one can build additional layers with using.
map5 f a b c d e =
map4 f a b c d
|> using e
This type defines the schema, encoder, and decoder for an elm type.