PackagesReroutesAboutlamdera.com

finos/morphir-elm/19.0.0

NOTE: the latest version is 22.0.2

Formerly Morgan-Stanley/morphir-elm. Search finds this package by either name.

READMEAboutVersionsSourceModules

Morphir.SDK.Aggregate

This module contains functions specifically designed to work with large data sets.

Aggregations

type alias Aggregation a key =
{ key : a -> key
, filter : a -> Bool
, operator : Operator a
}

Type that represents an aggregation on a type a with a key of key. It encapsulates the following information:

  • key is a function that gets the key of each a
  • filter is a function used for filtering items out before the aggregation. This can be set to always True to not do any filtering.
  • operator is the aggregation operation to apply (count, sum, average, ...)
groupBy : (a -> key) -> List a -> Dict key (List a)

Group a list of items into a dictionary. Grouping is done using a function that returns a key for each item. The resulting dictionary will use those keys as the key of each entry in the dictionary and values will be lists of items for each key.

testDataSet =
    [ TestInput1 "k1_1" "k2_1" 1
    , TestInput1 "k1_1" "k2_1" 2
    , TestInput1 "k1_1" "k2_2" 3
    , TestInput1 "k1_1" "k2_2" 4
    , TestInput1 "k1_2" "k2_1" 5
    , TestInput1 "k1_2" "k2_1" 6
    , TestInput1 "k1_2" "k2_2" 7
    , TestInput1 "k1_2" "k2_2" 8
    ]


testDataSet
    |> groupBy .key1
        {- == Dict.fromList
                    [ ( "k1_1"
                      , [ TestInput1 "k1_1" "k2_1" 1
                        , TestInput1 "k1_1" "k2_1" 2
                        , TestInput1 "k1_1" "k2_2" 3
                        , TestInput1 "k1_1" "k2_2" 4
                        ]
                    , ( "k1_2",
                      , [ TestInput1 "k1_2" "k2_1" 5
                        , TestInput1 "k1_2" "k2_1" 6
                        , TestInput1 "k1_2" "k2_2" 7
                        , TestInput1 "k1_2" "k2_2" 8
                        ]
                    ]
        -}
aggregate : (key -> Aggregator a Key0 -> b) -> Dict key (List a) -> List b

Aggregates a dictionary that contains lists of items as values into a list that contains exactly one item per key. The first argument is a function that takes a key and an aggregator and it should return a single item in the resulting list. The aggregator is a function that takes one of the aggregation functions in this module (count, sumOf, minimumOf, ...) and returns the aggregated value for the list of values in the input dictionary.

grouped =
    Dict.fromList
        [ ( "k1_1"
          , [ TestInput1 "k1_1" "k2_1" 1
            , TestInput1 "k1_1" "k2_1" 2
            , TestInput1 "k1_1" "k2_2" 3
            , TestInput1 "k1_1" "k2_2" 4
            ]
        , ( "k1_2",
          , [ TestInput1 "k1_2" "k2_1" 5
            , TestInput1 "k1_2" "k2_1" 6
            , TestInput1 "k1_2" "k2_2" 7
            , TestInput1 "k1_2" "k2_2" 8
            ]
        ]


grouped
    |> aggregate
        (\key inputs ->
            { key = key
            , count = inputs (count |> withFilter (\a -> a.value < 7))
            , sum = inputs (sumOf .value)
            , max = inputs (maximumOf .value)
            , min = inputs (minimumOf .value)
            }
        )
        {- ==
            [ { key = "k1_1", count = 4, sum = 10, max = 4, min = 1 }
            , { key = "k1_2", count = 2, sum = 26, max = 8, min = 5 }
            ]
        -}

This function is designed to be used in combination with groupBy.

testDataSet =
        [ TestInput1 "k1_1" "k2_1" 1
        , TestInput1 "k1_1" "k2_1" 2
        , TestInput1 "k1_1" "k2_2" 3
        , TestInput1 "k1_1" "k2_2" 4
        , TestInput1 "k1_2" "k2_1" 5
        , TestInput1 "k1_2" "k2_1" 6
        , TestInput1 "k1_2" "k2_2" 7
        , TestInput1 "k1_2" "k2_2" 8
        ]


    testDataSet
        |> groupBy .key1
        |> aggregate
            (\key inputs ->
                { key = key
                , count = inputs (count |> withFilter (\a -> a.value < 7))
                , sum = inputs (sumOf .value)
                , max = inputs (maximumOf .value)
                , min = inputs (minimumOf .value)
                }
            )
            { ==
                [ { key = "k1_1", count = 4, sum = 10, max = 4, min = 1 }
                , { key = "k1_2", count = 2, sum = 26, max = 8, min = 5 }
                ]
            }
aggregateMap : Aggregation a key1 -> (Float -> a -> b) -> List a -> List b

Map function that provides an aggregated value to the mapping function. The first argument is a tuple where the first element is a function that defines the aggregation key, the second element is predicate that allows you to filter out certain rows from the aggregation and the third argument is the aggregation operation to apply. Usage:

    testDataSet =
        [ TestInput1 "k1_1" "k2_1" 1
        , TestInput1 "k1_1" "k2_1" 2
        , TestInput1 "k1_1" "k2_2" 3
        , TestInput1 "k1_1" "k2_2" 4
        , TestInput1 "k1_2" "k2_1" 5
        , TestInput1 "k1_2" "k2_1" 6
        , TestInput1 "k1_2" "k2_2" 7
        , TestInput1 "k1_2" "k2_2" 8
        ]


    testDataSet
        |> aggregateMap
            (sumOf .value |> byKey .key1)
                (\\totalValue input ->
                    ( input, totalValue / input.value )
                )
        {- ==
            [ ( TestInput1 "k1_1" "k2_1" 1, 10 / 1 )
            , ( TestInput1 "k1_1" "k2_1" 2, 10 / 2 )
            , ( TestInput1 "k1_1" "k2_2" 3, 10 / 3 )
            , ( TestInput1 "k1_1" "k2_2" 4, 10 / 4 )
            , ( TestInput1 "k1_2" "k2_1" 5, 26 / 5 )
            , ( TestInput1 "k1_2" "k2_1" 6, 26 / 6 )
            , ( TestInput1 "k1_2" "k2_2" 7, 26 / 7 )
            , ( TestInput1 "k1_2" "k2_2" 8, 26 / 8 )
            ]
        -}
Aggregation a key1
-> Aggregation a key2
-> (Float -> Float -> a -> b)
-> List a
-> List b

Map function that provides two aggregated values to the mapping function. The first argument is a tuple where the first element is a function that defines the aggregation key, the second element is predicate that allows you to filter out certain rows from the aggregation and the third argument is the aggregation operation to apply. Usage:

    testDataSet =
        [ TestInput1 "k1_1" "k2_1" 1
        , TestInput1 "k1_1" "k2_1" 2
        , TestInput1 "k1_1" "k2_2" 3
        , TestInput1 "k1_1" "k2_2" 4
        , TestInput1 "k1_2" "k2_1" 5
        , TestInput1 "k1_2" "k2_1" 6
        , TestInput1 "k1_2" "k2_2" 7
        , TestInput1 "k1_2" "k2_2" 8
        ]


    testDataSet
        |> aggregateMap2
            (sumOf .value |> byKey .key1)
            (maximumOf .value |> byKey .key2)
            (\totalValue maxValue input ->
                ( input, totalValue * maxValue / input.value )
            )
        {- ==
            [ ( TestInput1 "k1_1" "k2_1" 1, 10 * 6 / 1 )
            , ( TestInput1 "k1_1" "k2_1" 2, 10 * 6 / 2 )
            , ( TestInput1 "k1_1" "k2_2" 3, 10 * 8 / 3 )
            , ( TestInput1 "k1_1" "k2_2" 4, 10 * 8 / 4 )
            , ( TestInput1 "k1_2" "k2_1" 5, 26 * 6 / 5 )
            , ( TestInput1 "k1_2" "k2_1" 6, 26 * 6 / 6 )
            , ( TestInput1 "k1_2" "k2_2" 7, 26 * 8 / 7 )
            , ( TestInput1 "k1_2" "k2_2" 8, 26 * 8 / 8 )
            ]
        -}
Aggregation a key1
-> Aggregation a key2
-> Aggregation a key3
-> (Float -> Float -> Float -> a -> b)
-> List a
-> List b

Map function that provides three aggregated values to the mapping function. The first argument is a tuple where the first element is a function that defines the aggregation key, the second element is predicate that allows you to filter out certain rows from the aggregation and the third argument is the aggregation operation to apply. Usage:

    testDataSet =
        [ TestInput1 "k1_1" "k2_1" 1
        , TestInput1 "k1_1" "k2_1" 2
        , TestInput1 "k1_1" "k2_2" 3
        , TestInput1 "k1_1" "k2_2" 4
        , TestInput1 "k1_2" "k2_1" 5
        , TestInput1 "k1_2" "k2_1" 6
        , TestInput1 "k1_2" "k2_2" 7
        , TestInput1 "k1_2" "k2_2" 8
        ]


    testDataSet
        |> aggregateMap3
            (sumOf .value |> byKey .key1)
            (maximumOf .value |> byKey .key2)
            (minimumOf .value |> byKey (key2 .key1 .key2))
            (\totalValue maxValue minValue input ->
                ( input, totalValue * maxValue / input.value + minValue )
            )
        {- ==
            [ ( TestInput1 "k1_1" "k2_1" 1, 10 * 6 / 1 + 1 )
            , ( TestInput1 "k1_1" "k2_1" 2, 10 * 6 / 2 + 1 )
            , ( TestInput1 "k1_1" "k2_2" 3, 10 * 8 / 3 + 3 )
            , ( TestInput1 "k1_1" "k2_2" 4, 10 * 8 / 4 + 3 )
            , ( TestInput1 "k1_2" "k2_1" 5, 26 * 6 / 5 + 5 )
            , ( TestInput1 "k1_2" "k2_1" 6, 26 * 6 / 6 + 5 )
            , ( TestInput1 "k1_2" "k2_2" 7, 26 * 8 / 7 + 7 )
            , ( TestInput1 "k1_2" "k2_2" 8, 26 * 8 / 8 + 7 )
            ]
        -}
Aggregation a key1
-> Aggregation a key2
-> Aggregation a key3
-> Aggregation a key4
-> (Float -> Float -> Float -> Float -> a -> b)
-> List a
-> List b

Map function that provides three aggregated values to the mapping function. The first argument is a tuple where the first element is a function that defines the aggregation key, the second element is predicate that allows you to filter out certain rows from the aggregation and the third argument is the aggregation operation to apply. Usage:

    testDataSet =
        [ TestInput1 "k1_1" "k2_1" 1
        , TestInput1 "k1_1" "k2_1" 2
        , TestInput1 "k1_1" "k2_2" 3
        , TestInput1 "k1_1" "k2_2" 4
        , TestInput1 "k1_2" "k2_1" 5
        , TestInput1 "k1_2" "k2_1" 6
        , TestInput1 "k1_2" "k2_2" 7
        , TestInput1 "k1_2" "k2_2" 8
        ]


    testDataSet
        |> aggregateMap4
            (sumOf .value |> byKey .key1)
            (maximumOf .value |> byKey .key2)
            (minimumOf .value |> byKey (key2 .key1 .key2))
            (averageOf .value |> byKey (key2 .key1 .key2))
            (\totalValue maxValue minValue average input ->
                ( input, totalValue * maxValue / input.value + minValue + average )
            )
       {-  ==
            [ ( TestInput1 "k1_1" "k2_1" 1, 10 * 6 / 1 + 1 + 1.5 )
            , ( TestInput1 "k1_1" "k2_1" 2, 10 * 6 / 2 + 1 + 1.5 )
            , ( TestInput1 "k1_1" "k2_2" 3, 10 * 8 / 3 + 3 + 3.5 )
            , ( TestInput1 "k1_1" "k2_2" 4, 10 * 8 / 4 + 3 + 3.5 )
            , ( TestInput1 "k1_2" "k2_1" 5, 26 * 6 / 5 + 5 + 5.5 )
            , ( TestInput1 "k1_2" "k2_1" 6, 26 * 6 / 6 + 5 + 5.5 )
            , ( TestInput1 "k1_2" "k2_2" 7, 26 * 8 / 7 + 7 + 7.5 )
            , ( TestInput1 "k1_2" "k2_2" 8, 26 * 8 / 8 + 7 + 7.5 )
            ]
       -}

Operators

count : Aggregation a Key0

Count the number of rows in a group.

sumOf : (a -> Float) -> Aggregation a Key0

Apply a function to each row that returns a numeric value and return the sum of the values.

minimumOf : (a -> Float) -> Aggregation a Key0

Apply a function to each row that returns a numeric value and return the minimum of the values.

maximumOf : (a -> Float) -> Aggregation a Key0

Apply a function to each row that returns a numeric value and return the maximum of the values.

averageOf : (a -> Float) -> Aggregation a Key0

Apply a function to each row that returns a numeric value and return the average of the values.

weightedAverageOf : (a -> Float) -> (a -> Float) -> Aggregation a Key0

Apply two functions to each row that returns a numeric value and return the weighted of the values using the first function to get the weights.

byKey : (a -> key) -> Aggregation a Key0 -> Aggregation a key

Changes the key of an aggregation. Usage:

count
    |> byKey .key1
    == { key = .key1
       , filter = always True
       , operator = Count
       }
withFilter : (a -> Bool) -> Aggregation a key -> Aggregation a key

Adds a filter to an aggregation. Usage:

count
    |> withFilter (\a -> a.value < 0)
    == { key = key0
       , filter = \a -> a.value < 0
       , operator = Count
       }

Utilities

TypedValue
-> TypedValue
-> TypedValue

constructAggregationCall transforms a Morphir.SDK.Aggregate groupBy and agggregate call into a single data structure

= AggregationCall
Name
(Maybe Name)
TypedValue

An AggregationCall represents a call to Morphir.SDK.Aggregate.aggregate

Its values are:

  • Group Key
    • i.e. the name of the column results are grouped by.
  • Returned Group Key
    • Optionally, the name to give the group column in the results.
  • Aggregate Values
    • i.e. the expressions that define columns in the output of aggregate.
  • Source Value
    • i.e. the Value that is being aggregated
= AggregateValue
Name
(Maybe TypedValue)
FQName
(Maybe TypedValue)

An AggregateValue represents a single aggregation used within the overall Aggregation call

Its values are:

  • Field Name
    • i.e. The column name in the output.
  • Filter
    • A Maybe Value of a filter expression, if it exists.
  • Aggregation Name
    • The Name of the aggregation function used (e.g. "sumOf")
  • Aggregation Arg
    • Any argument passed to the aggregation (e.g. ".ageOfItem" in "sumOf .AgeOfItem")

A ConstructAggregationError represents the ways constructAggregationCall may fail

Its values are:

  • TooManyKeyFields
    • i.e. in aggregate, 'key' was used to create multiple columns of the field(s) the data is grouped by, which can't be represented in Spark.
  • UnhandledValue
    • i.e. the 'aggregateBody' or 'groupKey' Values passed into constructAggregationCall could not be recognised as an aggregation.

Modules

About this package serverReroutesreroutes.json
Pages and docs rendering from package.elm-lang.org, © 2012-2020 Evan Czaplicki. © 2026 Lamdera