RawData

The RawData module exposes two functions

get : String -> Maybe RawData

and

getData : Int -> Int -> RawData -> Maybe Data

The first intelligently extracts a data table, column headers, and metadata from a string representing data in one of several formats -- csv, tab-delimited, or space-delimited. With the second, one can extract a list of Points in the xy plane from a data table.

A RawData value is a record of the following form:

type alias RawData =
    { metadata : List String
    , columnHeaders : List String
    , data : Table
    }

Here is how one can construct such a record from actual deta:

> get SampleData.temperature
     Just {
         columnHeaders = ["Year","Value"]
       , metadata = ["Global Land and Ocean Temperature Anomalies"
                     ,"January-December 1880-2016"
                     ,"Units: Degrees Celsius"
                    ]
       , rawData = [["1880","-0.12"],["1881","-0.07"],["1882","-0.08"]
                    ["1883","-0.15"], ...

To extract a list of points from the raw data, one proceeds as in the next example:

> get D.temperature |> Maybe.andThen (dataFromRawData 0 1)

type alias RawData =

{ metadata : List String

, columnHeaders : List String

, data : Table

}

A RawData value consists of metadata, columnHeaders, and data. The first two are lists of strings, while the last is a list of records, where a record is a lest of strings.

get : String -> Maybe RawData

get makes intelligent guesses, If it is successful

get spaceTest Just (["x","y"],[["0","0"],["1","0"],["1","1"],["0","1"]])

The function get accomplishes its work as follows. (1) it inspects the data and makes a guess about is format:

type DataFormat
    = SpaceDelimited
    | TabDelimited
    | CommaDelimited

Here CommaDelimited means Csv. (2) With this guess in hand the string is parsed into a Table value, i.e., a list of records. As is, it is usually in an unsatisfactory state. (3) Another informed guess is made about the the number of fields in the actual data, i.e., the number of columns of data. (4) The Table is filtered, removing rows that have too few or too many fields. (5) It is assumed that the real data is numerical, and that it starts on row k. An informed guess is made about the value of k, and the rows 1 through k - 1 are discarded. The actual data is now in hand, albeit as a table of strings. There is also enough information hand to extract the column headers -- the row of non-data just before the data starts, and the other metadata, the rows before the column headers .

The process just described is not fail-safe, but is it works quite frequently. One sometimes has to fix the data with a text editor.

getData : Int -> Int -> RawData -> Maybe Data

getData i j rawData_ extracts Data from RawData by extracting columns i and j of the RawData, trasforming these to lists of floats, and using these as the x and y coordinates of a list of Points.

getColumn : Int -> Table -> Maybe (List Float)

getColumn i table extracts a list of Floats from a table (List of Records) by extracting column i of the Table, transforming the result to lists of floats.

jxxcarlson/elm-stat/2.0.0

RawData

Modules