ElmTextSearch full text indexer

This is a full text indexing engine inspired by lunr.js and written in Elm language. See http://lunrjs.com/ for lunr.js

While ElmTextSearch has a good selection of tests this library is not battle tested and may contain some performance issues.

I am happy to hear about users of this package.

I am happy to receive contributions be they bug reports, pull requests, documention updates or examples.

v4.0.0 will not load indexes saved with old version.

If you do not use storeToValue storeToString fromString fromValue in ElmTextSearch this update is not likely to introduce issues.

The way that filters and transforms are applied to the content of documents has changed. This is to properly fix a bug reported see https://github.com/rluiten/elm-text-search/issues/10 where stop word filters werer not correctly applied. This means saved indexes from prevoius version of ElmTextSearch will not load in this version.

Defaults.indexVersion has changed value.

The reason this is a Major version bump is some generalisation was done to enable future support for loading and saving of older version and types of default index confgurations.

Packages

Several packages were created for this project and published seperately for this package to depend on.

trie * http://package.elm-lang.org/packages/rluiten/trie/latest
stemmer * http://package.elm-lang.org/packages/rluiten/stemmer/latest
sparsevector * http://package.elm-lang.org/packages/rluiten/sparsevector/latest

Parts of lunr.js were left out

* This does not have an event system. * Its internal data structure is not compatible.

Notes captured along way writing this.

lunr.js * tokenStore.remove does not decrement length, but it doesnt use length really only save/load * stemmer "lay" -> "lay" "try" -> "tri" is opposite to porter stemmer
porter stemmer erlang implementation * step5b does not use endsWithDoubleCons which is required afaik to pass the voc.txt output.txt cases

Example

See examples folder for three examples. First example is included inline here.

IndexNewAddSearch.elm

import ElmTextSearch
import Html exposing (Html, div, text)


{-| Example document type. -}
type alias ExampleDocType =
  { cid : String
  , title : String
  , author : String
  , body : String
  }


{-| Create an index with default configuration.
See ElmTextSearch.SimpleConfig documentation for parameter information.
-}
createNewIndexExample : ElmTextSearch.Index ExampleDocType
createNewIndexExample =
  ElmTextSearch.new
    { ref = .cid
    , fields =
        [ ( .title, 5.0 )
        , ( .body, 1.0 )
        ]
    , listFields = []
    }


{-| Add a document to an index. -}
resultUpdatedMyIndexAfterAdd :
  Result String (ElmTextSearch.Index ExampleDocType)
resultUpdatedMyIndexAfterAdd =
  ElmTextSearch.add
    { cid = "id1"
    , title = "First Title"
    , author = "Some Author"
    , body = "Words in this example document with explanations."
    }
    createNewIndexExample


{-| Search the index.

The result includes an updated Index because a search causes internal
caches to be updated to improve overall performance.
-}
resultSearchIndex :
  Result String
    ( ElmTextSearch.Index ExampleDocType
    , List (String, Float)
    )
resultSearchIndex =
  resultUpdatedMyIndexAfterAdd
    |> Result.andThen
      (ElmTextSearch.search "explanations")


{-| Display search result. -}
main =
  let
    -- want only the search results not the returned index
    searchResults = Result.map Tuple.second resultSearchIndex
  in
    div []
    [ text
        (
          "Result of searching for \"explanations\" is "
            ++ (toString searchResults)
        )
    ]

rluiten/elm-text-search/4.0.0