ngrams alternatives and similar packages
Based on the "Text processing" category.
Alternatively, view ngrams alternatives based on common mentions on social networks and blogs.
-
regex
An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs. -
-
WorkOS - The modern identity platform for B2B SaaS
Do you think we are missing an alternative of ngrams or a related project?
README
N-grams
This crate takes a sequence of tokens and generates an n-gram for it. For more information about n-grams, check wikipedia: https://en.wikipedia.org/wiki/N-gram
Note: The canonical version of this crate is hosted on Gitlab
Usage
Probably the easiest way to use it is to use the iterator adaptor. If your tokens are strings (&str, String, char, or Vec), you don't have to do anything other than generate the token stream:
use ngrams::Ngram;
let grams: Vec<_> = "one two three".split(' ').ngrams(2).collect();
// => vec![
// vec!["\u{2060}", "one"],
// vec!["one", "two"],
// vec!["two", "three"],
// vec!["three", "\u{2060}"],
// ]
(re: the "\u{2060}": We use the unicode WORD JOINER symbol as padding on the beginning and
end of the token stream.)
If your token type isn't one of the listed types, you can still use the
iterator adaptor by implementing the ngram::Pad trait for your type.