ORKG Shapes

go to ORKG: http://orkg.org/orkg/predicate/P43065

tokenizer

The method or tool used to preprocess text into tokens (e.g., Byte Pair Encoding, WordPiece).

• http://orkg.org/orkg/shapes/orkgc_C70000
• http://orkg.org/orkg/shapes/orkgc_Resource
• http://orkg.org/orkg/shapes/orkgc_Contribution