Class: Wikitext::Parser::Token

Inherits:

Object

• Object
• Wikitext::Parser::Token

Defined in:

doc/rdoc.rb

Overview

Token object representing a symbol found in the input stream during scanning. When you invoke the tokenize method you receive an array of Token instances.

This class exists purely for testing and diagnostic purposes; it is actually just a wrapper for the real token structure that is used internally. (In actual use the Wikitext extension doesn’t even use this class; it instead uses lightweight C structs under the hood for maximum speed and memory efficiency.)

Attributes

start

the location in memory (a character pointer into the input stream) where the token begins

stop

the location in memory (a character pointer) where the token ends

line_start

the line number where the token starts; numbering begins at line 1 (there is no line 0)

line_stop

the line number where the token ends

column_start

the column number where the token start; numbering beings at column 1 (there is no column 0)

column_stop

the column number where the token ends

code_point

for tokens outside the range of printable ASCII the UTF-32 code point corresponding to the token

token_type

the type of the token, from the possible set of token types returned by the types method

string_value

the textal content of the token as a Ruby String

Class Method Summary

• .types ⇒ Object

  Returns a hashof all token types by (numeric) value and (human-readable) name.

Class Method Details

.types ⇒ Object

Returns a hashof all token types by (numeric) value and (human-readable) name.

# File 'doc/rdoc.rb', line 424

def self.types
  # This is just a placeholder.
  # See token.c for the C source code to this method.
end