Package ZestyParser :: Module Tokens :: Class AbstractToken
[show private | hide private]
[frames | no frames]

Type AbstractToken

object --+
         |
        AbstractToken

Known Subclasses:
CompositeToken, Default, Defer, EncloseHelper, IndentHelper, Lookahead, Negative, Omit, Raw, RE, Skip, TakeToken, TokenSequence, TokenSeries, TokenWrapper

Base class from which most tokens defined in this module derive. Subclassing this is not required for writing tokens, since they can be any callable with certain semantics, but this class provides several useful services for creating reusable token classes, such as callback support and convenient operator overloading.
Method Summary
  __init__(self, desc, callback, to, as, name)
  __add__(self, other)
Allows you to construct TokenSequences with the + operator.
  __copy__(self)
  __imod__(self, val)
  __invert__(self)
  __mod__(self, val)
  __mul__(self, val)
Allows you to construct TokenSeries with the * operator.
  __or__(self, other)
Allows you to construct CompositeTokens with the | operator.
  __repr__(self)
  __rmul__(self, val)
Allows you to construct TokenSeries with the * operator.
  __rshift__(self, callback)
Convenience overloading for setting the callback of a token whose initializer you do not call directly, such as the result of combining tokens with + or |.
  __setattr__(self, name, value)
  __str__(self)
  __sub__(self, other)
Allows you to construct TokenSequences with the - operator, automatically padded with Whitespace.
  __xor__(self, message)
Overloading for setting the failMessage of a token.
    Inherited from object
  __delattr__(...)
x.__delattr__('name') <==> del x.name
  __getattribute__(...)
x.__getattribute__('name') <==> x.name
  __hash__(x)
x.__hash__() <==> hash(x)
  __new__(T, S, ...)
T.__new__(S, ...) -> a new object with type S, a subtype of T
  __reduce__(...)
helper for pickle
  __reduce_ex__(...)
helper for pickle

Instance Variable Summary
NoneType callback: An optional callable which, if not None, will be called whenever an instance matches successfully.
  desc: The generic "description" variable which stores the "essence" of any given instance.
NoneType to: An optional callable which, if not None, will be called in the same manner as a callback (after any callback and before returning to the parser instance), but will be passed only one argument: the data matched (or returned by the callback, if any).

Class Variable Summary
NoneType failMessage = None                                                                  
NoneType name = None                                                                  

Method Details

__add__(self, other)
(Addition operator)

Allows you to construct TokenSequences with the + operator.

__mul__(self, val)

Allows you to construct TokenSeries with the * operator. Operand can be:
  • int (a series of exactly this many)
  • (int, ) (a series of at least this many)
  • (x:int, y:int) a series of x to y
The constant Inf can be used in some of these -- * Inf yields a 0--infinity series, and * (x, Inf) yields an x--infinity series.

__or__(self, other)
(Or operator)

Allows you to construct CompositeTokens with the | operator.

__rmul__(self, val)

Allows you to construct TokenSeries with the * operator. Operand can be:
  • int (a series of exactly this many)
  • (int, ) (a series of at least this many)
  • (x:int, y:int) a series of x to y
The constant Inf can be used in some of these -- * Inf yields a 0--infinity series, and * (x, Inf) yields an x--infinity series.

__rshift__(self, callback)

Convenience overloading for setting the callback of a token whose initializer you do not call directly, such as the result of combining tokens with + or |.
Parameters:
callback - An AbstractToken-compatible callback.
           (type=callable)
Returns:
A copy of self with the callback property set to callback.

__sub__(self, other)
(Subtraction operator)

Allows you to construct TokenSequences with the - operator, automatically padded with Whitespace.

I realize it's a bit weird to use the - operator for this, but the main motivation is giving it the same precedence as +. Still, you can read it as a sort of "blank" (which is what the left and right tokens are being joined by), instead of "minus".

__xor__(self, message)

Overloading for setting the failMessage of a token.
Parameters:
message - The message to be raised with ParseError if this token fails to match.
           (type=str)
Returns:
A copy of self with the failMessage property set to callback.

Instance Variable Details

callback

An optional callable which, if not None, will be called whenever an instance matches successfully. It may take one, two, or three parameters, depending on its needs. If one, it will be passed whatever data the token matched (i.e. whatever it would normally have returned upon being called). If two, it will be passed the ZestyParser instance and the data. If three, it will be passed the parser, the data, and the what the parser's cursor was when this token started matching. Callbacks may raise NotMatched or ParseError with the usual behaviour. They should also return a value, which will be returned to the calling ZestyParser instance.
Type:
NoneType
Value:
None                                                                  

desc

The generic "description" variable which stores the "essence" of any given instance. Subclasses use this as needed.

to

An optional callable which, if not None, will be called in the same manner as a callback (after any callback and before returning to the parser instance), but will be passed only one argument: the data matched (or returned by the callback, if any). Its main purpose is to allow you to concisely do things like Token('[0-9]+', group=0, to=int) -- the builtin callable int will be passed the text matched by the regex, so the token will ultimately return an integer instead of a string or a regex match object. You can also use this property with AHT types, for more complex multi-stage parsing. See the n3.py and n3rdflib.py examples for a demonstration of this. (In previous versions, this was passed to the initializer as as, but this is deprecated because as will become a reserved word in Python 2.6. Change your code to use {to}.)
Type:
NoneType
Value:
None                                                                  

Class Variable Details

failMessage

Type:
NoneType
Value:
None                                                                  

name

Type:
NoneType
Value:
None                                                                  

Generated by Epydoc 2.1 on Thu Apr 26 01:32:22 2007 http://epydoc.sf.net