The need for buffering arises when the input cannot be mapped in memory all at
once: either it is too large, or it comes in a streaming fashion (like reading
from a socket). The usual technique in such cases is to allocate a fixed-sized
memory buffer and process input in chunks that fit into the buffer. When the
current chunk is processed, it is moved out and new data is moved in. In
practice it is somewhat more complex, because lexer state consists not of a
single input position, but a set of interrelated posiitons:

- cursor: the next input character to be read (``YYCURSOR`` in default API or
  ``YYSKIP``/``YYPEEK`` in generic API)

- limit: the position after the last available input character (``YYLIMIT`` in
  default API, implicitly handled by ``YYLESSTHAN`` in generic API)

- marker: the position of the most recent match, if any (``YYMARKER`` in default
  API or ``YYBACKUP``/``YYRESTORE`` in generic API)

- token: the start of the current lexeme (implicit in re2c API, as it is not
  needed for the normal lexer operation and can be defined and updated by the
  user)

- context marker: the position of the trailing context (``YYCTXMARKER`` in
  default API or ``YYBACKUPCTX``/``YYRESTORECTX`` in generic API)

- tag variables: submatch positions (defined with ``/*!stags:re2c*/`` and
  ``/*!mtags:re2c*/`` directives and
  ``YYSTAGP``/``YYSTAGN``/``YYMTAGP``/``YYMTAGN`` in generic API)

Not all these are used in every case, but if used, they must be updated by
``YYFILL``. All active positions are contained in the segment between token and
cursor, therefore everything between buffer start and token can be discarded,
the segment from token and up to limit should be moved to the beginning of
buffer, and the free space at the end of buffer should be filled with new data.
In order to avoid frequent ``YYFILL`` calls it is best to fill in as many input
characters as possible (even though fewer characters might suffice to resume the
lexer). The details of ``YYFILL`` implementation are slightly different
depending on which EOF handling method is used: the case of EOF rule is somewhat
simpler than the case of bounds-checking with padding. Also note that if
``-f --storable-state`` option is used, ``YYFILL`` has slightly different
semantics (desrbed in the section about storable state).
