Lines Matching defs:token
3001 ** complete if it ends with a semicolon token and is not a prefix of a
3004 ** independent tokens (they are part of the token in which they are
4203 ** ^If the most recent error references a specific token in the input
4205 ** of the start of that token. ^The byte offset returned by
4207 ** ^If the most recent error does not reference a specific token in the input
12814 ** to the column in which it occurs and *piOff the token offset of the
12815 ** first token of the phrase. Returns SQLITE_OK if successful, or an error
13052 ** followed by a "*" character, indicating that the last token
13053 ** returned by the tokenizer will be treated as a token prefix.
13061 ** For each token in the input string, the supplied callback xToken() must
13064 ** arguments are a pointer to a buffer containing the token text, and the
13065 ** size of the token in bytes. The 4th and 5th arguments are the byte offsets
13067 ** which the token is derived within the input.
13073 ** FTS5 assumes the xToken() callback is invoked for each token in the
13096 ** <ol><li> By mapping all synonyms to a single token. In this case, using
13098 ** same token for inputs "first" and "1st". Say that token is in
13115 ** first token in the MATCH query and FTS5 effectively runs a query
13127 ** provides multiple synonyms for each token. So that when a
13136 ** FTS index corresponding to both forms of the first token.
13141 ** is considered to supply a synonym for the previous token. For example,
13154 ** xToken() is called. Multiple synonyms may be specified for a single token
13157 ** single token.
13163 ** token "first" is substituted for "1st" by the tokenizer, then the query:
13168 ** will not match documents that contain the token "1st" (as the tokenizer
13179 ** token "1st", but not "first" (assuming the tokenizer is not able to
13203 const char *pToken, /* Pointer to buffer containing token */
13204 int nToken, /* Size of token in bytes */
13205 int iStart, /* Byte offset of token within input text */
13206 int iEnd /* Byte offset of end of token within input text */
13219 #define FTS5_TOKEN_COLOCATED 0x0001 /* Same position as prev. token */
15932 void *token; /* id that may be used to recursive triggers */
18165 ** Each token coming out of the lexer is an instance of
18175 const char *z; /* Text of the token. Not NULL-terminated! */
18176 unsigned int n; /* Number of characters in this token */
18245 ** Expr.op is the opcode. The integer parser token codes are reused
18353 int iOfst; /* else: start of token from start of statement */
18380 #define EP_DblQuoted 0x000080 /* token.z was originally in "..." */
19184 Token sLastToken; /* The last token parsed */
22586 void *token; /* Copy of SubProgram.token */
30869 ** The z string points to the first character of a token that is
30892 ** If pExpr has a byte offset for the start of a token, record that as
31923 /* Expressions of the form: CAST(pLeft AS token) */
33873 ** If the input token p is quoted, try to adjust the token to remove
82251 ** token (i.e. "5", "5.1", "'a string'"). If the expression can
90154 int n; /* Length of a token prefix */
90155 int nToken; /* Length of the parameter token */
97489 ** single trigger all have the same value for the SubProgram.token
97492 t = pProgram->token;
97493 for(pFrame=p->pFrame; pFrame && pFrame->token!=t; pFrame=pFrame->pParent);
97540 pFrame->token = pProgram->token;
106639 ** If dequote is true, then the token (if it exists) is dequoted.
106641 ** parameter is ignored if pToken is NULL or if the token does not
106646 ** can be translated into a 32-bit integer, then the token is not
106696 ** Allocate a new expression node from a zero-terminated token that has
107130 ** dupedExprNodeSize() Size of Expr + space for token
107131 ** dupedExprSize() Expr + token + subtree components
107791 ** statement handled by the parser. And so no token need be added
107792 ** to the token-map. */
110109 /* Expressions of the form: CAST(pLeft AS token) */
110999 ** This code depends on the fact that certain token values (ex: TK_EQ)
113007 ** the token that generated that element. The parse tree element
113019 ** to look up the actual token value that created some element in
113023 const void *p; /* Parse tree element created by token t */
113024 Token t; /* The token that created parse tree element p */
113046 ** rename-token list.
113048 ** 2. Dereferences each pointer in the rename-token list.
113085 ** the token pToken.
113118 ** with parse tree element pFrom. This function remaps the associated token
113403 ** corresponding rename-token from Parse object pParse and add it
113428 ** that matches the string in zOld, extract the corresponding rename-token
113498 ** This function edits SQL statement zSql, replacing each token identified
113510 const char *zNew, /* New token text */
113511 int bQuote /* True to always quote token */
113554 int iOff; /* Offset of token to replace in zOut */
113569 /* Dequote the double-quoted token. Then requote it again, this time
113571 ** original token within the input SQL was a single quote ('), then
113573 ** token. This is so that (SELECT "string"'alias') maps to
114449 ** table being edited, and token pName the name of the column to drop.
118373 ** Given a token, return a string that consists of the text of that
118374 ** token. Space to hold the returned string
118379 ** surround the body of the token are removed.
118430 ** The token *pName contains the name of a database (either "main" or
118456 ** This routine sets the *ppUnqual pointer to point at the token (pName1 or
120090 Token *pCons, /* The ',' token after the last column defn. */
120432 Token *pBegin, /* The CREATE token that begins the statement */
120433 Token *pName1, /* The token that holds the name of the view */
120434 Token *pName2, /* The token that holds the name of the view */
120472 ** This will force all the Expr.token.z values to be dynamically
120735 ** token for additional information.
121380 Token *pStart, /* The CREATE token that begins this statement */
122407 ** alias token. If the term is a subquery, then pSubquery is the
128365 Token tOld = { "old", 3 }; /* Literal "old" token */
128366 Token tNew = { "new", 3 }; /* Literal "new" token */
134629 Token *pId; /* Pointer to <id> token */
147216 TriggerStep *pStep /* The trigger containing the target token */
147607 pProgram->token = (void *)pTrigger;
150464 ** The parser calls this routine when it sees the first token
150474 ** The parser calls this routine for each token after the first token
150579 ** columns of the table to see if any of them contain the token "hidden".
150580 ** If so, set the Column COLFLAG_HIDDEN flag and remove the token from
166386 /* Construct a new Expr object from a single token */
166460 ***************** Begin token definitions *************************************/
166647 /**************** End token definitions ***************************************/
166664 ** token.
166774 ** current state and lookahead token. These tables are used to implement
166782 ** token onto the stack and goto state N.
167443 ** appears in the grammar, then ID becomes a fallback token for X, Y,
167445 ** but it does not parse, the type of the token is changed to ID and
167647 ** + The value of the token stored at this level of the stack.
167648 ** (In other words, the "major" token.)
167652 ** It is sometimes called the "minor" token.
167660 YYCODETYPE major; /* The major token value. This is the code
167661 ** number for the token at this stack level */
167662 YYMINORTYPE minor; /* The user-supplied minor token value. This
167663 ** is the value of the token */
168684 ** If there is a destructor routine associated with the token which
168746 ** is in state X and has a lookahead token Y. In a well-tested
168783 ** look-ahead token iLookAhead.
168786 YYCODETYPE iLookAhead, /* The look-ahead token */
168807 YYCODETYPE iFallback; /* Fallback token */
168848 ** look-ahead token iLookAhead.
168852 YYCODETYPE iLookAhead /* The look-ahead token */
168925 YYCODETYPE yyMajor, /* The major token to shift in */
168926 sqlite3ParserTOKENTYPE yyMinor /* The minor token to shift in */
169788 ** access to the lookahead token (if any). The yyLookahead will be YYNOCODE
169789 ** if the lookahead token has already been consumed. As this procedure is
169796 int yyLookahead, /* Lookahead token, or YYNOCODE if none */
169797 sqlite3ParserTOKENTYPE yyLookaheadToken /* Value of the lookahead token */
171387 int yymajor, /* The major type of the error token */
171388 sqlite3ParserTOKENTYPE yyminor /* The minor type of the error token */
171434 ** The second argument is the major token number. The third is
171435 ** the minor token. The fourth optional argument is whatever the
171442 ** <li> The major token number.
171443 ** <li> The minor token number.
171452 int yymajor, /* The major token code number */
171453 sqlite3ParserTOKENTYPE yyminor /* The value for the token */
171560 ** grammar defines an error token "ERROR".
171584 fprintf(yyTraceFILE,"%sDiscard input token %s\n",
171626 ** * Report an error message, and throw away the input token.
171628 ** * If the input token is $, then fail the parse.
171664 ** Return the fallback token corresponding to canonical token iToken, or
171817 ** it is a keyword. If it is a keyword, the token code of that keyword is
172119 ** return the integer n (the length of the token). */
172358 ** Return the id of the next token in string (*pz). Before returning, set
172359 ** (*pz) to point to the byte following the parsed token.
172383 ** whether the token should be treated as a keyword or an SQL identifier.
172396 ** * the following token is an identifier, or a keyword that can fallback
172398 ** * the token after than one is TK_AS.
172402 ** * the previous token was TK_RP, and
172403 ** * the next token is either TK_LP or an identifier.
172407 ** * the previous token was TK_RP, and
172408 ** * the next token is TK_LP.
172434 ** Return the length (in bytes) of the token that begins at z[0].
172435 ** Store the token type in *tokenType before returning.
172440 ** of the token. See the comment on the CC_ defines
172674 /* This token started out using characters that can appear in keywords,
172734 int n = 0; /* Length of the next token token */
172735 int tokenType; /* type of the next token */
172736 int lastTokenParsed = -1; /* type of the previous token */
172828 sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"", &x);
172907 int n; /* length of current token */
172908 int tokenType; /* type of current token */
172909 int prevType = 0; /* Previous non-whitespace token */
173110 ** (2) tkOTHER Any other SQL token.
173126 u8 token; /* Value of the next token */
173167 token = tkSEMI;
173175 token = tkWS;
173180 token = tkOTHER;
173187 token = tkWS;
173192 token = tkOTHER;
173197 token = tkWS;
173204 token = tkOTHER;
173214 token = tkOTHER;
173226 token = tkOTHER;
173231 token = tkCREATE;
173233 token = tkOTHER;
173239 token = tkTRIGGER;
173241 token = tkTEMP;
173243 token = tkTEMP;
173245 token = tkOTHER;
173251 token = tkEND;
173255 token = tkEXPLAIN;
173259 token = tkOTHER;
173264 token = tkOTHER;
173272 token = tkOTHER;
173277 state = trans[state][token];
178764 ** given term. Doclists hold docids and associated token positions.
178790 ** memory. A "position" is an index of a token in the token stream
179146 ** Retrieve the next token from the tokenizer cursor pCursor. This
179152 ** normalized version of the token (i.e. after any case-folding and/or
179154 ** of this buffer in bytes. The input text that generated the token is
179157 ** byte of the token in the input buffer. *piEndOffset should be set
179158 ** to the index of the first byte just past the end of the token in
179171 const char **ppToken, int *pnBytes, /* OUT: Normalized text for token */
179172 int *piStartOffset, /* OUT: Byte offset of token in input buffer */
179173 int *piEndOffset, /* OUT: Byte offset of end of token in input buffer */
179675 ** sequence. A single token is the base case and the most common case.
179680 char *z; /* Text of the token */
179682 int isPrefix; /* True if token ends with a "*" character */
179683 int bFirst; /* True if token must appear at position 0 */
179688 Fts3DeferredToken *pDeferred; /* Deferred token object for this token */
179689 Fts3MultiSegReader *pSegcsr; /* Segment-reader for this token */
179708 Fts3PhraseToken aToken[1]; /* One entry for each token in the phrase */
181951 ** Then the position list for this row for token 'b' would consist of:
181961 ** when the *pp1 token appears before the *pp2 token, but not more than nToken
181968 int nToken, /* Maximum difference in token positions */
182083 ** lists are the offsets of the last token in each phrase (tokens "1" and "2"
182092 int nRight, /* Maximum difference in token positions */
182093 int nLeft, /* Maximum difference in token positions */
182398 ** a token in position 0 (of any column). If so, it writes argument iDelta
183790 ** Allocate an Fts3MultiSegReader for each token in the expression headed
183837 ** Arguments pList/nList contain the doclist for token iToken of phrase p.
183951 int iPrev = -1; /* Token number of previous deferred token */
184269 ** Token pToken is an incrementally loaded token that is part of a
184270 ** multi-token phrase. Advance it to the next matching document in the
184279 Fts3Phrase *pPhrase, /* Phrase to advance token of */
184280 int iToken, /* Specific token to advance */
184315 ** * features at least one token that uses an incremental doclist, and
184340 ** one incremental token. In which case the bIncr flag is set. */
184361 /* Advance the iterator for each token in the phrase once. */
184462 ** each phrase in the expression (subject to deferred token processing).
184497 ** token in the FTS expression.
184507 Fts3Phrase *pPhrase; /* The phrase the token belongs to */
184508 int iToken; /* Position of token in phrase */
184509 Fts3PhraseToken *pToken; /* The token itself */
184512 int iCol; /* The column the token must match */
184634 ** called on each token to defer. Otherwise, an SQLite error code is
184656 ** causing any problems, the deferred token optimization is completely
184664 ** only 1 token, exit early. No tokens to defer in this case. */
184680 ** to retrieve the entire doclist for the token from the full-text index.
184683 ** a. The cheapest token in the entire query (i.e. the one visited by the
184686 ** b. Part of a multi-token phrase.
184688 ** After each token doclist is loaded, merge it with the others from the
184691 ** any phrase doclist for which 1 or more token doclists have been loaded.
184695 ** Then, for each token, defer it if loading the doclist would result in
184702 Fts3TokenAndCost *pTC = 0; /* Set to cheapest remaining token. */
184704 /* Set pTC to point to the cheapest remaining token. */
184730 /* Either this is the cheapest token in the entire query, or it is
184731 ** part of a multi-token phrase. Either way, the entire doclist will
184776 /* Allocate a MultiSegReader for each token in the expression. */
184917 ** where "*" may match any single token. The position list in this case
185271 ** token. Then, see if this row is really a match, considering deferred
185283 /* Free the position-lists accumulated for each deferred token above. */
186350 ** c) The old syntax supports the "-" token qualifier. This is not
186484 ** Extract the next token from buffer z (length n) using the tokenizer
186487 ** single token and set *ppExpr to point to it. If the end of the buffer is
186488 ** reached before a token is found, set *ppExpr to zero. It is the
186611 ** Fts3PhraseToken structures token buffers are all stored as a single
186792 ** user has supplied a token such as "ORacle". Continue.
186831 /* If control flows to this point, this must be a regular token, or
186832 ** the end of the input. Read a regular token using the sqlite3_tokenizer
186834 ** column specifier for the token.
186837 ** with a quoted phrase, only with a single token. Not sure if this was
188051 int iToken; /* index of next token to be returned */
188052 char *zToken; /* storage for current token */
188576 ** Characters that can be part of a token. We assume any character
188578 ** part of a token. In other words, delimiters all must have
188592 ** Extract the next token from a tokenization cursor. The cursor must
188597 const char **pzToken, /* OUT: *pzToken is the token text */
188598 int *pnBytes, /* OUT: Number of bytes in token */
188599 int *piStartOffset, /* OUT: Starting offset of token */
188600 int *piEndOffset, /* OUT: Ending offset of token */
188601 int *piPosition /* OUT: Position integer of token */
188798 /* Find the start of the next token. */
188914 ** list. For each token in the <input-string>, three elements are
188915 ** added to the returned list. The first is the token position, the
188916 ** second is the token text (folded, stemmed, etc.) and the third is the
188917 ** substring of <input-string> associated with the token. For example,
189234 int iToken; /* index of next token to be returned */
189235 char *pToken; /* storage for current token */
189343 ** Extract the next token from a tokenization cursor. The cursor must
189348 const char **ppToken, /* OUT: *ppToken is the token text */
189349 int *pnBytes, /* OUT: Number of bytes in token */
189350 int *piStartOffset, /* OUT: Starting offset of token */
189351 int *piEndOffset, /* OUT: Ending offset of token */
189352 int *piPosition /* OUT: Position integer of token */
189448 ** CREATE TABLE <tbl>(input, token, start, end, position)
189456 ** statement and returns one row for each token in the result. With
189460 ** token: A token from the input.
189461 ** start: Byte offset of the token within the input <string>.
189463 ** token within the input string.
189464 ** pos: Token offset of token within input.
189493 const char *zToken; /* Current 'token' value */
189572 #define FTS3_TOK_SCHEMA "CREATE TABLE x(input, token, start, end, position)"
189811 /* CREATE TABLE x(input, token, start, end, position) */
190001 Fts3PhraseToken *pToken; /* Pointer to corresponding expr token */
190002 int iCol; /* Column token must occur in */
190655 ** happen if there was no previous entry for this token.
190700 ** zText==0. In this case, add zero token entries to the hash table and
195221 char const *zToken; /* Buffer containing token */
195222 int nToken = 0; /* Number of bytes in token */
195224 int iPos = 0; /* Position of token in zText */
195262 ** The integrity-check works as follows. For each token and indexed token
195268 ** + The token (or token prefix) text itself,
195401 char const *zToken; /* Buffer containing token */
195402 int nToken = 0; /* Number of bytes in token */
195404 int iPos = 0; /* Position of token in zText */
195461 ** Add an entry for token pToken to the pCsr->pDeferred list.
195466 int iCol /* Column that token must appear in (or -1) */
195772 int iCurrent; /* First token of current snippet */
195786 int iPos; /* Index of first token in snippet */
196124 int *piToken, /* OUT: First token of proposed snippet */
196129 int iStart = pIter->iCurrent; /* First token of snippet */
196363 int *piPos, /* IN/OUT: First token of snippet */
196437 int iCurrent = 0; /* Current token number of document */
196438 int iEnd = 0; /* Byte offset of end of current token */
196440 int iPos = pFragment->iPos; /* First token of snippet */
196455 /* Open a token cursor on the document. */
196465 int iBegin = 0; /* Offset in zDoc of start of token */
196466 int iFin = 0; /* Offset in zDoc of end of token */
196483 /* Special case - the last token of the snippet is also the last token
196485 ** of the previous token and the end of the document to the output.
197372 int iMinPos = 0x7FFFFFFF; /* Position of next token */
197373 TermOffset *pTerm = 0; /* TermOffset associated with next token */
197553 int iToken; /* Index of next token to be returned */
197554 char *zToken; /* storage for current token */
197575 ** token characters (if bAlnum==1).
197666 ** considered a token character (not a separator).
197770 ** Extract the next token from a tokenization cursor. The cursor must
197777 int *piStart, /* OUT: Starting offset of token */
197778 int *piEnd, /* OUT: Ending offset of token */
197779 int *piPos /* OUT: Position integer of token */
197790 /* Scan past any delimiter characters before the start of the next token.
204657 ** Return the length of a token
205511 ** return non-zero on success and zero if the next token is not a number.
208082 ** Extract the next token from a tokenization cursor.
208086 const char **ppToken, /* OUT: *ppToken is the token text */
208087 int *pnBytes, /* OUT: Number of bytes in token */
208088 int *piStartOffset, /* OUT: Starting offset of token */
208089 int *piEndOffset, /* OUT: Ending offset of token */
208090 int *piPosition /* OUT: Position integer of token */
221475 ** to the column in which it occurs and *piOff the token offset of the
221476 ** first token of the phrase. Returns SQLITE_OK if successful, or an error
221713 ** followed by a "*" character, indicating that the last token
221714 ** returned by the tokenizer will be treated as a token prefix.
221722 ** For each token in the input string, the supplied callback xToken() must
221725 ** arguments are a pointer to a buffer containing the token text, and the
221726 ** size of the token in bytes. The 4th and 5th arguments are the byte offsets
221728 ** which the token is derived within the input.
221734 ** FTS5 assumes the xToken() callback is invoked for each token in the
221757 ** <ol><li> By mapping all synonyms to a single token. In this case, using
221759 ** same token for inputs "first" and "1st". Say that token is in
221776 ** first token in the MATCH query and FTS5 effectively runs a query
221788 ** provides multiple synonyms for each token. So that when a
221797 ** FTS index corresponding to both forms of the first token.
221802 ** is considered to supply a synonym for the previous token. For example,
221815 ** xToken() is called. Multiple synonyms may be specified for a single token
221818 ** single token.
221824 ** token "first" is substituted for "1st" by the tokenizer, then the query:
221829 ** will not match documents that contain the token "1st" (as the tokenizer
221840 ** token "1st", but not "first" (assuming the tokenizer is not able to
221864 const char *pToken, /* Pointer to buffer containing token */
221865 int nToken, /* Size of token in bytes */
221866 int iStart, /* Byte offset of token within input text */
221867 int iEnd /* Byte offset of end of token within input text */
221880 #define FTS5_TOKEN_COLOCATED 0x0001 /* Same position as prev. token */
222347 ** specified token or token prefix.
222358 ** The various operations on open token or token prefix iterators opened
222389 ** For an insert, it must be called once for each token in the new document.
222391 ** unique token in the document with an iCol value less than zero. The iPos
222396 int iCol, /* Column token appears in (-ve -> delete) */
222397 int iPos, /* Position of token within column */
222530 int iCol, /* Column token appears in (-ve -> delete) */
222531 int iPos, /* Position of token within column */
222860 ***************** Begin token definitions *************************************/
222878 /**************** End token definitions ***************************************/
222895 ** token.
222990 ** current state and lookahead token. These tables are used to implement
222998 ** token onto the stack and goto state N.
223097 ** appears in the grammar, then ID becomes a fallback token for X, Y,
223099 ** but it does not parse, the type of the token is changed to ID and
223116 ** + The value of the token stored at this level of the stack.
223117 ** (In other words, the "major" token.)
223121 ** It is sometimes called the "minor" token.
223129 fts5YYCODETYPE major; /* The major token value. This is the code
223130 ** number for the token at this stack level */
223131 fts5YYMINORTYPE minor; /* The user-supplied minor token value. This
223132 ** is the value of the token */
223420 ** If there is a destructor routine associated with the token which
223482 ** is in state X and has a lookahead token Y. In a well-tested
223519 ** look-ahead token iLookAhead.
223522 fts5YYCODETYPE iLookAhead, /* The look-ahead token */
223543 fts5YYCODETYPE iFallback; /* Fallback token */
223584 ** look-ahead token iLookAhead.
223588 fts5YYCODETYPE iLookAhead /* The look-ahead token */
223661 fts5YYCODETYPE fts5yyMajor, /* The major token to shift in */
223662 sqlite3Fts5ParserFTS5TOKENTYPE fts5yyMinor /* The minor token to shift in */
223770 ** access to the lookahead token (if any). The fts5yyLookahead will be fts5YYNOCODE
223771 ** if the lookahead token has already been consumed. As this procedure is
223778 int fts5yyLookahead, /* Lookahead token, or fts5YYNOCODE if none */
223779 sqlite3Fts5ParserFTS5TOKENTYPE fts5yyLookaheadToken /* Value of the lookahead token */
223995 int fts5yymajor, /* The major type of the error token */
223996 sqlite3Fts5ParserFTS5TOKENTYPE fts5yyminor /* The minor type of the error token */
224040 ** The second argument is the major token number. The third is
224041 ** the minor token. The fourth optional argument is whatever the
224048 ** <li> The major token number.
224049 ** <li> The minor token number.
224058 int fts5yymajor, /* The major token code number */
224059 sqlite3Fts5ParserFTS5TOKENTYPE fts5yyminor /* The value for the token */
224166 ** grammar defines an error token "ERROR".
224190 fprintf(fts5yyTraceFILE,"%sDiscard input token %s\n",
224232 ** * Report an error message, and throw away the input token.
224234 ** * If the input token is $, then fail the parse.
224270 ** Return the fallback token corresponding to canonical token iToken, or
224328 int iStart; /* First token in coalesced phrase instance */
224329 int iEnd; /* Last token in coalesced phrase instance */
224396 int iPos; /* Current token offset in zIn[] */
224397 int iRangeStart; /* First token to include */
224398 int iRangeEnd; /* If non-zero, last token to include */
224434 const char *pToken, /* Buffer containing token */
224435 int nToken, /* Size of token in bytes */
224436 int iStartOff, /* Start offset of token */
224437 int iEndOff /* End offset of token */
224536 int iPos; /* Current token position */
224539 int *aFirst; /* Array of first token in each sentence */
224565 ** For each such token, an entry is added to the SFinder.aFirst[] array.
224570 const char *pToken, /* Buffer containing token */
224571 int nToken, /* Size of token in bytes */
224572 int iStartOff, /* Start offset of token */
224573 int iEndOff /* End offset of token */
224674 int iBestStart = 0; /* First token of best snippet */
226102 ** The callback is invoked once for each token in the input text. The
226106 ** const char *pToken // Pointer to buffer containing token
226107 ** int nToken // Size of token in bytes
226108 ** int iStart // Byte offset of start of token within input text
226109 ** int iEnd // Byte offset of end of token within input text
226110 ** int iPos // Position of token in input (first token is 0)
226393 ** All token types in the generated fts5parse.h file are greater than 0.
226464 u8 bFirst; /* True if token must be first in column */
226482 ** One or more phrases that must appear within a certain token distance of
226522 ** Read the first token from the nul-terminated string at *pz.
226600 Fts5Token token;
226602 int t; /* Next token type */
226615 t = fts5ExprGetToken(&sParse, &z, &token);
226616 sqlite3Fts5Parser(pEngine, t, token, &sParse);
227409 int i, j; /* Phrase and token index, respectively */
227943 ** Set the "bFirst" flag on the first token of the phrase passed as the
228035 const char *pToken, /* Buffer containing token */
228036 int nToken, /* Size of token in bytes */
228037 int iUnused1, /* Start offset of token */
228038 int iUnused2 /* End offset of token */
228135 ** This function is called by the parser to process a string token. The
228177 /* This happens when parsing a token or quoted phrase that contains
228178 ** no token characters at all. (e.g ... MATCH '""'). */
228248 /* This happens when parsing a token or quoted phrase that contains
228249 ** no token characters at all. (e.g ... MATCH '""'). */
228286 ** is expected. If token pTok does not contain "NEAR", store an error
228403 char *z; /* Dequoted copy of token p */
229240 ** fts5 tables. This implies that all phrases must be at most 1 token
229286 const char *pToken, /* Pointer to buffer containing token */
229287 int nToken, /* Size of token in bytes */
229288 int iUnused1, /* Byte offset of token within input text */
229289 int iUnused2 /* Byte offset of end of token within input text */
229688 int iCol, /* Column token appears in (-ve -> delete) */
229689 int iPos, /* Position of token within column */
229690 char bByte, /* First byte of token */
235517 ** For an insert, it must be called once for each token in the new document.
235519 ** unique token in the document with an iCol value less than zero. The iPos
235524 int iCol, /* Column token appears in (-ve -> delete) */
235525 int iPos, /* Position of token within column */
235556 ** specified token or token prefix.
238859 const char *pUnused, /* Buffer containing token */
238860 int nUnused, /* Size of token in bytes */
238861 int iUnused1, /* Start offset of token */
238862 int iUnused2 /* End offset of token */
240127 const char *pToken, /* Buffer containing token */
240128 int nToken, /* Size of token in bytes */
240129 int iUnused1, /* Start offset of token */
240130 int iUnused2 /* End offset of token */
240576 const char *pToken, /* Buffer containing token */
240577 int nToken, /* Size of token in bytes */
240578 int iUnused1, /* Start offset of token */
240579 int iUnused2 /* End offset of token */
240946 ** For tokenizers with no "unicode" modifier, the set of token characters
241069 /* Count the token characters */
241088 /* Invoke the token callback */
241158 unsigned char aTokenChar[128]; /* ASCII range token characters */
241165 unsigned char aCategory[32]; /* True for token char categories */
241352 ** passed as the first argument, codepoint iCode is considered a token
241384 ** then the next token. */
241433 ** it is a token character, or break out of the loop if it is not. */
241443 /* An ascii-range separator character. End of token. */
241457 /* Invoke the token callback */
243116 aAscii[0] = 0; /* 0x00 is never a token character */