What is the exact definition of "Token?"

Question

I have problem to catch the real meaning of the term 'Token.'

In terms of software development, can you define it generically? (Does it have different meanings in terms of different contexts and languages?)

Thanks!

Solution 1

If you google define:token, you get (amongst others) the following two definitions that seem applicable:

• an individual instance of a type of symbol; "the word 'error' contains three tokens of `r'"
• something serving as a sign of something else

If you combine these two, you will land somewhere near what is commonly meant when talking about tokens in programming; a symbol representing something. Pretty vague, yes, but then it's used in many different contexts.

One example: you have an authentication system where a user logs on. When the system has authenticated the user, instead of repeating this process for every request, a token is created that represents the fact that the user is authenticated. This token is then used in subsequent requests. In this case the something is the fact the the user is authenticated, and the token represents this fact.

Solution 2

Tokens are: identifiers, keywords, literals, operators, and punctuators. But we can't consider White spaces and comments as tokens, though they act as separators for tokens.

Solution 3

In the compiler Lexical analyzer (or scanner ) : Reads the input stream and fuses characters of the source text into tokens of the language. Token : sequence of characters having a collective meaning. The character sequence forming a token is called the Lexeme.

this example might help
Consider the following assignment statement
newvalue = oldvalue + rate * 60
The lexical analyzer will generate the following tokens.

Token       Lexeme
Identifier  newvalue
assignop    =
Identifier  oldvalue
addop       +
identifier  rate
mulop       *
number      6

Solution 4

One of the meaning in terms of Software Development is the authentication signature which is issued by a server for a defined time interval

Solution 5

A token is a single element of a programming language. There are 5 token categories :

• Reserved words
• Operators
• Identifiers
• Constants
• Separators

View more solutions

Comments

[BoltClock]

Do you want a general definition or a definition specific to .NET?

[vel]

If it has a specific meaning for .Net, I would appreciate if you could include that as well. Thanks!

[vel]

Then if I use the word 'something' instead of Token it won't be wrong. :) I am afraid, I really don't get the point :(

[vel]

@Frederik: Thanks Frederik, good example. So, basically it represents a "specific instance" in the OOP context maybe, right?

[Fredrik Mörk]

@burak: again; depends on context, but yes in a way you could say that. A "specific instance" is "something" :)

[Admin]

Very nice definition of token. Just a question : Why should digits and numbers be considered as token? Isn't number enough?

[Fredrik Mörk]

@Ehsan I would say that a digits (such as 1 or 7) could be considered tokens with which you can produce numbers (such as 17). Depending on context, 1 or 7 could of course be considered to be numbers as well, consisting of single tokens.