Tokens In Python | The Lexical structure | CBSE Class 12

Introduction 

Let’s see

 

 

As so-called definitions of different books says:

A token in python is the smallest individual unit in a program and sometimes it is also called as lexical unit in Python programming

OR

In a passage of text individual words and punctuation marks are called tokens lexical units or lexical elements in python

But how we, as a beginner how can we understand the tokens in python.

Let’s have an example to understand

Suppose as a beginner you want a Python program to write any sentence on the screen, let’s say your name then you can do it very easily by using a simple sentence on your Python IDLE.

print(“my name”)

Now, you can see

There are some English words like:-

              print     TOKEN 1                     ( KEYWORD TOKEN)

              my name   TOKEN 2                 ( STRING TOKEN)

Here, we can see that there are two English words and we are denoting them as TOKEN 1 and TOKEN 2 and token one is a Keyword type token and Token 2 is an string type  token.

What does it means?

It means the tokens can be have more than one types    YES!!!!!, So now you have a right question to learn about tokens and that is :-

How many types of tokens are there in Python?

Basically a Python program breaks a logical line into a sequence of elementary lexical components and these components are known as tokens and each token corresponds to a substring of the logical line, so to know the different types of tokens that can be used while writing a Python program, please have a look at the following diagram.

So we can say that there are the following types of tokens in Python programming

1. Keyword

Keywords are predefined words in Python that have a specific meaning and cannot be used as identifiers. Python has 35 keywords, and they are reserved for specific uses within the language. These keywords include “if”, “else”, “while”, “for”, “class”, “def”, “return”, and more.

2. Identifiers

Identifiers are names given to variables, functions, classes, and other objects in a Python program. An identifier is a user-defined name that is used to identify a particular variable, function, or object in the program. Identifiers can be composed of letters, numbers, and underscores, but cannot begin with a number. It’s best practice to use descriptive names for your identifiers.

3. Literals

Literals are fixed values in a Python program that are used to represent specific data types. Examples of literals include strings, integers, floating-point numbers, and Boolean values. String literals are enclosed in single or double-quotes, and numeric literals are represented by a sequence of digits.

4. Operators

Operators are symbols that are used to perform operations on variables and literals. Python has several types of operators such as arithmetic operators, assignment operators, comparison operators, logical operators, and more. Some examples of operators include “+” for addition, “-” for subtraction, “*” for multiplication, “/” for division, and “%” for modulo.

5. Punctuators

Punctuations are symbols used to structure a Python program. Punctuations are used to indicate the beginning and end of blocks of code, define function arguments, and more. Examples of punctuations include parentheses, commas, colons, semicolons, and more.

6. Special literal 

Python has a special literal called None that is used to represent the absence of a value. None is an object in Python that is commonly used to indicate that a variable has no value or that a function returns no value. In this blog post, we will explore the special literal – None in Python, its characteristics, and its usage in Python programs.

TRICK : You can remember these types easily by considering all tokens as kILOP’s where

• • K :       stands for keywords

  •  I :        stands for identifiers

  • L :       stands for literals

  • O:        stands for operators

  • P:        stands for punctuators

Now, you have understood that when we write any command or statement in a Python program that will do something for us, and we write the command or statement by combining some tokens and that command is also meaningful to a computer and Python software and Python interpreter can understand it easily.

Exam Time

Here are some important questions with answers on the topic of “Tokens in Python”:

Q1. What are tokens in Python?

Answer: Tokens are the basic building blocks of Python programs. These building blocks are the smallest individual elements that are combined to create a complete Python program. A Python program consists of several types of tokens such as keywords, literals, identifiers, operators, and punctuations.

Q2. What are the different types of tokens in Python?

Answer: Python has several types of tokens that are used to construct a program. These include keywords, identifiers, literals, operators, and punctuations.

Q3. What are keywords in Python?

Answer: Keywords are predefined words in Python that have a specific meaning and cannot be used as identifiers. Python has 35 keywords, and they are reserved for specific uses within the language.

Q4. What are identifiers in Python?

Answer: Identifiers are names given to variables, functions, classes, and other objects in a Python program. An identifier is a user-defined name that is used to identify a particular variable, function, or object in the program.

Q5. What are literals in Python?

Answer:  Literals are fixed values in a Python program that are used to represent specific data types. Examples of literals include strings, integers, floating-point numbers, and Boolean values.

Q6. What are operators in Python?

Answer: Operators are symbols that are used to perform operations on variables and literals. Python has several types of operators such as arithmetic operators, assignment operators, comparison operators, logical operators, and more.

Q7. What are punctuations in Python?

Answer: Punctuations are symbols used to structure a Python program. Punctuations are used to indicate the beginning and end of blocks of code, define function arguments, and more.

Q8. How are tokens used in Python code?

Answer: Tokens are used to define the structure and behavior of a Python program. The keywords, identifiers, literals, operators, and punctuations are combined to form a program that performs a specific task.

Q9. What is the importance of understanding tokens in Python?

Answer: Understanding the different types of tokens and how they’re used is essential to write efficient and effective Python code. By understanding the different types of tokens, we can use them to construct programs that perform specific tasks efficiently.

Q10. What are some best practices for using tokens in Python code?

Answer: Some best practices for using tokens in Python code include using descriptive names for identifiers, following Python naming conventions, using whitespace and indentation to improve readability, and commenting your code to make it easier to understand.

Q11. What is the special literal – None in Python?

Answer: None is a built-in constant in Python that represents the absence of a value. It is used to indicate that a variable has no value or that a function returns no value. None is an object in Python, which means that it has its own data type. The data type of None is NoneType.

Q12. What are the Characteristics of the special literal – None?:

Answer

• None is a built-in constant in Python.
• None is an object in Python.
• None represents the absence of a value.
• The data type of None is NoneType.
• None is falsy, which means that it evaluates to False in Boolean context.

Q13. Explain the Usage of the special literal – None in Python ?:

Answer

1. Initializing variables: None is often used to initialize variables that do not have a value at the time of declaration. For example, if we want to declare a variable but do not have a value for it yet, we can initialize it to None.
2. Default return value: None is often used as the default return value for functions that do not return a value. For example, if we have a function that performs some operation but does not return a value, we can use None as the default return value.
3. Placeholder value: None is often used as a placeholder value in data structures such as lists, dictionaries, and tuples. For example, if we want to create a list with 10 elements but do not have a value for some of the elements yet, we can initialize those elements to None.
4. Comparison: None is often used in comparison operations to check if a variable has no value. For example, if we want to check if a variable is None, we can use the comparison operator ‘is’ or ‘is not’.

 

Now let’s learn these different types of tokens in python one by one in detail.

Keywords in Python – 

Identifiers in Python

Literals in Python

Operators in Python

Punctuators in Python

Special Literal in Python                 

---

Also Read in COMPUTER SCIENCE CLASS 12 CBSE

PYTHON: Data Types  

How to Learn Python free online 

Syllabus : computer science with python

Cloud computing