Wednesday, September 25, 2013
It is the process of dividing a string into sequences
of contiguous characters separated by any of the characters that are part of
delimiters.Basically tokenization word is derived from word "Token".
Token means a little part. For example if there is a string like that,
string st[]={"Programming Is Everything"};
Then if we divide a string into its parts like that
Programming // This is called a Token
Is // This is called a Token
Every thing // This is called a Token
The function which is used for this is "Strtok(sentence, " ")"
Here a simple program which can help you to understand this.
Program:
#include<iostream>
#include<conio.h>
#include<string.h>
using namespace std;
int main()
{
char arr[20]={"This is my book"};
char *ptr=arr;
ptr=strtok(arr, " "); // if there is a comma Between a sentence then we use a ","
cout<<ptr;
while(ptr!="Null")
{
ptr=strtok('\0', " ");
cout<<ptr<<endl;
}
getch();
}
.jpg)