Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Pinguino4550 + X.4] use of strtok() function
17-06-2014, 09:58 AM,
[Pinguino4550 + X.4] use of strtok() function

I use following setup: PIC18F4550 with Bootloader_v4.14_18f4550_X20MHz.hex, Pinguino IDE X.4 under Ubuntu.

In my application I try to extract sub-strings from a string separated by special characters (in my case I use character "|"). I found very convenient to use strtok() function, but it is not working as expected.

Below it is my code:

#include <stdio.h>
#include <string.h>

void setup()

void loop()
    char buffer[80];                 //here I keep my message
    char *sep = "|";                 // this is my separator
    char *extracted[30];          //here I want to store my sub-strings
    int i; //this is a general counter

    strcpy(buffer, "w0234|11|2|23042014*");     //this is the string I want to process
    Serial.printf("%s\n", buffer);                       //check that I loaded the string
    buffer[strlen(buffer)-1] = '\0';                     //remove "*" from the end of string
    Serial.printf("\r%s\n", buffer);                     //check the string w/o last position
    i = 0;
    extracted[i] = strtok(buffer, sep);                   //extract the first sub-string
    while (extracted[i]!=NULL) {                           //continue extraction until \NULL is reached
        extracted[i] = strtok(NULL, sep);
        Serial.printf("\r%d\t%s\n", i, extracted[i]);    //print extracted sub-strings
    Serial.printf("\rExtracted data is:\n");
    while (extracted[i]!=NULL) {                           //reprint all the sub-strings extracted
        Serial.printf("\r%d\t\t%s\n", i, extracted[i]);
  delay(2000);                                                 //transmit every 2 s the message on serial port

And here is the output on the serial console:

1 (null)
Extracted data is:
position value
0 w0234

I don't understand why strtok() is behaving like this. According to man page (I hope I understood it correct), strtok() should search into the string for the separator, when it finds the first separator, it should return the first sub-string AND replace the separator by \NULL and return a pointer to the next position after that \NULL. When reached end of the string a.k.a. finding \NULL (my understanding is that strings in C should be terminated with \NULL), strtok() will stop searching for separators and strtok() will return \NULL.

From the output I have on the serial console, seems like my code is finding the first separator (character "|"), it will copy the first sub-string ("w0234") in extracted[0], it will replace "|" with \NULL but not moving to the next position after \NULL, because extracted[1] = (null). At the next iteration, strtok() will find that \NULL (the one replacing "|"), so strtok() will stop executing.

Man pages for C are presenting strtok() as a pretty easy to use function, but perhaps for embedded / Pinguino / SDCC strtok() works in a different way.

Any ideas?
Thank you!
17-06-2014, 12:18 PM,
RE: [Pinguino4550 + X.4] use of strtok() function
No needs to #include anything with Pinguino, the IDE will do it for you.
I tried another example and get the same result as you. So I guess there is a bug in the strok function provided with SDCC.
If you have time to check it, here is the code :

#include <string.h>

char *
strtok (char *str, char *control)
  static char *s = NULL;
  register char *s1;

  if (str)
    s = str;

  if (!s)
    return NULL;

  while (*s)
      if (strchr (control, *s))
  s1 = s;

  while (*s)
      if (strchr (control, *s))
          *s = '\0';
          return s1;

  s = NULL;

  if (*s1)
    return s1;
    return NULL;

If you find the issue, go to the p8/pdl/string.pdl and find this line :
strtok strtok#include <string.h>
replace it with :
strtok pstrtok#include <pstrok.c>
where pstrok.c is where you put the new code and the new pstrok() function.
Save pstrok.c in p8/pinguino/core.
Hope it helps ...
It is easier to complain than it is to do, but it is better to do than it is to complain.
17-06-2014, 05:32 PM, (This post was last modified: 17-06-2014, 05:53 PM by YO2LYP.)
RE: [Pinguino4550 + X.4] use of strtok() function

Thank you for your fast reply!
I followed the steps you mentioned:
-change file string.h
-create file pstrtok.c

I compiled my code from the initial post and I got error 101:

error 101: too many parameters

for each line where strtok() is.
I will figure out later on why.

Meanwhile I found a workaround Smile

i = 0;
    extracted[0] = strtok(buffer, sep);       //first token extraction until first delimiter is found
                                                //(because strtok() works perfect for extracting the first token)
    token = buffer+(strlen(extracted[0]) + 1);   //save the rest of the string without first token
    while (extracted[i]!=NULL) {                   //extract the rest of the tokens
        extracted[i] = strtok(token, sep);           //next token
        token = token + (strlen(extracted[i]) + 1);  //save the rest of the string without the above token

And here is the serial output:

Initial string = w|023411|2|23042014
Delimiter = |
position content
0 w
1 023411
2 2
3 23042014

I hope this is helpful for other newbies like me Smile
Thank you and all the best!

Forum Jump:

Users browsing this thread: 1 Guest(s)