Pi pipe and filter serial out data

I am trying to build on a pi a simple internet quality measurement system using the ping command, piping it to a C filtering program which will filter data and output on serial port.
ping 8,8,8,8 outputs a string every second of delay through network.
I can pipe this output to a file like:
ping 8.8.8.8 | myfile.txt
I need to filter the data and so I want to pipe the ping output to a C code program.
ping 8.8.8.8 | ./myCprogram
I have tried reading the argc argV data in main() arguments but it does not do what I was hoping.
This is probably obvious to you guys so if you could point me in the right direction that would be nice.

Regards

2 Likes

Hi Clem

For a C program to be used the way you propose, you need to just read the standard input ‘file’.
The argc and argv values are only used to access the command line parameters

i.e. ./myCprog -v 42 would set argc to 2, and setup the argv array with argv[1] == “-v” and argv[2] == “42”
None of these are useful to you

You are trying to use your program as a ‘filter’ – it needs to read stdin, do some work, and (usually) send a result out to stdout. You can also read stdin, and that generate any other output such as write a statistic file, etc etc

The test_ascii.c program below (from a UNIX system) shows a way to read and process data from stdin. It is used like this

cat somefile.txt | ./test_ascii > invalidlinefile.txt

The cat program opens the file somefile.txt ( via argc/argv ), and writes the contents out of its’ stdout. This output is then piped into the stdin of test_ascii. The test_ascii program ignores ‘valid’ ascii characters, but will write out any input line that has an invalid character to its stdout. The > ‘diverts’ the output to a diskfile.

/*
test_ascii.c
stdin == file to be processed
stdout == lines from input with invalid ascii chars
*/
#include <stdio.h>    /* define stdin/stdout etc and getline() function */
#include <stdlib.h>   /* other generally useful functions */
#include <ctype.h>   /* is_xxxxx() character type functions */

ssize_t getline(char ** restrict linep, size_t * restrict linecap, FILE * restrict stream);

int main()
{
  FILE  *fp;         /* future use - open a file to write out 'clean' ascii data */
  char *line = NULL;
  size_t linecap = 0;
  ssize_t linelen;
  int i;

  while ((linelen = getline(&line, &linecap, stdin)) > 0) {
      for( i = 0 ; i< linelen; i++) {
            if(isalpha(line[i])) {           /* a-zA-Z are ok */
                   ;
            } else if(isdigit(line[i])) {    /* digit is ok */
                   ;
            } else if(ispunct(line[i])) {   /* punctuation is ok */
                   ;
            } else if ( line[i] == '\n') {    /* linefeed is ok -- carriage return is NOT ok */
                   ;
            } else if ( line[i] == ' ') {     /* simple space is ok -- tab char is NOT ok */
                   ;
            } else {
                   printf("--- %c %0x -------------- Error Line \n", line[i],line[i]);
                   fwrite(line, linelen, 1, stdout);
            }
      }    /* end for loop */
      free(line);
  }      /* end while */

  return (0);
}

So you need to check the appropriate C idiom for the Pi C compiler (it might be slightly different to the UNIX code libraries) to enable reading the data from the ping program into the stdin of your program. It may be able to do it line by line, or might be ‘better’ character by character.

Hope this helps

Murray

4 Likes

And as a follow-on …

If I wished to use the test_ascii program with more file management options I could setup to provide command line parameters as follows, and manage them via argc/argv :

-o outfilename
-e errorfilename
-i inputfilename

So a ‘complete’ use could be

test_ascii -i testfile.txt -o goodstuff.txt -e junk.txt

The -e option would change the existing fwrite(stdout) to use fopen(junk.txt) , and fwrite() to the new file descriptor instead of stdout.
The -o option would fopen(goodstuff.txt) and then fwrite() to this file descriptor instead to save the good text lines instead of dumping them
And the -i option would direct the program to fopen(testfile.txt) and fread() from that descriptor instead of reading from the stdin descriptor.

Any combination of these options would be valid, including none at all, which would let the program run as previously described.

Cheers
Murray

4 Likes

And a follow on to the follow on …

Just wrote a version as described above with all the command line options … Note it has been tested, but is definitely NOT bulletproof code in that there is almost no error checking for failed file operations etc.

Note: this code uses the C ternary operator as in ( test ? do if true : do if false ) to select the appropriate file pointer to read / write to.

And it will only pass as good characters letters a-zA-Z, digits 0-9, most punctuation characters, space characters but not tabs, and linefeeds but not carriage returns. More commandline options could be added to ‘control’ the valid characters if needed – exercise for the student :crazy_face: and yes I disliked my instructors when they said that too!

/** testascii.c
*
* added command line parameters
*/

/* refer style(9) */
#include <ctype.h>		/* for isXXXX functions */
#include <stdio.h>
#include <stdlib.h>		/* for free() and exit() */
#include <unistd.h>		/* for getopt() */

char           *my_infile;
char           *my_outfile;
char           *my_errfile;

enum TF {
	FALSE = 0,
	TRUE
};

enum TF	f_in = FALSE;
enum TF	f_out = FALSE;
enum TF	f_err = FALSE;

void  usage(void);
ssize_t getline(char ** restrict linep, size_t * restrict linecapp, FILE * restrict stream);

/** main(int argc, char *argv[])
The main() function for setting up and running the testascii process
argc count of commandline args
*argv pointer to char array of command args
returns an  integer status value

Can select operational modes from the command line
Or use as a filter
*/
int
main(int argc, char *argv[])
{
  int		ch;
  FILE  *fp_in, *fp_out, *fp_err;
  char *line = NULL;
  size_t linecap = 0;
  ssize_t linelen;
  int i;
  enum TF wrote_err = FALSE;

  /** \internal
  * handle command line
  */

  while ((ch = getopt(argc, argv, "e:i:o:h")) != -1) {
    switch (ch) {
    case 'e':
      /* write errors to here */
      my_errfile = optarg;
      f_err = TRUE;
      break;
    case 'i':
      /* Read this file */
      my_infile = optarg;
      f_in = TRUE;
      break;
    case 'o':
      /* write good lines to here */
      my_outfile = optarg;
      f_out = TRUE;
      break;
    case 'h':	/* the help file */
      /* FALLTHROUGH */
    case '?': /* invalid option */
      /* FALLTHROUGH */
    default:  /* if they put any other option on the commandline */
      usage();
      /* NOT REACHED */
      break;
    }
  }

  if(f_in == TRUE){
    fp_in = fopen(my_infile,"r");
  }
  if(f_err == TRUE){
    fp_err = fopen(my_errfile,"w");
  }
  if(f_out == TRUE){
    fp_out = fopen(my_outfile,"w");
  }

  while ( (linelen = getline(&line, &linecap, (f_in == TRUE ? fp_in : stdin))) > 0 ) {
    wrote_err = FALSE;
    for( i = 0 ; i < linelen; i++) {
      if(isalpha(line[i])) {
        ;
      } else if(isdigit(line[i])) {
        ;
      } else if(ispunct(line[i])) {
        ;
      } else if ( line[i] == '\n') {
        if(wrote_err == FALSE && f_out == TRUE){
          fprintf(fp_out, "%s", line);
        } else {
          ; /* do nothing */
        };
      } else if ( line[i] == ' ') {
        ;
      } else {
        fprintf((f_err == TRUE ? fp_err : stdout), "--- %c %0x ----- Error Line \n", line[i],line[i]);
        wrote_err = TRUE;
      }
    }
    free(line);
  }

  if(f_in == TRUE){
    fclose(fp_in);
  }
  if(f_err == TRUE){
    fclose(fp_err);
  }
  if(f_out == TRUE){
    fclose(fp_out);
  }

  return (0);
}

void
usage(void)
{
  (void)fprintf(stderr, "\
usage: test_ascii [-i infile] [-o outfile] [-e errfile] \n\
       test_ascii -h\n\
    -i infile - use this source file\n\
    -o outfile - use this output file\n\
    -e errfile - use this error file\n\
    -h            - show this help and exit\n");

  exit(99);  /* my usage exit code */
}

4 Likes

Thank you Murray125532 for most instructive posts

2 Likes

Hi Clem,

No problem - was a chance to re-work an old bit of code
And of course looking at it again this morning I realised that there is one more tweak that I could/should do. (moving goalposts - sigh)

The error output currently only prints out the invalid character - not the actual line
That can be added very simply where the lines below

become

(This depends on what one actually wants to capture and display…)

I did note that - depending on your use - you might need to manage properly closing the output files if you terminate the program with control-c. There are several ways of doing this, happy to talk more if you need help.

Cheers
Murray

2 Likes

Hi Murray,

Awesome posts on stdin, stdout and arguments! I’ll keep this one tucked away for when I need them next :slight_smile: .

I can say that I haven’t had any issues with UNIX-y stuff on a Pi like talking to serial ports directly, I think the only difference is the instructions that the C file gets compiled down to (arm64 vs x64).

1 Like

Hi James,

THere are other ways of managing the commandline stuff by using a for loop to process each of the argc count of argv strings, and doing your own string comparisons to find the option flags, filenames etc …

for(i = 1; i < argc; i++) {
    printf("%s\n", argv[i]);
}

noting that you start at i = 1, since argv[0] should be the program name, this can depend on the OS and compiler.

If @Clem73666 is processing his ping output and passing the result out of a serial port to something else, then managing the program exit should not be a concern, since the serial receiver would have done what it needed to do. But writing to an open file as in my code, a control-c signal does a hard abort, and files may not be flushed out and closed properly without trapping and handling the signal first.

The interesting corners to consider.

2 Likes

Hi @Clem73666 and @James

My ‘goto’ for programming info is often Beej’s guides - easily downloadable and with heaps of examples and great insights

Murray

1 Like