Spawning a console application and tracking its Standard Output

Toby Cosham, our Accounting Guru, has the first tech oriented blog post to contribute, helping us achieve our milestone of at least a post a week. Thanks Toby.

Sometimes programs we write must interact with a console application that does not integrate easily into our framework. The application may be designed for use in a manual environment, but you need to automate it. Any errors produced may be displayed only on the screen via Standard Output.

This happened to me recently. My program was written using the .NET framework. It is pretty simple to run another program from the .NET framework. I used System.Diagnostics.Process to launch the other software (I’ll call it ABCD) using Process.Start, and waited for it to complete using Process.WaitForExit.

The purpose of ABCD is to send and receive data over the internet. It is a command driven program, with an interface similar to FTP. After testing its responses, it became apparent that it was unreliable. Not only could it return errors about failing to connect to its server, but also logon failure and timeouts. In addition, there were some more obscure, but regular, failures – corrupted stream and handshake failure. There was no way to predict these failures, but trying again immediately almost always worked.

The requirements for my program included executing ABCD many times each day, starting at 1am and finishing at 7pm. If it did not work, immediate action needed to be taken. At MaxSoft, we have a library that handles sending the SMS, so sending the message was not a problem. However, since the SMS was to our CIO, it was important to identify the problem and attempt to rectify it first.

ABCD does not return an error code. It does not log its results to a database, or a file. The only way it notifies its results is directly to the screen. In addition, when it downloads files as a batch, the batch can contain multiple files of the same name. The first file gets overwritten by the second file with the same name. This meant that I could not pipe the standard output of ABCD to a file and interpret it after it had executed – some files would have been lost by then.

The answer was to use some of the features of System.Diagnostics.Process. This class allows for the Standard Output (which normally is displayed on the screen) to be captured as it occurs. It also allows for Standard Error to be captured, and for Standard Input to be written to. I passed the commands to ABCD by writing to the StandardInput stream, allowing me to protect the user id and password.

I then set Process.EnableRaisingEvent to true and called Process.BeginOutputReadLine. This activates the OutputDataReceived event, which I set to call my OutputDataReceived function.

I set up an array of strings for the expected result. The output was the same all the time, except the list of files produced, which I added as a place-holder string to the expected result. Each time OutputDataReceived was called, I compared the text that had been added to StandardOutput to the next item in the expected results. If it matched, I removed that section from the expected results and exited the OutputDataReceived function. If it did not match, I recorded the error and exited the function. When Process.WaitForExit completes, this error information is checked to determine if the process has run correctly or not. The placeholder string is used to identify when the list of files is being processed. This is only removed from the expected result when the following expected result is received.

Having written this, I was getting all errors reported. I wanted to reduce the calls our CIO received early in the morning (So did he), so I processed the error message. If any of the three errors that could be retried occurred, I looped back to where ABCD was being set up and ran it again. Along with providing the maximum number of retries, this eliminated alerts being generated on transient errors. If any other error occurred, ABCD was not rerun, and the alert was sent straight away.

Issues that I had resolved include:

- calling a console application using System.Diagnostics.Process

Process p = new Process();
ProcessStartInfo si = new ProcessStartInfo(application);
p.StartInfo = si;
si.Arguments = arguments;
si.UseShellExecute = false;
p.Start();

- retrieving and interpreting text output to the screen

si.RedirectStandardOutput = true;
p.EnableRaisingEvents = true;
p.OutputDataReceived += new DataReceivedEventHandler(OutputDataReceived);
p.BeginOutputReadLine();

- timeouts and other retryable errors

void OutputDataReceived(object sender, DataReceivedEventArgs e)
{
if (e.Data == null)
return;

// check e.Data for retryable errors and set flag indicating retry required
}

- preventing the console application for running forever by using WaitForExit with a time limit

if (!p.WaitForExit(Config.TimeoutSeconds * 1000))
{
p.Kill();
}
p.Close();