Restart When Output Stops #PowerShell

In command/program execution in PowerShell, there are situations where standard output stops (freezes/hangs). I implemented detection of this and, in such cases, stopping and re-executing the relevant program.

For example, the following is a PowerShell script that executes the ffmpeg command every 5 seconds.

ffmpeg -f dshow -i audio="Microphone (USB Microphone)" -y -t 00:00:05 "tmp.mp3"

Note that this introduction references a Stack Overflow answer.

I need to get the current state of the last line of the command output stream, then if the line remains unchanged (staled/freezing/hanging) over 5 seconds log “warning: the program is freezing. trying to restart…”, then stop the process and re-start the command. command line - Restart the process if its output unchanged in PowerShell - Super User

The above command executes for 5 seconds and ends, but with a simple implementation, it can be looped to run semi-permanently like this:

for ($i = 0; $i -gt -1; $i++) {
  $date = Get-Date -Format "yyyy-MM-ddTHH-mm-ss"
  $dir = "C:\_documents\WindowsPowerShell\data\recording\$($date.Substring(0,4))\$($date.Substring(5,2))\$($date.Substring(8,2))"
  $path = "$dir\$($date.Substring(11,2))-$($date.Substring(14,2))-$($date.Substring(17,2)).mp3"
  New-Item -ItemType Directory -Force -Path $dir
  ffmpeg -f dshow -i audio="Microphone (USB Microphone)" -y -t 00:10:00 -b:a 128k $path
}

However, the problem here is that for programs that hang/freeze during execution like the above, you need to prepare a mechanism that can automatically handle it when that happens.

If this were Linux instead of Windows, you could easily interfere with terminal console output and input by utilizing tmux or screen, but for PowerShell on Windows, there doesn’t seem to be anything as developed and suitable for general use as tmux.

By the way, what this article deals with is strictly handling “hang/output stop/freeze”, not forced program termination. If the command exits with an error code, there are other (simpler) implementation methods.

For example, for Node.js programs, by wrapping the program part as follows, you can re-execute the program when an error occurs and the command exits.

while ($true) {
    try {
        node .\notify_app\amazon_scrape.js
    } catch {
        Write-Host "An error occurred: $_"
        Write-Host "Restarting the program..."
    }
    # Optional delay between restarts (in seconds)
    Start-Sleep -Seconds 1
}

For Java, almost the same implementation is possible.

while ($true) {
    try {
        # Start your Java application using the java -jar command
        Start-Process -FilePath "java" -ArgumentList "-jar", $jarPath -NoNewWindow
        # Wait for the Java application to finish
        Wait-Process -Name "java" -ErrorAction SilentlyContinue
    }
    catch {
        Write-Host "An error occurred: $($_.Exception.Message)"
    }
}

Now, back to the point, in PowerShell, by using Start-Process to execute commands and interfering with the output log of that background process as follows, you can perform necessary processing.

$log = "C:\_documents\WindowsPowerShell\data\recording\output.log";
$micName = "Microphone (USB Microphone)";

$process = $null;

Function run-ffmpeg {param([string]$Log,[string]$MicName)
    $date = Get-Date -Format "yyyy-MM-ddTHH-mm-ss"
    $dir = "C:\_documents\WindowsPowerShell\data\recording\$($date.Substring(0,4))\$($date.Substring(5,2))\$($date.Substring(8,2))"
    $path = "$dir\$($date.Substring(11,2))-$($date.Substring(14,2))-$($date.Substring(17,2)).mp3"
    New-Item -ItemType Directory -Force -Path $dir
    Write-Host "Saving to file path: $path";

    $ffmpegCommand = "ffmpeg -f dshow -i audio='$MicName' -y -t 00:10:00 -b:a 128k $path 2>> $Log"

    $process = Start-Process powershell -ArgumentList "-NoProfile -ExecutionPolicy Bypass -Command $($ffmpegCommand)" -WindowStyle Hidden -PassThru;

    Write-Host "Process ID: $($process.Id)";
    # Stop-Process -Id $process.Id

    Start-Sleep -Seconds 4;
    logchecks $Log;
};

Function tail {Get-Content -Path $args[0] -Tail 1};
Function check {tail $args[0]};

Function logchecks {
    do {
        $checklog = check $args[0];
        Start-Sleep -Seconds 5;
        $rechecklog = check $args[0];
        if ($checklog -eq $rechecklog) {
            $message = "Restarting triggered. [reason: the last line of the log file has not changed after 5 seconds]";
            Write-Host $message;
            Write-Host "Stopping process with ID: $($process.Id)";
            
            # json, {type: "separator", date: date, message: message}
            $now = Get-Date -Format "yyyy-MM-ddTHH-mm-ss";
            Write-Output "{`"type`":`"separator`",`"date`":`"$now`",`"message`":`"$message`"}" 2>&1 | Tee-Object -FilePath $Log -Append

            # try to prevent the program from being quit
            try {
                Stop-Process -Id $process.Id;
            } catch {
                Write-Host "Process with ID: $($process.Id) has already been stopped";
            }
        }
    } while ($checklog -ne $rechecklog)

    run-ffmpeg -Log $log -MicName $micName;  
};

logchecks $log;

The execution flow of the above script is as follows:

1. The initial run-ffmpeg is executed by the logchecks function
2. The run-ffmpeg function executes the ffmpeg command and redirects standard output to a file
3. The logchecks function checks the last line of the output log every 5 seconds. If there’s no change from 5 seconds ago, it determines it’s frozen, stops the background process running the ffmpeg command, and executes the run-ffmpeg function again. If there’s a change, it checks again after 5 seconds and repeats the same process.

This enables handling when the log freezes.