Is it possible to terminate or stop a PowerShell pipeline from within a filter

Question

I have written a simple PowerShell filter that pushes the current object down the pipeline if its date is between the specified begin and end date. The objects coming down the pipeline are always in ascending date order so as soon as the date exceeds the specified end date I know my work is done and I would like to let tell the pipeline that the upstream commands can abandon their work so that the pipeline can finish its work. I am reading some very large log files and I will frequently want to examine just a portion of the log. I am pretty sure this is not possible but I wanted to ask to be sure.

Solution 1

It is possible to break a pipeline with anything that would otherwise break an outside loop or halt script execution altogether (like throwing an exception). The solution then is to wrap the pipeline in a loop that you can break if you need to stop the pipeline. For example, the below code will return the first item from the pipeline and then break the pipeline by breaking the outside do-while loop:

do {
    Get-ChildItem|% { $_;break }
} while ($false)

This functionality can be wrapped into a function like this, where the last line accomplishes the same thing as above:

function Breakable-Pipeline([ScriptBlock]$ScriptBlock) {
    do {
        . $ScriptBlock
    } while ($false)
}

Breakable-Pipeline { Get-ChildItem|% { $_;break } }

Solution 2

You can throw an exception when ending the pipeline.

gc demo.txt -ReadCount 1 | %{$num=0}{$num++; if($num -eq 5){throw "terminated pipeline!"}else{write-host $_}}

or

Look at this post about how to terminate a pipeline: https://web.archive.org/web/20160829015320/http://powershell.com/cs/blogs/tobias/archive/2010/01/01/cancelling-a-pipeline.aspx

Solution 3

It is not possible to stop an upstream command from a downstream command.. it will continue to filter out objects that do not match your criteria, but the first command will process everything it was set to process.

The workaround will be to do more filtering on the upstream cmdlet or function/filter. Working with log files makes it a bit more comoplicated, but perhaps using Select-String and a regular expression to filter out the undesired dates might work for you.

Unless you know how many lines you want to take and from where, the whole file will be read to check for the pattern.

Solution 4

Here's an - imperfect - implementation of a Stop-Pipeline cmdlet (requires PS v3+), gratefully adapted from this answer:

#requires -version 3
Filter Stop-Pipeline {
  $sp = { Select-Object -First 1 }.GetSteppablePipeline($MyInvocation.CommandOrigin)
  $sp.Begin($true)
  $sp.Process(0)
}

# Example
1..5 | % { if ($_ -gt 2) { Stop-Pipeline }; $_ } # -> 1, 2

Caveat: I don't fully understand how it works, though fundamentally it takes advantage of Select -First's ability to stop the pipeline prematurely (PS v3+). However, in this case there is one crucial difference to how Select -First terminates the pipeline: downstream cmdlets (commands later in the pipeline) do not get a chance to run their end blocks.
Therefore, aggregating cmdlets (those that must receive all input before producing output, such as Sort-Object, Group-Object, and Measure-Object) will not produce output if placed later in the same pipeline; e.g.:

# !! NO output, because Sort-Object never finishes.
1..5 | % { if ($_ -gt 2) { Stop-Pipeline }; $_ } | Sort-Object

---

Background info that may lead to a better solution:

Thanks to PetSerAl, my answer here shows how to produce the same exception that Select-Object -First uses internally to stop upstream cmdlets.

However, there the exception is thrown from inside the cmdlet that is itself connected to the pipeline to stop, which is not the case here:

Stop-Pipeline, as used in the examples above, is not connected to the pipeline that should be stopped (only the enclosing ForEach-Object (%) block is), so the question is: How can the exception be thrown in the context of the target pipeline?

Solution 5

If you're willing to use non-public members here is a way to stop the pipeline. It mimics what select-object does. invoke-method (alias im) is a function to invoke non-public methods. select-property (alias selp) is a function to select (similar to select-object) non-public properties - however it automatically acts like -ExpandProperty if only one matching property is found. (I wrote select-property and invoke-method at work, so can't share the source code of those).

# Get the system.management.automation assembly
$script:smaa=[appdomain]::currentdomain.getassemblies()|
         ? location -like "*system.management.automation*"
# Get the StopUpstreamCommandsException class 
$script:upcet=$smaa.gettypes()| ? name -like "*StopUpstreamCommandsException *"

function stop-pipeline {
    # Create a StopUpstreamCommandsException
    $upce = [activator]::CreateInstance($upcet,@($pscmdlet))

    $PipelineProcessor=$pscmdlet.CommandRuntime|select-property PipelineProcessor
    $commands = $PipelineProcessor|select-property commands
    $commandProcessor= $commands[0]

    $ci = $commandProcessor|select-property commandinfo
    $upce.RequestingCommandProcessor | im set_commandinfo @($ci)

    $cr = $commandProcessor|select-property commandruntime
    $upce.RequestingCommandProcessor| im set_commandruntime @($cr)

    $null = $PipelineProcessor|
        invoke-method recordfailure @($upce, $commandProcessor.command)

    if ($commands.count -gt 1) {
      $doCompletes = @()
      1..($commands.count-1) | % {
        write-debug "Stop-pipeline: added DoComplete for $($commands[$_])"
        $doCompletes += $commands[$_] | invoke-method DoComplete -returnClosure
      }
      foreach ($DoComplete in $doCompletes) {
        $null = & $DoComplete
      }
    }

    throw $upce
}

---

EDIT: per mklement0's comment:

Here is a link to the Nivot ink blog on a script on the "poke" module which similarly gives access to non-public members.

As far as additional comments, I don't have meaningful ones at this point. This code just mimics what a decompilation of select-object reveals. The original MS comments (if any) are of course not in the decompilation. Frankly I don't know the purpose of the various types the function uses. Getting that level of understanding would likely require a considerable amount of effort.

My suggestion: get Oisin's poke module. Tweak the code to use that module. And then try it out. If you like the way it works, then use it and don't worry how it works (that's what I did).

Note: I haven't studied "poke" in any depth, but my guess is that it doesn't have anything like -returnClosure. However adding that should be easy as this:

if (-not $returnClosure) {
  $methodInfo.Invoke($arguments)
} else {
  {$methodInfo.Invoke($arguments)}.GetNewClosure()
}

View more solutions

Comments

[Χpẘ]

See my answer for a way to do this using non-public members.

[mklement0]

Impressive, but mind-bending. If you're up for it, perhaps you can add more comments and also point to how others could implement their own versions of select-property and invoke-method.

[FSCKur]

This was great, but you have a space in your wildcard string, which breaks your code: "*StopUpstreamCommandsException *" May I suggest this as tidier: $script:upcet = [Powershell].Assembly.GetType('System.Management.Automation.‌​StopUpstreamCommands‌​Exception')