Split and collect files?

Is there any functionality in place that supports this? For example, chunking a file and scattering each chunk for processing, and collecting and concatenating the results? I'm surprised because I figured this was a common use case. Even suppose I have my own core modules that perform the mechanics of this, but I still don't see an easy way to write out a series of split files to a list, for later scattering.

input split > [input_a, input_b, input_c, ...] > scatter > process > collect

Am I doing it wrong?

Thanks!

Tagged:

Best Answer

Answers

Sign In or Register to comment.