Issue Details (XML | Word | Printable)

Key: BATCH-34
Type: New Feature New Feature
Status: Open Open
Priority: Minor Minor
Assignee: Robert Kasanicky
Reporter: Dave Syer
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Spring Batch

Support for multiple I/O files in a single jobRun for a particular scheduleDate.

Created: 09/Jul/07 11:26 AM   Updated: 17/Jul/08 06:44 AM
Component/s: Infrastructure
Affects Version/s: 1.0-m2
Fix Version/s: 2.0.0.M3

Time Tracking:
Original Estimate: 0.5d
Original Estimate - 0.5d
Remaining Estimate: 0.5d
Remaining Estimate - 0.5d
Time Spent: Not Specified
Remaining Estimate - 0.5d

File Attachments: 1. Text File flatfile_os_refactored_071024.patch (47 kB)



 Description  « Hide
Scenario: Read 100 rows from the database and write to file, Output file can have at most 50 lines(rows). So does spring batch support creating 2 files each containing 50 lines (filename.txt.001 & filename.txt.002).

This is supported, however, there isn't any kind of 'max record count' that would signal to the FileOutputSource that a new file should be created. The outputsource would simply need to be closed and reopened. However, a wrapper could easily be written that would do this for you.
Edit/Delete Message

 All   Comments   Work Log   Change History   FishEye   Related Builds      Sort Order: Ascending order - Click to sort in descending order
Peter Zozom added a comment - 25/Oct/07 08:39 AM
I have created new flat file output sources (see attached patch).
- SingleFlatFileOutputSource has same functionality as FlatFileOutputSource, but it is refactored to be extendable
- SplittableFlatFileOutputSource extends SingleFlatFileOutputSource and allows to specify 'max record count'. When record limit is reached new file is created. Output Source attaches part number to filename as extension so file names are filename.txt.001, filename.txt.002. Output source also supports rollback to last commit point located in different file than currently opened file (e.g. we are processing filename.txt.005 and last commit point was somwere in filename.txt.002)

Lucas Ward added a comment - 25/Oct/07 10:51 AM
Peter, I'll take a look at the solution as soon as possible. At first glance it seems to me that writing a simple wrapper for the input source would be the simplest approach. The wrapper would delegate read calls to it's inputSource and call close when the limit is reached, and open again on a new resource. However, extending may be an easier to configure solution. In my mind this is somewhat related to BATCH-16. The approach taken in the xml input sources was to use a static utils class to help with restarting, etc. However, it seems to me that common file handling concerns for reading in from both xml and flat files could be handled using a common abstract class. This would allow a wrapper, such as one that could be created to fix this issue, to work for both xml, flat file, and anything else that extended the common abstract file input source interface.

Lucas Ward added a comment - 13/Nov/07 09:36 AM
I understand your point about extending the output source so that you can span rollbacks and commits across files. However, I'm not sure if we should be doing that. Personally, I would prefer that any limit given simply means that we will close the file at the first commit point after the limit is reached. However, I would be interested to hear if anyone has a use case that requires an exact file size or record count.

Lucas Ward added a comment - 27/Jun/08 09:48 AM
Moving to 2.0, as there seems to be no requests for the feature.