Issue Details (XML | Word | Printable)

Key: BATCH-140
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Lucas Ward
Reporter: Lucas Ward
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
Spring Batch

File Input and Output should be refactored

Created: 25/Sep/07 09:19 PM   Updated: 07/Oct/07 11:40 PM
Component/s: Infrastructure
Affects Version/s: 1.0-m2
Fix Version/s: 1.0-m3
Security Level: Public (Public Issues)

Time Tracking:
Not Specified

Issue Links:
Depends
 


 Description  « Hide
File input and output sources should be refactored in order to create more sensible interfaces that are consistent with other input sources. For example, both the stax and sql input sources returns an object when read() is called. StAX does this by delegating to an OXM framework to map xml, and sql does the same using a passed in RowMapper. However, the FlatFileInputSources returned a FieldSet that then returns an object. It would be more sensible to inject a mapper into the file input sources and return a mapped object. In this way, all of the input sources would be consistent, which would require less stock ItemProviders, and would allow for BATCH-118 to be fixed, since the input source could catch a mapping exception and wrap it in another exception that contained the original line read in.

The same could be done with the FileOutputSource, by injecting an aggregator into it, similar to how the input has a tokenizer injected in.

 All   Comments   Work Log   Change History   FishEye   Builds      Sort Order: Ascending order - Click to sort in descending order
Dave Syer added a comment - 26/Sep/07 02:26 AM
One small problem with injecting FieldSetMapper into an InputSource arises in the case where the input consists of many different types of record - in that case what you really want is to take control in the ItemProvider, where application logic can aggregate the different record types into the correct types. There isn't a 1:1 correspondence between input lines and output objects, so you can't necessarily just throw a FieldSet into a mapper and get back an item.

I guess the answer is to either have a pass-thru mapper and let the item provider deal with it as it does now, or to encourage putting the complex logic inside the mapper. I guess either works, so people can just choose. In the latter case the item provider would get an Object back from the mapper and would have to make a decision then about whether it was a logical record terminator or not.

Lucas Ward added a comment - 02/Oct/07 10:51 AM
I think they could also pass in a composite FieldSetMapper to the InputSource, one that would delegate to other FieldSetMappers depending on the record type, probably using a map, similar to how it's done in the sample applications now.

Lucas Ward added a comment - 02/Oct/07 11:45 PM
I modified the SimpleFlatFileInputSource to have a FieldSetMapper wired in. If none is set, then an exception is thrown. I debated about automatically passing the FieldSet through, but decided that it was the minority of use cases and should be explicitly done by wiring rather than by default if none is set. This means that the FieldSetInputSource will need to be deleted. I'm still cleaning up some of the item providers and samples, but should be able to commit tomorrow.

Lucas Ward added a comment - 07/Oct/07 11:40 PM
SimpleFlatFileInputSource has been modified to no longer implement FieldSetInputSource (the interface has been deleted), and now requires that a mapper be wired in. In addition, any errors encountered while reading will be wrapped in a FileParsingException.