FAQ
add progress callback feature to the slow FileUtil operations with ability to cancel the work
---------------------------------------------------------------------------------------------

Key: HADOOP-5353
URL: https://issues.apache.org/jira/browse/HADOOP-5353
Project: Hadoop Core
Issue Type: Improvement
Components: fs
Affects Versions: 0.21.0
Reporter: Steve Loughran
Priority: Minor


This is something only of relevance of people doing front ends to FS operations, and as they could take the code in FSUtil and add something with this feature, its a blocker to none of them.

Current FileUtil.copy can take a long time to move large files around, but there is no progress indicator to GUIs, or a way to cancel the operation mid-way, short of interrupting the thread or closing the filesystem.

I propose a FileIOProgress interface to the copy ops, one that had a single method to notify listeners of bytes read and written, and the number of files handled.

{code}
interface FileIOProgress {
boolean progress(int files, long bytesRead, long bytesWritten);
}

The return value would be true to continue the operation, or false to stop the copy and leave the FS in whatever incomplete state it is in currently.

it could even be fancier: have beginFileOperation and endFileOperation callbacks to pass in the name of the current file being worked on, though I don't have a personal need for that.

GUIs could show progress bars and cancel buttons, other tools could use the interface to pass any cancellation notice upstream.

The FileUtil.copy operations would call this interface (blocking) after every block copy, so the frequency of invocation would depend on block size and network/disk speeds. Which is also why I don't propose having any percentage done indicators; it's too hard to predict percentage of time done for distributed file IO with any degree of accuracy.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedFeb 27, '09 at 2:31p
activeFeb 27, '09 at 2:31p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Steve Loughran (JIRA): 1 post

People

Translate

site design / logo © 2022 Grokbase