Process  I/O  and  File  NIO.2  Java  Fundamentals  

2014,  Tartu  

Neeme Praks @nemecec

Agenda  

1.  Process  I/O  1.  StarAng  Processes  2.  Handling  I/O  3.  Stopping  Processes  

2.  New  File  API  (NIO.2)  1.  Basics  2.  Metadata  

PROCESS  I/O  

Process  

•  A  process  is  an  instance  of  a  computer  program  that  is  being  executed.  

•  It  contains  the  program  code  and  its  current  acAvity.  

Process  Tools  

•  Windows  GUI:  Task  Manager,  Process  Explorer  •  Windows  CLI:  tasklist,  wmic  •  UNIX  GUI:  System  Monitor  •  UNIX  CLI:  ps,  top  •  Mac  OS  X:  Ac?vity  Monitor  •  Java:  jps  

Java  Process  APIs  

•  Two  APIs  – RunAme.exec()  

•  Since  the  beginning  (Java  1.0)  •  Error-­‐prone  and  inconvenient  

– ProcessBuilder  •  Since  Java  1.5  •  Much  beZer  than  RunAme.exec()  

3rd-­‐party  process  APIs  

•  Apache  commons-­‐exec  – CompaAble  with  Java  1.3+  – Verbose  API  – Very  mature,  no  acAve  development  

•  zt-­‐exec  – CompaAble  with  Java  1.5+  –  Inspired  by  ProcessBuilder  and  commons-­‐exec  – Concise  API  – Fixes  a  lot  of  small  issues  across  the  board  

StarAng  Process  in  Java  ProcessBuilder builder = new ProcessBuilder( "C:/Program Files (x86)/Google/Chrome/Application/chrome.exe”, "https://courses.cs.ut.ee/2014/javaFund/fall");builder.start();  

StarAng  Process  in  Java  (2)  

ProcessBuilder builder = new ProcessBuilder(”ps”);builder.start();    

Where  does  the  process  output  go?  

Process  Streams  

1.  Output  2.  Error  3.  Input  

•  They  NEED  to  be  processed!  

Process  Stream  Piping  Stream   My  Own  Process   My  Sub  Process  Output   OutputStream  

System.out  InputStream  Process.getInputStream()  

Error   OutputStream  System.err  

InputStream  Process.getErrorStream()  

Input   InputStream  System.in  

OutputStream  Process.getOutputStream()  

Inherit  Process  Standard  Output  

ProcessBuilder builder = new ProcessBuilder("ps");builder.redirectOutput( Redirect.INHERIT);builder.start();  

Inherit  All  Streams  

ProcessBuilder builder = new ProcessBuilder("ps");builder.inheritIO();builder.start();  

Redirect  Process  Output  to  File  

ProcessBuilder builder = new ProcessBuilder("ps");File file = new File("out.txt");builder.redirectOutput( Redirect.to(file));builder.start();  

Copy  Process  Output  ProcessBuilder builder = new ProcessBuilder("ps");Process process = builder.start();InputStream out = process.getInputStream();IOUtils.copy(out, System.out);  

Copy  Process  Error  ProcessBuilder builder = new ProcessBuilder("ls", "notfound");Process process = builder.start();InputStream err = process.getErrorStream();IOUtils.copy(err, System.err);  

Copy  Process  Output  and  Error  ProcessBuilder builder = new ProcessBuilder("ls", "notfound");builder.redirectErrorStream(true);Process process = builder.start();InputStream out = process.getInputStream();IOUtils.copy(out, System.out);  

Copy  Process  Output  and  Input  ProcessBuilder builder = new ProcessBuilder("head", "-n", "3");Process p = builder.start();//run in backgroundnew Thread(() -> copyStream( p.getInputStream(), System.out)).start();IOUtils.copy(System.in, p.getOutputStream());  

Process.getOutputStream()  may  be  buffered  –  it’s  not  flushed  immediately.  

Copy  with  Immediate  Flush  void copyStreamWithFlush( InputStream in, OutputStream out) { int i; while ((i = is.read()) != -1) { os.write(i); os.flush(); }}

Seang  Working  Directory  ProcessBuilder builder = new ProcessBuilder("ls");builder.directory(new File("/"));builder.redirectOutput(Redirect.INHERIT);builder.start();  

Seang  Environment  ProcessBuilder builder = new ProcessBuilder("env");builder.inheritIO();Map<String, String> env = builder.environment();env.put("foo", "bar");env.put("PATH", "/myBin" + File.pathSeparator + env.get("PATH"));builder.start();  

Seang  PATH  ProcessBuilder builder = new ProcessBuilder("Calculator");  Map<String, String> env = builder.environment();env.put("PATH", "/Applications/Calculator.app/Contents/MacOS” + File.pathSeparator + env.get("PATH"));  builder.start();  

Will  it  work?  

Seang  PATH  ProcessBuilder builder = new ProcessBuilder("Calculator");  Map<String, String> env = builder.environment();env.put("PATH", "/Applications/Calculator.app/Contents/MacOS” + File.pathSeparator + env.get("PATH"));  builder.start();  

IOException: Cannot run program "Calculator": error=2, No such file or directory

PATH  is  not  used  for  finding  the  command!  

Wait  for  Process  Stop  ProcessBuilder builder = new ProcessBuilder( "/Applications/Calculator.app" + "/Contents/MacOS/Calculator");Process process = builder.start();process.waitFor();  

Check  Process  Exit  Status  ProcessBuilder builder = new ProcessBuilder("cat", "test.txt");builder.inheritIO();Process process = builder.start();int exitCode = process.waitFor();//0 – success, 1 – file not foundif (exitCode != 0) { throw new IllegalStateException( "Exit code: " + exitCode);}  

Stop  Process  ProcessBuilder builder = new ProcessBuilder( "ping", "ut.ee");builder.redirectOutput(Redirect.INHERIT);Process process = builder.start();Thread.sleep(5000);process.destroy();  

Stop  Process  in  Finally  

Process process = builder.start();try { doStuff(process);}finally { process.destroy();}  

RunAme.exec()  

•  Since  Java  1.5  use  ProcessBuilder  instead  •  Issues:  

– Single  String  command  is  split  by  spaces  –  fails  if  arguments  contain  spaces  (quotes  don’t  help)  

– Environment  is  passed  as  String[]  not  Map  – Parent  environment  is  only  inherited  if  no  custom  variables  are  set  

Apache  Commons  Exec  

•  hZp://commons.apache.org/exec/    •  Enables  to  

– Pump  process  streams  – Destroy  a  process  on  Ameout  – Expect  certain  exit  values  – Handle  compleAon  or  failure  using  a  callback  – SubsAtute  variables  in  command  line  

•  JDK  1.3+  compaAble  

Stop  Process  on  Timeout  (commons-­‐exec)  

CommandLine cmdLine = new CommandLine( "/Applications/Calculator.app" + "/Contents/MacOS/Calculator");DefaultExecutor executor = new DefaultExecutor();executor.setWatchdog( new ExecuteWatchdog(10*1000));executor.execute(cmdLine);  

zt-­‐exec  

•  hZps://github.com/zeroturnaround/zt-­‐exec  •  Even  beZer  than  commons-­‐exec!  

–  Improved  API  •  One-­‐liners  for  common  but  complex  use-­‐cases  •  One-­‐liners  for  reading  process  output  to  string  •  Async  processes  (java.uAl.concurrent.Future)  

–  Improved  stream,  Ameout  and  exit  code  handling  •  JDK  1.5+  compaAble  •  Used  by  MongoDB,  documents4j  

Stop  Process  on  Timeout  (zt-­‐exec)  

try { new ProcessExecutor().command( "/Applications/Calculator.app" + "/Contents/MacOS/Calculator") .timeout(10, TimeUnit.SECONDS).execute();}catch (TimeoutException e) { // process is automatically destroyed}  

(Unix)  Process  Signals  Name   Number   Descrip?on  

HUP   1   Hang  up  (log  out)  INT   2   Interrupt  (Ctrl+C)  QUIT   3   Quit  (Ctrl+Break)  KILL   9   Non-­‐catchable,  non-­‐ignorable  kill  TERM   15   Somware  terminaAon  signal  

Java  Thread  Dump  Tools  

•  Ctrl+Break  •  jstack  <pid>  •  kill  –QUIT  <pid>  •  sendsignal.exe  <pid>  

Stopping  Gracefully  

•  Close  Window  •  Ctrl+C  •  kill  <pid>  •  taskkill  /PID  <pid>  

Stopping  Forcefully  

•  kill  -­‐KILL  <pid>  •  taskkill  /F  /PID  <pid>  

Process.destroy()  method  

•  By  Javadoc  it  forcibly  terminates  the  sub-­‐process.  

•  But  actually  –  It  forcefully  terminates  on  Windows  – and  gracefully  terminates  on  UNIX  

•  New  destroyForcibly()  method  in  Java  8      

Forceful  TerminaAon  on  UNIX  

•  Execute  external  ‘kill  –KILL  <pid>’  command  •  Geang  PID  via  Java  ReflecAon  API    class UNIXProcess extends Process {    private final int pid;  …  }  

•  Geang  PID  of  last  bash  command  my-­‐app  &  echo  $!  

NIO.2  (FILE  I/O)  

Based  On  

•  File  I/O  (Featuring  NIO.2)  Java  Tutorial  •  hZp://docs.oracle.com/javase/tutorial/essenAal/io/fileio.html    

What  is  NIO.2?  

•  Why  do  we  need  it?  •  New  I/O  API  introduced  in  Java  7  

– File  Paths  – File  OperaAons  – Managing  Metadata  – Walking  the  File  Tree  – Finding  Files  – Watching  a  Directory  for  Changes  

Other  Benefits  

•  Supports  symbolic  links  •  Improved  support  for  relaAve  paths  •  Improved  support  for  directory  lisAngs  •  Improved  error  handling  

Path  vs  OperaAons  

•  Old  File  I/O  – java.io.File  handles  both  

•  NIO.2  FILE  I/O  – java.nio.file.Path  handles  a  path  – java.nio.file.Files  handles  operaAons  

Paths  Paths.get("c:\\example\\data\\input.txt");Paths.get("/example/data/input.txt");Paths.get("example/data/input.txt");Paths.get("example\\data\\input.txt");Paths.get("example", "data", "input.txt");Paths.get("/", "example", "data", "input.txt");Paths.get( URI.create("file:/example/data/input.txt"));

Path  class Path { String toString(); Path getFileName(); Path getName(int index); int getNameCount(); Path subpath(int beginIndex, int endIndex); Path getParent(); Path getRoot(); //continues on the next slide}  

Path  (2)  

class Path { Path normalize(); boolean isAbsolute(); Path toAbsolutePath(); Path resolve(Path other); Path relativize(Path other); File toFile(); URI toUri(); //continues on the next slide}  

Path  (3)  Path toRealPath(LinkOption... Options)  enum LinkOption { NOFOLLOW_LINKS; }  path.toRealPath(); //follows symbolic linkspath.toRealPath(LinkOption.NOFOLLOW_LINKS);//doesn't follow symbolic links  

Checking  Files  class Files { boolean exists( Path path, LinkOption... options); boolean notExists( Path path, LinkOption... options); boolean isReadable(Path path); boolean isWritable(Path path); boolean isExecutable(Path path); boolean isSameFile(Path path, Path path2); //continues on the next slide}  

DeleAng  Files  class Files { void delete(Path path) throws IOException; boolean deleteIfExists(Path path) throws IOException; //continues on the next slide}  

May  throw  •  NoSuchFileExcepAon  •  DirectoryNotEmptyExcepAon  

Copying  Files  class Files { Path copy(Path source, Path target, CopyOption... options); long copy(InputStream in, OutputStream out); long copy(InputStream in, Path target, CopyOption... options); long copy(Path source, OutputStream out); //continues on the next slide}  

Copying  Files  (2)  enum StandardCopyOption implements CopyOption { REPLACE_EXISTING, COPY_ATTRIBUTES, ATOMIC_MOVE;}enum LinkOption implements CopyOption { NOFOLLOW_LINKS;}  

Moving  Files  class Files { Path move(Path source, Path target, CopyOption... options);}    Supported  opAons:  •  REPLACE_EXISTING  •  ATOMIC_MOVE  

LisAng  Directories  try (DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) { for (Path file: stream) System.out.println(file.getFileName());} catch (IOException | DirectoryIteratorException x) { System.err.println(x);}  

Filtering  Directory  LisAngs  DirectoryStream<Path> stream = Files.newDirectoryStream(dir, new Filter<Path>() { boolean accept(Path path) { return Files.isDirectory(path); } } );  DirectoryStream<Path> stream = Files.newDirectoryStream(dir, "*.class");  

Reading  and  WriAng  Files  class Files { byte[] readAllBytes(Path path); List<String> readAllLines( Path path, Charset cs); Path write( Path p, byte[] bytes, OpenOption... oo); Path write( Path path, Iterable<? extends CharSequence> lines, Charset c, OpenOption... oo);}  

NIO.2  FILE  METADATA  

Basic  File  AZributes  class Files { long size(Path path); boolean isDirectory( Path path, LinkOption... lo); boolean isRegularFile( Path path, LinkOption... lo); boolean isSymbolicLink(Path path); boolean isHidden(Path path); //to be continued}  

Last  Modified  Time  FileTime time = Files.getLastModifiedTime(path);long millis = time.toMillis();long millis = System.currentTimeMillis();FileTime time = FileTime.fromMillis(millis);Files.setLastModifiedTime(path, time);

File  Owner  UserPrincipal owner = Files.getOwner(path);String name = owner.getName();  

String name = "James";UserPrincipal owner = path.getFileSystem(). getUserPrincipalLookupService(). lookupPrincipalByName(name);Files.setOwner(path, owner);}  

POSIX  File  Permissions  

OWNER_READ   OWNER_WRITE   OWNER_EXECUTE  

GROUP_READ   GROUP_WRITE   GROUP_EXECUTE  

OTHERS_READ   OTHERS_WRITE   OTHERS_EXECUTE  

Geang  POSIX  File  Permissions  

Set<PosixFilePermission> perms = Files.getPosixFilePermissions(path);boolean ownerRead = perms.contains( PosixFilePermission.OWNER_READ);System.out.println(perms);System.out.println( PosixFilePermissions.toString(perms));

Seang  POSIX  File  Permissions  Set<PosixFilePermission> perms = new HashSet<PosixFilePermission>();perms.add(PosixFilePermission.OWNER_READ);Files.setPosixFilePermissions(path, perms);Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rwxr-xr-x");Files.setPosixFilePermissions(path, perms);  

Single  File  AZributes  class Files { Object getAttribute(Path path, String attribute, LinkOption... options); Path setAttribute(Path path, String attribute, Object value, LinkOption... options); //to be continued}  

Reading  File  AZributes  class Files { <A extends BasicFileAttributes> A readAttributes( Path path, Class<A> type, LinkOption... options); //to be continued}  

File  AZribute  Views  class Files { <V extends FileAttributeView> V getFileAttributeView( Path path, Class<V> type, LinkOption... options); //that's it}  

Supported  AZributes  

FileAZributeView  BasicFileAZributeView   BasicFileAZributes  DosFileAZributeView   DosFileAZributes  PosixFileAZributeView   PosixFileAZributes  FileOwnerAZributeView  AclFileAZributeView  UserDefinedFileAZributeView  

DOS  File  AZributes  DosFileAttributes attr = Files.readAttributes( path, DosFileAttributes.class);      interface DosFileAttributes extends BasicFileAttributes { boolean isReadOnly(); boolean isHidden(); boolean isArchive(); boolean isSystem();}  

Summary  

•  Use  ProcessBuilder  or  zt-­‐exec  to  start  sub  processes  

•  Handle  sub  process  I/O  correctly  – Either  Inherit  I/O  (since  Java  7)  – Or  pump  the  streams  (zt-­‐exec,  commons-­‐exec)  

•  Remember  to  stop  your  sub  process    

Summary  (2)  

•  Use  NIO.2  File  API  for  –  faster  directory  lisAngs  and  beZer  charset  support  – beZer  metadata  –  listening  for  changes  – finding  files  

QUESTIONS?  

Homework  #3  

Write  a  Java  program  which  mirrors  a  directory  (not  recursively)  1.  Take  the  source  and  target  directory  paths  as  

input  arguments.  1.  Fail  if  source  directory  does  not  exist.  2.  Create  the  target  directory  if  it  does  not  exist.  

Homework  #3  (2)  

2.  On  startup  compare  directory  lisAngs  (use  java.nio.file.DirectoryStream)  and...  1.  Copy  new  files  from  source  to  target  directory.  2.  Copy  updated  (check  last  modified  Ames)  files  

from  source  to  target  directory.  3.  Remove  files  from  target  directory  which  have  

been  removed  from  source  directory.  4.  Ignore  sub  directories  in  both  source  and  target  

directory.  

Homework  #3  (3)  

3.  Watch  source  directory  for  changes  (use  java.nio.file.WatchService)  and  mirror  them  into  the  target  directory.  1.  Exit  when  the  source  directory  gets  removed  

(watch  key  is  invalid  –  tested  on  UNIX  not  Windows).  

4.  Don’t  scan  enAre  directory  on  each  event  

Homework  #3  (4)  

5.   Use  Java  7  File  APIs  (java.nio.file)  instead  of  the  old  (java.io.File).  

6.  Print  a  message  about  each  file  operaAon  you  make.  

7.  Source  directory  must  not  be  altered.  

Homework  #3  (5)  

Use  jf-­‐skeleton  in  GitHub  hZps://github.com/zeroturnaround/jf-­‐skeleton/    Submit  zipped  source  code  via  e-­‐mail  Address:    jf@zeroturnaround.com      Deadline:  next  Monday,  6th  of  October