Cheat-Sheet

some useful Matlab commands 

when working with a remote server

INITIALIZATION

>> conn remotely on

starts SSH-based encrypted/secure connection with remote server (this will automatically launch a new server if the last one is down)

>> conn remotely off

ends connection with remote server


REMOTE EXECUTION

>> conn_server cmd

starts remote command-line prompt (type "quit" to return to local command-line prompt)

>> [out1, out2, ...] = conn_server('cmd', expression);

evaluates expression in remote server

>> conn_server('run', fcn_name, fcn_arg1, fcn_arg2, …);

runs function/script in remote server (blocks execution until function finishes)

>> conn_server(‘run_immediatereturn', fcn_name, fcn_arg1, fcn_arg2, …);

runs function/script in remote server (does not block execution)

>> [out1, out2, …] = conn_server('run', fcn_name, fcn_arg1, fcn_arg2, …);

runs function in remote server, returns output arguments locally

>> [out1, out2, …] = conn_server(‘run_keep', fcn_name, fcn_arg1, fcn_arg2, …);

runs function in remote server, keeping output arguments remotely (returns link to remote variables)


REMOTE JOB SUBMISSION

>> conn('submit', fcn_name, fcn_arg1, fcn_arg2, ... );

runs function in remote server HPC/cluster environment (blocks execution)

>> status = conn('submit', fcn_name, fcn_arg1, fcn_arg2, ... );

runs function in remote server HPC/cluster environment (does not block execution)



https://www.conn-toolbox.org/resources/documentation

FILE TRANSFER

>> conn_server('push', file_local, file_remote);

copies local file to remote server

>> conn_server('pull', file_remote, file_local);

copies file in remote server to local computer

>> conn_server_ssh('folderpush', folder_local, folder_remote);

copies local directory to remote server using SCP

>> conn_server_ssh('folderpull', folder_remote, folder_local);

copies directory in remote server to local computer using SCP


CONFIGURATION

>> conn remotely setup

installs remote-access options (run once in server)

>> edit ~/connserverinfo.json

edits remote-access options (run in server)

>> conn jobmanager settings

change remote server HPC/cluster environment options


ADVANCED COMMUNICATIONS

>> conn_server('connect', IP , SERVER_ID); 

connects to CONN server through existing channel (SERVER_ID = portnumberCONNprivatekey)

>> conn_server('start' [, PORT, PUBLICKEY]);

initializes CONN server and open communication channel (run in server)

>> conn_server disconnect

closes communication channel to remote server

>> conn_server exit

terminates remote server and closes communication channel

remote storage

FILE CACHE

(note: use /CONNSERVER/* format to refer to files in server)

>> file_local = conn_cache('pull', file_remote);

creates local-copy of a remote file

>> conn_cache('push', file_remote);

updates a remote file with any changes to its local-copy

>> file_local = conn_cache('new', file_remote);

creates local-copy of a yet-to-be-created remote file

>> conn_cache clearall

clears all locally-cached files


GENERAL FILE UTILITIES

(note: use /CONNSERVER/* format to refer to files in server)

>> data = conn_loadmatfile(filename, ...)

loads data from a MAT-file (same syntax as Matlab's load function)

>> conn_savematfile(filename, ...)

saves data to a MAT-file (same syntax as Matlab's save function)

>> conn_fileutils( fcn_name, ...)

miscelaneous file management utility functions (overloading original fcn_name functions to add support for files/folders located within remote server): 

  conn_fileutils('fileread', filename)         : reads text file (see fileread)

  conn_fileutils('filewrite', filename, str) : creates text file with cell array str (one line per element)

  conn_fileutils('fileappend', filename, str) : appends text file with cell array str (one line per element)

  conn_fileutils('textread', filename) : reads formatted data from text file (see textread)

  conn_fileutils('copyfile', source, target)  : copies file (within same-machine transfers only)

  conn_fileutils('movefile', source, target)  : moves file

  conn_fileutils('renamefile', source, target) : renames file

  conn_fileutils('deletefile', filename)  : deletes file

  conn_fileutils('emptyfile', filename)    : creates empty file

  conn_fileutils('mkdir', dirname)    : creates new folder (see mkdir)

  conn_fileutils('rmdir', dirname)          : removes folder (see rmdir)

  conn_fileutils('linkdir', dirname, link)  : creates symbolic-link to folder

  conn_fileutils('dir', dirname)               : lists files in folder (see dir)

  conn_fileutils('isdir', dirname)       : returns true if dirname is a directory (see isdir)

  conn_fileutils('cd', dirname)           : changes current working directory (see cd)

  conn_fileutils('homedir')              : returns user-specific home directory

  conn_fileutils('uigetfile', ...)          : GUI to select a file (see uigetfile)

  conn_fileutils('uiputfile', ...)                  : GUI to select a new filename (see uiputfile)

  conn_fileutils('imread', filename,...)     : reads image file (see imread)

  conn_fileutils('imwrite', A, filename,...) : writes image file (see imwrite)

  conn_fileutils('imopen', filename)     : displays image file in OS-viewer


LOCAL/REMOTE PATHS

>> conn_server('mnt', mount_point); 

mounts remote server root directory in mount_point local directory

default mount point name = /CONNSERVER

>> conn_server('util_isremotefile', filename)

returns 1 if filename is in remote-storage format (located within mount point)

>> conn_server('util_remotefile', filename)

conforms filename to remote-storage format (e.g. /data -> /CONNSERVER/data)

>> conn_server('util_localfile', filename)

conforms filename to local-storage format (e.g. /CONNSERVER/data -> /data)

>> conn_server('util_isremotevar', var)

returns 1 if variable var is a link to a remote variable


MRI FILE UTILITIES

(note: use /CONNSERVER/* format to refer to files in server)

 >> conn_display(filepath)

displays SPM or CONN analysis results

 >> conn_slice_display(filename, ...)

slice-display view of NIFTI file filename

 >> conn_mesh_display(filename, ...)

3D-display view of NIFTI file filename

 >> conn_mtx_display(filename, ...)

matrix-display view of NIFTI file filename

 >> data = conn_vol_read(filename)

reads volume NIFTI file filename

 >> conn_vol_write(filename, data)

writes volume NIFTI file filename

 >> data = conn_surf_read(filename)

reads surface NIFTI file filename

 >> conn_surf_write(filename, data)

writes surface NIFTI file filename

 >> data = conn_mtx_read(filename)

reads matrix NIFTI file filename

 >> conn_mtx_write(filename, data)

writes matrix NIFTI file filename

Matlab programming tips

Tips when adding remote-storage functionality to an existing function that is designed to work with local files only:


Tip #1: 

Make the function run remotely if it needs to work with remote files

(this works best when the function call requires minimal transfer of information)


Original function syntax:

  function fileout = myfunction(filein, varargin)

      ...

      ... % the code here works with files locally/normally

      ...

   end

New function syntax:

  function fileout = myfunction(filein, varargin)

     % if the input file is remote, run this same function in server

    if any(conn_server('util_isremotefile',filein)), 

         fileout = conn_server('util_remotefile',conn_server('run',mfilename,conn_server('util_localfile',filein),varargin{:})); 

         return

     end

     % otherwise continue normally


      ...

      ... % the code here works with files locally/normally

      ...


   end

Tip #2: 

Make the function bring/pull remote files to work with them locally, and update those remote files if necessary at the end

(this works best when the function needs to combine/mix remote and local files)


Original function syntax:

   function varargout = fcn(filein, varargin)

      ...

      ... % the code here works with files locally/normally

      ...

   end

New function syntax:

   function varargout = fcn(filein, varargin)

      % create local cache/copy of remote file if necessary

      isremotefile=conn_server('util_isremotefile',filein);

      if isremotefile, 

         remotefilename=filein; 

         filein=conn_cache('pull',remotefilename); % note: change 'pull' to 'new' if the remote file does not exist yet

      end


      ...

      ... % the code here works with files locally/normally

      ...


      % update remote file with any changes if necessary

      if isremotefile, conn_cache('push',remotefilename); end

   end

Tip #3: 

If a function manages files using functions that follow template #1-3 conventions then it does not need any additional changes to work with remote-storage files

Tips when adding remote-execution functionality to an existing function that works with large variables:


Tip #4: 

Make the function run remotely if it needs to work with remote variables

(this works best when the function is an intermediate function in a larger pipeline of functions)


Original function syntax:

   function output = fcn(input, varargin)

      ...

      ... % the code here works with "input" and "output" variables locally/normally

      ...

   end

New function syntax:

   function output = fcn(input, varargin)

      % if the input variable is a link to a remote variable, run this same function in server and keep output in server as well

      if conn_server('util_isremotevar',input), 

         output=conn_server('run_keep',mfilename,input,varargin{:}); 

         return

      end

     % otherwise continue normally


      ...

      ... % the code here works with "input" and "output" variables locally/normally

      ...


   end