Embedded Files
INITIALIZATION
>> conn remotely on
starts SSH-based encrypted/secure connection with remote server (this will automatically launch a new server if the last one is down)
>> conn remotely off
ends connection with remote server
REMOTE EXECUTION
>> conn_server cmd
starts remote command-line prompt (type "quit" to return to local command-line prompt)
>> [out1, out2, ...] = conn_server('cmd', expression);
evaluates expression in remote server
>> conn_server('run', fcn_name, fcn_arg1, fcn_arg2, …);
runs function/script in remote server (blocks execution until function finishes)
>> conn_server(‘run_immediatereturn', fcn_name, fcn_arg1, fcn_arg2, …);
runs function/script in remote server (does not block execution)
>> [out1, out2, …] = conn_server('run', fcn_name, fcn_arg1, fcn_arg2, …);
runs function in remote server, returns output arguments locally
>> [out1, out2, …] = conn_server(‘run_keep', fcn_name, fcn_arg1, fcn_arg2, …);
runs function in remote server, keeping output arguments remotely (returns link to remote variables)
REMOTE JOB SUBMISSION
>> conn('submit', fcn_name, fcn_arg1, fcn_arg2, ... );
runs function in remote server HPC/cluster environment (blocks execution)
>> status = conn('submit', fcn_name, fcn_arg1, fcn_arg2, ... );
runs function in remote server HPC/cluster environment (does not block execution)
https://www.conn-toolbox.org/resources/documentation
FILE TRANSFER
>> conn_server('push', file_local, file_remote);
copies local file to remote server
>> conn_server('pull', file_remote, file_local);
copies file in remote server to local computer
>> conn_server_ssh('folderpush', folder_local, folder_remote);
copies local directory to remote server using SCP
>> conn_server_ssh('folderpull', folder_remote, folder_local);
copies directory in remote server to local computer using SCP
CONFIGURATION
>> conn remotely setup;
installs remote-access options (run once in server)
>> edit ~/connserverinfo.json
edits remote-access options (run in server)
>> conn jobmanager settings;
change remote server HPC/cluster environment options
ADVANCED COMMUNICATIONS
>> conn_server('connect', IP , SERVER_ID);
connects to CONN server through existing channel (SERVER_ID = portnumberCONNprivatekey)
>> conn_server('start' [, PORT, PUBLICKEY]);
initializes CONN server and open communication channel (run in server)
>> conn_server disconnect
closes communication channel to remote server
>> conn_server exit
terminates remote server and closes communication channel
remote storage
remote storage
FILE CACHE
(note: use /CONNSERVER/* format to refer to files in server)
>> file_local = conn_cache('pull', file_remote);
creates local-copy of a remote file
>> conn_cache('push', file_remote);
updates a remote file with any changes to its local-copy
>> file_local = conn_cache('new', file_remote);
creates local-copy of a yet-to-be-created remote file
>> conn_cache clearall
clears all locally-cached files
GENERAL FILE UTILITIES
(note: use /CONNSERVER/* format to refer to files in server)
>> data = conn_loadmatfile(filename, ...)
loads data from a MAT-file (same syntax as Matlab's load function)
>> conn_savematfile(filename, ...)
saves data to a MAT-file (same syntax as Matlab's save function)
>> conn_fileutils( fcn_name, ...)
miscelaneous file management utility functions (overloading original fcn_name functions to add support for files/folders located within remote server):
conn_fileutils('fileread', filename) : reads text file (see fileread)
conn_fileutils('filewrite', filename, str) : creates text file with cell array str (one line per element)
conn_fileutils('fileappend', filename, str) : appends text file with cell array str (one line per element)
conn_fileutils('textread', filename) : reads formatted data from text file (see textread)
conn_fileutils('copyfile', source, target) : copies file (within same-machine transfers only)
conn_fileutils('movefile', source, target) : moves file
conn_fileutils('renamefile', source, target) : renames file
conn_fileutils('deletefile', filename) : deletes file
conn_fileutils('emptyfile', filename) : creates empty file
conn_fileutils('mkdir', dirname) : creates new folder (see mkdir)
conn_fileutils('rmdir', dirname) : removes folder (see rmdir)
conn_fileutils('linkdir', dirname, link) : creates symbolic-link to folder
conn_fileutils('dir', dirname) : lists files in folder (see dir)
conn_fileutils('isdir', dirname) : returns true if dirname is a directory (see isdir)
conn_fileutils('cd', dirname) : changes current working directory (see cd)
conn_fileutils('homedir') : returns user-specific home directory
conn_fileutils('uigetfile', ...) : GUI to select a file (see uigetfile)
conn_fileutils('uiputfile', ...) : GUI to select a new filename (see uiputfile)
conn_fileutils('imread', filename,...) : reads image file (see imread)
conn_fileutils('imwrite', A, filename,...) : writes image file (see imwrite)
conn_fileutils('imopen', filename) : displays image file in OS-viewer
LOCAL/REMOTE PATHS
>> conn_server('mnt', mount_point);
mounts remote server root directory in mount_point local directory
default mount point name = /CONNSERVER
>> conn_server('util_isremotefile', filename)
returns 1 if filename is in remote-storage format (located within mount point)
>> conn_server('util_remotefile', filename)
conforms filename to remote-storage format (e.g. /data -> /CONNSERVER/data)
>> conn_server('util_localfile', filename)
conforms filename to local-storage format (e.g. /CONNSERVER/data -> /data)
>> conn_server('util_isremotevar', var)
returns 1 if variable var is a link to a remote variable
MRI FILE UTILITIES
(note: use /CONNSERVER/* format to refer to files in server)
>> conn_display(filepath)
displays SPM or CONN analysis results
>> conn_slice_display(filename, ...)
slice-display view of NIFTI file filename
>> conn_mesh_display(filename, ...)
3D-display view of NIFTI file filename
>> conn_mtx_display(filename, ...)
matrix-display view of NIFTI file filename
>> data = conn_vol_read(filename)
reads volume NIFTI file filename
>> conn_vol_write(filename, data)
writes volume NIFTI file filename
>> data = conn_surf_read(filename)
reads surface NIFTI file filename
>> conn_surf_write(filename, data)
writes surface NIFTI file filename
>> data = conn_mtx_read(filename)
reads matrix NIFTI file filename
>> conn_mtx_write(filename, data)
writes matrix NIFTI file filename
Matlab programming tips
Matlab programming tips
Tips when adding remote-storage functionality to an existing function that is designed to work with local files only:
Tip #1:
Tip #1:
Make the function run remotely if it needs to work with remote files
(this works best when the function call requires minimal transfer of information)
Original function syntax:
function fileout = myfunction(filein, varargin)
...
... % the code here works with files locally/normally
...
end
New function syntax:
function fileout = myfunction(filein, varargin)
% if the input file is remote, run this same function in server
if any(conn_server('util_isremotefile',filein)),
fileout = conn_server('util_remotefile',conn_server('run',mfilename,conn_server('util_localfile',filein),varargin{:}));
return
end
% otherwise continue normally
...
... % the code here works with files locally/normally
...
end
Tip #2:
Tip #2:
Make the function bring/pull remote files to work with them locally, and update those remote files if necessary at the end
(this works best when the function needs to combine/mix remote and local files)
Original function syntax:
function varargout = fcn(filein, varargin)
...
... % the code here works with files locally/normally
...
end
New function syntax:
function varargout = fcn(filein, varargin)
% create local cache/copy of remote file if necessary
isremotefile=conn_server('util_isremotefile',filein);
if isremotefile,
remotefilename=filein;
filein=conn_cache('pull',remotefilename); % note: change 'pull' to 'new' if the remote file does not exist yet
end
...
... % the code here works with files locally/normally
...
% update remote file with any changes if necessary
if isremotefile, conn_cache('push',remotefilename); end
end
Tip #3:
Tip #3:
If a function manages files using functions that follow template #1-3 conventions then it does not need any additional changes to work with remote-storage files
Tips when adding remote-execution functionality to an existing function that works with large variables:
Tip #4:
Tip #4:
Make the function run remotely if it needs to work with remote variables
(this works best when the function is an intermediate function in a larger pipeline of functions)
Original function syntax:
function output = fcn(input, varargin)
...
... % the code here works with "input" and "output" variables locally/normally
...
end
New function syntax:
function output = fcn(input, varargin)
% if the input variable is a link to a remote variable, run this same function in server and keep output in server as well
if conn_server('util_isremotevar',input),
output=conn_server('run_keep',mfilename,input,varargin{:});
return
end
% otherwise continue normally
...
... % the code here works with "input" and "output" variables locally/normally
...
end
Page updated
Google Sites
Report abuse