Globus is a powerful tool for robustly and securely managing data transfers to and from collaborators and within UAB Research Computing. Globus is recommended for most single-use, day-to-day data transfer use-cases.
UAB Research Computing uses High Assurance Endpoints and Collections, meaning there are additional security measures in place to reduce risk and move toward HIPAA compliance. Generally speaking, if you have used Globus in the past, the data transfer interface has not changed, but there are a few new restrictions.
- You will be prompted to prove authorization each time you access a UAB Research Computing endpoint, collection or attempt to download files to your local machine from such an endpoint or collection. If you are already logged in with Single Sign-On (SSO) the process is simple. If not, you will need to authenticate with SSO.
- Bookmarks are not allowed in High Assurance endpoints and collections.
For more detailed information on High Assurance please see the Globus official pages below:
Setting up Globus Connect Personal¶
Globus Connect Personal is software meant to be installed on local machines such as laptops, desktops, workstations and self-owned, local-scale servers. Globus maintains excellent documentation for installation on MacOS, Linux and Windows.
To verify your installation is complete, please visit https://app.globus.org and log in. Click "Endpoints" in the left-hand navigation pane and then click the "Administered By You" tab. Look in the table for the endpoint you just created.
Globus Identities is a concept helping to map Globus Accounts (one per person) to institutions (one or more per person).
Most UAB researchers will have a single identity, their UAB identity, tied to their blazerid. Some researchers may have external collaborations or appointments that provide additional entities which need access to other endpoints on Globus.
To manage your identities, navigate to https://app.globus.org/account/identities and sign in.
To use UAB Research Computing endpoints and collections, you will need to ensure you are using you UAB identity.
Moving Data Between Endpoints¶
Log in to the Globus App online at https://app.globus.org using UAB Single Sign-On (SSO). Start typing "University of Alabama at Birmingham" into the "Use your existing organizational login" text box and selected it when it appears in the list.
Click File Manager in the left-hand navigation pane.
Ensure the center icon for the "Panels" selection is picked.
Click the "Search" icon in the "Collection" text box near the top-left or top-right of the page to locate an endpoint. There are multiple ways to find an endpoint. For some endpoints you may be asked to log in, which is true of all UAB endpoints. Some UAB endpoints may also require that you be on the UAB Campus VPN.
Begin typing in the box to search for an endpoint. To find UAB-related endpoints, search for "UAB". There are two Cheaha endpoints
- Cheaha cluster on-campus (UAB Science DMZ) for machines that are either on the UAB Campus Network, or connected to the UAB Campus VPN.
- Cheaha cluster off-campus (UAB Science DMZ) for machines that are not on the UAB Campus Network and not on the UAB Campus VPN.
The "Recent" tab shows endpoints that have most recently been used.
The "Bookmarks" tab shows a list of endpoint bookmarks. Bookmarks may not reference folders within UAB Research Computing or other High Assurance endpoints or collections.
The "Your Collections" tab shows all endpoints owned by you. For most researchers this will be one or more Globus Connect Personal endpoints.
The "Shared With You" tab shows any private endpoints that have been shared with you by other users, possibly collaborators.
The "More Options" tab will show a brief text on installing Globus Connect Personal.
When an endpoint has been selected you will see a list of folders and files on the default path for that endpoint in the bottom box. You can use the "Path" box to type a path to find the files you are looking for.
Repeat the process of selecting an endpoint for the other "Collection" text box.
When both endpoints have been selected and you have chosen the correct paths for each endpoint, select files and/or folders on the side you wish to transfer FROM. We will call this side the source endpoint, and the other side the target endpoint. Selections may be made by clicking the checkboxes that appear when you hover over each file or folder.
When all files and folders have been selected from the source endpoint, click the "Start" button on the source endpoint side. This will start a transfer process from source to target. The files will be placed in the currently open path on the target endpoint.
A green pop-up notification will appear indicating the transfer has started. Click "View details >" to be taken to the status of the transfer. You can also check on the status of any transfers by clicking the "Activity" button in the left-hand navigation pane.
File permissions from the source will not be copied to the destination. Please read more at this ask.ci FAQ.
Transfer and Sync Options¶
Between the two "Start" buttons on the "File Manager" page is a "Transfer & Sync Options" drop down menu. Click that button to change the options. More information on each option. A brief summary of the options are...
- sync - Sync files only, rather than create new files.
- delete files - Delete any files on the target that are not on the source. Useful for forcing identical filesystems when syncing.
- preserve source - Copies file "modified time" metadata.
- verify integrity - Verifies that checksums are identical on source and target after transfer completes. Highly recommended to have this checked.
- encrypt transfer - Encrypts data before leaving source and decrypts after arriving at destination. Recommended for all transfers, required and enforced for all UAB endpoints.
- skip files - Skips source files that cause errors during the transfer. Otherwise the entire transfer will stop when an error is encountered.
- quota fail - Fails instead of retries when the target storage quota is exceeded.
- File Not Found - This may mean that a file was not readable by Globus. Check that the file hasn't moved or changed names during the transfer. It is recommended to not modify files while they are being transferred by Globus.
- Permission Denied - Globus is not able to access the files because permissions do not allow it. For Globus Connect Personal, be sure the containing folder is on the "Accessible Folders" list. Be sure that your Cheaha account has access to read the file.
Project Space Permissions¶
Globus does not preserve permissions nor ownership when data is transferred, instead using whatever permissions are default at the target location, and making the owner the authenticated user who initiated the transfer. Typically this is not an issue, but may cause problems for project directories. Please see our Project Directory Permissions Section for more information.
A Globus FAQ is available for additional information on endpoints and transfers.
UAB Researcher Computing has subscriptions to connectors for cloud services and other types of filesystems.
UAB Box Connector¶
To use the UAB Box Connector, search for an endpoint like usual and enter "UAB Box" into the search box. Select the endpoint labeled "UAB Box". You should see a list of files and folders that are available to you at https://uab.app.box.com. File transfers work as they would with any other endpoint or collection.
Long-term Storage S3 (LTS) Connector¶
LTS behaves differently from other file systems and comes with a few possible pitfalls. Keep in mind the following three rules: (1) all data must be in buckets, (2) buckets are only allowed in the root folder, and (3) buckets must have unique names.
To use the UAB LTS Connector, search for an endpoint like usual and enter "UAB LTS" into the search box. Select the endpoint labeled "UAB Research Computing LTS (Long Term Storage aka S3)". If you have stored data within LTS already you should see a list of folders, otherwise you will see an empty space where folders may be placed. Each folder corresponds to a bucket in LTS. To create a bucket, click "New Folder" in the "File Manager" window in Globus. Note that buckets must have globally unique names. Read on for more information about possible pitfalls.
Data Must be in Buckets¶
All data transferred to LTS must be placed in a bucket, and may not be placed directly into the root directory. Attempting to move data to the root directory will result in an unhelpful error message in the "Activity" window.
Clicking on the "view event log" link shows the following.
Error (transfer) Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) Server: m-a201b5.9ad93.a567.data.globus.org:443 Command: STOR /test.log Message: The connection to the server was broken --- Details: an end-of-file was reached\nglobus_xio: An end of file occurred\n
Buckets Must Have Globally Unique Names¶
When creating new buckets, the name must be unique across all buckets on the system. At first this may sound very restrictive, but it is quite simple to deal with in practice. See our LTS section on good naming practice for how to avoid duplicate names.
If a duplicate bucket name is entered, a long error message will appear in a small space next to the new bucket name. The message reads like the following, expanded for readability.
Bad Gateway: Endpoint Error, Error (mkdir) Endpoint: UAB Research Computing LTS (Long Term Storage aka S3) (184408b4-d04b-4513-9912-8feeb6adcab3) Server: m-b81a79.9ad93.a567.data.globus.org:443 Command: MKD /test/ Message: Fatal FTP Response --- Details: 553- GlobusError: v=1 c=PATH_EXISTS\r\n553- GridFTP-Path: (null)\r\n553-globus_gridftp_server_s3_base: S3 Error accessing "": ErrorBucketAlreadyExists: ErrorBucketAlreadyExists: \r\n553 End.\r\n
To save a bookmark, use the File Manager interface to select an endpoint and navigate to a path on that endpoint. Then click the bookmark button as shown below.
To manage bookmarks, click "Bookmarks" in the left-hand navigation pane. Click the "Pencil" icon to edit a bookmark. Click the "Trash Bin" icon to delete a bookmark.
It is not possible to create bookmarks within High Assurance Endpoints.
Managing Shared Collections from a Globus Connect Personal Endpoint¶
It is NOT RECOMMENDED to make Globus Connect Personal endpoints public as this is insecure. It is more difficult to manage access controls for the entire Globus Connect Personal endpoint than for a shared collection. Shared collections make it simpler to share different data with distinct collaborators, and to manage who has access to what data. Be secure, use shared collections!
Creating a Shared Collection¶
Click "Endpoints" in the left-hand navigation pane.
Click the "Administered By You" tab.
In the table, find the endpoint you wish to share data from and click its name. You will be taken to the page for that endpoint.
Click the "Collections" tab.
Click the "Add a Guest Collection" button.
Fill out the form.
- Manually enter a path or click the Browse button to select a folder.
- Give a short but memorable name for your shared collection. This information will be useful for your collaborators.
- Optionally fill in a more detailed description of the shared collection for your records.
- Optionally fill in searchable keywords.
Click "Create Share" to move to the next step. You will be taken to the page for the newly created collection, which is now a full-fledged endpoint. Any further references to "an endpoint" will be about the newly created, shared collection.
Make sure you are on the "Permissions" tab. You should see a permissions table with your name in the first row.
Click "Add Permissions -- Share With" to share your endpoint with other users.
Fill out the form.
- Optionally enter a path within the shared endpoint or use the Browse button. If you leave the path as just a slash, the entire shared endpoint will be shared with these users.
Select who to share with.
- User - One or more users.
- Group - All members of a group.
All Users - All globus users.
This will expose information to everyone on Globus!
Search for users to add, or a group, depending on your choice above. You should be able to find any globus user using the search box.
Be certain of which user you are selecting! Check the email address domain.
If adding users, optionally enter a message so they know why they are being added.
- Select permissions. Read is automatically selected and cannot be changed. Write permissions are optional.
Click "Add Permission" to add permissions for these users or groups. You will be returned to the page for the shared endpoint and should be on the "Permissions" tab and should see the user or group in the table.
Deleting a Shared Collection¶
Click "Endpoints" in the left-hand navigation pane, then c
Click the "Administered By You" tab.
Click the right caret ">" icon at the right side of the row with the endpoint you wish to delete. You will be taken to the information page for that endpoint.
Click "X Delete Endpoint" and a confirmation dialog will open at the top of the page. Respond to the dialog to delete the endpoint, or to cancel.