tutorials/03-Working-with-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3598e82",
   "metadata": {},
   "source": [
    "# Working with data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26ff36c3",
   "metadata": {},
   "source": [
    "On your computer you can open an Explorer which will show you all the files and the directories. In iRODS we have something similar: we have data objects which you can see for the moment as a file and we have collections which are similar to directories.\n",
    "\n",
    "In the course of this and the next tutorials it will become clear that this is only an analogy and that there is more to data objects and collections."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "458b2f0f",
   "metadata": {},
   "source": [
    "## Create an iRODS session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed555077",
   "metadata": {},
   "source": [
    "Again, to work from your Laptop with data in iRODS you will need a `session` as describes in [01-Setup-and-connect](01-Setup-and-connect.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5ef4e10",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.interactive import interactive_auth\n",
    "session = interactive_auth()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "172e153a",
   "metadata": {},
   "source": [
    "## Create a collection and upload a file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9645865a",
   "metadata": {},
   "source": [
    "We will now create a new collection on iRODS in your current iRODS working location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "716769b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.path import IrodsPath\n",
    "from ibridges.data_operations import create_collection\n",
    "irods_path = IrodsPath(session, '~')\n",
    "print(\"Current working location:\", irods_path)\n",
    "coll_path = irods_path.joinpath('demo')\n",
    "print(\"New collection name:\", coll_path)\n",
    "coll = create_collection(session, coll_path)\n",
    "print(\"New collection is created:\", coll_path.collection_exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef9c2fd",
   "metadata": {},
   "source": [
    "Now we will upload a file to that new iRODS collection. In this tutorial we assume that you have a file called `demofile.txt` in your home:\n",
    "- Linux: `/home/<user>/demofile.txt`\n",
    "- Mac: `/Users/<user>/demofile.txt`\n",
    "- Windows: `C:\\Users\\<user>\\demofile.txt`\n",
    "\n",
    "Let's check that first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbe34a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "local_file = Path.home().joinpath('demofile.txt')\n",
    "if local_file.is_file():\n",
    "    print('You are good to follow the next steps!')\n",
    "else:\n",
    "    print(f'Please create a file {local_file} before you continue')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf9182b",
   "metadata": {},
   "source": [
    "Now we can upload that file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6be04e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges import upload\n",
    "ops = upload(session, local_file, coll_path, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ace2c2",
   "metadata": {},
   "source": [
    "You can also see that we get some information what was changed on the iRODS server. **All uploads, downloads and synchronisations will return such a dictionary.** You can also just retrieve the changes, without actually carrying them out by setting the flag `dry-run=True`.\n",
    "\n",
    "We now created a new data object in iRODS:\n",
    "\n",
    "<img src=\"img/DataObject1.png\" width=\"400\">\n",
    "\n",
    "How can we now be sure that the file is uploaded? Let's inspect the collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91b73ddd",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(coll_path.name)\n",
    "print(coll_path.collection_exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e41763a3",
   "metadata": {},
   "source": [
    "We can get the list of `subcollections` and `data_objects` behind an `IrodsPath` like below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48777591",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Current subcollections in {str(coll_path)}:', coll_path.collection.subcollections)\n",
    "print(f'Current data objects in {str(coll_path)}:', coll_path.collection.data_objects)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7897fc6d",
   "metadata": {},
   "source": [
    "**Note,** that if the `IrodsPath` points to a data object you need to use `path.dataobject` tob get the object the path points to. We will see that later on in the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3380dc",
   "metadata": {},
   "source": [
    "## Download the file and the collection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef68a87a",
   "metadata": {},
   "source": [
    "Of course we can also download the data object we have just created. We will download it to your `Downloads` directory on you computer. Let us first retrieve the changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7e770d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_path = Path.home().joinpath('Downloads')\n",
    "assert local_path.is_dir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d4a6914",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges import download\n",
    "ops = download(session, coll_path.joinpath('demofile.txt'), local_path, dry_run=True, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a53bfc",
   "metadata": {},
   "source": [
    "Now let's really download the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d67068c",
   "metadata": {},
   "outputs": [],
   "source": [
    "ops = download(session, coll_path.joinpath('demofile.txt'), local_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f320463",
   "metadata": {},
   "source": [
    "What will happen if we download again?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c755c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "download(session, coll_path.joinpath('demofile.txt'), local_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13e5c96d",
   "metadata": {},
   "source": [
    "You will receive an Exception `OVERWRITE_WITHOUT_FORCE_FLAG`. This means that the file is already there.  \n",
    "\n",
    "**Note, both the upload and the download function do not overwrite existing data by default.** \n",
    "\n",
    "You can force to overwrite the existing data by setting the flag `overwrite`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22496425",
   "metadata": {},
   "outputs": [],
   "source": [
    "ops = download(session, coll_path.joinpath('demofile.txt'), local_path, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b021f026",
   "metadata": {},
   "source": [
    "You also see that no data was transferred. *ibridges* comapres the source and the destination and if the data is exactly the same (checked by so-called [checksums](https://en.wikipedia.org/wiki/Checksum)) the respective data will be ommitted. In the next section we will show you, how to see beforehand, which data will be updated in a transfer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c9925f",
   "metadata": {},
   "source": [
    "### Check if you need to up or download"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19ee9c03",
   "metadata": {},
   "source": [
    "The two functions `upload` and `download` have a parameter `dry_run`. With this parameter you can first check what would be changed, i.e. which data would be updated and which collections/folders would be created in a real up- or download:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8f80d9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ops = download(session, coll_path.joinpath('demofile.txt'), local_path, dry_run=True, overwrite=True)\n",
    "ops.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6de3b9a0",
   "metadata": {},
   "source": [
    "The dictionary will fill the lists behind the keys depending on whether you want to upload or download data. In the example above we want to download data from iRODS to our local file system. In this case only the keys `'create_dir'` and `'download'` will be populated.\n",
    "\n",
    "The `ops` are always returned, not only in a `dry-run`. they can also serve as a summary, what was changed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff8110e",
   "metadata": {},
   "source": [
    "## System metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74c72c43",
   "metadata": {},
   "source": [
    "Both, collections and data objects in iRODS are labeled automatically with some extra information. This information is called *system metadata*. We already saw the `name` and the `path` for the collection. But there is more!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f3e9a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Collection {coll_path.name} was created on {coll_path.collection.create_time}')\n",
    "print(f'The collection was last modified on {coll_path.collection.modify_time}')\n",
    "print(f'The collection was uploaded by and is owned by {coll_path.collection.owner_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19779512",
   "metadata": {},
   "source": [
    "We can also get some system information about the data object we created by uploading the demo file. To inspect a data object (and also a collection) we have to retrieve it from iRODS. We will use the `IrodsPath` to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd40cf7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_path = IrodsPath(session, coll_path.joinpath('demofile.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58906168",
   "metadata": {},
   "source": [
    "Now we can inspect the object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908b0f87",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Data object {obj_path.name} was created on {obj_path.dataobject.create_time}')\n",
    "print(f'Data object {obj_path.name} full path in iRODS is {obj_path}')\n",
    "print(f'The data object was created on {obj_path.dataobject.create_time}')\n",
    "print(f'The data object was last modified on {obj_path.dataobject.modify_time}')\n",
    "print(f'The data object was uploaded by and is owned by {obj_path.dataobject.owner_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "007b5097",
   "metadata": {},
   "source": [
    "**Data objects also carry a `size` and a `checksum`**, with which you can check whether the data reached iRODS completely. Checksums are particularly useful. With the size you can only check whether the length of your local file matches the on in iRODS. However, you cannot see if the contents is really the same. E.g. the length of 'Hello' and 'Hallo' are the same, but they do differ. With a checksum you can detect this. A checksum is a digest of the contents of a file.\n",
    "\n",
    "**Note, both functions, `upload` and `download`, will always calculate and compare the checksums between your computer and iRODS**, to make sure that the data is transferred correctly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73727623",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Data object size', obj.size)\n",
    "print('Data object checksum', obj.checksum)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d605fa",
   "metadata": {},
   "source": [
    "The iRODS full path `obj.path` is the address with which you can get the full data object. Currently, this is the uploaded file and its system metadata\n",
    "\n",
    "<img src=\"img/DataObject2.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3614078",
   "metadata": {},
   "source": [
    "## Data object replicas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a206e8",
   "metadata": {},
   "source": [
    "Usually when we upload a file to another computer we create one new item which can be found under the path that we used to upload the data. \n",
    "\n",
    "Some iRODS systems distribute the uploaded data to different storage systems. That means, that the file is stored on several storage systems, but you have one path under which you can find the data and address the data. This is called in iRODS a data object `replica`. When you download data the system decides for you, which is the most advantageous replica to give to you in terms of speed and integrity.\n",
    "\n",
    "<img src=\"img/DataObject3.png\" width=\"400\">\n",
    "\n",
    "Let us inspect how many replicas of our file we have in iRODS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4ce4f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibridges.data_operations import obj_replicas\n",
    "obj_replicas(obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9438ebba",
   "metadata": {},
   "source": [
    "In Yoda you might have to wait for some time until the data is replicated.\n",
    "\n",
    "The structure of the answer is a list, where each element of this list is structured like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b33598a",
   "metadata": {},
   "outputs": [],
   "source": [
    "item = obj_replicas(obj)[0]\n",
    "print('Relplica index:', item[0])\n",
    "print('Storage system:', item[1])\n",
    "print('Replica cecksum', item[2])\n",
    "print('Replica size', item[3])\n",
    "print('Replica status', item[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e94ef61",
   "metadata": {},
   "source": [
    "The replica status tells you of this particular copy of the data is verified and ok. It should say `good`. In all other cases please send the whole replica information including the path to your local iRODS helpdesk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1143fc1f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}