R image: update python, pangeo-notebook and rstudio-server

[emiliom]

Update these core packages to recent versions. First time we update these since 2022.

See #90 (comment)

This update is in support of the OHW-espanol event in a couple of weeks. ref oceanhackweek/Hub-Management#6

[emiliom]

@abkfenris the build_push check failed:

Traceback (most recent call last):
  File "/opt/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

I built the image locally (make r-build) without any problems. Could you help me debug this??

Thanks!

[abkfenris]

You'd think conda would be able to find conda...

[abkfenris]

Hmm, it's a pretty old miniconda image, so I wonder if there are somehow multiple versions of conda conflicting.

--- a/r/Dockerfile
+++ b/r/Dockerfile
@@ -1,5 +1,5 @@
 #syntax=docker/dockerfile:1.3
-FROM continuumio/miniconda3:4.12.0@sha256:58b1c7df8d69655ffec017ede784a075e3c2e9feff0fc50ef65300fc75aa45ae
+FROM continuumio/miniconda3:24.9.2-0@sha256:39b820af05f0f9bb986ca145a0d077bbd3bd866256277dd869fefb93bd03cd21

 ENV NB_USER jovyan

I tried changing a handful of things, but in my tinkering this morning I couldn't get Rstudio to launch from Jupyterlab by running make r-lab.

[emiliom]

Thanks. I should've tried to running make r-lab too, but forgot about it. I tried it now, and sure enough, I run into the same error as on the GH action (and same one you're seeing, presumably):

File "/opt/conda/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

Hmm.

[emiliom]

For what it's worth, I followed the steps in https://hub.docker.com/r/continuumio/miniconda3 to pull the latest miniconda3 image and launch jupyter notebook server. In a new notebook, from conda.cli import main worked. It also worked on the python shell.

[emiliom]

I landed on a solution: update conda before doing conda install. After adding conda update -n root conda && \ right before conda install, I'm able to launch jupyter lab with make r-lab!

But on to the next problem: when I launch the RStudio kernel from jlab, I get a 500: Internal Server Error. Sigh.

And the build_push check failed with the same error as before. So the conda update fixed that immediate problem on my end, locally, but not on the CI. Hmm.

Actually, re-reading your message:

I tried changing a handful of things, but in my tinkering this morning I couldn't get Rstudio to launch from Jupyterlab by running make r-lab.

I now see that you were already at that stage, where you could launch jlab but not launch Rstudio from jlab. I thought what you meant was that you couldn't launch jlab. I'll keep digging.

[emiliom]

I updated the miniconda image to the version you listed, then explicitly added two library deb packages that were causing more subtle errors (including with nb_conda_kernels, I think): libarchive13 and libfmt-dev.

The CI tests are passing now! But Rstudio is still not launching successfully from jupyter lab. Rather than an immediate 500 error, as before, there's a wait of a few seconds, then the 500 error plus "The error was: could not start rstudio in time".

I added libarchive13 to try to address this error when launching jupyter lab with make r-lab:

conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory)

This error doesn't prevent jlab from launching and behaving normally (it seems). But I figured it's something I could address. Adding libarchive13 didn't fix it (I also tried libarchive-dev). I'm going to try one more strategy I just ran into to solve the libarchive.so error.

[abkfenris]

If nb_conda_kernels is causing issues, we should be able to get rid of it for this image. It shouldn't be necessary to run Rstudio. I think I tested with it removed, but still couldn't get anything other than a 500 trying to launch Rstudio.

[abkfenris]

I might explore the links off of here: https://discourse.jupyter.org/t/a-template-repository-for-running-latest-rstudio-r-on-mybinder-org-your-jupyterhub/12235

[emiliom]

I've pushed my last commits for the day. I've now eliminated all conda / libmamba errors. The last set of errors involved conflicts between libmamba and libarchive coming from conda-forge vs main channels.

There are no clear errors left in the run-time logs from make r-build and make r-lab, other than these warnings on make r-lab, which I assume are not critical.

[W 2024-11-14 23:25:32.326 ServerApp] No web browser found: Error('could not locate runnable browser').
...
[W 2024-11-14 23:26:33.071 LabApp] Could not determine jupyterlab build status without nodejs

BUT, RStudio still doesn't launch 😞. I still get this behavior: there's a wait of a few seconds, then the 500 error plus "The error was: could not start rstudio in time".

And the errors emitted when I click on the RStudio launcher in Jupyter Lab:

r-1  | [E 2024-11-14 23:26:54.725 ServerApp] {
r-1  |       "Host": "127.0.0.1:8080",
r-1  |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
r-1  |       "Referer": "http://127.0.0.1:8080/lab",
r-1  |       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0"
r-1  |     }
r-1  | [E 2024-11-14 23:26:54.725 ServerApp] 500 GET /rstudio/ (931ac30145db4858a308de6b5c2be215@172.19.0.1) 15033.48ms referer=http://127.0.0.1:8080/lab

[emiliom]

Sigh, the build_push check failed; I was expecting it to work. But it looks like it's a temporary blip on the CI:

ERROR: failed to solve: rpc error: code = Unknown desc = write /opt/conda/lib/python3.12/site-packages/babel/locale-data/ta.dat: no space left on device
Error: buildx failed with: ERROR: failed to solve: rpc error: code = Unknown desc = write /opt/conda/lib/python3.12/site-packages/babel/locale-data/ta.dat: no space left on device

I left nb_conda_kernels on the conda env, and everything is fine.

I might explore the links off of here: https://discourse.jupyter.org/t/a-template-repository-for-running-latest-rstudio-r-on-mybinder-org-your-jupyterhub/12235

I saw that earlier, but I didn't look into it.

[emiliom]

FYI the "no space left on device" error I ran into earlier turned out to be a very real error.

It looks like it's a docker error involving the GitHub limit of 14 GB for images. But I implemented a solution (adding this step: rm -rf /opt/hostedtoolcache in build-image.yml) that apparently worked, since the build check got through. For reference:

• https://github.com/orgs/community/discussions/25678#discussioncomment-5242449
• GitHub Actions failes with: "no space left on device" actions/runner-images#2875

That's just a CI problem, though. I'm still working on the core problem of not being able to launch RStudio from Jupyter Lab in local tests. I'll try some things today.

[abkfenris]

Ya, I've run into that a few times as they update the underlying image and have lots of language versions installed.

It might also be possibly to build on top of rocker, and then launch right into RStudio. https://github.com/search?q=repo%3A2i2c-org%2Finfrastructure%20rstudio&type=code

[abkfenris]

https://infrastructure.2i2c.org/howto/features/rocker/

[abkfenris]

It looks like their user package installer can now throw errors and stop a build rather than silently passing them. https://rocker-project.org/use/extending.html#install2.r

[emiliom]

I started looking into rocker a couple of days ago. It does look like the way to go going forward. But for me at least, making that switch now would be taking on too much beyond my depth for the upcoming event (starts next Monday 11/25).

I've tried downgrading pangeo-notebook, the Python version and the rstudio-server package, but still haven't made progress. I'm very, very close to giving up and falling back to our old setup. I'll ask on the 2i2c Slack (that I just joined) and see if that leads somewhere, before giving up by tomorrow.

[agoose77]

@emiliom 👋

@jnywong cross-posted your message on 2i2c's Slack into one of our private channels, and I took a quick look.

When I try this PR, the jupyter server instance throws a traceback:

2024-11-20T09:43:44.819637Z [rserver] ERROR system error 13 (Permission denied) [path: /home/jovyan/.cache/rstudio, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:848; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:845
TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/rstudio/logging.conf'. Logging to '/home/jovyan/.local/share/rstudio/log/rserver.log'.
rserver[1831]: ERROR system error 13 (Permission denied) [path: /home/jovyan/.cache/rstudio, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:848; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:845

If you chown the cache directory such that NB_USER has the appropriate permissions, you will then be able to launch RStudio from within the image:

RUN chown -R ${NB_USER} ${HOME}/.cache
USER ${NB_USER}

Long-term, I agree that moving to something like Rocker seems like a good suggestion, but hopefully this unblocks you? Feel free to ping me if you run into any more problems. I'm not an R expert, but I am very familiar with containers & the Jupyter stack.

[emiliom]

OMG, that did it!!! Thank you so much @agoose77!! This is a huge relief for me.

I'll update the PR. Again, THANK YOU!!

[agoose77]

@emiliom a pleasure! I haven't tested beyond "does it launch into RStudio". Keeping fingers crossed 🤞