Skip to content

Commit ad873f4

Browse files
JSKenyonJSKenyon
committed
Merge in master. Resolve conflicts. PFB not installed at present.
2 parents a781f1d + 4a40099 commit ad873f4

22 files changed

+462
-129
lines changed

‎.github/workflows/python-package.yml

+7-1
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,13 @@ jobs:
3939
poetry run flake8 cultcargo tests --count --select=E9,F63,F7,F82 --show-source --statistics
4040
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
4141
poetry run flake8 cultcargo tests --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
42-
- name: Test command
42+
- name: Test stimela command
4343
run: |
4444
poetry run stimela --help
4545
poetry run stimela doc tests/test-recipe.yml
46+
- name: Install builder deps
47+
run: |
48+
poetry install --with builder --no-interaction
49+
- name: List all quay.io images
50+
run: |
51+
poetry run cultcargo/builder/build-cargo.py -a -l

‎cultcargo/builder/build-cargo.py

+297-80
Large diffs are not rendered by default.

‎cultcargo/builder/cargo-manifest.yml

+39-16
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,32 @@
1-
assign:
2-
CULT_REGISTRY: quay.io/stimela2
3-
CULT_VERSION: 0.0.2
1+
metadata:
2+
# name of python package
3+
PACKAGE: cult-cargo
4+
# its repository
5+
GITHUB_REPOSITORY: caracal-pipeline/cult-cargo
6+
# registry to use. Use module::filename.yml::variable format to pull from a config file inside a python module
7+
REGISTRY: cultcargo.genesis::cult-cargo-base.yml::vars.cult-cargo.images.registry
8+
# image bundle ersion. Use module::filename.yml::variable format to pull from a config file inside a python module
9+
BUNDLE_VERSION: cultcargo.genesis::cult-cargo-base.yml::vars.cult-cargo.images.version
10+
# prefix to be removed from image version when comparing to python package version
11+
BUNDLE_VERSION_PREFIX: cc
12+
# path to images. Use module::filename to refer to content of module
13+
BASE_IMAGE_PATH: cultcargo::images
414

15+
assign:
516
# standard variables used in templated Docker files
617
pre_install: ''
718
post_install: ''
819
extra_deps: ''
20+
21+
# base image for generic Python-based packages
22+
base_python_image: python-astro:3.9
23+
# corresponding python binary
24+
python: python3.9
925

1026
images:
1127
base-cult:
1228
versions:
13-
base:
14-
version: '{CULT_VERSION}'
29+
base: {}
1530

1631
wsclean:
1732
versions:
@@ -39,7 +54,6 @@ images:
3954

4055
python-astro:
4156
versions:
42-
'3.10': {}
4357
'3.9': {}
4458
'3.8': {}
4559

@@ -83,12 +97,10 @@ images:
8397
versions:
8498
# '0.3.0':
8599
# package: crystalball==0.3.0
86-
# latest:
87-
# package: crystalball
100+
# latest:
101+
# package: crystalball # doesn't work?
88102
master:
89103
package: git+https://github.com/caracal-pipeline/crystalball
90-
issue-51:
91-
package: git+https://github.com/caracal-pipeline/crystalball@issue-51
92104

93105
tricolour:
94106
# assign:
@@ -122,15 +134,21 @@ images:
122134

123135
quartical:
124136
versions:
125-
base:
126-
package: git+https://github.com/ratt-ru/QuartiCal@v0.2.1-dev
127-
# experimental:
128-
# package: git+https://github.com/ratt-ru/QuartiCal@v0.2.1-experimental-dask-ms
137+
'0.2.2':
138+
package: quartical==0.2.2
139+
latest:
140+
package: quartical
141+
master:
142+
package: git+https://github.com/ratt-ru/QuartiCal
129143

130144
cubical:
131145
versions:
132-
base:
133-
package: git+https://github.com/ratt-ru/CubiCal@noxcal
146+
'1.6.4':
147+
package: cubical==1.6.4
148+
latest:
149+
package: cubical
150+
master:
151+
package: git+https://github.com/ratt-ru/CubiCal
134152

135153
pfb-clean:
136154
versions:
@@ -161,3 +179,8 @@ images:
161179
package: aimfast==1.3.4
162180
base:
163181
package: git+https://github.com/Athanaseus/aimfast.git
182+
183+
rfinder:
184+
versions:
185+
master:
186+
package: git+https://github.com/caracal-pipeline/RFInder.git

‎cultcargo/casa-flag.yml

+2
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ cabs:
1414
dtype: MS
1515
required: true
1616
nom_de_guerre: vis
17+
writable: true
18+
write_parent_dir: true
1719
versionname:
1820
info: "flag version name"
1921
mode:

‎cultcargo/fitstool.yml

+2
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ cabs:
3939
dtype: bool
4040
diff:
4141
dtype: bool
42+
prod:
43+
dtype: bool
4244
force:
4345
dtype: bool
4446
default: true

‎cultcargo/genesis/cult-cargo-base.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ vars:
22
cult-cargo:
33
images:
44
registry: quay.io/stimela2
5-
version: cc0.0.2
5+
version: cc0.1.2

‎cultcargo/genesis/quartical/argument_schema.yaml

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
input_ms:
22
path:
33
required: true
4-
dtype: str
4+
dtype: URI
5+
writable: true
56
info:
67
Path to input measurement set.
78

‎cultcargo/genesis/wsclean/__init__.py

+13-6
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ def img_output(imagetype, desc, path, glob=True, must_exist=False):
1717

1818

1919
def make_stimela_schema(params: Dict[str, Any], inputs: Dict[str, Parameter], outputs: Dict[str, Parameter]):
20-
"""Augments a schema for stimela based on solver.terms"""
20+
"""Augments a schema for stimela based on wsclean settings"""
2121

2222
# predict mode has no outputs
2323
if params.get('predict'):
@@ -29,12 +29,19 @@ def make_stimela_schema(params: Dict[str, Any], inputs: Dict[str, Parameter], ou
2929
nchan = params.get('nchan', 1)
3030
multichan = params.get('multi.chan', not isinstance(nchan, int) or nchan > 1)
3131

32-
stokes = params.get('pol', "I").upper()
32+
stokes = params.get('pol')
33+
34+
if stokes is None:
35+
stokes = ["I"] # shouldn't matter, multistokes will be False unless explicit
36+
elif isinstance(stokes, str):
37+
stokes = stokes.upper()
38+
# if just IQUV characters, break apart into list
39+
if all(p in "IQUV" for p in stokes):
40+
stokes = list(stokes)
41+
else:
42+
stokes = [stokes]
3343
# multi.stokes can be set explicitly
34-
multistokes = params.get('multi.stokes', stokes != "I")
35-
# if explicitly multi-pol, set Stokes list to "IQUV" if it's not explicitly set to something else
36-
if multistokes and stokes == "I":
37-
stokes = "IQUV"
44+
multistokes = params.get('multi.stokes', False) or len(stokes) > 1
3845

3946
# ntime -- if not an integer, assume runtime evaluation and >=2 then
4047
ntime = params.get('intervals-out', 1)

‎cultcargo/genesis/wsclean/wsclean-base.yml

+10-4
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,18 @@ lib:
2929
dtype: List[int]
3030
policies:
3131
repeat: ' '
32-
pol:
33-
choices: [I, IQ, IV, IQUV]
34-
default: I
32+
pol:
33+
dtype: Union[str, List[str]]
34+
element_choices: [I,Q,U,V,QU,QUV,IQ,IV,IQU,IQUV,XX,XY,YX,YY,LL,LR,RL,RR]
35+
policies:
36+
repeat: ","
3537
join-polarizations:
3638
dtype: bool
3739
link-polarizations:
38-
choices: [I, Q, U, V, IQ, IU, QU, IV]
40+
dtype: Union[str, List[str]]
41+
element_choices: [I,Q,U,V,QU,QUV,IQ,IV,IQU,IQUV,XX,XY,YX,YY,LL,LR,RL,RR]
42+
policies:
43+
repeat: ","
3944
intervals-out:
4045
dtype: int
4146
threads:
@@ -132,6 +137,7 @@ lib:
132137
temp-dir:
133138
dtype: Directory
134139
skip_freshness_checks: true
140+
writable: true
135141
log-time:
136142
dtype: bool
137143
interval:

‎cultcargo/images/Dockerfile.generic.pip

+2-4
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,8 @@
1-
FROM {CULT_REGISTRY}/python-astro:3.8-cc{CULT_VERSION}
2-
3-
RUN pip install -U pip setuptools wheel
1+
FROM {REGISTRY}/{base_python_image}-{BUNDLE_VERSION}
42

53
{pre_install}
64

7-
RUN pip install --no-cache-dir {package} {extra_deps}
5+
RUN {python} -mpip install --no-cache-dir {package} {extra_deps}
86

97
{post_install}
108

‎cultcargo/images/casa/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/casa:_base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/casa:_base-{BUNDLE_VERSION}
22

33
RUN wget -nv {url}/{tarball} && tar zxf {tarball} && rm {tarball}
44

‎cultcargo/images/casa/Dockerfile.base

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/base-cult:base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/base-cult:base-{BUNDLE_VERSION}
22

33
RUN apt-get update && apt-get -y upgrade && \
44
apt-get -y install libfreetype6 \

‎cultcargo/images/casa6/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/python:3.8
1+
FROM {REGISTRY}/base-cult:{BUNDLE_VERSION}
22

33
RUN pip install --no-cache-dir casatools=={wheel_version} casatasks=={wheel_version}
44

‎cultcargo/images/python-astro/Dockerfile

+9-8
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,25 @@
1-
FROM {CULT_REGISTRY}/base-cult:base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/base-cult:base-{BUNDLE_VERSION}
22

33
ENV DEBIAN_FRONTEND=noninteractive
44

55
RUN apt-get update && apt-get install apt-utils && \
6-
apt-get install -y casacore-data casacore-tools python{VERSION} python3-pip python3-matplotlib \
6+
apt-get install -y casacore-data casacore-tools python{VERSION}-dev \
7+
python3-pip python3-matplotlib \
78
&& rm -rf /var/lib/apt/lists/* \
89
&& rm -rf /var/cache/apt/archives
910

11+
RUN update-alternatives --install /usr/bin/python python /usr/bin/python{VERSION} 1
12+
1013
# upgrade pip
11-
RUN pip install --no-cache-dir -U pip setuptools wheel
14+
RUN python{VERSION} -mpip install --no-cache-dir -U pip setuptools wheel
1215

1316
# add useful astro stuff
14-
RUN pip install --no-cache-dir \
17+
RUN python{VERSION} -mpip install --no-cache-dir \
1518
python-casacore astropy astroplan regions astro-tigger-lsm \
1619
owlcat dask-ms scipy omegaconf bdsf msutils
1720

18-
# change this to master or release version when ready
19-
RUN pip install --no-cache-dir git+https://github.com/caracal-pipeline/stimela@singularize
20-
21-
RUN ln -s /usr/bin/python{VERSION} /usr/bin/python
21+
# add stimela -- useful for scabha.schema_utils
22+
RUN python{VERSION} -mpip install --no-cache-dir git+https://github.com/caracal-pipeline/stimela@2.0rc12
2223

2324
CMD /usr/bin/python{VERSION}
2425

‎cultcargo/images/rfinder/Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../Dockerfile.generic.pip

‎cultcargo/images/tricolour/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/base-cult:base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/base-cult:base-{BUNDLE_VERSION}
22

33
RUN pip install -U pip setuptools wheel
44

‎cultcargo/images/wsclean/Dockerfile.build

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/base-cult:base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/base-cult:base-{BUNDLE_VERSION}
22

33
RUN apt-get update && \
44
apt-get -y install casacore-dev libgsl-dev libhdf5-dev \

‎cultcargo/images/wsclean/Dockerfile.build.2x

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM {CULT_REGISTRY}/base-cult:base-cc{CULT_VERSION}
1+
FROM {REGISTRY}/base-cult:base-{BUNDLE_VERSION}
22

33
RUN apt-get update && \
44
apt-get -y install casacore-dev libgsl-dev libhdf5-dev \

‎cultcargo/msutils.yml

+2
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ cabs:
1313
msname:
1414
dtype: MS
1515
required: true
16+
writable: true
1617
fromcol:
1718
dtype: str
1819
required: true
@@ -31,6 +32,7 @@ cabs:
3132
msname:
3233
dtype: MS
3334
required: true
35+
writable: true
3436
colname:
3537
dtype: str
3638
required: true

‎cultcargo/rfinder.yml

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
_include:
2+
- genesis/cult-cargo-base.yml
3+
4+
cabs:
5+
rfinder:
6+
info: Tool for identify the presence of RFI in an observation, flagging and visualisation (https://github.com/caracal-pipeline/RFInder.git)
7+
command: rfinder
8+
image:
9+
_use: vars.cult-cargo.images
10+
name: rfinder
11+
policies:
12+
prefix: '--'
13+
inputs:
14+
ms:
15+
dtype: MS
16+
info: input Measuremnt Set file
17+
nom_de_guerre: input
18+
input_dir:
19+
dtype: Directory
20+
info: select working directory (MS file assumed to be here)
21+
default: './'
22+
output_dir:
23+
dtype: Directory
24+
info: select output directory
25+
label:
26+
dtype: str
27+
info: "ouput folder is called - rfi_<polarization>_<label>"
28+
field:
29+
dtype: int
30+
info: select field ID of MS file to analyze
31+
default: 0
32+
polarization:
33+
dtype: str
34+
choices: [xx, yy, xy, yx, q]
35+
default: q
36+
info: select stokes parameter to determine rfi presence
37+
ncpu:
38+
dtype: int
39+
info: Number of cpu to use when generating summary stats
40+
summary_options:
41+
dtype: Union[str, List[str]]
42+
info: Flag stats to determine
43+
element_choices: [ant,corr,scan,freq]
44+
default: [ant,corr]
45+
policies:
46+
repeat: list
47+
freq_bin:
48+
dtype: int
49+
info: Number of frequencies to bin into a single channel
50+
plot_summary:
51+
dtype: bool
52+
info: plot percentage of RFI
53+
default: true
54+
plot_details:
55+
dtype: bool
56+
info: plot percentage of RFI, 'rfi', or noise, 'noise', or factor of noise increase
57+
default: false
58+
rfimode:
59+
dtype: str
60+
choices: [use_flags, rms_clip]
61+
info: use_flags
62+
outputs:
63+
outfile:
64+
dtype: Directory
65+
implicit: =GLOB("rfi*{current.label}")

‎cultcargo/tricolour.yml

+1
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ cabs:
1616
dtype: MS
1717
policies:
1818
positional: true
19+
writable: true
1920
config:
2021
dtype: File
2122
ignore-flags:

0 commit comments

Comments
 (0)