model-to-file-to-reconstituted-model-serailization

[david-thrower]

Searlization of the model to file and re-loading the model post - the keras changes that broke this (handling of custom objects such as iRoPE embedding).

[david-thrower]

It just may work to append https://github.com/david-thrower/cerebros-core-algorithm-alpha/blob/main/phishing_email_detection_gpt2.py with:

# Get best model
best_model_found = cerebros_automl.get_best_model()

# Define custom objects
CUSTOM_OBJECTS = {
    'NewTokenizerLayer': NewTokenizerLayer,
    'RotaryEmbedding': RotaryEmbedding,
    'InterleavedRoPE': InterleavedRoPE,
    'GPT2Layer': GPT2Layer,
} # One concern I have is whether some of the custom gating functions also need included ....

model_path = "best_model.keras"

# Save entire model
print("Saving model...")
best_model_found.save(model_path, save_format='keras')

print("Loading model...")
reconstituted_model = tf.keras.models.load_model(model_path, custom_objects=CUSTOM_OBJECTS)

[david-thrower]

Holds before we merge:

Hold 1: Question (de-duplication in a node's upstream connectivity / incoming connections)

@Thunderblok

cerebros/keras_export.py lines 74 through 76

        # Deduplicate while preserving order
        seen = set()
        inputs = [x for x in inputs if not (x in seen or seen.add(x))]

If I am following this right, if I had the graph:

inp = Input()
dense_level1_unit1 = Dense(2, 'gelu')(inp)
dense_level1_unit2 = Dense(5, 'gelu' )(inp,inp,inp)(inp)

### Pay attention to this node <<<<-------------------------------------<<<<<<<<<<<<
cat_level_3_unit_1 = Concatenate(axis=1)(
    [ inp,
      inp, 
      dense_level1_unit1,
      dense_level1_unit1, 
      dense_level1_unit2, 
      dense_level1_unit2, 
])
dense_level_3_unit_1 =  Dense(5, 'gelu' )(cat_level_2_unit_1)

And this were capturing the upstream connectivity of cat_level_2_unit_1

...

The node cat_level_3_unit_1 has intentional duplication, the concatenation of [inp, inp, dense_level1_unit1, dense_level1_unit1, dense_level1_unit2, dense_level1_unit2 ]
...

Would this be searilizing this node as [inp, dense_level1_unit1, dense_level1_unit2 ]? If so, this is not correct, as the Cerebros graph has a hyperdesnse-sparse lattice of connectivity in all layers and should not be de-duplicated. If I am mis-reding this, hopefully you or the AI can elaborate where I am missing this.

Hold 2: Test cerebros/keras_export.py using a complex model

(e.g. a Cerebros model, possibly completed, still reviewing the code changes ..)

Hold 3: cerebros/rope/irope_torch.py appears it may be incomplete.

• This appears to be a RoPE embedding
• The continuation to make an interleaed RoPE or iRoPE needs to be added.

cici-requirements.txt: Need to upgrade scikit-learn, but need to resolve the issue of performance degradation seen on #164 with the version used there ...

... (under construction)

Hold 4 cerebros/simplecerebrosrandomsearch/simple_cerebros_random_search.py

Mixed use of self.seed += 1 (Programmaticlly set by SimpleRandomSearch(... seed=[1234]))

set_global_seed()
self.seed += 1

My concerns on this: The imported utility: cerebros/utils/random.py

• uses os.envoron["foo"] = "bar" to set environment variables.

    os.environ["PYTHONHASHSEED"] = str(seed)
    ... 
             os.environ["TF_DETERMINISTIC_OPS"] = "1"
            os.environ["TF_CUDNN_DETERMINISTIC"] = "1"

There are 4 potential concerns with that said:

• In managed containers (e.g. in Kubeflow, other Kubernetes environemnts), the environment variables may be managed by the system's orchestrator or [once the orchestrator has started the container] may be immutable for security reasons.
• This code is nested in try .. PASS blocks, meaning that if it was unsuccessful, it would be unsuccessful silently.
• To create a fall - back that preempts the seed setting in the Python API would basically be where we might as well just make that the primary seed control for the pytorch, numpy, jax, and tf frameworks.
• When we run this in distributed hyperparameter tuning, this adds additional nuances to controlling the setting of the seed.

Hold 5 cerebros/torch_builder.py >> _map_activation(name: str) -> nn.Module: (Question)

• Add softmax
• Add softplus

Cleared for merge:

• cerebros/simplecerebrosrandomsearch/simple_cerebros_random_search.py ✅ (other than aforementioned dependencies)

Hold 6 Resolved and passing tests ✅:

Hold 6: Change in subpoicess deployment confirmed bug: 🐞👾; Already fixed on 7ff75fe, pending CICD tests

p=Process()
processes.append(p)

VS

processes.append(Process(...))

... and the difference in how args are being passed to subprocesses

... I think this is an issue I discovered the hard way before (like in 2022). Most docs say to do this the first way, but in our case it just doesn't work...

Why I'm concerned:

• Commit 7cdb41f
• In the job https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/17118391352/job/48553751107
• The test in automerge.yaml that runs regression-example-ames-no-preproc.py

The job runs as normal and prints out:

Run python3 regression-example-ames-no-preproc.py
2025-08-21 05:55:52.204819: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1755755752.219619    2316 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1755755752.224700    2316 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1755755752.237074    2316 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755755752.237092    2316 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755755752.237094    2316 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1755755752.237095    2316 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-08-21 05:55:56.427789: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
Global task progress:   0%|          | 0/7 [00:00<?, ?it/s]/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = os.fork()

... But the blogs stop printing further logs.

I look at an older test on the same file:

https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/17116989447/job/48549947090
from a commit 1 hour 50 min ago (at the time of this writing), and it has not exited. These normally take 2 - 5 min to exit. Confirmed bug; To Do: Revert changes to cerebros/simplecerebrosrandomsearch/simple_cerebros_random_search.py line 600:

              # Old / correct
                processes.append(
                    Process(target=self.run_moity_permutations(spec=spec, subtrial_number=subtrial_number, lock=lock)))
               # AI Generated comments, I think: (# Fix Process target: do not call function immediately; pass args)
               # New version, needs revered back to old:
                p = Process(target=self.run_moity_permutations,
                            args=(spec, int(subtrial_number), lock))
                processes.append(p)

[david-thrower]

@sashakolpakov FYI on this conversation.