AbstractPhila PRO

Fable is interested in distilling bert's behavior at the core of an AlephLM experiment series to test the effectiveness of distillation into the AlephLM structure, and it sounds like a good idea coming off the direct experiment of it.

We will be testing arm coupling, decoupling, and testing the entire AMOE behavior as an internalized structural invariant built directly into the core of the AlephLM system. Essentially, WHEN this is successful, we will be controlling our MOE system from stem to stern.

So here we are. AlephLM is being controlled by fable on a runpod for the next few days. Updates arrive as they arrive. The big book of laws that Opus, Fable, and myself have cobbled will expand. We'll eliminate some, validate some, and learn new as we go like usual.

Every major system is heavily analyzed for structural and behavioral responses as we go like usual. We have a large set of geometry measuring mechanisms to test failures if they occur for why they fail as well unless we encounter an anomaly. So far the majority of anomalies have either been catalogued and turned into structure, eliminated from the potential of running into, sidestepped using deviations or forms of loss, or repaired architecturally and the laws updated to reflect the anomaly.

AlephLM will be an LLM before you know it. Stay tuned.

Massive success with both CaptionBert-v2 variants. They are both standalone and independent students with a similar residual effect.

https://huggingface.co/AbstractPhil/captionbert-8192-v2
https://huggingface.co/AbstractPhil/captionbert-8192-v2-b

This marks the end of the Bert teacher/student preprocessing for captionbert until a large discovery or innovation is discovered based on the AMOE research. In that case I may need the features in the future, but for now they are only necessary for analysis and potential ablation.

There are a few limitations.

CaptionBert-V2-B doesn't understand how to use V2-A's AMOE arms as effectively as A, and vise-versa. However, they ARE compliant with their twin's arms. All of which were trained independent of each other.

The core difference between the two was a logistics problem. Not all the ModernBert features landed. Some silently OOM'd and the audit slipped through the cracks. Once I approached it, I trained a model with the missing shards omitting the other teachers as well - so about 8m missing features or so.

B was issued the full corpus, which showed the full corpus didn't enhance the outcome for the base student. The structure was done with the data already and the cosine similarity mildly improved, pre-AMOE introduction.

The REAL strength came from attaching the AMOE systems to the B. They defeated A's benchmarks and pushed closer to MiniLM. A couple of out of scope too-small datasets were used, dropped a little between A and B, so there's legitimate differences for out of scope usage that have been unaddressed.

I'm not using MiniLM as a teacher, primarily because I want to have a legitimate industry-used marker as something to target.