The end-to-end autonomous driving model research I did last semester didn’t end in particularly satisfying results — many factors involved. I’ll only dissect my own causes. One: last semester I was stuck in some uncanny state of ultra-high-intensity Coursework stacked on un-raisable efficiency, leaving me with no time at all — in December it genuinely felt like even stretching my day to 36 hours wouldn’t get the work done. In the end, the semester closed out hastily with a bad cold plus deferred exams. Two: broadly attributable to my experience still being too thin — I made plenty of detour-taking mistakes, though in a sense that also trained me up.
A Change of Direction
This semester I didn’t much feel like continuing to play in that area; I wanted something more undergrad-friendly and more fun. So I set my sights on VL, the L being for LLM. LLMs are not just a red-hot topic — humanity, at least as of 2025, still doesn’t clearly know what tasks an LLM can actually accomplish.
Abstract thought may not need natural language as its carrier, but when it comes to complex CoT I honestly can’t imagine how you’d manage without natural language as the medium. Worth noting, though: many complex real-world tasks may well require some form of abstract semantic understanding and reasoning to pull off. Whether an LLM can — and how it can — be worked into one step of an assistive pipeline is, in my view, a field that is extremely frontier and thoroughly black-box.
Exploring SLAM
HQ assigned me to fold VL into SLAM (Simultaneously Locating and Mapping). Okay — it carries a strong whiff of an industry-commissioned side project, but it does at least count as a meaningful direction. We discussed a lot, mostly circling around VL’s black-box nature. It’s too new: never mind a survey, there isn’t even one decent published work.
The SLAM pipeline, including front-end sensor signal processing and back-end pose-graph optimization Image source: MathWorks
The most production-grade VL-into-a-model work out there is AutoVLA from UCLA at NeurIPS2025. Setting aside that it’s action-oriented, they still haven’t released the source code even now — even though they said it would be out last December 💦. We can only grope our own way forward, and what we’re groping toward is even paradigm-level. For instance: is it G(SLAM(x) + VL(x)) or SLAM(x + VL(x))?
We brainstormed a lot of approaches in the group, some wildly fanciful or engineering-explode-y. And by the end of discussion, the method that could land fastest was even just ripping the model head off someone else’s VL-Navigation work and modding it into a SLAM task head.
Later I also thought of some optimizations around vision alignment points — basically a grab-bag of ways to exploit VL priors.
Going by the requirements from the industry side, the scenarios are extremely extreme, but that may involve some trade secrets — conflict of interest, I’ll leave it there.
But sometimes I sink into a bit of nihilism: why don’t we just design an elegant and efficient method to solve this / why don’t we use one ultra-small CNN to do the thing? I don’t know either — build first, see later. Can a CNN even understand human speech? (jk) A prompt engineer is still an engineer.
Plans Going Forward
Some good news: I’ll be staying in Shenzhen for the whole winter break. Also, my course load next semester looks set to halve, so I’ll have plenty of time to open up this line of work. This week I may explore a few more papers and reproduce some things — and maybe also slack off for a while, to patch up the deep hollow finals week gouged out of me.
I might reproduce the current mainstream SLAM models, and then run some experiments on a few of the simpler ideas? I’m not sure how far I’ll get before the New Year. But with CC I really am several times faster — I’ve even run the most minimal of experiments already. Though my brainpower burn rate has also climbed quite a bit; it’s like a booster shoving you hard from behind.
In any case, I find this research far more interesting — partly because it’s more abstract, the objectively-meaningful kind of abstract, and partly for that particular excitement and freedom of exploring the unknown.
Stay tuned!