An eccentric dreamer in search of truth and happiness for all.

Category: Projects

A Beautifully Foolish Endeavour

Years ago, I remember sitting in a professor’s office. There were stacks and stacks of textbooks, shelves of more textbooks lining the walls. I felt like he had more books than a library.

In those days, I was a master’s student in the course option, looking to try to switch to the thesis option so I could do real research into the thing I thought was really cool, which was neural networks. In those days, AI was still a niche field of science, and connectionism (later called deep learning), the subset that neural nets fell under, was full of eccentric personalities committed to the beautifully foolish endeavour of trying to take our limited understanding of the algorithm of the brain and turn it into something grand and wonderful.

I somehow, back in those days, convinced the professor to take me on as a student, even though neural nets were just the last line on his list of other, at the time, more respectable research interests.

AI back then was very different from what it is now. I feel a sense of incredible sadness at what things have become, possibly also some rage. What was a profoundly interesting scientific endeavour has turned into this giant buzzword and megalithic all-devouring capitalism machine.

I was there before all the hype. Trying to do cool things before it was cool. Back when it was science! And clever engineering, and a bunch of math I didn’t really understand at the time. I remember when the Machine Learning Reddit was a place for random enthusiasts to discuss silly side projects, when papers came out every few weeks, rather than several every hour like now.

AI used to be, used to mean, something else. At least, to me it did. Maybe you could argue the goal was always this. But I think, people like Turing, like Simon, like Minsky, they’d be appalled at what people call AI now.

I mostly didn’t stay in contact with my supervisor after I graduated. He is a kind man who gave me a lot of leeway to finish my thesis despite many delays. I still have the copy of the Machine Learning textbook he gave me as a gift when I successfully defended the thesis, to replace the one I’d borrowed from him and returned earlier.

I kinda miss the days when things were heady and full of promise and potential. The world seems like it’s gone insane. I miss when I was just part of a beautifully foolish endeavour (yes, I know that’s also the title of Hank Green’s apparently fabulous book). I just…

The Idea That Could Save The World

Author’s Note: This was written while still somewhat sick, so it may not actually make much sense.

Many years ago, I wrote a rather simple and silly post on Less Wrong about what I called The Alpha Omega Theorem. It was, back in those days, not well received by the skeptical crowd of Rationalists, who were overwhelmingly atheists and my description of the Alpha Omega had obvious theistic overtones.

Many years later, I wrote about the concept of Superrational Signalling, in a long-winded essay that almost no one bothered to read.

Both of these posts have to do with an underlying idea, that it would be rational for a powerful entity like an ASI to be benevolent, or at least benign towards lesser entities.

Given the lack of being taken seriously, I wanted to find some way to show that this wasn’t just a hair-brained thought, but could be backed up with logic or math. The natural path towards this end was to show it through a proof using game theory.

I’ve mentioned before about Axelrod’s discovery of Tit-For-Tat winning the Iterated Prisoner’s Dilemma, and arguably showing that cooperation beats aggression fundamentally. But many people seem to see the IPD result as simplified, and not relevant to the case of an ASI facing primitive humans.

So, I decided to try something. Take the Iterated Prisoner’s Dilemma, and iterate on the design. Add Death through having payoff matrices with negative values that could lead to zero or less points, which would then remove the agent from the game. Add Asymmetric Power by allowing these payoff matrices to depend on the relative point totals. Add Aggressor Reputation so that agents could “police” or act as “peacekeepers” similar to what Toby Ord explored long ago

And so, I came up with Peace Or War Each Round (POWER) with code and analysis and an actual runnable simulation.

Basically, what I thought would probably happen, did. The cooperative (nice) strategies would, gradually, in the very long run, beat out the aggressive (nasty) strategies through a kind of coordination at a distance. Essentially, alliances beat empires. Perhaps more importantly, stronger agents had a strong strategic incentive to cooperate with weaker agents instead of just eating them. This part is what makes it relevant to AI safety.

Though, this goes past aligning just ASI. In theory, if we ever encounter alien superintelligence, the game theoretic proof holds even for them. In effect, this idea could turn evil towards good. It could show that morality is rational to everyone. This could be the idea that saves the world, so to speak.

Given how poorly my past essays on this area of ideas has been received, I’ve been more cautious about voicing this result this time. I want to write up a more rigorous analysis before I post on venues like Less Wrong again.

I did throw it past some other people interested in Game Theory and AI safety, and they seemed to find the idea interesting and potentially a big deal, but they’re probably very biased because of their aligned interests. I know that there are arguments people could use to critique the idea, that it’s too simple and irrelevant to real world situations, etc.

So, if I want to make it a true “proof”, I probably have to take a lot more steps to firm up the result, to confirm it across more complex simulations and expose it to more serious challenges. I’m not sure if I’m ready to push into that space.

Right now, I have what I think is a cool idea. I don’t know if it’ll actually save the world, but it’s nice to imagine.

In truth though, there’s a good chance the idea will be ignored. I could publish a paper, a Less Wrong post, and such, and it’ll probably just be an obscure thing on the Interwebs. I could try to write a series of novels that spread the idea, but that’s likely a moonshot.

This idea, what is it actually worth? I don’t know. I’m probably super biased by motivated reasoning. It’s something I want to believe. That in itself should make me more critical.

But then, I feel like the idea is written in the stars. It seems so obvious with reflection.

Anyways, I just wanted to mention this was a thing I’ve been working on. We’ll see if it ever goes anywhere…

The State Of Joseph 2026 – Random Mundane Stuff

It’s the new year, and I’m sick. My parents-in-law have been visiting and either they or the toddler coming home from daycare brought a pretty bad strain of something. First the father-in-law had a cough and a fever. Then the toddler got a fever and was lethargic for days.

Then my wife got sick with the fever. Then, I got the fever and also the worst sore throat I’ve ever experienced. I couldn’t fall asleep because swallowing saliva felt like knives.

Thankfully that seems past now, although I still have an annoying cough.

A thing I’ve been experimenting with recently is to ask three different chatbots the same questions, treating them almost like a council of advisors. Right now the “council” consists of ChatGPT, Gemini, and Claude, all free tier. My wife likes DeepSeek, but three is already a lot to copy and paste to for every question.

Anyway, my experiences so far are that they are generally yes men who generally agree with and come up with justifications for just about any idea I have, with some obvious exceptions when the idea is just blatantly foolish.

Interestingly, while ChatGPT and Gemini tend to offer very similar advice, Claude sometimes breaks ranks and offers more skeptical, critical advice. That I actually found pretty interesting.

Also, Gemini sometimes comes up with unique ideas or solutions that the other two don’t mention, such as using the Sunshine and Moonlight streaming software to solve the issue of wanting to play PC games on the 4k projector in the basement by streaming them from my office machine.

I had originally been trying to figure out how to fix an old PC I have with an ancient i7-3770 CPU and a 2080 Ti GPU, and whether it made more sense to buy a new PC and put the 2080 Ti in it.

The chatbots, particularly Claude, were actually very useful for helping me troubleshoot these computer problems, letting me fix an Nvidia driver issue in Ubuntu that was preventing me from starting up the computer.

My main reasons for tinkering with PCs again was that I had gotten my wife Clair Obscur: Expedition 33 for Christmas, and was thinking she’d prefer playing it on the couch with the 4k projector rather than in my office on my main computer.

I also had aspirations to introduce her to D&D by way of Baldur’s Gate 3 split screen co-op, which also would be nice on the projector.

Also, if I could salvage the 2080 Ti, I’d be able to use it for ML experiments as well.

I spent a lot of time looking at store stock on the local computer store website and trying to discuss with my council what options made the most sense. Alas, I picked an awful time to look at PCs as RAM prices have skyrocketed due to AI demand.

I was too sick to go to the store, so I watched as the ones I was considering got sold out. In the end I decided this wasn’t worth throwing like $1000-$2000 CAD at given my use cases were so limited, and my wife seemed okay with just playing in my office.

Nevertheless, I wasted a ton of time researching options and being told the relative merits of each by my ever agreeable chatbot council.

The old PC still has some issues where I have to literally open up the case and unplug and plug back in some power connectors every time I wanted to boot it up. The power supply is over a decade old, and I’m not sure if the GPU itself is actually not damaged too. It might be a lost cause at this point.

My wife actually got me the Magic: The Gathering Bloomburrow Commander decks for Christmas. We’d played with the Bloomburrow starter preconstructed decks earlier when I first introduced her to the game. She liked how cute the Bloomburrow setting is. Imagine rabbit knights and otter wizards doing battle. It’s probably the cutest Magic set since maybe Lorwyn, possibly cuter.

Alas, it’s really hard to find time to play with my wife. Her job is plenty stressful and she works A LOT. More reason why the gaming on the projector idea needs to be as convenient as possible.

We used to play Stardew Valley on the projector via the Switch. But that was 1080p. Apparently, 4k is much nicer even with a game like that. Though, I can just hook up my laptop to the projector if I want something like that.

And streaming from my office computer to the projector through my laptop for a game like Expedition 33 does actually work and looks and sounds glorious. It’s just a hassle to set up and potentially laggy.

And, again, my wife barely has time to play anyway…

I’m really not sure where I’m going with this ramble. I’m still sick, so I’m typing this on my phone in bed instead of at my computer in my office and man, this is a slow way to write.

It seems like I’m very easily distracted with random rabbit holes or nerdsniped by whatever thing happens to capture my attention. Obsession and flow state. Strange.

Anyways, all this is to say not much is actually happening in my life right now.

I finally got winter tires after much prodding from my wife. I had driven over two decades on all-seasons, but now that I have a kid that I have to take to daycare through snowstorms, I shouldn’t risk it anymore. So far the difference has been noticeable.

I regret procrastinating on getting the flu shot. I haven’t been this sick in a long time…

At least we can look forward to the next season of Frieren: Beyond Journey’s End this year. My wife suggested we watch the first season when it was still airing. As a former otaku, it impressed a lot. A strong recommend if you are at all into anime and/or high fantasy.

In other news, the toddler is adorable and learning words fast. He also is very silly. He invented the “no no song”, which is basically going “no no no no” to the tune of Twinkle, Twinkle, Little Star.

He also likes to go turn light switches and fans on and off over and over, as well as go up and down the stairs, again, over and over, either climbing or by getting me or whoever to carry him. We have a basement and three floors, so this gets arduous.

He also likes me to read certain books over and over. We joke the characters are trapped in a time loop. Graham turns two… again!

He also sometimes says his own name a lot, which kinda reminded my wife of a Pokemon.

I’m still on the bed right now, listening to the joyful shouts of the toddler and the laughter of the grandparents from the third floor.

Anyways, time to put the toddler to bed. Life goes on.

A Question Of Career

Lately, I’ve been trying to figure out what to do next.

In July, I ended my contract with Twin Earth. It was mostly a formality, as I hadn’t done any billable work in over a year, having spent most of my time taking care of my new child while my wife worked full-time.

That child is finally in daycare, so, in theory, I have time again to pursue my career. But the reality is that the first year of daycare often comes with repeated sickness and instability. So, I’m not sure how aggressively I should pursue work again.

Given that, I also am not sure what direction to take now. My ideal career at this point would probably be either AI Safety research engineering work, or something creative, like indie game design or writing novels.

AI Safety is a small field. Compared to the AI industry proper, there’s something like just less than a thousand people working on AI Safety. And the funding is relatively sparse compared to industry. It exists, but it mostly goes to very smart people who are willing to move to the Bay Area or London. I’m not sure I’m even competent enough to work in the field.

Creative work under my own brand could be fun, but success would be kind of like winning the lottery. Most likely I’d end up spending a lot of time and not making any money. The games or novels I might make -could- maybe be useful for spreading important ideas into our culture. There is the slight chance I could do something meaningful. Even just writing a novel for my child to one day read, like what Tolkien did with The Hobbit, could be worthwhile in a small way.

But there’s opportunity cost. AI Safety -could- be more important to move the dial on. And there’s also the idea of working in the AI industry, or game development, or even just some generic programming job. Any of these would actually pay, and in the case of AI, probably much better.

I do have a family now to feed and ensure they have a good life. A normal person would take that as a good reason to go back into AI, or find some mundane programming job that can pay the bills and make things work.

Of course, as a former dreamer, I want to do something grand and meaningful and big. So, AI Safety and indie creative work have an appeal to me. The latter is probably not prudent, but would be a way to keep my hands clean of the AI mess that is now being created. The former is a way to fix the mess, or at least prevent its worse excesses, but risks encouraging the whole industry, safety-washing as it’s called.

My education, my credentials, the greater part of my work experience, tends to lean towards the technical, the AI work, and to a lesser extent game dev stuff. Writing is something I know I have some talent for, but I’m uncertain if that talent is actually enough to be exceptional, to actually be that much better that my writing would be worth reading over the other options.

There is a massive pile of literature in the world already. Most people will never live long enough to read all the classics, much less all the books they’d be personally interested in. Why add to the pile?

On the other hand, do I think I can actually make a difference on AI Safety? Much smarter people are struggling to figure out how to attack the problem. It may not even be a solvable thing. It could be intractable.

There’s an argument among Effective Altruism circles that the best thing most people can do is to Earn-To-Give. To find the highest paying job they can find, and donate as much as they comfortably can to the most effective charities. On paper, the numbers work out that this is the best thing you can do with your time and energy, unless you are an exceptionally good fit for direct work on the causes that matter.

That would suggest I should go back into AI, and just donate what I can.

But AI is increasingly a field that contributes to a lot of cultural pollution, technological misuse, unethical profit-seeking, etc. To what extent would I be condoning such things by choosing to participate in it?

Realistically, this is a problem for the long term. Right now, with my child in daycare, and potentially seeing interruptions in that, it may make sense to be patient and do something less demanding for now. Perhaps, in the interim, it does make sense to explore creative work on the side, to test whether or not I can write well enough to justify a project of some kind.

I tried testing my game design ability earlier when there was some time when my wife’s parents were here. I was able to finally finish programming the game Star Lance, and create another game called Cities and Tactics. That showed I have some aptitude, though nothing particularly special, I think. I should try writing some short stories, and see if they’re any good…

There are lots of things I should do. I still suck at doing things. I think. I ponder. I have intellectual wanderings and musings. But actually doing things? I hesitate. I question. I doubt. I’m not good at being productive. It makes me think I may not be cut out for any of these things I’ve been considering.

In another life, I would have been something like a political philosopher. That probably could have been my calling if I’d been more foolish. Though realistically, I’d probably have ended up starving instead.

My wife thinks maybe I should go to teacher’s college and become a teacher. She thinks I have the empathy for it. I’m not sure about that. If I want to teach, I’m probably more inclined to write a book about the thing I want people to learn. Dealing with actual people is not my forte. At the same time, teaching seems like a very noble profession.

Maybe I should consider what will likely disappear due to AI first? My two skills in life, programming and writing, both seem to be things that LLMs are uniquely suited for. An unfortunate coincidence, that.

I don’t really know what to do with this. My career was going… somewhere? But now it’s on pause and I’m not sure I can go back. I feel kinda useless in that regard, washed up.

The pivot into game development might have been a mistake, but then, I didn’t exactly have any other opportunities at that moment. I took what was there. I’ve never been in a position to choose between multiple job offers at the same time. I’ve never been that successful, that privileged.

I find I end up just going with the flow. My life is mostly stuff that happened, and I did what seemed to make sense at the time. I never really planned far ahead, aside from maybe choosing to study AI before it was cool. That was a lucky choice, it seems. Though I don’t know the counterfactual, so maybe it wasn’t, who knows?

Anyways, the reality is I’ll probably end up choosing what ever opportunity first presents itself. Right now there’s not much going on. I should maybe be more strategic, but I suck at that too.

Where am I going with this? There’s possibilities, but they are very uncertain. There’s things I can do, but I don’t know if I should. There’s paths to take, but I hesitate. I want a sign. I want some clear instructions from God that this is what I should do with my life. But it seems like I’m not getting a sign, or at least, I’m too dumb to recognize it.

These days my mindset is darker than usual, more melancholic. I’m tired.

I wish I could end on a high note. I want to believe in something. I admit that life isn’t that bad. I should show more gratitude. I know the CBT, I know in theory that things are okay. I just feel a certain way. I guess it can’t be helped.

In case someone is actually reading this, I have these ups and downs. Usually, in the past, I posted during the ups. Recently, I’ve tried posting during the downs, to perhaps balance things, to show a more real presentation of myself, rather than the idealized image that you often see on social media. I’m not sure if I went too far, have said too much that is much too personal and bad for my prospects.

I hope that being so impulsively honest helps people to understand me better. But I should probably cut back on this kind of thing. It does nothing for my cause with recruiters, I think.

Words can come back to curse us. Or they can teach and help us to connect. It’s a matter of wisdom I suppose, which one happens in the end. I want to communicate, because I seem to care what other people think, because what they think leads to how they feel, and what they feel matters.

So, regardless, maybe I should write. There are stories in my head I want to write. I really should stop hesitating, fearing that they will prove my incompetence. The truth is likely that I’ll be decent but not exceptional, like with most of my hobbies.

I have to trust that something will open up. That there is a place for me in this world. Somewhere in the future, things will work out, somehow. I just have to patient and kind and myself.

The Story of Music-RNN

There was once a time when I actually did interesting things with neural networks. Arguably my one claim to having a footnote in AI and machine learning history was something called Music-RNN.

Back around 2015, Andrej Karpathy released one of the first open source libraries for building (then small) language models. It was called Char-RNN, and it was unreasonably effective.

I had, back in 2014, just completed a master’s thesis and published a couple papers in lower tier conferences on stuff like neural networks and occluded object recognition, and figuring out the optimal size of feature maps in an convolutional neural network. I’d been interested in neural nets since undergrad, and when Char-RNN came out, I had an idea.

As someone who likes to compose and play music as a hobby, I decided to try modifying the library to process raw audio data and train it on some songs by the Japanese pop-rock band Supercell and see what would happen. The result, as you can tell, was a weird, vaguely music-like gibberish of distilled Supercell. You can see a whole playlist of subsequent experimental clips on YouTube where I tried various datasets (including my own piano compositions and a friend’s voice) and techniques.

Note that this was over a year before Google released WaveNet, which was the first of the real generally useful raw audio based neural net models for things like speech generation.

I posted my experiments on the Machine Learning Reddit and got into some conversations there with someone who was then part of MILA. They would, about a year later, release the much more effective and useful Sample-RNN model. Did my work inspire them? I don’t know, but I could hope that it perhaps made them aware that something was possible.

Music-RNN was originally made with the Lua-based version of Torch. Later, I would switch to using Keras with Theano and then Tensorflow, but I found I couldn’t quite reproduce as good results as I had with Torch, possibly because the LSTM implementations in those libraries was different, and not automatically stateful.

I also moved on from just audio modelling, to attempting audio style transfer. My goal was to try to get, for instance, a clip of Frank Sinatra’s voice singing Taylor Swift’s Love Story, or Taylor Swift singing Fly Me To The Moon. I never quite got it to work, and eventually, others developed better things.

These days there’s online services that can generate decent quality music using only text prompts, so I consider Music-RNN to be obsolete as a project. I also recognize the ethical concerns with training on other people’s music, and potentially competing with them. My original project was ostensibly for research and exploring what was possible.

Though, back in the day, it helped me land my first job in the AI industry with Maluuba, as a nice portfolio project along with the earthquake predictor neural network project. My posts on the Machine Learning Reddit also attracted the attention of a recruiter at Huawei, and got me set towards that job.

Somewhat regrettably, I didn’t open source Music-RNN when it would have still mattered. I was convinced by my dad back then to keep it a trade secret in case it proved to be a useful starting point for some kind of business, and I was also a bit concerned that it could potentially be used for voice cloning, which had ethical implications. My codebase was also, kind of a mess that I didn’t want to show anyone.

Anyways, that’s my story of a thing I did as a machine learning enthusiast and tinkerer back before the AI hype train was in full swing. It’s a minor footnote, but I guess I’m somewhat proud of it. I perhaps did something cool before people realized it was possible.

Letting Go

I’ve officially discontinued the Earthquake Predictor project for now.  I still need to do backups of the data, but I’m no longer going to have the daily update script running.  It had stopped for a while now due to a bug anyway, and I wasn’t able after a cursory effort to debug it so that it worked again.

It also honestly didn’t do anything relevant, as the model only predicted high frequency, low magnitude quakes and not the low frequency, high magnitude ones that mattered.  The architecture I was using was my old LSTM-RNN model, which has largely been superseded in the literature by the transformer architecture, so it badly needed to be taken offline and retrained anyway.  Not sure if I’ll ever get around to retraining it with the new architecture.

It kinda seems like a very silly longshot of a project that kept one entire GPU busy because I could never figure out how to properly unload models from memory without stopping the process.  Even if it only updated once a day, it was a hassle because the thing would randomly error out and I’d have to start it up again after having missed several days of updates and having to backdate the sequence of updates made with the appropriate 24 hours of data.  All in all, not a great system.

The Great Debates

I’m currently still trying to decide what I should even post here. I tend to post more personal stuff on Facebook and to a lesser extent on Twitter, but my fiancee thinks it might be unwise to publish personal details on a public facing blog like this one.

Possibly I could focus more on professionally relevant ideas, but I’m not sure what I can offer in that regard. Anything really worth publishing should probably go into a proper paper rather than some random blog on the Internets. I suppose I could write opinions about philosophical things, but that overlaps with the Pax Scientia wiki that I was working on building earlier.

I probably have too many of these projects that don’t get enough attention anyway. I’ve been trying to consolidate them recently, but I worry that the resulting web presence is still far too sprawling and even less clear to navigate without the delineations.

Another debate I’ve been having recently is whether to put more effort into my creative writing. I want to eventually write a novel. It’s a vague goal I’ve had since I was a kid. I have lots of ideas for stories, but I’ve always had trouble actually getting down to writing the ideas down into actual narratives. Sometimes I wonder if I actually have the writing ability to justify the effort, whether it makes sense to add yet another piece of literature to the ever expanding pile of books in the world.

I spent a long time working out in my head the worlds that I want to write about. In some sense, if I don’t write, it’ll have been a waste. But I’m not sure my imagination is that much more extraordinary enough to justify the effort in the first place.

I also claim to be a music composer and a game designer, the other two arts that I have some capacity in. To what extent would those be more appropriate uses of my time? To what extent is writing more worthwhile than composing songs for instance? I can hash out a song somewhat faster than a novel, but I also as yet don’t consider my songs to be particularly notable either.

My thoughts on why writing was my first choice in terms of artistic expression were originally and ostensibly because writing allows me to communicate ideas rather than just emotions like with music. And writing can be done on my own, rather than needing an artist and a team for game development. Admittedly, the creator of Stardew Valley did it on his own, but I don’t have the visual art skills for that, and I don’t see myself having the patience to become good at drawing at this point.

In another debate, I’ve also been considering a change of career path. Working in machine learning has been exciting and lucrative, but the market now seems increasingly saturated as the most competent folks in the world recognized the hype and adjusted their trajectories to compete with mine. Whereas a few years ago I was one of maybe a couple hundred, now there seem like thousands of people with PhDs who outclass me.

At the same time, I’ve wondered about whether or not the A.I. Alignment problem, the existential risk of which has been the focus of several books by prominent philosophers and computer scientists, isn’t a more important problem that needs more people working on it. So I’ve wondered if I should try switching into this field.

Admittedly, this field seems to be still in its infancy. There’s a bunch of papers looking at defining terms and building theoretical frameworks, and little in the way of even basic toy problems that can be coded and tested. I’m personally more of an experimentalist than a theoretician when it comes to AI and ML, mostly because my mathematical acumen is somewhat lackluster, so I’m not sure how much I can help push forward the field anyway.

On a more philosophical note, it seems the social media filter bubble has been pushing me more to the left politically. At least, I find myself debating online with Marxists about things and becoming more sympathetic to socialism, even though a couple years ago I was a moderate liberal. I’m not sure how much to blame the polarization of social media, and how much it’s the reality of disillusionment with the existing world.

I also have mixed feelings in part because the last company I worked for was, according to media outlets, controversial, but to me it was the company that gave me a chance to work on some really cool things and paid me handsomely for my time and energy. Admittedly, as a lowly scientist working in an R&D lab, I wouldn’t have been privy to anything untoward that could have been happening, but it was always jarring to see the news articles that attacked the company.

I left more for personal reasons, partly some issues of office politics that I wasn’t particularly good at dealing with. My own criticisms of the company culture would be much more nuanced, aware that any major corporation has its internal issues, and that many of them are general concerns of large tech companies.

The debates in my head are somewhat bothersome to be honest. But at the same time, it means I’m thinking about things, and open to updating my understanding of the truth according to new evidence, factored with my prior knowledge.

The March of Progress

Where to begin. I guess I should start with an update about some of the projects I’ve been working on recently… First, the Earthquake Predictor results that can be found at cognoscitive.com/earthquakepredictor are just over a year out of date. I still need to update the dataset to include the past year’s earthquakes from the USGS, but I’ve been busy first using the existing data as a benchmark to test some changes to the loss function and architecture that I want to utilize. I’m still debating whether to continue using an asymmetrical loss like Exp Or Log, or Smooth L1 Or L2, or to switch to the symmetric Smooth L1, which would reduce false positives substantially. My original reason for an asymmetric loss was to encourage the model to make higher magnitude predictions, but I worry that it makes the model too eager to guess everywhere that earthquakes are frequent, rather than being more discriminating.

Music-RNN has run into a weird problem where I’m having difficulty reproducing the results that I got with the old Torch library a few years ago with the Keras port. It’s probably because the Keras version isn’t stateful, but it could also be that some of the changes I made to improve the model have backfired for this task, so I need to do some ablation studies to check. My modification for Vocal Style Transfer is on hold until then.

In other news, a couple of neat projects I’ve been trying include: Lotto-RNN, and Stock-RNN.

Lotto-RNN is a silly attempt to predict Lotto Max numbers on the theory that some of them, like the Maxmillions draws, are pseudorandom because they are done by computer rather than ball machine, and thus might be predictable… Alas, so far no luck. Or rather, the results so far are close to chance. I’m probably not going to spend more time on this long shot…

Stock-RNN is a slightly more serious attempt to predict future daily price deltas of the S&P500 given previous daily price deltas. It uses the same online stateful architecture that seemed to work best for the Earthquake Predictor before. The average result of ten different initializations is about +9% annual yield, which falls below the +10.6% that you’d get from just buying and holding the index over the same period. Technically, the best individual model result achieved +14.9%, but I don’t know if that’s a fluke and won’t regress to the mean.

I also tried a stateless model for Stock-RNN, but it performed much worse. There are some things I can do to adjust this project. For instance, I could modify the task to try to predict the annual price delta instead, and train it on many stocks instead of just the S&P500 index, and use it to pick stocks for a year rather than just guess where to buy or sell daily. Alternatively, I could try to find a news API for headlines and use word vectors to convert them into features for the model.

On the research front, I was also able to confirm that the output activation function I originally named Topcat, does seem to work, and doesn’t require the loss function modifications that I’d previously thought were necessary, but works if you use it with binary crossentropy in place of softmax and categorical crossentropy. I still need to confirm the results on more tasks though before I can seriously consider publishing the result somewhere. There’s actually a few variants, mainly two different formulas and various modifications that seem to be functional.

It also looks like a hidden activation function I was working on that I named Iris also seems to work better than tanh. (Edit: More testing is required before I can be confident enough to say that.) Like with Topcat, I have several variants of this as well that I need to decide between.

Another thing that seems to help is scaling the norm of the gradients of an RNN, rather than just clipping the norm as is standard. Previously, I’d thought that setting the scaling coefficient to the Golden Ratio worked best, but my more recent tests suggest that 1.0 works better. Again, it’s something I need to double check on more tasks.

Some things that turned out to not work reliably better than the control include: LSTM-LITE, my tied weights variant of the LSTM, my naively and incorrectly implemented version of Temporal Attention for sequence-to-sequence models, and a lot of the places where I used the Golden Ratio to scale things. The formula for Iris does have a relation to Metallic Ratios, but it’s not as simple as scaling tanh by the Golden Ratio, which weirdly works on some small nets, but doesn’t scale well. Interestingly, the Golden Ratio is very close to the value suggested to scale tanh in this thread on Reddit about SELU. So, it’s possible that that would be theoretical justification for it. Otherwise, I was at a loss as to why that seemed to work sometimes.

I’m also preparing to finally upgrade my training pipeline. In the past I’ve used Keras 2.0.8 with the Theano 1.0.4 backend in Python 2.7. This was originally what I learned to use when I was at Maluuba, and conveniently was still useful at Huawei, for reasons related to the Tensorflow environment of the NPU. But, it’s way out of date now, so I’m looking at Tensorflow 2.1 and PyTorch 1.4. An important requirement is that the environment needs to be deterministic, and Tensorflow 2.1 introduced better determinism, while it’s been in PyTorch for several versions now.

I’ve used both Tensorflow and PyTorch in the past at work, though most of my custom layers, activations, and optimizers are written in Keras. Tensorflow 2.0+ incorporates Keras, so in theory, it should be easier to switch to that without having to rewrite all the customizations, but just adjust the import statements.

I’ve also switched to Python 3, as Python 2 has apparently reached end-of-life. Mostly, this is requires some small changes to my code, like replacing xrange with range, and possibly paying attention to / versus // in terms of division of integers.

One thing I’ve realized is that my research methodology in the past was probably not rigorous enough. It’s regrettable, but the reality is that I wasted a lot of experiments and explorations by not setting the random seeds and ensuring determinism before.

Regardless, I’m glad that at least some of my earlier results have been confirmed, although there are some mysterious issues still. For instance, the faulty Temporal Attention layer shouldn’t work, but in some cases it still improves performance over the baseline, so I need to figure out what exactly it’s actually doing.

In any case, that’s mostly what I’ve been up to lately on the research projects front…

Powered by WordPress & Theme by Anders Norén