They demonstrate the model on a "minecraft-like" domain (introduced earlier by someone else) where there are resources in the world and the agent has tasks.

The MDP does not change, it's dynamics are the same, with or without the RM, as they are with or without a standard reward model. Additionally, the rewards from the RM can be non-Markovian with respect to the MDP because they inherently have a kind of memory or where you've been, limited to the agents "movement" (almost "in it's mind") about where it is along the goals for this task.

So by specifying a reward machine you are augmenting the state space of the MDP with higher level goals/subgoals/concepts that provide structure about what is good and what isn't.

Fascinating idea, why not? Why are we hiding the reward from the agent really?

Apply a set of logical rules to the state space to obtain a finite set of states.

reward is a constant number assigned to each set of states

[Icarte, JAIR, 2022] "Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning"

[Icarte, PMLR, 2018] "Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning"

strong statement, what do people think about this? is it accepted by anyone or dismissed?

"Finding Optimal Solutions to Rubik's Cub e Using Pattern Databases" by Richard E. Korf, AAAI 1997.

The famous "Korf Algorithm" for finding the optimal solution to any Rubik's Cube state.

but not "never"

really? so that's how they make money.

Question: what kind of bias does this introduce into the model?

• which topics and questions grt trained on?
• what is the goal of training? truth? clickability?

Blog post from OpenAI in Jan 2022 explaining some of the approaches they use to train, reduce and tube their LLM for particular tasks. This was all precursor to the ChatGPT system we now see.

"Talking About Large Language Models" by Murray Shanahan

[Nam, NeurIPS, 2022]. "Reinforcement Learning with State ObservationCosts in Action-Contingent Noiselessly Observable Markov Decision Processes"

Lee et. al. - NeurIPS 2022 "Multi-Game Decision Transformers"

[Neumann, Gros, NeurIPS, 2022] - "SCALING LAWS FOR A MULTI-AGENT REINFORCEMENT LEARNING MODEL"

Kind of ambiguous to use 10K when one of the most important variables is K.

one timestep corresponds to three tokens

Using the attention mechanism to determine global dependencies between input and output instead of using recurrent links to past states. This is the essence of their new idea.

"Burn Severity in Canada's Mountain National Parks: Patterns, Drivers, and Predictions" Weiwei Wang, Xianli Wang, et al Geophysical Research Letters

"On the Opportunities and Risks of Foundation Models" This is a large report by the Center for Research on Foundation Models at Stanford. They are creating and promoting the use of these models and trying to coin this name for them. They are also simply called large pre-trained models. So take it with a grain of salt, but also it has a lot of information about what they are, why they work so well in some domains and how they are changing the nature of ML research and application.

AAAI 2022 Paper : Decentralized Mean Field Games Happy to discuss online.

S. Ganapathi Subramanian, M. Taylor, M. Crowley, and P. Poupart., “Decentralized mean field games,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-2022), vol. 36, pp. 9439–9447, February 2022. 1.

A recent overview of RL methods used for autonomous driving.

They didn't try to train the CNN from scratch.

Discussion on

Bellinger C, Drozdyuk A, Crowley M, Tamblyn I. Balancing Information with Observation Costs in Deep Reinforcement Learning. Proceedings of the Canadian Conference on Artificial Intelligence [Internet]. 2022 May 27; Available from: https://caiac.pubpub.org/pub/0jmy7gpd

Interesting sounding high level paper about the limits and constraints on general intelligence and how this might relate to the struggles AI/ML research has had historically.

Another piece to the "what can we do with eligibility traces" puzzle for Deep RL.

Question: What happened to Eligibility Traces in the Deep RL era? This paper highlights some of the reasons they are not used widely and proposes a way they could still be effective.

Question: What happened to Eligibility Traces in the Deep RL era? This paper highlights some of the reasons they are not used widely and proposes a way they could still be effective.

Hypothesis page to discuss this high level description of DeepMind's new Gato framework.

The paper that introduced the MineRL challenge dataset.

Their target images are very large and there is a known (supervised) label for the entire image, but no labels for parts of an image (e.g. where is the tumor exactly?). So the powerful property of their method is the ability to learn what parts of the image relate to the label on it's own.

I agree with this.

I see what he means but I wouldn't go this far. Canadians do have a seperate cultural identity. It is defined by its lack of definiton and certainty, in contrast to American certainty. This ks why it isore resilient. It cannot have certainty because our nation was founded on "two solitudes" of French and English, Catholic and Protestant, and also the very different, though equally destructive relationship of the Eurooean colonizers with the Indigenous Peoples of Canada.

The flaw was an assumption that everyone had the nation's best interests at heart, that they all wanted the same thing deep down.

Diversity is Strength. This is really one of their founding principles, even in its hypocrisy. For them the diversity was in religious faith and ways of thinking but did not include gender, ethnicity or anything else. In time this changed and it is the only reason America has done so well.

The minimal necessary to construct a Turing machine

Whats the difference?