the agent learns skills that increase in complexity

σ

−V ∗

(1 − λ)v(st+1) + λV λt+

zkt , zkt+1

zkt + T k(zt, at), zkt+1

Each feature map mkt = [Eext(st)]k can be interpreted as an object mask correspondingto one particular object slot, where [. . .]k denotes selection of the k-th feature map

For each targe

On each iteration, the single predicate that improves Jsurr the most is added to the se

number of blocks (5 for Easy, 6 forMedium, and 7 for Hard

spatialreasoning capabilities of VLMs

as illustrated inFig. 7

This implies that generating I and G ismore challenging than O

We found from the outputs that ViLaIn tends to omitsome propositions in this domain, making the PDs invalid

recall

Forthe Hanoi domain, L is identical through all problems.

The three pegs arenamed by the number from left to right (e.g., peg1, peg2,and peg3)

All four PDDL problem descriptions represent the planning problem of stacking one blockonto another

Objects can be picked up, dropped and movedaround by the agent.

we call a PDDL Planner

In real life, the en-vironment is often described with natural languagetexts.

nother limitation stems from the inherent randomnesswithin LLMs, c

The decoder D reconstructs an image ˆx0 = D(z0), from which thepolicy π predicts the action a0.

setting anew state of the art for methods without lookahead search

Somewhat surprisingly, the lowest scores in Blocksworldare associated with BlockAmbiguity and KStacksColor; thesetwo problems require the LLM to associate objects basedon their color and we had apriori expected the LLM to becapable of such associations and perform well on this task.

fan-in of the layer.

qst , qat are positive over S and A respectively

ze(x)

3 × 3 blocks

Our proposal distribution q(z = k|x) is deterministic, and bydefining a simple uniform prior over z we obtain a KL divergence constant and equal to log K.

orcesthe autoencoder to focus on object positions

This is notdirectly feasible with conventional policy gradient formula-tions

At is an estimator of the advantage function at timestep t

quality

Thedifference is that we only record objects that are either actionarguments or in contact with them.

A “world” frameserves as a default frame of reference for every object in the environment

However, these approachesassume high-level actions to be provided as input.

A is an uncountably infinite set of primitivedeterministic actions.

One can view the noise vector z in such a GAN as a featurevector, containing some representation of the transition to o′ from o.

Wedefine action predicates PA = {left(1), left(2), right(1), right(2), jump(1), idle(1), ...} and state predicatesPS = {type, closeby, ...}

This dataset will contain a set of tuples, (s, a, s′), of states,actions, and next states

More recently, Chenet al. (2022) explored a variant of DreamerV2 where a Transformer replaces the recurrent network inthe RSSM

straight-through estimator

bject state vectors

DeepMind Lab dataset

showing good variability over the irrelevant factors

blue wall

small, round, red

few example images of an apple paired with the symbol “apple”

unlabeled set of image pairs

vψ (sτ )

ataset of past experience

finite imagination horizon

blocks1-5 (arm, 5 blocks)

in many casesoptimally

In one case, the input data corresponds to one or morestate graphs Gi assumed to originate from hidden planninginstances Pi = 〈D, Ii〉 that need to be uncovered

The latter approaches are less likely to generate crisp represen-tations due to the dependence on images

a latent policy via behavior cloning

π(ot)

Moveover

is

before and after the action of inter-est is taken

which predicts which action at was taken by the agent between consecutive obser-vations ot and ot+1

prior distribution over programs likely to solve tasks inthe domain

positive probability

earch for programs

find best program

We selected the tasks on which Tassa et al. (2018) reportnon-zero performance from image inputs

Finally, we call a PDDL Planner as the de-terministic solver to obtain A, a plan to accomplishthe goal CSL under the predefined scenario.

Task descriptions are constructed using PDDL and symbolicplans are generated using the FAST-DOWNWARD planner

Ourset-up automatically parses LLM-generated language intoa program using our synthetic grammar

The P+S modeloutputs executable PDDL actions

Even with high Exec, some task GCR are low, becausesome tasks have multiple appropriate goal states, but weonly evaluate against a single “true” goal

SR is the fraction of executionsthat achieved all task-relevant goal-conditions

We provide the available objects in theenvironment as a list of strings

“The bowl can also be a container to fillwater”, will be added to the task planner.

perfectly match gold visual semantic plans us-ing only the text directives as input

Generated strings from all models arepost-processed for common errors in sequence-to-sequence models, including token doubling,completing missing bigrams (e.g. “pick <arg1>”→ “pick up <arg1>”), and heuristics for addingmissing argument tags

The ALFRED dataset contains 6,574gold command sequences

astly, the goal predicates for each problem were generatedfrom the "PDDL parameters" field of every data sample.

hus, we have created a PDDL domain file usingour knowledge of the objects and actions in the ALFRED world and a PDDL problem file for eachsample

Since in our task we ignorethe vision part of the data, we might encounter some duplicates between our datasets

ALFWorld uses PDDL - Planning DomainDefinition Language (McDermott et al., 1998) to describe each scene from ALFRED and to constructan equivalent text game using the TextWorld engine.