nothing happens

[LightCheat]

Hi, when I execute your code, I will prompt for nothing happens. I would like to ask if this question is normal. I have been executing it for about an hour and it has always been prompted with nothing happens. I am using Alfworld version 0.3.3, but the same issue occurs when I downgrade to 0.2.2. I hope to receive your answer, thanks

In addition, there will be a warning prompt, and I am not sure if this warning will affect the normal execution of the program.

[chanhee-luke]

Hi! Can you check if the plans are being generated normally? From the screenshot it looks like it but just to double check.

Also what is your computing information?

[LightCheat]

Hello, thank you very much for your prompt reply. I am not sure if the plan is fully generated. The following is a complete screenshot of the program.

[LightCheat]

Additionally, I have reconfigured a Python environment and this time I am using the AFLWORLD that has been modified in your project. But when I was running aflworld play thor, I found that neither go to nor navigation could execute properly. The following is a screenshot of the operation:

[LightCheat]

In the end, I used the method in # 7 to replace the get_object method with the original method in alfworld. As you can see from the screenshot, Navigation can be used now. But I ran run_eval. py again and found that the same situation as before, "nothing happy," still occurred. I am trying to make changes and looking forward to your reply. Thank you.
My computer configuration is: ubuntu18.04 Python=3.9 alfworld=0.2.2 (using your project)

[chanhee-luke]

Weird, seems like interaction is bugged.
I'll try to replicate it with alfworld==0.2.2 and get back to you soon.

Just to double check, the config file is not modified right?

[LightCheat]

Thank you very much for your reply. Does the configuration file refer to the file gpt4_base_config.yaml? I did not make any changes to this file.

[LightCheat]

Hello, I tried to add a label at the end of the sub plan, and from the screenshot, it can be seen that some sub tasks have been identified and will not generate "nothing happy". But that's not right, and I'm not sure what to do afterwards. I will continue to try to make modifications and hope to receive your guidance. Thank you.

[BatmanofZuhandArrgh]

Hi i am having this same issue
Output of step is this: Output of step: observations: ['Nothing happens.'] None [False] {'admissible_commands': [['go to fridge 1', 'go to cabinet 1', 'go to countertop 1', 'go to toaster 1', 'go to coffeemachine 1', 'go to countertop 2', 'go to cabinet 2', 'go to stoveburner 1', 'go to stoveburner 2', 'go to cabinet 3', 'go to cabinet 4', 'go to microwave 1', 'go to countertop 3', 'go to sink 1', 'go to sink 2', 'go to shelf 1', 'go to shelf 2', 'go to drawer 1', 'go to drawer 2', 'go to drawer 3', 'go to shelf 3', 'go to stoveburner 3', 'go to garbagecan 1', 'inventory', 'look']], 'won': [False], 'goal_condition_success_rate': [0.0], 'extra.gamefile': ['/home/anhnguyen/Documents/VLM-Planner/alfworld/data/json_2.1.1/valid_unseen/pick_heat_then_place_in_recep-Mug-None-CoffeeMachine-10/trial_T20190907_171933_349922'], 'expert_plan': [[]]}

It seems that the (my version of) env.step will only intake low-level actions. Not Navigation toaster, but go to toaster 1 @LightCheat

[BatmanofZuhandArrgh]

I think the code is missing the step where it converts high level plan (high_pddl) to low level actions (low_actions)

[chanhee-luke]

Hi, I found that there is a bug where the updated OpenAI package has a conflict with grounding the objects (for interaction actions). We’re working actively to fix this since but it needs some overhaul.

In the meantime, high level plan generation works totally fine. Sorry and please be patience!

[BatmanofZuhandArrgh]

HI @chanhee-luke , i think the OpenAI version problem is ok, right? I think the high level plan actions are fine, i'm just having trouble finding where they convert to low level ones

[BatmanofZuhandArrgh]

Sorry idk if we can call them "low_level". I just mean mapping "Navigation" to "go to", maybe this old vocabulary was used in alfworld==0.2.2? I'm using alfworld==0.3.3 right now, the newest one so maybe that vocab doesn't work anymore?
Anyway, I'm thinking I'm gonna write a mapping function from the llm output to the input to the step function. @chanhee-luke can u point me to where this conversion function is, if it exist? (maybe in env.step?)

EDIT: I think i found it. The parse_command() in alfworld/alfworld/agents/controller/base.py in between version 0.2 and 0.3. It's causing more conflict, i'll try to see what i can do

[LightCheat]

Sorry idk if we can call them "low_level". I just mean mapping "Navigation" to "go to", maybe this old vocabulary was used in alfworld==0.2.2? I'm using alfworld==0.3.3 right now, the newest one so maybe that vocab doesn't work anymore? Anyway, I'm thinking I'm gonna write a mapping function from the llm output to the input to the step function. @chanhee-luke can u point me to where this conversion function is, if it exist? (maybe in env.step?)

EDIT: I think i found it. The parse_command() in alfworld/alfworld/agents/base.py in between version 0.2 and 0.3. It's causing more conflict, i'll try to see what i can do

Hi, you can try replacing the file alfworld/alfworld/agents/base.py with the same file in the same path in this repository. The author has modified this file and redefined go to as navigation.

[BatmanofZuhandArrgh]

@LightCheat do u mean alfworld/alfworld/agents/controller/base.py?
Have u got it to work? Is that what u tried?
This still happened.
New plan!!!
Act 5: Navigation cabinet
Obs 5: Nothing happens.
Generated plans: [' Navigation shower', ' PickupObject soapbar', ' Navigation cabinet', ' OpenObject cabinet', ' PutObject soapbar cabinet', ' CloseObject cabinet', ' Navigation countertop', ' PickupObject soapbar', ' Navigation cabinet', ' OpenObject cabinet', ' PutObject soapbar cabinet', ' CloseObject cabinet']
Plan: Navigation shower
Traceback (most recent call last):
File "/home/anhnguyen/Documents/VLM-Planner/alfworld/alfworld/agents/controller/oracle.py", line 176, in step
self.curr_loc = recep['locs']
TypeError: 'NoneType' object is not subscriptable

I think we gotta input the object index as well

[LightCheat]

@LightCheat do u mean alfworld/alfworld/agents/controller/base.py? Have u got it to work? Is that what u tried? This still happened. New plan!!! Act 5: Navigation cabinet Obs 5: Nothing happens. Generated plans: [' Navigation shower', ' PickupObject soapbar', ' Navigation cabinet', ' OpenObject cabinet', ' PutObject soapbar cabinet', ' CloseObject cabinet', ' Navigation countertop', ' PickupObject soapbar', ' Navigation cabinet', ' OpenObject cabinet', ' PutObject soapbar cabinet', ' CloseObject cabinet'] Plan: Navigation shower Traceback (most recent call last): File "/home/anhnguyen/Documents/VLM-Planner/alfworld/alfworld/agents/controller/oracle.py", line 176, in step self.curr_loc = recep['locs'] TypeError: 'NoneType' object is not subscriptable

I think we gotta input the object index as well

I haven't solved this problem, you can take a look at the screenshot of my above answer. I tried to add a number after the subtasks, and some of them can be executed normally. But I have no way to make any further modifications.

[BatmanofZuhandArrgh]

In the end, I used the method in # 7 to replace the get_object method with the original method in alfworld. As you can see from the screenshot, Navigation can be used now. But I ran run_eval. py again and found that the same situation as before, "nothing happy," still occurred. I am trying to make changes and looking forward to your reply. Thank you. My computer configuration is: ubuntu18.04 Python=3.9 alfworld=0.2.2 (using your project)

Thank u for your reply. I'm not sure how this is supposed to add indices to the objects the robot see?

[LightCheat]

In the end, I used the method in # 7 to replace the get_object method with the original method in alfworld. As you can see from the screenshot, Navigation can be used now. But I ran run_eval. py again and found that the same situation as before, "nothing happy," still occurred. I am trying to make changes and looking forward to your reply. Thank you. My computer configuration is: ubuntu18.04 Python=3.9 alfworld=0.2.2 (using your project)

Thank u for your reply. I'm not sure how this is supposed to add indices to the objects the robot see?

The method I used is very simple, I added an additional sentence of plan=plan+'1', just to test if my idea is correct. After the test was over, I was not sure what to do next, so I gave up making further modifications.

[BatmanofZuhandArrgh]

If we can have the num_id of the objects that are visible:
init_vis_objs = [obj["objectType"] for obj in env.envs[0].env.last_event.metadata['objects'] if obj['visible']]
We can use that to map actions and find admissible action that the agent can see. Like object visible is TV with num_id = 2, llm output is "Navigation TV", find in admissible action "go to TV 2"
I'll work on this after dinner lol

[LightCheat]

If we can have the num_id of the objects that are visible: init_vis_objs = [obj["objectType"] for obj in env.envs[0].env.last_event.metadata['objects'] if obj['visible']] We can use that to map actions and find admissible action that the agent can see. Like object visible is TV with num_id = 2, llm output is "Navigation TV", find in admissible action "go to TV 2" I'll work on this after dinner lol

Yes, it can be continued through this method. But I believe that if there are multiple objectives, the order may not be easy to grasp, and it may also lead to errors in other parts of the author's code, so I gave up on making subsequent modifications.

[BatmanofZuhandArrgh]

Wait, I assume that in LLM-Planner, it just knows that an object like fridge exists, but not know which fridge (1 or 2) it's going to, right? Cuz from the issues and code I'm reading in alfworld's repo alfworld/alfworld#16, they generate object index sequentially on the fly. Can you confirm that the robot doesn't go to the objects it sees, but just go to a random object? @chanhee-luke correct me if i'm wrong, pls

[chanhee-luke]

Hi @BatmanofZuhandArrgh, for LLM-Planner, the priority of the object selection is 1) objects that are visible, 2) objects that exist in the environment. Usually there aren't many cases of multiple instances of same object type (e.g. single fridge) so it has minor impact to the overall success rate.

The main translation from the generated high level plan to Ai2Thor compatible plans happen in

LLM-Planner/alfworld/alfworld/agents/controller/base.py

Line 225 in ace9a93

def parse_command(self, action_str):

We are currently working on replacing the ALFWorld backbone since it's not easily compatible with other datasets/simulators and MLLMs so in the meantime I recommend using this codebase (https://github.com/lbaa2022/LLMTaskPlanning). Please look out for the revamped codebase soon.

[BatmanofZuhandArrgh]

Hi @BatmanofZuhandArrgh, for LLM-Planner, the priority of the object selection is 1) objects that are visible, 2) objects that exist in the environment. Usually there aren't many cases of multiple instances of same object type (e.g. single fridge) so it has minor impact to the overall success rate.

The main translation from the generated high level plan to Ai2Thor compatible plans happen in

LLM-Planner/alfworld/alfworld/agents/controller/base.py

Line 225 in ace9a93

def parse_command(self, action_str):

We are currently working on replacing the ALFWorld backbone since it's not easily compatible with other datasets/simulators and MLLMs so in the meantime I recommend using this codebase (https://github.com/lbaa2022/LLMTaskPlanning). Please look out for the revamped codebase soon.

Hi @chanhee-luke , thank you for your reply. My problem here is that from
[obj["objectType"] for obj in env.envs[0].env.last_event.metadata['objects'] if obj['visible']]
We know what object exist in the field-of-view of the robot, and its object_id. However we don't know if it's apple 1 or apple 2. I have played some Thor game and there are scenes with 7 drawers, each with different content inside of it. Maybe our 192 cases don't have that many receptibles? If not, do you know where the logic the robot used to prioritize visible objects is? Because from what I understand, the llm outputs Navigation fridge, with no index?

[chanhee-luke]

First of all, the ALFRED task is designed so that usually you can interact with any object of the same kind to complete the task. Which means that if you need to slice an apple, you can slice any apple in the room to be counted as successful.

The ALFWorld simulator keeps track of visible objects by default so the code is accessing that visible object information from the simulator. If there are multiple visible objects of the same type (e.g. multiple apples) it randomly chooses one. LLM-Planner does not output index and the code doesn't use index at all.

[BatmanofZuhandArrgh]

Okie thank u so much for the live tech support lol @chanhee-luke

[lxsy-xcy]

So, does this means this codebase cannot be reproduced without bug? I mean the Nothing happen is because of the bugs or the capacity of the llm-planner?

[chanhee-luke]

Hi @lxsy-xcy . No you can still generate high level plans and compare them with the results on the paper. We have all the prompts used in the codebase. Our work aims to show the potential of the LLM in high level planning. Thanks for the interest!

[lxsy-xcy]

thanks! But does this codebase can have a accurate success rate of the high level planning? I got a lot "0", I don't know if that means the llm-planner cannot solve this question or just because of the ALFWORLD part have some bugs?

[chanhee-luke]

No since high-level planning SR != SR. There could be issues of the low-level planner that is independent of the high-level planning. ALFWORLD part do seem to have bugs. We are trying to decouple it from the LLM-Planner so people can use it independently of any low-level planner.

[lxsy-xcy]

Okie! Thank you for your reply