DtypeWarning: Specify dtype option on import or set low_memory=False

[DenisValcke]

We have been trying to retrieve streets from the data but keep getting the same error when making the attempt:

Error: sys:1: DtypeWarning: Columns (5,8,9,10,12,13,15,16,17) have mixed types. Specify dtype option on import or set low_memory=False.

The command we are using is:

python /app/best/filter/filter.py /app/best/data/data.csv /app/best/data/streets.csv --output_type street --postcode 1020

We have tried several zipcodes but with no success.

Sidenote, is it possible to omit the --postcode flag so that we extract all available streets? We actually need a list of streets instead of a list of addresses.

edit: We are using FROM python:3.7.4-alpine as base, the base of our Docker setup is pretty much the same as the one included in this repo.

[nielsbril]

@JosseVanDelm
I have found a solution for this issue. As the error says, you should specify the datatypes when using the read_csv method.

So line 65 of filter/filter.py should be file = pd.read_csv(args.input_file, dtype='unicode').

This seems to work. I could open a PR if you want, but I don't have write permissions on this repository to create a branch.

@DenisValcke
The --postcode filter is optional, when not specifying it the script returns all streets for all zipcodes.

[JosseVanDelm]

Thank you for reporting this issue @DenisValcke
Sadly, I currently do not have the time to look into this issue.
Please try out the solution that @nielsbril provided and let us know if it resolves your issue.

Also thank you @nielsbril for pointing out your solution and the fact that you can not open a pull request.
I don't have rights to set the permissions either.
@jbelien or @barthanssens can we maybe set this repo up so people can contribute more easily?

[jbelien]

AFAIK, anyone can create a PR !
But obviously not create a branch. @nielsbril you can fork this repository, create a branch on your fork and then from the new branch "push" a PR in this repository.

[nielsbril]

Alright, I'll fork the repo and open a PR 🙂