The graph couldn't be sorted in topological order - during tgan.fit

[jzkoh]

Hi,

When I run the experiment or tgan.fit, there will be warning messages as such below, about The graph couldn't be sorted in topological order.

Should this be of concern?

Thank you.

�[32m[1010 06:26:22 @base.py:275]�[0m Start Epoch 1 ...

  0%|          |0/5000[00:00<?,?it/s]2019-10-10 06:26:26.595311: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-10-10 06:26:26.868907: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-10-10 06:26:26.987556: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] layout failed: Invalid argument: The graph couldn't be sorted in topological order.
2019-10-10 06:26:27.797393: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] remapper failed: Invalid argument: The graph couldn't be sorted in topological order.
2019-10-10 06:26:27.850938: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] arithmetic_optimizer failed: Invalid argument: The graph couldn't be sorted in topological order.
2019-10-10 06:26:27.897542: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-10-10 06:26:27.986491: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2019-10-10 06:26:35.334083: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0

[pvk-developer]

Hello @jzkoh and thank you for reporting this problem.

We would need some more information in order to analyze this problem:

• Packages and versions that you are using in your virtualenv ? If you are using pip you can run the following command in order to retrieve this information: pip freeze .
• Are you using GPU or CPU?
• Does this occur running the README example and data shown ?
• If does not happen on the README, could you provide a sample of the data that you are using for us to investigate the problem?

[jzkoh]

Hi, thank you for your prompt reply!
I am using the AWS p2.xlarge instance and running this with 1 GPU.
I am running this on the mimic dataset (mimic-3d) from this Kaggle website https://www.kaggle.com/srikanth403/mimic-3d/data

These are my packages:

absl-py==0.8.0
alabaster==0.7.10
anaconda-client==1.6.14
anaconda-project==0.8.2
asn1crypto==0.24.0
astor==0.8.0
astroid==2.3.0
astropy==3.2.1
atomicwrites==1.3.0
attrs==19.1.0
autovizwidget==0.12.9
Babel==2.5.3
backcall==0.1.0
backports.shutil-get-terminal-size==1.0.0
bcrypt==3.1.7
beautifulsoup4==4.6.0
bitarray==0.8.1
bleach==3.1.0
bokeh==1.0.4
boto==2.48.0
boto3==1.9.195
botocore==1.12.195
Bottleneck==1.2.1
cached-property==1.5.1
certifi==2019.6.16
cffi==1.11.5
chardet==3.0.4
click==6.7
cloudpickle==0.5.3
clyent==1.2.2
colorama==0.3.9
confuse==1.0.0
contextlib2==0.5.5
copulas==0.2.3
cryptography==2.7
cycler==0.10.0
cymem==2.0.2
Cython==0.28.2
cytoolz==0.9.0.1
dask==0.17.5
dataclasses==0.6
datashape==0.5.4
decorator==4.4.0
defusedxml==0.6.0
dill==0.2.9
distributed==1.21.8
docker==3.7.3
docker-compose==1.24.1
docker-pycreds==0.4.0
dockerpty==0.4.1
docopt==0.6.2
docutils==0.14
entrypoints==0.3
environment-kernels==1.1.1
et-xmlfile==1.0.1
exrex==0.10.5
Faker==2.0.2
fastcache==1.0.2
fastprogress==0.1.21
filelock==3.0.4
Flask==1.0.2
Flask-Cors==3.0.4
gast==0.2.2
gevent==1.3.0
glob2==0.6
gmpy2==2.0.8
google-pasta==0.1.7
greenlet==0.4.13
grpcio==1.24.0
h5py==2.10.0
hdijupyterutils==0.12.9
heapdict==1.0.0
html5lib==1.0.1
htmlmin==0.1.12
idna==2.6
imageio==2.3.0
imagesize==1.0.0
importlib-metadata==0.23
ipykernel==4.8.2
ipyparallel==6.2.2
ipython==6.4.0
ipython-genutils==0.2.0
ipywidgets==7.4.0
isort==4.3.21
itsdangerous==0.24
jdcal==1.4
jedi==0.12.0
Jinja2==2.10.1
jmespath==0.9.4
joblib==0.13.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.3
jupyter-console==5.2.0
jupyter-core==4.5.0
jupyterlab==0.32.1
jupyterlab-launcher==0.10.5
Keras==2.3.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.2
llvmlite==0.29.0
locket==0.2.0
lxml==4.2.1
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mccabe==0.6.1
missingno==0.4.2
mistune==0.8.4
mkl-fft==1.0.6
mkl-random==1.0.1
mock==3.0.5
more-itertools==7.2.0
mpmath==1.0.0
msgpack==0.6.2
msgpack-numpy==0.4.4.3
msgpack-python==0.5.6
multipledispatch==0.5.0
murmurhash==1.0.2
nb-conda==2.2.1
nb-conda-kernels==2.2.2
nbconvert==5.6.0
nbformat==4.4.0
networkx==2.1
nltk==3.3
nose==1.3.7
notebook==5.5.0
numba==0.45.1
numexpr==2.6.5
numpy==1.16.5
numpydoc==0.8.0
nvidia-ml-py3==7.352.0
olefile==0.45.1
onnx==1.4.1
openpyxl==2.5.3
opt-einsum==3.1.0
packaging==19.2
pandas==0.24.2
pandas-profiling==2.3.0
pandocfilters==1.4.2
paramiko==2.6.0
parso==0.2.0
partd==0.3.8
path.py==11.0.1
pathlib2==2.3.2
patsy==0.5.0
pep8==1.7.1
pexpect==4.5.0
phik==0.9.8
pickleshare==0.7.4
Pillow==5.2.0
pkginfo==1.4.2
plac==0.9.6
plotly==4.0.0
pluggy==0.13.0
ply==3.11
preshed==2.0.1
prompt-toolkit==1.0.15
protobuf==3.9.2
protobuf3-to-dict==0.1.5
psutil==5.4.5
psycopg2==2.7.5
ptyprocess==0.5.2
py==1.8.0
py4j==0.10.7
pycodestyle==2.4.0
pycosat==0.6.3
pycparser==2.18
pycrypto==2.6.1
pycurl==7.43.0.1
pydicom==1.3.0
pyflakes==1.6.0
pygal==2.4.0
Pygments==2.4.2
pykerberos==1.2.1
pylint==2.4.1
PyNaCl==1.3.0
pyodbc==4.0.23
pyOpenSSL==18.0.0
pyparsing==2.4.2
pyrsistent==0.15.4
PySocks==1.6.8
pyspark==2.3.2
pytest==5.1.3
pytest-arraydiff==0.2
pytest-astropy==0.3.0
pytest-doctestplus==0.1.3
pytest-openfiles==0.3.0
pytest-pylint==0.14.1
pytest-remotedata==0.2.1
python-dateutil==2.8.0
pytz==2019.2
PyWavelets==0.5.2
PyYAML==5.1.2
pyzmq==18.1.0
QtAwesome==0.4.4
qtconsole==4.3.1
QtPy==1.4.1
rdt==0.1.3
regex==2018.1.10
requests==2.20.0
requests-kerberos==0.12.0
retrying==1.3.3
rope==0.10.7
ruamel-yaml==0.15.35
runipy==0.1.5
s3fs==0.1.5
s3transfer==0.2.1
sagemaker==1.34.1
sagemaker-pyspark==1.2.4
scikit-learn==0.21.3
scipy==1.3.1
sdv==0.1.2
seaborn==0.9.0
Send2Trash==1.5.0
simplegeneric==0.8.1
SimpleITK==1.2.2
singledispatch==3.4.0.3
six==1.12.0
snowballstemmer==1.2.1
sortedcollections==0.6.1
sortedcontainers==1.5.10
spacy==2.0.18
sparkmagic==0.12.5
Sphinx==1.7.4
sphinxcontrib-websupport==1.0.1
spyder==3.2.8
SQLAlchemy==1.2.11
sympy==1.1.1
tables==3.4.3
tabulate==0.8.5
tblib==1.3.2
tensorboard==1.14.0
tensorflow==1.9.0
tensorflow-estimator==1.14.0
tensorflow-gpu==1.14.0
tensorpack==0.9.4
termcolor==1.1.0
terminado==0.8.1
testpath==0.4.2
text-unidecode==1.3
texttable==0.9.1
tgan==0.1.0
thinc==6.12.1
toolz==0.9.0
torch==1.1.0
torchvision==0.2.2
tornado==5.1.1
tqdm==4.36.1
traitlets==4.3.2
typed-ast==1.2.0
typing==3.6.4
typing-extensions==3.7.4
ujson==1.35
unicodecsv==0.14.1
urllib3==1.23
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.56.0
Werkzeug==0.16.0
widgetsnbextension==3.4.2
wrapt==1.11.2
xlrd==1.1.0
XlsxWriter==1.0.4
xlwt==1.3.0
zict==0.1.3
zipp==0.6.0