8 - Deep Learning: crash during start training

I’m getting this crash when I try to run start training. It seems that some dependencies might be missing or out of date.

$ roslaunch my_rosbots_training start_training.launch
… logging to /home/user/.ros/log/981c6f60-e018-11ea-89da-02658400611f/roslaunch-rosdscomputer-12832.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://rosdscomputer:36069/

SUMMARY

PARAMETERS

  • /rosbots/alpha: 0.01
  • /rosbots/alpha_decay: 0.01
  • /rosbots/angular_speed: 0.45
  • /rosbots/batch_size: 64
  • /rosbots/cur_lane_offset: 1.51
  • /rosbots/end_episode_points: 200
  • /rosbots/episodes_training: 1000
  • /rosbots/epsilon: 1.0
  • /rosbots/epsilon_decay: 0.995
  • /rosbots/epsilon_min: 0.01
  • /rosbots/follow_lane_reward: 5
  • /rosbots/gamma: 1.0
  • /rosbots/init_linear_forward_speed: 0.0
  • /rosbots/init_linear_turn_speed: 0.0
  • /rosbots/left_right_reward: 2.5
  • /rosbots/linear_forward_speed: 1.5
  • /rosbots/linear_turn_speed: 0.1
  • /rosbots/look_ahead_distance: 100
  • /rosbots/max_lane_offset: 1.5
  • /rosbots/min_episodes: 100
  • /rosbots/monitor: True
  • /rosbots/n_actions: 3
  • /rosbots/n_observations: 19200
  • /rosbots/n_win_ticks: 250
  • /rosbots/number_decimals_precision_obs: 4
  • /rosbots/quiet: False
  • /rosbots/safe_lane_offset: -1.5
  • /rosbots/speed_step: 1.0
  • /rosbots/veer_off_reward: -10
  • /rosdistro: kinetic
  • /rosversion: 1.12.14

NODES
/
rosbots_n1try_algorithm (my_rosbots_training/start_training.py)

ROS_MASTER_URI=http://master:11311

process[rosbots_n1try_algorithm-1]: started with pid [13337]
/usr/local/lib/python2.7/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.25.9) or chardet (2.3.0) doesn’t match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
2020-08-17 00:29:15.778855: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libnvinfer.so.6’; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/home/user/catkin_ws/devel/lib:/home/simulations/public_sim_ws/devel/lib:/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/usr/local/cuda-10.2/lib64:/home/user/.catkin_ws_python3/src/TensorRT-7.0.0.11/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/usr/local/cuda-10.2/lib64:/home/user/.catkin_ws_python3/src/TensorRT-7.0.0.11/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/x86_64-linux-gnu/gazebo-7/plugins:/usr/lib/x86_64-linux-gnu/gazebo-7/plugins
2020-08-17 00:29:15.780820: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libnvinfer_plugin.so.6’;dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/home/user/catkin_ws/devel/lib:/home/simulations/public_sim_ws/devel/lib:/opt/ros/kinetic/lib:/opt/ros/kinetic/lib/x86_64-linux-gnu:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/usr/local/cuda-10.2/lib64:/home/user/.catkin_ws_python3/src/TensorRT-7.0.0.11/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/opt/ros/kinetic/share/euslisp/jskeus/eus//Linux64/lib:/usr/local/cuda-10.2/lib64:/home/user/.catkin_ws_python3/src/TensorRT-7.0.0.11/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/x86_64-linux-gnu/gazebo-7/plugins:/usr/lib/x86_64-linux-gnu/gazebo-7/plugins
2020-08-17 00:29:15.781523: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

Confirmed the error and we will try to solve it as soon as possible.
Sorry for the inconvenience