Setting up developm...
 
Notifications
Clear all

Setting up development system

24 Posts
2 Users
0 Likes
1,589 Views
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

Hey all.  I posted my Introduction over in the forums, and wanted to stimulate getting steps to put together whatever is needed for development.

I have a laptop with a fresh Ubuntu 20.04 install, and a VM on my desktop with 20.04 as well (the desktop has lots of RAM & CPU so a VM might actually perform better).  I've installed PyCharm as an IDE.

If I can get "next steps" then we can put together the bullet points of what's needed for a dev machine.  Or I can wait for a complete list, whichever is more convenient.

Reply
23 Replies
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

I see that there's a dev task to migrate from Python 2 to 3.  Is that complete?  Which version of Python should I install?

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

@Dangerdad137 welcome on board! 

Posted by: @dangerdad137

I see that there's a dev task to migrate from Python 2 to 3.  Is that complete?  Which version of Python should I install?

Yes that is correct. And this will be the upcoming release which will be 0.9.0. The python port is almost done. I am using python 3.7 but i think all version > 3.5 will work.

Just some notes about the project structure:

As you already discovered, the current working code is located at: https://github.com/mariolukas/FabScanPi-Server/tree/feature/python3

The new frontend code is located in this fork: https://github.com/KaZZacH/FabScanPi-Frontend

We need to get rid of the old frontend code. The old frontend code (FabScanPi-Frontend) is ugly and it's still written in coffee script and uses a grunt task to compile it to javascript. The code base is over 5 years old and didn't change since the first bottom up quick and dirty version. The backend code (FabScanPi-Server) much better.

The FabScanPi-Server acts as a webserver using Pythons tornado lib. A built version of the Frontend is located in the www folder of the FabScanPi-Server repository.

There is also a Firmware repository which holds the Arduino Firmware for the FabScanPi. The latest hex file is also located in the FabScanPi-Server firmware folder. https://github.com/mariolukas/FabScanPi-Firmware

A last repository keeps the build script for the Raspbian image. https://github.com/mariolukas/FabScanPi-Build-Raspbian

The last repository keeps all the files for the FabScan Pi case. https://github.com/mariolukas/FabScan-Case

As i mentioned in your welcome thread, i will try to do some documentation on how to setup the dev tool chain. 

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

I started to write some documentation on how you can setup pycharm for remote developing.

https://fabscanpi-server.readthedocs.io/en/develop/development_backend.html

It is not complete. Maybe some points are missing. But i think you will get an idea.

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

So far I've run into the following:

missing module 'cv2' -- I installed it with:

pip3 install opencv-python

Now I get the following when running from PyCharm:

  File "/home/mping/dev/FabScanPi-Server/src/fabscan/server/services/api/FSStreamHandler.py", line 44, in FSStreamHandler
@tornado.web.asynchronous
AttributeError: module 'tornado.web' has no attribute 'asynchronous'
Reply
Dangerdad137
(@dangerdad137)
Joined: 3 years ago

Active Member
Posts: 19

Removing the decorator resolves the error leaving the @tornado.gen.coroutine in place -- not sure if that's the correct solution.

 

Also, I'm wondering if I fubar'd the current installation for development should I have the standard installation image SD or something different on the pi?

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

I should add, I'm running on Ubuntu 20.04.
Python version is 3.8.2

tornado version is 6.0.4

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

First of all, thank you again for joining the project!

Have you seen the guide. It describes how you can setup pycharm for remote development on the pi. Where you are developing and writing your code on your ubuntu machine, but the code runs on the pi when you execute it. That is the best way to go until we have the dummy driver stuff. 

https://fabscanpi-server.readthedocs.io/en/develop/development_backend.html

Posted by: @dangerdad137

I should add, I'm running on Ubuntu 20.04.
Python version is 3.8.2

tornado version is 6.0.4

I think that doesn't matter. But whenever i decided to add dependencies i tried to use those which are available over apt-get /aptidude. I know that pip is the python package manager and normally i use it when i do some python development. But due the fact that we build debian packages later it is easier to fit the package dependencies when we use only debian packages. 

Most packages can be installed by using 

sudo apt-get install python3-<package-name>

e.g. python3-opencv, python3-tornado ... etc.

If you are able to run it with pip packages you are good to go. 

By the way i've never tried to run the code on another system than the raspberry. I expect no problems. There is only one dependency which might be a problem, and that is the picamera library. It is only run able on the pi because it was written to drive the raspberry pi camera module and is more or less an abstraction layer for the low level camera driver. I think when we replace it by dummy drivers it shouldn't be a problem anymore.

One more thing, it is important to have a OpenCV installation with built-in multicore support. Most Linux distributions on desktop systems have opencv packages where multicore is compiled with multicore support. However the pre-compiled pi packages are not. 

That is the reason why the fabscan repository brings own opencv packages: 

https://archive.fabscan.org/pool/main/o/opencv/

I recompiled them with other flags. I don't know why the guys behind the raspbian disto don't do that. There are only a few very old raspberries not able to handle more than one core. 

The newest opencv version for python3 is still not in my archive. But i will add it this weekend. 

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

Yep, I followed the guide, and set it up for remote run. The errors I'm getting come when go through the steps.  It appears a version of tornado removed the 'asynchronous' decorator.  What are the versions on your dev machine?

In particular, the easiest thing to do is match my setup to yours -- if that means installing a different Linux, that's no problem.

Also, I'm concerned the rpi image will start the server by default, making it impossible for me to run the server via PyCharm because the server will already be bound to the web port.  Should I just put default raspian on the dev pi?

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

Any progess here ?

Byt the ways, there was a bug in the last image ( from 2020-01-01). I just released a new image wich should work now. 

https://github.com/mariolukas/FabScanPi-Build-Raspbian/releases/tag/v.2020-05-27-00-08

The FabScan Software Version is the same, but i fixed a couple of related operating system stuff which caused some problems with the fabscan server. 

In Detail: 

FabScan needs a opencv version compiled with tbb (multiprocessing flag). Raspbian released a newer version of opencv than my latest, so my compiled opencv was overwritten by the official ones. I added package pinning to this images, it so that this will not happen again.

Reply
Dangerdad137
(@dangerdad137)
Joined: 3 years ago

Active Member
Posts: 19

@mario Apologies for the late response.  I got fixated in just trying to get a decent scan or calibration.  I'm running 0.8.2 after reinstalling the image from scratch.  I designed laser holders and a rpi camera holder than can have the angle be adjusted so I've actually got the center of the scan platform dead center of the camera view, and the lasers exactly crossing at the center.  I'll go ahead and install the latest version and see what I can.  I actually have 3 days off from work this week, and would like to spend a lot of time on this then.

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

@mario I installed the 0.8.2 image from above.  I've got the scanner at least calibrating, but so far haven't had success with a scan, but no matter.

 

Can you tell me exactly what your setup is?  It would be easier if I just prepared a dev machine with the same release and libraries you're using.

 

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

You should start with the latest testing version. I am going to release that one soon. The python3 port is almost done. 

Istallation: 

1. Download and install the latest SD Card Image from GitHub.

2. Switch to the testing packages in source list.

/etc/apt/sources.list.d/fabscanpi.list

change the entry

deb  http://archive.fabscan.org/  stable main

to

deb http://archive.fabscan.org/ testing main

Then update the package list and upgrade installed packages with: 

sudo apt-get update && sudo apt-get dist-upgrade

When the upgrade process is done, change some configuration things ( the default baudrate changed within the new version).

/etc/fabscanpi/default.config.json

open the config and modify the serial section to this: 

    "serial": {

        "plattform_type": "fabscanpi",

        "baudrate": 115200,

        "autoflash": "True",

        "port": "/dev/ttyAMA0",

        "flash_baudrate": 115200

    },

The old default value was 57600, now it is 115200. Do a server restart with

sudo systemctl restart fabscanpi-server 

Just check it the process is running.


ps -axf

FabScanPi Server should be listet. The new version also provided systemd instead of the old init scripts. You can use the following commands for process control now.

sudo systemctl start fabscanpi-server 

sudo systemctl stop fabscanpi-server 

sudo systemctl restart fabscanpi-server

And thats it. Afterwards you can follow the dev guide to get it running with pycharm. But keep in mind that the server should not run on the pi when you start it remotely from pycharm. You should stop the process on the pi before you start it from pycharm.

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

I updated the rpi as stated above.  Now that it's on 0.9.0+202006051157 I get the following from PyCharm when I attempt to run.

On branch feature/python3
Your branch is up to date with 'origin/feature/python3'.


File "/home/mping/dev/FabScanPi-Server/src/fabscan/server/FSScanServer.py", line 15, in <module>
from fabscan.FSVersion import __version__
ModuleNotFoundError: No module named 'fabscan.FSVersion'

Not sure how to proceed.
Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

I did some fiddling with the config settings and got the 0.9.0 running, but still no success from PyCharm

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago
Posted by: @dangerdad137

I did some fiddling with the config settings and got the 0.9.0 running, but still no success from PyCharm

Yes sure. I forgot to add this to the docs. You need to create the FSVersion.py file manually in the folder you linked to the deployment process. The content should be something like: 

__version__="0.9.1"

The Version File is normally created while the Release build process. I think this error should be catched in dev mode. We can set a static dev version number while developing. A simple import check in try catch block should do the job.

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

Getting better.  Here's the output in PyCharm when I try to run:

/usr/bin/python3.8 /home/mping/dev/FabScanPi-Server/src/fabscanpi-server --loglevel=debug --config=/home/pi/fabscan/src/fabscan/config/dev.config.json --settings=/home/pi/fabscan/src/fabscan/config/default.settings.json
sh: 1: /usr/bin/avrdude-autoreset: not found
2020-06-12 17:18:23,886 [41104:140427982456640] INFO - fabscan.server.FSScanServer: FabScanPi-Server 0.9.1
2020-06-12 17:18:23,886 [41104:140427982456640] CRITICAL - fabscan: Fatal error: [Errno 2] No such file or directory: '/home/pi/fabscan/src/fabscan/config/dev.config.json'

Not sure if I need to install avrdude-autoreset, or where to find it (I see the project on github).

Also not sure if I need to change the remote folder: /home/pi/fabscan/src/fabscan/config/ or if that should be deploying.  Otherwise as far as I can tell my settings match the dev setup directions. 

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

I followed the instructions here on the pi to get avrdude-autoreset, but no change in the PyCharm error:

https://github.com/mharizanov/avrdude-rpi

Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

Are you using the FabScanPi Image? ( https://github.com/mariolukas/FabScanPi-Build-Raspbian/releases )

avdrude-autoreset should be contained in the image. You can test it by running it when you are logged in. 

/usr/bin/avrdude-autoreset

Anyway, the file is also part of the FabScanPi Server Package. It is located here:

https://github.com/mariolukas/FabScanPi-Server/tree/develop/src/avrdude

The package installer copies it to /usr/bin and makes it executable with 

chmod a+x /usr/bin/avrdude-autoreset

 

 

Reply
Dangerdad137
(@dangerdad137)
Joined: 3 years ago

Active Member
Posts: 19

@mario Yes I'm using the latest image, then updated to latest. I've manually installed avrdude-reset now but am still getting the same error.

 

/usr/bin/python3.8 /snap/pycharm-professional/201/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 41111 --file /home/mping/dev/FabScanPi-Server/src/fabscanpi-server --loglevel=debug --config=/home/pi/fabscan/src/fabscan/config/dev.config.json --settings=/home/pi/fabscan/src/fabscan/config/default.settings.json
pydev debugger: process 2318 is connecting

Connected to pydev debugger (build 201.7846.77)
sh: 1: /usr/bin/avrdude-autoreset: not found
2020-06-14 09:09:01,385 [2318:140607759497024] INFO - fabscan.server.FSScanServer: FabScanPi-Server 0.9.1
2020-06-14 09:09:01,386 [2318:140607759497024] CRITICAL - fabscan: Fatal error: [Errno 2] No such file or directory: '/home/pi/fabscan/src/fabscan/config/dev.config.json'

Process finished with exit code 1
Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

Maybe my failure. I guess your pycharm is using your local python interpreter and not the remote one on the raspberry. You need to setup the remote python interpreter. I forgot to mention this in my pycharm setup description. I will try to add this to the docs. Meanwhile you can try to set it up by using this guide from JetBrains: https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html

The username is pi and password raspberry. IP Adress is of course the address of your pi. Afterwards you need to select this new added python interpreter as default in your run configuration.

Reply
Dangerdad137
(@dangerdad137)
Joined: 3 years ago

Active Member
Posts: 19

@mario  Awesome!  Based on this and some fiddling in PyCharm, I've got past that error.   I think I'm almost there.

ssh://pi@10.25.25.145:22/usr/bin/python3 -u /home/pi/dev/FabScanPi-Server/src/fabscanpi-server --loglevel=debug --config=/home/pi/dev/FabScanPi-Server/src/fabscan/config/dev.config.json --settings=/home/pi/dev/FabScanPi-Server/src/fabscan/config/default.settings.json
avrdude-original: Using autoreset DTR on GPIO Pin 7
sudo: /usr/bin/avrdude: command not found
2020-06-14 18:12:36,766 [9909:1996188368] INFO - fabscan.server.FSScanServer: FabScanPi-Server 0.9.1
2020-06-14 18:12:36,769 [9909:1996188368] DEBUG - fabscan.scanner.interfaces: Scanner Type is: laserscanner
2020-06-14 18:12:37,658 [9909:1996188368] DEBUG - fabscan.scanner.laserscanner.driver.FSSerial: Port in Config found
2020-06-14 18:12:37,660 [9909:1996188368] DEBUG - fabscan.scanner.laserscanner.driver.FSSerial: Latest available firmware version is: v.20200306
2020-06-14 18:12:37,778 [9909:1996188368] ERROR - fabscan.scanner.laserscanner.driver.FSSerial: Communication error on port /dev/ttyUSB0 try other flashing baudrate than 115200. Maybe corrupted bootloader.
2020-06-14 18:12:37,779 [9909:1996188368] ERROR - fabscan.scanner.laserscanner.driver.FSSerial: Fatal FabScanPi HAT or compatible connection error....'NoneType' object has no attribute 'isOpen'
2020-06-14 18:12:37,779 [9909:1996188368] INFO - fabscan.server.FSScanServer: FabScan Server Exit. Bye!

Process finished with exit code 0
Reply
mario
Posts: 570
Admin
(@mario)
Honorable Member
Joined: 4 years ago

Which kind of Hardware are you using? In case you are using one of the china cilclops you should have a closer look into your config file. It is important to set the correct baudrates for flashing and communication. The Arduino should be connected to the Raspberry. 

You can start with this values. 

"serial": {
"plattform_type": "ciclop",
"baudrate": 14400,
"autoflash": "True",
"port": "/dev/ttyUSB0",
"flash_baudrate": 115200
}

Otherwise we need to find some fitting settings. You can do a quick connection test with this command: 

sudo avrdude-autoreset -p m328p -b 115200 -carduino -P/dev/ttyUSB0

This command should list some information about the microcontroller on the arduino. If you are getting errors instead you need to find the error. Possible reasons are a wrong baudrate... try 57600, 38400, 14400 instead. But also the device might be wrong, you can try /dev/ttyUSB1 also.

When you are getting a nice output you can try to run it again with pycharm.

 

Reply
Dangerdad137
Posts: 19
Topic starter
(@dangerdad137)
Active Member
Joined: 3 years ago

Success!  I had the config as you specified, but there was one last avrdude thing going wrong.

I was able to run the server from PyCharm and connect to the web interface!

I'm up and running!

Reply
Share: