Setting up development system

I see that there's a dev task to migrate from Python 2 to 3.  Is that complete?  Which version of Python should I install?

@Dangerdad137 welcome on board! 

Posted by: @dangerdad137

I see that there's a dev task to migrate from Python 2 to 3.  Is that complete?  Which version of Python should I install?

Yes that is correct. And this will be the upcoming release which will be 0.9.0. The python port is almost done. I am using python 3.7 but i think all version > 3.5 will work.

Just some notes about the project structure:

As you already discovered, the current working code is located at: https://github.com/mariolukas/FabScanPi-Server/tree/feature/python3

The new frontend code is located in this fork: https://github.com/KaZZacH/FabScanPi-Frontend

We need to get rid of the old frontend code. The old frontend code (FabScanPi-Frontend) is ugly and it's still written in coffee script and uses a grunt task to compile it to javascript. The code base is over 5 years old and didn't change since the first bottom up quick and dirty version. The backend code (FabScanPi-Server) much better.

The FabScanPi-Server acts as a webserver using Pythons tornado lib. A built version of the Frontend is located in the www folder of the FabScanPi-Server repository.

There is also a Firmware repository which holds the Arduino Firmware for the FabScanPi. The latest hex file is also located in the FabScanPi-Server firmware folder. https://github.com/mariolukas/FabScanPi-Firmware

A last repository keeps the build script for the Raspbian image. https://github.com/mariolukas/FabScanPi-Build-Raspbian

The last repository keeps all the files for the FabScan Pi case. https://github.com/mariolukas/FabScan-Case

As i mentioned in your welcome thread, i will try to do some documentation on how to setup the dev tool chain. 

I started to write some documentation on how you can setup pycharm for remote developing.

https://fabscanpi-server.readthedocs.io/en/develop/development_backend.html

It is not complete. Maybe some points are missing. But i think you will get an idea.

So far I've run into the following:

missing module 'cv2' -- I installed it with:

pip3 install opencv-python

Now I get the following when running from PyCharm:

  File "/home/mping/dev/FabScanPi-Server/src/fabscan/server/services/api/FSStreamHandler.py", line 44, in FSStreamHandler
    @tornado.web.asynchronous
AttributeError: module 'tornado.web' has no attribute 'asynchronous'

I should add, I'm running on Ubuntu 20.04.
Python version is 3.8.2

tornado version is 6.0.4

First of all, thank you again for joining the project!

Have you seen the guide. It describes how you can setup pycharm for remote development on the pi. Where you are developing and writing your code on your ubuntu machine, but the code runs on the pi when you execute it. That is the best way to go until we have the dummy driver stuff. 

https://fabscanpi-server.readthedocs.io/en/develop/development_backend.html

Posted by: @dangerdad137

I should add, I'm running on Ubuntu 20.04.
Python version is 3.8.2

tornado version is 6.0.4

I think that doesn't matter. But whenever i decided to add dependencies i tried to use those which are available over apt-get /aptidude. I know that pip is the python package manager and normally i use it when i do some python development. But due the fact that we build debian packages later it is easier to fit the package dependencies when we use only debian packages. 

Most packages can be installed by using 

sudo apt-get install python3-<package-name>

e.g. python3-opencv, python3-tornado ... etc.

If you are able to run it with pip packages you are good to go. 

By the way i've never tried to run the code on another system than the raspberry. I expect no problems. There is only one dependency which might be a problem, and that is the picamera library. It is only run able on the pi because it was written to drive the raspberry pi camera module and is more or less an abstraction layer for the low level camera driver. I think when we replace it by dummy drivers it shouldn't be a problem anymore.

One more thing, it is important to have a OpenCV installation with built-in multicore support. Most Linux distributions on desktop systems have opencv packages where multicore is compiled with multicore support. However the pre-compiled pi packages are not. 

That is the reason why the fabscan repository brings own opencv packages: 

https://archive.fabscan.org/pool/main/o/opencv/

I recompiled them with other flags. I don't know why the guys behind the raspbian disto don't do that. There are only a few very old raspberries not able to handle more than one core. 

The newest opencv version for python3 is still not in my archive. But i will add it this weekend. 

Yep, I followed the guide, and set it up for remote run. The errors I'm getting come when go through the steps.  It appears a version of tornado removed the 'asynchronous' decorator.  What are the versions on your dev machine?

In particular, the easiest thing to do is match my setup to yours -- if that means installing a different Linux, that's no problem.

Also, I'm concerned the rpi image will start the server by default, making it impossible for me to run the server via PyCharm because the server will already be bound to the web port.  Should I just put default raspian on the dev pi?

Any progess here ?

Byt the ways, there was a bug in the last image ( from 2020-01-01). I just released a new image wich should work now. 

https://github.com/mariolukas/FabScanPi-Build-Raspbian/releases/tag/v.2020-05-27-00-08

The FabScan Software Version is the same, but i fixed a couple of related operating system stuff which caused some problems with the fabscan server. 

In Detail: 

FabScan needs a opencv version compiled with tbb (multiprocessing flag). Raspbian released a newer version of opencv than my latest, so my compiled opencv was overwritten by the official ones. I added package pinning to this images, it so that this will not happen again.

@mario I installed the 0.8.2 image from above.  I've got the scanner at least calibrating, but so far haven't had success with a scan, but no matter.

Can you tell me exactly what your setup is?  It would be easier if I just prepared a dev machine with the same release and libraries you're using.

You should start with the latest testing version. I am going to release that one soon. The python3 port is almost done. 

Istallation: 

1. Download and install the latest SD Card Image from GitHub.

2. Switch to the testing packages in source list.

/etc/apt/sources.list.d/fabscanpi.list

change the entry

deb  http://archive.fabscan.org/  stable main

to

deb http://archive.fabscan.org/ testing main

Then update the package list and upgrade installed packages with: 

sudo apt-get update && sudo apt-get dist-upgrade

When the upgrade process is done, change some configuration things ( the default baudrate changed within the new version).

/etc/fabscanpi/default.config.json

open the config and modify the serial section to this: 

    "serial": {

        "plattform_type": "fabscanpi",

        "baudrate": 115200,

        "autoflash": "True",

        "port": "/dev/ttyAMA0",

        "flash_baudrate": 115200

    },

The old default value was 57600, now it is 115200. Do a server restart with

sudo systemctl restart fabscanpi-server 

Just check it the process is running.

ps -axf

FabScanPi Server should be listet. The new version also provided systemd instead of the old init scripts. You can use the following commands for process control now.

sudo systemctl start fabscanpi-server 

sudo systemctl stop fabscanpi-server 

sudo systemctl restart fabscanpi-server

And thats it. Afterwards you can follow the dev guide to get it running with pycharm. But keep in mind that the server should not run on the pi when you start it remotely from pycharm. You should stop the process on the pi before you start it from pycharm.

I updated the rpi as stated above.  Now that it's on 0.9.0+202006051157 I get the following from PyCharm when I attempt to run.

On branch feature/python3
Your branch is up to date with 'origin/feature/python3'.

  File "/home/mping/dev/FabScanPi-Server/src/fabscan/server/FSScanServer.py", line 15, in <module>
    from fabscan.FSVersion import __version__
ModuleNotFoundError: No module named 'fabscan.FSVersion'

Not sure how to proceed.

I did some fiddling with the config settings and got the 0.9.0 running, but still no success from PyCharm

Posted by: @dangerdad137

I did some fiddling with the config settings and got the 0.9.0 running, but still no success from PyCharm

Yes sure. I forgot to add this to the docs. You need to create the FSVersion.py file manually in the folder you linked to the deployment process. The content should be something like: 

__version__="0.9.1"

The Version File is normally created while the Release build process. I think this error should be catched in dev mode. We can set a static dev version number while developing. A simple import check in try catch block should do the job.

Getting better.  Here's the output in PyCharm when I try to run:

/usr/bin/python3.8 /home/mping/dev/FabScanPi-Server/src/fabscanpi-server --loglevel=debug --config=/home/pi/fabscan/src/fabscan/config/dev.config.json --settings=/home/pi/fabscan/src/fabscan/config/default.settings.json
sh: 1: /usr/bin/avrdude-autoreset: not found
2020-06-12 17:18:23,886 [41104:140427982456640] INFO - fabscan.server.FSScanServer: FabScanPi-Server 0.9.1
2020-06-12 17:18:23,886 [41104:140427982456640] CRITICAL - fabscan: Fatal error: [Errno 2] No such file or directory: '/home/pi/fabscan/src/fabscan/config/dev.config.json'

Not sure if I need to install avrdude-autoreset, or where to find it (I see the project on github).

Also not sure if I need to change the remote folder: /home/pi/fabscan/src/fabscan/config/ or if that should be deploying.  Otherwise as far as I can tell my settings match the dev setup directions. 

I followed the instructions here on the pi to get avrdude-autoreset, but no change in the PyCharm error:

https://github.com/mharizanov/avrdude-rpi

Are you using the FabScanPi Image? ( https://github.com/mariolukas/FabScanPi-Build-Raspbian/releases )

avdrude-autoreset should be contained in the image. You can test it by running it when you are logged in. 

/usr/bin/avrdude-autoreset

Anyway, the file is also part of the FabScanPi Server Package. It is located here:

https://github.com/mariolukas/FabScanPi-Server/tree/develop/src/avrdude

The package installer copies it to /usr/bin and makes it executable with 

chmod a+x /usr/bin/avrdude-autoreset

Maybe my failure. I guess your pycharm is using your local python interpreter and not the remote one on the raspberry. You need to setup the remote python interpreter. I forgot to mention this in my pycharm setup description. I will try to add this to the docs. Meanwhile you can try to set it up by using this guide from JetBrains: https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html

The username is pi and password raspberry. IP Adress is of course the address of your pi. Afterwards you need to select this new added python interpreter as default in your run configuration.

Which kind of Hardware are you using? In case you are using one of the china cilclops you should have a closer look into your config file. It is important to set the correct baudrates for flashing and communication. The Arduino should be connected to the Raspberry. 

You can start with this values. 

"serial": {
    "plattform_type": "ciclop",
    "baudrate": 14400,
    "autoflash": "True",
    "port": "/dev/ttyUSB0",
    "flash_baudrate": 115200
}

Otherwise we need to find some fitting settings. You can do a quick connection test with this command: 

sudo avrdude-autoreset -p m328p -b 115200 -carduino -P/dev/ttyUSB0

This command should list some information about the microcontroller on the arduino. If you are getting errors instead you need to find the error. Possible reasons are a wrong baudrate... try 57600, 38400, 14400 instead. But also the device might be wrong, you can try /dev/ttyUSB1 also.

When you are getting a nice output you can try to run it again with pycharm.

Success!  I had the config as you specified, but there was one last avrdude thing going wrong.

I was able to run the server from PyCharm and connect to the web interface!

I'm up and running!