TART-ZA (TART in South Africa) remains uncalibrated even after running continuous calibration script

[danieludick]

After installing the latest object detection server and calibration software on my Ubuntu 18.04 PC, I am not having any luck with our TART system in SA.

For reference, the image produced looks as follows:

Just for reference, this calibration was run using the following:

sudo docker run --rm -d -e TART_LOGIN_PW=SAPWD -e TART_API=http://146.232.222.105 -v ~/calibration_results:/app --name=cal -it tart_calibration_server

Then I did:

docker exec -it cal bash

And when inside the bash attached to this docker machine, I executed the following:

sh /tart_calibrate.sh

The following output snippet was produced:

...
Calibration output is in /app/cal_2019_02_20_15_38_37.json
SUCCESS
Uploading new antenna positions
SUCCESS

There is a directory in my $HOME with the following content:

dludick@danie-ubuntu-desktop:~/calibration_results$ ls -als
total 28
4 drwxr-xr-x 5 dludick dludick 4096 Feb 21 07:00 .
4 drwxr-xr-x 55 dludick dludick 4096 Feb 18 09:22 ..
4 -rw-r--r-- 1 root root 3510 Feb 20 19:28 cal_2019_02_20_15_36_28.json
4 -rw-r--r-- 1 root root 3491 Feb 20 19:28 cal_2019_02_20_15_38_37.json
0 -rw-r--r-- 1 root root 0 Feb 21 07:00 calibrate.log
4 drwxr-xr-x 2 root root 4096 Feb 18 08:47 calibration_2019_02_18_06_47_55
4 drwxr-xr-x 2 root root 4096 Feb 18 08:50 calibration_2019_02_18_06_50_22
4 drwxr-xr-x 2 root root 4096 Feb 21 07:00 calibration_2019_02_21_05_00_37

I have attached now the file cal_2019_02_20_15_38_37.json (added .txt extension to get it on GitHub):

cal_2019_02_20_15_38_37.json.txt

[danieludick]

Just also to mention: The software on our Raspberry Pi for TART-ZA is still running the original code that was sent to us. After performing antenna position measurements (as documented here), I uploaded the following files to the Pi's directory: /home/pi/git/TART/hardware/rpi/24_ant_setup

calibrated_antenna_positions.json.txt
telescope_config.json.txt

[tmolteno]

Thanks for the report. The calibration minimum is returning -7 (this is a log of the function that is minimising) and usually we get around -11.0. A couple of questions.

1. Could you check the timing of the files. There appear to be two files created with close together dates (roughly 2.5 minutes apart) . Is there a chance that you're running two copies of the calibration script? (this shouldn't really stop the calibration working however).

2. Are the diagnose mode outputs showing all green in the telescope web application?

[danieludick]

The status map of the antennas show all green:

I did observe just after uploading this, that one of the status flags changed red for a short duration. but is back on green now.

Also seems there is some strong source nearby:

Also clear when looking at the spectra (here the antenna that shows a

[tmolteno]

There looks to be a fair amount of RF interference in those spectra. Here is a single antenna spectrum from a telescope at a quiet site:

[danieludick]

Thanks @tmolteno . Added now the calibration.log file

calibrate.log

[tmolteno]

Thanks for the log file, the calibration process is not doing a good job of finding good phases and gains. I will try and come up with a good way to check for RF interference.

The visibilities can be retrieved from the telescope via the API. Here is how to get them from our live telescope: (https://tart.elec.ac.nz/signal/api/v1/imaging/vis). This will list the real and imaginary parts of the visibilities.

If there is strong RFI, the visibility amplitudes may be approaching 1.0. Typically they should be around 0.2. What numbers are you getting?

I've just noticed that there is a very bright satellite (LUCH 5A) dominating the sky here. This may cause calibration issues.

[tmolteno]

You can monitor visibilities using the script in /github/TART/doc/calibration/radio_hub. The following command will show the antenna pair 2-3 visibility as a function of time.

python monitor_vis.py --api https://tart.elec.ac.nz/signal --i 2 --j 3

[danieludick]

Thanks @tmolteno . I will check the visibilities. Looking at the spectra, I also see that one of the Radio Hubs might be exhibiting on-board RFI (see Antenna 12 cluster).

We are planning to build also a new Radio Hub board. If you get a chance, will you please upload a bill of materials for the components.

[danieludick]

Plotted now the monitored visibilities using the monitor_vis.py script for antennas 2 and 3:

[tmolteno]

Hi @danieludick. The BOM has been uploaded to

 hardware/pcb/radio_hub_module/RadioHub2.1/RadioHub2.1_.html

Apologies for the delay.

[tmolteno]

Those visibilities look reasonable, in that the phase is slowly changing (Consistent with things moving over the sky). Could you run the command for a short baseline pair, and a long baseline pair (roughly east-west) and you should see the phase changing faster in the long pair, vs the short pair.