View Issue Details

IDProjectCategoryView StatusLast Update
0002155CCdciel[All Projects] Generalpublic19-07-23 09:26
ReporterhanAssigned ToPatrick Chevalley 
PrioritynormalSeverityfeatureReproducibilityhave not tried
Status resolvedResolutionfixed 
Product Version 
Target Version1.0Fixed in Version 
Summary0002155: Reducing lost time between exposures.
DescriptionI did some CCDCiel performance testing on the Raspberry Pi 3 and Pi 4. For a 4635 x3520 image a 7.5 seconds are lost between the exposures. It doesn't matter if the preview function is used or the images are saved to disk. The camera transfer option was set to ram disk. This performed a little better then the network option.

I'm just wondering if valuable time is lost by saving & loading files to the class 10 memory card in the Raspberry Pi. So I tried a ram disk to keep the files in memory. Both the temporary and capture folder where set to /mnt/ram_disk. There was no performance improvement in both preview and capturing images.

Is there a way to reduce the lost time in exposures? If the temporary and capture folder are the same, could time be saved by keeping the temporary image and just rename it? Does the preview function slow down the process or is this handled in a separate thread?
TagsNo tags attached.

Activities

han

19-07-13 10:49

reporter  

Pi 3 vs Pi 4 .png (9,910 bytes)
Pi 3 vs Pi 4 .png (9,910 bytes)

Patrick Chevalley

19-07-13 11:21

administrator   ~0005755

Yes, this is something I like to work on.

A few month ago I try to implement a way to not wait for the image to be saved on disk an show on screen before to start the new exposure.
This attempt failed because of conflict on the main thread access. But there is probably a way to implement that correctly.

You can show more detail on the timing for the different part of the Indi image reception. For that uncomment {$define camera_debug} at the top of cu_indicamera.pas.
This can give more information on the part that must be improved in priority.

Patrick Chevalley

19-07-13 13:37

administrator   ~0005756

After you uncomment {$define camera_debug} go to procedure T_indicamera.CreateIndiClient; and comment the line 273 to 276 to avoid to make big trace file with the indi protocol. This can affect the performance a lot.

Patrick Chevalley

19-07-13 17:26

administrator   ~0005757

I make a test to see how starting the next exposure before to save and display the image will affect this delay between exposure.

For that I create a new branch because at the moment it is likely to crash if some function like autofocus, dither, meridian flip is done as part of StartCaptureExposure() because it no more run in the main thread. Tested only with Indi, it may also crash with Ascom.
But I want to be sure this approach is effective to reduce the time lost between exposure before to do more work.
Also this is active only for a capture started from the Capture tool, the preview are still sequential.

To test this function do the following:
- git pull
- git checkout EarlyNextExposure
- build ccdciel as usual

To return to the standard code:
- git checkout master

On my test system (I5-3570 3.1Ghz) a capture log with the simulator set to an image size of 4000x3000 now show the following:

2019-07-13T17:18:28.286 1: Starting Light exposure 1/5 for 5 seconds
2019-07-13T17:18:34.977 1: Starting Light exposure 2/5 for 5 seconds
2019-07-13T17:18:36.624 1: Saved file /home/pch/Capture/test_1.fits
2019-07-13T17:18:41.670 1: Starting Light exposure 3/5 for 5 seconds
2019-07-13T17:18:43.308 1: Saved file /home/pch/Capture/test_2.fits
2019-07-13T17:18:48.385 1: Starting Light exposure 4/5 for 5 seconds

We can see the next exposure start about 1.6 second before the previous image is saved.

Can you make this test with the RPi and a real camera?

han

19-07-13 21:12

reporter   ~0005758

Hello Patrick,

It looks like my Pi 4 is now broken. It doesn't boot anymore. :(

I will check out the new branch on the Pi 3 & my AMD desktop running win7 and compare it with the normal branch.
If would be great if this work reliable. I assume the remaining "lost time" is defined by the image download time. In your case 1.6 seconds since the the image exposure interval is 6.6 seconds.

Han

Patrick Chevalley

19-07-13 22:10

administrator   ~0005759

Too bad for the Pi4 :(

I just commit a fix to make it work with Ascom and Alpaca.

We cannot totally avoid some delay between the exposures, you cannot start a new one before the image is downloaded from the camera and copied a first time in CCDciel.
This is because the Ascom driver release the previous imagearray when a new exposure is started and with Indi you know the data finish to be transferred from the camera to the server only when you receive the image blob in the client.

han

19-07-13 23:51

reporter   ~0005760

Last edited: 19-07-13 23:51

View 2 revisions

Okay I got organised and did a simulation run on my desktop for a 4635 x 3520 image, saved to an ordinary hard disk.

- For the master version lost time preview 10.1 seconds, lost time normal exposure 10.5 seconds

- For the early exposure version lost time preview 8.7 seconds, lost time normal exposure 3.5 seconds

 I'm a little puzzled about the 10.5 seconds, it was in an older version 7.5 seconds, but the new test version shows performance!

THERE IS ONE MAYOR PROBLEM. For short exposures of 0.01 seconds, only small files (3 kb) with header only are written.

 Will test tomorrow live with my cameras.

Patrick Chevalley

19-07-14 08:00

administrator   ~0005761

Thank you Han.

OK, your number confirm it is worth the effort to continue in this way.

But sure this is not finished, the problem with short exposure is the image file get replaced or cleared by the next exposure before it is written to disk. I will add a mutex/criticalsection to solve this problem.
Also at the moment the last file of a capture sequence is not written, this must be easy to fix.
I will work on this tonight.

han

19-07-14 12:46

reporter   ~0005762

The results:

old desktop, win7, 4635 x 3520 image, lost time in preview mode 9.3 sec, lost time imaging 3.6 sec
old desktop, win7, 2328 x 1760 image, lost time in preview mode 3.3 sec, lost time imaging 1.3 sec

This is a significant improvement for short exposure times.

Raspberry Pi3, no repeat possible. Program stops after first image. Button indicated "stop" and green light is still there . Log:
2019-07-14T12:37:14.076 3: ZWO CCD ASI1600MM-Cool: progress: 0.9
2019-07-14T12:37:14.326 3: ZWO CCD ASI1600MM-Cool: progress: 0.6
2019-07-14T12:37:14.576 3: ZWO CCD ASI1600MM-Cool: progress: 0.3
2019-07-14T12:37:14.827 3: ZWO CCD ASI1600MM-Cool: progress: 0.1
2019-07-14T12:37:16.190 3: ZWO CCD ASI1600MM-Cool: progress: 0.0
2019-07-14T12:37:16.398 3: ZWO CCD ASI1600MM-Cool: load stream
2019-07-14T12:37:17.623 3: ZWO CCD ASI1600MM-Cool: write headers
2019-07-14T12:37:17.626 3: ZWO CCD ASI1600MM-Cool: apply correction
2019-07-14T12:37:17.626 3: ZWO CCD ASI1600MM-Cool: display image
2019-07-14T12:37:17.642 3: ZWO CCD ASI1600MM-Cool: progress: 0.0
2019-07-14T12:37:20.500 1: Saved file /tmp/test_20190714_103720.fits

Then the program waits for ever for something.

Patrick Chevalley

19-07-14 21:59

administrator   ~0005763

I make a few more change, it must be much more reliable now, it is protected again overlaping the previous image before it is saved.
For me it work with short or long exposure, with Indi, Ascom and Alpaca.
Now this is also activated for the preview loop.

For one week more I can only test on x86_64 with the simulator, so I really appreciate your testing.

han

19-07-15 10:41

reporter   ~0005764

Last edited: 19-07-15 10:45

View 2 revisions

Unfortunately the problem has not been solved for the Pi 3. Both imaging and preview stop doing anything after the first image. The preview last indication is Downloading... See screenshot.

I tried to use the debugger but it stops at a linkage error I have to investigate.

The master version is working fine after a compilation on the Pi 3.



problem pi3.png (225,143 bytes)
problem pi3.png (225,143 bytes)

Patrick Chevalley

19-07-15 11:52

administrator   ~0005765

Can you try on the Pi3 with the Indi ccd simulator?

I tested remotely with my arm32 and arm64 virtual machine and both the preview and the capture work with the simulator.

han

19-07-15 12:26

reporter   ~0005766

Same problem with CCD simulator.

Maybe should give you remote access to my Pi 3. :)

Strange, I can't run CCDCiel with the debugger. Get attached error.

error linkage.txt (5,784 bytes)
Compile Project, Mode: Default, Target: ccdciel: Exit code 1, Errors: 1, Warnings: 2, Hints: 17
fu_starprofile.pas(950,8) Hint: Converting the operands to "Int64" before doing the multiply could prevent overflow errors.
u_annotation.pas(304,78) Note: Local variable "test" not used
cu_astrometry_engine.pas(520,56) Hint: Conversion between ordinals and pointers is not portable
cu_astrometry_engine.pas(585,56) Hint: Conversion between ordinals and pointers is not portable
cu_astrometry_engine.pas(652,56) Hint: Conversion between ordinals and pointers is not portable
cu_astrometry_engine.pas(710,56) Hint: Conversion between ordinals and pointers is not portable
cu_astrometry_engine.pas(720,56) Hint: Conversion between ordinals and pointers is not portable
pu_edittargets.pas(1534,33) Hint: Local variable "p1" of a managed type does not seem to be initialized
pu_edittargets.pas(1535,33) Hint: Local variable "p2" of a managed type does not seem to be initialized
cu_plan.pas(28,40) Hint: Unit "math" not used in cu_plan
cu_ascomfocuser.pas(510,5) Note: Local variable "i" not used
cu_ascomrest.pas(321,28) Hint: Mixing signed expressions and longwords gives a 64bit result
cu_ascomrest.pas(321,28) Hint: Converting the operands to "Int64" before doing the multiply could prevent overflow errors.
cu_ascomrestcamera.pas(30,47) Hint: Unit "math" not used in cu_ascomrestcamera
pu_main.pas(1039,5) Note: Local variable "mi" not used
pu_main.pas(9898,28) Hint: Conversion between ordinals and pointers is not portable
pu_main.pas(9899,34) Hint: Conversion between ordinals and pointers is not portable
ccdciel.lpr(86,0) Warning: "crtbegin.o" not found, this will probably cause a linking failure
ccdciel.lpr(86,0) Warning: "crtend.o" not found, this will probably cause a linking failure
                                   Update contents of section <name> with
                                   contents found in <file>
     --dump-section <name>=<file>  Dump the contents of section <name> into <file>
     --rename-section <old>=<new>[,<flags>] Rename section <old> to <new>
     --long-section-names {enable|disable|keep}
                                   Handle long section names in Coff objects.
     --change-leading-char         Force output format's leading character style
     --remove-leading-char         Remove leading character from global symbols
     --reverse-bytes=<num>         Reverse <num> bytes at a time, in output sections with content
     --redefine-sym <old>=<new>    Redefine symbol name <old> to <new>
     --redefine-syms <file>        --redefine-sym for all symbol pairs 
                                     listed in <file>
     --srec-len <number>           Restrict the length of generated Srecords
     --srec-forceS3                Restrict the type of generated Srecords to S3
     --strip-symbols <file>        -N for all symbols listed in <file>
     --strip-unneeded-symbols <file>
                                   --strip-unneeded-symbol for all symbols listed
                                     in <file>
     --keep-symbols <file>         -K for all symbols listed in <file>
     --localize-symbols <file>     -L for all symbols listed in <file>
     --globalize-symbols <file>    --globalize-symbol for all in <file>
     --keep-global-symbols <file>  -G for all symbols listed in <file>
     --weaken-symbols <file>       -W for all symbols listed in <file>
     --add-symbol <name>=[<section>:]<value>[,<flags>]  Add a symbol
     --alt-machine-code <index>    Use the target's <index>'th alternative machine
     --writable-text               Mark the output text as writable
     --readonly-text               Make the output text write protected
     --pure                        Mark the output file as demand paged
     --impure                      Mark the output file as impure
     --prefix-symbols <prefix>     Add <prefix> to start of every symbol name
     --prefix-sections <prefix>    Add <prefix> to start of every section name
     --prefix-alloc-sections <prefix>
                                   Add <prefix> to start of every allocatable
                                     section name
     --file-alignment <num>        Set PE file alignment to <num>
     --heap <reserve>[,<commit>]   Set PE reserve/commit heap to <reserve>/
                                   <commit>
     --image-base <address>        Set PE image base to <address>
     --section-alignment <num>     Set PE section alignment to <num>
     --stack <reserve>[,<commit>]  Set PE reserve/commit stack to <reserve>/
                                   <commit>
     --subsystem <name>[:<version>]
                                   Set PE subsystem to <name> [& <version>]
     --compress-debug-sections[={none|zlib|zlib-gnu|zlib-gabi}]
                                   Compress DWARF debug sections using zlib
     --decompress-debug-sections   Decompress DWARF debug sections using zlib
     --elf-stt-common=[yes|no]     Generate ELF common symbols with STT_COMMON
                                     type
  -M  --merge-notes                Remove redundant entries in note sections
      --no-merge-notes             Do not attempt to remove redundant notes (default)
  -v --verbose                     List all object files modified
  @<file>                          Read options from <file>
  -V --version                     Display this program's version number
  -h --help                        Display this output
     --info                        List object formats & architectures supported
/usr/bin/objcopy: supported targets: elf32-littlearm elf32-littlearm-fdpic elf32-bigarm elf32-bigarm-fdpic elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex
ccdciel.lpr(86,0) Error: Error while linking
error linkage.txt (5,784 bytes)

Patrick Chevalley

19-07-15 13:24

administrator   ~0005767

There can be random linking error on the RPi if there is not enough swap file, be sure you have a least 1GB of swap file.

You can try the version I compile this morning:
ARMHF:
https://vega.ap-i.net/tmp/ccdciel_0.9.59-1656_armhf.deb
ARM64:
https://vega.ap-i.net/tmp/ccdciel_0.9.59-1656_arm64.deb

han

19-07-15 14:00

reporter   ~0005768

I have version 1656 installed but still the same problem occurs for preview and capture. Using CCD simulator, the only difference I see is that the download of the second image is completed, so the message Download.., disappears and the viewer image is updated. Then all activities stop but the program is full functional and looping can be stopped.

Note that I have currently INDI 1.62 installed, but that works for the master, so that shouldn't be a problem.
Simulator is also complaining about GSC stars missing but that shouldn't be a problem either.

I will increase the swap size. Where in the source code, I should place a break for the debugger?

han

19-07-15 14:36

reporter   ~0005769

Increased the swap size to 1024 and checked it with free -m This didn't help with link error.

However uncheck two option in debugger (Display line number... , Use external gdb debug ...) did help. no i can compile with debugger.

So I have CCDciel running in the debugger. Which source code lines I should check?

han

19-07-15 16:25

reporter   ~0005770

I did a check again using the Windows compiler. No problem in Windows with the repeat of the preview and capture function.

I have tried to trace the problem in the Pi 3. However the event driven execution is difficult to follow and I have problems to follow the execution. The first image is saved but the second save is never executed. In contradiction the master code runs fine in the Pi 3 so the problem must be in the code.

I could give you remote access to the Pi 3 with Teamviewer program.

Patrick Chevalley

19-07-15 21:36

administrator   ~0005771

This is a difficult case to debug because a breakpoint introduce a delay that totally change how the whole thing work.
A problem I have a few days ago with Ascom on Windows was "solved itself" after I add a few msg() to trace the execution.
I think the less intrusive method is to add simple writeln() and look at the console output.

A sensitive place is T_camera.TryNextExposure that start the next exposure but wait until CameraProcessingImage is false, it respawn itself using QueueAsyncCall if CameraProcessingImage is true.
You can add just after the begin of this procedure : writeln('TryNextExposure '+BoolToStr(CameraProcessingImage, True))
As your exposure time is long (60sec.) the value must be false on the first call.

CameraProcessingImage is set to false in the finally clause of Tf_main.CameraNewImageAsync. It is good if you can add a writeln to be sure Tf_main.CameraNewImage is called, then at the begin of Tf_main.CameraNewImageAsync and in the finally clause.

I am not at home now, I not use my computer, so difficult to install Teamviewer.
This is not urgent to solve as we are in a sandbox branch. If not solved in the mean time I can test with my Pi2 next week.

han

19-07-15 23:59

reporter   ~0005772

I found the reason why it doesn't work. It doesn't work with camera option "ram disk". but it works with option download via network= "INDI"

As usual the cause is very trivial. I'm glad I found it because your Pi 2 would probably have worked as expected.

Test result with option INDI selected:

 2328 x 1760 image, master version, lost time 8.5 seconds/image
 2328 x 1760 image, early exposure version, lost time 3 seconds/image

As far i can remember the ram disk option was about 10 to 15% faster in the master version.

Take your time to fix it. My Pi 4 is anyhow gone and needs replacement which could take some time. Secondly you have probably more urgent things to do.

Patrick Chevalley

19-07-16 08:17

administrator   ~0005773

There is always a good reason!
I need to fix that, the current code is in the network blob receive function but not in the ram disk file read. I also need to add it to the websocket file read if you compile Indi with this option.

Patrick Chevalley

19-07-17 11:44

administrator   ~0005774

This was a really simple fix:
https://github.com/pchev/ccdciel/commit/dc9d6e50fc41a10cd424fac872e05631944d5288

For me it work with RAM disk now.

han

19-07-17 16:44

reporter   ~0005775

It works also here. The ram-disk option is a little faster then INDI transpor.

Ram disk
 2328 x 1760 image, master version, lost time 7.6 seconds/image
 2328 x 1760 image, early exposure version, lost time 4 seconds/image

 4635 x 3520 image, master version, lost time 15 seconds/image
 4635 x 3520 image, early exposure version, lost time 8.5 seconds/image

The lost time varies but the improvement is 3.5 and 7.5 seconds for each exposure depending on the image dimensions. For 50 seconds exposures this is significant improvement.

Will test it tonight at a club member observatory.

Patrick Chevalley

19-07-17 21:33

administrator   ~0005778

Thank you for the test, this is in a good way.

Now I want to take the time to check the logic on a paper diagram to be sure there is no risk of image overwrite or deadlock, if OK I can merge the change.

han

19-07-23 01:08

reporter   ~0005784

This night the sky was clear and the Windows version worked fine for me. No problems.

Patrick Chevalley

19-07-23 09:26

administrator   ~0005787

I return home and also use this version on Linux last night, no problem at all.

One more change I do is to add an option in the camera preference to disable this new behavior in the case this make problem with some camera or if someone really need a sequential processing.

I have merged and removed the temporary branch, be sure you return to master and you can prune the obsolete branches.

Issue History

Date Modified Username Field Change
19-07-13 10:49 han New Issue
19-07-13 10:49 han File Added: Pi 3 vs Pi 4 .png
19-07-13 11:21 Patrick Chevalley Note Added: 0005755
19-07-13 13:37 Patrick Chevalley Note Added: 0005756
19-07-13 17:26 Patrick Chevalley Assigned To => Patrick Chevalley
19-07-13 17:26 Patrick Chevalley Status new => feedback
19-07-13 17:26 Patrick Chevalley Note Added: 0005757
19-07-13 21:12 han Note Added: 0005758
19-07-13 21:12 han Status feedback => assigned
19-07-13 22:10 Patrick Chevalley Note Added: 0005759
19-07-13 23:51 han Note Added: 0005760
19-07-13 23:51 han Note Edited: 0005760 View Revisions
19-07-14 08:00 Patrick Chevalley Note Added: 0005761
19-07-14 12:46 han Note Added: 0005762
19-07-14 21:59 Patrick Chevalley Note Added: 0005763
19-07-15 10:41 han File Added: problem pi3.png
19-07-15 10:41 han Note Added: 0005764
19-07-15 10:45 han Note Edited: 0005764 View Revisions
19-07-15 11:52 Patrick Chevalley Note Added: 0005765
19-07-15 12:26 han File Added: error linkage.txt
19-07-15 12:26 han Note Added: 0005766
19-07-15 13:24 Patrick Chevalley Note Added: 0005767
19-07-15 14:00 han Note Added: 0005768
19-07-15 14:36 han Note Added: 0005769
19-07-15 16:25 han Note Added: 0005770
19-07-15 21:36 Patrick Chevalley Note Added: 0005771
19-07-15 23:59 han Note Added: 0005772
19-07-16 08:17 Patrick Chevalley Note Added: 0005773
19-07-17 11:44 Patrick Chevalley Note Added: 0005774
19-07-17 16:44 han Note Added: 0005775
19-07-17 21:33 Patrick Chevalley Note Added: 0005778
19-07-23 01:08 han Note Added: 0005784
19-07-23 09:26 Patrick Chevalley Status assigned => resolved
19-07-23 09:26 Patrick Chevalley Resolution open => fixed
19-07-23 09:26 Patrick Chevalley Target Version => 1.0
19-07-23 09:26 Patrick Chevalley Note Added: 0005787