View Issue Details

IDProjectCategoryView StatusLast Update
0001971CCdciel[All Projects] Generalpublic18-10-10 15:46
ReporterhanAssigned ToPatrick Chevalley 
PrioritynormalSeveritytweakReproducibilityhave not tried
Status resolvedResolutionfixed 
Product Version 
Target Version1.0Fixed in Version 
Summary0001971: HFD, FWHM calculation of faint stars, sigma factor used to measure the star flux/intensity.
DescriptionI'm currently playing with photometry in ASTAP and testing the code of the HFD routine which is identical in CCDCIEL. While doing so, I notice that for faint stars too much star flux is ignored. The program recognize star flux if the pixel value is 5 * sigma above the background. This looks now too rigid. The HDF value of very faint stars is therefore reported a little low. A sigma value of 3 look more appropriate but the detection box size has also an influence. A large box will allow some noise to be counted as flux.

95,45% is within 2 * sigma
99,73% is withing 3 * sigma
99,9937% is within 4 * sigma
99,999943% is within 5 * sigma

The 5*sigma is used twice in the CCDCIEL procedure TFits.GetHFD and TFits.GetHFD2 for

 - Find the center of gravity
- Calculated the HFD

Reducing the sigma from 5 to 3 in the HFD calculation will result in a more accurate HFD for faint stars. For the first step "find center of gravity", the size of the box counts. Maybe the sigma factor should be made dependent on the box size. Something I have to think about and experiment more before I would propose a change.

So I just wanted to drop the idea and will experiment further before proposing a change.

TagsNo tags attached.

Activities

han

18-08-24 21:41

reporter   ~0004828

Still studying the HFD routine. I did a CCDCiel field test with the improved code where signal is anything > 3* sigma. This worked fine. However I'm also considering an other change. The second test on "signal is anything > 3* sigma" can be removed. This will improve the HFD calculation of faint stars. The noise will be averaged and the star flux below 3* sigma will be included. After applying this second change the HFD values reported by the CCD inspector are more equal for bright and faint stars.

In a few days, I will attach the improved code including some additional comments including a description of the HFD calculation method.

Remove line 1863
  if snr>3 then
  begin
    for i:=-ri to ri do
      for j:=-ri to ri do
      begin
        Val:=value_subpixel(xc+i,yc+j)-bg;
////// if val>3*bg_standard_deviation then {>3 * sd should be signal }
        begin
          r:=sqrt(i*i+j*j);{distance for gravity center}
          SumVal:=SumVal+Val;{sumval will be star total flux value}
          SumValR:=SumValR+Val*r;
          if val>=valmax*0.5 then pixel_counter:=pixel_counter+1;{how many pixels are above half maximum for FWHM}
        end;
      end;

han

18-08-26 22:45

reporter   ~0004829

The above proposed changes have been tested. They work fine both in CCDciel and ASTAP. Attached an updated cu_fits.pas file with the changes included and extensive documentation in the code.

If I make an image of the same stars with 10,20 or,200 seconds the CCDinspector HFD stays almost the same. At 10 or 20 seocnds it is at 1.5. At 200 seconds is is 1.7. I assume the noise has still some influence.

I have studied the current algorithm again. The calculation assumes that the HFD line splits the star in equal portions of gravity. It is an approximation with a maximum of 6.4% error. Ths idea I picked up from Kazuhisa Miyashita.

While doing so, I made test program using a super large oversampled (5000x5000 pixel) Gaussian star. No need to look at.

How did we get here:
1) HFD was proposed by Harry Weber and Steve Brady See page 10 at http://www.ccdware.com/Files/ITS%20Paper.pdf
2) Kazuhisa Miyashita proposed the approximation formula to using the center of gravity calculation around the HFD line for video recordings. http://www005.upp.so-net.ne.jp/k_miyash/occ02/halffluxdiameter/halffluxdiameter_en.html
3) We added the sub-pixel method.

I wrote an article about HFD for Wikipedia which is currently in draft.
https://en.wikipedia.org/wiki/Draft:Half_flux_diameter

In the test program a more accurate method with a histogram is used but this is more CPU intensive and I don't think it is required. It would require to make small steps of maybe 0.05 pixel.

This is extra documentation in the cu_fits.pas code:

      {==========Notes on HFD calculation method=================
        https://en.wikipedia.org/wiki/Half_flux_diameter Currently in draft
        http://www005.upp.so-net.ne.jp/k_miyash/occ02/halffluxdiameter/halffluxdiameter_en.html by Kazuhisa Miyashita. No sub-pixel calculation
        https://www.lost-infinity.com/night-sky-image-processing-part-6-measuring-the-half-flux-diameter-hfd-of-a-star-a-simple-c-implementation/
        http://www.ccdware.com/Files/ITS%20Paper.pdf See page 10, HFD Measurement Algorithm

        HFD, Half Flux Diameter is defined as: The diameter of circle where total flux value of pixels inside it is equal the outside pixel's.
        HFR, half flux radius:=0.5*HFD
        The pixel_flux:=pixel_value - background.

        The approximation routine assumes that the HFD line divides the star in equal portions of gravity:
            sum(pixel_flux * (distance_from_the_centroid - HFR))=0
        This can be rewritten as
           sum(pixel_flux * distance_from_the_centroid) - sum(pixel_values * (HFR))=0
           or
           HFR:=sum(pixel_flux * distance_from_the_centroid))/sum(pixel_flux)
           HFD:=2*HFR

        This is not an exact method but a very efficient routine. Numerical checking with an a highly oversampled artificial Gaussian shaped star indicates the following:

        Perfect two dimensional Gaussian shape with σ=1: Numerical HFD=2.3548*σ Approximation 2.5066, an offset of +6.4%
        Homogeneous disk of a single value : Numerical HFD:=disk_diameter/sqrt(2) Approximation disk_diameter/1.5, an offset of -6.1%

        The approximation routine is robust and efficient.

        Since the number of pixels illuminated is small and the calculated center of star gravity is not at the center of an pixel, above summation should be calculated on sub-pixel level (as used here)
        or the image should be re-sampled to a higher resolution.

        A sufficient signal to noise is required to have valid HFD value due to background noise.

        Note that for perfect Gaussian shape both the HFD and FWHM are at the same 2.3548 σ.
        }


       {=============Notes on FWHM:=====================
          1) Determine the background level by the averaging the boarder pixels.
          2) Calculate the standard deviation of the background.

              Signal is anything 3 * standard deviation above background

          3) Determine the maximum signal level of region of interest.
          4) Count pixels which are equal or above half maximum level.
          5) Use the pixel count as area and calculate the diameter of that area as diameter:=2 *sqrt(count/pi).}

cu_fits.zip (16,204 bytes)
HFD algorithm test.zip (131,937 bytes)

han

18-09-06 22:30

reporter  

star_test_image.zip (8,120,375 bytes)
modified CCDCIEL.png (985,054 bytes)
modified CCDCIEL.png (985,054 bytes)
current CCDCIEL.png (837,377 bytes)
current CCDCIEL.png (837,377 bytes)
ASTAP hfd values.png (1,756,846 bytes)

han

18-09-06 22:30

reporter   ~0004833

Last edited: 18-09-10 22:39

View 3 revisions

Today, I made routine to produce perfect Gaussian shaped stars of different intensity against a noise background. See attached FITS file.

Attached three screen shots.

1 First screenshot shows the test image with the current CCDCIEL version. All artifical stars have the same HFD value but you can see that the HFD decrease if the SNR becomes lower. So this is not good.
2) Second is a screenshot of ASTAP. Showing a consistent HFD for each SNR value. The HFD become noisy but in average the values are correct.
3) Screenshot of a modified CCDCIEL. The HFD values are more noisy then ASTAP. The reason is that the test window is larger then in ASTAP. I have fixed it now by increasing the detection level in the center of gravity routine depending on the box size

The artificial image shows the following:
Artificial image, background has value 1000 with sigma 100 Gaussian noise
Top rows contain hotpixels of value 65535'
Rows below have stars all with the same HFD value of different intensity.



cu_fits version 2018-9-6.zip (16,205 bytes)

han

18-09-11 00:26

reporter  

ccdciel version 2018-9-18.zip (16,682,625 bytes)

han

18-09-11 00:26

reporter   ~0004834

Last edited: 18-09-11 00:27

View 2 revisions

More small proposed improvements in cu_fits.pas and pu_main.pas. The HFD value is now pretty consistent for low and high SNR star and the selected star detection size,

Also included two new improved FITS test files. I added the keywords datamax and datamin for correct readout in CCDCIEL.

I will do a field test the next clear night.



han

18-09-16 22:15

reporter   ~0004841

The routine is now reporting in all cases the correct HFD, but the image inspection is too sensitive and just show too many stars. I will have a look into that next week.

Patrick Chevalley

18-09-18 14:29

administrator   ~0004854

This look very good, thank you!

I will look at the code after I finish to fix the other bug.

han

18-09-25 15:51

reporter   ~0004886

I have made the last simple update to CU_fits. Replaced in TFits.GetStarList the minimum treshold by (snr>AutofocusMinSNR){minum SNR value} This reduces the number of stars to a reasonable amount and removes the very faint star which will have a less reliable HFD value.

I didn't further update attached pu_main.pas since the master pu_main.pas has been recently been updated. At the moment I can't remember what I did to pu_main.pas 3 weeks ago. Too many other complicated problems where on my mind asking my full attention. I assume you can find out with version control.

The two attached HFD test fits files where created by ASTAP. Creating HFD test images is now a standard tool in the pixel math tab of ASTAP stacking menu.

ps. the ASTAP solver can now do blind solving :) With the last magnitude control loop it it very fast

han

18-09-25 15:53

reporter  

ccdciel version 2018-9-25.zip (16,682,643 bytes)

Patrick Chevalley

18-09-25 19:08

administrator   ~0004887

Thank you Han.

I tested your change with my images and it work much better than the previous version.

I commit your new version in git.
Can you get this new version as a base if you want to make other changes.
https://github.com/pchev/ccdciel/commit/d701bfac749f4ddbb8108ed445f7a4277ff66578


About pu_main the only change you do is replacing the measurement window size at two places:
s:=min(max(16,round(3*med)),starwindow div fits.HeaderInfo.BinX);
by
s:=min(max(12,round(6*med)),starwindow div fits.HeaderInfo.BinX);

In pu_main I also set a default value AutofocusMinSNR=3 for the case people use the image inspection without configuring the auto-focus.

In cu_fits I revert a change you do to set the scaling factor to 1 for 64bit/pixel fits files. Any reason for that?
This is near line 1090:
 case FFitsInfo.bitpix of
      -64 : begin
- if Fdmax>Fdmin then
- c:=MaxWord/(Fdmax-Fdmin)
- else
+ // if Fdmax>Fdmin then
+ // c:=MaxWord/(Fdmax-Fdmin)
+ // else
              c:=1;

han

18-09-25 19:22

reporter   ~0004888

Hello Patrick,

There was a good reason for the pu_main window size.

Yes your correction/revert in cu_fits, line 1090 is correct. My change should have been temporary to understand what was happening but I forgot to remove the //

Will download the latest code next time.

Patrick Chevalley

18-10-10 15:46

administrator   ~0004944

I think this work very well now with the new code and this issue can be closed.

Issue History

Date Modified Username Field Change
18-08-16 13:43 han New Issue
18-08-24 21:41 han Note Added: 0004828
18-08-26 22:45 han File Added: cu_fits.zip
18-08-26 22:45 han File Added: HFD algorithm test.zip
18-08-26 22:45 han Note Added: 0004829
18-09-06 22:30 han File Added: current CCDCIEL.png
18-09-06 22:30 han File Added: ASTAP hfd values.png
18-09-06 22:30 han File Added: modified CCDCIEL.png
18-09-06 22:30 han File Added: star_test_image.zip
18-09-06 22:30 han File Added: cu_fits version 2018-9-6.zip
18-09-06 22:30 han Note Added: 0004833
18-09-06 22:35 han Note Edited: 0004833 View Revisions
18-09-10 22:39 han Note Edited: 0004833 View Revisions
18-09-11 00:26 han File Added: ccdciel version 2018-9-18.zip
18-09-11 00:26 han File Added: CCDCIEL version 2018-9-10.png
18-09-11 00:26 han Note Added: 0004834
18-09-11 00:27 han Note Edited: 0004834 View Revisions
18-09-16 22:15 han Note Added: 0004841
18-09-18 14:29 Patrick Chevalley Assigned To => Patrick Chevalley
18-09-18 14:29 Patrick Chevalley Status new => assigned
18-09-18 14:29 Patrick Chevalley Target Version => 1.0
18-09-18 14:29 Patrick Chevalley Note Added: 0004854
18-09-25 15:51 han Note Added: 0004886
18-09-25 15:53 han File Added: ccdciel version 2018-9-25.zip
18-09-25 19:08 Patrick Chevalley Note Added: 0004887
18-09-25 19:22 han Note Added: 0004888
18-10-10 15:46 Patrick Chevalley Status assigned => resolved
18-10-10 15:46 Patrick Chevalley Resolution open => fixed
18-10-10 15:46 Patrick Chevalley Note Added: 0004944