View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002225 | CCdciel | General | public | 19-11-10 12:51 | 19-11-13 08:44 |
| Reporter | han | Assigned To | Patrick Chevalley | ||
| Priority | normal | Severity | minor | Reproducibility | have not tried |
| Status | resolved | Resolution | fixed | ||
| Target Version | 1.0 | ||||
| Summary | 0002225: Support for new raw format .CR3 | ||||
| Description | There is a new raw .CR3 format. Samples are available at: https://www.photographyblog.com/reviews/canon_eos_m50_review/sample_images (scroll further down) The new extension is not supported in the CCDCiel file mask. Loading the file using a *.* file mask gives an error "Invalid file C:\temp\test\IMG_3119.CR3" . I tried also the new dcraw.exe provided using the command line. This new dcraw v0.9.28 converts the new format but the output is only black. According the dcraw Github page the CR3 format should be supported but it doesn't work. I assume you used the Github code to compile this new dcraw version. (I failed to do that) If so is it time to report this issue at the dcraw github page or do you have a solution? Han | ||||
| Tags | No tags attached. | ||||
|
|
Hello Han, I was working on that this morning :-) For CCDCiel, it use dcraw only as a last resort if it cannot do otherwise. The preferred way to decode the raw files is LibRaw. This as the advantage to not require temporary files, this make the decoding twice as fast on my computer. Because LibRaw is C++ and header are continuously changing with every version it is not possible to use directly from Pascal. So I make a simple interface library : https://github.com/pchev/ccdciel/tree/master/library/raw On Windows this library is linked statically to LibRaw so no other dll are required. On Linux it is dynamically linked to use the distribution LibRaw. The library is loaded by this unit: https://github.com/pchev/ccdciel/blob/master/src/u_libraw.pas Use is in cu_fits procedure RawToFits() : https://github.com/pchev/ccdciel/blob/master/src/cu_fits.pas#L2438 So this morning I upgrade the Windows build to use the new LibRaw snapshot with CR3 support, and add *.cr3 in the open dialog filters. You can try this change: https://vega.ap-i.net/tmp/ccdciel-0.9.64-1856-windows-x32.exe https://vega.ap-i.net/tmp/ccdciel-0.9.64-1856-windows-x64.exe For Linux this will work as soon the distribution packages are upgraded to the new version of LibRaw. |
|
|
I will try libraw. I was also interested in compiling dcraw for astap since I had only version 0.9.27 and was struggling to get the latest version compiled. I assume you used the latest code, so I will report my dcraw problems with CR3 format. The problem with dcraw is that it can't produce 16 bit fits files. I use 16 bit pgm as an intermediate format. Not efficient but that is my current setup. Maybe the libraw "dcraw_emu.exe" is a solution for me. I prefer command line interfaces. This dcraw_emu.exe will call the dll. Not an elegant solution. |
|
|
libraw doesn't support cr3 yet. https://www.libraw.org/node/2424 |
|
|
I have compiled the last version of dcraw for Windows from the last code available at https://www.dechifro.org/dcraw/ I can send you a .exe , do you need 32 or 64 bit or both? Maybe using the command line from LibRaw is a better solution, because I am not sure if dcraw development is still active. dcraw_emu.exe do not have the -D option for raw output. You have to use unprocessed_raw.exe instead. When using the command line I not use pgm output because Lazarus do not have a reader for this format, I use the -T option to output a 16bit tiff file. I just check it work with a cr3 files. I can also send you a compiled unprocessed_raw.exe 32 and 64bit. |
|
|
Yes, there is LibRaw support for CR3 but this is very new: https://www.libraw.org/news/libraw-snapshot-201910 You need master branch from libraw github: https://github.com/LibRaw/LibRaw |
|
|
Yes a copy dcraw.exe would be nice. :) If no effort send me both 32 and 64 bit version. Maybe the 64 bit version is faster. I assume the DCRAW development has slowed down or stopped. But if for the moment CR3 is supported there is no need to change. I use PGM because I did not have a 16 bit TIFF reader in the beginning. I will check if TIFF is more efficient using the FPimage units. Reading PGM is very easy, The code is in ASTAP, function load_PPM_PGM. |
|
|
Here is the latest dcraw 9.28 for win32 and win64. From my testing it accept a .cr3 file but the output is all black. So this format is still not supported by dcraw. |
|
|
This is the libraw utilities dcraw_emu.exe and unprocessed_raw.exe for win32 and win64. This work fine to read .cr3 files. They are statically linked, no dll is required. The compilation command are: make -f Makefile.mingw CXX=i686-w64-mingw32-g++ CC=i686-w64-mingw32-gcc LDADD="-static-libgcc -static-libstdc++" make -f Makefile.mingw CXX=x86_64-w64-mingw32-g++ CC=x86_64-w64-mingw32-gcc LDADD="-static-libgcc -static-libstdc++" dcraw_emu.exe as the option to produce a debayered image. unprocessed_raw.exe allow to get the raw data. |
|
|
Hello Patrick. The unprocessed_raw.exe works fastest and works for cr3 as expected. I get the fastest conversion using PGM as intermediate format. For reading TIFF, I use fpreadTIFF. If I remember well, the Lazarus unit for PGM, PPM doesn't support 16 bit, but reading PGM is easy. Conversion to FITS of 22x CR2 files using an intermediate format: dcraw.exe 32 bit 31 seconds (via PGM) dcraw.exe 64 bit 26 seconds (via PGM) unprocessed_raw.exe 20 seconds (via PGM) unprocessed_raw.exe 27 seconds (via TIFF) Han |
|
|
Does it make sense to adapt my Pascal function to read PGM to an array for CCDCiel? |
|
|
I not think pgm read is necessary in ccdciel. We use the raw conversion a bit differently, in astap you already have the raw files on disk to process them, it make sens to call dcraw or similar and read the pgm as this is faster. But in ccdciel the main use is when we receive a raw image from indi. In this case there is no file on disk, all is in memory. This is why I prefer to call libraw directly, this avoid to write the raw file on disk, have dcraw to read this file and write a pgm, then ccdciel to read the pgm. This can make a big difference on computer that have low file i/o performance like the RPi. |
|
|
I solved the loading of my libraw library on Mac, so now the Mac version also process CR3 files. I removed dcraw from my source because it is no more necessary to build it for CCDciel. Only for Linux we have to wait the new LibRaw version is packaged by the Linux distribution, CCDciel will process CR3 files without change when this is done. If you want to use unprocessed_raw on Linux beware it is packaged separately from LibRaw itself. On Debian/Ubuntu the package is libraw-bin, it install unprocessed_raw in /usr/lib/libraw/ , so not in the default PATH. Maybe you can add your own static version with CR3 support in /opt/astap/. |
|
|
Thanks Patrick, For ASTAP Windows, I added the compiled version you provided. That is now CR3 capable. For Linux, ASTAP will look for "unprocessed_raw" at the location you indicated. If not found it will show a popup suggesting to install libraw-bin. So CR3 will be available as soon this library is updated by the distribution or user. For the macOS version I, have added the older "unprocessed_raw" to the ASTAP app. So that will work with CR3 as soon i have an update "unprocessed_raw". So libraw is now fully implemented for ASTAP, that was a good move. I will wait for libraw updates which will probably come soon. I assume CCDCiel will save CR3 as fits so there is less urgency for me to support that. :) |
|
|
Yes, CCDciel always save the files as FITS even when they are received as raw from the camera driver. The reason is to save the same metadata in the FITS header, I not want specificity everywhere in the program to manage exif metadata. I attach here the Mac 64bit version of the latest unprocessed_raw, it work with CR3. This is very simple to build: git clone https://github.com/LibRaw/LibRaw.git cd LibRaw make -f Makefile.dist result is in bin directory and not depend on any specific library |
|
|
Just for reference, I tested to rebuild the last LibRaw from git to test CR3 loading from CCDciel on Linux. This work fine. This is NOT recommended for real use because you end with two incompatible version of libraw.so, the standard one in /usr/lib/x86_64-linux-gnu/ and one in /usr/local/lib/ because it is not possible to remove the libraw package without removing every graphic software on the system. But this is likely to make this software to crash. git clone https://github.com/LibRaw/LibRaw.git cd LibRaw autoreconf --install ./configure --prefix=/usr/local make sudo make install git clone https://github.com/pchev/ccdciel.git cd ccdciel/library/raw g++ -DLIBRAW_NOTHREADS -O3 -fPIC -w -I/usr/local/include -shared -Wl,-soname,libpasraw.so.1 -o libpasraw.so.1.1 pasraw.cpp -L/usr/local/lib -lraw -lm sudo cp libpasraw.so.1.1 /usr/lib/ sudo ldconfig |
|
|
Patrick, Thanks for al this work. I have used your macOS binary to update the ASTAP macOS version. The Linux instructions, i will keep in reserve for the case a user needs CR3 conversion. Han |
|
|
I think we are OK with raw CR3 now. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 19-11-10 12:51 | han | New Issue | |
| 19-11-10 14:43 | Patrick Chevalley | Assigned To | => Patrick Chevalley |
| 19-11-10 14:43 | Patrick Chevalley | Status | new => assigned |
| 19-11-10 14:43 | Patrick Chevalley | Target Version | => 1.0 |
| 19-11-10 14:43 | Patrick Chevalley | Note Added: 0006050 | |
| 19-11-10 14:59 | han | Note Added: 0006051 | |
| 19-11-10 15:30 | han | Note Added: 0006052 | |
| 19-11-10 15:43 | Patrick Chevalley | Note Added: 0006053 | |
| 19-11-10 15:46 | Patrick Chevalley | Note Added: 0006054 | |
| 19-11-10 17:21 | han | Note Added: 0006057 | |
| 19-11-10 18:12 | Patrick Chevalley | File Added: dcraw32.zip | |
| 19-11-10 18:12 | Patrick Chevalley | File Added: dcraw64.zip | |
| 19-11-10 18:12 | Patrick Chevalley | Note Added: 0006058 | |
| 19-11-10 18:18 | Patrick Chevalley | File Added: dcraw-emu-win32.zip | |
| 19-11-10 18:18 | Patrick Chevalley | File Added: dcraw-emu-win64.zip | |
| 19-11-10 18:18 | Patrick Chevalley | Note Added: 0006061 | |
| 19-11-10 23:10 | han | Note Added: 0006062 | |
| 19-11-10 23:49 | han | Note Added: 0006063 | |
| 19-11-12 12:54 | Patrick Chevalley | Note Added: 0006065 | |
| 19-11-12 20:16 | Patrick Chevalley | Note Added: 0006068 | |
| 19-11-12 20:58 | han | Note Added: 0006069 | |
| 19-11-12 22:23 | Patrick Chevalley | File Added: unprocessed_raw_mac.zip | |
| 19-11-12 22:23 | Patrick Chevalley | Note Added: 0006071 | |
| 19-11-12 23:33 | Patrick Chevalley | Note Added: 0006073 | |
| 19-11-13 08:40 | han | Note Added: 0006074 | |
| 19-11-13 08:44 | Patrick Chevalley | Status | assigned => resolved |
| 19-11-13 08:44 | Patrick Chevalley | Resolution | open => fixed |
| 19-11-13 08:44 | Patrick Chevalley | Note Added: 0006075 |
