simplify libjpeg build system?

Index > High Level Languages > simplify libjpeg build system?

Author

Thread

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 12 Apr 2018, 07:18

Build systems for libjpeg (and for pretty much everything in c) are so confusing. They are made to support every processor, os, and compiler, and because of that they are nearly unreadable.

I want to take libjpeg-turbo-1.5.3 source, and delete everything unrelated to my system (32 bit windows + vs2005 compiler). Just so I can understand what is going on. I will use CMakeLists.txt for that.

I wonder if I can delete, for example, turbojpeg-mapfile.jni . What it is for, java?

12 Apr 2018, 07:18

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20289
Location: In your JS exploiting you and your system

revolution 12 Apr 2018, 07:57

vivik wrote:

Build systems for libjpeg (and for pretty much everything in c) are so confusing. They are made to support every processor, os, and compiler, and because of that they are nearly unreadable.

Only "nearly" unreadable? You mean someone can actually read them? Shocked

12 Apr 2018, 07:57

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 12 Apr 2018, 16:18

How I compile libjpeg and libpng.

I have folders:

Code:

C:/_src
C:/_build
C:/_install

I dropped libjpeg-turbo-1.5.3 and libpng-1.6.34 in _src.
I created same folders in _build, and moved in them in command line.
And used this command:

Code:

cmake -G"Visual Studio 8 2005" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=C:/_install C:/_src/libjpeg-turbo-1.5.3

(Probably can use -DCMAKE_BUILD_TYPE=Release or -DCMAKE_BUILD_TYPE=RelWithDebInfo instead.)

After that, I opened the generated <something>.sln file, compiled <something>-static project in it, and then "compiled" the INSTALL project.

Now I have some header files in the _install/include and some object files in _install/lib (don't forget to add those paths in "tools -- options -- projects and solutions -- directories"). I can use them in my programs by adding those lines in them:

Code:

#include <jpeglib.h>
#pragma comment(lib,"jpeg-static.lib")

#include <png.h>
#pragma comment(lib,"libpng16_staticd.lib")

#include <zlib.h>
#pragma comment(lib,"zlibstaticd.lib")

"d" stands for "debug". I will switch to "release" versions once I'll get comfortable with everything.

(my cmake --version is 3.10.2. A reminder to not install anything more modern, since they are threating to deprecate vs2005 support)

Last edited by vivik on 12 Apr 2018, 16:43; edited 1 time in total

12 Apr 2018, 16:18

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 12 Apr 2018, 16:33

There is this comment in c:/_src/libjpeg-turbo-1.5.3/CMakeLists.txt , not sure what it means.

Code:

# This does nothing except when using MinGW.  CMAKE_BUILD_TYPE has no meaning
# in Visual Studio, and it always defaults to Debug when using NMake.
if(NOT CMAKE_BUILD_TYPE)
  set(CMAKE_BUILD_TYPE Release)
endif()

message(STATUS "CMAKE_BUILD_TYPE = ${CMAKE_BUILD_TYPE}")

# This only works if building from the command line.  There is currently no way
# to set a variable's value based on the build type when using Visual Studio.
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
  set(BUILD "${BUILD}d")
endif()

12 Apr 2018, 16:33

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 12 Apr 2018, 16:58

One thing that will make my life easier, is making sure that my static libraries and my main program use the same version of runtime library. In visual studio it is specified in "c/c++ -- code generation -- runtime library". I've set it to "multi-threaded debug /MTd". I hope I could disable it entirely, but, not now.

To set it in cmake, I will need something like this:

Code:

if(MSVC)
  set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT")
  set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")
endif()

Or use set(CMAKE_CXX_FLAGS "something")

Extra info: https://stackoverflow.com/questions/14148374/how-do-you-set-properties-in-a-vcproj-file-with-cmake

Huh, there already is something related to that in c:/_src/libjpeg-turbo-1.5.3/CMakeLists.txt

Code:

if(MSVC)
  option(WITH_CRT_DLL
    "Link all libjpeg-turbo libraries and executables with the C run-time DLL (msvcr*.dll) instead of the static C run-time library (libcmt*.lib.)  The default is to use the C run-time DLL only with the libraries and executables that need it."
    FALSE)
  if(NOT WITH_CRT_DLL)
    # Use the static C library for all build types
    foreach(var CMAKE_C_FLAGS CMAKE_C_FLAGS_DEBUG CMAKE_C_FLAGS_RELEASE
      CMAKE_C_FLAGS_MINSIZEREL CMAKE_C_FLAGS_RELWITHDEBINFO)
      if(${var} MATCHES "/MD")
        string(REGEX REPLACE "/MD" "/MT" ${var} "${${var}}")
      endif()
    endforeach()
  endif()
  add_definitions(-W3 -wd4996)
endif()

Replaced code above with this, for now:

Code:

if(MSVC)
  set(CMAKE_C_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /MT")
  set(CMAKE_C_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")
  set(CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} /MT")
  add_definitions(-W3 -wd4996) #LATER: what is this?
endif()

12 Apr 2018, 16:58

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 12 Apr 2018, 17:49

Need to write CMakeLists.txt for giflib, and clean it up for libjpeg, libpng and zlib.

Look how each library uses "option" from cmake.

Reference: https://cmake.org/cmake/help/v3.10/command/option.html

>Provide an option for the user to select as ON or OFF. If no initial value is provided, OFF is used.

libjpeg:

Code:

option(WITH_SIMD "Include SIMD extensions" TRUE)

libpng:

Code:

option(PNG_BUILD_ZLIB "Custom zlib Location, else find_package is used" OFF)

zlib:

Code:

option(ASM686 "Enable building i686 assembly implementation")

...What do you like more, ON OFF, or TRUE FALSE?

I guess I'll go for TRUE FALSE, they are harder to mix up.

12 Apr 2018, 17:49

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 14 Apr 2018, 17:24

I'd like libjpeg to also add "d" prefix for its debug build, just like libpng and zlib. This way I'll never have to recompile them again.

I'll add this in CMakeLists.txt:

Code:

set(CMAKE_DEBUG_POSTFIX "d")

Also, I'll include those libraries like this:

Code:

// https://stackoverflow.com/questions/2290509/debug-vs-ndebug/29253284
#ifdef _DEBUG

#include <jpeglib.h>
#pragma comment(lib,"jpeg-staticd.lib")

#include <png.h>
#pragma comment(lib,"libpng16_staticd.lib")

#include <zlib.h>
#pragma comment(lib,"zlibstaticd.lib")

#else

#include <jpeglib.h>
#pragma comment(lib,"jpeg-static.lib")

#include <png.h>
#pragma comment(lib,"libpng16_static.lib")

#include <zlib.h>
#pragma comment(lib,"zlibstatic.lib")

#endif

14 Apr 2018, 17:24

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 14 Apr 2018, 17:44

Also, I'll add _build.bat into each c:/_build subdirectory, with content like this:

Code:

cmake -G"Visual Studio 8 2005" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=C:/_install C:/_src/libjpeg-turbo-1.5.3
pause

This way I can just double click it when I need to rerun cmake. And I'll need to do that often, since I'm changing the build scripts themselves.

14 Apr 2018, 17:44

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 28 Apr 2018, 20:35

How to get what compiler flags cmake uses: https://stackoverflow.com/questions/24767450/using-cmake-how-do-you-determine-the-default-compiler-flags-per-build-type

Code:

message("CMAKE_C_FLAGS_DEBUG is ${CMAKE_C_FLAGS_DEBUG}")
message("CMAKE_C_FLAGS_RELEASE is ${CMAKE_C_FLAGS_RELEASE}")
message("CMAKE_C_FLAGS_RELWITHDEBINFO is ${CMAKE_C_FLAGS_RELWITHDEBINFO}")
message("CMAKE_C_FLAGS_MINSIZEREL is ${CMAKE_C_FLAGS_MINSIZEREL}")

message("CMAKE_CXX_FLAGS_DEBUG is ${CMAKE_CXX_FLAGS_DEBUG}")
message("CMAKE_CXX_FLAGS_RELEASE is ${CMAKE_CXX_FLAGS_RELEASE}")
message("CMAKE_CXX_FLAGS_RELWITHDEBINFO is ${CMAKE_CXX_FLAGS_RELWITHDEBINFO}")
message("CMAKE_CXX_FLAGS_MINSIZEREL is ${CMAKE_CXX_FLAGS_MINSIZEREL}")

Apparently it's this for me:

Code:

1>CMAKE_C_FLAGS_DEBUG is /MDd /Zi /Ob0 /Od /RTC1 /MTd /ZI /Od
1>CMAKE_C_FLAGS_RELEASE is /MD /O2 /Ob2 /DNDEBUG /MT
1>CMAKE_C_FLAGS_RELWITHDEBINFO is /MD /Zi /O2 /Ob1 /DNDEBUG /MT
1>CMAKE_C_FLAGS_MINSIZEREL is /MD /O1 /Ob1 /DNDEBUG
1>CMAKE_CXX_FLAGS_DEBUG is 
1>CMAKE_CXX_FLAGS_RELEASE is 
1>CMAKE_CXX_FLAGS_RELWITHDEBINFO is 
1>CMAKE_CXX_FLAGS_MINSIZEREL is

Because I used this:

Code:

##MINE
if(MSVC)
  set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} /MT")
  set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} /MTd /ZI /Od")
  set(CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} /MT")
  add_definitions(-W3 -wd4996) #LATER: what is this?
endif()

instead of that regex:

Code:

if(MSVC)
  option(WITH_CRT_DLL
    "Link all libjpeg-turbo libraries and executables with the C run-time DLL (msvcr*.dll) instead of the static C run-time library (libcmt*.lib.)  The default is to use the C run-time DLL only with the libraries and executables that need it."
    FALSE)
  if(NOT WITH_CRT_DLL)
    # Use the static C library for all build types
    foreach(var CMAKE_C_FLAGS CMAKE_C_FLAGS_DEBUG CMAKE_C_FLAGS_RELEASE
      CMAKE_C_FLAGS_MINSIZEREL CMAKE_C_FLAGS_RELWITHDEBINFO)
      if(${var} MATCHES "/MD")
        string(REGEX REPLACE "/MD" "/MT" ${var} "${${var}}")
      endif()
    endforeach()
  endif()
  add_definitions(-W3 -wd4996)
endif()

I mean, there are two mutually exclusive options, "/MDd" and "/MTd". Where did the first one came from, somewhere from cmake cache? I should try deleting everything and rebuilding it, let's see if something changes. Edit: nothing changed. Regex that is then.

28 Apr 2018, 20:35

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 28 Apr 2018, 21:12

I'm not sure how to debug libjpeg.

debugging requires .pdb file, and it should be in the same directory as the static library. By default this file is named vc80.pdb, so a potential conflict when you have more than one static library to debug. I can change it by changing "properties -- C/C++ -- output files -- program database file name" to "$(IntDir)\$(ProjectName)_vc80.pdb", but I'm not sure if I can actually set those from cmake. I can tell cmake to add /Fd"turbojpeg-static.dir\Debug\turbojpeg-static_vc80.pdb" as a new console parameter, but I doubt I can use those $(ProjectName).

.pdb files include file paths, so debugger will find source files easily. But preprocessor highlighting wouldn't work properly. I guess it will work if I'll include the whole project into my solution, instead of just static library.

I guess that c:/_install folder was a silly idea, I need to use more than one project in one solution instead.

I have no idea what I'm doing.

28 Apr 2018, 21:12

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 30 Apr 2018, 10:41

Okay, this worked, I can run libjpeg in the debugger.

I inserted my own program's project into libjpeg solution. Rerunning cmake does update libjpeg projects, but it doesn't touch the solution itself, so I can do that safely.

I removed "#pragma comment(lib,"turbojpeg-staticd.lib")", and changed project properties:

Code:

C/C++ -- general -- additional include directories
C:\_build\libjpeg-turbo-1.5.3;C:\_src\libjpeg-turbo-1.5.3;

linker -- input -- additional dependencies
user32.lib C:\_build\libjpeg-turbo-1.5.3\debug\jpeg-staticd.lib

Also I rightclicked my main project, and "Set as startup project", and rightclick -- "Project dependencies" -- added turbojpeg-static, and also rightclick -- "Project build order" -- set my project as last one. Now everything gest neatly recompiled as necessary. For some reason all .asm files in the simd folder get recompiled far more often than necessary (on every little change in CMakeLists.txt), I don't know what's up with that.

Yeah, by the way, I can debug .asm files too. I changed this code in CMakeLists.txt

Code:

# This does nothing except when using MinGW.  CMAKE_BUILD_TYPE has no meaning
# in Visual Studio, and it always defaults to Debug when using NMake.
if(NOT CMAKE_BUILD_TYPE)
  # set(CMAKE_BUILD_TYPE Release)
  ##MINE
  set(CMAKE_BUILD_TYPE "Debug")
endif()

And since there already is this in the simd/CMakeLists.txt:

Code:

# This only works if building from the command line.  There is currently no way
# to set a variable's value based on the build type when using the MSVC IDE.
if(CMAKE_BUILD_TYPE STREQUAL "Debug"
  OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
  set(NAFLAGS ${NAFLAGS} -g)
endif()

all .obj files contain debug info. All to add that -g option.

30 Apr 2018, 10:41

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 30 Apr 2018, 11:03

This isn't related to build system, but.

libjpeg-turbo supports 3 algorithms for forward or inverse discrete cosine transform: fast integer, slow integer, and floating. According to https://libjpeg-turbo.org/About/SIMDCoverage and https://github.com/libjpeg-turbo/libjpeg-turbo , you use fast integer for most images, and slow integer/floating for images with quality 98 and above (or if you just need quality). I guess if I'll ever come across image like that, I'll reencode it to 75 quality or to png, maybe. (Just seeking for a code to throw out.)

EDIT: not sure about this though. There is this comment about TJFLAG_FASTDCT and TJFLAG_ACCURATEDCT in turbojpeg.h :

Quote:

libjpeg[-turbo] uses the fast
* algorithm by default when compressing, because this has been shown to have
* only a very slight effect on accuracy, but it uses the accurate algorithm
* when decompressing, because this has been shown to have a larger effect.

It's odd that this simplified turbojpeg api gives only 2 options for dct, not sure which one of them is which. EDIT2: fast is fast integer, accurate is slow integer. Float unused. See jsimd_x386.c, jsimd_idct_ifast and jsimd_idct_islow.

And about arithmetic coding. libjpeg supports using it instead of huffman coding, it will make files 10% smaller, but will make decoding 900% slower. According to this: http://www.rw-designer.com/entry/1311 , didn't check it myself. There is an option to remove arithmetic coding support from libjpeg by setting WITH_ARITH_ENC and WITH_ARITH_DEC to false, this will remove 3 .c files from the program. It's not the best what can be done, since there will still be arithmetic coding related code here and there, for example I seen this in jdmarker.c :

Code:

  for (i = 0; i < NUM_ARITH_TBLS; i++) {
    cinfo->arith_dc_L[i] = 0;
    cinfo->arith_dc_U[i] = 1;
    cinfo->arith_ac_K[i] = 5;
  }

Last edited by vivik on 01 May 2018, 17:19; edited 2 times in total

30 Apr 2018, 11:03

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 30 Apr 2018, 11:26

I need to understand how to downsample an image. There must already be code for that in libjpeg, since it does it with color during compression with 4:2:0 chroma subsampling, each 2x2 pixel block of color becomes just 1 pixel.

Maybe it's as simple as setting an output resolution somewhere.

Need to compress an image, and see where the code goes.

Heard somewhere that if you decompress an image with 4:2:0 chroma subsampling (there is also 4:4:4 chroma subsampling, which means color goes as is. There are a few other subsamplings), it will downsample luma (monochrome) down, but will use chroma as is. This should be faster.

Also sometimes you can upload raw YUV to the videocard texture, direct3d9 and direct3d11 supported that. And you can also use shaders for displaying separate luma and chroma textures as one. Too confusing right now for me. There is a mark in directx9 sdk about YUV textures being a legacy from PC98 computers.

30 Apr 2018, 11:26

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 30 Apr 2018, 11:40

About downsampling. Downsampling to 50% is simple, it means turning 4 pixels into 1, so, simply taking 25% of each. Downsampling to anything from 99.99% to 50.0001% is done by billinear filtering (other types of filtering are useful for games, billinear looks fine to me for now, I don't rotate anything), it uses something like 20% + 20% + 30% + 30% for each pixel, there are some unreadable formulas on wikipedia for that. Downsampling to anything beyond 50% requires doing one or more downsampling to exactly 50% first.

Troublesome points for me right now are: what if i need to downsample 3x4 image down? Should I downsample it to 1x2, or to 2x2? Should I just pretend that this image is actually 4x4, and each extra pixel is just black? Or should I use the more expensive billinear filtering here?

Also, can this be speed up with simd?

Should I downsample on cpu, or I somehow should downsample on gpu instead? I'm merely making an image viewer. There is a library for downsampling on cpu (something python related, and simd accelerated, https://github.com/uploadcare/pillow-simd , SSE4 AVX2, https://blog.uploadcare.com/the-fastest-production-ready-image-resize-out-there-part-0-7c974d520ad9 ), and there is a library for loading jpeg in videocard, with the help of shaders. https://www.youtube.com/watch?v=UsXyRwhnJXc ), and maybe there is a way to use "computation on gpu" somehow.

I'll make the simpliest way first, and will see if I'm fine with it.

Hm, here are a lot of benchmarks here... http://python-pillow.org/pillow-perf/

30 Apr 2018, 11:40

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 30 Apr 2018, 13:40

https://github.com/jcupitt/libvips/wiki/Why-is-libvips-quick

>The convolution operator, for example, will examine the matrix and the image and at run-time write a short SSE3 program to implement exactly that convolution on exactly that image.

oh wow

30 Apr 2018, 13:40

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 01 May 2018, 17:58

Yes, libjpeg-turbo can downsample images. It will even do simplified version of inverse discrete cosine transform (IDCT). It usually extracts each 8x8 block as is, but it can produce 4x4 2x2 1x1 blocks instead. Probably math is even simplier in that case, and those cases are simd accelerated. Proof: jsimd_can_idct_4x4 and jsimd_idct_4x4 in jsimd_x368.c, and jidctred.c

Need to play with different images now...

png and gif don't have capacity like that. Png needs every pixel in the buffer, because it depends on pixels around currenty decompressed one. And gif, in gif you need to first decompress pallete, and then create rgb image from that pallete. Also since it's animated, subframes may not lay as neatly on the 2x2 border, harder to downsample on the fly. For both png and gif, you decompress image as is, and downsample that as a separate step.

EDIT: Hm, libjpeg doesn't resize EXACTLY to the size I need, it's a bit smaller. Oh well. At least it's fast.

01 May 2018, 17:58

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 02 May 2018, 07:34

Actually no, [url=#203834]#203834[/url] is false,

>I inserted my own program's project into libjpeg solution. Rerunning cmake does update libjpeg projects, but it doesn't touch the solution itself, so I can do that safely.

cmake can delete projects from the solution. I just disabled simd in CMakeLists.txt , and this happened.

This means, I'll have to re-add my main project in the solution every time I'll change something in CMakeLists.txt .

I have to create my main project in CMakeLists.txt itself. Something like this, i guess?

Code:

add_executable(_main--ddraw main.cpp)
set_property(TARGET _main--ddraw PROPERTY RUNTIME_OUTPUT_DIRECTORY "c:/_main--ddraw3")

...but in different order, i guess...

02 May 2018, 07:34

vivik

Joined: 29 Oct 2016
Posts: 671

vivik 04 May 2018, 10:07

How to read CMakeLists.txt files.

The most important things in it are: add_subdirectory , add_executable , add_library . If you are lost, go straight for those 3.

04 May 2018, 10:07

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum