What is the good approach to sort a buffer that contains bitmap pixels by one of the color channels?

Let's suppose I have 4 pixels in format BBGGRRAA:
A01020FF C0C0C0FF 406080FF 7AE321FF

and arranged by green channel should be
A01020FF 406080FF C0C0C0FF 7AE321FF

When I say a good approach I am thinking about an easy implementation also not just fast execution and memory used. I think I am looking for a balanced solution between.

Choose sorting algorithm you like most and instead of directly comparing the values compare the same values but bitwise-anded with appropriate mask.

I would myself use a bitwise algorithm, where you first arrange the list so that items having the topmost bit 0 are before the items having the highest bit 1, and then for each of these two sub-lists you repeat the process, but looking at the bit below, and so on.To arrange the list this way, you can scan from the beginning of the list until you reach an element with the highest bit set, and from the top of the list until you reach an element with highest bit zeroed, and if the former is before the latter in the list, you swap them and continue scanning.

In your case you'd be looking at the highest bit of the green channel, and continue (perhaps recursively) to cover all 8 bits. So at first you'd be testing bit 800000h, during the next step you would be testing 400000h, and so on, ending on the 010000h bit.

For assembly programming I have always been finding this algorithm very natural and easy, in fact in my early days of programming I had been using it without even realizing that it is simply a special case of Radix sort that one may learn during formal education.

Note, however, that this algorithm is not stable, so if you also need stability, you might need to consider another one.