AVX2 depth buffer check and 32-bit to 24-bit data conversion

Introduction

I have been tinkering with my C pseudo REYES software renderer again lately, i especially worked on some SIMD optimizations to compose multiple render buffers into a single buffer with a depth check, the code heavily use Intel x86 / x86-64 AVX2 instructions which works on 256 bits data, in my case it is able to process 8 pixels at a time "branchless" which greatly enhance the performance of my software renderer.

The idea was to replace this compositing loop (depth buffer is a series of float values and i/o buffer are a series of bytes) :

by an optimized version which works on 8 pixels at a time, my idea was to use AVX2 compare instruction (vcmpps) which produce a mask and give that mask to a data output instruction (vpmaskmovd) which is able to output selectively based on the mask value.

My buffers can be 32-bit or 24-bit so i needed to make the code works for both type of data, the 32-bit code took me an evening to figure out, the 24-bit version took two days and proved to be harder because some AVX2 instructions are limited such as vpshufb and there is no mask store instructions for 8 bits data, all of this were added in AVX-512 alas my CPU (i7 7600) doesn't handle that kind of instruction set.

32-bit (4 bpp; RGBA) C code

// to use Intel Intrinsics with GCC : #include <x86intrin.h>
const int vec_step = 8;
const int loop_step = bpp * vec_step;
const __m256i mask_24bpp = _mm256_set_epi8(-1, -1, -1, -1, -1, -1, -1, -1, 28, 28, 28, 24, 24, 24, 20, 20, 20, 16, 16, 16, 12, 12, 12, 8, 8, 8, 4, 4, 4, 0, 0, 0);
for (int i = 0; i < len; i += loop_step) {
    // compare z value
    __m256 ymm0 = _mm256_loadu_ps((const mfloat_t *)input_depth_buffer);
    __m256 ymm1 = _mm256_loadu_ps((const mfloat_t *)output_depth_buffer);
    __m256 ymm2 = _mm256_cmp_ps(ymm0, ymm1, _CMP_GT_OS);

    // copy input buffer following mask value
    __m256i ymm3 = _mm256_loadu_si256((const __m256i_u *)input_buffer);
    _mm256_maskstore_epi32((int *)output_buffer, _mm256_castps_si256(ymm2), ymm3);

    // copy depth following mask value
    _mm256_maskstore_ps(output_depth_buffer, _mm256_castps_si256(ymm2), ymm0);

    input_depth_buffer += vec_step;
    output_depth_buffer += vec_step;

    input_buffer += loop_step;
    output_buffer += loop_step;
}

24-bit (3 bpp; RGB) C code