Photoshop :: Multi Channel Color Separation?

For a long time I have been struggling to find a good workflow for what I would call "multi channel color separation". What I want is to reduce a full color RGB image into an image with only a limited number (e.g. 5-8) well-chosen colors, for instance to use as set of masks for screen print. (I'd also like to use this as an effect in itself, mimicking a typical 'screen print' or 'litho' effect on an inkjet print)

The idea is to create a set of monochrome layers or spot color channels. Colors are to be selected by myself (so not necessarily RGB or CMYK). Each channel/layer has to be a continuous tone that I transform myself into a dedicated half tone screen (e.g. with the Andromeda screen filter).

Till sofar it is clear to me. My issue is: How to separate the image in the predefined set of colors?

I can not just select a color range: I can adjust a "tolerance" but that is too general. I need the pixels of with a specific hue but with the full range of lightness (e.g. all the purple pixels that range from dark/medium purple to very light purple). The dark colors can be made dark with a black channel. I'm not sure how to deal with the saturation range.

An approach that gives sometimes a result that is to some extent acceptable is to transfer from RGB mode to an Indexed Color mode and use a custom palette. However, this does not give much control on the halftone screening process.

I envision for instance an approach of filling a layer with a solid given color, select in one way or another only those parts that correspond to the original image and subtract that layer somehow from the image. This process is then to be repeated for each color. Up till now I have not been able to figure the right options to do this. But maybe there is a better way to do it.

One final wish: it would be great if the process can take into account that some layers in the final print are (semi-)transparent. In my opinion this would exclude just selecting a color and deleting de pixels.