Picking

Picking - What did I just click?

While you know exactly what object or element is being drawn to the framebuffer during rendering, a little more complicated thing is to know what element or object a pixel in the final render belongs to. Imagine having rendered a set of points and lines which are displayed in your GUI. Now you click on one of the points in your GUI and want to get some more detailed information about it or highlight it. How do you know what you just clicked?

There are different approaches to deal with this, you could for example back transform the mouse coordinates from screen space into 'world' or 'object' space and find the elements that contain the click coordinates. However this can become a quite complicated task depending on the number of elements and the complexity of the displayed objects.

A simpler technique, that JPlotter supports out of the box, is called picking. In this technique every element that is drawn can have a custom color assigned to it with which it is being drawn. Using the color of a pixel you can lookup what object or element it corresponds to since you assigned the color earlier. Of course this picking color is not the same color that you want the element to appear in, for example you want to have 20 green lines and still be able to differentiate between them, which is why the element is rendered twice. Once with the actual color you want it to appear in, and again with the picking color that serves as an ID for the element. The picking color image is never shown and only used to look up IDs from pixels. Fortunately this type of double rendering can be done very efficiently in one go by OpenGL so that it does not affect rendering performance at all.

visible render	picking color render

In the table above you can see how the picking color rendering looks like compared to the normal rendering. It can be seen that only the points of the scatter plot are rendered in picking color which is due to all other elements (e.g. the coordinate system lines) have transparent a transparent picking color. By default every element's picking color is transparent. It may seem as if most of the elements in the picking rendering are of the same color, but in fact each point is colored differently (the colors range from 0x000001 to 0x000096 in hexadecimal RGB).

Getting the pixel value

In order to get the color of a pixel within an FBOCanvas you can use the method

getPixel(int x, int y, boolean picking, int areaSize).

This method lets you specify if you want to know the picking color or the actual color as well as an area size. The area size parameter is used to read pixels in a whole area around the specified (x,y) location and the method will then return the most prominent color in that area. This allows for a bit of inaccuracy when dealing with mouse inputs, since the user may have clicked just a few pixels off target.

Assigning picking color

In JPlotter there are 4 major Renderable classes from which a visualization is built, i.e. Text, Points, Lines and Triangles.

• Text
  • use the method setPickColor(int pickColor)
• Points, Lines, Triangles
  • these are collections of either PointDetails, SegmentDetails or TriangleDetails which all feature the method setPickColor(int pickColor)
  • The *Deatils objects are returned upon adding stuff, e.g. public PointDetails addPoint(double x, double y) or public ArrayList<SegmentDetails> addLineStrip(Point2D...points)

The picking colors are specified as integers which are interpreted as an RGB value of 24 bits with 8 bits for each channel. This kind of integer is referred to as integer packed ARGB color value in the javadocs and looks like this:

Alpha	Red	Green	Blue
bits 31..24	bits 23..16	bits 15..8	bits 7..0

So the opaque color red is 0xffff0000 as hexadecimal integer, green is 0xff00ff00, black is 0xff000000 and invisible is 0x0 = 0. Transparent colors are not allowed for picking since they would blend during rendering when overlapping and result in a new color which is why you dont have to specify the alpha channel when using the setPickColor(int pickColor) methods. This allows you to use the integers within [0x1 .. 0xffffff] ( in decimal [1 .. 16777215]) as values for picking colors and 0 as the value for no color assigned since that's completely transparent, which leaves you with roughly 16.7 million color IDs.

Since generating and managing these integer IDs together with their mappings is a bit tedious, the PickingRegistry class can be used to generate unique picking colors and remember a corresponding object for each. When doing so you don't need to think about all these bits and stuff from 10 seconds ago, sorry for even mentioning it.