I am happy to inform you that a new version of Ab4d.SharpEngine, the cross-platform 3D rendering engine, was published today.
This version was actually planned to be released in November or December. However, due to the recent resolution of critical issues and the implementation of exciting new features that our users requested, I decided to publish a new official version sooner than planned.
Significant improvements have been made to improve the stability and reliability of the engine. Numerous new tests have been created to check these aspects. Some issues were resolved by modifying the Ab4d.SharpEngine, but there were also some issues that were due to bugs in the Vulkan validation layers or Intel drivers. To check the stability of the engine by yourself, one test is now available to all users. If you start the new samples for Avalonia, WPF or WinUI, then you will find a new "Advanced options" section in the Settings window. Among other options, there is also a new "Show samples TEST runner" CheckBox. When checked, a new TEST button will appear. If clicked, it starts showing random samples as quickly as possible (on ApplicationIdle). You can use that to see how long the engine can run.
This version also brings some major new features. The most interesting is support for rendering 3D models from scanned 2D images from MRI or CT scanners. See the following screenshots:
The 3D model is rendered by using the new VolumeMaterialEffect that can show the 3D model from any camera angle, despite the slices being made from only one direction (one slice image is shown in the upper right corner of the screenshot). Based on the values from the scanned slices, it is possible to assign different colors and transparencies to different parts of the model - in this sample, this is used to distinguish between hard and soft tissue.
To show the 3D model, first, a 3D texture is generated from all of the scanned 2D images. Then the new CubeSlicer class is used to generate a mesh that slices a 3D cube from the camera's position to the other end of the 3D cube. Then this mesh is rendered by using the new VolumeMaterial that samples the value from the 3D texture (values from the scanned images) and converts the values into colors by using a transfer function texture (for each value in the scanned image, a color is assigned).
The sample can also be used to show how the 3D slices are generated. This can be best observed by yourself by starting the new sample and selecting "Show slice lines" or "Show slices mesh" options. You can also quickly see that by watching the following video.
The code from the sample can be used as a generic method for displaying 3D models from scanned 2D images. However, if you require any special settings or tweaks, please let me know, and I will try to adjust the code accordingly.
Another great new feature is support for super-fast hit-testing of complex meshes by using octree structures. This feature does not look that interesting, but it can be very useful for many of the users of the Ab4d.SharpEngine.
Usually, when performing hit testing, the engine generates a 3D ray from the mouse (or pointer) position to the 3D scene. Then the engine checks which SceneNodes may be intersected by the ray (by checking the bounding box). For those SceneNodes, the ray intersection is checked for each triangle of the SceneNode. When showing complex meshes with millions of triangles, this process can be very slow.
Octree structure helps solve that by organizing the triangles into a multi-level structure where each level divides the parent's 3D space into eight areas (or OctreeNode objects). The first level contains only one area, the second level contains eight areas, the third level 64 areas and the fourth level 512 areas (it is possible to define more than four levels; this is recommended for very complex meshes). Each of those areas has its own bounding box. So when doing hit testing, the area's bounding boxes are used to determine which areas are intersected by the ray. Then only the triangles in those areas are tested for intersection. This may reduce the number of checks by many thousands.
The following screenshot from the new sample shows how each level divides the space into 8 areas (see the bottom right corner to see how many triangles are in each of the levels)
The MeshOctree object can further optimize the distribution of the triangles by slightly expanding the areas so more triangles fall into lower areas. The following screenshot shows the distribution of triangles after that optimization (note that more triangles are distributed in the lower levels):
For example, the unoptimized image shows that the first level contains 728 triangles. Those triangles are always tested for intersection with the ray. But in the optimized versions, the first and the second levels have no triangles. To see the full distribution of triangles, the Octree sample prints the "Octree statistics" to the Visual Studio Output window. Here is a shortened part of that (the numbers in the list show how many triangles are in each area or OctreeNode):
Level 1: Triangles count: 0
Level 2: Triangles count: 0
Level 3: Triangles count: 113: 0, 0, 0, 0, 0, 7, 0, 0, ... (and other values)
Level 4: Triangles count: 3983: 4, 11, 52, 12, 8, 4, 12, ... (and other values)
Total triangles count: 4096
So, for example, if the ray intersects only the bounding box of the first area in the fourth level, then only 4 triangle-ray intersections are performed (11 intersection checks are done when only the second area is hit). Now, compared to 4096 checks that would need to be performed when no octree is used.
But using an octree is not free. For complex meshes, generating all the required OctreeNodes may take some time.
By default, the MeshOctree object is generated when hit-testing a mesh that has more than 512 positions. This is defined by the HitTestOptions.MeshPositionsCountForOctreeGeneration property. You can increase this value or set it to int.MaxValue to prevent automatic octree generation. If your SceneNode is not supposed to be hit-testable, you can also disable octree generation by setting the SceneNode.IsHitTestVisible property to false. You can also manually generate the octree by calling the CreateOctree method on the mesh object (do not forget to assign the created MeshOctree to the Octree property).
In the HitTestOptions class, there are also some other new properties that define the global settings for octree generation (for example, you can set the max level count and generate the octree at mesh initialization instead of hit-test time).
So, the octree structure can provide a huge hit-testing performance boost. However, if you do not need that and you experience a longer initialization time, you can disable the octree generation.
The last new feature that I wanted to mention was also developed for one of the customers and is also related to hit-testing. This customer is showing a mesh with millions of triangles and he needs to know which triangle is hit by the mouse. Usually, using OctTree would be perfect for that scenario, but he is changing the mesh (smoothing) on each frame. This means that on each frame, the MeshOctree needs to be regenerated. This was very slow and significantly impacted the frame rate.
The solution was to render the scene to an ID bitmap where each triangle would get its own color. So instead of regenerating the MeshOctree, the frame only needs to be rendered again and then the index of the triangle can be determined by the pixel color under the mouse. This is much faster and allows the customer to perform real-time smoothing of the triangles under the mouse.
The following screenshot shows the new sample that demonstrates the triangle hit-testing that uses the ID bitmap (the mouse is not shown, but the triangle under the mouse is shown by a red polyline):
This version also brings some other new features. As always, to see the list of all the changes and fixes, check the Ab4d.SharpEngine change log.
I know that at this time of the year, many of you have already been expecting to be able to try the first version of Ab4d.SharpEngine for the web browser. But because of the issues in the core Ab43.SharpEngine library, and because some time was spent implementing the new features that are mentioned here, the web version is not yet ready for the first public version.
However, a lot of work was still done for the web version. Therefore, I am very happy to announce that today I have published the web page where you can check how the web version of the Ab4d.SharpEngine works. You can check it in the Ab4d.SharpEngine for web pre-alpha demo page.
Here is also a screenshot:
This demo shows that the engine already supports rendering meshes with standard material (with or without a specular effect). It also supports hit testing, rendering 3D lines and camera manipulation with mouse or touch (on mobile devices).
When the demo page is first opened, it takes a few seconds until the page is loaded (most time is spent downloading and compiling the self-contained .NET WASM file). But on subsequent page loads, the 3D scene is shown immediately.
You can also test how the engine works when rendering many SceneNodes. With each click on the "Add many objects" 1000 BoxModelNode objects are added to the scene. You can observe the rendering times by opening the DevTools (F12) and check the log messages in the Console tab. You will see that even though the engine is running in the browser, it can be very fast - on my computer, it is rendering more than 5000 objects in around 7 ms (rendering individual objects and not using mesh instancing).
As seen from this demo, many of the major features are already implemented. The code just needs a little bit of additional work. Also, documentation about the web project setup and a quick usage guide need to be written. I think that this can be completed before the end of this month. Then a new NuGet package and the source code for the sample project will be published. This will also allow you to test the engine with your own web projects.
Some of you were asking about the license of the new Ab4d.SharpEngine for the web. The good news is that there will continue to be only one license for desktop, mobile and web platforms. This means that the existing customer will be able to start using the new web version without any additional costs.