any example showing how to use the wrapper?

[ttsesm]

@sampotter I would like to use the embree lib for raycasting since it provides a nice accelerated solution. However, I am not quite sure how to use your wrapper. Would be possible to write a small example showing some use case. For example considering that someone has the indices and faces of a mesh object in numpy arrays how to create the corresponding mesh in embree by using this wrapper?

[sampotter]

This wrapper attempts to mimic the Embree C API closely. So, if you want to figure out how to do something, my goal is for a user to be able to consult the Embree API first.

That said, this wrapper is in a prototype stage and hasn't been used or tested thoroughly yet.

Here's some code that I'm using currently for raytracing, somewhat out of context:

device = embree.Device()
geometry = device.make_geometry(embree.GeometryType.Triangle)
scene = device.make_scene()

vertex_buffer = geometry.set_new_buffer(
    embree.BufferType.Vertex, # buf_type
    0, # slot
    embree.Format.Float3, # fmt
    3*np.dtype('float32').itemsize, # byte_stride
    V.shape[0], # item_count
)
vertex_buffer[:] = V[:]

index_buffer = geometry.set_new_buffer(
    embree.BufferType.Index, # buf_type
    0, # slot
    embree.Format.Uint3, # fmt
    3*np.dtype('uint32').itemsize, # byte_stride,
    F.shape[0]
)
index_buffer[:] = F[:]

geometry.commit()
geom_id = scene.attach_geometry(geometry)
geometry.release()

scene.commit()

n0, p0 = N[i0], P[i0]
dP = P - p0

mask = np.logical_and(dP@n0 >= 0, np.sum(dP*N, axis=1) <= 0)
P_mask = P[mask]

M = sum(mask)

rayhit = embree.RayHit1M(M)
rayhit.tnear[:] = 0
rayhit.tfar[:] = np.inf
rayhit.prim_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.geom_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.org[:] = p0 + eps*n0
rayhit.dir[:] = P_mask - p0

context = embree.IntersectContext()

scene.intersect1M(context, rayhit)

H = np.zeros((P.shape[0],), dtype=bool)
I_mask = np.where(mask)[0]
H[I_mask] = rayhit.prim_id == I_mask

In this example, I'm shooting some rays from perturbed triangle centroids to other triangles and seeing which rays actually hit their target triangles. In this case, I've loaded an OBJ using another Python library (doesn't really matter which), and then used Embree's API to tell it about the mesh's vertices and faces.

If you aren't familiar with Embree, if you consult the Embree API, you'll find that the code above matches what is done in Embree's provided examples.

Please let me know if you run into any trouble, and feel free to post code and mesh files here.

Please note: the entire API isn't wrapped yet, but should be very easy to wrap more functions. If a function from the Embree API that you need is missing, please say so. You can check embree.pyx to see if a function has been wrapped.

[ttsesm]

@sampotter thanks for the prompt response and the example. However, just before we go to code examples I would like to clarify the installation procedure. How someone should install the wrapper?

I tried to run pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" . from my virtual environment where C:\Program Files\Intel\Embree3 is the path to my Embree3 installation but it failed giving me some errors (I can post the terminal output if needed), thus I guess there is something I am missing.

[sampotter]

No problem.

Can you tell me a little more about your setup? I see that you're using Windows, which should work in principle, but is less familiar to me. Are you using Anaconda?

At this point, I haven't tested this wrapper on Windows, so we'll need to figure out an appropriate workflow for getting it installed.

And yes, please post the build output.

[ttsesm]

Yes, currently I am on a windows machine. I know that usually with this kind of things in linux is more straight forward. However, in principle someone should be able to install the wrapper on both systems.

Anyways back to the point, in my project I am using Pycharm where I have set both a conda and a virtualenv environment, the pip command above I've run it from my virtualenv environment.

As you correctly figured out I am on a windows 10 machine, with Python 3.8, and I have installed Embree v3 by using the windows installer.

This is the output from the build:

>>pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .
c:\users\ttsesm\desktop\dev\radiosity\r_venv\lib\site-packages\pip\_internal\commands\install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Processing c:\users\ttsesm\desktop\dev\python-embree
Skipping wheel build for embree, due to binaries being disabled for it.
Installing collected packages: embree
    Running setup.py install for embree ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"'; __file_
_='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"
'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.txt' --single-version-externally-ma
naged --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree'
         cwd: C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\
    Complete output (14 lines):
    running build_ext
    building 'embree' extension
    creating build
    creating build\temp.win-amd64-3.8
    creating build\temp.win-amd64-3.8\Release
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/opt/local/include "-IC:\Program Files\Intel\Embree3\include" -Ic:\users\ttsesm\desktop\dev\radiosity\r_venv\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Com
munity\VC\Tools\MSVC\14.26.28801\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x
86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\
10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcembree.c /Fobuild\temp.win-amd64-3.8\Release\embree.obj
    embree.c
    embree.c(9654): warning C4013: 'posix_memalign' undefined; assuming extern returning int
    creating C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\build\lib.win-amd64-3.8
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:/opt/local/lib "/LIBPATH:C:\Program F
iles\Intel\Embree3\lib" /LIBPATH:c:\users\ttsesm\desktop\dev\radiosity\r_venv\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38 /LIBPAT
H:c:\users\ttsesm\desktop\dev\radiosity\r_venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio
\2019\Community\VC\Tools\MSVC\14.26.28801\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Wind
ows Kits\10\lib\10.0.18362.0\um\x64" embree3.lib /EXPORT:PyInit_embree build\temp.win-amd64-3.8\Release\embree.obj /OUT:build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.li
b
       Creating library build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.exp
    embree.obj : error LNK2001: unresolved external symbol posix_memalign
    build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build
-dtrlbuol\\setup.py'"'"'; __file__='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close(
);exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.tx
t' --single-version-externally-managed --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree' Check the logs for full command output.

[sampotter]

How did you install Embree?

[ttsesm]

Do you think that it might have to do with the Embree installation which is linked against the Visual Studio 2013 (VC12) and Visual Studio 2015/2017 (VC14) runtime?

I installed it by using the .msi windows installer from here https://www.embree.org/downloads.html

[sampotter]

No, your installation doesn't appear to be a problem.

It can't find posix_memalign, which makes sense. It looks like I can use _aligned_malloc instead on Windows. I'll need to try building this on Windows, which should be pretty easy for me right now.

If you want to try debugging this yourself, look for instances of posix_memalign in embree.pyx and see if you can replace them with calls to _aligned_malloc. It will be necessary to add an aligned malloc that's platform independent, where the appropriate function calls are selected using preprocessor macros.

[ttsesm]

Actually it seems that it is not a direct replacement, since the set of input parameters are different. I found this stackoverflow thread https://stackoverflow.com/questions/33696092/whats-the-correct-replacement-for-posix-memalign-in-windows where they discussing the same issue and the provided solution seems to do the trick.

[sampotter]

That's what I was looking at, as well. :-)

To provide some context, when using Embree, things that are allocated on the heap need to be aligned appropriately for SIMD. This is explained pretty clearly in the Embree API, which I linked to in my first reply.

Regardless of the exact API of the function, what's needed is a malloc that can align allocated memory to the right byte boundary for use with SIMD.

[ttsesm]

@sampotter do you think that you could use the C's malloc, realloc and free or the Python's C-API memory allocation functions instead of the posix one.

Based on this https://tianrunhe.wordpress.com/2012/04/23/aligned-malloc-in-c/ it seems that aligned_malloc can be done with calls to malloc.

Here are some other reference links:

https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html

https://github.com/EveryTimeIWill18/Cython_Repo/blob/master/Cython_Tutorial_Three.ipynb?utm_campaign=News&utm_medium=Community&utm_source=DataCamp.com

https://github.com/linux-rdma/rdma-core/blob/master/pyverbs/mem_alloc.pyx

I couldn't find any way to use calls to _aligned_malloc, which is valid for windows.

I also found the following link which could be usefull:

numpy/numpy#5312

[sampotter]

Hey @ttsesm, I just pushed a commit that should fix this. I switched from posix_memalign to aligned_alloc on POSIX, and _aligned_malloc on Windows, which has the same API. I was able to compile without any trouble on Windows but haven't tested this. Let me know how this works.

[sampotter]

Hmm, thought I had fixed this, but my changed cause some problems on macOS, which I wasn't able to figure out (in the small amount of time I budgeted for it).

I created a new branch off of develop (aligned_allocation) where we can try to figure this out. The latest commit there is working on Mac, but haven't had a chance to try it on Windows or Linux yet.

[ttsesm]

Thanks @sampotter, I've managed to make it compile in Windows (I do not have access to a linux machine currently). I had to make the following change though:

if errno == EINVAL:
     raise Exception('bad alignment')
elif errno == ENOMEM:

in line

python-embree/embree.pyx

Line 19 in 802b5f4

if errno == ENOMEM:

otherwise it was complaining in line

python-embree/embree.pyx

Line 803 in 802b5f4

if errno == EINVAL:

that EINVAL was not declared.

[sampotter]

Thanks for catching that, and glad you were able to get it to compile. If you put together a pull request I'll be happy to accept it.