A windows procedure must have the following signature:
LRESULT CALLBACK MyWndProc(HWND, UINT, WPARAM, LPARAM);
A dialog procedure must have the following signature:
INT_PTR CALLBACK MyDlgProc(HWND, UINT, WPARAM, LPARAM);
If you need to cast your windows or dialog procedure then you should double-check the signature. If it is correct, there should be no need to cast unless you are attempting to put it into a LONG_PTR
.
In x64, INT_PTR
and LRESULT
are essentially the same; but in Win32 with /Wp64
, they are treated differently.
Along the same lines as DLSetWindowLongPtr
and GetWindowLongPtr
(in maxsdk
\include\3dsmaxport.h), there are also DLGetWindowProc
and SetWindowProc
. DLGetWindowLongPtr
and SetWindowLongPtr
also have overloaded versions that will accept a WNDPROC
properly. If you want to put a WNDPROC
into GWLP_USERDATA
, you can explicitly specify it, otherwise it is routed to GWLP_WNDPROC
.
In versions of 3ds Max previous to version 9 there were cases where windows procedures return INT_PTR
instead of LRESULT
, and these were being routed quietly to GWLP_USERDATA
in Win32 builds. This changed the pointer to MyClass
* with a pointer to a function which is undesirable. Since 3ds Max there is an overloaded version which accepts this signature and routes it to GWLP_WNDPROC
as well.