I found another post referring to this that helped me out. I don't think you can call the endpoint for object categories. You have to send a call to the project level and manually update form order.
the order of the custom forms can be found under the endpoint OBJCAT (Object Categories).
As you can see in the API-Explorer, the corresponding object (e.g. project) is saved in the field objID and the order as an integer in the field categoryOrder.
As there seems to be no Misc Action, I would try to use the Custom API Call module sending a PUT request to update this.
I found another post referring to this that helped me out. I don't think you can call the endpoint for object categories. You have to send a call to the project level and manually update form order.