I'm trying to learn USB linux device drivers to implement a project idea I have:

Primary device is a Rasp Pi with USB port

Secondary device is a custom USB stick that I'm going to design with some LEDs and buttons. It will have some kind of either Microcontroller, another Rasp Pi, or a USB controller IC based on how I need to implement the USB gadget driver.

How should the USB host and gadget drivers be implemented? The gadget driver needs to be implemented on the USB stick or can it also be implemented in the primary device, such that I can drive the LED output and read the buttons input from USB stick on the primary device?