In general, when the hydraulic fluids flow continuously between the pump and the actuator (normally motor) without going to the reservoir, then the system is called as closed loop hydraulic system.
On the other hand, if the fluid from the actuator (normally cylinder) first goes to the reservoir and then the fresh fluid from the reservoir flow to the pump inlet, then the system is called the open loop circuit.
Please refer the above picture and note that, for open loop system, a directional control valve is used for setting the direction of movement of the actuator.
In closed loop system, one additional pump is used for making up the circuit fluid. And the direction of the direction of the movement of the actuator is controlled by the swash plate of the variable displacement pump.
The open loop hydraulic system has advantage of less heat generation and on the other hand the closed loop circuit is preferred for better (precise) response of the actuation.